How you can make the network yente work for you Need to find the IP address of a network node? Let ARP be your matchmaker

March  1997

This month, we'll give you the network handy-person's guide to ARP, starting with an overview of how the protocol fits into the TCP/IP world. We'll look at how Solaris manages the ARP protocol and associated caches, and see how ARP deals with routers and other devices that don't pass along its gentle probes for data. We'll conclude with a list of symptoms and fixes for common ARP-related woes. (2,800 words)

Mail this article to a friend

In the Broadway show and movie "Fiddler On The Roof," five young women turn to Yente, the arranger of marriages in their small Russian town. Yente's job is that of the matchmaker, finding the best fit for a young lady, or at least a job close enough to call complete. Are we crass enough to take this slice of classic show business and turn it into a network analogy? Right -- of course right!

Whether you realize it or not, the address resolution protocol (ARP) is hard at work constantly matching IP addresses to their Ethernet hardware addresses on your network. When ARP is working well, packets flow in both directions, across routers and bridges without a moment's notice. If you're suffering from ARP difficulties, you'll see intermittent grand pauses in network traffic, reductions in usable network bandwidth, and generally poor performance for various network applications.

Like Yente, ARP has to deal with liars, two- and three-timers, and other less-than-trusty network citizens like those middlemen we usually call routers. Figuring out who's telling the truth, who needs a sharp reprimand, and what's going wrong is, of course, your job. Right? Of course right!

A many-to-one splendored thing: Mapping IP addresses
First of all, it's permissable to sound out the protocol acronym, rhyming it with "harp," even if you do sound like Network the dog with a bit error in his bark. ARP can also be used as a verb as in, "Who is arping for that address?" It's not part of the IP protocol per se but instead is a peer protocol that is nearly always included in an implementation of TCP/IP. ARP is described in gory detail in RFC 826.

ARP's job appears simple on the surface: it finds the hardware or MAC (media access control) address corresponding to a given IP address. ARP builds up relations of 32-bit IP addresses into the 48-bit MAC address space, maintaining these many-to-one mappings on a per-host basis. Each machine that runs TCP/IP needs to track the mappings constructed up by ARP traffic, building its own map of the local network. The fact that ARP management represents the ultimate in distributed state is what also makes it prone to strange failure modes.

A bit of background on MAC addresses is in order before explaining ARP's tasks. A hardware, or MAC, address is assigned per host - one per system, no matter how many network interfaces it has. The MAC address uniquely identifies the system in the "universe" of Ethernet devices. MAC addresses are written as a series of 6 colon-separated hex values, for example, 8:0:20:74:b:6. The upper 24 bits are reserved for a vendor ID and the lower 24 bits represent a unique sequence number within that vendor's space. You'll find all of your Sun systems have MAC addresses that start with 8:0:20. RFC XXX, Assigned Numbers, provides a listing of MAC address prefixes and vendors, allowing you to match up a network packet and the flavor of machine it came from. You'll also see that Sun assigns one MAC address per system, burned into the ID or EEPROM of the system, a fact that startles most people adding additional network interfaces.

The IP-MAC relationship is N:1 when a given host has IP addresses on multiple network interfaces but uses the default MAC address on each one. For the most part, a given MAC address maps back to one IP address per network -- so a network client can maintain a 1:1 table of IP addresses and associated hardware addresses. Looking at it another way, IP addresses are logical conventions, assigned by system administrators to make the TCP/IP protocol work. MAC addresses are physical conventions, assigned by the vendor; ARP's job is to bridge the logical world onto the physical one.

Who are you? ARP in action
Here's how a typical ARP request and reply play out:

• Our network node huey needs to send an Ethernet packet to server duey. It knows duey's IP address, having located it via DNS, but has no idea how to find it on the network. Ethernet frames use MAC addresses to specific source and destination, so huey needs duey's MAC address to fill in the Ethernet frame header and zip the packet on its way.

• Looking in its routing table, huey sees that duey is on the same network, so it consults the cache of already-known IP:MAC address mappings. Unfortunately, duey isn't listed, so huey has to shout for help.

• huey sends an ARP broadcast, asking the entire local network for the MAC address corresponding to duey's IP address. The ARP broadcast is sent to MAC address ff:ff:ff:ff:ff:ff, the Ethernet broadcast address. duey receives the ARP request, recognizes its own IP address in the request header, and returns its MAC address to huey. Note that the replies are unicast - they go from ARP target back to the requester, not to the whole network. duey simply fills in the missing IP information and reverses sender and receiver fields in the Ethernet frame header; an attempt to ARP huey back would result in a deadlock.

• huey receives the ARP reply, updates its local cache of ARP information, and sends the packet on to huey. Future conversations with huey will use the cached MAC address and avoid the round-trip ARP conversation.

• If duey is smart, it will have cached huey's MAC address from the initial ARP request, so that IP traffic from duey back to huey doesn't require an ARP in the reverse direction. Solaris 2.x caches information from ARP requests to eliminate some of the ARP traffic.

Not all network protocol suites require ARP: DECnet uses an algorithm to convert DECnet area:node addresses into Ethernet MAC addresses. This requires the DECnet protocol to change the local MAC address to match the DECnet configuration, a change that can confuse bridges if DECnet is enabled after the node has already spoken TCP/IP with its neighbors. ARP traffic may seem like an unreasonable bit of overhead added to the startup of each conversation with a new host. It's the price paid for flexibility -- you can assign IP addresses as you like to your hosts, even re-using the same addresses on disjoint networks, without upsetting the inner workings of the Ethernet one bit.

Digging data out of ARP
ARP maintains its cache of IP:MAC mappings on each machine. Dump it out using arp -a:

luey% arp -a
Net to Media Table
Device   IP Address               Mask      Flags   Phys Addr 
------ -------------------- --------------- ----- ---------------
le0    duey                 255.255.255.255       08:00:20:1d:0b:92
le0    129.151.128.250      255.255.255.255 U     
le0    huey                 255.255.255.255 SP    08:00:20:21:15:13
le0    fred                 255.255.255.255       08:00:20:1a:f2:02
le0    224.0.0.0            240.0.0.0       SM    01:00:5e:00:00:00

You can get a slightly better report from ndd:

luey% ndd -get /dev/arp arp_cache_report
ifname   proto addr      proto mask      hardware addr     flags
le0      196.9.8.177 255.255.255.255 08:00:20:1f:34:9b
le0      196.9.8.185 255.255.255.255 08:00:20:04:cd:1e
le0      196.9.8.005 255.255.255.255 08:00:20:18:1c:e5
le0      196.9.8.012 255.255.255.255 08:00:20:11:27:af
le0      196.9.8.011 255.255.255.255 08:00:20:7a:5c:2c
le0      196.9.8.049 255.255.255.255 08:00:20:21:15:13 PERM PUBLISH
le0      196.9.8.059 255.255.255.255 08:00:20:1a:f2:02
le0      196.9.8.099 255.255.255.255 08:00:20:1e:26:5c
le0      224.000.000.000 240.000.000.000 01:00:5e:00:00:00 PERM MAPPING

Completed entries represent queries that returned at least one reply. Those marked "U" or "UNRESOLVED" are from queries that were never answered, because the IP address is not on the local network or because the host owning it was down or not listening at the moment the request was sent.

Entries marked "S" (static) or "PERM" will never be removed from the cache; typically these are inserted on behalf of devices (like printers) that have limited TCP/IP implementations and don't answer their ARP requests, or to work around network configuration quirks. Create a permanent entry using arp as root:

luey# arp -s 196.9.8.250 06:1:4:bc:a:34 

A "published" entry is one for which the local host acts as a proxy server. When the local host sees an ARP request come by for that IP address, it answers with the MAC address in the cache, not its own MAC address. Publishing ARP entries allows one machine to answer ARP requests for machines that can't hear the broadcasts because they're on the far side of a router, or because they're not listening. Published entries are marked with a "P" or "PUBLISH" in the arp cache report. Load ARP information from a table using arp -f to publish several ARP addresses at once.

Using permanent entries prevents the local machine from sending ARP requests; creating a published entry allows the local machine to answer ARP requests coming from other machines, on behalf of hosts that can't handle their own ARP traffic. We'll discuss proxy ARP configurations and the networks that require them next month.

ARP ages entries in the cache. An entry that is not completed within 5 minutes is removed, and entries not in use are flushed every 20 minutes. These timeouts are governed by the variables ip_ire_cleanup_interval and ip_ire_flush_interval, settable with the ndd interface:

luey# ndd -set /dev/ip ip_ire_flush_interval 600000

Note that the values are in milliseconds.

Given that ARP is relatively simple, it seems it should have few failure modes. However, it sits at the heart of each nascent TCP/IP connection, and flakey ARP behavior is the root cause of many network dog-days. The first thing that can go wrong is that your router or switch gets upset with a multi-homed machine's ARP replies.

Love me two times: Multi-homed hosts and MAC addresses
How and why might you want to change your own ARP information? If you're connecting multiple network interfaces to a switch or hub that wants to see a unique MAC address on each port, you'll need to modify the default Sun configuration. Start by dumping out the MAC address for each interface using ifconfig -a:

luey# ifconfig -a
lo0: flags=849 mtu 8232
	inet 127.0.0.1 netmask ff000000 
qe0: flags=863 mtu 1500
	inet 196.9.8.49 netmask ffffff00 broadcast 196.9.8.255
	ether 8:0:20:21:15:13 
qe1: flags=863 mtu 1500
	inet 196.9.7.47 netmask ffffff00 broadcast 196.9.7.255
	ether 8:0:20:21:15:13 

You'll notice that the same Ethernet address shows up for every interface, creating an N:1 mapping for that machine that ARP manages just fine on multiple networks. When you connect those interfaces to the same network, for example, multiple ports of a switch - to gain improved input bandwidth to the server using 2 IP addresses on dedicated switch segments - the switch is likely to be confused by seeing the same MAC address show up on multiple ports.

The easiest way to fix this behavior is to assign each interface its own MAC address. The "ether" parameter to ifconfig allows you to set the MAC address on a per-interface basis. One easy configuration is to add a digit to the first hex value in the default MAC address:

luey# ifconfig qe1 duey-qe1 ether 18:0:20:4:56:7
luey# ifconfig qe2 duey-qe2 ether 28:0:20:4:56:7
luey# ifconfig qe3 duey-qe2 ether 38:0:20:4:56:7

Be careful not to set the last bit in the first value in the MAC address, as this bit is reserved for indicating multicast addresses. For example, 18:0:20:x:x:x is a valid station address but 9:0:20:x:x:x is a multicast address.

Wary the scary middleman: ARP and routers
Dealing with single hosts and switches is relatively simple. How does ARP play when there's a router involved? ARP is the magic glue that guides the packet from the sender to the router without cheating on the value of the real, destination IP address. Here's how it works:

• Our local node huey has to send a packet to node faraway, at IP address 196.7.8.23. Consulting the IP routing table, huey determines that the packet needs to go to router cisco45 at IP address 196.7.9.1.

• If required, huey sends an ARP request for cisco45's MAC address, corresponding to its IP address determined from the routing tables.

• Using the router's MAC address, huey constructs the outbound packet with cisco45's MAC address in the Ethernet destination field, and faraway's IP address in the destination IP field. The packet then goes from huey to the router, where it gets forwarded to another interface based on the destination IP address. The router may also have to ARP for the next hop's MAC address on the next network.

There are two rules to be taken from this quick look at packet routing. First, never try to associate MAC addresses and IP addresses merely from the contents of an average, TCP or UDP packet. The only real answers come from ARP replies. Traffic that is incident to a router will have the router's MAC address but the real, final destination's IP address, confusing you if you want to build IP:MAC mappings that way. Second, keeping a full and useful ARP cache reduces network latency by taking an extra round-trip request-reply pair out of the connection set up. We'll wrap up with some tips on keeping ARP running smoothly and detecting trouble spots when it's not.

ARPing up the wrong tree: Dealing with trouble in Network doggie land
ARP can produce a whole host of problems that create grand pauses or performance bottlenecks across an entire network:

• The SunOS 4.1.3 and eariler ARP cache is only 96 entries deep, arranged in a 16x6 array. If the last digits in your IP addresses don't cover all 16 buckets evenly, you'll only use part of the ARP cache. The worst possible case is to number your machines .1, .17, .33 and so on, as they'll all use the same ARP line in the cache. There is a patch to increase the size of the ARP cache, bundled into later SunOS 4.1.x releases, highly recommended for networks with more than 100 nodes on them.

• It's possible to get more than one reply for an ARP request. The last reply to arrive makes it into the cache, so a slow machine at the "end" of the network that is overly eager in answering ARPs or has a bad published ARP table will overwrite valid ARP entries. You'll see this problem as intermittent network failures; machines will suddenly stop receiving data when their servers get the wrong MAC address from another machine on the network. Watch the network traffic using "snoop -broadcast", and look for valid ARP requests followed by precisely one reply from the right machine. If you get multiple replies, check the configurations on the machines that replied but should have remained silent.

• Any machine configured with a broadcast address as its IP address will answer ARP requests for nearly every machine; the broadcast IP address appears to match nearly every possible ARP target on the network. This is of particular importance when switching from a 0s-based broadcast address to a 1s based address; make sure all of the hosts agree on the proper format of an IP broadcast address.

• A bogus published ARP entry can also lead to confusion, where the server appears to be sending valid data to the client, but it goes to the wrong client because the server has cached the wrong ARP information. You'll see packet traffic on the network, but it will be destined for the wrong host. Watch the initial ARP exchanges between server and client machine to catch the culprit; try flushing the server's ARP cache using arp -f to force the server to ARP for all IP addresses.

ARP is one of the protocols you'd like to learn about and forget, but each time you have a puzzling network performance problem that can't be explained by application or system software design, make sure your network traffic is flowing to the right places. Everybody loves the network that does as its supposed to, quietly and unevently, like a good doggie companion.

Resources

• Take a look at Hal Stern's other Sysadmin
  /sunworldonline/common/swol-backissues-columns.html#sysadmin
• Other system administration-related stories
  /sunworldonline/common/swol-siteindex.html#systemadmin

If you have technical problems with this magazine, contact webmaster@sunworld.com

URL: http://www.sunworld.com/swol-03-1997/swol-03-sysadmin.html
Last modified: