How you can make the network yente work for you
Need to find the IP address of a network node? Let ARP be your matchmaker
This month, we'll give you the network handy-person's guide to ARP, starting with an overview of how the protocol fits into the TCP/IP world. We'll look at how Solaris manages the ARP protocol and associated caches, and see how ARP deals with routers and other devices that don't pass along its gentle probes for data. We'll conclude with a list of symptoms and fixes for common ARP-related woes. (2,800 words)
In the Broadway show and movie "Fiddler On The Roof," five young women turn to Yente, the arranger of marriages in their small Russian town. Yente's job is that of the matchmaker, finding the best fit for a young lady, or at least a job close enough to call complete. Are we crass enough to take this slice of classic show business and turn it into a network analogy? Right -- of course right!
Whether you realize it or not, the address resolution protocol (ARP) is hard at work constantly matching IP addresses to their Ethernet hardware addresses on your network. When ARP is working well, packets flow in both directions, across routers and bridges without a moment's notice. If you're suffering from ARP difficulties, you'll see intermittent grand pauses in network traffic, reductions in usable network bandwidth, and generally poor performance for various network applications.
Like Yente, ARP has to deal with liars, two- and three-timers, and other less-than-trusty network citizens like those middlemen we usually call routers. Figuring out who's telling the truth, who needs a sharp reprimand, and what's going wrong is, of course, your job. Right? Of course right!
A many-to-one splendored thing: Mapping IP addresses
First of all, it's permissable to sound out the protocol acronym, rhyming it with "harp," even if you do sound like Network the dog with a bit error in his bark. ARP can also be used as a verb as in, "Who is arping for that address?" It's not part of the IP protocol per se but instead is a peer protocol that is nearly always included in an implementation of TCP/IP. ARP is described in gory detail in RFC 826.
ARP's job appears simple on the surface: it finds the hardware or MAC (media access control) address corresponding to a given IP address. ARP builds up relations of 32-bit IP addresses into the 48-bit MAC address space, maintaining these many-to-one mappings on a per-host basis. Each machine that runs TCP/IP needs to track the mappings constructed up by ARP traffic, building its own map of the local network. The fact that ARP management represents the ultimate in distributed state is what also makes it prone to strange failure modes.
A bit of background on MAC addresses is in order before explaining ARP's tasks. A hardware, or MAC, address is assigned per host - one per system, no matter how many network interfaces it has. The MAC address uniquely identifies the system in the "universe" of Ethernet devices. MAC addresses are written as a series of 6 colon-separated hex values, for example, 8:0:20:74:b:6. The upper 24 bits are reserved for a vendor ID and the lower 24 bits represent a unique sequence number within that vendor's space. You'll find all of your Sun systems have MAC addresses that start with 8:0:20. RFC XXX, Assigned Numbers, provides a listing of MAC address prefixes and vendors, allowing you to match up a network packet and the flavor of machine it came from. You'll also see that Sun assigns one MAC address per system, burned into the ID or EEPROM of the system, a fact that startles most people adding additional network interfaces.
The IP-MAC relationship is N:1 when a given host has IP addresses on multiple network interfaces but uses the default MAC address on each one. For the most part, a given MAC address maps back to one IP address per network -- so a network client can maintain a 1:1 table of IP addresses and associated hardware addresses. Looking at it another way, IP addresses are logical conventions, assigned by system administrators to make the TCP/IP protocol work. MAC addresses are physical conventions, assigned by the vendor; ARP's job is to bridge the logical world onto the physical one.
Who are you? ARP in action
Here's how a typical ARP request and reply play out:
hueyneeds to send an Ethernet packet to server
duey. It knows duey's IP address, having located it via DNS, but has no idea how to find it on the network. Ethernet frames use MAC addresses to specific source and destination, so huey needs duey's MAC address to fill in the Ethernet frame header and zip the packet on its way.
Not all network protocol suites require ARP: DECnet uses an algorithm to convert DECnet area:node addresses into Ethernet MAC addresses. This requires the DECnet protocol to change the local MAC address to match the DECnet configuration, a change that can confuse bridges if DECnet is enabled after the node has already spoken TCP/IP with its neighbors. ARP traffic may seem like an unreasonable bit of overhead added to the startup of each conversation with a new host. It's the price paid for flexibility -- you can assign IP addresses as you like to your hosts, even re-using the same addresses on disjoint networks, without upsetting the inner workings of the Ethernet one bit.
Digging data out of ARP
ARP maintains its cache of IP:MAC mappings on each machine. Dump it out using
luey% arp -a Net to Media Table Device IP Address Mask Flags Phys Addr ------ -------------------- --------------- ----- --------------- le0 duey 255.255.255.255 08:00:20:1d:0b:92 le0 188.8.131.52 255.255.255.255 U le0 huey 255.255.255.255 SP 08:00:20:21:15:13 le0 fred 255.255.255.255 08:00:20:1a:f2:02 le0 184.108.40.206 240.0.0.0 SM 01:00:5e:00:00:00
You can get a slightly better report from
luey% ndd -get /dev/arp arp_cache_report ifname proto addr proto mask hardware addr flags le0 220.127.116.11 255.255.255.255 08:00:20:1f:34:9b le0 18.104.22.168 255.255.255.255 08:00:20:04:cd:1e le0 196.9.8.005 255.255.255.255 08:00:20:18:1c:e5 le0 196.9.8.012 255.255.255.255 08:00:20:11:27:af le0 196.9.8.011 255.255.255.255 08:00:20:7a:5c:2c le0 196.9.8.049 255.255.255.255 08:00:20:21:15:13 PERM PUBLISH le0 196.9.8.059 255.255.255.255 08:00:20:1a:f2:02 le0 196.9.8.099 255.255.255.255 08:00:20:1e:26:5c le0 224.000.000.000 240.000.000.000 01:00:5e:00:00:00 PERM MAPPING
Completed entries represent queries that returned at least one reply. Those marked "U" or "UNRESOLVED" are from queries that were never answered, because the IP address is not on the local network or because the host owning it was down or not listening at the moment the request was sent.
Entries marked "S" (static) or "PERM" will never be
removed from the cache; typically these are inserted on behalf of
devices (like printers) that have limited TCP/IP implementations
and don't answer their ARP requests, or to work around network configuration
quirks. Create a permanent entry using
luey# arp -s 22.214.171.124 06:1:4:bc:a:34
A "published" entry is one for which the local
host acts as a proxy server. When the local host sees an ARP request
come by for that IP address, it answers with the MAC address in
the cache, not its own MAC address. Publishing ARP entries allows
one machine to answer ARP requests for machines that can't hear the
broadcasts because they're on the far side of a router, or because
they're not listening. Published entries are marked with a "P" or
"PUBLISH" in the arp cache report. Load ARP information from a table
arp -f to publish several ARP addresses at once.
Using permanent entries prevents the local machine from sending ARP requests; creating a published entry allows the local machine to answer ARP requests coming from other machines, on behalf of hosts that can't handle their own ARP traffic. We'll discuss proxy ARP configurations and the networks that require them next month.
ARP ages entries in the cache. An entry that is not completed within 5 minutes is removed, and entries not in use are flushed every 20 minutes. These timeouts are governed by the variables ip_ire_cleanup_interval and ip_ire_flush_interval, settable with the ndd interface:
luey# ndd -set /dev/ip ip_ire_flush_interval 600000
Note that the values are in milliseconds.
Given that ARP is relatively simple, it seems it should have few failure modes. However, it sits at the heart of each nascent TCP/IP connection, and flakey ARP behavior is the root cause of many network dog-days. The first thing that can go wrong is that your router or switch gets upset with a multi-homed machine's ARP replies.
Love me two times: Multi-homed hosts and MAC addresses
How and why might you want to change your own ARP information? If you're connecting multiple network interfaces to a switch or hub that wants to see a unique MAC address on each port, you'll need to modify the default Sun configuration. Start by dumping out the MAC address for each interface using
luey# ifconfig -a lo0: flags=849
mtu 8232 inet 127.0.0.1 netmask ff000000 qe0: flags=863 mtu 1500 inet 126.96.36.199 netmask ffffff00 broadcast 188.8.131.52 ether 8:0:20:21:15:13 qe1: flags=863 mtu 1500 inet 184.108.40.206 netmask ffffff00 broadcast 220.127.116.11 ether 8:0:20:21:15:13
You'll notice that the same Ethernet address shows up for every interface, creating an N:1 mapping for that machine that ARP manages just fine on multiple networks. When you connect those interfaces to the same network, for example, multiple ports of a switch - to gain improved input bandwidth to the server using 2 IP addresses on dedicated switch segments - the switch is likely to be confused by seeing the same MAC address show up on multiple ports.
The easiest way to fix this behavior is to assign each interface its
own MAC address. The "ether" parameter to
allows you to set the MAC address on a per-interface basis. One
easy configuration is to add a digit to the first hex value in the
default MAC address:
luey# ifconfig qe1 duey-qe1 ether 18:0:20:4:56:7 luey# ifconfig qe2 duey-qe2 ether 28:0:20:4:56:7 luey# ifconfig qe3 duey-qe2 ether 38:0:20:4:56:7
Be careful not to set the last bit in the first value in the MAC address, as this bit is reserved for indicating multicast addresses. For example, 18:0:20:x:x:x is a valid station address but 9:0:20:x:x:x is a multicast address.
Wary the scary middleman: ARP and routers
Dealing with single hosts and switches is relatively simple. How does ARP play when there's a router involved? ARP is the magic glue that guides the packet from the sender to the router without cheating on the value of the real, destination IP address. Here's how it works:
There are two rules to be taken from this quick look at packet routing. First, never try to associate MAC addresses and IP addresses merely from the contents of an average, TCP or UDP packet. The only real answers come from ARP replies. Traffic that is incident to a router will have the router's MAC address but the real, final destination's IP address, confusing you if you want to build IP:MAC mappings that way. Second, keeping a full and useful ARP cache reduces network latency by taking an extra round-trip request-reply pair out of the connection set up. We'll wrap up with some tips on keeping ARP running smoothly and detecting trouble spots when it's not.
ARPing up the wrong tree: Dealing with trouble in Network doggie land
ARP can produce a whole host of problems that create grand pauses or performance bottlenecks across an entire network:
arp -fto force the server to ARP for all IP addresses.
ARP is one of the protocols you'd like to learn about and forget, but each time you have a puzzling network performance problem that can't be explained by application or system software design, make sure your network traffic is flowing to the right places. Everybody loves the network that does as its supposed to, quietly and unevently, like a good doggie companion.
If you have technical problems with this magazine, contact firstname.lastname@example.org