May 1999

ROUTER EVOLUTION:
The Specialization Of Network Elements

BY PAUL DOOLAN

George Orwell wrote that "all animals are equal but some animals are more equal than others." Translating from political allegory at the end of WWII to telecommunications at the dawn of the new millennium, we might paraphrase his famous aphorism and apply it to Quality of Service (QoS) as "all packets are equal but some packets are more equal than others." Perhaps the single most troublesome issue in the whole convergence space is that of Quality of Service (QoS).

The telephone network provides one level of QoS: you either get a 64kbps voice channel or you do not. In the voice network, depending on the country you live in, there is a small probability that you will be unable to complete a call because of lack of common equipment resources (digit receivers, trunks, call processors, etc.). As the network becomes busier this probability increases. We call this a "loss system;" in overload the call is "lost" and the carrier loses revenue.

Packet switched networks, on the other hand, are "delay systems." If the network is lightly loaded, you may inject traffic into it and receive traffic from it fairly freely. By contrast, when the network is heavily loaded, traffic can experience substantial delays. Indeed, as packet switching networks go into overload, we find that some packets never get to their destination at all. To all intents and purposes they experience infinite delay.

Since our ears are sensitive to delays in speech communication, we might wonder whether delay systems, or at least those exhibiting high delay variability, are suitable transports for voice. It turns out that they are not, and so considerable effort is being spent trying to improve the performance of IP networks in this regard. Before we look at that work, however, it will be useful to consider just how IP networks work.

IP NETWORKS AND ROUTERS
An individual IP host has an address that consists of two parts: one part identifies the network to which the host belongs, the other part identifies the particular host on that network. Networks are interconnected by IP packet switches or routers. A router knows the identities of the networks to which it is directly attached. Through participation in routing protocols, for example OSPF (open shortest path first) and BGP (border gateway protocol), a router learns the identities of networks that are (directly or indirectly) available via the other routers to which it is attached.

We speak of routers having links to networks and to other routers. We typically talk about these links in terms of their speeds and we differentiate the network topologically into "access" and "core" or "backbone" based on these factors. For example, my company's access to our IP service provider is via a T1 link (1.54 Mbps), and within the service provider's network the backbone routers may be connected at OC3 (155Mpbs), OC12 (622Mbps), or OC48 (2.4 Gbps) speeds.

A router's job is, in essence, incredibly simple. It inspects packets arriving on its incoming links, chooses the correct outgoing link, and queues the packets onto the appropriate link. The link will either be one that goes directly to the network the packet is destined for - a local network - or a link to another router. In the majority of contemporary networks, the choice of outgoing link is determined solely by the network part of the Destination Address (DA) in the IP packet. (Like an envelope in the regular U.S. mail system, IP packets have source and destination addresses.)

In actual fact, routers are incredibly complex machines, and their complexity is in large part related to the management and servicing of the queues associated with their output links. More than any other factor, queue management, particularly in the face of overload, affects the QoS a router or network is able to provide. We will see later how routers have evolved beyond the simple model discussed here, but first let's look at QoS in IP networks.

IP QoS
Contemporary IP networks offer a best effort service: the network simply promises to try to deliver all packets to their destinations as best it can. A network with unlimited resources can, by definition, deliver all offered traffic to its destination with no delay or loss. Such a network would be adequate for even the most demanding applications, since all packets would get the best possible service.

However, when we say "network resources," we are referring in the abstract to very concrete and expensive assets: the links, routers, and switches that make up a network infrastructure. Leaving aside the laws of physics, in the real world it would require an infinite supply of money to build a network with infinite resources, and so any real network has limited resources. This means that as the traffic load approaches and exceeds the network's capacity, there will be competition between packets for the network resources.

In reality, demand exceeds supply. Since we know that some of the traffic on the network is more sensitive to delay than other traffic, the goal of much of the QoS work in the IP community is to provide a network that offers service differentiation. With the onset of congestion (competition) in the network, some packets must be given preference over others. It is somewhat ironic that having spent much of the late '80s and early '90s trying to make networks behave fairly - IP networks behave "fairly" with techniques like Fair Queuing and Random Early Detection - we are now exerting perhaps even greater efforts researching how to make them behave unfairly.

In the Internet Engineering Task Force (IETF) there are two working groups that are commonly mentioned in the same breath as differentiated services and QoS: the Differential Services Working Group (DiffServ) and the MultiProtocol Label Switching Working Group (MPLS).

DIFFERENT APPROACHES - SAME PROBLEMS
Before we look at DiffServ and MPLS, let's spend a minute thinking about what it is we are trying to achieve. VoIP is an example of a traffic type that requires control over both the absolute delay in delivering packets and the variability in that delay to ensure that received data is useful. The goal then would seem to be that at each queuing point we arrange that the queue managers preferentially select VoIP packets for transmission over, say, packets belonging to a file transfer (FTP) session.

But how do we do this? There are three separate, if not quite orthogonal, issues here:

• Packets need to be classified as worthy or unworthy of special treatment.
• Network devices need to be told how to treat special packets.
• Network devices need to implement active queue management techniques to provide the required treatments or behaviors.

In slightly more abstract terms, we have classification, marking, signaling, and queue management issues to deal with. So how do DiffServ and MPLS measure up?

DIFFSERV
In looking at providing QoS, the DiffServ working group decided that it couldn't possibly deal with specifying all the likely services that applications might need now or in the future. What it has managed to propose is a way of marking packets using a field in the IP packet header called the Differential Services Code Point (DSCP), and associating Per Hop Behaviors (PHBs) with those code points that allow the synthesis of services. Of particular interest to us is the "Expedited Forwarding" PHB since it can be used to build low latency and low jitter services.

Expedited forwarding requires the DiffServ routers in a network to serve the outgoing queues faster than traffic arrives. In practice this means reserving bandwidth at each of the nodes and policing traffic entering the network so that the offered load stays below the reserved bandwidth (otherwise queues begin to build up). Policing schemes rely on having a contract - expressed, for instance, in bits per second - between the traffic source and the network. The network will only accept traffic that stays within the contract. Alternatively, the network will mark excess traffic in such a way that it can be discarded under overload as being "out of contract." DiffServ only suggests possible mechanisms for the policing and queue management on the lofty basis that the IETF is about specifying protocols and not telling people how to build routers.

So what does DiffServ give us? Not much more really than a way of marking packets to indicate which of a limited number of queuing treatments should be applied. It does have the nice scaling property that packet classification and marking are done at the edge of the network, and that the DiffServ routers in the core act on the DSCP markers in deciding on queuing treatments, thus avoiding having to do complex packet classification themselves at high speed. We will see that MPLS places somewhat similar burdens on the edge routers.

MPLS
MPLS works by setting up virtual circuit-like paths, called Label Switched Paths (LSPs), through the network. Specific QoS requirements for an LSP can be signaled as the paths are set up. This capability, in concert with the ability to "explicitly route" or dictate the path that these LSPs follow through the network, has led to comparison with trunks in the voice world. In fact some vendors are talking about using MPLS to set up just such things - LSPs with known delay characteristics through which VoIP sessions are "trunked" from one voice gateway to another. MPLS is superior to DiffServ in this regard.

MPLS places the same requirements on the edge routers as DiffServ, in that they have to classify and label the packets. Although MPLS itself doesn't specify this behavior, it is clear that these label edge routers would be the logical place to police traffic flows.

ANYONE REMEMBER ATM?
Although DiffServ and MPLS provide some of the tools necessary to provide QoS through IP networks, neither of them have the same extensive capabilities in this regard as ATM. The principle deficiency is in the area of QoS routing. While DiffServ and MPLS may deal with marking or labeling, neither of them address the key issue of how to find a path through the network that supports a given QoS requirement.

The ATM PNNI routing protocol has this capability, and there have been suggestions that integrating IP routing with PNNI, the so-called Integrated PNNI approach, would help solve this problem. While MPLS proponents like to contrast its simplicity against ATM, it clearly isn't a zero sum game. The simplicity comes at the cost of certain capabilities: in this case, QoS routing which people are busily building into the IP routing protocols.

ROUTER EVOLUTION
As in Darwinian systems, routers are evolving by becoming more specialized. For a decade, router development was principally one of scaling; i.e., the machines were made faster and they had more ports and more memory for buffers and route tables. However, destination-based, best effort forwarding remained the norm. The machines in the access and core parts of the network differed principally in scaling respects, not in terms of providing different capabilities.

This is now changing. New routers are emerging with capabilities optimized for their specific place in the network. In broad terms we see complex functions like packet classification taking place at the edge, while the core machines concentrate on forwarding huge amounts of data at high speed based on simpler forwarding criteria like MPLS labels or a combination of DA and DSCP.

Additionally, the edge routers that touch the end customer are beginning to support multi-service capabilities, such as network-based IP virtual private networks. As convergence occurs, "the rubber meets the road" somewhere. Increasingly that place looks to be on multi-service platforms located at the edge of the carrier's network.

For example, if we consider the migration from circuit voice to VoIP, the familiar compression and packetization has to occur somewhere. It makes sense to many carriers to use existing TDM facilities to bring voice and data traffic from the customer premise to the Carrier Office or POP and convert the voice from circuit to packet at that point. For this reason, we see the voice compression and conversion capabilities familiar in the CTI market making an appearance inside what has traditionally been simply packet switching equipment. This is a space that no one currently owns. It will be interesting to see if the winners are voice vendors who make the leap to handling IP or the established IP vendors who successfully integrate voice capabilities into their platforms.

AN ENGINEERED SOLUTION
While we wait for IP QoS routing to be developed, can we provide voice-grade QoS in our IP networks today? The answer is yes, but at some cost. The essential trick is to over-engineer or under-subscribe the network. In doing this, link and consequently buffer occupancy in the routers is kept low and so therefore is delay. This is clearly an expensive way to run a network, but since many of the new carriers touting VoIP are also those planning to deploy high density WDM on their fiber networks, maybe inefficient use of bandwidth is less of an issue than it has been in the past. In any case, the IP community will continue to work to solve this problem, and the economies obtained by using a single converged network will continue to cause many carriers to embrace IP as the voice transport for the future.

Paul Doolan is vice president and chief technology officer of Ennovate Networks and is an active member of the Internet Engineering Task Force's (IETF) Multi-Protocol Label Switching (MPLS) Working Group, for which he is presently co-authoring the MPLS standard framework document. Doolan is also the editor of the Label Distribution Protocol (LDP) within the IETF's LDP Design Team. He also is a co-author of the recently published book, Switching In The Internet, published by Morgan Kauffman. Ennovate Networks was founded in 1997 to design, develop, build, market, and support premium IP provisioning products that will help satisfy the increasing demands of Network Service Providers (NSPs) to provision new and innovative services for their corporate customers. For more information, visit the company's Web site at www.ennovatenetworks.com.