1.1
Convergence
Data centers use different network fabrics to meet the quality of service require- ments of different workload types. There are three main workload types, namely, general networking traffic, storage traffic, and inter-processor traffic. General net- working traffic is handled by Ethernet, storage traffic is handled by Fibre Channel
(FC), and inter-processor traffic is handled by Infiniband (IB). Each network fabric is designed to meet the throughput and latency requirements of its workload. Gen- eral networking traffic is very mixed and includes web traffic along with email and socket applications, so there are few quality of service requirements. Storage traffic between storage devices and servers requires high I/O rates, high capacity and non- disruptive data delivery. Inter-processor traffic requires low latency. General network traffic does not have stringent latency requirements and is not drastically impacted by packet loss. Storage and inter-processor traffic have low latency requirements and require guaranteed delivery of packets. Figure 1-1 shows an example of a converged datacenter. Unfortunately there is a major down-side to having several fabrics in a single data
1Figure 1-1: Converged Datacenter center. There is a high cost to managing and maintaining the various fabrics in a data center. Having several fabrics in a single data center results in high heat generation due to additional heat generating hardware as well as reduced air flow, so cooling costs are high. The cost of purchasing different types of equipment for each technology is high. Furthermore, staff with various skills and expertise are needed to maintain the different fabrics, so technical and management costs rise. As a result, there is a move to using the same fabric for transmitting different workloads. The fabric must be partitioned into virtual fabrics, where each virtual fabric link is reserved for a particular workload type. Ethernet is the fabric of choice since it is the cheapest and most ubiquitous of the three fabrics. However, a major disadvantage of Ethernet is that under high traffic conditions congestion occurs and Ethernet may start dropping packets which is unacceptable for most storage and inter-processor traffic. In order
2to transmit different workloads on Ethernet, it is necessary to ensure that Ethernet becomes “lossless”.
Ethernet does provide flow control mechanisms to reduce packet loss during con- gestion. One mechanism classifies traffic workloads into eight priority levels (0-7). The lower priority traffic classes have to wait until higher priority traffic classes are trans- mitted, so there is less chance of packet loss in high priority workloads. Workloads with strict performance constraints, such as SAN traffic, are given a high priority.
Another flow control mechanism available is Ethernet PAUSE [5]. When a computer
(sender) transmits messages faster than some part of the network (receiver) can pro- cess the message, the receiver transmits a PAUSE frame back to its senders. The
PAUSE message results in the sender halting transmission of data for a specified pe- riod of time. Unfortunately, the Ethernet PAUSE mechanism does not differentiate between senders, so all senders have to stop transmitting, not just the sender that was overwhelming the receiver. The Ethernet PAUSE mechanism is not sophisticated enough to handle multiple traffic classes on the fabric. With Ethernet, a specific traf- fic class can only avoid losing packets using higher-level protocols like TCP/IP that recover from packet loss. However, these reliable protocols have too much overhead to be useful for storage and inter-processor traffic classes. There is a clear need for a low-level, sophisticated congestion reduction mechanism that differentiates between traffic classes.
1.2
Data Center Bridging
Data Center Bridging is a new set of standards defined by the IEEE standards body to handle the smooth transmission of multiple traffic classes on Ethernet. DCB defines four protocols, namely, Enhanced Transmission Selection (ETS), Priority-based Flow
Control (PFC), Congestion Notification (CN) and DCB eXchange Protocol (DCBX).
3ETS provides a guaranteed bandwidth allocation for the eight traffic classes. PFC allows for the independent pausing of traffic from specific classes. For example, if a switch is getting overwhelmed by packets from a specific class, then PFC allows the switch to send PAUSE signals back to the transmitter of this traffic class. PFC is a link by link flow control control protocol, and only works between devices that are directly connected. A disadvantage of PFC’s link by link behavior is “congestion spreading” — the congested node sends PAUSE frames to its sender neighbor node; the neighbor reduces its transmission rate, so the neighbor gets congested and sends
PAUSE frames to its sender neighbor; in this manner, the congestion from the first congestion point spreads through the network. To prevent congestion spreading, CN allows a congestion point to send a message across the network that tells the originator of the heavy traffic to slow down. DCBX provides a mechanism to ease configuration of the network. DCBX does not affect performance on the network. It only simplifies the job of network administrators.
The DCB standards seem to have solved the problem of transmitting multiple classes of traffic streams on Ethernet. Unfortunately, the problem still persists due to a hardware shortfall, namely, switches not supporting eight traffic queues. In order to recognize eight traffic classes, switches must implement eight queues for every port. This is both cost and space expensive, so most switches only support two traffic queues. Therefore, the eight traffic classes are partitioned into two traffic queues, and each queue handles up to four traffic classes. If any traffic class starts causing congestion, the DCB protocols will throttle all the traffic classes assigned to the corresponding queue. The DCB protocols are ineffective without the supporting hardware. Even if hardware technology improves and switches start supporting eight queues, there is another problem, namely, handling transmission of multiple streams within
4the same traffic class. The Internet supports several workloads - Netflix, Skype,
HTTP file transfer, and BitTorrent are a small sample of typical workloads on a university subnet. Since there are far more than eight workloads, several types of workloads are assigned to the same traffic class. For example, all storage workloads could be in one traffic class. Within this class, there could be two streams. The first stream may contain packets relating to a large file’s transmission, while the second stream may contain packets relating to small modifications to files. It is important to schedule the two streams so that there is minimal packet loss and some degree of fairness. DCB is also a new technology and there is currently little research on the per- formance of real DCB networks. DCB drastically alters the performance characteris- tics of Ethernet by dividing the network into independent traffic classes that can be grouped and paused individually. It is important to understand the performance char- acteristics of DCB enabled Ethernet. In order to understand the benefits of DCB, it is important to evaluate the performance characteristics of each protocol within
DCB and understand how the protocols interact with each other and with existing protocols. The three main protocols in DCB networks are: PFC, ETS and CN. Both
PFC and ETS can be implemented with minor changes to existing Ethernet devices, so these protocols have already been deployed in devices. CN is significantly more challenging to implement and currently there are no devices that deploy it. Therefore, at this time, only simulation or analytical analysis of the CN protocol is possible.