Traffic Delay in Ethernet over SONET/SDH Guylain Barlow, Product Manager Innocor Ltd.

Ethernet over SONET/SDH (EoS), or the combination of Virtual Concatenation (VCAT), Link Capacity Adjustment Scheme (LCAS), and Generic Framing Procedures (GFP), enables legacy transport networks to evolve into data-friendly environments, particularly for Ethernet and storage services. There are several types of delays that affect EoS systems and networks, and impact the selection of test methods. The impact of delay depends on the type of service delivered through a network. As an example, broadcast video is very sensitive to delay variations and less sensitive to transfer delay while interactive video is sensitive to both. The contributing factors to delay can be divided into two main areas. The first is propagation delay, which is the principal cause of differential delay between the path members that constitute a VCAT group (VCG). The second aspect comes from the data nature of the GFP encapsulated EoS traffic, which can introduce emission and queuing delay. This paper does not consider additional contributing factors, such as failure recovery and dynamic bandwidth adjustments. Propagation Delay The transmission of information always involves a transfer delay in the form of latency. The most basic form of delay, propagation delay, occurs in fiber optic systems relative to the speed of light. In this case, because the transmission medium is silica glass, light travels at a lower speed than in a vacuum. The difference in speed is accounted for by the index of refraction of the core material in the optical fiber. Of note is the fact that propagation delay is independent from the link rate and depends on distance. An example of propagation delay across the Atlantic Ocean, or about 5,500km: Distance * Refraction Index/ Speed of Light = 5,500km * 1.46 / 2.998 * 10^ 5 km/s= 26.78ms. With VCAT, the available bandwidth for a group is a multiple of the bandwidth of a constituent SONET/SDH path. For example, the bandwidth for a VCG using STS3c/VC-4 paths is a multiple of 149.76Mbits/sec. As a result, a Gigabit Ethernet service using EoS and 7 STS-3c/VC-4 members consumes a bandwidth of 1.048Gbits/sec. Since each VCAT path member is an independent entity in the SONET/SDH network, the paths in a VCG may take different physical routes and reflect different propagation delay values. The overall VCG propagation delay then corresponds to that of the slowest path member. In addition to propagation delay, there is also a processing delay component, generally small, which occurs when going through network equipment.

Figure 1 VCG Members

Figure 1 provides an example of physical routing for a 4-member VCG. The difference in propagation delay values between VCG members forms the basis for the concept of differential delay. Differential Delay Differential delay is the relative arrival time measurement between the members of a VCG. This means that in an EoS network, buffering is used at the terminating end of a VCG in order to re-align the data stream. For high-order VCAT paths, including STS-1nV, STS-3c-nV, VC-3-nV, and VC-4-nV where n is the number of path members, the differential delay is measured by examining the Multi-Frame Indicator (MFI) present in the path overhead of each VCG member. The MFI, which is located in the H4 byte as in figure 2, increments frame-after-frame for each member of a VCG.

Figure 2 H4 Byte in High-Order VCAT

The information in Figure 2 illustrates that it takes a cycle of 16 frames (2ms) to gather the full MFI information. In addition, the absolute maximum compensation between members is 512ms, as the twostage 12-bit MFI value increments from 0 to 4095 (4096 * 125us = 512ms). The MFI values are aligned across the VCG members at the origination point. At the destination point, the evaluation of differences in MFI values between members will reflect the differential delay between any two members that are part of a VCG. Differential delay tends to reflect stable values although factors such as pointer adjustments and environmental factors impacting propagation delay on optical fibers may cause variations. From a test perspective, it is possible to use equipment that will simulate differential delays without the use of long distance fiber. This involves the manipulation of MFI values for path members and the buffering of the payload information carried by the VCG in order to maintain flow order. The evaluation of low-order (VT1.5-nV, VT2-nv, VC-11-nV, VC-12-nV) differential delay uses the Frame Count information, which is spread across multiple SONET/SDH frames.

Figure 3 Z7/K4 byte in Low-Order VCAT

Figure 3 illustrates the VCAT Frame Count information found in bit 2 of the Z7/K4 byte. As illustrated, the H4 byte is then used to group four SONET/SDH frames into a superframe that contains pointer information (V1, V2, V3, V4) and four low-order overhead bytes. This includes the Z7/K4 byte, where the VCAT information is spread across 32 superframes, which results in a Frame Count cycle of 128 frames (16ms). The low-order maximum compensation between members is 512ms (32 * 16ms) as the 5-bit Frame Count value increments from 0 to 31. The low-order differential delay is evaluated by monitoring of the Frame Count information for each path member. Emission Delay GFP provides a mechanism to map Ethernet onto SONET/SDH. The GFP functions apply at the edge of the EoS network, which limits the GFP framing impact on delay. Two mapping modes are available in the form of Frame-Mapped GFP (GFP-F) and Transparent GFP (GFP-T). GFP-F maps Ethernet Media Access Control (MAC) clients into GFP frames and maintains a one-to-one relationship between the MAC Protocol Data Unit (PDU) and the GFP frame. In the case of GFP-T, the incoming information from clients like Gigabit Ethernet or Fiber Channel is mapped into fixed-length GFP frames. The GFP-T frame transmission takes place without waiting for the reception of an entire client data frame. To help achieve this, GFP-T performs pad stuffing to compensate for clocking differences between client and line. The use of GFP-F involves the buffering of MAC PDUs before transmission, and this process induces emission delay otherwise called transmission delay. An example:

Using 7 STS-3c/VC-4 channels for an aggregate bandwidth of 1.048 Gbits/sec and a MAC PDU size of 1,518 bytes plus 8 bytes of GFP overhead results in an emission delay of (1,518 + 8) * 8 bits/bytes / 1.048Gbits/sec = 11.6us. The impact of emission delay increases as the bandwidth decreases or the frame size increases. The same example with 7 VT1.5/VC-11 totaling 11.2Mbits/sec of bandwidth produces an emission delay of 1.09ms. The difference between GFP-T and GFP-F in terms of emission delay is analogous to the Ethernet concept of transparent bridging, compared to store and forward bridges. Queuing Delay EoS is often described as a point-to-point transport mechanism that provides a cutthrough link between Ethernet islands. However, there are cases where the overall service integrates a MAC switching function, which involves additional delays including queuing, and a small processing delay for switching. One reason to introduce MAC switching in an EoS network, which can be based on regular MAC or Resilient Packet Ring (RPR) MAC, is to more efficiently share the bandwidth as provided by VCAT. It also enables the statistical multiplexing of bursty traffic and spatial reuse but results in higher service complexity. Performing MAC switching can involve classes of service and priority queues, which implies queuing delay. This type of delay depends primarily on bandwidth utilization and link rate. It is a complex topic that involves many variables such as the traffic-scheduling algorithm.

Figure 4 Queuing Delay Example

Figure 4 shows a diagram that provides a comparison of queuing delay values, for a specific service class, based on connection bandwidth and bandwidth utilization. The purpose is to illustrate the exponential nature of queuing delay and its diminishing impact as connection bandwidth increases. In the case of VCAT, the link bandwidth may take intermediate values based on the VCG size. The delay components of propagation,

processing, emission, and queuing are additive to result in total latency. To evaluate latency in an EoS network, a universal technique may be used. Latency Measurements RFC 2544, as outlined by the IETF (www.ietf.org), documents a method for evaluating latency. The RFC recommends the use of fixed frame sizes for each test, which keeps the emission delay constant. In addition, the RFC recommends that the traffic throughput, which corresponds to bandwidth utilization, be kept constant for the duration of a test. This effectively limits queuing delay variations, should it be a factor, in order to provide more stable measurements. The RFC also addresses the concepts of bit forwarding and store and forward. Bit forwarding is the time interval when the first bit of a frame reaches the system and the first bit of the same frame leaves the system. Whereas store and forward is the time interval when the last bit of a frame reaches the system and the first bit of the same frame leaves the system. In actuality, the difference between bit forwarding (GFP-T) and store and forward (GFP-F) latency is a fixed factor corresponding to the emission delay. Test devices calculate the latency by inserting timestamps into frames and measuring the elapsed time when the frame arrives at destination. This measurement can be for a unidirectional flow or round trip. For constant bit rate services, an additional test item is the evaluation of latency variations between consecutive frames, also called latency variance or data jitter. In summary, several delay factors impact Ethernet transport in EoS. From a service perspective, the inherent components of propagation, processing, emission, and queuing delay combine to produce the total latency. The evaluation of latency and data jitter is accomplished using timestamps to evaluate the transfer delay time and its variance. VCAT, which aggregates multiple SONET/SDH path members, introduces a new delay component in the form of differential delay. This differential delay is the relative time difference at the traffic destination point between VCG path members. This difference is primarily attributed to propagation delay. It is measured by monitoring the differences in MFI, for high-order, or Frame Count values, for low-order, between VCG path members. As in the example of broadcast and interactive video services, performing these measurements will help to ensure adequate service based delay characteristics.