____________________________________________________________________________________________________
1880 S. Flatiron Ct., Ste. M, Boulder, CO, 80301 •(303) 444-7765 • FAX (303) 444-7767 • www.gigabytex.com
White Paper
CONGESTION and DELAY HAMPER the GLOBAL INTERNET
By Bill Gibson, Chief Technical Officer, Niwot Networks, Inc. April 5, 2000
(revised July 25, 2000)
The Global Internet promises world wide efficient and
inexpensive communications for individuals and businesses. Measurements of the evolving Global Internet
show congestion (packet loss) and long round-trip delays (ping times) are facts
of life in this environment. TCP (Transmission Control Protocol) is the basic
transport for Internet applications such as email, web browsing, and file
transfer. This paper presents Niwot
Networks’ research on the performance of TCP over Global Internet conditions.
When an application program sends a stream of data, the
sending TCP divides that data into packets, sends those packets, and the
receiving TCP reassembles the data stream from received packets. With TCP, if
packets are lost the sending side must retransmit them.
Measurements at www.internettrafficreport.com
show
round-trip delay (ping time) on the Global Internet from 80 milliseconds
(within North America) to over 600 milliseconds (to South America). Congestion
is another name for packet loss, and measurements show from 0 per cent to 8 per
cent of packets being lost Globally.
Our results show that TCP is significantly slowed by either
packet loss or delay, and that the slowing is dramatic under Global Internet
conditions where packet loss and delay occur together.
This chart shows the dramatic drop in throughput of TCP file transfer over a T1 link with various ping time(delay) and packet loss (congestion). Tests were done between NT4(Service pack 6a) Pentium workstations. Software was Niwot’s Gigabyte Express with compression disabled. Delay simulated by a TTC 1010B Delay simulator. Packets dropped by custom NT4 WAN drivers.
Problem Summary: When faced with the typical Global Internet ping time of 300 milliseconds and 1 out of 25 packets being lost (4 per cent) an Internet user with a T1 connection who might expect 10 megabytes/minute throughput on a clean local connection will find he is getting less than 10 per cent or under 1 megabytes/minute. In the case of a 600 millisecond (satellite) ping time and one in 12 packets being lost (8.5 per cent) theT1 throughput falls even farther, to less than 3 per cent or under .3 megabytes/minute.
What can be done?
Parallel Operations: If a customer is transmitting hundreds of files, multiple FTP or
Gigabyte Express clients can run in parallel to move the files. Each file moves slowly, but the total
throughput can be acceptable.
Special Products for Satellites: Mentat (www.mentat.com) makes a
gateway that efficiently handles most of the issues with satellite delay and
packet loss and is installed at either end of the satellite link. Existing
applications may gain an advantage when the transmission path includes a
satellite link equipped with Mentat technology.
Redundant Extensions to TCP: Niwot’s RELIA technology (patent
pending) is an extension to TCP which adds redundant information that
enables the receiver to reconstruct
lost information without requiring the time delay of retransmission. The first application to support RELIA is Niwot’s
Gigabyte Express for Windows version 5.0
Measured Packet Loss and Delay: From
www.InternetTrafficReport.com, these two charts show recent 30 day
International Packet Loss measurements and Delay measurements:
Global Packet loss measured in July 2000.
Global
Ping Time measured in July 2000..
Where are packets being lost? It is hard to identify where any particular packet on the Internet is lost. The prime suspects are:
Congestion: The Internet Standards treat packet loss and congestion as synonyms. Routers discard incoming packets that can’t be stored or transmitted. Imagine an Ethernet (10 megabit/sec) pipe feeding a T1 (1.536 megabit) router. Anytime the average feed from the Ethernet exceeds 1.536 megabits/sec, packets will be lost. This is normal congestion, (ie. packets lost) because the average sum of the inputs to a router exceeds the capacity of its output. When the output connection is a costly nation-to-nation or satellite link, it becomes very expensive to make the pipe big enough so packets won’t be lost.
Bit errors: As information packets
move from place to place, there is always a chance that some bits will be
modified. Each packet has a mathematical sum of the bits it contains appended
to it. When a receiving router or station receives a packet whose contents and the
appended sum don’t agree, that packet is discarded. This is most likely on satellite links (when the sun is near the
satellite, or there is an electrical or rain storm) and on “last mile” plain
old telephone service links, but it can occur anyplace in the journey from
source to destination.
Deliberate Discard: ATM networks guarantee that voice or video connections won’t lose bits. Internet packet traffic moves over these same networks, and if it looks like there are too many packets to get all the voice and video through without missing a bit, packets are discarded until there is room for the voice and video. Similarly Cisco and Nortel backbone routers offer packet discard policies, so the operator of the router can decide which types of traffic will suffer lost packets as the router approaches congestion.
What is causing
the delay? 600 milliseconds is a typical ping time when a satellite is in the
path. 80 milliseconds is a typical North American ping time when going between
national backbones. Most of this delay cannot be avoided.
Speed of Light: In the
case of a synchronous satellite, the speed of light isn’t fast enough. For a geosynchronous satellite it takes 250
milliseconds for the signal to get from the earth to the satellite and
back(22000 miles). If the return path also goes via satellite, the ping time
will be at least 500 milliseconds (600 milliseconds is a typical ping time when
a satellite is in the path). The speed
of light in a fiber optic cable works out to 10 millisec per thousand miles, for
a ping time (due to the speed of light) of 60 millisec on a coast-to-coast (US)
fiber link.
Router in and out time: Routers receive packets before forwarding them. If a router is sending a 1500 byte packet at T1(1.536 megabits/sec) the time from the first bit to the last bit is 7.8 milliseconds. If the router is storing packets waiting to send them, then the delay time increases. We believe this is the reason the measured Global Internet ping time varies.
Why is TCP so slow? TCP does a wonderful job on long distance slow data links and on high speed local data links. It was not designed for long distance high speed data links.
Congestion Avoidance: TCP assumes that all packet loss is caused by congestion and responds by reducing the transmission rate.
Slow Start: When a TCP connection starts (or re-starts if more than one packet has been lost) it sends one packet, waits for the acknowledgment, then sends 2, then 4...and ramps up its transmission pace. Each step in the ramp consumes a round trip delay.
Data Acknowledgments: The TCP receiver sends an acknowledgment to the sender whenever a segment of information is received. The sender does not assume any data is lost until a multiple of round trip time has elapsed without receiving an acknowledgment, or until it has received multiple duplicate acknowledgments.
Window Size: TCP can
only send a certain amount of data before it must stop transmitting
and wait for an acknowledgment. That amount of data is called the window size.
The standard window size in TCP is limited to 64 kilobytes. RFC1323 allows larger windows, but it is not
yet usable by applications running on Microsoft platforms.
TESTING DETAILS:
Target Test Conditions:
· Ping times of 10 millisec,
80 millisec, 300 millisec, and 600 milliseconds. This covers the range of
delays expected.
· TCP file transfers of a
large file(3 megabytes). A large file is used to reduce the effect of start-up
and closing handshakes on the results.
· Packet loss from none to 8
per cent. This covers the range of packet loss expected. The meaning of “x%
packet loss” is ambiguous, since the measurements don’t specify in which
direction of the round trip the packets are being lost, and the measurements
can’t tell the difference of 4 per cent due to 1 packet being lost out of every
25 as compared to a burst of 8 packets being lost out of every 200. We tested
packet loss conditions of: none(0 per cent), 1 out of 100(1 per cent), 1 out of
50 (2 per cent), 1 out of 25 (4 per cent), and 1 out of 12(8.5 per cent) as well as 4 out of 100(a bursty 4
per cent) and 8 out of 100(a bursty 8 per cent). These packets are lost in the
direction the files are being transferred.
· Link data rate of 1.536
megabits/sec(T1). This rate was chosen because it represents a “good” typical
business connection to the Internet.
Test Platform:
· Route using Windows NT4 as
an IP router at each end of the delay link.
· Delay using TTC 1010B
Digital Delay Simulator.
· Packet loss accomplished by
programming Niwot’s WAN drivers on the file receiving side to lose the desired
percentage and burstiness of packets.
· File Transfer Software running
on Pentium computers with Windows NT4 Service Pack 6a installed. Measured both Niwot’s Gigabyte Express(tm) file
transfer software with compression turned off and Microsoft-standard FTP file
transfer sofware. Results for FTP were inferior to Gigabyte Express in the 80
millisecond and longer environment. and are not charted. The full results follow:
|
3/10/00 |
|
|
|
|
|
Sending a 3.22
Meg file |
|
|
|
|
|
Throughput in
Megabytes/minute |
|
|
|
|
|
(a megabyte is
1,000,000 bytes) |
|
|
|
|
|
Link rate 1536 |
|
|
|
|
|
NT4 to NT4 using FTP |
|
|
|
|
|
Packet Loss one
way |
|
|
|
|
|
Ping time |
10ms |
80ms |
300ms |
600ms |
|
No pkt. loss |
10.70 |
4.65 |
1.51 |
0.76 |
|
1 in 100 |
9.83 |
3.86 |
1.17 |
0.66 |
|
1 in 50 |
9.90 |
3.20 |
1.05 |
0.54 |
|
1 in 25 |
9.95 |
2.59 |
0.89 |
0.47 |
|
1 in 12 |
6.06 |
1.50 |
0.48 |
0.26 |
|
4 in 100 |
7.10 |
2.88 |
1.03 |
0.54 |
|
8 in 100 |
1.74 |
1.10 |
0.60 |
0.38 |
|
3/10/00 |
|
|
|
|
|
Sending a 3.22
Meg file |
|
|
|
|
|
Throughput in
Megabytes/minute |
|
|
|
|
|
(a megabyte is
1,000,000 bytes) |
|
|
|
|
|
Link rate 1536 |
|
|
|
|
|
NT4 to NT4 using Gigabyte Express |
|
|
|
|
|
Packet Loss one
way |
|
|
|
|
|
Ping time |
10ms |
80ms |
300ms |
600ms |
|
No pkt. loss |
10.08 |
8.20 |
3.41 |
1.93 |
|
1 in 100 |
9.99 |
6.08 |
2.08 |
1.15 |
|
1 in 50 |
9.75 |
4.73 |
1.50 |
0.80 |
|
1 in 25 |
9.87 |
2.88 |
0.95 |
0.51 |
|
1 in 12 |
6.30 |
1.28 |
0.56 |
0.26 |
|
4 in 100 |
9.67 |
3.79 |
1.26 |
0.58 |
|
8 in 100 |
5.02 |
2.24 |
0.97 |
0.49 |
NIWOT
NETWORKS, INC. OVERVIEW
Company
History:
Niwot
Networks was formed in 1988 to provide high speed communication boards and
drivers for pc-based routers, with an emphasis on customer-supported product
enhancements and bootstrap financing.
These first routers were Novell-based and customer complaints about
performance over thousand-mile distances led to the development of DFT (Direct
File Transfer) which provided full theoretical throughput over T1 terrestrial
lines, and DFT/Mac for full theoretical throughput over T1 satellite links. With the addition of on-the-fly compression
and support of Internet Protocols these products evolved into today’s Gigabyte
ExpressÔ for Windows and
Macintosh. These are simply the fastest
long-haul file transfer products available today, and Niwot hardware is no
longer required to achieve full throughput.
While
making Gigabyte Express the fastest way to move files over the Internet, Niwot
found that packet loss and delay on the Internet hobbles the performance of all
Internet applications. This performance
can be dramatically improved with Niwot’s RELIA technology(patent pending).
Gigabyte Express™
Gigabyte
Express file transfer software has been available since 1995, and today is
offered for Windows and Macintosh platforms.
Gigabyte Express is the fastest way to move large files over long
distances for PC-to-PC, Mac-to-Mac, or PC-to-Mac transfers. Niwot’s Gigabyte Express is now the easiest,
fastest, and most trustworthy way to move large files over the Internet and
private Intranets. The product features
on-the-fly lossless compression licensed from STAC. Our customers typically report throughput of 2 to 5 times FTP.
In
addition to lossless compression and high performance, Gigabyte Express™ File Transfer offers features for ease of
use, unattended operation, and control of the amount of bandwidth used.
Niwot’s RELIA™ technology is an extension to TCP which adds redundant information that enables the receiver to reconstruct lost information without requiring the time delay of retransmission. The first application to support RELIA is Niwot’s Gigabyte Express for Windows version 5.0 RELIA™ technlogy is especially beneficial over international and satellite Internet Protocol links.