The Truth about KERMIT NEWS
Soon after ZMODEM was introduced in 1986, protocols claiming to outperform ZMODEM appeared almost monthly. Most have faded into obscurity, but a few have found niches in specialized applications and proprietary environments.
Columbia University designed Kermit for the sixties' generation of mainframes and the pioneering networks that supported them. These design tradeoffs earned Kermit a reputation for slow file transfers. Indeed, it was slow XMODEM, YMODEM, and Kermit file transfers that prompted GTE Telenet to fund the development of 1986 ZMODEM.
Since then, advances in computing and networking have largely supplanted the legacy mainframes that needed Kermit. With interest and funding waning, it was time for Columbia's Kermit group to address different markets.
Columbia changed the Kermit specification to allow hackers to tweek the protocol for speed. Permission for developers to include Columbia code in their programs was quietly withdrawn. Columbia's Kermit group had become a nonprofit business unit.
Columbia revealed this change in its July 1993 issue of Kermit News. This promotional issue began with an announcement that Columbia University now accepts Credit Card orders. Under the heading "Kermit Saves You Money" Columbia University invited businesses to switch from other vendors' shareware and commercial software to Columbia's product. Columbia used its non profit mail permit to distribute this promotional literature to a large mailing list. Their taxpayer subsidized mail permit gives Columbia a crucial advantage over competitors.
In The Truth about Kermit File Transfer Performance Columbia University proudly announced that Columbia's Kermit software "outperforms X-, Y-, and ZMODEM protocol transfers every time" (Kermit News p.14). To lend credence to this assertion, Kermit developer Frank da Cruz supplied results of "True-Life Benchmarks" performed at Columbia University.
One would expect benchmarks published by Columbia University to be fair and accurate. Unfortunately, these benchmarks leave much to be desired.
The documentation for sz 3.24 states:
LICENSE AGREEMENT
This is a copyrighted shareware program. Commercial use
of this program is subject to licensing conditions
detailed in the rz.c source code. "Commercial Use"
includes use of this program to send files to any
commercial or shareware program not published by Omen
Technology INC.
Using sz to send files to
Procomm and Telix (both commercial programs) for
the specific purposes of writing a promotional piece to increase
market share is as commercial as one can get.
Had Columbia University honored Omen's Copyright,
Omen Technology could have advised on the benchmarking,
ensuring a fair and accurate report instead of the misinformation
that was published.
XMODEM and YMODEM
transmit file data verbatim, without any control character
prefixing or other modifications.
Data patterns do not
affect X/YMODEM throughput on direct connections.
Under these conditions,
X/YMODEM throughput is affected only by the file length.
The time required for start-up and end of file processing is relatively
less important for
longer files.
Deviations from the trend of
these data points are small. In the absence of modem
compression, line noise, and other perturbations, a
stable protocol runs as smoothly as clockwork.
Columbia's data plotted above show relatively large random variations. These variations, which are not a function of file size, show an experimental error of more than 20 per cent.
"The results given in the article are exactly as
reported by the software, and each trial was run three
or more times. Theory is fine, but in the real world
all sorts of factors can cause variations."
But 20 per cent error in controlled benchmark tests?
These tests did not involve any modems or telephone lines!
How real is a 20 per cent jitter in real life testing?
I once ran a series of downloads of b17mh.gif.
I was testing
Omen Technology's Professional-YAM running in a Windows
3.11 DOS Box.
A total of 100 megabytes
were sent
in 291 downloads
over a 16 hour period.
During this time Omen's Unix system performed its normal
activities, including netnews processing, information utility
downloads, database processing, and a 1.2 gigabyte system
backup. Even with all this activity there was only a 6 per cent
difference between the fastest and slowest download. Excluding
the top and bottom one percentile, the difference was only 3 per
cent. Columbia suggests that all sorts of unstated external
factors can cause the large variations only they see.
Columbia's Kermit significantly understates the time it takes to
pull down a file. In a typical case, Kermit reported a file transfer
time of 112.98 seconds when in fact the machines were tied up more
than nine seconds longer. Professional-YAM's ZMODEM self
reported transfer time was much more accurate, understating the
actual time by less than one half second.
Some explanation is in order here. Based on recommendations by Kermit's authors, I prepared a C-Kermit initialization file to replicate Columbia's benchmark tests. Initialization files are vital to Columbia Kermit; without a customized initialization file Kermit will not provide fast and reliable file transfers.
set terminal bytesize 8 set parity none SET BUFFER 30000 30000 ; Allocate packet buffers set send packet-length 5000 set receive packet-length 5000 set window-size 6 set block 3 set transfer cancelation off set file type binary set delay 0 set control unprefix all set control prefix 0 1 129 define asend echo START, apc rec, send \%1, echo DONE, exitI timed the C-Kermit/MSKermit combination with the shell script "tks":
kermit quit >/dev/null 2>&1 time kermit -C "asend $@"
The first line in the shell script ensures that C-Kermit and its initialization file are resident in the disk cache for instant access for the timing run. Compared to sz, Unix C-Kermit is a massive program that takes some time to load. Frank da Cruz doesn't believe this real-world overhead should be counted in "True-Life Benchmarks", so I agreed to handicap the race for Kermit's benefit.
When the command "tks uuencode" is given, MSKermit immediately switches to its file transfer screen.
Receiving: In progressAfter some seconds data transmission starts. After transfers complete, MSKermit returns to terminal emulation. A further second or two passes before the "DONE" message appears as Unix Kermit finishes. Using Columbia's uuencode benchmark file, MSKermit 3.13 reports a file transfer time of 4.62 seconds. Timed with a stopwatch, 13.7 seconds pass between the appearance of the Kermit file transfer screen and the "DONE" message. The Unix time command reported 13.87 seconds. Columbia would have you believe the 4.62 second figure, never mind the fact that Columbia Kermit really takes 14 seconds to get the job done.
When this issue was raised on the Usenet comp.dcom.modems newsgroup, Kermit disciples acknowledged this overhead. They could not suggest ways to get rid of this Kermit overhead. They merely suggested we ignore this Kermit overhead to make the tests "fair" for Kermit. This "True-Life Benchmarks" mind set is not a real world attitude.
Considering this measured Kermit overhead, the claimed 3 to 9 second Kermit file transfers reported in Tables 2-5 must be viewed with suspicion.
To avoid cheating, file transfer times should be measured with the Unix "time" command and verified with a stopwatch.
Some of Columbia's results were simply so preposterous I could not resist having a wee bit of fun with them. We noticed that cutting the communications speed from 19200 to 9600 and introducing noise actually increased the speed of a Columbia Kermit file transfer from 2648 to 2736 characters per second (Kermit News, p.13, Tables 2 and 5). This is the moral equivalent of lowering the speed limit by half and installing speed bumps every block only to find the same trip takes less time than before. When I announced the "Columbia Stochastic Telepathic Kermit Hyperprotocol" to "explain" Columbia's "True-Life Benchmarks" the Kermit faithful rather lost their sense of humor. They flamed about the "news release" but could not explain how Columbia came up with such crazy results. Frank's response: "This is indeed what happened. Explain it any way you like."
To avoid a fair comparison with ZMODEM,
Columbia selected files with redundant data
that respond unusually well to Kermit's run length compression.
Selecting the uuencode Sun Sparc binary for a benchmark took some chutzpa. Uuencode is a tiny program stored in a 24576 byte file. All but a few thousand bytes are nulls. (Before you complain to Sun about the inefficiency of their executable format please note that this is an atypical file!)
Naturally this preponderance of empty nulls does wonders for Kermit transfer speeds. Had Columbia's protocol boffins read the sz 3.24 documentation and followed its instructions, they would have discovered that ZMODEM compression does even better. On direct connect transfers, Kermit gets 320 per cent efficiency on this file; ZMODEM compression gets an even better 363 per cent efficiency. A separate compression program does even better at 646 per cent, twice as good as Kermit.
Less egregious, but still atypical, is the selection of kermit.exe as a benchmark file. As the above chart shows, kermit.exe is more compressible than stsplus.exe, whose compressibility is typical of DOS executables.
For fairness, I included ZMODEM-90TM compression in the chart above. To quote Frank dz Cruz, "The results speak for themselves."
B17mh.gif
is typical of compressed files. It has a flat
distribution of codes. Its size is ideal for high speed protocol tests.
The picture is a B17 bombardier's view shortly after takeoff
that instantly convinced a Doubting Thomas the file was legitimate.
So why is Kermit not the fastest when it only prefixes a few control characters? Because Kermit prefixes more than you tell it to. Several other characters are quoted by Kermit itself. Columbia claims that only 0, 1, and 129 need to be prefixed, but this doesn't work sending to MSKermit 3.11. The following recommendations made by Frank da Cruz do work:
SET CONTROL UNPREFIX ALL
SET CONTROL PREFIX 1 13 129 141
This makes C-Kermit prefix 8 characters, 0, 1, d, 7e, 81, 8d, a3, and fe (hex). One can understand why YMODEM-g is fastest since it does not prefix ANY control characters. MobyTurboTM ZMODEM is next with one control character prefixed, followed by vanilla ZMODEM which prefixes 5 control characters (7 when dialing out). Columbia's Kermit-94 not only prefixes 8 characters, but suffers from the startup and shutdown delays mentioned above.
What is not reasonable is the incredibly low 66% efficiency Columbia reported for an ASCII ZMODEM transfer. Standard ZMODEM is 99% efficient transferring ASCII files. ZMODEM does not prefix any of the characters appearing in ASCII files, and the CRC-32 protocol overhead is about 6 out of every 1024 characters (0.6%). A standard ZMODEM single file transfer involves 5 exchanges for a total delay time of about one second. So how did Columbia get ZMODEM to take more than a minute longer than expected to transfer IKCKER.DOC than expected? Columbia reported "the connection is clean and no errors occurred". But their results are definitely in error.
Click here for a larger graph.
Contrary to Columbia's claims, ZMODEM-90TM automatically detects 7-bit paths. Few applications involve true 7-bit paths, but ZMODEM-90TM works well here also.
ZMODEM-90TM optimizes performance with two 7-bit submodes:
The most impressive test at the ShootOut, and the likely reason nobody from Columbia University chose to appear, was Kermit's spectacular total failure in the noise test. After all Frank da Cruz has published about alleged ZMODEM reliability problems it would have been interesting to witness his embarrassment when Kermit decisively failed Frank's very own test, correctly executed.
Columbia's "True Life Benchmark" noise test consisted of a noise burst every two seconds on a 9600 bps modem link. This can be caused by a "clock slip" between telephone central offices. BBS users rarely see such line noise on 9600 and faster connections because today's high speed modems provide link level error correction. With today's modems, line disturbances manifest themselves as slow throughput and disconnects, not gibberish on the screen. But Columbia University lives in a different time period; Frank da Cruz claims clock slips are a common problem on New York's phone lines.
The Kermit News "True-Life Benchmarks" specified a packet length of 5000 to minimize Kermit's high per packet overhead. When we attempted to replicate Columbia's 9600 bps plus noise test, Kermit failed every time. When Kermit uses the specified 5000 byte packets with noise bursts every 2 seconds, no data packets get through. Why? It takes 5 seconds to send a 5000 byte packet at 9600 bps. Under Columbia's test conditions, every packet would suffer at least 2 noise bursts, and none could get through. Unlike ZMODEM, Kermit cannot resend a packet with a different size, so Kermit has to resend the same 5000 byte packet over and over until it gives up. This is a fundamental limitation of the Kermit protocol.
Columbia's Frank da Cruz has not been able to explain why his private tests showed Kermit working well under these mathematically impossible conditions. He still insists Columbia's tests were honest and fair, but neither he nor Columbia University have agreed to repeat them in public.
Assuming Columbia's tests were not completely faked, the only
explanation is that their results are merely an artifact of the
two-line terminal server and flawed assumptions used in
Columbia's "True Life Benchmark" tests. This lightly loaded
server buffers Kermit packets at 10000000 bps Ethernet speeds
instead of the stated 9600 bps modem speed.
Frank da Cruz has
admitted that the noise bursts Columbia used in their tests were
not
injected into the modem's analog data stream as a real world clock
slip would be. Instead the noise bursts were injected into the
buffered Unix data stream.
Because Unix C-Kermit buffers packets and outputs each in a single
atomic operation, the Unix buffering forced the noise bursts to go
BETWEEN Kermit packets where they would not cause errors very often.
This artificial redistribution of noise bursts also explains the
better than expected XMODEM and YMODEM performance Columbia
reported. (Normally, ZMODEM transfers do not benefit from such
artificial conditions.)
I repeated Columbia University's "True Life Benchmark" tests according to the description published in Kermit News. I was not able to replicate Columbia's results because the the noise bursts I added were at equal intervals as they would be with a real world two second clock slip. Kermit failed this test every time the test was run according to Columbia's published conditions.
Few users see such conditions today. With higher speed error correcting modems selling for less than a speeding ticket, real world conditions bear little resemblance to Columbia's tests.
Although Frank da Cruz did not read the sz documentation carefully enough to give ZMODEM a fair chance, he did read a 1988 ZMODEM Protocol Overview carefully enough to complain about the document's comments on Kermit. This old document mentions that ZMODEM is not optimized for the rare situations where delays and error rates are both high. It is no surprise Frank attempted a supremely unrealistic test that simulates conditions rarely seen today.
In contrast, ZMODEM is optimized for the real world.
I made an informal survey of the guests at the Protocol ShootOut. As so many have noted, today line impairments manifest themselves mostly as interrupted calls. When this happens ZMODEM's Crash Recovery is useful, allowing resumption of interrupted file transfers. Many users think Crash Recovery is ZMODEM's most important feature. Crash Recovery was an integral part of ZMODEM since its introduction in 1986. Since then Omen Technology has refined Crash Recovery to a high degree of reliability. Since Kermit was not designed to support Crash Recovery, Columbia chose not to discuss it, instead relying on bogus tests incorrectly simulating conditions rarely seen today.
What ZMODEM detractors don't know or won't tell you about file transfers may hurt you.
-Chuck Forsberg
Since these points were first brought up, Columbia University's Frank da Cruz has repeatedly attacked the speed and reliability of ZMODEM in his Internet postings. He continues to reference Columbia's defective "True-Life Benchmarks".
In return, I have challenged him to allow me the same courtesy I offered him, to monitor a repeat of Columbia's "True-Life Benchmarks". This would seem a reasonable request, considering Columbia started this controversy by mailing misleading sales literature to many thousands of purchasers at public expense. At each challenge, Frank either disappears, changes the subject, or offers a lame excuse. I suspect he knows exactly who would be embarrassed by a fair and honest repeat of the so-called "True-Life Benchmarks".