Back in the early 80's I couldn't get the speed I wanted downloading files from Unix to PCs using YMODEM. Increasing the YMODEM block size to 1024 helped, but downloads were still too slow. Moving a file fifteen feet shouldn't take so long. So I developed YMODEM-g. (The "g" in YMODEM-g stood for "go-go-go!")
YMODEM-g gets its speed advantage over YMODEM by sending all the data blocks without waiting to hear whether the receiver actually got any of them. YMODEM-g blindly bets its bippy on success. By assuming an absolutely error-free data link, YMODEM-g screams when conditions are perfect.
YMODEM-g was a simple ad hoc hack to increase transfer speed by ignoring traffic lights. YMODEM-g sends the data on a screaming joy ride, cutting corners on two wheels. Unfortunately, the slightest hiccup spills the whole game.
When YMODEM hit the boards in 1985, YMODEM-g was but a little read chapter in ymodem.doc. As time passed, the proliferation of error correcting modems encouraged many programmers to offer YMODEM-g. Some of these creations were better than others. Some didn't even bother to follow the YMODEM-g protocol, they just borrowed the name. (Maybe more of them would have gotten it right if I'd put the YMODEM-g description near the beginning of the document.)
Unfortunately, YMODEM-g is not an error correcting protocol. YMODEM does not know how to back up to the last good block, so death is YMODEM-g's one and only response to transmission errors. A single fly in YMODEM-g's ointment destroys everything. Seemingly innocuous aberrations such as disk drive retries, slower than usual TSR interrupts, or interference from display activity can abort a YMODEM-g transfer. Compared to ZMODEM, YMODEM-g comes with a price.
As I write this, line crews are untangling power lines from fallen trees. During the storm, power flickered dozens of times at Omen Technology's Sauvie Island World Corporate Headquarters. Many TeleGodzilla downloads were cut off in the middle of a file. ZMODEM users just called back and resumed their downloads from where they were disconnected. XMODEM and YMODEM users, many of whom were downloading DSZ, no doubt wish they already had ZMODEM.
Crash Recovery was not discussed when I began work on ZMODEM back in 1985. While contemplating ZMODEM technology with friends and associates, I was told the BLAST protocol was supposed to allow recovery of interrupted file transfers. That was such a neat idea we decided then and there to develop a working Crash Recovery for ZMODEM.
Two conditions must be met for successful crash recovery. First, the receiver must tell the sending program where in the file to pick up the transfer. Adding crash recovery to other protocols requires extra logic because there is no simple correspondence between protocol packet number and location within the file. Adding crash recovery to ZMODEM was easy because ZMODEM already "thinks" in terms of the actual file position.
Successful crash recovery also requires that the receiver's copy of the file match the sender's copy up to the point where the transfer was cut off. If your transfer is cut off and you dial right back in, you can be fairly confident the sender's copy of the file hasn't changed. If you don't call back right away the file may have changed. Omen Technology products check for this by comparing a portion of both files before allowing a resumption from where the receiver's copy left off. If the comparison detects a difference between the two files, the transfer starts over to guarantee an undamaged file. Skimping on this safety check may result in a patchwork quilt made from pieces of unrelated files.
Omen's Intelligent Crash RecoveryTM (-rr option) allows more or all of the files to be compared without transmitting the actual file contents. The sender and receiver take a 32 bit CRC on the files and compare those numbers. The "zmodem R" parameter controls how much of the files to compare. The default of 0 compares the entire file. The chance of different files producing the same 32 bit CRC is small; an independent comparison of another segment further enhances reliability.
Why is YMODEM-g (when it works) faster than ZMODEM on ZIP files?
Over the years 128 byte XMODEM blocks grew into YMODEM-1k 1024 byte blocks. 96 byte Kermit packets stretched out to thousands and thousands of bytes. People learned that long blocks push data faster than short blocks. But when it comes to ZMODEM, everything you know about block length is wrong.
In the context of start-stop protocols such as XMODEM, YMODEM and JMODEM, long blocks do indeed move data faster than short blocks. Streaming ZMODEM takes this a step further with an infinite block length, bound only by the length of the file.
To make this article easy to understand, I've left out many important "details".
ZMODEM splits data into 1024 byte subpackets for two reasons. First, subpackets allow the receiver to verify the accuracy of received data without tying up a lot of memory.
Second, and more important, subpackets permit ZMODEM to check for errors often without incurring the high overhead of conventional packets. When modems make errors, longer packets waste time because it takes longer to detect errors and more bytes must be discarded to correct the errors. ZMODEM can change the subpacket length to speed up error recovery without incurring the extreme overhead of shorter packets. When the error rate rises to once every several seconds (not a terrific connection, but far from the worst possible), ZMODEM can reduce the subpacket length to 128 bytes or less. This generates an overhead of 5 per cent,
whereas 128 byte XMODEM blocks can generate 170 per cent or more overhead on a V.32 connection.
The overhead imposed by 1024 byte ZMODEM subpackets is one half per cent. Changing this to 8192 would reduce overhead to one tenth per cent. There isn't that much difference between 0.005 and 0.001. It isn't going to make a noticeable speed difference except to give error recovery as slow as JMODEM and Super8K.
______________________________________________________ | OVERHEAD vs LENGTH | | (Length Related Factors) | | (9600 bps V.32 Connection) | |_______|_____________________________________________| | LENGTH| OVERHEAD | |_____________________________________________________| | XMODEM | |_______|________|____________________________________| | 128 | 170% | **********************************| | 1024 | 20% | **** | |_____________________________________________________| | ZMODEM | |_______|________|____________________________________| | 128 | 5% | | | | 1024 | .5%| . | |_______|________|____________________________________|But I digress. YMODEM-g can't recover from line hits anyway. So why does YMODEM-g beat ZMODEM on ZIP compressed files?
ZMODEM was developed to operate over packet switched networks. Several control characters are reserved for controlling these networks. ZMODEM is "network friendly" because it does not use these network control characters. This is why ZMODEM works in many applications where XMODEM and YMODEM cannot be used. When one of these characters appears in the data, it is QUOTED. When ZMODEM quotes a character, it sends two characters in its place. This coding increases the number of bytes the modems must transmit when sending a file. ZMODEM quoting increases overhead on compressed files by about 3 per cent. The quoting overhead is several times the overhead generated by 1024 byte subpackets. To speed up ZMODEM, we must quote fewer characters.
_____________________________ | ZMODEM OVERHEAD FACTORS | | (Compressed Files) | |_____________________|______| | 1024 Byte Subpackets| .5%| | Character Quoting | 3 %| |_____________________|______|ZMODEM-90TM adds MobyTurboTM (ZMODEM is boffo turbo by itself) to close the speed gap between ZMODEM and less reliable protocols without sacrificing any of ZMODEM's reliability. With MobyTurbo, ZMODEM-90TM does not escape network control characters. MobyTurbo reduces the character quoting overhead on compressed files to one half per cent. The total overhead is about one percent on long ZIP files. In comparison tests with YMODEM-g, the difference between YMODEM-g and MobyTurbo was generally less than the precision of the measurements, obscuring the theoretical 0.005. MobyTurbo's performance should satisfy even the most enthusiastic YMODEM-g fans.
MobyTurbo can be used in most applications that allow ZMODEM. Any environment that supports YMODEM-g will also support MobyTurbo. Thanks to 32 bit error checking and ZMODEM's famous "bullet resistant" error recovery, MobyTurbo does not require error-correcting modems.
MobyTurbo supports the full ZMODEM feature set including AutoDownload, Crash Recovery and selective file transfers.
MobyTurbo is available in Omen Technology (DSZ, ZCOMM and Professional-YAM) and Aladdin products.
Unfortunately, YMODEM-g is not an error correcting protocol.
YMODEM-g's one and only response to transmission errors is death.
Everything you know about block length is wrong
MobyTurbo's performance should satisfy even the most enthusiastic YMODEM-g fans.
ZMODEM-90TM, MobyTurboTM, and Intelligent Crash RecoveryTM are Omen Technology trademarks.