Recent Developments in ZMODEM


By Chuck Forsberg

In early 1986, Telenet funded a project to develop a new file transfer protocol to alleviate the throughput problems network customers were experiencing with XMODEM and Kermit file transfers. Designing ZMODEM from scratch allowed me to use the best ideas from X.PC, HDLC, BISYNC, Kermit, and dozens of other protocols, while avoiding many of their shortcomings.

Since then ZMODEM has been incorporated into hundreds of programs. ZMODEM's speed, reliability, and ease of use has made it the protocol of choice for thousands of bulletin boards, GEnie, BIX, Portal, Delphi, and other information utilities.

From the beginning, ZMODEM was designed to be extended. Important reliability, performance and compatibility extensions have been developed since the Telenet project. These extensions accelerate downloads by 5 to 30 percent in many applications, with some files downloading many times faster.

New programs can exploit these extensions without sacrificing downward compatibility with older programs. This article summarizes the most important of these ZMODEM extensions.

CRC-32

Ease of implementation was one of ZMODEM's original design goals. XMODEM CRC technology was used because XMODEM routines were widely available in many programming languages.

XMODEM's CRC polynomial is many times less reliable than good 16 bit CRC's. This wasn't an issue with XMODEM because XMODEM itself was inherently unreliable under stress. When ZMODEM's speed and "bullet-proof" robustness soon found use in applications too harsh for XMODEM, ZMODEM's robustness revealed XMODEM CRC to be too inaccurate.

The first major extension to ZMODEM was the adoption of 32 bit CRC.

ZMODEM uses the 32-bit CRC specified by ANSI X3.66, FIPS PUB 71, and FED-STD-1003. A table driven calculation keeps processing overhead low. ZMODEM CRC-32 is five orders of magnitude more accurate than XMODEM CRC, and billions of times more sensitive than 1 byte XMODEM and Kermit checksums. The extra protection of CRC-32 is vital in high speed applications.

ZMODEM retains XMODEM CRC capability to accommodate old programs.

Compression

Compression techniques compact the redundant information in data files to reduce storage and transmission time. ZMODEM-90TM extensions include "on the fly" compression that boosts throughput on listings and other suitable files. Compression percentages range from -1% on already compressed files to more than 1000 percent (10 times speedup) on the Personal Computing Magazine text file benchmark.

MobyTurboTM Accelerator

Many files downloaded from bulletin boards are already compressed with PKZIP or other compression programs.

ZMODEM was developed to operate over packet switched networks that use control characters for network control. When one of these characters appears in the data, ZMODEM protects the network by replacing the offending character with a two character sequence (quoting). Standard ZMODEM quoting increases overhead on compressed files by about 3 percent.

The protection of control character quoting and its overhead are not needed in many applications. ZMODEM-90TM offers MobyTurboTM to close the speed gap between ZMODEM and less reliable protocols without sacrificing ZMODEM's historical robustness and reliability. MobyTurbo reduces the character quoting overhead on compressed files to 0.5%. In comparison tests the speed difference between YMODEM-g and MobyTurboTM is less than 1 percent. Many users feel 1% is a low price to pay to get Crash Recovery, automatic downloads, and the safety of 32 bit CRC error checking,

	      _____________________________
	     |      OVERHEAD FACTORS      |
	     |     (Compressed Files)     |
	     |_____________________|______|
	     | 1024 Byte Subpackets|   .5%|
	     | Character Quoting   |  3  %|
	     | MobyTurbo Quoting   |   .5%|
	     |_____________________|______|

Reduced Overhead

Some networks and modem concentrators reserve control characters not protected by ZMODEM defaults. Previously the solution was to protect all control characters, resulting in protocol overhead approaching that of Kermit.

ZMODEM-90TM extensions provide individual control of the protected control characters, avoiding the high overhead of quoting all control characters.

Window Management

Many information utilities are accessed via complex packet switched networks. These networks may behave more like a balloon than a pipe as they pass data from a fast mainframe to a relatively slow modem. Undelivered kilobytes swell the network's memory banks, and error correction is impaired.

ZMODEM allows the sender to limit this ballooning by regulating the rate of transmission to accommodate the slowest segment of the network. ZMODEM accomplishes this by commanding the receiver to acknowledge data sub-blocks as they are received, and waiting for more acknowledgements when the receiver is too far behind (window too large).

The optimum window size depends on the network characteristics and modem error rate. With ZMODEM-90 the receiver can override the sender's window size according to local conditions. Users with error correcting modems can increase throughput by increasing the window size.

These ZMODEM-90 extensions give GEnie downloads 5 to 30 percent faster than public domain ZMODEM.

7-BIT Environments

A pair of ZMODEM-90 programs with 7-bit support will automatically detect a 7-bit environment and switch to one of two 7-bit path compatible encodings.

The default 7-bit ZMODEM uses RLE compression and 8th bit quoting. This mode resembles Kermit encoding, but is more efficient because the RLE encoding is optimized and fewer control characters are quoted. This default is well suited for text files.

ZMODEM Pack-7 packs 4 bytes into 5 printing characters. This is more efficient than quoting for sending compressed files. ZMODEM Pack-7 is efficient enough to beat Kermit on ZIP files, even when Kermit is allowed to use all 8 bits.

A 75136 byte ZIP file was sent between two adjacent machines directly connected at 2400 bps. These tests demonstrate the difference in protocol performance under ideal conditions.

	   ___________________________________
	  |            8-Bit line            |
	  |____|_____________________________|
	  | cps|  Protocol                   |
	  |____|_____________________________|
	  | 182|  Kermit long packet=2000    |
	  | 231|  ZMODEM (defaults)          |
	  | 237|  ZMODEM-90TM MobyTurboTM|
	  |____|_____________________________|

	     _______________________________
	    |          7-Bit line          |
	    |____|_________________________|
	    | cps|  Protocol               |
	    |____|_________________________|
	    | 156|  Kermit long packet=2000|
	    | 190|  ZMODEM-90TM PACK-7   |
	    |____|_________________________|
Kermit transfers used one byte checksum and 2000 byte packets. ZMODEM used 32 Bit (four byte) CRC.

Intelligent Crash RecoveryTM

Crash Recovery allows an interrupted file transfer to be completed without throwing away the portion transferred before the interruption. Crash Recovery has been a favorite ZMODEM feature since 1986.

Accurate crash recovery requires that the receiver's copy of the file match the sender's copy up to the point where the transfer was cut off. If you don't call back instantly the file may change, and simply resuming the transfer will corrupt the file. If this is a concern, choose a program that verifies the accuracy of Crash Recovery.

Intelligent Crash RecoveryTM (-rr option) allows files to be compared without transmitting the actual file contents. The sender and receiver take a 32 bit CRC on the files and compare those numbers. The "zmodem R" parameter controls how much of the files to compare. The default of 0 compares the entire file. The chance of different files producing the same 32 bit CRC is small; an independent comparison of another segment further enhances reliability.

Currently, ASCII file translation (Unix to DOS format, etc.) does not allow Crash Recovery.

A future extension should remove this restriction.

Total Transfer Display

ZMODEM and YMODEM now support a count of the files remaining and their total size to the receiver. This allows the receiver to estimate remaining transfer time, updated as conditions change. This information is optional and may not be provided when its collection would cause an unacceptable delay starting a transfer.

VAX/VMS Specific Programs

Previous versions of the RZ (Receive ZMODEM) and SZ (Send ZMODEM) file transfer programs were poorly adapted for Digital Equipment's VAX/VMS minicomputer operating system.

A new VMS RZ writes variable length CR terminated records for ASCII files and fixed 512 byte records for binary files. These record formats are better suited for standard VMS utilities.

A -i option may be given to the sender or receiver to force VMS Stream_LF record format. Stream_LF format preserves the exact data and file length. Stream_LF is suitable for C programs and PC oriented file server and archive applications.

VMS SZ now supports wild cards and subdirectories.

These programs are available from Omen Technology Incorporated.

ZMODEM-90TM, MobyTurboTM, and Intelligent Crash RecoveryTM are Omen Technology trademarks.

>>>Return to the Omen Home Page.