Almost all the cryptography we use every day – the kind that protects your bank account, your messages, your connection to this very site – rests on a bet: that certain mathematical problems are too slow or too hard to solve. Factoring six-hundred-digit numbers, computing discrete logarithms over elliptic curves: nobody today knows how to do it fast enough, and so we sleep soundly. But it's a peace with an expiry date, because “hard” is never “impossible”. The hardware changes, a cleverer algorithm comes along, quantum computing looms on the horizon, and the bet has to be renegotiated from scratch.

There is one, single exception. A cipher that isn't hard to break: it's impossible, and not in the advertising sense of the word. There's a proof, a theorem in the full sense. It's called the one-time pad, the single-use notebook. In a world full of threats to privacy and of governments playing “big brother” without knowing the basics of modern cryptography, it's worth telling the story of this cipher and of how, for a century, the real world has found a way to sidestep it every single time without ever scratching it.

An invention from 1917

It all begins in a room at AT&T.

In 1917 Gilbert Vernam, an engineer at Bell Labs, invents a cipher to protect teleprinter communications. Vernam's idea is elegant and simple: electrically combine each character of the message with a character read from a punched paper tape – what we can call “the key”. The combination is a reversible operation: whoever holds the same tape undoes it by running it backwards and recovers the message; whoever doesn't have it sees only noise. If the tape says one thing and the message says another, what comes out on the wire is a sum of the two, unreadable to anyone without that tape.

The structure is already very solid, but the conceptual leap is made by a soldier. Captain Joseph Mauborgne, of the U.S. Army Signal Corps, realises that if the sequence on the tape were completely random, cryptanalysis would become enormously harder. It's the intuition that turns Vernam's cipher into the true one-time pad: not just “any” key, repeated or derived from a phrase, but a truly random key, as long as the message and used only once. Hence the name: a notebook of sheets, each sheet a key, each sheet torn off and destroyed after use.

The intuition is right, but it hangs unproven. For a quarter of a century no one manages to explain why it works so well: the proof is missing that the system is not merely very robust, but mathematically unassailable.

Two lines running in the same year

Here the story forks, and the two lines that set off run in parallel for the rest of the tale. On one side there's the theory, which between 1941 and 1949 arrives at locking down the cipher with a proof. On the other there's the practice, which in those very same years begins to sabotage it. It's worth keeping both in view, because the whole point of the affair lies precisely in the distance between the two.

The line of the proof. The proof comes from the man who, in those same years, is inventing the entire theory of information from scratch: Claude Shannon. The special status of the Vernam-Mauborgne pad is established by him, about twenty-five years after its invention. In his Communication Theory of Secrecy Systems – circulated as a classified report in 1945 and published openly in 1949 – Shannon proves that the one-time pad is unassailable because every possible plaintext is equally probable given any ciphertext. This property is called perfect secrecy.

But Shannon isn't the first to get there. The Soviet mathematician Vladimir Kotelnikov – the same one behind the sampling theorem – had independently proved the absolute security of the one-time pad as early as 1941, in the very months of the Nazi invasion of the Soviet Union, in a report that, as far as anyone can tell, remains classified to this day. Unable to share it with the Allies, Shannon would have to rediscover the same properties on his own a few years later. The USSR, then, possesses the mathematical proof that its own code is unbreakable earlier and better than anyone else.

The line of the sabotage. In the very same span of time in which it deposits that proof, the Soviet apparatus begins to demolish it with its own hands. Between late 1943 and the end of the conflict, the surge in communications traffic and the relocation of Soviet industry towards the Urals push the personnel to reuse the cipher notebooks. To this is added the often-cited hypothesis of a more concentrated episode: with German troops just outside Moscow between 1941 and 1942, staff at the Centre are said to have printed more than one copy of the same key material. Immediate contributing cause or prolonged drift, the result is the same – and fatal, namely: the moment the same key enciphers two different messages, the one-time pad ceases to exist and becomes an ordinary cipher, attackable with statistics.

Let's stop a moment, because this is the crack that everything else passes through, and it's worth seeing up close. Call M₁ and M₂ two plaintexts, K the reused key, and let's use XOR (addition without carry, the operation the pad uses to combine key and text). The two ciphertexts are C₁ = M₁ ⊕ K and C₂ = M₂ ⊕ K. An attacker knows neither the messages nor the key: they know only C₁ and C₂, fished out of the ether. But if they combine them with each other, something remarkable happens:

C₁ ⊕ C₂ = (M₁ ⊕ K) ⊕ (M₂ ⊕ K) = M₁ ⊕ M₂

The key cancels out. It vanishes, because K ⊕ K is zero. What's left to the attacker is the XOR of the two plaintexts, with no trace of the secret remaining: and from there, knowing the structure of the underlying code and the statistics of the language – which letters and words recur, where the spaces fall – one begins to separate the two messages and reconstruct them. The perfection of the one-time pad lies entirely in the “one“: one key, one message. Reusing it a second time doesn't weaken it a little. It cancels it.

Theory and sabotage, the proof and its demolition, in the same country and in the same years. It's the irony on which the entire story of VENONA rests.

VENONA: forty years inside a crack

On the other side of the ocean, someone finds that crack. In 1943 the U.S. Army Signal Intelligence Service launches an ultra-secret project against Soviet diplomatic traffic, believed impregnable precisely because it was enciphered with a one-time pad. It changes name a couple of times – JADE, then BRIDE – before settling on a word chosen at random, with no meaning: VENONA.

For a year, near-total darkness. The first breach is opened, in late 1943, by Lieutenant Richard Hallock, who works on Soviet commercial traffic – the so-called “Trade”, the messages dealing with matters of exchange – and notices that the Soviets are reusing the pad pages. Hallock and his colleagues recover a significant quantity of key tables, but it's only the outer layer of an onion. The messages aren't enciphered with the pad alone: they are first converted into numbers via a complex code, and then super-enciphered by adding the numerical stream of the one-time pad. With the pad broken, there remains the codebook to reconstruct, the one that turns words into figures.

To that work sets a young analyst, Meredith Gardner – a linguist, a veteran of breaking Japanese diplomatic codes, who after the war taught himself Russian. It's he who penetrates what will turn out to be NKVD traffic, and later GRU, reconstructing the code piece by piece. The first reconstructions of the codebook he owes, by his own admission, to the linguist Marie Meyer; and it's worth recalling, against a certain spy-novel imagery, that most of the cryptanalysts were young women. The dates of the breakthrough are precise and close together: on 13 December 1946 Gardner reads a KGB message discussing the 1944 American presidential campaign; a week later, on 20 December 1946, he breaks another – sent to Moscow two years earlier – containing the list of the leading scientists working on the Manhattan Project. The proof that the atomic bomb had been infiltrated.

As early as 1945 the very existence of VENONA had been revealed to the USSR by Bill Weisband, a Soviet agent embedded in the U.S. Army's SIGINT. The Soviets, that is, knew they could be read. But the mathematics of the pad offers no retroactive remedy: the messages already intercepted and enciphered with reused keys remained vulnerable regardless, whatever Moscow knew. There was no flaw to close; there was only an error already committed, frozen in the ether, that the Americans would go on squeezing for decades.

Putting a name to the ghosts

From there the work is slow and stubborn. In the summer of 1947 Gardner produces enough results to write a “Special Report” laying out the project's potential to the intelligence leadership: it's that document which triggers the long collaboration between cryptanalysts and the FBI to put names to the spies. Because even with the pad broken and the code reconstructed, one last layer remains: the people hidden behind the cryptonyms, the code names. New York is TYRE, Washington CARTHAGE, San Francisco BABYLON, in an imagery drawn from classical antiquity. Some cryptonyms appear dozens of times and are fairly transparent; others once or twice only, and remain unknown to this day. The system, after all, betrays the same vice as the pad: sometimes the KGB reused the same covername for different people, untangled only by context or geography. Reuse as the recurring sin, at every level.

When the cryptonyms find a face, the history of the Cold War is rewritten. Twenty-one decrypted cables, all from 1944-45, concern Julius Rosenberg, first “Antenna” and then “Liberal”. The Britons Kim Philby, Donald Maclean and Guy Burgess of the Cambridge Five fall, and with them dozens of other Soviet sources inside the American and British governments. And yet, in proportion, the haul is tiny: the Venona Papers are some 3,000 cables, forced slowly over a project lasting nearly forty years, from 1943 to 1980 – three thousand messages out of several hundred thousand intercepted, a laughable fraction, made possible only by the reuse of the keys.

Important note: the cipher was not broken. The mathematics of Shannon and Kotelnikov held perfectly. No one “solved” the one-time pad. It was the wartime logistics of the Soviet apparatus – notebooks reprinted under pressure, keys reused to save time – that handed Moscow's secrets to the Americans for the next forty years. The theory was perfect. The hand that applied it was not.

The numbers still floating in the ether today

You might think a technique of teleprinters and paper notebooks is archaeology. It isn't. The one-time pad, precisely because it's the only system that requires no computer and fears no amount of the enemy's computing power, is still today the tool of whoever must communicate with an agent in the field without leaving a trace. All it takes is a radio.

Anyone with a shortwave receiver can stumble into them: synthetic voices, often female, reciting endless sequences of numbers in groups of five, sometimes preceded by a little tune or a signature melody. They're called number stations, they've been transmitting for almost a century and for decades were an officially non-existent mystery. They aren't an urban legend: several espionage trials have confirmed their use, and the messages are typically enciphered with a one-time pad. Those groups of five digits are exactly the method of the cipher notebooks: the services have long converted words and phrases into groups of digits via a codebook, and then combine secret numbers with them by modular addition – the same carry-less sum we saw cancel out in VENONA, only done by hand on paper instead of electrically on tape.

And if the number stations are the voice that calls the agent, at the other end of the line there was hardware built precisely to answer without being found: devices like the Soviet R-353 “Proton” (below), which pre-encoded the message onto a magnetic tape and then fired it off in Morse at very high speed – up to 250 words per minute – so as not to leave whoever was listening the time to locate the transmitter with radio direction finding. It's the same principle as the pad, but on the physical plane: if the one-time pad makes the content unassailable, the burst transmission makes the position of the sender unassailable. Two ways of vanishing, both perfect as long as the human link holds.

The case that brought them to court is Cuban. The station nicknamed “¡Atención!” – catalogued by radio listeners as HM01 – becomes the first in the world to be officially and publicly accused of transmitting to spies, at the centre of a federal trial after the FBI, on 12 September 1998, dismantles the Cuban Wasp Network. The arrests number ten; the five who go to trial will go down in history as the “Cuban Five”.

And here the lesson repeats itself. The one-time pad is mathematically perfect, and indeed the FBI doesn't break it. It doesn't need to: when it arrests the network, it recovers the pad software, the shortwave receivers and the decryption keys on floppy disk. Not only that. The enciphered messages found in the apartments matched the transmissions the FBI had recorded from the ether during surveillance. No one forced anything: the prosecution had in hand both the intercepted signal and the key taken from the drawer, and simply overlaid them – the two things every tradecraft manual insists you never keep together. The mathematically unbreakable secret defeated by a lock.

And it's precisely here that the system's residual strength lies, when used well: the transmissions leave no digital trace, receiving a signal is not in itself proof of a crime, and the moment the pad is used and destroyed the cryptographic trail is cut. Nothing to intercept, nothing to decrypt after the fact. It's not archive theory: ¡Atención! is transmitting to this day, nearly thirty years on from those arrests. Provided, always, that no one forgets a floppy in a drawer.

The perfect cipher lives only on the blackboard

Line up the pieces of this story: the one-time pad is, literally, the only encryption method humanity has ever conceived whose unbreakability we can prove. Used correctly, it remains among the most solid ever invented. And in a century it has never been beaten on its own ground – mathematics. It has always been beaten on ours.

VENONA doesn't stem from a weakness of the cipher, but from the logistics of an apparatus under wartime pressure. The Wasp Network doesn't fall because someone decrypted its numbers, but because it leaves the disks in a drawer. The weak point is never the algorithm. It's the key to be distributed, the randomness hard to produce, the discipline impossible to maintain, the physical medium that survives and accuses. It's always the friction of the real world against an idea born frictionless.

On the blackboard, in the clean abstraction where keys are random by definition, distribute themselves by magic and destroy themselves without leaving ash. But we live elsewhere: a world of approximate generators, of reprinted notebooks, of forgotten floppies, of exhausted people who at three in the morning cut a corner. The only mathematically unbreakable cipher we possess works perfectly everywhere – except in reality.

And perhaps this is the most honest thing cryptography can teach us, well beyond cryptography: perfection is provable, but it is not inhabitable.

(This article was born from a private experiment: I tried to write my own implementation of the one-time pad in Python. The code worked, but in doing it I reconstructed, without meaning to, exactly the errors you've read about here – the not-random-enough generator, the reused key, the details left in the clear. Touching with my own hands how easy it is to get it wrong was the best way to understand why, in a century, the perfect cipher has never held up on contact with us, poor imperfect human beings.)

Discuss...

References and further reading

The theory

• Claude E. Shannon, Communication Theory of Secrecy Systems, Bell System Technical Journal, vol. 28, no. 4, October 1949, pp. 656-715. The paper that proves perfect secrecy.
• Claude E. Shannon, A Mathematical Theory of Cryptography, Memorandum MM 45-110-02, Bell Laboratories, 1 September 1945. The classified version, four years ahead of the open publication.
• Claude E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, vol. 27, 1948. The twin work that founds information theory.
• Vladimir A. Kotelnikov, report on the absolute security of the Vernam cipher, 1941 (still classified). Independent proof, earlier than Shannon's.

The history (VENONA)

• National Security Agency, VENONA: An Overview (Cryptologic Almanac, 50th Anniversary Series) – declassified official document.
• National Security Agency, The Venona Story – official monograph on the cryptonyms and Gardner's decryptions.
• NSA-CIA, Venona: Soviet Espionage and the American Response, 1939-1957, ed. R. L. Benson and M. Warner, 1996. The collection of the roughly 2,900 messages translated and made public from July 1995.
• John Earl Haynes and Harvey Klehr, Venona: Decoding Soviet Espionage in America, Yale University Press, 1999. The reference work on the identifications.
• Chronology of Gardner's decryptions (13 and 20 December 1946), the roles of Marie Meyer, Genevieve Feinstein, Cecil Phillips and Richard Hallock, pad reuse and the revelation of VENONA to the USSR by Bill Weisband (1945).

The number stations

• Records of the Wasp Network / Cuban Five trial (arrests 12 September 1998; trial material declassified from 2001), on the use of the “¡Atención!” / HM01 station as evidence.
• Crypto Museum, technical page on the Soviet R-353 “Proton” spy transceiver.
• Priyom.org and ENIGMA 2000 – communities of radio listeners who catalogue and still monitor shortwave transmissions.

#Cryptography #OneTimePad #VENONA #ColdWar #Espionage #InformationTheory #Shannon #NumberStations #History #Writing