I Wanted the Phones to Stop Dropping. I Got WhatsApp Instead.

The curious story of Erlang, told from inside the head of the man who built it.

A note on form. Joe Armstrong, co-creator of the Erlang programming language, died in April 2019. What follows is a work of narrative nonfiction told in an imagined first-person voice. The events, dates, and figures are drawn from the historical record, from Armstrong's own papers, talks, interviews, and blog posts, and from the accounts of the people who worked beside him. Where he is quoted directly, the words are his. The rest is reconstruction: how it might have felt, from where he stood. He would, I think, have preferred that we say so plainly.

Stockholm, February 2014

The news reached me the way news reaches everyone now, on the small glass rectangle in my pocket. Facebook was buying WhatsApp for $19 billion.

I am not a man who follows acquisitions. I follow bugs. But this one I read twice, and then I put the phone down on the kitchen table and looked at it for a while, because I knew something about that phone that the newspapers did not think to mention.

WhatsApp had about 450 million people using it every month. It moved tens of billions of messages a day. The entire engineering staff was around fifty people. Fifty. You could fit them in a decent-sized seminar room at KTH and still have chairs left over. Every one of those engineers was responsible for something like ten million human beings, and the thing that made that arithmetic possible was a programming language that three of us had cobbled together in a laboratory in Kista in 1986, to stop telephone calls from dropping.

I was sixty-three years old. I had spent most of my life being told that Erlang was too strange, too academic, too Swedish, too functional, too old, too small. That morning, the strangest language in the world was carrying more human conversation than any piece of software in history, and the people who had built the app on top of it had never once asked my permission.

That is the best kind of success. The kind that does not need you.

A Physicist Walks Into a Telephone Company

I was born in Bournemouth, on the south coast of England, at the end of 1950. I trained as a physicist, not a programmer. Physics teaches you a particular humility. The universe does not care about your elegant theory. Things fall. Things break. Your job is to describe what actually happens, not what you wish would happen. I never quite lost that habit, and it turned out to be the whole trick.

By 1985 I had drifted north, first to Kiruna in the Swedish Arctic, where I wrote software for radar that stared at the aurora, and then to Ericsson's Computer Science Laboratory outside Stockholm. Ericsson made telephone exchanges, the big humming cabinets that connected your call to your mother's. The laboratory had been handed a problem by a man named Bjarne Däcker, and the problem was this: writing software for those cabinets took forever, and the software had to be perfect, and it never was.

You have to understand what a telephone exchange is asked to do. It handles thousands of calls at once, each one its own little world. Hardware fails constantly, because hardware is made of atoms. The exchange is never allowed to stop. Not for a bug, not for an upgrade, not for anything. When the exchange stops, a woman somewhere cannot reach an ambulance. Those were the rules, and they were not negotiable, and I found I rather liked that. Most software has no rules. Telecom had rules.

Bjarne's team spent three years trying more than twenty existing languages against those rules. Lisp, Prolog, a dozen others. None of them fit. So we did the arrogant thing. Mike Williams, Robert Virding, and I started writing our own.

The first version, in 1986, was a little interpreter I wrote in Prolog. It was slow enough to be embarrassing. But it had the right shape. Everything was a process, thousands of tiny isolated processes that could not touch one another's memory and talked only by sending messages. If one died, it died alone. Mike then wrote a proper virtual machine in C so the thing would run at a speed a telephone company could tolerate. He called it JAM, for Joe's Abstract Machine, which I have always found faintly ridiculous and secretly loved.

We named the language Erlang. Officially it was for Agner Krarup Erlang, the Danish mathematician whose queueing theory the entire telephone industry rests on. Unofficially it was short for Ericsson Language. We let people believe whichever one they liked.

The Movie

In 1990, someone at Ericsson decided the world needed a film about us.

If you have never seen "Erlang: The Movie," I envy you the experience of seeing it for the first time. It is eleven minutes long. Mike, Robert, and I wear enormous telephone headsets. There is a synthesizer soundtrack that sounds like a lift in a Swedish bank. I place a call to Mike through an Erlang switch. "Hello, Mike." "Hello, Joe." We deliberately trigger a bug in the middle of the conversation. Robert fixes the code. We load the fix into the running switch, and the call never drops. Then we say goodbye, stiffly, like men who have been told by a director to say goodbye.

The thing is, the trick in the film was real, and it was the whole point of the language. A telecom system cannot be shut down for an upgrade. So Erlang could hold two versions of the same code at once. The processes already running the old version finished what they were doing. Every new call went to the new version. When the last process on the old code was done, the old code quietly went away. Nobody's call dropped. We called it hot code loading. Most languages still cannot do it, thirty-five years later.

The community made a tongue-in-cheek sequel in 2013, with the original cast, and I have made peace with the fact that this film may outlive everything else I ever did.

Nine Nines

Erlang spent the early nineties escaping the laboratory. We used it to prototype software for the MD110, Ericsson's corporate exchange. In 1993 the company set up a little subsidiary, Erlang Systems, to sell the language to outsiders. For a while it felt like Erlang might become a product.

Then the company had a disaster, and the disaster saved us.

Through the early nineties, Ericsson had poured a fortune into AXE-N, a grand next-generation switch built the conventional way. In late 1995 it collapsed. Cancelled. Years of work, gone. But Ericsson still needed a broadband switch, and it needed one quickly, and someone with courage decided the replacement would be written in Erlang.

That project became the AXD301. To make Erlang fit for something that size, a dedicated team spent 1996 building what we called the Open Telecom Platform, or OTP. OTP was everything we had learned about not falling over, packaged into reusable parts. Standard ways to build a server. Standard ways to build a state machine. And above all, supervisors: processes whose only job was to watch other processes and restart them when they died.

The AXD301 shipped in the late nineties with more than two million lines of Erlang inside it. Somewhere in a British Telecom network it produced a number that has followed me around ever since: nine nines. 99.9999999 percent availability. That is about 31 milliseconds of downtime a year. People have argued about that figure for two decades, and they are right to, because it describes a specific network over a specific period and nothing more. But I will tell you what I know for certain. The switch did not fall over. In telecom, that is the only compliment that matters.

The Memo

In 1998, the same year the AXD301 was proving us right, Ericsson banned Erlang.

Not the product. The language. A memo from Ericsson Radio Systems declared that Erlang was not to be used for new product development. The reasoning was not technical. Choosing a programming language, the memo said, was a longer-term commitment than choosing a processor or an operating system. C++ and Java were standard. You could hire for them. You could outsource them. Erlang was homegrown and known by a few hundred people on the planet, most of whom I could name.

I have thought about that memo for a long time, and I have come to believe the people who wrote it were not stupid. They were doing what large companies do, which is to reduce the number of ways they can be surprised. What they could not see, and what I could not see either, was that being surprised was about to become the entire business of the internet.

So a group of us left. We founded a company called Bluetail and used Erlang to build fault-tolerant internet infrastructure for people who had never heard of a telephone exchange. Ericsson, to keep the peace with us and with the outsiders who had grown attached to the language, released Erlang and OTP as open source in December 1998. I do not think anyone in the room understood what they had just done. They had let the thing out.

Bluetail was sold within two years to Alteon WebSystems for about $140 million, at what turned out to be the exact top of the dot-com bubble. Then Nortel bought Alteon. Then the bubble burst. In 2001, Nortel laid us off.

Fifty-Two, Unemployed, at the Kitchen Table

There is a particular quiet in a house when you have been made redundant in your fifties. I filled it by going back to school.

For two years I sat and wrote a PhD thesis, which I finished in 2003 and called Making Reliable Distributed Systems in the Presence of Software Errors. That title is not modest, but it is accurate. It was fifteen years of telecom scar tissue, finally written down as a theory.

The theory begins with an admission most programmers would rather not make. Your program has bugs in it. It will always have bugs in it. The bigger it gets, the more it has. Every school of defensive programming is a way of pretending otherwise, of wrapping your beautiful logic in layers of error handling that anticipate every failure except the one that actually happens at three in the morning.

So stop pretending. Let it crash.

When a process hits something it did not expect, it should die, immediately and without ceremony. Because it shares no memory with anyone, its death corrupts nothing. Its supervisor notices, and starts a clean copy with a known-good state. That is all. It is exactly what you do to a misbehaving piece of hardware: you turn it off and turn it on again. I merely suggested that software deserved the same dignity.

The doctrine of surviving a crash was written by a man who had just been through one. I did not plan that symmetry, but I have never been able to unsee it.

The Gorilla

I am told I am famous for a gorilla.

By the late nineties, object-oriented programming had won everything. Java and C++ modeled the world as objects that bundled data with behavior and reached into one another's insides to change them. I thought this was a catastrophe for anything that had to run forever, and I said so, and at some point I said it like this:

"The problem with object-oriented languages is they've got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle."

The joke stuck because it is precise. Ask an object for one small thing and you drag along its parent and the parent's parent and the entire mutable, shared, locked-and-double-locked jungle of the running program. When something in that jungle goes wrong, everything goes wrong together. Erlang's processes were my answer. Isolated. Immutable. Talking only by post.

Here is the twist I did not expect. Years later I came to understand that Erlang might be the only truly object-oriented language, because its processes are exactly what objects were supposed to be: isolated things that communicate only by messages. Alan Kay, the man who coined the term, has said roughly the same thing, and has grumbled that what he had in mind looked a great deal more like Erlang than like C++. So the man who mocked objects had built the purest object system in the industry. I just called them processes, and I was too stubborn to notice.

A Post on a Mailing List

In June 2009, a man I had never heard of named Jan Koum posted a question on the mailing list for ejabberd, an open-source chat server written in Erlang. He wanted to know about client access controls. It was the kind of question you get every week. Nobody thought anything of it.

Jan Koum and Brian Acton had just founded WhatsApp. They wanted a messaging app that did one thing well: deliver the message. Always. Even when the phone was on a bad 2G connection in a basement. Even when it had been switched off for a week. They took ejabberd, and they took Erlang, and then they took both of them somewhere no one had ever gone.

I want to be honest about my role in this, which was nothing. I did not consult for WhatsApp. I did not know Rick Reed, the engineer who did most of the impossible things, until long after he had done them. What I did was build a tool in 1986 for a problem that would not exist for twenty-five years. A global messaging app is a telephone exchange. It is millions of simultaneous connections that must never drop, running on hardware that always fails, upgraded while people are still talking. It was the same problem. It was always the same problem.

By 2011, WhatsApp was holding a million simultaneous connections on a single server. That was already absurd. Then Rick and his team spent a month hunting lock contention through the operating system, through the virtual machine Mike had started, through their own code, and got to two million. In a record run in early 2012, one machine peaked at 2.8 million people connected at once. They gave each user one Erlang process, the way we had given each telephone call one Erlang process, and the machine did not fall over.

On New Year's Eve 2012, when half the planet says the same thing to each other at midnight, WhatsApp carried 18 billion messages in a single day. The servers, as far as anyone could tell, did not notice.

There is one more thing about that $19 billion, and it is the part I enjoy most. In 2008, Facebook had launched its own chat on Erlang, on the very same ejabberd foundation. Their engineers wrote admiring posts about it. Then they rewrote the whole thing in C++, because too few people at the company could work in our strange little language. Six years later they wrote the largest cheque in the history of software for a company that had made the opposite bet and never looked back. I try not to be smug about this. I do not always succeed.

The Children

The thing about letting something out is that it stops being yours.

In 2012 a Brazilian named José Valim, who had spent years making Ruby on Rails pleasant to use, decided that our virtual machine deserved a language people actually enjoyed writing. He called it Elixir. It ran on the same BEAM, used the same OTP, and could call every line of Erlang ever written, but it looked like something from this century instead of something from a Prolog textbook. I had spent twenty-five years being told Erlang's syntax was unpleasant. José simply agreed, and fixed it, and did not ask me.

Elixir brought people I never expected. Web developers. Startups. A chat platform for gamers called Discord built its entire real-time system on it, and by 2017 was writing about running five million people at once. Two years later it was eleven million, and one hot spot in the code could not keep up, so they rewrote that one piece in Rust and called it from Elixir. That is the pattern now. The BEAM holds the connections and survives the crashes. Something fast and unforgiving does the arithmetic. I think that is exactly right. I never wanted Erlang to do everything. I wanted it to never fall over.

Then in 2016 a young Englishman named Louis Pilfold built Gleam, which gave our machine the one thing I had never given it: a strict, compile-time type system that catches your mistakes before they run. Its mascot is a small pink starfish named Lucy. I find I cannot object to anything with a starfish.

Not every story ended kindly. A company called Basho built Riak, a distributed database in Erlang that ran, among other things, the central patient records of Britain's National Health Service. Riak worked. Basho did not. In 2017 it collapsed into receivership, and the code was rescued by Bet365, a betting firm that had already staked its business on Erlang and bought the remains simply to set them free. The software outlived the company. In a language built around letting things crash, that felt like the correct ending.

What I Actually Learned

People ask me for the secret, and they are disappointed by the answer, because the answer is not clever.

We did not set out to build the backbone of the social internet. We set out to stop phone calls from dropping. To do that, we had to accept three things that programmers hate accepting. Hardware fails. Software has bugs. You are not smart enough to anticipate the ways either will go wrong. Once you accept those, the design writes itself. Isolate everything, so one failure cannot spread. Let the broken thing die, so it cannot lie about its state. Have something watching, ready to start a clean copy. Never stop the machine to fix it.

Everything else was consequence. The nine nines were a consequence. WhatsApp was a consequence. Fifty engineers carrying half a billion people was a consequence.

I trained as a physicist, and physics taught me that you do not get to argue with what happens. You describe it and you build accordingly. Every telephone exchange in Sweden knew that in 1986. It took the rest of the software industry rather longer.

Reliability is not writing perfect code. Nobody can. Reliability is building a system that survives your imperfections gracefully, and then going home to dinner while it does.

Afterword

Joe Armstrong died on April 20, 2019, at the age of 68, of an infection complicated by pulmonary fibrosis. The tributes came from well beyond the Erlang world, from people who had never written a line of the language but had absorbed its ideas through Elixir, through WhatsApp, or through the actor frameworks that borrowed from it.

The system he described has kept running without him.

RabbitMQ, the message broker written in Erlang in 2007, still carries much of the traffic between the world's enterprise systems. A great many companies that have never heard of Armstrong depend on his ideas every time two of their services talk to each other.

Discord, still running on the BEAM, has become the place where developers gather and where a new generation of AI companies launched their products. Midjourney, for a time, existed almost entirely as a Discord bot. Its millions of image requests flowed through Elixir processes descended directly from Armstrong's telephone switches.

WhatsApp kept building on the foundation. Meta wrote a new type checker for Erlang, called eqWAlizer, so that hundreds of engineers with no functional programming background could work safely inside the codebase. The company then replaced 160,000 lines of C++ media-parsing code with 90,000 lines of Rust, keeping Erlang as the orchestrator and handing the dangerous work of parsing untrusted files to a language that cannot overflow a buffer.

The language's children grew up. Gleam reached version 1.0 in 2024. The Elixir community built Nx and Axon, and the BEAM now runs machine learning pipelines on GPUs.

And "Erlang: The Movie" is still on YouTube, headsets and all.

The phone calls, as far as anyone can tell, are still connected.

Sources

1. A History of Erlang, Joe Armstrong, HOPL III (2007), https://dl.acm.org/doi/10.1145/1238844.1238850
2. Making Reliable Distributed Systems in the Presence of Software Errors, Joe Armstrong, PhD thesis, KTH (2003), https://erlang.org/download/armstrong_thesis_2003.pdf
3. Joe Armstrong (programmer), Wikipedia, https://en.wikipedia.org/wiki/Joe_Armstrong_(programmer)
4. Erlang (programming language), Wikipedia, https://en.wikipedia.org/wiki/Erlang_(programming_language)
5. Three Decades with Erlang, Happi Hacking, https://happihacking.com/blog/posts/2023/erlang-history/
6. The Rise and Fall of Erlang at Ericsson AB, Imad Alihodzic, https://iknek.github.io/blog/the-erlang-story/
7. Twenty Years of Open Source Erlang, Erlang Solutions, https://www.erlang-solutions.com/blog/twenty-years-of-open-source-erlang/
8. Erlang: The Movie (1990), Ericsson, https://www.youtube.com/watch?v=xrIjfIjssLE
9. Erlang: The Movie II: The Sequel (2013), https://www.youtube.com/watch?v=rRbY3TMUcgQ
10. Write no classes! Joe Armstrong, Hacker News, https://news.ycombinator.com/item?id=5205441
11. Objects, Smalltalk and Erlang: Joe Armstrong on why Erlang might be the only object-oriented language, InfoQ (2010), https://www.infoq.com/news/2010/07/objects-smalltalk-erlang
12. Inside Erlang, The Rare Programming Language Behind WhatsApp's Success, Fast Company, https://www.fastcompany.com/3026758/inside-erlang-the-rare-programming-language-behind-whatsapps-success
13. The WhatsApp Architecture Facebook Bought For $19 Billion, High Scalability, https://highscalability.com/the-whatsapp-architecture-facebook-bought-for-19-billion/
14. How WhatsApp Grew to Nearly 500 Million Users, 11,000 Cores, and 70 Million Messages a Second, High Scalability, https://highscalability.com/how-whatsapp-grew-to-nearly-500-million-users-11000-cores-an/
15. Scaling to Millions of Simultaneous Connections, Rick Reed, WhatsApp (2012), https://www.slideshare.net/slideshow/scaling-to-millions-of-simultaneous-connections-by-rick-reed-from-whatsapp/52848143
16. WhatsApp, Facebook, Erlang and realtime messaging: It all started with ejabberd, ProcessOne, https://www.process-one.net/blog/whatsapp-facebook-erlang-and-realtime-messaging-it-all-started-with-ejabberd/
17. WhatsApp Processed A Record 18 Billion Messages On Last Day of 2012, The Next Web (2013), https://thenextweb.com/insider/2013/01/02/whatsapp-processed-record-18-billion-messages-on-last-day-of-2012
18. New Facebook Chat Feature Scales to 70 Million Users Using Erlang, High Scalability (2008), https://highscalability.com/new-facebook-chat-feature-scales-to-70-million-users-using-e/
19. Erlang at Facebook: Chat Architecture, Eugene Letuchy, Erlang Factory (2009), http://www.erlang-factory.com/upload/presentations/31/EugeneLetuchy-ErlangatFacebook.pdf
20. History of Elixir, Practical Elixir, https://practical-elixir.woojiahao.com/history-of-elixir
21. How Discord Scaled Elixir to 5,000,000 Concurrent Users, Discord Engineering (2017), https://discord.com/blog/how-discord-scaled-elixir-to-5-000-000-concurrent-users
22. Using Rust to Scale Elixir for 11 Million Concurrent Users, Discord Engineering (2019), https://discord.com/blog/using-rust-to-scale-elixir-for-11-million-concurrent-users
23. Gleam (programming language), Wikipedia, https://en.wikipedia.org/wiki/Gleam_(programming_language)
24. If you wagered Bet365 would buy up Basho's remains, you'd be a big winner right now, The Register (2017), https://www.theregister.com/2017/08/25/bet365_to_buy_basho_release_code/
25. Type-checking Erlang and Elixir, Erlang Solutions, https://www.erlang-solutions.com/blog/type-checking-erlang-and-elixir/
26. Rust at Scale: An Added Layer of Security for WhatsApp, Engineering at Meta (2026), https://engineering.fb.com/2026/01/27/security/rust-at-scale-security-whatsapp/
27. Why do ML on the Erlang VM?, Underjord, https://underjord.io/why-ml-on-erlang.html
28. An Introduction to RabbitMQ, Erlang Solutions on YouTube, https://www.youtube.com/watch?v=Hx7o3pZoy0g

Image Credits

1. Bank of Strowger selectors from a local exchange, Aachen museum. Túrelio, CC BY-SA 3.0 de, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:HebdrehwaehlerbatterieOrtsvermittlung_4954.jpg
2. Robert Virding and Joe Armstrong, 2013. Robert Scoble's Building 43, CC BY 3.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Robert_Virding_and_Joe_Armstrong,_2013.jpg
3. Ericsson headquarters in Kista, Stockholm, 2010. Holger Ellgaard, CC BY-SA 3.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Ericsson_huvudkontor_Kista_2010.jpg
4. Ericsson AXD301 ATM switch. Keyhanjkm777, CC BY-SA 4.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Ericsson-AXD-301.jpg
5. Joe Armstrong, 2009. Yareeh, CC BY 2.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Joe_Armstrong_(3352929823).jpg
6. OTP supervision tree diagram. Original illustration for this article.
7. Mountain gorilla, Volcanoes National Park, Rwanda. Charles J. Sharp, Sharp Photography, CC BY-SA 4.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Mountain_gorilla_(Gorilla_beringei_beringei)_female_2.jpg
8. WhatsApp logo. WhatsApp, public domain, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:WhatsApp.svg
9. Jan Koum at 4 Years From Now, Barcelona, February 2014. Dan Taylor for tech.eu, CC BY 2.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Jan_Koum_-_Whatsapp_-_4_Years_from_Now_(12791847664).jpg
10. Elixir logo. José Valim, elixir-lang.org, public domain, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Official_Elixir_logo.png
11. Lucy, the Gleam mascot. Louis Pilfold, Gleam project, Apache License 2.0, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Gleam_Lucy.svg
12. Erlang logo. Public domain, via Wikimedia Commons. https://commons.wikimedia.org/wiki/File:Erlang_logo.svg