How WhatsApp scaled to 1 billion users with only 50 engineers
How to quickly learn new programming languages and frameworks. Plus, an dive into HDFS and how it works.
| 18 |
Hey Everyone,
Today we’ll be talking about
How WhatsApp scaled to 1 billion users with only 50 engineers.
WhatsApp’s Engineering Culture
WhatsApp’s Tech Stack
Erlang
FreeBSD
SoftLayer
How to quickly learn new programming languages and frameworks
This is from a recent discussion on Hacker News. We summarized some of the best comments.
Plus, some tech snippets on
An in-depth dive to Hadoop Distributed File System (HDFS)
Concurrency in Java vs. Go
An amazing free textbook on math for programmers (covers Calculus, Linear Algebra and Discrete Math)
Plus, we have a solution to our last coding interview question on creating a Spiral Matrix and a new interview question from Facebook.
Quastor Daily is a free Software Engineering newsletter sends out Technical Deep Dives, summaries of Engineering Blog Posts, and FAANG Interview questions (with detailed solutions).
How WhatsApp served 1 billion users with only 50 engineers.
In 2016, WhatsApp reached more than a billion users and had the following load stats
42 billion messages sent daily
1.6 billion pictures sent daily
250 million videos sent daily
They managed to serve this scale with only 50 engineers.
Here’s a dive into the engineering culture and tech stack that made this possible.
Engineering Culture
WhatsApp’s Engineering culture consists of 3 main principles
Keep Things Small
Keep Things Simple
Have a Single Minded Focus on the Mission
Keep Things Small
WhatsApp consciously keeps the engineering staff small to only about 50 engineers.
Individual engineering teams are also small, consisting of 1 - 3 engineers and teams are each given a great deal of autonomy.
In terms of servers, WhatsApp prefers to use a smaller number of servers and vertically scale each server to the highest extent possible.
Their goal was previously to have 1 million users for every server (but that’s become more difficult as they’ve added more features to the app and as users are generating more activity on a per-user basis).
Having a fewer number of servers means fewer things breaking down, which makes it easier for the team to handle.
The same goes for the software side where they limit the total number of systems and components in production.
That means fewer systems that have to be developed, deployed and supported.
There aren’t many systems/components that are developed and then put into maintenance mode (to eventually become orphans until something goes wrong).
Keep Things Simple
WhatsApp uses the mantra Just Enough Engineering.
They avoid over-investing in systems and components.
Instead, they focus on building just enough for scalability, security and reliability.
One of the key factors when they make technical choices is “what is the simplest approach?”
Also, they avoid investing in automation unless it’s completely necessary.
Have a Single Minded Focus on the Mission
Product Design at WhatsApp is incredibly focused.
It’s dedicated to delivering a core communications app with a great UI.
They avoid extra bells and whistles and don’t implement features that aren’t exclusively focused on core communications.
The simpler product makes it much easier to maintain and scale.
Tech Stack
The tech stack revolves around 3 core components: Erlang, FreeBSD and SoftLayer.
Erlang
Erlang is the programming language of choice for WhatsApp’s backend systems.
Erlang was designed for concurrency from the start, and fault tolerance is a first class feature of the language.
You can read more about Erlang’s fault tolerance here.
Developer productivity with Erlang is also extremely high. However, it is a functional language, so it takes a little getting used to if you’re not familiar with the paradigm.
The language is very concise and it’s easy to do things with very few lines of code.
The OTP (Open Telecom Platform) is a collection of open source middleware, libraries and tools for Erlang.
WhatsApp tries to avoid dependencies as much as possible, but they do make use of Mnesia, a distributed database that’s part of OTP.
Erlang also brings the ability to hotswap code. You can take new application code and load it into a running application without restarting the application.
This makes the iteration cycle very quick and allows WhatsApp to release quick fixes and have extremely long uptimes for their services.
To see exactly how WhatsApp’s backend is built with Erlang, you can watch this talk from 2018.
FreeBSD
FreeBSD is the OS WhatsApp uses for their servers.
The decision to use FreeBSD was made by the founders of WhatsApp, based on their previous experience at Yahoo!
The founders (and a lot of the early team) all used to be part of Yahoo!, where FreeBSD was used extensively.
To see exactly how WhatsApp uses FreeBSD, you can watch this talk.
Just note, the talk is from 2014, so some things may be out of date now.
SoftLayer
SoftLayer is the hosting platform that WhatsApp was using in 2016.
They picked SoftLayer for two main reasons
The availability of FreeBSD as a first class operating system.
The ability to order and operate bare metal servers.
However, SoftLayer is owned by IBM (part of IBM public cloud), and WhatsApp has since moved off SoftLayer to use Facebook’s infrastructure.
They made the transition in 2017.
Tools and Techniques Used to Increase Scalability
You can view the full talk on WhatsApp’s engineering here.
Get more specific details from this High Scalability post on WhatsApp Engineering.
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.
How to learn new programming languages and frameworks quickly
This is an interesting discussion on Hacker News for a process to learn new things quickly.
Here are some of the top answers summarized.
For a new programming language, there are a couple of standard things you should try to implement.
Write several programs where you
Read/write to a file
Turn a structured object into JSON
Parse JSON into an object
A basic script that can be run from the CLI, parses flags/args, reads stdin
Send a HTTP request
Implement a very basic web server
When you’re doing these exercises, do not copy/paste any code.
You should be typing out the code yourself.
Try to bias towards reading (blogs, documentation, code, books, etc.) over watching video tutorials. Reading is a lot faster than watching video and you can be far more efficient with your time.
A 3 step process for learning new frameworks is to divide it up into a week’s work (10 - 15 hour total commitment)
Prepare - Spend 1 hour Monday through Thursday reading documentation, books, and watching content. This way you’ll have time to sleep on the concepts.
Plan - On Friday, spend 1 hour preparing a small project idea for the weekend that will use all the concepts you learned over the week.
Project - Spend 4 - 6 hours building the project over the weekend.
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.
Tech Snippets
A Programmer's Introduction to Mathematics - If you’re looking to learn more about areas like machine learning, you might have to brush up on your math background. Jeremy Kun wrote a fantastic textbook on math for programmers that covers topics in Calculus (Single and Multivariable), Linear Algebra, and Discrete Math.
You can set your own price when you purchase the e-book, so you can pay anything from $0 to $100.
Why you can have millions of Goroutines but only thousands of Java Threads
If you’ve been working with JVM based languages for a while, you’ve probably come across a situation where you’ve reached the limit of the number of concurrent threads you can have.
On your personal computer, this limit is usually around ~10,000 threads.
On the other hand, you can have more than a hundred million goroutines on a laptop with Go.
This article explores why you can have so many more Goroutines than threads.
There’s two main reasons why
The JVM delegates threading to operating system threads. Go implements its own scheduler that allows many Goroutines to run on the same OS thread.
The JVM defaults to a 1 MB stack per thread. Go’s stacks are dynamically sized and a new goroutine will have a stack of about 4 KB.
An in-depth dive into HDFS (Hadoop Distributed File System)
The article describes the architecture of HDFS and also goes into the experience of using HDFS to manage 40 petabytes of data at Yahoo!
Interview Question
Implement a BST Iterator class that represents an iterator over the in-order traversal of a Binary Search Tree.
Implement the following methods
BSTIterator- constructor. The root of the BST will be passed in as a parameter. The pointer should be initialized to a non-existent number small than any number in the BST.hasNext- Returns true if there exists a number in the traversal to the right of the pointer, otherwise returns false.next- Moves the pointer to the right, then returns the number at the pointer.
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.
Previous Solution
As a reminder, here’s our last question
Given a positive integer i, generate an nxn matrix filled with elements from 1 to i^2 in spiral order.
Example
Input: i = 3
Output: [[1,2,3],[8,9,4],[7,6,5]]
Here’s the question in LeetCode.
Solution
We can solve this question by creating an i x i matrix and then looping through it in a spiral fashion and setting each element to a value from 1 to i^2.
In order to loop through in spiral fashion, we’ll first create 4 variables: left, right, top, bot.
These 4 variables represent pointers to the 4 different sides of our matrix (left column, right column, top row, bottom row).
Left will point to the 0th column, right will point to the last column, top will point to the 0th row and bottom will point to the last row.
When we iterate through our matrix in spiral order, we’ll do it in the following way
Iterate through the top row. Now, increment the top row.
Iterate through the right column. Now, decrement the right column.
Iterate through the bottom row. Now, decrement the bottom row.
Iterate through the left column, Now, increment the left column.
We’ll iterate through these steps while the left pointer is less than or equal to the right pointer and the top pointer is less than or equal to the bottom pointer.
When our loop terminates, we can return the completed matrix.
Here’s the Python 3 code.
Quastor Daily is a free Software Engineering newsletter sends out FAANG Interview questions (with detailed solutions), Technical Deep Dives and summaries of Engineering Blog Posts.