.tech Podcast - Delivering at scale for Meta

Tugberk Ugurlu from Meta gives us a sneak peek into life as a software engineer at one of the biggest software companies in the world. He tells us about engineering culture, writing software at scale, the development ecosystem, programming languages and even... the hiring process! Tugberk joined Meta in April 2021. He joined the London office, but is working from home, as many of us nowadays. He is a part of the Catalog Platform group, which is responsible for business product catalogs on Instagram and Facebook. Their team takes care of data flow, storage and scalability of this important feature.

Engineering culture

Meta has a "bottom up" culture: engineers have a lot of input into the planning process and they are trusted to make their own decisions.

All of the products engineers develop work for incredible scale, so the tooling that Meta engineers have available to them has been designed with this in mind. This supports Meta engineers to focus on the problems they are trying to solve, rather than getting generic solutions in place. Great tooling reduces lead time and keeps engineers happy as well.

Organisationally, teams decide for themselves how to structure and support their work to best serve their work and goals. Tugberk's team works on two-week sprints and work is based on goals that the team has defined already. There is also a company wide "better engineering" initiative which allows engineers to spend 20% of their time to improve existing services. This allows teams to schedule in maintenance and tech debt work.

Recognition is a key part of engineering culture as well. Tooling is set up for engineering recognition as well, allowing teams to give props and thanks to each other publicly, in a company wide forum or feed. "Better engineering" champions are team members that have been working hard on their team's service maintenance.

Internally, Meta uses Workplace, which is a modified version of Facebook for all their communication and collaboration. There are like/reaction buttons on everything, including pull requests, allowing teams to send each other Gifs and other fun messages.

Writing software for scale

In terms of volume, most of the traffic Meta works at is in the billions. This is where the great tooling really makes a difference. All of the services, databases and pipelines are built for the scale first.

Teams have some building blocks that are already designed to be scalable, which they can leverage in their work. Scale, replication, sharding and logging are things that teams don't have to think about, as they are already in place. It may seem daunting at first to deal with such a large scale system, but on the other hand really efficient tools are provided to help solve these problems.

On the code level, even small performance improvements have the potential to save the company millions of dollars. Thinking about the data patterns is key to pipelines at Meta. Optimising reads and writes, choosing tradeoffs in consistency and throttling are all things engineers will need to consider during their work.

Development ecosystem

Source control for repositories is a purpose built Mercurial version. Code is organised into 4 or 5 monorepos that are used across the teams. Thousands of commits go into these repos every hour from around the globe. The swap from Git to Mercurial took a little while for Tugberk, but he's used to it now.

Engineers use a purpose built version of VSCode. From there, they can connect to a remote, on demand, dedicated server. The monorepo code will be on the server once it is ready, allowing engineers to search through the code.

Builds are also optimised to only rebuild the parts that have been modified, instead of rebuilding an entire system, which would be time consuming and expensive.

Releases propagate through the system in around 5 to 6 hours, using canary releases and rollbacks in case of issues. They make use of feature flags to release new features to employees first, to test out and get feedback on.

All of the internal tooling is purpose built for the incredible scale that Meta work at.

Programming languages

All backend systems are written in the Hack programming language, which is a PHP based language developed by Meta. There is also some C++ and Python, as well as Rust and Go for some smaller projects.

Hack has similar concepts to C#, like static typing and async/await. It was developed by teams at Facebook to address some of the scalability issues of the original/legacy PHP code.

Instead of rewriting the whole codebase into something faster like C++, they opted to replace the underlying PHP virtual machine (VM) with a faster, custom written VM in C++, called HHVM.

After understanding the key concepts, Tugberk now finds it easy and natural to write code in Hack.

Hiring process

Depending on the level of the role, there will be a few live coding interviews to test coding and problem solving skills. The coding environment doesn't auto-complete or compile the code during the interview. The candidate can use any language they want. The interviewer doesn't actually care if the solution will compile, they are only interested in problem breakdown, general approach and ability to learn.

The coding interviews are followed up by one or more system design interviews, as well as a behavioural interview. The behavioural interview assesses communication skills and growth mindset.

Meta values core skills and ability to learn, not previous experience with particular technologies. This way of interviewing line up with our values at Form3 as well.

by Adelina Simion Technology Evangelist