After their six month anniversary, our senior engineers take on the extra responsibility of joining the oncall rota. We know this is a crucial aspect of engineering life and it is essential that everyone on the engineering team has good oncall experiences! This page gives you some detailed insight into our approach to oncall at Form3.
We operate under a true DevOps culture, so our engineers are responsible for building, maintaining and supporting our high volume platform. We firmly believe that supporting a platform oncall is a great learning experience in how to debug and fix a complex system, at scale.
In general, the responsibilities of the engineer oncall are:
Our engineers are in charge of maintaining the monitoring and alerting of their services, ensuring their services are not noisy and that their alerts are meaningful and actionable.
First off, let's cover some of the tooling we use for our oncall rotas. All of our engineers have company phones (currently iPhones). All necessary company software and accounts available on these devices.
The main tools that engineers use oncall are:
We are always improving our documentation and tooling to ensure that it is up to highest standards.
It's important to us that our engineers never feel alone, pressured or unsupported when they're oncall. We have dedicated incident managers to support the engineer oncall throughout the lifecycle of an incident.
Typically, our incident process consists of:
Our platform runs critical payments for our clients, so we provide round the clock oncall support. Everyone from the engineering group takes part in the rota - including our leads, heads of engineering and the executive team!
We divide oncall shifts between office hours and out of hours shifts. During office hours, the engineer oncall has the support of the entire team, but is the first point of contact in the case of incident or error. Out of hours shifts run on a daily rotation, while office hours shift can be longer, depending on the team.
As out of hours shifts are less convenient than office hours shifts, these are remunerated on top of engineer base salary. Engineers are paid for making themselves available for a shift, even if they are not paged. We also give our engineers time in lieu if they are called out for a longer time to fix a challenging issue.
Team size will impact the frequency of out of hours on-call. Typical team size at Form3 is 4-6 engineers, meaning you should expect to take on 7 out of hours on-call shifts every 4-6 weeks. The team members support each to accommodate for annual leave and personal schedules so swapping shifts or a part of a shift is normal and easy to do with PagerDuty.
We provide support and training for engineers going on call. While the process varies across our teams, the oncall onboarding generally follows this structure:
We look for the best candidates from around the world to come and join us as senior engineers. We regularly promote engineers into technical leadership roles who have a demonstrated track record in delivery, communication, team work and technical ability. Form3 is a great place to come learn, excel and grow your career!