Bootstrapping a successful engineering organisation

Secrets management in build and deployment pipelines

Secrets are required to build and deploy software. Normally when you start out, there are some simple, secure ways to inject secrets into your build and deployment pipelines. For example, with Travis you can use the CLI to commit an encrypted file to a repo, or add an encrypted environment variable. In GitHub actions, you can add a secret to a repo, which you can safely use later in your workflows.

However, it doesn't take long before you've got lots of secrets in lots of different places. When this happens, the secrets themselves become hard to manage, and hard to maintain. You can't easily revoke or rotate them, or roll out new secrets automatically.

Ideally, when bootstrapping an engineering organisation, you want to establish a good way of injecting secrets into your entire build and deployment pipeline estate from day one.

Read more about my approach to secrets management in pipelines here!

Distributed management, DevOps, and hiring

You might be wondering what management, DevOps, and hiring have in common at Form3 such that I have grouped them together! One of the philosophies underpinning the organisation's foundation was the distribution and delegation of responsibility to a largely self-organising team.

This means that, as we have grown, our large team has crystallised into multiple independent, and self organising units. Each unit is organised around a particular functional area, and has complete autonomy to run and improve this area.

The entire engineering organisation was begun as a DevOps team, with a single engineering population responsible for building, deploying, and supporting our platform. This has led to a culture of ownership and pride, and means that each team has everything it needs to operate autonomously.

This same approach can be seen in many other areas of the engineering organisation at Form3, but it is most notable in our hiring process. There is no hiring manager in charge of engineering hiring at Form3, nor is there a fixed panel of interviewers. Instead, every engineer who has been with the team for longer than 6 months is involved in the process. This means that most of our engineers are also interviewers, and are involved in recruitment activities. This might sound like a lot of non-engineering work for members of our team, however each of interviews is split into three 30 minute segments, each of which is conducted by a different engineer. In practice, an engineer can expect to do a couple of these per month, which means the burden of hiring is greatly reduced, and is distributed across our engineering team.

This hiring example just serves to demonstrate how the founding team decided to distribute and democratise a range of responsibilities across the entire engineering organisation as it grew. Rather than keeping control of these important activities in the hands of a select few, they have become part of the fabric of our engineering community, and not a bottleneck beholden to a small group of decision makers.

Decision making and design

With a small team, it's easy to quickly make decisions and act on them. Decisions can be made over a cup of coffee, and system architecture can be roughed out on a whiteboard. However, when more people join your team this doesn't scale well, and it isn't very inclusive. Furthermore, the further in the past that you made decisions, the more times people will ask, "why do we do it that way?".

Having a scalable process in place for distributed decision making and design is essential--especially in a remote setting--to making sure everyone is included in shaping the trajectory of your product. It also ensures that future generations of your team can look back on the fruits of your labour, and understand why things are the way they are.

Read more about my approach to decision making and design here!

Infrastructure as code

Although taken for granted now, infrastructure as code was in its infancy when Form3 was founded. Despite its novelty, codified infrastructure (using Terraform) is completely ubiquitous at Form3. Everything from our AWS infrastructure to the repos in our GitHub organisation are managed through Terraform. Anything you can imagine with an API is managed via Terraform (apart from perhaps Slack channels!).

Beyond the obvious benefits this brings, it means that access to a wide variety of systems is democratised. Everyone can inspect, understand, and contribute to our PagerDuty configuration, network infrastructure, Grafana dashboards, Logz alerts, etc. via a uniform tool chain.

This has scaled very well with the size of the team, and has maintained a culture that we all own everything equally.

Environment access

You'll probably start with a development environment. Then you might need a testing environment. Soon enough you'll have a production environment, and in no time at all you'll have a fleet of environments to manage. For each environment, your engineers will need a variety of different means of access. For example:

• SSH access to compute nodes.
• Control plane access (e.g. via kubectl) to Kubernetes.
• Database access to your database servers.
• Command line or web console access to your cloud provider.

For some environments, you'll want engineers to have unrestricted access. In others, you'll want access to be tightly controlled, possibly with some kind of privilege escalation and auditing.

Access to your environments is a fundamental component to your engineering organisation, and getting this right at the beginning will allow your team to grow and build out a platform in a secure, safe way.

Blog post to follow!

Documentation

When there are only two engineers in your team, it hardly seems necessary to maintain documentation. Often knowledge is "tribal", and is passed on by word of mouth. This scales well in the beginning, when you add a modest number of people to your organisation, but sooner or later it breaks down. At this point, if you never built a culture of written documentation, you have two problems:

1. You have no documentation.
2. No-one wants to (or is practised at) writing it.

Starting your organisation with a healthy attitude towards written documentation will pay dividends as your engineering team, and product, grows.

Blog post to follow!

Testing

Something Form3 does very well, and has done well from the beginning, is automated testing of the software the team builds. This is thanks to a relatively simple formula consisting of:

• Mainly component-based integration tests.
• The use of real dependencies where possible, or external mocks where required (e.g. localstack).
• Continuous soak tests in pre-production environments.
• Few or no unit tests.

This last point can be a bit controversial, but it has led to a platform that is routinely delivered into production with very rare functional defects. Functional problems are almost always found during development, thanks to the very robust nature of full-stack integration tests. This technique is employed across Form3, to great effect.

Non-functional problems relating to concurrency or load, are generally picked up by the continuous soak tests we run using f1.

As a result, from day one, we've been able to build and deploy our platform with confidence, with little operational burden on the engineering team, and no manual testers.

Blog post to follow!

Secrets management in applications

Secrets are required to run your applications. You might start off by using your deployment pipeline to deliver secrets into a runtime environment (via environment variables), although that's obviously not very secure. You might use secret deployment tools like Sealed Secrets to conveniently manage secrets from source control. Both of these approaches, whilst simple, don't allow you to manage a fleet of secrets en-masse.

Pretty soon you will want to start rotating secrets, just like you'll need to in your CI pipelines, and eventually you might want to start using shorter-lived secrets to secure your environment. Building in something that can issue secrets to your applications at runtime, from the very beginning, will set your engineering organisation up for success in the future.

Blog post to follow!

Incident response

Your software and platform will fail, and it will probably fail often! You want to be prepared for this from the beginning, so that everyone on the team understands how to respond to incidents, and has the tools and experience to do so.

Part of this is tooling, but a greater part is your organisation's attitude towards incident response, and how it is organised internally.

Blog post to follow!

Observability

Building applications without an observability platform leaves your engineers blind to how their software is performing, and makes it almost impossible to manage your product in production. Establishing a means of observing your software from day 1 is essential.

Blog post to follow!

Building and managing compute capacity

There are lots of ways to do this, and you'll probably have a preference. As long as you make a sensible choice for your organisation, I don't think you can get this wrong at the beginning. You can always change it later too.

Service communication and messaging

Again, this is probably dependent on your preferences and domain. I don't think the choice of these technologies is critical to your success.

Orchestrating containers

There are also lots of ways of doing this. It's probably harder to change this later if you do get it wrong in the beginning, but this a technology choice you can make based on the tools available on the market.

Security considerations

These will become very important! However, when you're starting a new engineering organisation I don't think you need to solve every problem all at once. If you can get secrets management and environment access right from the beginning, then I think you'll have solved some of the larger challenges, and you can fill in the blanks as your organisation grows.

Programming languages

Choosing a set of programming languages to use is probably one of the more visible choices that need to be made when starting an engineering organisation. Form3 started with Java, and later moved to Go. I don't think a bad choice can really be made here, because it's always possible to move to a different language later. Ultimately, this choice is probably based on your preferences, your experiences, and the type of software you're planning to build.

Database engines

Whilst this is an important consideration, I think it's specific to your domain and preferences.

Based on my observations and reflections of working for a successful engineering organisation, some of the biggest challenges to get right at the beginning are:

• Secrets management (in CI, and your environments).
• Environment access.
• Observability.
• Incident response.
• Decision making and design.
• Documentation.
• Testing.

In my future posts, I will address each of these themes in more detail to try to provide some practical examples of how you can achieve success.

I think these are the essential ingredients to bootstrapping a successful engineering organisation, and I see signs of these themes everywhere at Form3.