Blogs· 4min August 16, 2022
Evelina Vrabie joins us to share her insights into measuring the success of engineering teams. She tells us about the role of an engineering manager as well as the four types of success. Then, she walks us through how to measure productivity and high performance through research-based frameworks.
Evelina Vrabie is an Engineering Manager at Hopin. She is an engineering leader with over 15 years of experience, with a strong entrepreneurial, management and technical background. In her, Master's degree she explored the use of artificial intelligence in work therapy for high performance environments. Her work culminated in co-founding Touco Labs, a FinTech with the purpose of helping personal finance. Evelina's blog is called Jumpstart and she often publishes for CTO Craft as well. Follow these resources to connect with her.
Evelina begins by sharing what an engineering manager or leader means to her. Her view is that technical leaders bring their unique perspectives on technology to help the strategic execution of their companies. In practice, that means:
In order to achieve this, leaders have to develop the ability to understand and shape a particular function in a company. In a small startup, leadership roles often span across many areas from product, to technology to finance. In larger companies, leadership roles often become more specialised, but still require an understanding on other functionalities. This can be very difficult to do, as leaders often need to take into account problems that aren't even in the technology domain.
Evelina believes that the success of an individual or a team needs to start with where they are, their context and their organisation. Understanding the company goals and strategies, as well as the fact that they will be different at different times. There are always multiple successful paths or strategies, each coming at a different cost. Just like in a complex, distributed system, which is never fully healthy, nor is an organisation fully successful. Success is variable, especially in technology startups which are often compared to rollercoasters.
There are at least four types of success that Evelina has dealt with in her experience as an engineering leader:
All of these types of successes and outcomes require engineering leaders to employ different types of measurement metrics.
As we begin to look at measuring software delivery success, Evelina recommends that leaders start by identifying first the problems they need to solve. Then, they can proceed to identifying the outcomes they want to measure. Once these outcomes are identified, they can be placed into a framework like OKRs:
As technology leaders, we need to understand things that are outside of technology. One example Evelina gives is understanding the basics of a marketing funnel, when trying to improve the number of users for a service.
Teams should be heavily involved when setting the success strategy, especially in the metric setting part of the process. The engineers on your team should understand their success metrics very well. Leaders need to understand that their teams never exist in a vacuum and are always affected by factors outside of only technology.
High performing organisations invest in software delivery success and organisational success. When it comes to organisational success, the folks behind the book "Accelerate: The Science of Lean Software and DevOps" have been pushing new research on scaling high performing technology organisations. The whole idea behind this research is to not even start with measuring as a goal, instead we need to ensure that the company has in place at least a dozen key capabilities that are proven to increase both delivery and organisational performance. Evelina lists some common examples of these capabilities:
Once these capabilities are in place, we can begin to choose the metrics that we care about. We need to be careful about how we use these metrics and break them down accordingly as well. No matter how many metrics we know and we employ, Evelina thinks leaders should understand that blockages are not caused by technology and code, they are caused by people.
Evelina recommends the article "When Incentives Fail. A Story about Rats, Cobras, Nails, and Atrocities". This article provides many examples from human history of Goodhart's law, which states:
"When a measure becomes a target, it ceases to be a good measure"
Evelina summarises at least 3 types of anti-patterns:
Less than half of startups achieve both organisational and delivery success. Evelina points us to The SPACE of Developer Productivity framework, which has the headline:
There's more to it than you think.
Most technology leaders, which have limited exposure to research, often only emphasise activity metrics, completely disregarding people metrics. The lack of organisational performance leads to loss of productivity, burnout and attrition. Both of these need to be considered if we want to see an impact on performance.
How do you phrase high performance? Evelina thinks of it as:
High performance = Workplace satisfaction * Intrinsic motivation
As technology leaders, we can focus on providing the two factors to this equation:
Evelina thinks we lack good metrics for measuring the motivation and satisfaction in the technology industry. We often think of happiness as a soft metrics, when in fact happiness at work is directly translatable to revenue because software is written by humans - happy people are motivated to deliver good products that others love!
The SPACE framework recommends measuring invisible work as well. Aside from activity metrics, we need to look at communication and collaboration, as well as satisfaction and well-being. We need to learn to communicate timely, accurately and succinctly. This is not an art, it's a science. Very few technology companies invest in communication training, which is especially important for fully remote companies.
Evelina thinks referring to technical debt by this term diminishes our understanding of it. Technical debt is in fact scale work and risk work. Prioritising technical debt is a Goldilocks problem: doing it too early limits our opportunity to do work and gain the market, while doing it too late will degrade performance, again leading to a loss of market share. We need to find the just right moment. It has at least three steps:
Evelina has written an interesting 2-part series on "Data-driven negotiation with SLIs, SLOs and Error Budgets", which you can read on her blog.
Evelina ends with five highlights for engineering leaders:
Adelina is a polyglot engineer and developer relations professional, with a decade of technical experience at multiple startups in London. She started her career as a Java backend engineer, converted later to Go, and then transitioned to a full-time developer relations role. She has published multiple online courses about Go on the LinkedIn Learning platform, helping thousands of developers up-skill with Go. She has a passion for public speaking, having presented on cloud architectures at major European conferences. Adelina holds an MSc. Mathematical Modelling and Computing degree.
Blogs · 4 min
In this blogpost, David introduces us to the five W's of information gathering - Who? What? When? Where? Why? Answering the five Ws helps Incident Managers get a deeper understanding of the cause and impact of incidents, not just their remedy, leading to more robust solutions. Fixing the cause of an outage is only just the beginning and the five Ws pave the way for team collaboration during investigations.
July 26, 2023
Blogs · 4 min
Patrycja, Artur and Marcin are engineers at Form3 and some of our most accomplished speakers. They join us to discuss their motivations for taking up the challenge of becoming conference speakers, tell us how to find events to speak at and share their best advice for preparing engaging talks. They offer advice for new and experienced speakers alike.
July 19, 2023
Blogs · 4 min
Andy recently wrote about what he considers to be the key ingredients for bootstrapping a new engineering organisation. Many of these ingredients are about what you use to build your organisation with. However, this post describes one key element of how you do it. This blog post describes one way of making decisions and designing new changes which scales well with a growing team, and includes everyone in the process.
June 28, 2023