.tech Podcast - Measuring the success of engineering teams

Evelina Vrabie is an Engineering Manager at Hopin. She is an engineering leader with over 15 years of experience, with a strong entrepreneurial, management and technical background. In her, Master's degree she explored the use of artificial intelligence in work therapy for high performance environments. Her work culminated in co-founding Touco Labs, a FinTech with the purpose of helping personal finance. Evelina's blog is called Jumpstart and  she often publishes for CTO Craft as well. Follow these resources to connect with her.

Evelina begins by sharing what an engineering manager or leader means to her. Her view is that technical leaders bring their unique perspectives on technology to help the strategic execution of their companies. In practice, that means:

• Helping find a way to calibrate product and business technical strategy
• Supporting the generation of new ideas, most often by combining existing technologies to obtain a new result
• Making tradeoffs and creating a balance portfolio of product/feature work and technical debt

In order to achieve this, leaders have to develop the ability to understand and shape a particular function in a company. In a small startup, leadership roles often span across many areas from product, to technology to finance. In larger companies, leadership roles often become more specialised, but still require an understanding on other functionalities. This can be very difficult to do, as leaders often need to take into account problems that aren't even in the technology domain.

Evelina believes that the success of an individual or a team needs to start with where they are, their context and their organisation. Understanding the company goals and strategies, as well as the fact that they will be different at different times. There are always multiple successful paths or strategies, each coming at a different cost. Just like in a complex, distributed system, which is never fully healthy, nor is an organisation fully successful. Success is variable, especially in technology startups which are often compared to rollercoasters.

There are at least four types of success that Evelina has dealt with in her experience as an engineering leader:

1. Commercial success: the company is generating enough revenue to justify investment and growth
2. Product success: the company is offering products and services without which customers would be truly unhappy
3. Software delivery success: the company delivers these products and services as effectively as possible
4. Cultural/organisational success: the company offers a workplace where employees feel they belong and work on challenging and rewarding work. This is perhaps the most underrated type of success

All of these types of successes and outcomes require engineering leaders to employ different types of measurement metrics.

As we begin to look at measuring software delivery success, Evelina recommends that leaders start by identifying first the problems they need to solve. Then, they can proceed to identifying the outcomes they want to measure. Once these outcomes are identified, they can be placed into a framework like OKRs:

• Choose a series of objectives
• Create a series of key results and initiatives to deliver those objectives
• Choose a set of metrics to measure progress towards those key results

As technology leaders, we need to understand things that are outside of technology. One example Evelina gives is understanding the basics of a marketing funnel, when trying to improve the number of users for a service.

Teams should be heavily involved when setting the success strategy, especially in the metric setting part of the process. The engineers on your team should understand their success metrics very well. Leaders need to understand that their teams never exist in a vacuum and are always affected by factors outside of only technology.

High performing organisations invest in software delivery success and organisational success. When it comes to organisational success, the folks behind the book "Accelerate: The Science of Lean Software and DevOps" have been pushing new research on scaling high performing technology organisations. The whole idea behind this research is to not even start with measuring as a goal, instead we need to ensure that the company has in place at least a dozen key capabilities that are proven to increase both delivery and organisational performance. Evelina lists some common examples of these capabilities:

• Continuous delivery: CI, version control, automated deployments, test automation etc.
• Architecture: loosely coupled, test and deploy on demand, empowered teams which can change their tools
• Product & Process: continuously gathering customer feedback, ensuring that teams have good visibility into the work flow and the value their deliver, working in small batches
• Lean management & Monitoring: lightweight change approval process, monitoring across the application, code reviews

Once these capabilities are in place, we can begin to choose the metrics that we care about. We need to be careful about how we use these metrics and break them down accordingly as well. No matter how many metrics we know and we employ, Evelina thinks leaders should understand that blockages are not caused by technology and code, they are caused by people.

Evelina recommends the article "When Incentives Fail. A Story about Rats, Cobras, Nails, and Atrocities". This article provides many examples from human history of Goodhart's law, which states:

"When a measure becomes a target, it ceases to be a good measure"

Evelina summarises at least 3 types of anti-patterns:

• Having metrics for the sake of metrics
• Having metrics imposed externally
• Having metrics that are linked to the wrong incentives

Less than half of startups achieve both organisational and delivery success. Evelina points us to The SPACE of Developer Productivity framework, which has the headline:

There's more to it than you think.

Most technology leaders, which have limited exposure to research, often only emphasise activity metrics, completely disregarding people metrics. The lack of organisational performance leads to loss of productivity, burnout and attrition. Both of these need to be considered if we want to see an impact on performance.

How do you phrase high performance? Evelina thinks of it as:

High performance = Workplace satisfaction * Intrinsic motivation

As technology leaders, we can focus on providing the two factors to this equation:

• Workplace satisfaction through meaningful work, psychological safety, good feedback, recognition and reward etc.
• Intrinsic motivation through coaching other people to increase their own motivation

Evelina thinks we lack good metrics for measuring the motivation and satisfaction in the technology industry. We often think of happiness as a soft metrics, when in fact happiness at work is directly translatable to revenue because software is written by humans - happy people are motivated to deliver good products that others love!

The SPACE framework recommends measuring invisible work as well. Aside from activity metrics, we need to look at communication and collaboration, as well as satisfaction and well-being. We need to learn to communicate timely, accurately and succinctly. This is not an art, it's a science. Very few technology companies invest in communication training, which is especially important for fully remote companies.

Evelina thinks referring to technical debt by this term diminishes our understanding of it. Technical debt is in fact scale work and risk work. Prioritising technical debt is a Goldilocks problem: doing it too early limits our opportunity to do work and gain the market, while doing it too late will degrade performance, again leading to a loss of market share. We need to find the just right moment. It has at least three steps:

• Identify the use-case
• Identify the right outcomes and compare them to other work
• Set some reasonable thresholds

Evelina has written an interesting 2-part series on "Data-driven negotiation with SLIs, SLOs and Error Budgets", which you can read on her blog.

Evelina ends with five highlights for engineering leaders:

• Expand your knowledge and understanding outside technology
• Understand where the technology function sits within a company, and how it relates to other functions
• Combine the different types of engineering success to build a high performing organisation
• Find the right use-cases, outcomes and metrics, but use them only as a to tell us the trend and the direction we are moving
• There are two sides of high performance, activity side and the people side