Metrics and measures are always on the agenda when I work with organizations new to agile. We're bringing in a new way for them to get work completed, so it would make sense that the old ways of measuring wouldn't necessarily make sense anymore. The unfortunate thing is that you get what you measure, so you really have to think about what you want, what do you want to achieve?
If your goal is to get more stuff out the door, velocity is a great measure, but it is extremely easy to abuse and manipulate. But, you might ask, what value is getting stuff out the door if it's not the right stuff? How do you determine if you're building the right stuff? There are a bunch of different tools available to come up with a value or relative value, Weighted Shorted Job First (WSJF) is popular because of the popularity of Scaled Agile Framework (SAFe), but it can be complicated for people just starting out. Affinity estimation is a good tool for finding relative value to determine what you should be working on.
So, theoretically, you now have a way to know what stuff is the right stuff, and determine if you're actually developing it on a predictable cadence. In order to ensure you’re not trying to go too fast, if you’re measuring velocity, you should look to something that counteracts that measure, like quality. Using a counter-measure like this helps ensure that you aren’t taking anything too far. You’re not pushing the team to deliver too much, and you’re also not pushing for gold-plated code at the expense of quantity delivered.
Once you are delivering the right things on a regular cadence, you can move on to second level problems. The first of these is to examine if you are delivering optimally. Do the metrics we have allow you to determine that? I’d argue that you don’t have that information with the information you’ve got on hand. There are a few new things you can keep an eye on in order to better understand your system. As your organization matures, you can measure the efficiency of the system by measuring flow via flow efficiency. While this is a Kanban idea, that doesn’t mean it’s not applicable outside of Kanban systems.
To get to flow efficiency you need to track two new metrics, lead time and overall work time. Lead time is the time from the customer request to the time it’s delivered to the customer. Think of this as the time from when you order the pizza to when it shows up on your doorstep. Overall work time is measured by the time working in any states. Taking the pizza analogy further, this would be from the time the store starts making your pizza until it is ready to deliver. If it takes 30 minutes from order to door, and 10 from start of making to ready to deliver, then the flow efficiency of that system is 10/30 = or 33% efficiency.
Armed with this information, you can start looking at the process to determine if there are any bottlenecks or other places where it can be streamlined and improved. Starting to look at the process of building things will allow you to increase the amount you’re delivering, and with the other metrics you’ve already been capturing, you can ensure that it’s all quality. Starting with these metrics should get you through your first year of agile development.