Velocity and Story Points

Right now in (zoom) conference rooms across the globe, some engineering manager or product manager is talking about, defending, or setting goals around Velocity. The engineers in the room are playing along, maybe debating the pros and cons of using Fibonacci or T-Shirt sizes as the metric by which they declare the “story points” of planned work. At the same time, sometimes in the same room, product managers or company leadership is asking for a description of upcoming work in terms of “milestones” and an estimate (and commitment!) for when the upcoming work will be completed. The meeting ends and everyone agrees and then deadlines are missed, scope is cut, tech debt is ignored, and shortcuts are taken to get the desired product changes into production sooner. So what happened? Where is the disconnect between engineering velocity and shipping The Thing?

Firstly, we should talk about how velocity is measured in most engineering teams (and indeed at Justworks). Typically, we talk about velocity in terms of Story Points. I asked around for definitions of what this story point estimate means and got a couple different answers. To some, it’s an expression of amount of time the work will take. To others, it describes the relative complexity of the work. Either way it is an estimate on some definition of “Effort,” be it body or brain. During the scrum process, engineers go through the process of estimating the story points of tickets we are ready or just about ready to work on. Every two weeks (or whatever), the team goes through the process of picking tickets that are ready to be worked on and pulling them into a queue. Which tickets we bring in and in what order is usually the result of a group conversation. Sometimes certain work has to happen before others. Sometimes we just want to bring this thing into the sprint before that other thing for nebulous reasons. Finally, the team looks at how many total Story Points we’ve brought into the sprint and takes a pulse check on if its the Right Amount of Work for the next two weeks. Looking at historical Total Story Points Shipped Per Sprint becomes a de facto representation of the team’s velocity.

At some point, after some number of sprints (and thus some number of story points worked on), new features or system improvements can be considered complete and the team can celebrate the delivery on milestones the rest of the company is patiently waiting on. Hurray!

So we’re meticulously estimating and tracking the amount of Work We Do and then reporting on that. Is this what we actually want? Does the sales team care if our velocity is “60 story points a sprint”? or do they only really care about when the feature they’ve already sold is going to be in production? Why is our goal to do a certain amount of work? Do people care how hard we’re working each sprint if we’re not shipping anything that matters?

The problem seems to be that teams have higher level goals of shipping something which the rest of the organization sees as some Valuable Thing. And separately, the way the product team plans their day to day work is by stacking it up with Effort (either real actual hard work, or inflated effort so as to fill up the Effort-Velocity Bucket).

There is no connection between the metrics we use to measure our day to day progress and the larger goal we hope to achieve. Said another way, there is nothing preventing the team from filling the sprint with hard/complex/time consuming tasks that don’t make any meaningful progress towards the larger milestones/objectives/features/product. (Note: I am not claiming that engineers here or anywhere would ever do such a thing, but you have to wonder: if engineers are measured on their velocity for raises or promotions, is there a chance that some subconscious over-estimating is happening?)

“But Cullen,” you exclaim, “product managers have a say in what gets brought into the sprint! This prevents engineers from pulling in things that don’t matter!”

The issue, as I see it, is that this now creates a conflict between engineering and Product. Things that engineers consider tech debt are ignored for tickets that ladder up to the new feature. Furthermore, product managers now become the arbiter of the order in which things get built. Sure, the team can have conversations and make a case for one particular thing before another but at the end of the day, some one gets to issue the edict that this ticket is more important (or…valuable hint hint) than another ticket.

So what is to be done? At a macro level (quarterly, annually, and in relation to other teams in the org), a particular Pod’s velocity actually is Value Delivered. What did you ship? What impact did it have on the business? The Value Delivered is the thing that matters to the business, so why don’t we estimate the value of the work we are planning?

Introducing the Dual Metric Estimation Sprint Planning System (real name TBD). I would never advocate for getting rid of the story point estimate. Calibrating as a team on the effort required to do some unit of work gives us a lot of useful information. But I believe we should also be estimating the Value Impact of the work we are doing. Going further, I suggest that we should be filling up the sprint based on Value Points Estimate instead of Effort (story) Point Estimates. This strategy gives us some useful new dynamics within the pod and in sharing out our progress to the rest of the org.

Firstly, it creates a more “same sides” feeling between the product manager and the engineers. Now, PMs are on the hook for estimating and defending the value of chunks of work (tickets or epics, that’s an implementation detail).  And engineers’ progress and performance is measured by that same value estimate. Additionally, engineers can present technical work as an expression of value delivered to the company (eg. this test improvement will save us 8 engineering hours a week, that’s money and time saved), giving the team a way to compare two different types of work. The effort estimate then becomes the way we rank work of similar value. For example if two tickets are of equal value, why wouldn’t we do the lower effort thing first? That way, we get closer to hitting our Sprint Value Delivery Goal. If a chunk of work is estimated as medium value but high effort, its now on the team to figure out either how to make it more valuable, or how to reduce the effort (work getting broken up into smaller units of valuable deliveries is another nice side effect of this process).

Secondly, because of our natural inclination to game systems, the product team is now encouraged to figure out how they can get a certain amount of value shipped to production with the least amount of effort. Now we’re being Agile! The team will be working together, PMs, engineering, design, QA and all the rest to focus on getting the value points per sprint they’ve all committed to.

Thirdly, when speaking to the rest of the org, the team can talk about the “points” completed each sprint and have it actually mean something. Instead of saying “we’ve been really really busy”, the team can say “we’ve shipped this much value.” Now the conversation is on a level playing field. The rest of the org understands the concept of value, and they can agree or disagree about how valuable something was or was not but at least we now have a metric we can all use to talk about work shipped.

At the end of the day, there’s a lot to be unpacked here. I still haven’t fully addressed how the Sprint Value Delivery Goal improves the hit rate of deadlines. In my experience, the team-unifying outcomes of using Value as a measure of velocity has been worth it regardless of actual or theoretical improvements to shipping The Whole Feature on time. At the very least, we’ll all have a shared sprintly target to aim at that is expressed in the same metric as the quarterly goal: Shipping Stuff That Matters.

Delivering Value to the business is the reason we exist as a team in the first place, maybe we should be using that as the measurement of our impact.