Recently I was asked about EVM, specifically CPI and SPI, for Agile projects. The question stopped me in my tracks because …. it makes no sense. Why would I collect metrics that estimate my progress when I can prove actual progress by delivering working software every iteration (sprint)? I’ll outline my thoughts in this post to show why EVM makes no sense for Agile.
First, EVM was developed by the US Government in the 1960’s to solve a particular problem. They had many large, multi-year Waterfall projects with outside vendors and no visibility into progress. Because there was no way to measure how much of the product was complete (Waterfall meant they got everything at the end), if they stayed with Waterfall, the only thing they could measure was the progress of the project. EVM is based on a fundamental assumption that we can plan everything up front, and therefore if the project progresses as expected, then we will get the product we paid for.
Let me be very clear. EVM treats the project as the deliverable and measures progress of the project. EVM in no way measures completion of the product, which is what we are actually delivering to the customer.
The fundamental assumption on which EVM is based is (in many cases) not a valid assumption. In many cases we cannot plan all the work up front. Also, we all know of projects that progressed exactly as planned, but the delivered product was a failure. The project succeeded, the product did not.
But let’s be fair. As long as the choice for software development is Waterfall, there really is no other way to measure progress.
Agile gives us a fundamentally different approach. We can still measure plan versus actual, but we measure a different plan against a different actual. We measure the user stories selected for the iteration (sprint) against the actual software we completed. Does the delivered software do what the user stories said it must do? We measure the actual benefit received by real users using the real software against the cost of developing the software. Do the actual benefits match what we expected? Is the cost reasonable compared to the benefit? Should we continue investing in this product?
Agile answers the question the customer really cares about:
Did I receive the right product at the right time for the right cost?
Our customers do not care about how well the project went. They don’t see projects. They see products. They see what the product does, they see what the product costs. They want the product when they need it, not 6 months from now.
I’ll close this by illustrating with a scenario. I have 2 projects with 10 people each, with an average labor charge of $100 an hour. Each project is expected to cost $1,000,000 and complete in 6 months.
Project 1 is Waterfall. The project plans calls for the requirements phase to be completed after 2 months for a cost of $330,000.
Project 2 is Agile (of some flavor). They are doing 2 week iterations (sprints). Each iteration (sprint) should cost $80,000 if everyone is billing full time to the project. The first two iterations (sprints), the team expects to complete work that has an estimated benefit of $400,000 a year by saving 80 hours a week of labor time on a task performed by 40 people in the company (2 hours per person).
Project 1 completes the requirements phase in 7 weeks for a cost of $280,000. Everyone is happy. We are ahead of schedule and spend, so the project continues.
Project 2 completes the two sprints. The software matches what the user stories described. The team spent $90,000 (they had to call in a specialist to fix a weird problem). The software is given to 10 of the people who will be using it. The users save on average 1 hour per week using the new software. The cost was $90,000, the benefit is only $200,000, but still a good ratio and the users are happy. The new software is released to all 40 users. Based on lessons learned, the team reviews the cost and benefits of the next user stories to see if there is value in continuing.
If I look at EVM for Project 2, we completed what we planned, but we have overspent. The users are already using the software and receiving enough benefit that it was worth the cost. We have this information 1 month into a 6 month project. We can choose whether or not to continue spending money.
EVM for Project 1 looks better. We are on schedule and budget. But we have no idea if we are doing the right thing or if the benefit will be worth the cost. We will not know this for another 4 months. We have no basis for deciding if the cost is worth the benefit or if we should continue spending money.
There is so many false statements in this blog on Agile and EVM. EVM measures progress by looking at satisfied deliverables. Agile or not projects still have cost, schedule, and quality constraints. With agile the focus shifts from the traditional but the question of what features will make it into the final release when the time and money are gone is still best answered with EVM. How well are you spending time? How well are you spending money?
The AGILE + EVM track at EVM WORLD (evmworld.org) will continue to explore how the best meld these two concepts with case studies of how its done and why it works.
Just to be picky, but at $80k/sprint, for two sprints + $10k for a specialist, the cost would be $170k for the $200k benefit, rather than $90k, which isn’t quite as good a ratio. Doesn’t detract from the point, though, which I agree with.
Thanks Chris for the catch! I did the math in my head late at night – oops!
I’ve been product owner for real products, and how I work is – show me working software quickly, let’s refine our mutual understanding, then get it into users’ hands for the real feedback. In short cycles, I know my real cost and can quickly measure my true benefit, then decide if I want to continue the work or not. There is no project to set up, measure against, and tear down. It is a continuous stream of work until the cost/benefit ratio is no longer there. This agility in process allows for tremendous business agility, which in many industries is vital for remaining in business.
It is a very different model than planning ahead, estimating the cost, then reporting the actual against the plan. Instead, the focus is on working, production quality software and the cost/benefit ratio, and not the tasks that created the software. I have seen this model at work in companies from startup to Fortune 50 over a couple of decades. 🙂