Agile and the Iron Triangle of Software Requirements

Software companies make fundamental tradeoffs between cost, quality, and functionality (the Iron Triangle) all the time. These tradeoffs happen in the software itself, of course, as anyone who’s ever heard this analogy will tell you. But they also happen in the process of making the software. In software development processes, cost = effort, resources, and time, and functionality = functionality, of course. But what is quality?

I don’t mean this to sound like a Zen primer or a TQM discussion, but software quality is much more than the negative condition of having no open bugs. It’s also the positive condition of building the right thing. Generally speaking, this means product strategy and product requirements. And here is where the arguments usually start, because it’s far from clear how to build The Right Thing. Traditional software development tries to ensure that The Right Thing gets built by requiring a multi-stage sign off and by providing extremely detailed documents at each stage of the process—a marketing requirements document for initial concept sign off, followed by a functional requirement, a technical requirement, a detailed test plan, and so forth. Between each of those stage gates is a lot of work. Many months may pass between initial concept and development.

So using rigorous stage gates to build in “quality” comes at a cost, especially in time. This is particularly risky in a new market space, where time to market and time to benefit is key. If you commit to a lengthy waterfall process, you lock yourself into a feature set that commits the team to a path of several months or even years to build the Right Thing.

But in a year’s time, the market may have shifted out from under you, and not just once. Clearly a way of building software is needed that doesn’t require so heavy a use of resources to define requirements and that can allow the Right Thing to be built more quickly, and that allows for more opportunities for the team to “change its mind” without wasting a lot of work. Agile software development methodologies seek to provide those opportunities by defining the work product to be built in terms of “iterations,” where the software gets built in discrete, short sprints rather than one. Something concrete gets built in each iteration, but the team is free to change its mind about what to build in the next iteration—and it may need to, either in response to feedback about the product of an earlier iteration, or because things have changed in the outside world.

To support a rapidly iterating software product, though, you need a requirements process that moves quickly as well. That’s where Agile gets its emphasis on “stories.” An Agile story is a simple way to specify a requirement, and it follows a pattern of “As a user type, I want to do something because of an important reason.” It’s a streamlined but easy and effective way to specify functional requirements. The idea is that each story is a standalone piece of functionality that delivers value to a user. Therefore, every discussion about choosing features for a release discusses trades between things that actually benefit the user, which is a much easier decision to make than choosing between a functional requirement and a technical requirement.

As you might guess, adopting Stories as a requirements methodology has its own challenges. In this case, the tradeoff between requirements “features” (detailed documentation) and requirements “quality” (preserving flexibility by not overspecifying) has real downstream impacts on other parts of the organization. For instance, Marketing teams are used to using the MRDs for product features as input to their messaging campaigns. And every team in the organization misses the stage gates as a chance to come in and provide feedback about what is to be built. So the challenge becomes how to build a process that remains flexible and adaptable, doesn’t get in the way of the participants, and ensures that the right deliverable gets built and brought to market correctly.

There are some pretty good suggestions out there on how to do this, as it turns out. Googling agile and mrd gets you a variety of good discussions and ideas, including the following:

  • Only write an MRD when you need to (Luke Hohmann)
  • Let your prototype be your requirements doc when you can
  • Demo each “sprint”’s results to everyone in the organization while sharing the prototypes for the next iteration (both from SVPG)
  • Keep your requirements at one level. Don’t write high level requirements documents that are then translated into low level documents. These are a good example of Conway’s Law (Roger Cauvin)
  • Use a quick 10-question checklist as a way to structure an opportunity assessment (SVPG again)
  • Minimize the impact of your iterations on your users by communicating with them (SVPG once again)

I will try to expand on some of these topics in future posts.

Radical transparency, part 1

My last post about Agile software development processes and corporate culture addressed one of the major challenges of adopting Agile within a company: the need to get the buy-in of other stakeholders, such as sales, marketing, and the executive team, who think in terms of specs-as-contracts rather than features as conversations. I got feedback from at least one reader who said, quite rightly, that this is very hard to do on both sides. It’s hard for developers to be open and transparent with sales and marketing, and even with product management. I’ll talk a little bit about the second challenge today, and circle back around to the first one another time.

Why should developers have difficulty achieving transparent communication with product management? It shouldn’t be hard, right? After all, product management is the face of the product to the rest of the company, and the face of the market to development. It’s a role that is accustomed by definition to speaking to multiple audiences, and spends more time than just about any non-technical role in communication with engineering. So it should be easy to build that conversation, right?

Sadly, all too often, this isn’t true. Product managers, as has been said eloquently by others, are detail oriented and prone to micromanaging. They tend to stomp on the engineer’s turf by talking implementation and second-guessing the developer’s expertise. Worse, product managers force firm commitments with outside stakeholders on delivery dates for functionality of uncertain scope; often, they actually make the commitment without consulting development. Then when reality strikes, the project becomes a deathmarch and the developer is working nights and weekends trying to make an unrealistic release date that the product manager casually committed to. Lastly, remember that Agile as a whole came out of a deep dissatisfaction with the way that software development projects were relating to the market, and for some product managers were the embodiment of that problem—feature creep, churn, deal-driven “vision” that whiplashes development according to the whims of the latest sales prospect.

Building trust between developers and product managers is really hard for both the structural reasons cited above, and any trust that’s built is going to be fragile at best. So how do you get over these hurdles?

That’s a little like asking, How can I be a good spouse? Because the relationship between product managers and developers has to be a long term one. It’s built interaction by interaction, by showing that one party can trust the other to do its part, by sharing responsibility. And by being open: about the market challenges, about the problems faced by sales, about the competitive landscape, and about the technical challenges that stand in the way of success.

One very simple way to build trust and get to a shared understanding of goals is to build in frequent working meetings. Many Agile processes, such as Scrum, have these built-in already in the form of daily or thrice-weekly standups, where developers can articulate progress and discuss challenges. If your team is doing this, participate. Make sure that you communicate an appropriate level of detail about what you are doing about the release. Often times, developers have no idea that you’re off talking to customers about the features that they are working on, and many find the opportunity to get detailed feedback helpful. But save the discussions of features outside the iteration, future roadmap concerns, and other things for other meetings, probably with the dev manager and maybe an architect or a designer. Keep focused on the business at hand. But even more than talking, it’s important to listen. I found the most satisfactory standups to be the ones where I actively listened, maybe asked clarifying questions, but mostly communicated to the team that I respected their time and energy and judgment. Once the developers see you in that context and get that you really understand where their challenges are, there will be one less barrier between you and them.

If Agile is about conversations, who’s listening?

The other category I planned to start almost two weeks ago was this one—it’s high time that I started writing more systematically about product management. And what better place to start than with the latest craze, agile software development?

As Steve Johnson at the Product Management blog at Pragmatic Marketing is fond of pointing out, Agile (as it’s usually capitalized) isn’t exactly new, having appeared on the horizon back in 2001 with the Agile Manifesto. But it has taken a long time to achieve anything like industry momentum. Why? In my experience, it’s because Agile requires a culture change throughout the whole company, not just in engineering and product management but also support, product marketing, and (most importantly) sales.

So why should the impact on all these other groups make it harder to adopt agile development processes? Well, let’s look at some of the top level statements of the Agile Manifesto:

  • Individuals and interactions over processes and tools: If your software development process depends on ongoing communication with the key members, what about organizations like sales who aren’t part of the ongoing conversation?
  • Responding to change over following a plan: Often in agile, this translates into much less complete planning artifacts. What happens when sales starts to fill in the blanks in the plans themselves—say, when talking to a customer?

The reality, of course, is that going Agile doesn’t lessen the requirement to communicate frequently with sales and other external stakeholders. If anything, it increases the importance of doing so. If the point of Agile is to deliver frequent software releases that add real value to the customer, then including the customer and the parts of the organization who talk with current and potential customers every day is absolutely critical to success.

In fact, Agile as a philosophy shares a lot with another manifesto of similar vintage, the Cluetrain Manifesto, which states that markets are conversations too. See a theme? There is a definite evolution in both Agile and Cluetrain away from central planning and delivery of work products (internal deliverables or actual products) as fiats, made more palatable with heavy doses of promotion and/or process, and toward ongoing conversation, collaboration, and cooperation with the other stakeholders in the process of creating value.

So at some level these two types of conversations—developers with product managers, marketers with the market—really need to become one. And that has profound implications on organizational structures, marketing plans, and nearly everything else that companies do. Is it any wonder that Agile took so long to take off?

I’ll be writing a little more about Agile from time to time, as I reflect on the lessons I learned from adopting an agile development process at iET Solutions and those that I learn at my new gig, Veracode.

Where Everyone (Wants to) Know Your Name

A few months ago, I wrote about single sign-in and why AOL and Microsoft are both trying to be the Internet’s major providers of it. Yesterday, Sun announced they were jumping on the bandwagon with digital identity services. It’s surprising that it took Sun as long as it did to come to the party, given their ambitions as an Internet platform company. Why did they wait so long? What’s so important about single sign-on?

When I was a programmer, I used to hate one thing about debugging my application: Every time I wanted to run it to test my code fixes, I had to type in my user name and password. We couldn’t do anything nifty at that point like tying it to some automated central login — the military still didn’t fully trust NT security, and half our user base was running on Windows 95 or 98, which weren’t designed to be bulletproof when it came to authenticating users.

So I did what any self respecting developer would do when he got a loud complaint from his user (me): I hacked my local code base so that it automatically supplied my username and password when I ran. Single sign-in, for sure–I was the only one who could sign-in.

There’s definitely a user benefit to only having to log on once to access information, even when you’re talking about logging into your computer and only one other system. But what about the web? Every merchant, chat room, vendor site, newspaper, whatever site in existence wants you to sign in somewhere. I calculated the other day that it takes visits to four web sites to pay our monthly bills on line. One of those sites, our bank, consolidates information from at least ten other billing agents behind its “single sign-in.” By doing that consolidation, our bank has reduced the number of user name and passwords that I have to remember from fourteen to four. Do they have my business for a long time? You bet.

Microsoft has announced part of its business strategy behind the .NET initiative. While it will be working with service providers across the Internet to deliver tons of value to the end customer, it will be the customer, as value recipient, who will pay for the service. This is probably good, since it avoids all the known bad business models (advertising supported services, VC funded free software, etc.) that have caused so many dot-coms to implode. But how does Microsoft convince customers that paying for these services is worth it? I think single sign-in is one of the benefits they’re betting that customers will pay for. And I think Sun just woke up and realized that a business shift is occurring, and they are about to miss it.