A report on the usability testing for WordPress 2.7

Jane Wells at WordPress opens the kimono further on the decision to redesign the administrative user interface for 2.7. I griped a while back that it seemed like the design changes were being made in a non-systematic way, but it turns out there were solid usability testing exercises prior to the surveys we’ve all seen about structuring the navigation menu. She also includes a detailed usability report covering the eye tracking studies and the different stages of the process. Sounds like the new administrative screen is genuinely a usability improvement in terms of the metrics that matter (task success and time on task).

I also found it interesting that participants had a strong emotional connection to the 2.5 admin interface even when the formal usability metrics proved that they had trouble using it! Apparently good graphic design (my attraction to the UI) helps to mitigate formal usability flaws–a point worth remembering.

Test driving Google Reader

One of the downsides of being an early adopter in some areas is that I’m a late adopter in many others. I was using a desktop RSS aggregator back in 2002 (Radio Userland, then NetNewsWire) and so came late to the web-based news aggregator market. When I did hop on board, I used Bloglines, one of the early web based aggregators, and so missed out on Google Reader. I’ve stuck with Bloglines because it works and because it works well on the iPhone.

Yesterday, Bloglines wasn’t working. I haven’t seen anything posted about this, but while the site’s UI was up I didn’t get any new results for any of my 175 feeds from about 11 AM on. So in the early afternoon I decided to give Google Reader a spin.

One of the nice things about feed readers is that it’s pretty easy to take all your feeds to a new reader, thanks to OPML (one of Dave Winer’s many innovations in this area). Most feed readers support exporting your feed list to OPML, a structured XML format, and support importing feed lists from OPML. So you can pack up your feeds and easily bring them to a new place–minimizing vendor lock-in. I did that with my Bloglines feeds and was up and running quickly in Google Reader.

One thing that struck me almost immediately was the poorer UI in Google Reader. While it uses the same left pane navigation–right pane reading metaphor as Bloglines, the left pane is cluttered with a bunch of stuff on the top–starred items, trends, shared items and notes, a big help pane, and THEN your list of feeds. Bloglines’ feed list takes up the whole left pane and is just your content–much easier to manage–while other information like your personal blog and “clippings” are in separate tabs. If you’re just interested in reading feeds, Bloglines’ navigation is easier and less cluttered.

The right pane UI is a little better too, imho. I find the separate drop-shadowed feed boxes in the expanded view (what NetNewsWire used to call “smash view”) distracting; Bloglines’ zebrastriped list is visually flatter and doesn’t get in the way of the content. And I can’t imagine a use for the list view for most of my RSS feeds; though perhaps the notification-only ones are better suited for this kind of presentation, I can’t imagine trying to read BoingBoing or even Krugman this way.

Google Reader does feel a little snappier–feeds update more frequently and quicker. But the reading experience is actually slower, because items don’t get marked as read on display, but only if you scroll them off the screen. That might be beneficial for some people, but I’m a quick scanner and like to run through the feed list quickly. And because Google Reader doesn’t fetch all the items in a folder at once, dynamically fetching items as the user scrolls, there’s no way to quickly scroll to the bottom and read everything all at once. You have to wait for the fetch to catch up, then scroll to the bottom again.

So this morning I was pleased to see Bloglines is back online. I’ll still test out the Google Reader iPhone experience, because there are things that don’t quite work for me in Bloglines’s. But I’ll be continuing to use Bloglines in my browser.

Technical Debt part II: Security debt

I wrote previously about “technical debt,” the concept that the decision to defer necessary technical work (adopting an updated version of a new component, refactoring code to reduce cruft, etc.) accumulates across releases until it absorbs a project team’s entire capability to develop code. You “pay interest” on technical debt because it’s much harder and consumes many more resources to make a necessary technical change the further downstream you get from the point where the change becomes necessary.

It occured to me today that there’s a specific flavor of technical debt, security debt, that is both more insidious and much easier to see in operation, because we have so many prominent examples of it. It might not have cost the developers of Windows too much more to make the OS more secure at design time, but some of the decisions were deferred, until the point where you had whole features introduced to address security deficiencies in prior features, and the six month long security push that postponed Vista’s launch while the team took care of outstanding security issues in the already-shipped version of the OS.

What’s interesting about security debt to me is that it balloons over time. My once-favorite mix sharing site, Art of the Mix, is a good example. The guy who developed it didn’t really understand SQL injection or XSS, or at least didn’t code defensively against them, and it’s become a hive of malware as a result–and is now flagged as a “reported attack site” and blocked by Firefox 3. So, to carry the metaphor to its logical conclusion, the site’s security debt drove it into a kind of “bankruptcy” when it proved susceptible to drive-by SQL injection attacks.

So how do you avoid incurring security debt? Learning good development practices is a good start; keeping up on the prevalent attacks–the current risk space–is another. But there’s one key thing to remember about security debt: in many cases fixing the underlying flaw that permits exploitation is far far cheaper than getting hacked, or even putting bandaids like web application firewalls in place.

WordPress gives a window into user experience design

With the WordPress 2.7 Navigation Options Survey, the fine folks at WordPress.org have opened the kimono on one of the trickiest product management tasks: user experience design. The context: the administrative interface of WordPress. The UI was famously redesigned earlier this year by Happy Cog studios, who applied a rigorous information architecture along with a highly readable visual style. So why redesign now?

Well, it appears that users didn’t like the way the dashboard used screen real estate. While the WordPress team doesn’t describe what the users complained about, the key navigation options are currently along the top, and I would guess that users who have widescreen monitors are pointing out that horizontal screen real estate is less precious than vertical. So the team has created a survey to get user feedback about some design options.

This is a tricky task, and it could have been made a little easier by some better user requirements gathering. For instance, what the team is fundamentally trying to do in identifying top-level command categories is classically served by “card sorting,” a classic usability design exercise. They might get better feedback by doing a card-sort study, either offline or with a software package like WebSort.

Second, the presentation of the choices doesn’t include a control. It assumes that all users prefer the vertical menu and presents variations on that option. Adding an option for the existing horizontal menu might present some valuable information on how users feel about the existing option.

My opinion may be tainted by my personal preferences; I’m one who finds the current administrative interface design preferable to what I’ve seen so far of the new direction. But regardless of my personal feelings, there’s something to be said for rigorous user centered design in determining the next direction.

What does “beta” mean for Software as a Service?

Steve Johnson at Pragmatic Marketing points to an interesting article on five different types of betas. One of Steve’s commenters suggests there is a sixth kind, the SaaS beta:

…ratchet up your release cycles to monthly, then you can call it a ‘release’ or a ‘beta.’ Either way customers get their hands on the new functionality. If they don’t like what they get you’ll hear about it.

The good news about SaaS is that, by eliminating concerns about customer migration and installation costs, you can truly embrace the frequent releases recommended by most agile methodologies. The bad news is that four to six week release cycles don’t leave much time for customer feedback, and so most of it comes after the update has been pushed, when sales and customers start pushing back on some of the changes (or asking for more).

One way around this is inherent in the agile model itself. By breaking down new functionality into small chunks for release, you can take customer feedback as each chunk is delivered. You may be wrong with every release, but you won’t be as dramatically wrong as if you waited six months before getting customer feedback, and you’ll be able to quickly find and correct the areas where you were wrong with each new functional push.

In the meantime, you can take your overall plan for the functionality that you hope to have completely delivered over three to four releases, whether in the form of a design prototype or even a set of slides, in front of your key customers and get their feedback, and prioritize any changes into upcoming releases.

Becoming a product manager when you aren’t one already

Being one of the top Google hits for “product manager resume” has its responsibilities as well as its perks. I occasionally, as I was today, get contacted by people trying to figure out things about the product management career path, and sometimes they ask really good questions. Today my correspondent asked, essentially, how to become a product manager. It’s trickier than it sounds, since you can’t really go to school to become one, and there’s not (yet) an independent certification that you can study for at night to build the skills, as there is for project management. So how do you build the credibility to become a product manager when it seems you have to be one to become one?

There are two ways that I’m aware of to become a product manager:

  1. Start as an entry level product manager at a firm, generally as an outside hire.
  2. Get promoted internally to the position.

Option #2 is hard, particularly if you’re coming from a technical discipline (I’ll explain why in a minute). Most firms that I’ve worked at have a hard time figuring out how to take someone out of what they have always done and put them into a role where they could contribute, but which would be a completely different classification.

#1 is also hard, because you have to convince the new company to hire you without the experience. This is where the MBA helps; it’s a signal to a prospective employer that, despite a lack of experience, you have the basics of what it takes. This is actually why I took the MBA–I knew I wanted to learn some business fundamentals about finance and marketing that I wasn’t getting as an engagement manager and architect in my job at a consulting firm.

Are there ways to do #1 without having an MBA or a product management job beforehand? This is probably harder than option #2. Basically, you have to come up with some other way to prove that you are product management material. The good news is that you can do this by looking at what a product manager does and figuring out how do to some of the things inside the scope of your current employment. Maybe you can get some opportunities to work directly with customers and taking their feedback. Maybe you can build a track record of being really, really good at documenting what customers want and building user requirements. Maybe you can take some opportunities to build internal business cases for taking on a new project.

So why is it so hard to become a product manager from a technical background? Why do some people find it easier to move from a marketing position than from a development position into product management? The reason is easy: the skill set required for marketing and product management overlap, to an extent. Marketing folks need to look at the market, figure out what companies’ problems are, and identify what it takes to get them to consider your product (oversimplifying grossly). Product managers look at the market, identify what business’s and users’ problems are, and identify how to build a product (or modify an existing product) to solve those problems and meet those needs. The first two skill sets are common; the last skill set is where there are specific disciplines that come into play.

I’m sure I’ve insulted just about every marketer or product manager out there with what I’ve just written. Anyone want to take a crack at removing the noxious generalizations?

The danger of outsourcing…

…your bookmarks. Del.icio.us is offline and my whole morning routine is off. Okay, so instead of tagging these two links I’ll post them to my blog instead.

First, for those new product managers out there, as well as those that have been the copy machine once too often, check out the free ebook from Pragmatic Marketing, The Strategic Role of Product Management. There’s nothing new here; in fact, it’s all stuff you’ve seen before, on Steve’s blog or in other Pragmatic publications. But it distills a bunch of lessons on why product management matters to a single document that makes a compelling story.

Okay, but once you get management buyin of the strategic importance of product management, how do you avoid getting bogged down in minutiae? How can you stay strategic? One answer comes courtesy of the Good Product Manager: Delegate tactical responsibilities. The methods to do so are simple even if you don’t have direct reports: transfer knowledge, teach to fish, and examine priorities constantly to ensure that the “urgent task” really needs doing.

The non-linear cost of bad software development

I ran across an interesting concept in my reading today: technical debt, and its cousin design debt. The concept is basically the application of the Second Law of Thermodynamics to software development. As you develop software, you affect the entropy of the code. Feature development typically increases entropy, while refactoring and explicit design activities decrease entropy.

Why do we care about entropy in software code? Code with high entropy is harder to maintain, harder to fix bugs in, and harder to add features to. It basically increases the cost and time to get new releases of the software out.

The concept of design debt argues that this kind of entropy is additive across releases, and that each time you perform entropy positive actions you increase the amount of work needed to dig out and make the code maintainable again.

I’ve lived this, for sure, and I suspect most others have too. But what makes it really interesting is thinking about it dynamically, where it is made clear that design debt decreases the profitability of a project. I think it’s even worse than it appears in the diagram, because the diagram neglects the time dimension. As the cost of development increases, more than likely the time to develop also increases—which means that Domain Evolution proceeds even farther while you are trying to catch up. This means that you have to increase the number of features even more, but that incurs a higher design debt still. It’s an unpleasant positive feedback loop.

Design mistakes cost

I’ve stopped reading Jakob Nielsen on a regular basis, so I missed this: Top-10 Application-Design Mistakes. As it turns out, this is one of the few of Jakob’s Alertboxes that I agree with more than disagree with. Iterative design, paper prototypes, decide what your app should do, beware nonstandard GUI controls, design for the user rather than the back-end system, etc.

Number two particularly amuses me. I was on a business trip with someone who was bitten, hard, by this bug (on a different travel site). His boss booked his travel, and didn’t pay attention to the fact that the position of the months on the calendar changed between the Start and End date fields. Worse, the travel was in February in a non-leap year, so there wasn’t even a difference in date numbers to clue him in (since the Wednesday in March was exactly 28 days after the Wednesday in April). Result? A very long delay for our friend at McCarran Airport in Las Vegas trying to straighten the problem out, so that my friend could get back 30 days earlier than his ticket specified.

Usability mistakes cost.

The Ribbon: a study in good product design practices

When I came to my new company, I didn’t mind going from Vista back to Windows XP. There were a few tools and features that I missed, but ultimately one version of Windows isn’t too much different than another.

But I would have caused a serious uproar if I had to use something other than Office 2007.

It’s pretty rare for me to feel passionate about Office software. The last version of Word that I really, really liked was probably Word 5.1a for the Mac, back in 1992. After that I got proficient with some newer features, most notably working with references and TOC formatting for an 800 page software design book that I put together from individual documents in a ridiculous amount of time back in the late 90s. But it got harder and harder to like Word, and I found the time I spent typing in plain old text boxes on the Web to be a refreshing change. I even adopted Notepad and other text editors as my quick writing tool of choice, and still find I prefer working in plain text to having to think about formatting as I write.

So what changed my perception of Office? Why am I no longer afraid to set foot in Word? Why do I actually evangelize PowerPoint 2007? I think the answer is that the Office team, Jensen Harris among them, sat down and looked seriously at how people used, or didn’t use, Office, and made some hard choices about how to change the software to fix what they found.

Jensen had a great presentation at Mix that explains the process that the Office team used to come up with the new Ribbon-centered design for Office 2007. I’ve only looked at the slides so far, not the actual video, but by themselves they’re pretty inspiring.

What are the takeaway principles? I think the outline of the presentation spells it out:

  1. Define the problem by talking to actual users. Don’t rely on the conventional wisdom to tell you whether there is a problem with your software. Conventional wisdom would have told Microsoft that Office was a cash cow with no issues and no room to improve the brand, that they should just keep marketing the product the same way and keep stacking on task panes.
  2. Get data about the problem. Start with internal observations (menu and command count), then go to external observations to define a hypothesis and an approach based on user interaction data and the emotional tone of how your users approach and think about your product.
  3. Define the design goals, and draft a list of principles that guide the solution. If the solution is too complex to define up front, you’ll need guiding principles to help you make decisions along the way as you iterate.
  4. Prototype. A prototype can be created a number of ways, from plain old paper to Photoshop to RAD tools. Use what’s appropriate and don’t discount the low tech solutions. At my current gig, most of our prototypes are created in Visio.
  5. Evaluate the prototypes. I love Jensen’s list for this process because it sums up about two semesters’ worth of marketing education from my MBA program:
    • Beta users
    • Anecdotal feedback (blogs, forums)
    • Benchmarks and Metrics
    • Observations and Interviews
    • Usability studies (around the world and remote)
    • Card Sorts and Paper Prototypes
    • Surveys
    • Longitudinal Usability Studies
    • Long-Term Deployments (5 months+)
    • Truman Show
    • SQM (Customer Improvement Program) [automated data collection from volunteers]
  6. Finally, build in room for iteration. You probably won’t get it right the first time, but you’ll learn from every trial.

I’m excited about Jensen’s summation of the experience because it is high-visibility verification that structured software design matters. It’s a great story to tell your management team, your development team, or that guy in QA who complains about changes to the screens.

I already know this approach works because I’ve used it, or at least a subset of it. At my last gig, this is what our design process looked like for our last release:

  • Problem: Users couldn’t find functionality in our software that was already there. Some of the functionality confused users. Prospects didn’t enjoy working with our product as much as with competitive products.
  • Design goals: Improve discoverability and make the product more enjoyable to use. Design tenets: reduce the number of toolbars; have consistent places and ways to expose functionality; borrow UI conventions from products that the user is familiar with; use a familiar overall convention to frame what the user is doing and make them at ease; eliminate features that weren’t adding value.
  • Prototyping: Paper, Photoshop, and quick XAML apps. In some cases, I mocked up parts of the UI using sticky notes to move around different modules or parts of the screen design, so we could figure out how to get the features in the most convenient places.
  • Evaluate prototypes: Here customer presentations and beta users made the biggest difference. I would have loved to have the budget for more formal usability testing.
  • Iterate: We rebuilt our development process to allow for course corrections along the way.

The process isn’t rocket science, but it works, and so will your product if you think honestly about every step along the way.

An open look into the mind of an iPhone product manager

Apple has posted its application form for the new iPhone enterprise tools, whatever they are (Apple has been awfully nonspecific on that point). This is cool for a bunch of reasons:

  1. It’s an open, transparent beta process for a piece of enterprise technology.
  2. From Apple. When was the last time you heard any of the words in the first point from Apple?
  3. And there is a fairly detailed feature list too!

Well, prospective feature list. And of course I should note that there’s a chance this comes from a marketing manager who’s looking to write case studies. But still: cool.

How to say “no” to feature requests

I am in awe of another product manager: the Cranky Product Manager. She posts infrequently but many of her posts are brilliant.

One from last year on “how to say no” to customer or sales feature requests caught my attention. Many of the comments on the post were insightful and mirrored my experience with dealing with feature requests. Generally if you can turn the conversation away from a specific feature and toward the business issue the customer is trying to solve, you can usually either (a) point out another way the software can already do what the customer wants, or (b) give a more informed answer about what it will take to make it happen. The trick is getting away from the specific feature, which the customer and the sales guy may have spent some time building up as The Solution to the problem. To do this, Joel Spolsky’s five Whys is helpful.

Being open with your roadmap is important when you can; it shows the customer that you are thoughtful about the features that do go into the product, gets them thinking about the relative importance of the feature, and makes them feel included in the future of the product.

What makes a good product manager?

While there are whole blogs devoted to the topic of good product management, I don’t think anyone has a good answer for how to interview for a good product manager that doesn’t boil down to “it depends.” There are sound reasons for that: There is a continuum of roles that a “product manager” can play, ranging from almost entirely strategic/market focused to almost entirely development/product focused to somewhere in the middle. The roles I have played have been at all ends of the spectrum, or all roles at once; at most companies the role falls somewhere along the continuum depending on company size and organization. The competencies that you look for during the interview depend strongly on how you expect the product manager to function in your organization.

That said, here are some things to look for in a product manager, divided into categories:

  1. General
    • Communication skills – one of the major roles of a product manager is being able to speak Sales to Development and vice versa and be understood
    • Decisive – need to take input and synthesize a clear direction even if it’s unclear what the right way to go is.
    • Negotiation experience – important when brokering a deadlock between priorities, or between sales and development.
    • Business savvy – understand how the product can make money. Prioritize according to business needs, both long and short term
    • Market savvy – understand the competition, the strategic landscape, and how the product fits
    • Driven to learn – do they keep up with what’s going on in the industry? Ask them what are the challenges for product managers in an Agile environment and you’ll quickly get a sense for their depth and how much they think about their role.
  2. Development-facing specific
    • Tech savvy – I would say that the ability to quickly pick up and understand the challenges of new technologies is more important than specific experience in the languages your development team is using. I was a PowerBuilder dev, but I was able to understand the peculiarities and challenges that my development team experienced in C++ vs. C# vs. XAML.
    • Project management – Tracking, detail orientation. This can make or break a product manager on a day in, day out basis.
    • Facilitation and meeting management – In a lot of teams the engineering lead drives this almost entirely. But it’s a critical skill to have in your pocket, from requirements gathering to release post-mortems to gathering agreement between different stakeholders on what is to be done on a particular problem.
  3. Market-facing specific
    • Experience working with analysts – Some companies have separate AR roles; some don’t. It’s a pretty good bet your product manager will need to face a skeptical analyst to describe the product’s direction at some point, and it’s a good indicator if a candidate has had that experience under her belt already.
    • Strong demo skills — even if the PM doesn’t do sales directly, they might do conferences, and at a minimum they will need to champion the work that development is doing.

Finally, there’s a really good article on Pragmatic Marketing that sums up some value drivers for good product managers as: Attitude, Knowledge, Communication, Customers, Managing, Decision Making. It’s worth looking into.

Feeling like a copier

Great product manager joke from Art Petty (via Steve Johnson at Pragmatic):

Q: Why is a Product Manager like the Office Photocopy Machine?

A: Because when it works no one notices and when something goes wrong, everyone wants to kick it.

Heh. Heh. TANJ, indeed.

Of course, many a product manager has done the “Danse Russe” (“I am lonely, lonely, I was born to be lonely, I am best so!”), but I’ve found an interesting dynamic working as the second product manager at my company. In addition to the extra pair of hands to manage the workload, there is some reinforcement and validation. Of course there’s always the danger that that turns into “us vs. them,” so that is something to watch. But it really does help having a sanity check on team interactions, process questions, requirement prioritization, and all the other caution areas that product managers face.

Agile and the Iron Triangle of Software Requirements

Software companies make fundamental tradeoffs between cost, quality, and functionality (the Iron Triangle) all the time. These tradeoffs happen in the software itself, of course, as anyone who’s ever heard this analogy will tell you. But they also happen in the process of making the software. In software development processes, cost = effort, resources, and time, and functionality = functionality, of course. But what is quality?

I don’t mean this to sound like a Zen primer or a TQM discussion, but software quality is much more than the negative condition of having no open bugs. It’s also the positive condition of building the right thing. Generally speaking, this means product strategy and product requirements. And here is where the arguments usually start, because it’s far from clear how to build The Right Thing. Traditional software development tries to ensure that The Right Thing gets built by requiring a multi-stage sign off and by providing extremely detailed documents at each stage of the process—a marketing requirements document for initial concept sign off, followed by a functional requirement, a technical requirement, a detailed test plan, and so forth. Between each of those stage gates is a lot of work. Many months may pass between initial concept and development.

So using rigorous stage gates to build in “quality” comes at a cost, especially in time. This is particularly risky in a new market space, where time to market and time to benefit is key. If you commit to a lengthy waterfall process, you lock yourself into a feature set that commits the team to a path of several months or even years to build the Right Thing.

But in a year’s time, the market may have shifted out from under you, and not just once. Clearly a way of building software is needed that doesn’t require so heavy a use of resources to define requirements and that can allow the Right Thing to be built more quickly, and that allows for more opportunities for the team to “change its mind” without wasting a lot of work. Agile software development methodologies seek to provide those opportunities by defining the work product to be built in terms of “iterations,” where the software gets built in discrete, short sprints rather than one. Something concrete gets built in each iteration, but the team is free to change its mind about what to build in the next iteration—and it may need to, either in response to feedback about the product of an earlier iteration, or because things have changed in the outside world.

To support a rapidly iterating software product, though, you need a requirements process that moves quickly as well. That’s where Agile gets its emphasis on “stories.” An Agile story is a simple way to specify a requirement, and it follows a pattern of “As a user type, I want to do something because of an important reason.” It’s a streamlined but easy and effective way to specify functional requirements. The idea is that each story is a standalone piece of functionality that delivers value to a user. Therefore, every discussion about choosing features for a release discusses trades between things that actually benefit the user, which is a much easier decision to make than choosing between a functional requirement and a technical requirement.

As you might guess, adopting Stories as a requirements methodology has its own challenges. In this case, the tradeoff between requirements “features” (detailed documentation) and requirements “quality” (preserving flexibility by not overspecifying) has real downstream impacts on other parts of the organization. For instance, Marketing teams are used to using the MRDs for product features as input to their messaging campaigns. And every team in the organization misses the stage gates as a chance to come in and provide feedback about what is to be built. So the challenge becomes how to build a process that remains flexible and adaptable, doesn’t get in the way of the participants, and ensures that the right deliverable gets built and brought to market correctly.

There are some pretty good suggestions out there on how to do this, as it turns out. Googling agile and mrd gets you a variety of good discussions and ideas, including the following:

  • Only write an MRD when you need to (Luke Hohmann)
  • Let your prototype be your requirements doc when you can
  • Demo each “sprint”’s results to everyone in the organization while sharing the prototypes for the next iteration (both from SVPG)
  • Keep your requirements at one level. Don’t write high level requirements documents that are then translated into low level documents. These are a good example of Conway’s Law (Roger Cauvin)
  • Use a quick 10-question checklist as a way to structure an opportunity assessment (SVPG again)
  • Minimize the impact of your iterations on your users by communicating with them (SVPG once again)

I will try to expand on some of these topics in future posts.