Dec 17, 2013
Since I got into software development back in 1995, I have been exposed to many different software development methodologies. I started out with a highly structured waterfall model (which is hard to avoid with fixed bid projects). Whenever possible, I have advocated agile approaches and I have been on teams that have practiced agile methods like Scrum with varying degrees of fidelity. Overall, I love the values of agile but the agile methodologies that I know of don't quite feel right for the kind of work I do now.
These days my job focuses on managing several different web sites and applications. Officially, you could say that these applications are in "maintenance mode" but the truth is that they were always in maintenance mode. All of these applications have been built on the philosophy of a minimum viable product. That is, we developed a minimal solution to test a) that the problem actually exists and b) that our idea is an effective way of solving it. The initial solutions were just complete enough to answer these questions and create a foundation for an evolutionary process towards continuous improvement. Everything since has been "maintenance."
The agile development methodologies that I know of don't quite fit because they all seem to focus on a notion of defining a 2-3 week "release" or "sprint" and putting the entire team onto cadence that begins with planning and ends with some kind of review. But my teams do not work that way. We release one feature at a time. We don't wait for some large milestone to push new code into production. In fact, we push a new feature as soon as it is ready. We see an un-published feature as a huge liability. Worse than simply not being able to get the benefit of the feature or see how the audience responds to it, unpublished code is a risk. When code sits in an unpublished state it needs to be maintained like working code. Otherwise, when you deploy it, you are likely to break other things. Also, large deployments are riskier than small deployments and make problems harder to diagnose.
We try to define features to be small as possible. Some features are "below the waterline." That is, they are not visible to the end user and are only deployed to support another feature in the future. A very common example of this pattern is that we usually do our database work up front. For example, if we need to build a new page design, we will start by deploying the new fields that the design requires. Then our content team can add the data. Later, once the data is in place, we can deploy our page templates and other application logic that relies on the new data. We also commonly deploy infrastructure upgrades ahead of the features that they support. We also try phasing in features with an initial simplistic implementation to test out the concept and then a series of enhancements to reach the full potential.
When you are maintaining a live application, the line between a bug and enhancement is blurred and the same team should responsible for both. Anyone on the team needs to be able to pull herself away from the feature she is working on to fix an urgent issue. This is another reason why keeping things small is very important. It is also why it is so important to have a very clearly defined prioritization protocol. Ours is:
-
Site down
-
Feature broken with no work around
-
Feature broken with work around
-
Revenue opportunity
-
Customer Usability
-
Operations Usability
-
Brand consistency
The product manager is in charge of setting the priorities and assigning high priority issues that need to get taken care of ASAP.
The lines between coding, content, infrastructure, and support are also blurred. Sometimes an issue is noticed and the cause is not immediately apparent. For example, let's say a customer cannot download a file. It could be that the file was not added in the first place. The code to retrieve the file could be broken. The server could be out of space and the file could not be saved. It could be a usability issue and the customer was clicking on the wrong link. Triaging and coordinating this work is critical needs to be core to the methodology.
Refactoring is another big part of the methodology. You often run into situations where a feature gets a lot of traction but not in the way that you expected. You might need to change how it was built to allow it to evolve in the right direction. You will also need to refactor your successful experiments to support active use. The concept of a "user story" (common in agile methodologies) does not lend itself well to code refactoring.
As you can see, traditional agile methodologies do not accommodate the day to day activities required to maintain and extend an actively used web application. This is because they are designed at their core to support software development projects where the team has the luxury of ignoring the operational aspects of running the application as a service. But that luxury comes at a price: alienation from critical feedback by actual consumers of the application. In agile, a "customer" role filters and simplifies end user needs and tastes. This customer makes a lot of guesses about what the end customers want. The guesses tend to be big and there is no process for correcting the wrong ones. The agile approaches that I know of seem better for building packaged software or for integration companies that are brought in to build something and leave. These are cases where you cannot or don't want to be told that you need improve something right after you thought you built it to specification.
Here is a very real world example to illustrate that point. We use LivePerson to chat with customers who are on our sites. Sometimes there are cases when a customer gets stuck because he does not notice or cannot understand an error message — such as one telling him that he needs to enter something into a certain field. If it is common enough, we will use that feedback to design, implement, and deploy a fix that makes the validation message more visible and understandable. Fixing this issue could be hugely important to conversions. But think about how this problem might be solved in Scrum. A "user story" would be added to the backlog. At some point, this story makes it into the next sprint. It needs to wait for the rest of the sprint to finish before it is deployed. If you do two-week sprints, that could be up to a four-week turn around. You might say that you would handle this in a maintenance track that runs in parallel to your real development. But I would counter that tweaks like this are where applications succeed and fail and should be the focus of (not the exception to) your process. In fact, I think you should think of everything as a tweak.
Despite the fact that lots of organizations are maintaining sophisticated web applications and want to experiment with new features as fast as possible, I have found little writing on formal methodologies to support this. The best writing comes from the DevOps community but they don't directly often address the actual software development as much as I would hope. I like Eric Ries's chapter on continuous deployment in Web Operations: Keeping the Data On Time
. Speaking of Eric Ries, lean startup literature is also a very good resource. But this community emphasizes customer development and doesn't explore the architecture/implementation side as much as I would like.
What really inspired me to write this post was this relatively obscure research paper called "An Online Evolutionary Approach to Developing Internet Services." This fit the bill when I was looking for a term to describe how we work. The word "evolution" is perfect. It captures the idea that development is a series of small modifications that respond to stresses and opportunities. Each modification is tested in the real world. The ineffective ones die off. The good ones are the foundation for future progress. It is true natural selection. We just try to be a little less random than biological evolution.
I would love to hear your feedback on this. I go back and forth between thinking that we are outliers for working in this way to thinking that this process is so universal that nobody thinks about it. I also think there is a lot to discuss with regards to testing, communicating changes to users, and localizing the solution.
Jun 10, 2013
Estimation has long been a point of embarrassment for the software development community. A big part of that is unrealistic expectations set by association with construction and manufacturing industries. Software is unlike building houses and widgets because every software application is unique and unprecedented, whereas construction and manufacturing most often involve assembling well-understood components in repeatable ways. Truly innovative construction projects, like the Big Dig, and the Sydney Opera House, regularly overrun their estimates (by 3 times and 15 times respectively in these cases). They shouldn't be compared with building out a subdivision either.
A more appropriate analogy is a moving truck. If you have ever hired movers to help you move between homes, you know that the process starts by someone walking through your house and looking over your stuff. This person may pull out a tape measure a few times but she is not measuring everything and she is certainly not doing things like seeing if your ottoman can be neatly wedged underneath your dinner table. The real purpose of the exercise is to know how big a truck (and team) to bring on moving day. It is up to the movers to make the most of the space and there is a real art to that. Sometimes the moving team can see right away that the job is going to be a real squeeze and they think through every piece like a game of Tetris. But sometimes the truck is just too small, you need to call in another truck. Either that or pulverize your furniture into very compact dust.
The main point of this analogy is that precise software estimation is impossible at the detail level. It takes people who have done lots of projects to give a decent "truck-size" estimate, and even then they can be wrong. In fact, each task estimate will likely be wrong. Accuracy is achieved only at the aggregate level when the underestimates cancel out the overestimates. Just as important, an experienced team will be quick to identify the need to adjust: either by increasing the scope (size of the truck), by leaving low value items on the curb (decreasing scope), or by changing how things are packed together. The earlier you are aware of your constraints the more options you will have. Only inexperienced teams wait until the truck is full to realize that some adjustments need to be made.
Related: Work Breakdown Structure vs. Deadlines
Aug 08, 2011
I am trying to get a friend of mine to quit his job. No, I don't want him to join the massive ranks of the unemployed. I want him to move to a job that appreciates his talents and efforts. My friend (let's call him Bob) is totally dedicated to his profession. He is continually trying to improve his skills and productivity and wants to help his colleagues achieve better results too. Nobody brings more thought and interest to any task he faces. But he is surrounded on all sides by people who just don't care. They are satisfied by doing mediocre work. If something they build doesn't immediately fall over, they consider it done. If they can assign responsibility to someone else, they will. If nobody else notices a problem, it never happened.
It's frustrating for Bob. He can't accomplish his goals without help or at least cooperation. His company doesn't recognize the difference between his performance and that of his peers. In fact, he is constantly getting passed over. It doesn't feel fair. But it is fair. Bob is just playing the wrong game. Queue the metaphor...
Imagine that you are a world class tennis player. You live and breath tennis. You go into every volley trying to do your best and to improve upon the last one. You invest in coaching and resources to improve your game. If something might help you play better, (like a different diet, a different racket, or even one of those magnetic bracelets) you will try it.
One day, a friend challenges you to a game of Wii Tennis. You start to play tennis as you know how. Your knees are bent. You move your feet. You swing with explosive power. You are playing great tennis... but you are losing. You look over and you see your friend. He is slumped on the couch barely moving his controller with a flick of wrist. He looks like he couldn't walk across a tennis court without losing his breath.
As pathetic as he looks, Wii thinks your opponent is a better tennis player. Wii doesn't care about his form, his position, or how much leverage he has over the ball. Wii only cares about timing and maybe a couple other factors. Your friends knows this and he is not wasting any energy on what Wii doesn't care about. Your friend is the better Wii Tennis player because he has figured out how to do less but still satisfy the minimal inputs that Wii has been programmed to observe.
Bob is trying to play real tennis in his company's Wii Tennis tournament. He is not exactly losing, but he should be winning. The frustrating part is that he could be doing a lot less and get the same results. When he is on a team, he is constantly questioning whether he should be compensating for others by fixing their work — like in a doubles game when you see your partner is not running so you cover more of the court. His extra efforts only result in bumping into things and annoying people who just want to "sit on the couch and casually wave their hands."
Now the big question: who is playing the wrong game? It is difficult to tell if Bob's company would perform better if it paid attention to the finer aspects of how people worked and measured performance more holistically. It is unknown whether the company's customers would appreciate a higher quality product. But there are two things I know. 1) Bob wants to play real tennis, not Wii Tennis. 2) He doesn't have the clout to change the game that the rest of the company is playing.
The reason why I am writing this post on this blog is that I think that this story is very relevant to the pursuit of any type of organizational change, be it adopting Agile development or Enterprise 2.0. Companies can delude themselves into thinking they are playing a different game than they actually are. They may think they have a bunch of real tennis players who are striving for excellence and victory on a dynamic playing field; instead, they have a bunch of Wii tennis players who are looking for ways to minimize their personal/professional investment to only what is recognized. These companies get confused when they put a tool or way of working out in the wild and nobody adopts it; they shouldn't be.
Organizational change is much harder with the Wii Tennis company because you have to actively incentivize every behavior you want to see. It is not enough to say "this will make our company more effective." You have to say "do this specific thing and get points towards your performance review;" and you better not be lying because Wii Tennis players will see right through that. Finding real tennis players like Bob is hard to do — especially if you work for a large company. Real tennis players need a lot of room to move around and large companies tend to compartmentalize people. They need to be challenged in different ways to keep things interesting. Their rewards need to be tied to company achievement. Plus, other employees will get annoyed when you change the game that they have gotten so good at.
When building a company or even just a team within a company, think about what game you are playing. If your industry is commoditized and the difference between average and excellent is under appreciated by the market, it may pay to go after the Wii Tennis players and be very specific about expectations and incentives. If you are competing in a dynamic industry with a large upside, build a team of competitors who will take ownership of optimizing their personal performances and also the effectiveness of the overall team. They will innovate and try new tools and techniques that are offered. Most importantly, they will not stop a practice if it is difficult to do — they will only stop if it is ineffective or if they find something more effective.
May 24, 2011
Over the past year I have been doing some side work building a web application for a startup. The project has been very interesting and the process has helped me stay in touch with my inner developer. It has also allowed me to practice agile product management philosophy to an extreme.
Last week I read Andy Hunt's article "Uncomfortable with Agile" and I have been ruminating on that feeling of being in a constant state of discomfort. If you click through and read the article, you will learn that discomfort is a good thing. The alternative is blissfully trusting a process — and then finding that you were wasting your time. Agile revolves around challenge. You challenge your assumptions and your limitations. That discomfort that you feel is really the sense of awareness that everything is in play and you are personally are accountable for a successful outcome.
Recently, I have been feeling like architecture in an agile environment is a little like wrestling or aikido. The goal is to maintain your balance and stay on your feet. Your opponent (the customer) pushes requirements in one direction and you respond by building up the architecture in that area like you might move your feet to stop yourself from being knocked over. Then, all of the sudden, the customer changes direction and you need to quickly adjust. The trick is to never over-commit in one direction or the other. Over-committing will put your balance at risk because, if the direction of force changes, all your weight will be on the wrong foot.
We see over-committing in architecture all the time. You go in thinking one feature will be the center of the application but then you find that little thing you built on the side as an afterthought is the killer feature. This is especially true with startups that are constantly "pivoting" or redefining their product. In my experience, most over-commitment sins are done in the name of performance optimization. You structure the data or the code in such a way that you can cut some corners and shave some processing time. Sometimes these optimizations are absolutely necessary but they leave you exposed for the next weight-shift. When this happens, your next iteration is spent regaining the balance of the application (refactoring) rather than adding new features.
The next time you are in charge of the technical design of an agile project, think of it as an aikido match. Fluidly adjust to changing requirements but try to stay away from positions that extend too far from a position of balance. Otherwise plan to take time getting back on your feet.
Jul 21, 2010
One of the most common points of friction between project managers and developers is planning work. Most programmers hate creating work breakdown structures (WBS). You can't blame them, accurately predicting steps and effort required to build undesigned software is impossible. Yes, you heard that right. Software development planning is impossible — at least for someone who likes precision, which most programmers do.
The problem is that every software development project is a unique collection of thousands of tiny details that each have the potential to suck up enormous amounts of time. The traditional, PMI-sanctioned WBS technique forces developers to name all the activities that will be required, sequence them with dependancies, and then create an estimate of each one. The assumption is that if you did the planning right, you should just be able to follow the steps and come out the other end on time and on budget. This also implies that if you didn't blindly follow the steps, the project plan was wrong — or you were too incompetent to follow the steps correctly. But with the fluid nature of software development, the project plan is always wrong. I used to think that precision would increase with finer granularity. The more lines in the project plan, the more accurate it would be. But now I think the opposite is true. The more tasks you add, the more guesses you make and the greater the overall variance. Even if you guessed every task right, there were probably just as many tasks that you forgot to add. And there are also lots of steps that you find you didn't need to do too.
While predicting a WBS is impossible, developers can get better at setting and meeting deadlines. There is a small nuance between setting a deadline and estimating tasks in a WBS. On the outside, the difference is so small that no one will notice. Nobody will care because they just want to know when the work will get done. But there is a difference. The WBS technique forces a linear accounting of all the work that needs to be done. Creating a deadline is more like adding a constraint (that you hope is reasonable) to help guide and prioritize the work that you wind up doing. Comparing the two is like comparing launching a rocket to flying a plane. PMI-style planning is like shooting a rocket: doing all the calculation at the beginning and then hoping that you accounted for everything before ignition. Setting a deadline turns the rocket into an airplane by adding a pilot that can steer. Realizing you can make adjustments after take-off transforms the pre-flight calculations from a fixed flight path to a map that you can use to make in-flight decisions. A deadline (either the final deadline or an intermediate milestone) is where you think you can be at a certain point of time (or after a certain amount of effort). When creating a deadline for yourself, you don't try to think of every possible task it will take. It is more like eyeballing distances than counting steps.
I became conscious of this distinction the other day when I was on a bike ride. I take pride in the fact that I usually get home within a few minutes of the time I tell my wife I will be back. Lots of times I pull in right at the minute. Putting on my planner hat, if I was asked how long a bike ride would take, I would want to know the exact route and measure the distance and slope and windspeed and make assumptions about average speed. When I put on my cycling helmet, I realize that most of those variables are under my control. I can shorten the route. I can ride faster. I can take an alternate road to stay out of a headwind. Because I know my cycling ability and the terrain so well, I make these adjustments without even thinking about it.
I know you are thinking that software development is not like riding a bike. There are all these externally imposed requirements, constraints, and dependencies that need to be accounted for. But think back and ask yourself: how many of these factors are added specifically for the purpose of creating the WBS? I feel like developers work against themselves by asking for more and more estimation inputs and being more prescriptive of how they will work. There is no way that every detail can be accounted for and every detail that you do add will constrain your ability to make adjustments.
For estimation purposes, requirements should represent boundaries of an acceptable solution. With this understanding, a developer needs to produce a reasonable deadline based on similar work and explain any assumptions made. An overall deadline or intermediate milestone shouldn't be overly ambitious. It should account for unknowns. If a deadline is not acceptable, scale back the scope until an acceptable deadline can be achieved. Through the course of the project, new information is going to present itself: the client is more particular than he was able to articulate; the available components are not as good as expected; new features are added to the scope. When any of these things happen, you make adjustments. You might be able to work a little more efficiently. You might be able to scale down scope in other areas. You might be able to delegate work back to the client. Or, you might just have to extend the deadline.
These adjustments require a decent partnership between the developer and the client where the deadline is jointly owned. It doesn't work when one party feels like the other is obligated to deliver no matter what. In the bicycle analogy, when two people go for a ride, they decide where they want to go. Usually the conversation plays out where one rider asks the other what sort of ride he is up for. The second rider may say he needs to get back in 2 hours and wants to get in some climbing. The first rider will suggest a route that he is familiar with. When they encounter construction that makes a road impassable, they may be able to find an alternative route that is just as good; they can hammer home over a longer route in a paceline; or they can call home to say that they are going to be late. Whether the first rider should have known about the construction is debatable (Did the construction just start? Was the overall distance to ambitious? Did the route not allow for adjustments?) but debating is not going to get anyone home sooner.
With experience, you do get better at making more realistic deadlines. And, more importantly, you also get better with time management. You will build an awareness of where you are in the overall process and know early if you are falling behind schedule. In the cycling analogy, you periodically glance at the clock, your current speed, the slope of the road, and which way the wind is blowing. In software development, you are looking at things like the calendar, the productivity, and the rate of defect identification. With this information rolling around in your subconscious, you start thinking about options instinctively. The client perception is that you planned well. But you really didn't. You managed time well. The up front estimate was just one of the many constraints that you juggled when developing the solution.
Apr 09, 2009
My friend and colleague Brian Kelly and I are starting an Agile Developer Book Club in New York. The first meeting will be on April 29th (location TBD) and we will start by reading one of my favorite books on software development: The Pragmatic Programmer: From Journeyman to Master
. After that, we will move onto other books in that genre. The general idea is to assemble a cross-language group of developers that want to improve their craft by learning from great books and each other. Each meeting we will discuss a section of the book. Beyond that, it is up to the group to figure out the details.
If you live or work in New York City and want to hang out with some people who also want to be great programmers, please join the Meetup group and block April 29th on your calendar. If you are not in New York, you should still read The Pragmatic Programmer and talk about it to whoever will listen.
Note: in case you were wondering, I have not relocated to New York. I still live in the heart of the beautiful Pioneer Valley but spend a lot of time working in New York.
Mar 06, 2007
I frequently hear the terms "Proof of Concept" (POC), "Prototype", and "Pilot" used interchangeably. This is unfortunate because they mean different things and serve very different functions. In fact, I think that if you want to successfully implement technology, you should be doing all three. The good news is that, at least to some level, you probably are. But the extent to which you are benefiting from POCs, prototypes, and pilots depends on the level of consciousness and deliberateness you practice. By "deliberateness," I mean having an understanding of purpose. I use "consciousness" to say that you pay attention to the process and are aware of the results. More on that later.
Proof of Concept
A Proof of Concept (POC) is a small exercise to test a discrete design idea or assumption. Software developers tend to do POCs instinctively when they experiment with technology. An example of a POC is testing whether one technology talks to another. To be deliberate, a POC should clearly state what it is to be proven and to what degree. For example, it may not be enough that a message can be transfered from system A to system B if the complete solution calls a specific level of performance and reliability. You may need to test under certain conditions. POC's often take the outward form of a simple "Hello World" but that may not be enough if the system needs to support an extended character set.
The results of a POC need to be measurable so that they can be fed into the decision making process. Your decision should be based on thresholds like a performance level that is acceptable. Thats the consciousness part. It may not be enough to for someone to just say "yeah, we tested that and it works." You want the results of the test to see how or how well it works. A POC may involve secondary research if you trust the source. Maybe another group tested the same concept. If their process was as rigorous as yours, you may be able to use their results. There are also user interface POCs that explore the usability of a specific UI control or visual metaphor. These need to be designed and tested with the same degree of thoroughness.
Prototype
A Prototype is a more fleshed out system that tries to simulate the full system or at least a material part of it. Prototypes are used to test the viability or usefulness of a system or subsystem. You plunk an intended user of the projected system in front of the prototype and he/she/it (the user might be another system) should be able to visualize the experience of using the complete system. The completeness of the prototype depends on what you are testing. For example, you might want to build a three dimensional physical model to test an innovative new shape for a telephone handset. If there is question as to whether the person can hear a speaker which has been designed in an unconventional place, the prototype may need a speaker. Or possibly, you could do a POC of just holding a speaker in that position and testing audibility in different conditions. Pretty much all manufactured goods go through a prototyping process because that is usually the best way to figure out the feasibility and cost of manufacturing the product. For software, prototypes can have varying degrees of fidelity from visual mockups to working software.
You need to be deliberate in defining what you hope to gain through building the prototype and the minimal level of completeness that it would take to get that information. There is some disagreement as to whether a prototype should be constructed to be thrown away or whether it should be the initial building blocks of the complete system. I think it depends on a number of things. You should invest as little as possible in the prototype otherwise the sunk costs might make you stick with a bad idea longer than you should. If that means building it out of a technology which is agile to use but unsuitable for full scale deployment, then use the easy technology with the expectation of throwing it away. However, if you are dead sure on the core idea and you just want to prototype different variations, it may be more practical to build the core technology framework and then build prototypes off of that.
If you go through the hassle and expense of building a prototype, use it. Get it in front of your intended audience and listen to what they say. Don't just listen. Hear them. If you are not doing to do that, don't call it a prototype. Call it an iteration.
Pilot
A Pilot uses the full production system and tests it against a subset of the general intended audience. The reason for doing a pilot is to get a better understanding of how the product will be used in the field and to refine the product. Maybe there are some open ended questions about scalability that only a live audience using the product can answer. If you are doing a Pilot (or Beta) for a reason other than to build hype, make sure that you target the right group of people and you are prepared to collect the right metrics. You may consider prepping your pilot group to evaluate the full breadth of the application and give them channels to provide feedback. If there are particular areas of the application that you really want to test, call attention to them. Determine what kind of quantitative metrics you will be collecting and have the tools and time to analyze them. If you are going to do a Pilot you should be prepared to act on the information that you collect and fold refinements into a general release. You should probably have a process to prioritize enhancements ideas and a threshold for determine what enhancements to implement.
POCs, Prototypes, and Pilots are all important tactics in implementing technology. However, just using the words is not enough. You need to engage in these activities deliberately and methodically for them to be of any value. And you will never realize that value unless you consciously record the results and turn this data into usable, actionable information.