Software Engineering Overview at Divergent Labs

This article describes the software development approach used at Divergent Labs. In the early part of my career I focused on small projects ranging in size from $500 to $5,000 each, completing a dozen or more of them per year. Through that experience I learned how to manage rapid life cycles efficiently and comprehensively. The process I developed works for larger projects too, and so it became standard for the work we do.

Linear project flow for accuracy

Creative work is by necessity error-prone and involves a lot of mistakes that get fixed. Even a senior programmer could create multiple bugs in a single line of code in the first pass, and perhaps 70-90% of code that gets written is revised either immediately or later in the project. The same is true in the planning stages – a lot of the first draft of any document gets revised. So it is fair to say that most of the steps taken by software engineers turn out to be the “wrong” steps, and it is an iterative process like any other creative work.

The key to streamlining an inherently error-filled process like this is not to stop making mistakes, but rather to make sure that errors are caught early. Ideally 90% of code bugs are caught by the author in the same day that they are created, so no one else ever sees them. But when too many mistakes linger without being rapidly fixed, it bogs down the whole project and can make it necessary to go backwards a long way to fix them. Going backwards like that is a cost that someone bears (either the contractor or customer depending on the type of contract), and it cannot be estimated in advance.

The biggest lesson in all of this is to get it right the first time through, and only travel once through planning to building to testing. So, we try to proceed through the phases once without backing up, but while doing that, we may revise the details thousands of times (rapidly) during the project.

Deliverables

There are a number of milestones that make up the system, each with a deliverable. These are:

  • scope – A scope defines the size of a product, in order to estimate the overall hours of effort. This is what is written in the contract and forms the basis for the estimate or fixed price. Examples of statements of scope would be “a database system with up to 20 tables” or “up to 30 data entry screens” or “automatic invoice printing capability”. The scoping process only needs as much detail as necessary to estimate the effort.
  • requirements – A requirements document lists in non-technical language exactly what the product will accomplish, while carefully omitting any statements about how it will be accomplished. It takes longer than any other phase because there are usually more people involved. Examples of statements of requirements would be “will track the grades of students for each class by semester” or “obtain shipping address from the accounting system but allow the user to override it”. The most important thing about requirements is to make sure it is limited to statements of purpose, and does not include design points, screens or other visual elements.
  • architecture – Architecture is comprised of the list of main components and the languages they use to speak with each other. The architecture phase determines, among other things, whether the project will be web based and/or installable apps, what format the data lives in, and where it lives.
  • design – “Design” (in this approach) is the detail of the components – all the exact database fields, entries on each screen, background processes, and so on.  (More detail on design methods here)
  • documentation – Documentation follows from the design document, but only those parts that are needed for ongoing reference. It is set up at the beginning of the building phase, and then added to as the coding progresses.
  • product – This is the executable product, plus all the scripts, data, reference material, and other assets that go along with it.

As an example, consider an inventory management system. The scope might take a couple days back and forth to work out, and the requirements might take 20 hours over a duration of 3-5 weeks. After that milestone is approved, the architecture will only take a matter of hours, and the detail design might take a few days. Then the coding might take a couple weeks, and validation and changes might take another 3-4 weeks.

Common mistakes in the sequence and milestones are:

  • Using “agile” as an excuse to stop planning ahead – While the agile methodology is a good way to organize teams, it should not mean that the project is globally iterative (constantly going backwards) with nothing ever being pinned down.
  • Doing architecture too early – A lot of processes put the architecture earlier along with the scope. For example, they might scope out a “web based inventory system to be written as a WordPress plugin”. But the problem with that is that once the requirements are fully known and approved, it may turn out that the pre-conceived architecture is not actually the best way to achieve the requirements.
  • Failure to do requirements; jumping right to design
  • Failure to do design at all
  • Failure to separate requirements and design, or making the design just a more detailed version of requirements.
  • Documentation as an afterthought – When documentation isn’t done along with building, it won’t get done at all. Then future work on the product is slowed down by having to reverse-engineer the product.

Mental tracks

Within software development, there’s a number of different kinds of work, or mental tracks. At one time you may have to crank up one aspect of your brain or personality to focus on one type of outcome, and at another time it’s different. It’s helpful in my experience to allocate a large chunk of time to get into these tracks, and not attempt to bounce between them a lot on one day.

  • discovering – In the requirements process as well as during bug fixing, the developer has to be in the receptive listening stance of discovery, always making the assumption that the customer or user is right, and not forcing much structure on things. Discovering is social with the developer as a student, being quiet, studying or asking only to clarify.
  • playing – In the design phase (particularly in prototyping) and sometimes during coding, there’s an exploratory stance, wondering how things might be best done. It’s a receptive dialog with the tools and technology, sometimes wandering and often helped by going on walks for a change in scenery. The attitude of playing a game makes it fun and makes us more creative.
  • controlling – The controlling mental track is used when you know how to do things and you are the master, meticulously converting detail aspects of what needs to happen into aspects of the built system. This is by necessity solitary and interruptions are terrible when you are in this track. The developer makes choices quickly from experience and sticks with the choices, forcing the system to work with a heavy hand.
  • validating – A mental track used throughout a software life cycle is asking “really”? Is this scope reasonable? Is this requirement really unambiguous? Did you really mean that? Is that the best data structure? Does it work if I try to enter bad data? It requires adopting an extremely critical, skeptical mindset against some proposition, including those that we ourselves have invented.

It takes time to develop each of these mental acuities, and it’s important to get into them separately to do them well, rather than always trying to be in all tracks. One of the mental tracks that I’m not including in the list is fighting, even though it’s pretty common for a developer to fight against a poorly-made tool or an unrealistic deadline. When in a fighting stance, this shuts down the ability to do any of the needed mental tracks really well.

Pipelining multiple projects

The intensity of effort on the engineering side is concentrated in the technical design and coding phases. Before and after that, the work load is more minimal. With overlapping projects, it’s important to know how to pipeline them so they each get the right amount of attention, and so that one project doesn’t eclipse another one.

To manage the pipeline, I use this priority order: (1) overhead tasks; (2) review and validation of already-built projects; (3) estimating and requirements on new projects; (4) design and building of the current project. By consistently applying this priority system, which makes the current project the lowest priority, old projects don’t drag on and there’s less time spent waiting on a bottleneck.

Change management

Most projects are changes to existing products, such as new features, and therefore change is the norm. One of the criticisms of a linear project flow is that it can’t be reactive to changed requirements and new understanding. I still favor a linear project flow, but there are two main things to understand so that it can work. The first is that if there is significant risk in having to go back to requirements during the project cycle, then probably the scope was too large to begin with. We should not as individuals or teams approach a project that is larger than our ability to capture it in written requirements and estimate it. If it is too large, it becomes the master of us, and we end up in fighting mode. So if we have a very large project, we have to carve out a doable piece and go through the phases for that one piece in a forward-only manner, and complete it. At that point we should have a better understanding of the next project scope that can be done.

The second and related point of understanding is that for an existing project, each change is a separate project, with its own requirements, design, code and test phases. With the proper pipelining, this is an efficient way to manage change.