The Software Studio: A Software Production Process

"The Fog Creek approach really is that programmers are talent... the programmers here are like the stars and the directors of a movie."

Joel Spolsky on Fog Creek Software

Having worked over the last few years on a relatively unstructured project, I have a growing angst to elaborate a clear outline by which high quality software may be produced without resorting to tears. This framework is inspired by the film production process in having distinct production phases. The key idea is to drop the notion of a 'life cycle' (which I critqued in a previous post) and instead think of a delivered product. The main difference between film and software is substantial: while film is designed to be consumed without regard to consumers feedback, software is interactive and heavily depends on user feedback. The middle ground between the two is game development, which essentially amounts to interactive entertainment though the 'feedback' is typically transitory - the purchase of games is an indication of its popularity while for software it is continuous use that demonstrates satisfaction and popularity among users.

Therefore, the goal of this outline is the finished product. Obviously, as with ordinary software practice, it is expected that the developers will offer support for a limited duration after which it will be assumed that all major operational bugs can be lived with.

Most commentators will discretise the film production process into five stages:

• Development, in which the producers source the script to be realised into a film. Without a good, well written story nothing much is worthwhile. The right story opens up the financing needed to fund the whole enterprise.
• Pre-production entails the establishment of the production company together with all the artistic and technical concerns which will be required to realised the producers vision. This stage ellaborates the script into various outputs, each of which describes how a technical aspect of the final product will be produced. It is also the stage during which the talent for the production is sourced, not restricted to the actors, but the directors, cinematographers, lighting artists, decor, costume artists and all the hands to staff the deck.
• Production is where the bulk of the time and money is spent and involves the actual filming on location in costume as well as recording of the auditory experience (music, sound effects etc.).
• Post-production is where all the elements from production are combined into the final product to be presented to the consumer.
• Distribution fascilitates the last-mile provisioning to the consumer.

These five phases are distinct in time. Development must be completed prior to the commencement of pre-production and this is enabled by the producers securing financing for the project. Production is usually realised according to a strict schedule (albeit with some wiggle room contingent on prevailing circumstances such as weather, mishaps on set etc.) and may suffer considerable setbacks due to unavoidable delays. For example, in 2020, filming of Mission Impossible 7 had to be suspended multiple times due to the Covid-19 pandemic leading to significant postponement of its release stretching into 2022. Post-production can only begin once all elements are present to be assembled though the post-production team may use stubs for some of the elements which may require revision. It goes without saying, that only the completed film may be distributed.

In much the same way, I am persuaded that software production should follow a similar process. Despite the popularity of agile methodologies, I think that these disciplines and practices may apply only to one or a few of the stages. In parallel, I propose the following production process analogous to film production.

• Development would consist of assembling the set of user stories that will be addressed by the proposed software. These should be evidence-based in that there should be a solid basis for proceeding. Only eligible projects may be green lit to proceed to...
• Pre-production, in which design elements would be developed as well as extensive UX design ellaborated. The main elements that come to mind would be: mockups, prototypes to be used for UX design, infrastructure designs, information schemas, the design language consisting of typography, colour schemes, imagery and/or illustrations, software frameworks to be used, essential libraries including outline any greenfield libraries to be developed alongside. The deliverable of the this stage would be a design document encompassing all the elements upon which the final design would be built.
• Production is where most of the agile methodologies would fit in. Whether the product owner would like to use Scrum, Kanban, Extreme Programming or what have you, a team consisting of all the required experts will need to assembled on schedule to realise the design. If done through Scrum then each sprint would offer the client the opportunity to feedback to the scrum team without interrupting their production flow. Sprints should not be run indefinitely but should have a start and end, by which time the scrum team is expected to deliver on the product. Otherwise the team runs into sprint fatigue. All versions at this point should be considered 'development releases'.
• Post-production would then involve dotting all the i's and crossing all the t's of the final product so as to address hanging issues. However, by this point the the product is 'feature complete' and ready for deployment. A road test with a handful of real users in a production environment will demonstrate whether the product meets expectations. This is the domain of the alpha, beta and release candidates.
• Finally, distribution would be the production configuration, whereby the system would be configured for production use. During this last phase, training would be provided for all end-users whether they will be consumptive users (click, click, click) or production users (developers, users of APIs). It goes without saying that there should be a support period during which a skeleton development team (not the full production team) will fix bugs and make minor adjustments.

Under such a model, it will be practical for 'software studios' to have only a handful of staff employed permanently, whose main responsibilities will concentrate on the first two phases. The production and post-production phases will depend on contract staff, typically paid per project. However, the studio will need permanent staff with considerable industry experience to serve as architects and project leads who set the tone of every project. Such employees will play a vital role by utilising the studios accruing pool of libraries (visual, code, patterns, infrastructure) which accelerate the production process.

Incidentally, such an approach hints back at the oft maligned waterfall method. I think that waterfall fit well in a world that aimed to be rational prior to the exponential growth in complexity of software systems. While agile has been praised for addressing some of the pitfalls, it should be clear from the plethora of agile methodologies that no clear solution has been arrived at that is able to scale independent of the particular practitioners involved i.e. multiple practitioners given the same project may all succeed to realise the project though their contributions are unlikey to be interchangeable. To me this is a sign that there is still considerable room for growth in our understanding of how to reliably build large scale software systems in a practionar-agnostic way. My proposal above shows that what is needed is a hybrid model which addresses both the need for a rational approach (the pre-production stage is basically waterfall) and an adaptive approach (the production stage lends itself to agile) under the condition that the production comes to a close with a working system.