How I Learned to Stop Worrying and Love Engineering Documentation
All tech companies rely on organizational memory. At Nebo, we know this to be true. As Linda Argote, author of Organizational Learning (Berlin: Springer, 2005) explains, this type of memory depends upon "repositories of knowledge" comprised of "individuals, including managers, technical support staff, and direct production workers; the organization's technology, including its layout, hardware, and software; the organization's structure, routines, and methods of coordination; and the organization's culture" (p. 74). Translation: organizational memory gives well, everyone, an avenue to access their history.
Maintaining these memory repositories is essential for businesses to retrieve key project details—vendors, client contacts, SOWs, even years later. Complex engineering projects in particular require a companies’ organizational memory to be accurate, adequately descriptive and up-to-date. For a team of engineers, organizational memory takes the form of documentation and is critical for project’s success.
Organizational Memory is Not Optional
In the hustle-and-bustle of looming deadlines and negotiating new business, we often see that recording information essential to project maintenance gets sidelined. What you end up with is a sole individual holding all of the critical knowledge concerning a product. This information is lost if they leave the organization or take a vacation at the wrong time. Another issue many organizations face is tracking down the location of records. Rather than a central repository, oftentimes archives are stored in a variety of locations, siloed within the semi-private filing systems of multiple departments. For organizations interested in maintaining robust ongoing client relationships, adopting documentation practices for recording crucial data and streamlining access must take priority.
As a human centered agency, organizational memory serves a particularly important function at Nebo. Fostering long-lasting partnerships with clients means never assuming that the end of a project is the end of a relationship we have built. If we’re doing our job well, the completion of one campaign should transition into the beginning of another. So, recording and maintaining detailed documentation is a unique and important part of the way we conduct all our digital marketing projects. We take organizational memory into consideration from the start, especially in the case of digital apps developed by our engineering team.
Better Than Agile
Software documentation is the most concrete and important form of organizational memory that engineers are responsible for consuming and creating. Yet, the lights of Agile development have often downplayed the importance of this form of organizational memory to the engineering project cycle. According to Scott W. Ambler, a leader in the Agile community, documentation should always be minimal, if it’s present at all. Ambler argues: “You should only create a document if it fulfills a clear, important and immediate goal of your overall project efforts.”
Recently, there has been a movement among experts in Agile and Agile adjacent methodologies like XP, Scrum, and Kanban to expand and revalue software documentation. Tom Thompson, Senior Technical Writer at Schneider-Electric, argues that documentation is not only essential, it should happen in parallel with development. Thompson makes it clear that the process of writing documentation can actually save time by eliminating work. “When the engineers and writers collaborate in an iterative process, they can learn from each other and make the whole process more efficient…[Moreover,] Getting technical writers involved early is a great way to get feedback on your design. If your documentation team can't figure out a feature, your customers probably won't either.” It’s fair to say that documentation as a tool of organizational memory possesses wide-ranging benefits.
A New Model for Documentation
This year the engineers at Nebo decided to reevaluate the role that documentation plays in our development process. As a professional software engineer with a background in technical writing and communication, I was excited to take a steering role in assessing and redefining our documentation practices.
After much deliberation and planning, the engineering team approved several changes that we implemented during an all-day documentation writing and revising event (or, “Great Document-Palooza,” as we called it). Although working on documentation can be an admittedly boring task, we made the best of it by gathering together in a single conference room, playing music, enjoying a pizza lunch and joking around as a team. It ended up being a lot of fun.
Our core goals for this initiative centered on optimizing accessibility and accuracy. I began by interviewing individual members of the engineering team to determine the perceived strengths and weaknesses of our current system. From there, I interviewed Damon Borozny, Senior Director of Project Management at Nebo, about his experience using engineering documentation to get a sense of what non-engineers require from these docs. What became immediately clear from both groups was the need for a template with unambiguous and recurrent categories that enabled users to avoid redundancies, improved searchability and simplified the task of writing documentation.
The improved documentation that came out of our pizza-fueled Great Document-Palozza has been a great success because it benefits all of us at Nebo by improving our organizational memory.
Over the course of a month I created a template and revised it several times based on stakeholder feedback. I also mocked up a revised documentation example by updating a sample internal project’s existing documentation. These materials incorporated suggestions prioritized by the team at the same time they employed user experience and technical writing best practices.
Perhaps most importantly, our new template improved searchability by establishing categories in a predictable, logical and consistent manner. Uniform and generic headers ensure that the document’s parts are marked clearly, and they are legible to a broader audience of non-engineers. For instance, instead of using a heading titled “Postgres,” all documentation now includes a category for “Database Management System.” By beginning with a template, where appropriate, the document writer simply adds the type of management system (Postgres, MySQL, etc.) beneath this existing generic heading. This genericizing of our headings extends to categories such as the project’s “CMS” (the engineer can write Drupal, Wordpress, Nebo’s CMS4, etc.), “Hosting” (Rackspace, DigitalOcean, etc.), “Custom Functionality” (Certificates, Templates, etc.) and third-party "Accounts" (Instagram, Gmail, etc.). These categories are arranged roughly in order of interest from general employees (PMs, SEO) to engineers (front-end, back-end), with code and database specific information appearing towards the bottom. Although these changes are relatively minor, they simplify composing, understanding and finding information tremendously.
Another improvement was adding a category to our documentation template for recording a comprehensive list of project managers and engineers that have contributed to a project. Previously, this information could be discovered by reviewing the Statement of Work, task tickets and, in the case of the engineers, code committed on GitLab. Now it all exists in a single, centralized location. We also added a category linking to the SOW, Tactical Plan and CMS User Guide. If our team needs to update a site or bring new people onto a project, everything they require to get up to speed is now in one place.
Although this documentation template is intended for internal use, it has also proven valuable for composing the instruction documentation we provide to our clients. Instead of tracking down and reviewing several sets of notes, our documentation now serves as a one-stop-shop to find all the information necessary for us to give clients. In fact, because the documentation template includes a “Custom Functionality” section, all special instructions are readily available to add to these important manuals.
What We Learned
The process of re-thinking and updating our engineering documentation at Nebo has taught me much about not only the importance, but also the limits of organizational memory. Organizations must take a thoughtful and balanced approach to determining the role, power and structure of memory. We’ve all experienced how insufficient records can lead to more work and, in some cases, significant problems. Let’s be clear though. Recording and storing information about completed projects should not take up a significant amount of time or resources. Once the assignment wraps up, the meeting agendas, minutes, post-mortem reports and manuals detailing obsolete products and design kits have little value and are rarely (if ever) looked at again. Therefore, belaboring how the minutiae of a project is archived poses little to no benefit. The key is to achieve a balance that works well for your organization, and then stick with it. A robust documentation strategy requires planned intent and clearly stated purpose.
Accurate, Accessible and Useful
Updating and improving Nebo’s engineering documentation process convinced our team that this task must be iterative, ongoing and also suited to cultivating long-term and human-centered relationships with clients. Marketing works best when it does not operate on a one-and-done model. It makes sense to treat all of our clients as though we will work with them again soon, either on retainer or as part of a new campaign. Thankfully, current business trends have discredited the “Always Be Closing” mindset, which is focused wholly on securing new contracts, as it does not align with our company ethos of quality service.
Thought leaders now recognize that budgeting time to keep appropriate records (as determined by the team) is not only a good idea from a project management standpoint, it also cultivates positive client relationships. The engineering team is committed to using the template model to create new and update all past project documentation. We also hope to schedule another engineering team Great Document-Palooza in the coming months to ensure that everyone continues to find our records accurate, accessible and useful. We haven’t decided if we’ll share our pizza yet.