Managing Creativity and Complexity

Most people who enter the IT profession want to be “developers”. They want to create new and exciting capabilities and not be stuck supporting existing systems. Sounds exciting but managing the creative process is a manager’s worst nightmare. How does a manager address the following questions?

1. What is the quality of the results from the creative process?
2. How long will it take?
3. How does anyone know when they are done being creative?
4. What skills are required to be creative and who has them?
5. When does the creative process occur within the life-cycle?
6. How do we ensure consistency of application behavior when each individual developer is making their own creative decisions?

Failure to manage creativity injects a substantial amount of uncertainty and risk into our projects. Today’s development tools allow developers to design and develop at the same time. They have thousands of options to choose from including database design, colors, graphics, object types, etc. that are arranged into a continuously mutating prototype. By the time they are forced to complete the assignment, the components have already been modified hundreds of times resulting in code that is highly complex, heavily modified, inadequately tested, late and over budget.

Does this sound familiar? What impact does this approach have on your projects? Imagine the chaos that would result if we constructed buildings or cars using this approach. Would you want to drive one of these cars?

What can be done to manage creativity and complexity to solve this problem? Take a lesson from the construction industry. Architects complete the creative rendering based on general requirements from the customer. They do not ask the customer to design the building nor do they allow the plumber and electrician to design their own solutions. Once the Architect finishes the creative rendering, a draftsman creates detailed blueprints, and everyone else’s job is to build it according to the blueprint. How do we implement this approach in IT projects?

1. Create position and career path for a design architect who can assume responsibility for the creative process similar to a building architect or a design engineer in the automotive industry.
2. Impose a deadline on the creative process.
3. Keep the creative process at a high level. PMI describes “progressive elaboration” as the mechanism for gradually defining lower levels of detail. Evolving detail is not the same as constantly changing the design. Specifications should simply provide lower level details for implementing the initial design.
4. Provide the customer with an opportunity to view the initial “rendering” and concur before detailed design or development.
5. Defer changes till the end unless they will result in significant re-work. Once the initial capability is build, organize the changes and implement them as a follow-on project.
6. Build the application using a modular approach that allows flexibility and allows future changes to be organized by module.

These suggestions are common sense. Why aren’t more organizations taking this approach? Simple: Everyone wants to be a developer even if they lack the design skills to be effective. Senior people are assigned to support production systems while new systems are designed and developed by junior staff. Finally, we are also in a hurry to begin development so we don’t spend the time to develop a workable approach as expressed by one of the oldest jokes in the IT industry: “You start coding while I go figure out what they want”. I heard this more than 20 years ago and it is still true today.

Effective Software Development

Why do projects miss their deadlines and exceed their budgets? Are we poor planners or are we poor managers? One of the basic problems that complicates project estimating, scheduling, and management is the failure to manage the creativity.

The manufacturing and construction industries recognized this problem a long time ago. One hundred years ago, they employed highly skilled craftsmen and each product was a reflection of the design and creativity of the individual. Today, the creative process is the responsibility of engineers and architects. The job of plumbers, electricians, and manufacturing line workers is to build the product according to the design.

Many IT projects allow creativity to be distributed across the life-cycle of a project. Today’s object-oriented development languages provide significant options for artistic and functional creativity. The problem: How do you know when someone will finish being creative? These activities are impossible to estimate. Creative choices by individuals may result in a lack of consistency which confuses users. Creative re-work also adds costs and negatively impacts quality because the frequent changes are not adequately tested.

Common Development Practices

1. IT organizations develop Software in response to a unique set of requirements similar to a fabrication plant. They employ skilled craftsmen who utilize their creative skills to develop software that is customized for the stated requirements. This approach is similar to the use of craftsman by the manufacturing industry prior to the 20th century (with similar results). Each product is unique and quality and efficiency are the responsibility of the individual craftsman and difficult to measure.

2. Functionality is based on the unique requirements as defined by the requester. The quality and efficiency of the design and the final product is the direct result of the creativity and skill level of the individual developers. As a result, quality and cost can be unpredictable.

3. The effort (and the cost) for designing and developing these unique “one off” applications is significant. As a result, new applications are rarely authorized and we continue to operate and maintain older legacy applications. Support costs are also higher because the resulting applications require special knowledge to operate and support because they are all uniquely different in their basic design.

4. Collecting metrics to facilitate estimating and scheduling is difficult. Creative activities are difficult to measure and metrics generation requires additional effort that is typically resisted. Even if metrics are collected, they are useful in defining and measuring quality but the existence of metrics to not ensure quality.

5. Repeatable processes are marginally useful in defining and managing creative fabrication activities. They are very useful in managing repeatable assembly activities.

6. Development tools change frequently. The decision to utilize a new tool is very often made by a developer without serious consideration for the available knowledge base and future support challenges.

7. Most of these new Object-Oriented languages are based on low-level rudimentary languages like C++ and Basic. They facilitate code re-usability at the object level (e.g. drop down boxes) but they hinder re-usability at the functional level. In the past, we could develop an online interactive program that updated a database and clone this program to develop other similar functions. This level of re-usability is much more difficult with Object Oriented languages.

8. These new Object Oriented Languages are much more difficult to learn than their predecessors. Inexperienced developers are more likely to have learned the newer languages but they do not have the design skills required to develop new applications. IT organizations do not train their experienced developers on newer technologies. As a result, the average skill level available for newer technologies is not adequate for developing complex applications.

The Solution: Manage Creativity and Plan Re-Usability

“Object Oriented” languages claim to provide re-usability. Unfortunately, their re-usable components are low-level objects. They provide re-usable functions for creating a drop-down list but do not provide a re-usable function for updating information about people.
IT needs to be more proactive in developing and utilizing re-usable components. We also need to manage and control creative activities that are difficult to estimate and can have significant impacts on the schedule, quality, risk, and cost of a project. The following recommendations apply to each technology area:

1. Create re-usable specifications and functions. Most systems maintain data about people, places, things, and processes. When designing screens and databases to maintain this type of information, choose design options that provide the greatest potential for re-use in future applications. Most applications track people and organizations. Unfortunately, these functions were designed to be unique to each application and are not re-usable across applications. Re-usability should be a high priority design consideration.

2. Use bench resources and trainees to create/enhance re-usable modules. Gradually, an inventory of high quality modules will be available for each technology area that will facilitate future development.

3. Maintain an inventory of re-usable modules. Modules cannot be re-used if developers are not aware of their existence. This inventory should also include re-usable specifications and documentation.

4. Define requirements and functional specifications based on existing re-usable functions. This facilitates the rapid demonstration of the solution and maximizes the possibility for re-using existing functionality. By re-using specs and modules, we will reduce the cost of designing and developing these functions for every application.

Summary
The recommendations I am describing are old solutions. We have discarded these proven solutions because newer Object Oriented technologies have claimed to be re-usable but they only provide re-usability at the individual object level (e.g. drop down box). Re-using functional components requires a proactive effort on the part of IT organizations but the benefits can be significant. If we implement these recommendations, we will be following the example of the manufacturing industry by assembling components to build systems and reducing dependence on the creative skills of craftsmen.