October 22, 1998 - Jack Harich

Purpose

The purpose of this diagram is to design the exact high level change flow needed for effective continuous change. Since software should be architecture driven, we start with modeling the system architecture first. Then we work backward to the process needed to support changes to the system using that architecture. Hence the process is architecture driven, which is not true for most popular processes.

This flow chart is a further evolution of previous ones. The earlier document on Continuous Change Architecture had a small diagram that has become the system diagram we see above. The document covering Configurability and Change Point Theory laid the groundwork for Controlled Injections and the singular role of the Param Editor and Parameters.

Note the three Change Points - Injections of Parameters, Components or Infrastructure. The process feeds into the system architecture at only these well defined points.

The diagram tries to have icon importance reflected in icon size. The largest is the system we are producing. The second largest are Controlled Injection Review and System Engine. Think of the Reviews as very carefully injecting change into what the Engine is running, with the precision and expertise of a brain surgeon. Here's an email reply from me on this subject which should give you a real feel for our goal here:

Christian Cryder wrote: (10/7/98 on the JSL list)

> As technology grows, complexity increases at a rate of change which
> accelerates exponentially. Yikes!

Yes and no. If you don't approach system size and diversity with an eye
to keeping it under comprehension, apparent complexity does grow
exponentially. But properly applied design can tame this well. We are
just not seeing this done often.

For example suppose all your systems had a standard foundational
architecture. Suppose they all had the identical inter-system
collaboration mechanism. Suppose all substance was either infrastructure
or components. Suppose injecting a new type of component or
infrastructure into the enterprise system was a hotly contested event,
subject to severe restrictions. You would see far more robust,
comprehendible and easy to use system, from the perspective of all user
types.

System Architecture

This is partitioned into parameters, components, infrastructure and the system engine. We have split true components into two types, low and high level, components and infrastructure. This allows illustrating what changes from system to system as an inverted pyramid, and allows treating infrastructure as a thing in itself. The parameters, components and infrastructure follow layered architecture, ie lower layers cannot call upward except via low coupling techniques like events. They form an inverted pyramid because there are far more parameters than components, and far more components than infrastructure. The lower in the pyramid you go, the less change you see and the more important stability and quality is.

The system engine is the system microkernel. It seldom changes, is very small, and is the foundation which runs the other system partitions. The exact system engine I use is the one in the Bean Assembler, since it supports parameter driven components, hierarchial system organization and composition, and is domain neutral.

Controlled Injection Review

This is used to keep the system design and quality at a very high level of excellence, while allowing the easiest and fastest process possible. The system Architects lead the reviews, accompanied by the Assemblers and Analysts, and occasional Users. These reviews are much like inspections, whose goal is to prevent bad requirements, design or code from going any further. They also serve to educate others about the architectural vision and system design details, as well as good practice. In practice the main benefit of reviews is educational. This becomes obvious when one sees a team working at a higher and higher level as time goes by, during their reviews.

When a system is young, most Change Proposals will be on Infrastructure and Components. But as it matures, more and more will be on Parameters. Eventually most will be Parameters. If the Param Editor(s) are good enough, the edits will be look and choose, self validating and very easy to do. Thus most will be done by Users and Analysts, and system growth will be easy, fast, cheap and highly effective.

If the Param Editor is good enough, it will validate all change as it occurs, leading to:

Change Requests & Proposals

Change Requests are for change too complex to be handled simply by parameters, or involving changes to Components or Infrastrucrure.

We want a flexible document vehicle here, that can handle all types of system change that needs careful planning for correct implementaiton. This includes Bug Fixes, New Features, New Products, Project Phases, Change Sets, etc. The important thing is to capture all these in a uniform manner for uniform processing. We want a polymorphic document.

Some change types will have already undergone work before they become a Change Request. For example a New Product will already have a vision, management support and some analysis. A Bug Fix probably started out as a Bug Report, was confirmed to exist, and was resolved by escalating it to a Change Request.

Depending on the change type, a lot of magic can happen between the Change Request and Change Proposal steps. For example a New Version will require a project plan, change sets per milestone, models per change, prototypes, etc, such as the usual life cycle method artifacts. But please note that even a life cycle method lives inside of the larger Change Process & System Architecture. This allows a lighter weight implementation of a method's steps, because the method now has pre-existing infrastructure.

A Change Proposal is a document completely ready for review and injection, ie implementation. Like all documents, it may never even exist on paper but eke its way through electronic workflow.

The reason we are reviewing Change Proposals and not Code is the latter is far more expensive, and the former is far more educational. I commonly say "The battle is usually won or lost at the design level, not the code level," to get those stuck at a lower level into thinking on a hgher plane. This process enforces that statement. Code will be reviewed, but less often and so is not in this model.

We may rename Change Proposal to Injection Proposal. (Comments?) They contain the original Change Request, a unit and integration test plan, the use case, features and risk resolutions they are meeting, various artifacts such as models, and crystal clarity on what will be changed and how.

Param Editor

As the model shows, this is the single most important tool. It's purpose is to easily and reliably edit a set of parameters that drive a reusable piece of software. Pssst, here's a secret - I didn't know this tool was the bottleneck until I did this model. Currently we hand edit parameters as text. This has loads of awkwardness, especially as file size grows or novel functionality is desired. It is very hard for non-experts to do.

We envision a visual tree of parameter nodes which can be read and navigated with ease. Adding new nodes is done by selecting from the node types allowed for that node. Editing node propeties is done by something similar to a property sheet. Nodes can be copied, pasted, deleted and disabled/enabled. The latter allows alternative approaches to be easily compared or saved, or even be dynamic.

It's possible we will convert the parameters to XML. This would open the door to using resusable software and concepts here, and all sorts of future benefits. But XML is a far more complex beast that our parameter format, and has different goals. Its prime, but unadmitted goal is a better HTML. It's secondary goal is a universal structured data format in ASCII text, not binary. This is fine. A problem is XML is complex and lacks features our format has, such as the ability to convert from a object back to the exact original parameter format with ease.

An interesting advantage of continuing to not use XML is we will not be restricted by the path XML's evolution takes.

A mature system of well designed and battle tested components and infrastructure will have nearly all work done via parameter edits. We predict Param Editor work time will be over 95% of total system development for truly mature systems once we have a really good Param Editor. Hmmmm, better get to work right now! :-)

People Roles

User - Far and away the most important role is the User. The process is User centric, so we carefully support their goals, which are to easily use and change the system to meet their needs. If there are needs the user cannot express themselves, they get assistance from an Analyst. The two of them can build most systems with no code whatsoever, once decent parameter driven components are in place. This is "programming" of the future, and we should all prepare for it.

Analyst - The second most important role is the Analyst. They do Param Edits that a user cannot, or train users to do this themselves. They create Change Requests that are too complex for a user to create themselves. More importantly, they spot new and better ways to do things and create Change Requests to get it done, such as more appropriate components or infrastructure. They are the eyes and ears of the rest of the system. And perhaps most importantly, they participate in Controlled Injection Reviews. This allows their input and educates them to system abilities, potential, needs, etc.

Assembler - The third most important role is the Assembler. These wizards assembler systems from thin air by creating parameters with completely define a system, using no code. Their finger tips twiddle pure declarative knowledge, aka parameters, much as a spider weaves a web of by extruding gossamer filaments. Since the components collaborate via Message events, and there are lots of these, our merry wizards are composing a fine, delicate web of Messages and Components, driven by Parameters, all masterminded by an invisible System Engine. This role takes deep domain and reusables knowledge. Just as ancient wizards connected magic and reality, past and future, fact and fantasy, so the Assemblers connect domains and reusables, users and systems.

Architect - The least important role is the Architect. All they do is sit idly by and make a few deft flourishes to ensure the process and architecture proceeds as planned, and is adjusted as needed, such as by whipping out a few new components when needed or proactively, fabricating some fabulous new infrastructure once how to build the next generation becomes obvious, or tweaking the System Engine. They also steer system and personal growth to best meet the long term needs of the customer at large.

A comment on the first draft of this was "The Architect sure doesn't do much." They were not reading between the lines, and seeing the silly parody of relative importance. The problem I'm seeing is most coding cowboys want to be the Architect. As Eric Kohlenberg said, "Everybody wants to be the Architect." So I'm downplaying that perception, as well as designing a process and architecture that is not heavily dependent on lots of architects, since good ones are scarce. This process and architecture is not a "build every system from scratch every time" approach, but more a "build every system from reusables, and follow the same foundational architecture every time" style, which inherently takes less heroic effort and genius. Actually the above roles are all equally important, just as every person in the universe is equally important and interesting, once you get to know them.

Schedule Management

All this must smoothly tie into work schedules and a project plan. Since the entire process is either Change Request (CR) flow or direct injections with the Param Editor, all we need to schedule is CRs. Working backward from that premise, for a project schedule we map milestones to groups of CRs (Change Sets) or Controlled Injection Reviews, and map tasks to CRs or CPs, and their preparation, progress and completion.

Lo and behold, we now have A Beautiful Thing. One large goal of any systematic approach is seamless transitions, plus clarity. We have that for Schedule to Process to Change Points to System.

More will be added to this once we've stumbled a bit. Stay tuned. Comments are welcome. Ya learn by trying....