This is a brief description of what seem to be the abstract essentials of software engineering design. First, what is general to all engineering design, including software. Then, what is specific to software engineering design. It seeks some answers to the question: “What are you doing when you are doing engineering – when you are designing an engineering solution to a problem?”.
It is mainly distilled and extrapolated from the broadly mechanical-engineering perspective of ‘Engineering Design: A Systematic Approach’ by Pahl and Beitz. The basic, and perhaps best, way to understand software's special qualities is to relate it to established knowledge.
The scope is restricted to a minimal core of engineering. Requirements formation and management are excluded, both assumed taken care of elsewhere. These are important but not truly distinctive to engineering proper. The motto or axiom here is an intensional, mechanistic, activity-based definition of engineering: that it is ‘assembling by understanding and experiment’.
(3000 words)
There are two overall goals in engineering design. The first is to maximise fulfilment of the requirements, including all product needs. The second is to minimise the effort of design work. This includes all production work, both for immediate needs, and in anticipation of future needs. These goals could be summarised as: ‘getting things right, and getting things done’.
Everything done in design must serve at least one of these. They set the general measure to which all activities can be referred.
Background:
The essential purpose of engineering design is: to find a way to join what is wanted with what is possible – what the client wants to do with what the material/medium can do.
Vitruvius mentions this millienia ago:
In architecture [particularly engineering in today's sense], as in other arts, two considerations must be constantly kept in view; namely, the intention, and the matter used to express that intention: but the intention is founded on a conviction that the matter wrought will fully suit the purpose;
– ‘De Architectura’; Vitruvius; 25 BCE. – Book 1, Chapter 1, section 3.
Each of these two is a field of knowledge itself. Requirements can be broken down and re-arranged, and are as important as the material, though have a different kind of complexity. Material knowledge is the core of engineering design – it is determinate, and understood logically. Requirements knowledge is indeterminate, and more a matter of sense. Depending on how separated the working style, the requirements side may be handled by product design or architecture – i.e. the sub-discipline focused on outer, usable features.
This essential purpose of uniting these two sides sets an essential broad process – or group of activities or aspects – for design:
Background:
The principle of engineering design is to realise, or reify, abstract structures. Rational ideas are expressed, formed, or codified in the material/medium – not merely layed on top of, but aligned and interlocked with the native structure or properties of the material.
The most important aspect is separating things. This simplifies the task, and even makes it possible at all. Such dividing/discretisation/decomposition is essential to determinate activity – to think is pretty much to make something distinct, separate. Crucial to engineering design is working on just one thing at a time: isolation. Doing more than one thing should be avoided. Separating isolates: parts of the solution do not interfere, and can be developed more easily, and effort can be parallelised.
Really, design is neither separating nor joining alone, but making a structure or system that does both, clearly and sharply. Design creates a system of ‘joints’. But in concious effort, the first step is to separate.
Background:
Engineering design has a general, basic process, that can be seen as following from the above aspects.
The overall goals lead to the first guide. Requirements must be fulfilled, but effort minimised, so: the design must create only what is particular to the project, and reuse all that is general and shared with other projects. (‘Reuse the general, create the particular.’) ‘Reuse’ here means ‘import’-reuse: finding parts already made that fit the current needs – concepts, standards, rules, components, anything. With that, as little as possible effort is spent on the required new, ad hoc, particular design.
And the essential purpose of engineering design is effectively: assemble available parts to make something new. So the difference between ‘reuse’ and ‘create’ in the above first guide is only one of level: ‘reuse’ means fill a gap in the design with a ready-made part; ‘create’ means fill a gap by designing a part – and that is really assembling other, more primitive, ready-made parts.
And the principle of reifying (separating and joining), elaborated and extended over time, gives the basic process: a recursive analysis into reused, known parts:
Where the top is the single whole system, and the bottom is the finest practical properties of the material. However, the direction is not really dictated: it can be as much a synthesis from, as an analysis into, known parts.
The process has two main dynamics entwined: hierarchical progression – composition of sub-parts, and experimental progression – iteration of variation and selection.
Both dynamics are essential. Experimental iteration is necessary because if information and understanding were sufficient to avoid iteration, the process could be automated and so would not be design. Design handles only insufficient information and understanding, and so requires iteration. And hierarchical composition is necessary too. Analysis/synthesis is required, as is iteration, and an iterated analysis/synthesis makes a hierarchy.
The experimental progression is not pure iteration, it is variant. Iterations do not repeat exactly the same activities each time. The purpose is to get new information so some things can then be done better. Also, being combined with decomposition splits the iteration. Some parts are OK, some not, and only a portion need iteration at all. Iteration is not simply about following process steps, but attending to the parts of the product.
The hierarchical progression is not only analysis, but synthesis too – it is bi-directional. Composition does not wholly descend from broad to detailed. The overall purpose of joining intention to material means there is always some detailed to broad direction too. Design can start with reusable components at various levels, and progress is arranged around them; and since there is inherent dividing, different parts can develop at different rates too.
The idea of ‘process’ is really a simplification. Engineering design is not a sequence of steps. It is a system of operations/transformations, that reflects the multi-dimensional properties and relations of the parts the design works on. A design is put together not by following steps, but by following the parts and material.
Design and management have a particular relationship. Decomposition and iteration usefully divide-up the design process, allowing assessment-points and decision-points. So management can see progress, and decide to continue, halt, or change project aims. However, the structure of decomposition is technically led – so management must follow the design process, not dictate it.
The optimum of engineering design emerges from the description of goal, purpose, and principle. The goal of proportionally maximising the reuse of ready-made parts and minimising ad hoc creation, along with the purpose of uniting intent and material, implies: finding the simplest way of creating novelty that maximally fits requirements – that is the characteristic of elegant, great design. This seems to leave out the principle of realising abstracts: good design may appear by sheer inspiration. But if there is any intention – as there must be in anything properly called design – that means some abstraction, it is just not systematically subdivided.
Engineering design can be defined as: ‘Assembling things by understanding their properties and using experiment where needed.’. How does this ‘understanding and experiment’ happen? Understanding sets an organising abstraction of the idea (this is the reifying principle, above), and experiment sets the particular parameter values – or, understanding sketches the idea, experiment fixes the details.
(The goals, purpose, and principle are all different aspects of a single complex activity.)
Background:
The software design process has two products: algorithm and representation. These are the obvious basic features that distinguish it from other engineerings.
Algorithm is primary and truly essential. Software engineering is founded on fulfilling user requirements by processing, transferring, and storing data; and doing so economically (within acceptable time and space), accurately, reliably, securely (Vitruvius might have summarised all as Fidelitas). These are the fundamental design features. And algorithmics is determinate: its design work rests on knowledge that is scientific in character, centered on theory and complexity of computation. This is equivalent to mechanics in physical engineering, and is similarly the core of software engineering.
Representation is harder to rank, though quite clear itself. It is the visible form of the algorithmic structure, and can be chosen and changed separately from it. Representation exists to make the software manipulable for human designers: to be comprehensible and modifiable. Other engineerings have descriptions, diagrams, etc., but representation is distinctive for software in its emphasis. Representation dominates software design because: first, its importance in software's especially iterative process; second, algorithm is often underconstrained by modern computational power. These create ambivalence. Representation is secondary and inessential, because it is only to assist the engineering process, rather than constitute the usable product. Yet circumstances mean it is the most important aspect.
Software design has some distinctive extensions to general engineering design. These emerge from its basic properties being expressed through the elements of general design.
There are two prominent aspects. Software has no final material form or product: having no end-point makes the design process continual – with very frequent adaptation. And software is easily copyable and communicable: having no ‘location’ makes the design process global – with very wide re-use of components. Both these are evident in actual practice.
They are both concerns of the second product, representation. Other engineerings may have more minimal forms of these, but software has them saliently.
When there is no end-point, the shape of the process changes. What would be done at the end must be done over the whole course: moving from initial to fully realised states is done repeatedly. Software design therefore naturally has a process of many fine-grained iterations – it becomes a stream of versions, of large and small degree.
This is a useful attribute. The ideal of the general purpose of design (described previously) would include being able to respond instantly to changes in requirements (and in components to build with). This becomes nearly attainable with software.
(There is both one course of development, and also many separate ones. Each iteration is a sub-part of an overall development process that contains and coordinates work and components for all.)
To handle this continualness, software design is prompted to add an extra concern or dimension. Not only must it define the form, but the way the form can change – it must also design a way of growth and development for parts. There are two extremes of change. It can be ordered and connected, in which case it can be planned for and pre-set. Or it can be discontinuous and substantial, and so cannot be predicted or prepared for, and must be responded to. Both must be handled by the design process/activities. But currently there seems less well-developed (or just less known and practiced) theory underneath this, compared to the other design product, algorithm.
The ordered, connected kind of development can be handled with fixed programming forms. An abstraction can be set up-front, and its parameters, instances or whatever varies, are arranged over the course of development. Up-front design is only possible if the bounds of the abstraction can be defined ahead with certainty. In a sense, this is not change at all, just spreading fixed structures over time.
At the other extreme, real change in requirements can only be handled with real change in software. Refactoring is a means of handling real change. Refactoring is not so much a technique, as the fundamental form of responding to iterative change of the substantial kind. Up-front design is preparatory/anticipatory, refactoring is responsive. Up-front design arranges things to suit the future, refactoring arranges the current state into a good form. (Refactoring's use of well-defined, logical transforms is not essential to its identity, it is just good use of the medium's natural properties.) One might be done after the other, but they are essentially different activities.
Globalness is similar to Continualness.
When there is no location, the reach of the process changes. Whatever would be at a site is spread out over the whole field or industry: production of parts is done dispersedly. Software design therefore naturally has a process of wide distribution and communication – it becomes a single network of interrelated parts and components, of close or far degree.
This is a useful attribute. The ideal of the general goals of design (described previously) would include being able to share work across the whole field. This becomes nearly attainable with software.
(There are many separate programs, but there is also only one. Each project is a sub-part of a global software design process that produces and communicates components for all.)
Increasing software development ‘productivity’ is truly about reuse. It is about focusing only on what is unique to the particular requirements, and doing nothing else. This is how programming in high-level languages is more productive than programming in assembly. It produces no difference in distinct software volume – it just factors the work so the low-level sub-tasks are done once and reused, instead of being done repeatedly.
Software development has a significant large-scale economic aspect. Reuse works by exploiting copyability – copyability is like a free, limitless natural resource. Optimal development is about best using this. The ideal, extreme would be: no work is ever done more than once across the whole world. That would give the greatest gains in productivity.
The basic means to realise this are the same for globalness as for continualness: up-front design and refactoring. They would just be working across projects instead of time. One is appropriate for the common, predictable, and fixed, and the other for real, substantial, variation and change.
Evolution has a special relationship to design, and even more so to software design. Both evolution and design create viable, functional structures, but by different means; and with different strengths and weaknesses. And the particular needs of evolution are especially well suited to the capabilities of software.
Design depends on clear understanding of requirements. If you know what you want, design is appropriate, and better than evolution. But this condition is never entirely true. Requirements are necessarily somewhat vague, because they can only accurately describe the known: how things were done in the past. They cannot determinately predict what might be good in the future. But design always creates some amount of newness, so there is always some room for evolution.
Evolution has no dependence on understanding, it will work under very limited conditions. The problem with evolution is it cannot very well be controlled. But where it is not clear what is wanted, control is not useful, and there evolution has a role.
Evolution is a default, fall-back for design – a very robust and complete one.
How does evolution fit with design? Evolution has a fundamental similarity to design: procedurally, both have variation and selection. Design includes these in a larger process, so that is the obvious way to merge-in evolution. Where design is dominated by experimentation, that part can be replaced by evolution.
To incorporate evolution requires exposing the ‘product’ to the ‘user’ or problem domain. This means: the design process must be expanded beyond simple product development bounds – so it works across releases, globally and continually. And, evolution has random variation and statistical selection (probably its main strengths). These characteristics suggest: not choosing the variation, but making it random; and not having single versions, but releasing multiple variants. These would both allow evolution to have full effect.
As more is understood of natural evolution's mechanisms, other techniques can be adapted to design. But all are based on iteration, which software is uniquely able to provide.
The technicalities, the special properties, of software shape everything else related to it. And this frame of influence orients the different layers of the subject.
Computer science – mainly computational theory, and algorithmic complexity – underlies software engineering design, and in turn software engineering design underlies: programming languages, components/libraries, development processes, development tools.
These exist to serve the purpose of engineering design, so how they are considered, implemented, and performed follows from and depends on understanding it. It could be said that software engineering design is the most abstract general conception of all those parts of software development. To improve those dependent subjects, we must primarily ask how they are driven by software engineering design.