The Review Is Not the End of the Problem
In the previous post, I argued that architecture reviews are not merely approval gates. They are attempts to recover decision memory by understanding why certain constraints exist, which futures were preserved or closed, and which invariants the architecture was supposed to protect.
However, there is an additional problem.
Reviews, metrics, dashboards, scorecards, compliance matrices, and risk registers are not direct views of the architecture. They are representations layered on top of other representations. That makes them necessary, but dangerous.
A program can have a complete model, a successful review, a green dashboard, and a populated risk register. Then integration begins, and the architecture fails in ways none of those artifacts revealed.
The problem is not that models, metrics, or reviews are useless. They are essential. The problem is that they can be treated as if they provide direct access to the architecture, when they are usually several representational steps removed from it.
The architecture is not the dashboard.
The Representation Stack
The real architecture of a system is the causally ordered constraint structure that shapes what futures remain admissible. This includes technical structure, interface semantics, verification burden, organizational coupling, manufacturing constraints, regulatory obligations, and governance pathways. No single artifact captures all of that. So we use representations.
A simplified representation stack looks like this:
Architecture → Model → Metric → Dashboard → Decision
Each step is useful. Each step also loses something.
The first step projects the architecture into a form that can be seen, discussed, analyzed, and shared. The next step evaluates or scores the view of the architecture resulting from the projection. The next step compresses the evaluation into a status artifact. The final step turns the status artifact into institutional action.
The farther up the representation stack we move, the more actionable the artifact becomes and consequently the more dangerous it becomes if the artifact’s limits have been stripped away.
First-Order Representations
First-order representations are direct projections of architecture. Examples include:
- SysML models
- Architecture diagrams
- Operational views
- Requirements structures
- Interface control documents
- Design structure matrices
- Functional decompositions
- Verification models
- Trade-space models.
These artifacts are not optional. Complex engineering would be impossible without them. Standards such as ISO/IEC/IEEE 42010 explicitly frame architecture description in terms of stakeholders, concerns, views, viewpoints, and models [ISO11]. Systems engineering practice similarly relies on representations to coordinate architecture, requirements, verification, and lifecycle processes [WRF+23].
However, first-order representations are still projections. They abstract. They omit. They freeze a dynamic situation. They impose a viewpoint. They privilege some concerns over others.
A system model may capture interfaces while missing important informal organizational dependencies. A requirements structure may preserve contractual traceability while obscuring operational coupling. A design structure matrix may represent technical dependency while excluding governance or certification propagation. A diagram may be clear precisely because it has compressed away consequential detail.
This is not a defect. It is the condition of representation.
The defect appears when the organization forgets that projection has occurred.
Second-Order Representations
Second-order representations are representations of first-order representations.
They include
- Architecture review findings
- Technical debt scores
- Readiness levels
- Risk registers
- Maturity assessments
- Compliance matrices
- Executive dashboards
- Governance summaries
- Stoplight status charts.
These artifacts usually do not evaluate the architecture directly. They evaluate what has been represented, scored, reviewed, summarized, or claimed about the architecture.
That distinction matters.
A metric derived from a model inherits the model’s omissions and limitations. A dashboard based on the metric inherits both the model’s omissions and limitations and the metric’s valuation assumptions. A governance decision based on the dashboard inherits the entire chain.
Metrics are not direct access to architecture. They are second-order evidence whose validity depends on provenance, calibration, and the fidelity of the representations from which they were derived. In many cases, the metrics themselves are proxies for what actually needs measuring.
This is where Goodhart’s Law and Campbell’s Law become architecturally relevant. When measures become targets, they can lose their value as measures [Goo75, Str97]. Campbell warned that the more an indicator is used for decision-making, the more it can distort the process it is intended to monitor [Cam79]. Muller makes a similar argument about metric fixation in organizations [Mul18]. Porter shows why numbers acquire institutional authority by appearing objective and transportable [Por95].
In architecture, this means a score can become more powerful than the model. A dashboard can become more powerful than the review. An approval can become more powerful than the evidence.
The organization does not need to misunderstand the architecture directly. It only needs to put too much trust in a purified summary of a lossy projection.
Authority Amplification
Second-order representations often acquire more institutional authority than the first-order representations from which they were derived.
This is understandable. Senior leaders cannot inspect every model element, interface definition, verification dependency, or trade study. They need summaries. Governance needs compression. Programs need status. Budgets need categories.
But compression changes what can be seen.
- A green dashboard may mean the architecture is healthy. It may also mean the organization has successfully compressed uncertainty out of view.
- A compliance matrix may show that every requirement has a verification method. It may not show whether the verification strategy preserves the architectural invariants that matter.
- A technical debt score may show improvement. It may not show that debt has migrated from software implementation into verification burden, certification risk, or operational fragility.
- A risk register may contain known risks. It may not contain the risk created by the representation itself.
This is second-order fidelity loss which I define as the additional loss, distortion, or false confidence introduced when representations of architectural representations are treated as architectural evidence without tracing their dependence on the original projection.
Several subtypes of this fidelity loss are common
- Inherited projection loss: the second-order artifact inherits omissions from the first-order model
- Compression loss: complex structure is collapsed into a score, status, or category
- Valuation mismatch: the artifact measures the wrong architectural concern
- Calibration drift: the metric or review rubric no longer corresponds to actual system behavior.
- Authority amplification: the summary becomes more trusted than its evidence
- Limitation masking: uncertainty and things that cannot be established disappear from the artifact
- Optimization capture: teams optimize the score rather than the architecture
- Review theater: the representation is shaped to pass the review rather than reveal the architecture
Classification systems and categories do not merely describe organizational reality; they shape it [BS99]. Once a status category exists, people begin managing toward the category. Once a maturity score exists, people begin optimizing the score. Once a review criterion exists, presentations are shaped to satisfy it.
The representation becomes part of the architecture’s governance environment.
Representations Should Carry Their Limitations Forward
The answer is not to abandon metrics, dashboards, reviews, or scorecards. That would be impossible and unwise. Large systems require mediated visibility.
The answer is to make higher-order representations carry their limitations. A responsible architectural representation should preserve
- Provenance
- Scope limits
- Uncertainty
- Abstraction losses
- Omitted couplings
- Stale assumptions
- Calibration limits
- Decisions the architecture supports
- Decisions the architecture cannot support
- Things the architecture does not or cannot establish
This matters especially for reviews. A responsible architecture review should not only say what it found. It should say what its findings do not establish.
For example
This review supports the current interface decomposition under the modeled operational scenarios. It does not establish verification sufficiency, manufacturing feasibility, certification evidence continuity, cybersecurity resilience, or lifecycle cost stability.
That is a more architecturally honest statement than a simple approval. It preserves the boundary of the evidence. It prevents a second-order artifact from expanding beyond its valid use. It keeps institutional action coupled to the real architecture rather than to a purified status artifact.
What This Means for Metrics
Architectural metrics should be treated as recovered evidence, not primitive measurements. A metric measuring the cost of change, for example, may be derived from a model of coupling. But that metric is only as good as the representation chain beneath it
latent coupling → model entries → propagation calculation → change-cost score → management decision
At each step, something can be lost or distorted.
So a metric should not travel alone. It should carry a recovery protocol to include answering
- What first-order representation is it derived from?
- What architectural feature is it supposed to measure?
- What does it omit?
- What invariant does it help preserve?
- What limitations does it carry?
- What decision is it valid for?
- What decision is it invalid for?
- What cannot be established by this metric?
This does not weaken the metric. In fact, it strengthens it. A metric with provenance and limitations is more trustworthy than a metric pretending to be context-free.
The Architect as Fidelity Steward
Architecture work is often described as model creation, design coordination, interface management, or technical decision-making. Those are all part of the work. But they are not the whole work.
The architect’s deeper responsibility is to preserve fidelity across the representational hierarchy.
Architecture becomes a model. The model becomes a metric. The metric becomes a dashboard. The dashboard becomes a governance decision. At every transition, the architect or systems engineer must ask
- What was lost?
- What was compressed?
- What was assumed?
- What limitations should have traveled upward but did not?
The dashboard is not the architecture. The review is not the architecture. The metric is not the architecture. They are instruments for seeing architecture. Like all good instruments, they must be calibrated, bounded, and understood before they are trusted.
References
[BS99] Geoffrey C. Bowker and Susan Leigh Star. Sorting Things Out: Classification and Its Consequences. MIT Press, Cambridge, MA, 1999.
[Cam79] Donald T. Campbell. Assessing the impact of planned social change. Technical report, The Public Affairs Center, Dartmouth College, 1979.
[Goo75] Charles A. E. Goodhart. Problems of monetary management: The U.K. experience. Papers in Monetary Economics, (1), 1975.
[ISO11] ISO/IEC/IEEE 42010:2011 Systems and Software Engineering – Architecture Description, 2011.
[Mul18] Jerry Z. Muller. The Tyranny of Metrics. Princeton University Press, Princeton, 2018.
[Por95] Theodore M. Porter. Trust in Numbers: The Pursuit of Objectivity in Science and Public Life. Princeton University Press, Princeton, 1995.
[Str97] Marilyn Strathern. “Improving Ratings”: Audit in the British University system. European Review, 5(3):305–321, 1997.
[WRF+23] David D. Walden, Garry J. Roedler, Kevin J. Forsberg, R. Douglas Hamelin, and Thomas M. Shortell, editors. INCOSE Systems Engineering Handbook: A Guide for System Life Cycle Processes and Activities. Wiley, 5 edition, 2023.
Leave a Reply