11  The Middlegame: Choosing Representation Under Pressure

Opening principles become useful only when plausible designs are compared against actual consumers. This chapter follows one semantic-document problem through three defensible representations:

1. a property-rich model;
2. typed known nodes with an extension bag;
3. a validated open bag.

None is the inevitable S7 answer. Streaming does not force a particular owner architecture, a large field count does not disqualify native classes, and metamodel size alone does not justify a schema engine. The choice turns on guarantees and operations.

11.1 The position

Suppose a package works with style documents containing named sources, typed layers, paint settings, references from layers to sources, and embedded expressions. It must support:

• constructing known fragments;
• adding and reordering layers while preserving references;
• reading documents from a changing external specification;
• writing them back without casually dropping extensions;
• producing at least one engine projection;
• possibly adding inspection or analysis later.

Before choosing classes, apply the shared opening workflow.

11.1.1 Meanings and consumers

The stable meanings are source identity, layer identity, layer kind, source references, paint declarations, and expression payloads. Consumers include constructors, invariant-preserving transformations, serializer/parser boundaries, the external validator, and perhaps a later analysis.

11.1.2 Representative values and variants

Sketch one source, one known fill layer, one known line layer, one unknown future layer, one literal paint value, and one opaque expression. Draw containment separately from the layer-to-source reference.

11.1.3 Property inventory

For every modeled field, record shape, presence, nullability, emptiness, defaults, and wire omission. An engine default must not automatically become an R constructor default: omission may carry version-sensitive meaning.

11.1.4 Rule placement

• scalar shape and bounded literal rules belong to properties;
• relationships among a layer’s known fields belong to that layer;
• unique ids and resolvable source references belong to the root;
• expression validity and full conformance may remain with the engine;
• version compatibility belongs at parsing or compilation boundaries.

11.1.5 Operations and laws

The package needs add_layer(), reorder, parse, and serialize. It must name its round-trip relation and unknown-content policy. If it later interprets expressions or supports several engines, those consumers may change the representation choice.

All three models can now be evaluated fairly.

11.2 Candidate A: property-rich native model

At the first design review, Candidate A is the natural proposal. Users will construct layers in R, the package promises invariant-preserving edits, and future interface or documentation generation could consume declared property metadata. A native model appears to put every known fact where S7 can reflect and validate it.

Known layer kinds are classes, and meaningful paint entries are S7 properties with local validators. Repeated scalar, enum, color, or expression contracts can be shared as property definitions or generated from an owned specification.

11.2.1 What it buys

• constructor signatures and reflection expose known fields directly;
• local edits can re-run precise property and class validators;
• documentation and interfaces can project declared metadata;
• differences among layer kinds remain available for dispatch;
• typos in known property names fail at the native boundary.

11.2.2 What it costs

• specification changes require regenerating or editing declarations;
• presence and omission semantics must survive defaults and serialization;
• generated code needs an explicit authority and review strategy;
• unknown future fields still need a policy;
• metadata may duplicate an external schema unless one source is authoritative and agreement is tested.

11.2.3 When it wins

Choose this model when users construct and edit known fields heavily, native introspection matters, several projections consume property metadata, or layer-specific operations need distinctions. The number of fields is not itself an argument against it. Generation from an owned semantic specification can make a large native surface maintainable.

11.3 Candidate B: typed known nodes plus extension bag

The next review introduces real documents from newer engine versions. Known layer kinds still drive add_layer(), reordering, and eventual analysis, but unrecognized paint keys and future layer fields must survive a read-modify- write cycle. Candidate B is plausible because it keeps nominal meaning at the operation boundary while declining to pretend that this release understands every foreign field.

Major concepts and known variants are classes, while unknown or intentionally extensible content is stored separately in a disciplined bag. Known paint entries might be properties or a validated known-key map; foreign entries remain inert.

11.3.1 What it buys

• nominal identity and dispatch remain available where operations use them;
• foreign keys can be preserved without claiming to understand them;
• the native surface can focus on stable, high-value meanings;
• version churn is localized to parsers, key tables, and merge rules.

11.3.2 What it costs

• authority is split between typed and extension state;
• collisions need a deterministic rule: typed value wins, extension wins, or rejection;
• transformations and serializers must preserve extensions deliberately;
• unknown content cannot participate in core logic without promotion;
• read-modify-write cycles need tests proving extensions survive.

11.3.3 When it wins

Choose the hybrid when known concepts drive operations but round-trip fidelity across evolving foreign documents also matters. It is often a strong compromise, but the merge policy is part of the public contract.

11.4 Candidate C: validated open bag

Candidate C becomes plausible when the team asks whether native construction is truly the product. If most work is ingest, conformance checking, small patches, and pass-through serialization, translating a rapidly changing external document into dozens of classes may add ceremony without removing interpretation from any important consumer.

The document or selected nodes remain lists. A schema, generated key table, or boundary validator checks known constraints, while operations use accessors and explicit validation stages.

11.4.1 What it buys

• close fidelity to an open or changing external representation;
• straightforward preservation of unknown keys;
• schema updates can change data tables rather than class signatures;
• low translation overhead when the package mostly transports documents.

11.4.2 What it costs

• nominal meaning and dispatch axes are weaker;
• arbitrary list edits can bypass validity unless operations revalidate;
• consumers may repeat lookups, presence checks, and interpretation;
• reflection reveals list shape rather than domain intent;
• aggregate errors can arrive later and with less local context.

11.4.3 When it wins

Choose the open bag when transport, validation, and pass-through are the main product and native semantic operations are few. It can be the correct stable architecture under external-schema authority.

11.5 Comparing candidates by consumer

11.5.1 Construction

A property-rich model gives the strongest discoverable constructor surface. The hybrid gives strong constructors for known concepts and an extension escape hatch. The open bag minimizes translation but depends more heavily on boundary validation and documentation.

11.5.2 Supported transformation

For add_layer(), all three can enforce root-level id and reference laws if the operation goes through a validated root. The difference is bypass risk: native replacement paths can revalidate declared contracts, while ordinary list mutation requires convention or an explicit validation pass.

11.5.3 External evolution

The property-rich model can evolve safely when generation and versioning are disciplined. The hybrid absorbs unknown additions but must preserve and merge them. The open bag accepts change most easily but exposes fewer guarantees.

11.5.4 Projection and interface generation

Property-rich declarations help when documentation, schema, and UI consumers need the same metadata. A hybrid can project the typed portion and label extensions separately. An open bag should project from its external schema rather than infer contracts from observed values.

11.5.5 Analysis and dispatch

If legend generation, field usage, or renderer behavior varies by layer kind, typed variants earn their cost. If serialization is the only consumer, open data plus validation may suffice.

11.6 An annotated choice

Assume the first release promises known-layer construction, invariant-preserving reorder/add operations, round-trip preservation of foreign keys, and one engine serializer. It does not interpret expressions.

A typed-known-plus-extension model is reasonable under those promises:

1. source and known layer identities are classes because operations validate or dispatch by kind;
2. ids and source references use shared property contracts;
3. known paint literals are checked from declared metadata;
4. expressions remain opaque and delegated to the external validator;
5. unknown fields on known layer variants are retained in an extension bag, while wholly unknown variants become an explicit unknown_layer carrying their uninterpreted source record;
6. the root checks unique ids and references;
7. named mutators use S7::set_props() so root validation follows edits;
8. serialization rejects collisions between typed fields and extension keys.

The list is the end of a decision chain, not a scorecard result. Start with the promised operations: they need stable layer identity and root-level reference laws, which rules out an entirely unstructured core. Next apply external evolution: foreign keys must survive even when the package cannot interpret them, which rules out a closed property-only model. Put the split where behavior changes—typed layer variants above it, inert extension data below it. Then place checks by visibility: variant-local paint rules on each known layer, id and source laws on the style root, expression conformance at the external engine. Finally choose round-trip equality: known fields compare semantically, while unknown fields require representational preservation. Each step constrains the next; none follows from class count alone.

This is contingent, not a destination every design approaches. If interface generation becomes central, more properties may move into the native model. If lossless transport becomes the product, more nodes may move toward validated bags. If expression analysis becomes a product, opaque expressions may become typed nodes.

11.6.1 One implementation slice

The following load-bearing slice was verified with S7 0.2.2 on R 4.6.1: insertion order, aggregate rejection of duplicate ids, and aggregate rejection of missing source references were exercised. It deliberately models only variant identity, aggregate laws, and the supported edit; parsing, paint validation, and serialization remain separate boundaries.

layer <- S7::new_class(
  "layer",
  abstract = TRUE,
  properties = list(
    id = S7::class_character,
    source = S7::class_character,
    extensions = S7::new_property(S7::class_list, default = list())
  )
)

fill_layer <- S7::new_class(
  "fill_layer",
  parent = layer,
  properties = list(
    paint = S7::new_property(S7::class_list, default = list())
  )
)

line_layer <- S7::new_class(
  "line_layer",
  parent = layer,
  properties = list(
    paint = S7::new_property(S7::class_list, default = list())
  )
)

unknown_layer <- S7::new_class(
  "unknown_layer",
  parent = layer,
  properties = list(raw = S7::class_list)
)

layer_list <- S7::new_property(
  class = S7::class_list,
  default = list(),
  validator = function(value) {
    valid <- vapply(
      value,
      S7::S7_inherits,
      logical(1),
      class = layer
    )
    if (!all(valid)) {
      "every element must inherit from layer"
    }
  }
)

style <- S7::new_class(
  "style",
  properties = list(
    sources = S7::new_property(S7::class_list, default = list()),
    layers = layer_list,
    extensions = S7::new_property(S7::class_list, default = list())
  ),
  validator = function(self) {
    ids <- vapply(self@layers, function(x) x@id, character(1))
    refs <- vapply(self@layers, function(x) x@source, character(1))
    messages <- character()

    if (anyDuplicated(ids)) {
      messages <- c(messages, "layer ids must be unique")
    }

    missing_refs <- setdiff(refs, names(self@sources))
    if (length(missing_refs)) {
      messages <- c(
        messages,
        paste("layers reference unknown sources:", paste(missing_refs, collapse = ", "))
      )
    }

    if (length(messages)) messages
  }
)

add_layer <- function(x, value, before = NULL) {
  if (!S7::S7_inherits(value, layer)) {
    cli::cli_abort("{.arg value} must inherit from {.cls layer}.")
  }

  layers <- x@layers
  if (is.null(before)) {
    layers <- c(layers, list(value))
  } else {
    ids <- vapply(layers, function(item) item@id, character(1))
    position <- match(before, ids)
    if (is.na(position)) {
      cli::cli_abort("No layer with id {.val {before}} exists.")
    }
    layers <- append(layers, list(value), after = position - 1L)
  }

  S7::set_props(x, layers = layers)
}

add_layer() owns the operation precondition, then performs one grouped replacement through S7::set_props(). Root validation consequently rejects a duplicate id or unresolved source after the proposed edit; the mutator does not duplicate those laws. Additional related root fields should be included in the same set_props() call rather than replaced one at a time.

The parser for an unknown layer extracts the common id and source fields needed by aggregate laws and stores the complete original record in raw. Supported edits treat that record as inert, and the serializer emits it unchanged unless a documented operation explicitly promotes it to a known variant. The layer_list contract ensures the aggregate validator never dereferences an arbitrary list element as though it were a layer.

There is an important limit to that guarantee. If a semantic class inherits from a parent with its own mutation API—most notably a data-frame or tibble base—then [, replacement methods, and ecosystem verbs become inherited side doors that may bypass subclass validation or drop the subclass. Base-type inheritance can still be honest, but “construction certifies validity” then applies only to guarded front doors; the package must document which parent operations preserve the class and where invariants are re-established.

11.7 Conversion pressure: three different jobs

11.7.1 Faithful conversion

Use S7::convert() when two representations preserve the same meaning without policy, diagnostics, or contextual choices. The acceptance law should be a genuine inverse or clearly named equivalence.

11.7.2 Adapter or compiler

Use parse_style(), adapt_style(), or compile_style() when work is tolerant, lossy, contextual, one-to-many, version-aware, or diagnostic-producing. A parser may preserve unknown variants that a compiler later rejects. Naming policy is more honest than calling this conversion.

11.7.3 Earned intermediate representation

Do not create a universal style IR by unioning every foreign field. Earn an IR when at least two real inputs or outputs reveal stable shared semantics and operations consume those semantics. sqlr earns its middle representation through rendering, reflection, semantic comparison, and diffing. One serializer does not provide the same evidence.

11.8 Acceptance laws

• parsing and serializing preserves unknown extensions under the chosen equality relation;
• supported edits preserve untouched extensions;
• duplicate layer ids and missing source references fail at the root;
• typed/extension key collisions fail explicitly;
• every known layer variant supports required operations;
• opaque expressions pass through unchanged but are not claimed valid;
• external conformance is checked after serialization;
• consumers do not repeat intrinsic id, presence, or reference checks.

11.9 What remains unsettled

This comparison does not decide how large property metadata should become, when an id deserves promotion from shared property to class, whether expression analysis will justify typed nodes, how version migration should work, or which equality relation users need. Those remain in the open-questions ledger.

11.10 Keeping the decision live

Record consequential choices in a five-field decision journal:

1. Decision — the representation or boundary being chosen.
2. Pressure — the concrete consumer, law, or cost forcing attention.
3. Options — the plausible candidates considered.
4. Choice — the selected option and its accepted concessions.
5. Doctrine — the principle followed or overruled, with the reason.

For this scenario, the entry would say that layer variants remain typed while unknown fields use extension bags; the pressure is native edit operations plus foreign round-trip fidelity; the alternatives are the three candidates above; the concession is split authority; and the doctrine trade is honest alternatives balanced against external-schema authority.

Review the journal at complexity checkpoints, not by calendar: the first mutator tests invariant placement; the first parser/serializer pair tests presence and equality together; the second output target tests whether an intermediate representation is emerging; the tenth repeated declaration tests whether generation now amortizes; and the first external implementation tests whether generics plus laws form a real contract. A checkpoint may move the design toward any candidate. The balanced conclusion remains: complexity and field count are pressures to price, not automatic votes for or against native classes.