7  Documents, Schemas, and Languages: Authority at the Wire

7.1 A document model is a contract between representations

Map styles, configuration files, database schemas, and query trees are declarative values consumed by another system. A useful model identifies:

  • semantic nodes and their containment;
  • symbolic references that cross the containment tree;
  • the aggregate boundary that can check those references;
  • operations such as transform, render, inspect, and serialize;
  • the external format or engine whose authority still applies.

A root can certify unique names and resolvable references only when it sees the complete collection. It cannot certify behavior owned by a renderer, database, or version of an external specification. Chapter 4’s invariant visibility rule applies at the wire as well as inside ordinary values.

This recurring anatomy is worth making explicit. Composition edges form an ownership tree: a style contains sources and layers, or a catalog contains collections and items. Symbolic references add a second graph over that tree: a layer names a source, and an item links to a collection. Those references behave like foreign keys. Their uniqueness and resolution laws belong to the aggregate root, the lowest object that can see both definitions and uses. That root is naturally the unit of consistency, mutation, and serialization.

Boundary constructors complete the pattern by lifting objects users already have into document nodes. The public stacbuildr API, for example, provides constructors such as item_from_terra() and item_from_sf() rather than requiring users to transcribe raster and geometry metadata into nested lists. The constructor interprets the native object once, normalizes it, and enters the document model through its supported front door.

7.2 Documents as programs, within limits

A declarative document can often be treated as a program for another system: a map style is interpreted by a rendering engine, a SQL model by a database, and a query tree by a compiler. The package that builds it then resembles a front end. It parses convenient inputs, constructs a structured representation, checks the static facts it owns, and renders once at the boundary. This analogy explains why string templating is weak: strings erase the structure on which local checks and transformations depend.

The analogy is bounded, however. A STAC item is also an interchange record, and a configuration document may chiefly be persisted data. The package may preserve a foreign language without interpreting it, and the external engine still owns runtime behavior and full conformance. Calling every serializer a compiler would obscure those authority limits. Use the compiler vocabulary when semantic lowering or target policy is real; otherwise parser, adapter, and serializer are more accurate names.

There is no single correct relationship between an S7 model and a schema. Four architectures recur.

7.3 Four schema and model architectures

An external schema owns the content contract. The package validates a foreign document with the schema’s tooling, then stores or wraps the resulting list-like content for navigation and extension.

This architecture fits heterogeneous schemas supplied at runtime or systems whose main job is schema-mediated transport. The S7 properties may describe the wrapper and schema machinery while the external schema describes the contained entries. Editing arbitrary nested content does not automatically preserve conformance unless edits pass through another validation boundary.

Its advantage is fidelity to an authority that can change independently. Its cost is a deliberately weak semantic core: navigation and preservation may be available while domain dispatch and invariant-rich transformations are not. It works best when the package is a transport or validation tool rather than an alternate implementation of the foreign language.

An external schema or interface definition generates classes, field maps, or serializers that mirror the wire. This is appropriate when faithful coverage of a large, changing transport surface is itself the requirement.

Generated transport classes should be named and documented as transport types. They absorb foreign naming and version churn; an adapter can translate the subset with domain meaning into a hand-designed model. Generation does not establish semantics absent from the source, and generated code can still contain generator bugs. Test it against the external contract.

This is an anti-corruption layer, not a claim that two class graphs should remain isomorphic forever. Regenerate the transport side when the foreign schema changes; evolve the semantic side when the package’s concepts change. The adapter is where version differences, omission policy, and unsupported foreign constructs become explicit.

The model is organized around concepts and operations rather than wire objects. It may combine several foreign fields into one semantic value, split one foreign object into several concepts, or omit incidental structure.

This architecture earns its cost when nominal identity, invariants, generic operations, or multiple targets matter. The external schema remains evidence and a boundary authority, not the class diagram. Translation is often an adapter or compiler rather than a faithful conversion.

Because it organizes around meaning, this model can be smaller than the wire surface and deeper where operations need it. The trade-off is an explicit translation burden: losses, preservation of unknown fields, and the equality relation for round trips can no longer be assumed from structural similarity.

The package owns an inspectable semantic specification from which it derives some combination of classes, validators, schemas, documentation, interfaces, and renderers.

This is stronger than reflecting arbitrary validator closures. Only explicitly represented knowledge can be projected reliably: a declaration carrying minimum, maximum, membership, presence, and default policy can be inspected; an opaque function may enforce the same rule without exposing it. Generated projections can still disagree because translators can be incomplete or wrong. Shared declarations remove independent definitions and make agreement testable; they do not guarantee agreement for free.

Generation from an owned semantic specification is different from mirroring incidental foreign structure. The former can legitimately produce native domain classes when the specification actually contains their meaning.

This architecture is appropriate when a package truly owns the declaration and several consumers need it. A property specification might project to an S7 validator, JSON Schema, documentation, and a form. The declaration reduces independent definitions, but every projector remains code that can be incomplete or wrong. Cross-projection agreement tests are therefore part of the architecture, not optional reassurance.

These architectures can coexist. A package may receive an external transport document, preserve its unknown fields, adapt known content into a semantic model, and render that model through projections derived from an owned specification.

7.4 The roles of external authority

An external authority can participate in at least three ways.

Wire gate. An official schema or validator decides whether input or output conforms to a published format. This is a boundary check, not a replacement for domain invariants.

Runtime metadata provider. A database catalog, driver registry, or remote capability endpoint supplies allowed values and limits only after context is known. Preparation resolves a request against that metadata, much like prepare(specification, dataset) in Chapter 4.

Test oracle. An independent parser, renderer, or validator checks generated artifacts in tests. It can reveal disagreement without becoming the package’s in-memory representation.

State which role applies. “The schema is authoritative” is incomplete unless readers know whether it governs wire acceptance, dynamic capabilities, test comparison, or all three.

7.5 Typed and open are per-node choices

Typing depth should follow the guarantees and operations required at each node.

  • Use a typed class where node identity, invariants, references, or dispatch matter.
  • Use a validated open bag where keys are intentionally extensible but local shape rules still matter.
  • Keep an embedded language opaque when the package only preserves it.
  • Use a typed AST only when inspection, transformation, or multi-target compilation requires node semantics.

A document can therefore have a typed root and typed reference-bearing nodes, an open metadata bag, and opaque expression payloads. “Typed document” need not mean that every leaf has a class; “open document” need not mean that nothing is checked.

Every open bag should state its guarantees: key syntax, value types, duplicate policy, recursion limits, and whether unknown entries survive rendering.

7.6 Parser, adapter, compiler, and conversion

Name a translation according to its contract.

A parser interprets external syntax or loose data and returns known, unknown, or partial document variants with diagnostics. It owns syntax, version gating, normalization, and unknown-key policy.

An adapter maps between existing representations under an explicit policy. It may be contextual or lossy, but it does not imply that one representation is source code for the other.

A compiler translates a semantic model into a target representation and applies target policy. It may reject a model that parsed successfully, lower one node into many target nodes, or report unsupported features.

S7::convert() is best reserved for faithful representation changes with a clear source and target class. A contextual, diagnostic-producing, lossy, or one-to-many translation deserves a named function.

A serializer projects an accepted in-memory representation to a wire shape. A to_wire() method owns wire names, required type tags and version constants, omission versus explicit null, ordering or canonicalization policy, recursive descent, and preservation of declared extension content.

Its inverse, from_wire(), is a parser rather than object restoration by fiat. It owns syntax decoding, version gates, coercion to canonical values, unknown-key handling, migrations, diagnostics, and construction through the current front door. Keeping those responsibilities in one pair prevents wire knowledge from leaking into constructors, mutators, and render methods throughout the class graph.

For each translation, document:

  • what meaning and foreign content are preserved;
  • what is normalized or deliberately lost;
  • which context and target policy are applied;
  • what diagnostics can be produced;
  • which equality relation defines success.

Do not introduce a shared intermediate representation merely by taking the union of several foreign formats. An IR is earned when multiple real inputs or outputs reveal stable shared semantics.

7.7 Round trips require named equivalence

Two directions usually need different laws.

For a semantic model:

parse(render(model, dialect), dialect) ≈ model

For a foreign document:

render(parse(document), dialect) ≈ document

The symbol ≈ must be defined. Possible relations include:

  • exact representation identity, such as identical();
  • canonical equality after normalization;
  • semantic equivalence that ignores non-meaningful order or generated names;
  • wire equivalence after defaults and omission rules are resolved;
  • preservation equivalence that additionally requires unknown extensions to survive.

Most semantic document models promise canonical or semantic equivalence, not exact identity. A reflected database model may omit engine-specific details; a renderer may choose canonical whitespace; a parser may resolve defaults. Those can be sound round trips if the promised equivalence names the losses.

Reflection is also bounded. It reports what the external system exposes and what the reflector understands; it does not reconstruct every original authoring decision. Tests should compare within that declared observation boundary.

The public typedjson package offers a concrete round-trip design. For a supported value x, it tests identical(json_read(json_write(x, path)), x). Ordinary values keep ordinary JSON spellings—1L becomes 1, while the double 1 becomes 1.0—and only information JSON cannot carry is annotated. Thus NA_real_ receives a typed tag, character() records its empty-vector type, and an S7 object records the class identity needed to find current class code when reading. The worked point is not that every public format should preserve all R values. It is that the format states an exact equivalence, adds machinery only for distinctions needed by that equivalence, and tests the law over generated as well as curated values.

7.8 Native serialization is not a durable wire contract

saveRDS() is useful for trusted, version-bound caches and test fixtures. It is a poor default for a public or long-lived interchange format. S7 currently serializes the class specification with each object, so an object can retain an older class definition after package code evolves. R’s serialization version identifies the stream format; it does not migrate domain classes (Vaughan 2024; R Core Team 2026).

Native serialization is also a trust boundary. R versions before 4.4.0 were affected by CVE-2024-27322 (National Vulnerability Database 2024; CERT Coordination Center 2024), and current RDS files can still contain executable or session-bound objects. Do not accept untrusted RDS as inert data. Prefer a bounded plain-data format, an explicit schema version, and current-code parsing and migration. When RDS is deliberately used as a cache, record relevant versions, constrain its source, and make cache invalidation cheaper than heroic migration.

7.9 Typed state and raw content need authority rules

Preserving raw input beside typed state helps with unknown extensions and forward compatibility, but it creates two possible sources for the same field. The model must choose an authority rule.

Common policies include:

Typed wins for known fields. Rendering regenerates known fields from typed state and carries only unknown raw fields through. Mutators cannot leave a stale raw copy of a known field authoritative.

Raw snapshot is provenance only. Original bytes or parsed content are retained for diagnostics and audit but never merged back into output.

Lossless editor with synchronized patches. Edits update both a typed view and retained syntax tree through dedicated operations. This is substantially more complex because comments, order, formatting, and unknown nodes may all be meaningful to the editor.

Avoid an unspecified merge such as “typed values overwrite raw values where present.” Define collisions recursively:

  • how known keys are recognized at each node;
  • whether deleting a typed optional field removes or reveals a raw value;
  • where unknown keys are retained;
  • whether typed normalization changes raw spelling;
  • what happens when an extension later becomes a known field;
  • which version’s defaults apply.

The safest general policy for a semantic model is typed authority for known content, isolated extension bags for unknown content, and raw snapshots used only as provenance. Other policies are valid when their additional guarantees justify the synchronization machinery.

Consider a style document parsed from version 1. It contains a known layer with opacity = 0.5 and an unknown vendor key. A user changes the typed opacity to 0.8; meanwhile version 2 of the specification adopts the vendor key as a standard field with a default. Under “typed wins,” serialization must emit 0.8, preserve the still-unknown extension only in locations where it does not collide, and apply defaults according to the target version—not resurrect the raw 0.5 or silently let the old extension override the new known field. Under “raw snapshot is provenance,” none of the original parsed tree is merged at all. This scenario is why authority, collision, deletion, and version rules must be specified recursively rather than summarized as “merge raw and typed.”

7.10 Model and dialect as a compact example

Consider a small semantic query model with named relations and references, plus two dialect values. The model owns relational meaning and aggregate reference checks. The dialect owns quoting and target capabilities.

render(model, dialect) uses multiple dispatch because behavior depends on both concepts. A compiler may reject a valid model when a dialect lacks a feature, or lower it under a documented policy. parse() may preserve an unknown clause for inspection while compilation rejects that clause.

The acceptance laws can then be explicit:

  • supported transformations preserve model invariants;
  • every published model–dialect pair has method coverage;
  • unsupported pairs fail through a documented fallback;
  • render-then-parse preserves semantic equivalence;
  • unknown clauses survive only when preservation equivalence is promised;
  • an external engine or parser serves as a test oracle for emitted syntax.

The example connects all three chapters: Chapter 4 establishes what the model’s construction certifies, Chapter 5 defines the rendering operation and coverage contract, and this chapter defines authority and round-trip policy.

7.11 Embedded languages and safe generation

Use the shallowest representation that serves the consumers:

  1. opaque data for pass-through;
  2. constructor functions that build a target form;
  3. typed AST nodes and separate compiler passes for analysis or multiple targets.

Code generation should likewise follow represented knowledge. Enum catalogs, wire-name maps, repetitive transport fields, and projections from an owned semantic specification are good candidates. Domain distinctions absent from the source still require design.

Generated validators, classes, schemas, and documentation need direct tests. Inspectability helps debugging but does not prove equivalence. Never evaluate untrusted document content as R code; keep unknown extensions inert, and bound the depth and size of recursive input before walking it.

The ladder becomes concrete in query systems. At the opaque rung, a package stores a SQL fragment or foreign expression and promises only to preserve it. At the combinator rung, functions such as select(), filter(), or expr_match() build a valid target-shaped tree without giving every node a domain class. At the typed-AST rung, node identity supports independent analysis and compilation passes. The public quickr project takes this deeper route for typed R-to-Fortran translation, while dbplyr builds relational expressions and lowers them to database-specific SQL. Their depth is justified because analysis and multi-target translation are part of the product; a pass-through field with one renderer would not earn the same machinery.

7.12 Metaprogramming guardrails

R makes small languages easy to build, so generated checks and classes need stricter design discipline, not less. Checks as data can be useful when quoted, composable conditions are assembled into one inspectable validator. The generated function should print as ordinary understandable code and be tested directly. Splicing language objects created inside the package is appropriate; eval(parse(paste0(...))) and evaluation of document-supplied text are not.

Classes as data are equally legitimate because S7 classes are first-class values. A table can generate a regular family of enum or transport classes. Generation should stop where per-class meaning begins: if most generated classes need exceptions, readable hand-written definitions are the better model. In both patterns, generate structural regularity, keep inputs trusted, and test the generated artifact rather than only the factory.