Source formats — detection, grammar, and drift policy

What each ingestion format's parser accepts, how it maps into the ASTs, and — most importantly — the policy for everything it does not model. Read this before changing anything under src/ingest/.

Two formats are covered today: TMDL semantic models (ingest::semantic_model → TabularDatabase) and PBIR reports (ingest::report → ReportModel, its own section below).

Detection

ingest::semantic_model(path) accepts:

• a .SemanticModel item folder (its definition/ subfolder is located automatically), or
• a definition/ folder itself (any directory that directly contains a model.tmdl, or is named definition).

Anything else is Error::UnsupportedFormat. The item's display name is read from .platform (metadata.displayName) beside definition/ — TMDL itself records no usable model name (model.tmdl names its root object Model). A missing or unreadable .platform yields None; a name is provenance, never liveness.

TMDL grammar subset

Stage 1 (src/ingest/tmdl.rs) scans tab-indented lines into a generic node tree; stage 2 maps known descriptors into TabularDatabase. Tab indentation only — a leading-tab count is the depth; spaces after the tabs are content.

Object headers, mapped into the AST:

Property forms: key: value (scalar), key = value (raw), bare key (flag, e.g. isHidden), annotation N = v, ref table N (fixes table order; unreferenced table files append in file-name order). cultures/ is never read. Names are single-quoted when they contain spaces or punctuation ('Sales Order'), with '' as an escaped quote; unquoting happens in the format layer. Column references are Table.Column, 'Quoted Table'.Column, 'Quoted Table'.'Quoted Column', or a bare (possibly quoted) Column for same-table references such as sortByColumn.

Multi-line expressions. A key = line with nothing after it captures the following deeper-indented block verbatim (blank lines inside preserved, dedented by the block's own first-line indent). The block closes at the first non-blank line above that indent — which is how a sibling property at depth+1, such as a measure's formatString, closes the expression. When the expression's first line sits on the header itself (measure M = VAR … followed by lines below the property level, i.e. deeper than depth+1), the value continues as first line + "\n" + block — the shape PBI Desktop serializes for multi-line DAX (see samples/…/Owners.tmdl).

Only two things are Error::Tmdl, never notices: a line that tokenizes to nothing (e.g. starting with : or =), and a flag-shaped node whose key is a known descriptor that requires a name (a bare table). Everything else unexpected is a skip notice; a run never fails because of drift.

PBIR reports

PBIR is not one grammar but a folder of one-object-per-file JSON documents, each carrying a $schema URL whose version drifts across Power BI releases (preview format; real exports run ahead of the published schemas at microsoft/json-schemas). There is therefore no fixed grammar: parsing is per-file key policies — keys the AST models are parsed, keys deliberately unmodeled are silent, anything else is UnknownProperty drift (see the Keys tables in src/ingest/pbir.rs). Files are read in a fixed order (report.json → definition.pbir → reportExtensions.json → pages in folder-name order → bookmarks in file-name order, then the phone layout's pages in folder-name order) so notices are deterministic.

Detection. ingest::report(path) accepts a .Report item folder (its definition/ is located automatically) or a definition/ folder itself (any directory directly containing a report.json). When the report ships a phone layout — a definition.mobile/ folder beside definition/ containing a pages/ folder (issue #49) — its pages are parsed like the desktop tree's and land in ReportModel::mobile_pages; the layout is optional and anchor-less, so any absence is silent. A report is parsed standalone — the semantic model it references need not sit beside it, because several reports can share one model. The reference is read from definition.pbir beside definition/ (byPath.path or byConnection.connectionString; the schema demands exactly one); any absence or drift yields DatasetReference::Unresolved — a pairing is never fabricated. Display name from .platform, as on the model side.

What each file contributes.

Persisted automatic filters. A visual's own filter normally lives in the container's filterConfig, but an automatic filter persists only after the filter pane has been expanded in the report's authoring history — and then it appears as a filter property inside the formatting objects (objects.general[].properties.filter). Both shapes join the visual's filters; the other objects properties are conditional formatting, whose fields are collected structurally (a FillRule input, an icon rule's comparison operands).

Filters, aliases, and condition trees. One filter-entry shape serves every scope: the filtered field under field (bookmark states spell it expression), and the condition under filter — a FilterDefinition (Version: 2, From, Where). From maps query aliases to entities; a SourceRef.Source resolves through it case-insensitively, a SourceRef.Entity names the table directly, and a hierarchy on a date variation sources its table through PropertyVariationSource, whose Property (the varied column) and Name (the variation) are carried on the FieldTarget::HierarchyLevel for the graph's variation resolution. An alias that matches no From entry yields FieldTarget::Written plus an UnresolvedAlias notice — the alias is not a table name, so it is never written in as one. Condition trees are walked structurally, not schema-driven: known field containers (Column, Measure, Aggregation with function codes 0–8, Min/Max/Percentile, Hierarchy/HierarchyLevel) are extracted wherever they nest, Literal values are data and never references, and a Where clause's Target arrays are references too. Known non-drift shapes that yield no reference: ScopedEval wrappers (unwrapped transparently), RangePercent bounds in formatting rules (they reuse the Min/Max keys for gradient ends), and visual-calculation sources (below).

Bookmark staleness. Power BI leaves a deleted page's section inside every bookmark that captured it — the saved state survives, but nothing can ever navigate to or re-apply it. A bookmark section therefore counts as a binding only when its page exists, judged by both sources: pages.json pageOrder and the pages/ folders. The live set is their case-insensitive union (a page named by either source is real; stripping its bookmarks' bindings would under-count roots), a disagreement between the sources is itself a StaleState notice, and a section outside the set is skipped whole — filters and saved projections — with one StaleState notice naming the bookmark and section. A bookmark whose activeSection is stale gets the same treatment (folded into the section's notice when they coincide). Report-level explorationState.filters are page-independent and always bind.

Errors. Only the anchor report.json can fail the run (Error::Io / Error::Json) — it is what makes the folder a report. An unreadable page, visual, or bookmark file, a malformed filter field, an unknown property: all notices, never failures.

PBIR known gaps

• Visual calculations. A field sourced from Subquery, SelectRef, or TransformTableRef names a visual-calculation local, not a model object, so it produces no binding and no notice; a subquery's own Select columns are walked with the subquery's aliases, so the model fields behind a calculation do bind. A SelectRef name (a calculation output referenced elsewhere in the same visual) cannot be resolved here, and a NativeVisualCalculation projection in a field well is not yet modeled.
• Bookmark-saved display state. singleVisual.orderBy (saved sort), saved formatting merges (singleVisual.objects, explorationState.objects — a conditional-formatting rule changed only inside a bookmark would be missed), and highlight.selection are deliberately unmodeled.
• Tooltip pages are read from section expr literals; any other spelling yields a MalformedValue notice rather than a silent loss.

Drift policy: two tiers of skipping

Ingestion entry points return Ingested<T> — the parsed value plus a Vec<SkipNotice> (path, TMDL line number or JSON pointer, SkipKind, detail). Notices are warnings as data: core never prints, and the CLI decides presentation. They are collected on every run, not only in debug builds, because a silent skip can surface later as a false "unused" finding.

Tier 1 — deliberately unmodeled, silent. Keys on the curated lists (the TMDL ignore list below; the PBIR Keys tables in src/ingest/pbir.rs) are skipped without a notice. Everything on them is metadata that cannot consume a model object, so skipping it cannot cause a false "unused" finding. The tests hold this honest in both directions: the samples tests fail if a list misses something the samples carry; the golden fixtures fail if a modeled key lands on a list.

Tier 2 — unexpected drift, noticed. An unknown object (root descriptor this crate does not know, unknown file or directory under definition/), an unknown property not on a list, a modeled value that fails to parse (MalformedValue), or a PBIR query alias that cannot be resolved (UnresolvedAlias).

The ignore list

Universal metadata: lineageTag, sourceLineageTag, changedProperty, description, annotation, extendedProperty. Extended properties carry no liveness of their own: the dynamic M parameter binding marker on a column (ParameterMetadata with "kind": 1) is the anonymous half of that binding — the authoritative half is the parameter expression's parameterValuesColumn property, which is modeled (see graph.md, "Dynamic M parameters"). The other ParameterMetadata shapes mark field parameters ("kind": 2) and what-if parameters ("version": 0), whose objects stay alive through ordinary DAX and report references.

Columns: dataType, formatString (static), summarizeBy, sourceColumn, sourceProviderType (the provider-side type of a DirectQuery column), dataCategory, isKey, isNameInferred, isDataTypeInferred, isUnique, isNullable, isDefaultLabel, isDefaultImage, isAvailableInMdx, keepUniqueRows, relatedColumnDetails, tableDetailPosition.

Measures: displayFolder, excludeFromModelRefresh. Tables: excludeFromModelRefresh, showAsVariationsOnly (engine-only visibility of the auto date/time machinery — the machinery itself is identified by the __PBI_LocalDateTable/__PBI_TemplateDateTable annotations, the isPrivate flag, and the LocalDateTable_/DateTableTemplate_ name prefixes, all mapped onto Table flags).

Model/database: culture, sourceQueryCulture, defaultPowerBIDataSourceVersion, discourageImplicitMeasures, dataAccessOptions, valueFilterBehavior, compatibilityLevel, createOrReplace, retainDataTillForceCalculate. Power Query query groups (queryGroup) are unmodeled in every form they take: block declarations in model.tmdl and membership properties on expressions and partitions.

Cultures (folder never read; keys listed for stray uses): cultureInfo, linguisticMetadata, contentType.

Partitions: mode. Roles: modelPermission. KPIs: statusGraphic. Hierarchies/levels: ordinal. Relationships: crossFilteringBehavior, fromCardinality, toCardinality, joinOnDateBehavior, hideArrows, securityFilteringBehavior, reliability.

Refresh policies (TMDL refreshPolicy, a table-level object) are modeled for their two expression properties only: sourceExpression (the RangeStart/RangeEnd-filtered query new policy-range partitions are created from) and pollingExpression (change detection). Both are evaluated at refresh time — deleting what they reference breaks refresh — so they flow through the same keep-alive pipeline as partition M (sourceExpression always; pollingExpression additionally through the DAX lexer, which resolves the measure references of the measure-based change-detection form). The policy's scalar vocabulary — policyType (kept for diagnostics), incrementalGranularity, incrementalPeriods, incrementalPeriodsOffset, rollingWindowGranularity, rollingWindowPeriods — names no model object and stays silent; an unrecognized key inside the policy is ordinary Tier-2 drift.

Verified spellings (validated against the AS engine)

No samples/ model exercises these, so the spellings were verified by loading probe models in the Analysis Services TMDL engine (via tomix-cli's tx load), and the golden fixture is held to the same standard — it loads clean in the engine:

• KPI: kpi blocks carry statusGraphic plus targetExpression, statusExpression, trendExpression. The short forms (target =, status =, trend =) are not valid TMDL — the engine rejects them — so this parser notices them as drift rather than mapping them.
• Detail rows: measures use detailRowsDefinition; tables use defaultDetailRowsDefinition (the engine rejects the measure spelling on a table).
• Calculation-group selection expressions: noSelectionExpression and multipleOrEmptySelectionExpression are objects — the expression itself, with an optional nested formatStringDefinition child for its dynamic format string. Standalone format-string spellings are rejected by the engine.
• tablePermission: the filter is the = expression (inline or block); a filterExpression child property also loads. A permission with no filter is just tablePermission <table>. All three shapes verified.
• relatedColumnDetails: a nameless object under a column with one groupByColumn: <column> per grouped column (the shape in samples/…/Toggle for breakdown.tmdl); it feeds Column::group_by_columns.
• calendar: a named object under a table (calendars require compatibility level 1701). Its column bindings live inside nameless calendarColumnGroup objects in two shapes — a time-related group lists plain column: <column> lines, and a time-unit association carries the unit as the group's = value plus primaryColumn:/associatedColumn: references. All of them name columns of the owning table and all feed Calendar::columns; the shapes were verified against the Analysis Services engine by round-tripping through tomix-cli.
• The engine also resolves relationship fromColumn/toColumn against the model's tables, and rejects /// doc comments (descriptions) on tablePermission. This parser does not cross-validate references — missing targets are the graph layer's findings, not parse failures — but the golden fixture keeps self-consistent references to stay engine-loadable.

TMDL known gaps

• Refresh policies are parsed at table level, matching TOM (Table.RefreshPolicy) and the folder TMDL Desktop serializes. A model that nests refreshPolicy under a partition is therefore not the modeled position; the drift policy notices it (unknown property 'refreshPolicy' on partition …) rather than silently reading it — the correct signal, since the policy's expressions keep objects alive.
• Date variations are modeled (Column::variations): a variation object on a date column declares the relationship and the table-qualified default hierarchy through which the engine serves the column — for auto date/time, a hidden relationship to an engine-generated LocalDateTable_*. The graph layer resolves a report's date-hierarchy binding written over the varied column through this declaration, so a hierarchy reachable only through a variation is no longer mis-reported as unused. Only showAsVariationsOnly (engine-only table visibility) stays on the ignore list; the machinery's identity comes from the __PBI_LocalDateTable / __PBI_TemplateDateTable annotations, isPrivate, and the name prefixes, all mapped onto Table flags.
• ColumnKind::CalculatedTableColumn has no sampled TMDL form; the column kind is not mapped. If it appears, the drift policy notices it — which is the correct signal, not silence. (The table-level calendar object, once in the same boat, is mapped now — see above.)
• A multi-line expression that continues at exactly the property level (depth+1) after a non-empty = value is indistinguishable from properties and reads as a sibling; TMDL serialization keeps expression bodies below the property level, so this has not been observed.