Building a regulatory reporting pipeline that survives
Most reporting failures are not calculation errors. They are architecture decisions taken two years earlier by someone solving a different problem. A regulatory reporting pipeline has five stages, and the value of getting it right is not the first submission — it is the twentieth, after two framework releases, a product launch and a change of vendor. This is the shape that survives those, and the shortcuts that do not.
1. The five stages
| Stage | What it does | Owned by |
|---|---|---|
| 1. Extract | Pull source records at a defined point in time | Engineering |
| 2. Model | Normalise into an internal reporting data model, independent of any one return | Engineering + reporting |
| 3. Map | Translate the model into the data points a specific return requires | Reporting |
| 4. Validate | Run the framework rules, then the authority’s own | Reporting |
| 5. Submit and archive | File, capture the acknowledgement, retain the file and its inputs | Reporting + operations |
2. Pin the extract, or nothing else is reproducible
The single most valuable discipline is that stage 1 produces an immutable snapshot tied to a reference date, stored, and never re-derived. Everything downstream runs against the snapshot.
Without it, two things become impossible. You cannot reproduce a historical submission, because the source has moved on. And you cannot safely file a correction, because rebuilding from live data silently changes figures unrelated to the error you were fixing — the classic failure mode in a full-replacement regime.
The snapshot should carry three pieces of metadata as first-class fields: the reference date, the extraction timestamp, and the framework version the submission will be built against. The first two are not the same thing, and conflating them is what makes late submissions hard.
3. The mapping layer is an artefact, not code
The mapping — which source field, through which transformation, becomes which data point — is the asset with the longest useful life in the whole pipeline. It outlives staff, vendors and framework versions. Treating it as an implementation detail buried in code is how firms end up unable to answer “how is this figure derived?” without reading a query.
What a maintainable mapping records, per data point:
- the source field or fields and the system of record;
- the transformation, including every dimensional qualifier — because a data point is a metric plus its dimensions, not a cell position;
- the framework version it is valid for;
- the owner, and the date of last review;
- any known gap where the source cannot produce a required qualifier, and how it is currently derived.
That last line is the one that turns a reporting team’s private knowledge into an institutional record. A derived dimension is not automatically wrong, but a supervisory question about a single figure eventually asks how each of its qualifiers was determined.
4. Validate twice, in the right order
Framework validation rules are published with the framework and change on their own cycle — for the European Banking Authority’s frameworks, updated or small validation-rule packages are published quarterly. National authorities then commonly apply additional checks inside their own collection channels, and those are the ones absent from a vendor tool’s rule set.
The order that saves deadlines is: run the framework rules early, against last period’s accepted file as well as this period’s, to separate “our data changed” from “the rules changed”; then submit a test file to the authority’s channel as far ahead of the remittance date as the channel allows. A validation pass that happens only on the day of filing has no room to act on what it finds.
5. What to archive, and why
Facts: a supervisor queries a single figure in a return filed fourteen months ago. The reporting lead who built it has left.
What is needed to answer: the submitted file, the acknowledgement, the pinned extract it was built from, the mapping version in force at that reference date, and the validation output. If all five exist, the answer takes an afternoon. If the extract was not pinned, the answer is a reconstruction with a caveat.
What the practitioner does: archives all five together, keyed by reference date rather than by submission date — because corrections, restatements and queries all arrive addressed to a reference date, and an archive organised by when you filed cannot answer a question about what you filed for.
FAQ
Is an intermediate data model worth it for one return?
For exactly one return, arguably not. For the second, yes — and almost nobody ends up with one. The model is what makes cross-return reconciliation and framework upgrades tractable.
Why pin the extract?
So a submission is reproducible and a correction changes only what you intended. Rebuilding from live data for a historical reference date silently moves unrelated figures.
What should be keyed to the reference date?
The extract, the framework version, the mapping version, the validation output and the archive itself. Corrections and supervisory queries are always addressed to a reference date.
Related: Mapping source data to a template · Validation rules · Reference date vs remittance date
