Data Pipelines

Lana's reporting pipeline runs inside the application. The operational database publishes public events, the outbox relay writes those events into the Postgres data warehouse, dbt transforms the warehouse tables, and the Rust reporting jobs render downloadable report files.

Overview

The pipeline provides:

• Event-sourced warehouse tables for reporting and analytics.
• dbt transformations in report/dbt_lana_pg_dw.
• Regulatory and operational report files generated by Rust jobs in core/report.
• CI checks that compare public event schemas, dbt staging models, and report definitions.

Postgres DW

The data warehouse is populated from the public event outbox. Report generation does not depend on external landing tables or a separate Python orchestrator.

The application needs DW_PG_CON when the data warehouse is enabled. The same DSN is passed to dbt when the Rust report runner invokes the dbt command line tool. Without this setting, manual and end-of-day report generation are disabled for that deployment.

dbt Transformations

The dbt project lives at:

report/dbt_lana_pg_dw/
├── models/
│   ├── staging/
│   ├── intermediate/
│   └── outputs/
├── macros/
├── seeds/
├── dbt_project.yml
└── profiles.yml

The model tiers are:

Tier	Purpose
Staging	Normalize public event tables and typed payload fields.
Intermediate	Reconstruct entity state, intervals, balances, and regulatory business logic.
Outputs	Produce the final report relations or as-of SQL functions consumed by the Rust report runner.

As-of reports use the as_of_sql_function materialization so the report runner can call the output relation with a target date.

Report Generation

Manual and end-of-day report jobs are ordinary Lana jobs. They:

1. Run dbt deps and dbt build for the required Postgres dbt resources.
2. Query the dbt output table or SQL function from the Postgres DW.
3. Render CSV, XML, or text outputs from the report definition in core/report/report-definitions/reports.yml.
4. Upload the files to configured storage and persist ReportRun / Report records for the admin panel.

CI Checks

The data pipeline workflow validates the Postgres dbt project without provisioning external warehouse infrastructure:

• dev/check-pg-dw.py validates pg-dw staging and rollup models against public event schemas.
• ci/tasks/validate-report-columns.sh parses the Postgres dbt manifest and checks that as-of report definitions point at as_of_sql_function models using the expected as-of filter macro.