Skip to main content
Domain: Datasets
Depends on: Data Schema
RFCs: RFC-001, RFC-015

​
Abstract

This spec defines how data of a dataset is represented in memory and at rest, the system columns that all data in ODF carries to support the bitemporal event model, the changelog representation of retractions and corrections, and the rules according to which a dataset’s schema is allowed to evolve over time.

​
In-memory and On-disk Formats

We differentiate two data formats by their purpose:
  • In-memory format - used when passing data around between the subsystems (e.g. when communicates with the ).
  • On-disk format - used for data at rest and by query engines when querying data from the entire dataset.
For our in-memory format we chose Apache Arrow, as it is:
  • Purposefully designed for minimal overhead interoperability between data processing systems
  • Minimizes copying of data
  • Hardware efficient
  • Supports streaming
For our on-disk format we choose Apache Parquet, as it is:
  • Fast to decode into Apache Arrow
  • Space and IO efficient thanks to the built-in compression
  • Efficient for querying thanks to the columnar structure and built-in basic indexing
Data on disk is stored in the multiple “part” files. Once a part file is written it is immutable for the entire lifetime of the dataset.

​
Common Data Schema

All data in the system is guaranteed to have the following columns: Representation:
TODO:
  • We are not allowing non-UTC-adjusted timestamps yet as Parquet does not offer a way to encode the timezone, meaning we need a reliable way to pass timezone information between different engines through some other means (e.g. Parquet metadata). Having naive/local timestamps without enforcing that they are accompanied by the specific timezone would be too error-prone.

​
Representation of Retractions and Corrections

are differentiated from regular in the via special op column, carrying an . Valid operation types are: The retract and correct-from events must carry the same exact data fields (all fields excluding offset, op, and system_time) as the event that is being retracted. The correct-to event carries the new values for the event being corrected. This event must immediately follow the correct-from event. This model allows data engines that operate on changelog / diff events that simultaneously carry both old and new values for event during a correction to reconstruct such events easily by sequentially reading the . A pair of correct-from and correct-to events must belong to the same - they cannot span across slices. The order of records in a dataset, as defined by their , is significant: a retraction or correction always follows the event it applies to, and correction pairs must stay adjacent. Records therefore must not be reordered (e.g. sorted by ) when they are processed or written, as that may produce an invalid stream. Not every stream needs to use all operation types. The following are all valid subsets of this model:
  • Append-only stream - contains only append events
  • Retract stream - contains only append and retract events, where a change of a previously published event is represented as its retraction followed by an append of the new value
  • Changelog stream - uses all operation types, differentiating retractions from corrections
See also:

​
Schema Evolution

may change when:
  • A different process is specified in the
  • A transformation of the is changed
  • A of one of the inputs of the changes
In general, changes are restricted to only adding new columns. Removing and renaming columns or changing data types would break downstream data consumers and therefore not allowed. A migration procedure should be used instead when it’s desired to introduce breaking changes in the .
Last modified on October 15, 2026