Purpose
Open Data Fabric by design stores all data in the append-only event log format, always preserving the entire history. Unfortunately, a lot of data in the world is not stored or exposed this way. Some organizations may share their data in the form of periodic database dumps, while some choose to provide it as a log of changes between current and the previous export. When ingesting data from external sources, can choose between different that define how to combine the newly-ingested data with the existing one.Types
Ledger
strategy should be used for data sources containing append-only event streams. New data exports can have new rows added, but once data already made it into one export it should never change or disappear. A user-specified primary key is used to identify which events were already seen, not to duplicate them.Example
Imagine a data publisher that exposes dataset with historical populations of major cities around the world, for example:year column), so it’s a good sign that we’re dealing with a ledger data.
So we use the following root dataset manifest to ingest it:
ledger merge strategy with composite primary key (year, country, city), meaning that we expect only one record per city per year. We also set eventTimeColumn to use year as the source of event times.
The resulting dataset when ingested will look like this:
+A in op column identifies the and in this case all we have is Append operations.
Now let’s say after a census in Vancouver our source data changes to:
Note that the old events from 2019 were recognized as ones we already seen and were not added again.
Snapshot
strategy can be used for data exports that are taken periodically and contain only the latest state snapshot of the observed entity or system. Over time such exports can have new rows added, and old rows either removed or modified. It’s important to understand that publishers that use such formats routinely lose information. When a record in the database is updated, or one DB dump is replaced with another we not only lose the history of previous values, but you also lose the context of why those changes happened. This is a really bad default! Thesnapshot strategy transforms such data sources into a history of changes by performing the change data capture. It relies on a user-specified primary key to correlate the rows between the two snapshots.
A new event is added into the output stream whenever:
- A row with a certain primary key appears for the first time
- A row with a certain key disappears from the snapshot
- A row data associated with a certain key has changed
-R) and correction (-C / +C) events.
The Snapshot strategy also requires special treatment in regards to the event time. Since snapshot-style data exports represent the state of some system at a certain time - it is important to know what that time was. This time is usually captured in some form of metadata (e.g. in the name of the file, in the URL, or the HTTP caching headers. It should be possible to extract and propagate this time into a data column.
Example
Imagine a data publisher that exposes dataset with current populations of major cities around the world, for example:snapshot data.
So we use the following root dataset manifest to ingest it:
snapshot merge strategy with composite primary key (country, city). We also specify the eventTime of kind FromMetadata, instructing the ingest to use time from the caching headers as the event time of new records.
The resulting dataset when ingested will look like this:
Notice that since it’s the first time we ingested data both records have
op == '+A' values, i.e. “append” operation.
Now let’s say after a census in Vancouver our source data changes to:
The two added rows represent the
'-C' correct-from event that carries the old value that was changed, and '+C' correct-to event that carries the new value that was observed. Similarly the '-R' retract event can be issued for a record that disappeared in the source data.
This model is extremely powerful, as it not only provides us the full history of how data was evolving over time, but allows all downstream computations to automatically react to corrections and retractions.