Introduces a sourceState field into AddData event to store the state of the generic data source from which the data is being added, allowing for incremental ingestion.
ODF implementations often need to store the state of the data source from which data is ingested to avoid expensive recomputations.For example when using SetPollingSource source state can be in the form of:
ETag of Last-Modified header for HTTP sources
filename of the last file matched by glob pattern
or a height of the block of indexed blockchain.
Currently, we don’t have anywhere to store such data.
When ingesting data, an ODF implementation will be able to attach opaque state data to the AddData event in order to resume ingestion most efficiently on the next iteration. The source state data will allow differentiating the kind of state that is preserved (similarly to MIME type) and will specify the identity of the source.
New SourceState metadata fragment will be introduced.The schema of AddData metadata event will:
be extended with an optional sourceState field
have outputData field made optional (as it’s possible for ingest to produce no new data but update source state, watermark, or checkpoint)
Two predefined source state kinds will be added:
odf/etag - for state identifiers similar to ETag HTTP header
odf/last-modified - for RFC3338 timestamps with meaning similar to Last-Modified HTTP header
One predefined source ID will be added:
odf/polling - referring to the source specified in the SetPollingSource metadata event
Both fields will be plain strings and not enums allowing different implementations to define their own extensions.It should be safe to ignore the source state that an implementation does not understand.Example SourceState:
Compatibility: Backwards-compatible, forwards-incompatible