A named group of accounts. Members are assigned via the Member relation. Groups can be granted roles on resources, allowing permissions to be managed at the group level rather than per-account.
Container for key-value secrets. Every key must be a string. Values may be strings with raw unencrypted data or objects that signify the encoding.Map of string keys to arbitrary values.
Container for key-value variables. Every key must be a string. Values shoud reference fields in SecretSets and VariableSets.Map of string keys to arbitrary values.
Container for key-value variables. Every key must be a string. Values may be raw strings or objects that incorporate the encoding.Map of string keys to arbitrary values.
This schema aims to be a human-friendly variant of Arrow. Arrow currently specifies only the flatbuffer format which has many legacy to it and is not suited to be defined by humans, so we had to define our own schema format. While inspired by Arrow - this format makes a clear separation between logical data types and encoding (physical layout) of data in the chunks.
Defines a logical type of the field. Logical type determines the semantics and boudaries of a type and how it can be operated on, without a concern about encoding and physical layout of the data in chunks.
A fixed-point decimal number with a specified precision and scale.
Property
Type
Required
Format
Description
precision
integer
✔️
uint32
Total number of decimal digits that can be stored.
scale
integer
✔️
int32
Number of digits after the decimal point. In certain situations, scale could be negative number. For negative scale, it is the number of padding 0 to the right of the digits.
For example the number 12300 could be treated as a decimal has precision 3 and scale -2.
The unit of the timestamp value that determines its precision.
Default: Millisecond
timezone
string
The timezone is an optional string indicating the name of a timezone one of
* As used in the Olson timezone database (the “tz database” or “tzdata”), such as “America/New_York”. * An absolute timezone offset of the form “+XX:XX” or “-XX:XX”, such as “+07:30”.
Container for custom key-value extension attributes. Every key must be in the form of <domain>/<path> (e.g. kamu.dev/archetype) in order to fully disambiguate the value in the face of multiple extensions. Values may be any valid JSON including nested objects.Map of string keys to arbitrary values.
Describes checkpoint written during this transaction, if any. If an engine operation resulted in no updates to the checkpoint, but checkpoint is still relevant for subsequent runs - a hash of the previous checkpoint should be specified.
Last watermark of the output data stream, if any. Initial blocks may not have watermarks, but once watermark is set - all subsequent blocks should either carry the same watermark or specify a new (greater) one. Thus, watermarks are monotonically non-decreasing.
The state of the source the data was added from to allow fast resuming. If the state did not change but is still relevant for subsequent runs it should be carried, i.e. only the last state per source is considered when resuming.
Controls whether the dataset is readable by users who did not authenticate. Materialized into the ReBAC attribute store by the controller.Type: boolean
Controls whether the dataset is readable by any authenticated user. Materialized into the ReBAC attribute store by the Dataset controller.Type: boolean
Access role granted to a subject on a dataset. Note: in future this fixed enum schema will likely be replaced by a reference to a DatasetRole resources that defines granular permissions on different actions available on a dataset.
Describes checkpoint written during this transaction, if any. If an engine operation resulted in no updates to the checkpoint, but checkpoint is still relevant for subsequent runs - a hash of the previous checkpoint should be specified.
Last watermark of the output data stream, if any. Initial blocks may not have watermarks, but once watermark is set - all subsequent blocks should either carry the same watermark or specify a new (greater) one. Thus, watermarks are monotonically non-decreasing.
Last block of the input dataset that was previously incorporated into the derivative transformation, if any. Must be equal to the last non-empty newBlockHash. Together with newBlockHash defines a half-open (prevBlockHash, newBlockHash] interval of blocks that will be considered in this transaction.
Hash of the last block that will be incorporated into the derivative transformation. When present, defines a half-open (prevBlockHash, newBlockHash] interval of blocks that will be considered in this transaction.
prevOffset
integer
uint64
Last data record offset in the input dataset that was previously incorporated into the derivative transformation, if any. Must be equal to the last non-empty newOffset. Together with newOffset defines a half-open (prevOffset, newOffset] interval of data records that will be considered in this transaction.
newOffset
integer
uint64
Offset of the last data record that will be incorporated into the derivative transformation, if any. When present, defines a half-open (prevOffset, newOffset] interval of data records that will be considered in this transaction.
Identifier of the engine used for this transformation.
version
string
Version of the engine to use.
query
string
SQL query the result of which will be used as an output. This is a convenience property meant only for defining queries by hand. When stored in the metadata this property will never be set and instead will be converted into a single-iter queries array.
Specifies multi-step SQL transformations. Each step acts as a shorthand for CREATE TEMPORARY VIEW <alias> AS (<query>). Last query in the array should have no alias and will be treated as an output.
A local or remote dataset reference. When block is accepted this MUST be in the form of a DatasetId to guarantee reproducibility, as aliases can change over time.
alias
string
An alias under which this input will be available in queries. Will be populated from datasetRef if not provided before resolving it to DatasetId.
Condition capturing what caused this FlowRun to be scheduled. Set by the controller at creation time; never written by users. In case of a retry, the causes of the original run are preserved.
The trigger will fire upon first observed event. If another event arrives withing the cooldown interval the firing will be postponed until cooldown interval ends. I.e. trigger is guaranteed to fire, but may batch multiple events together into one flow run.
cooldownMaxBatch
integer
uint64
If an event is observed a cooldownMaxBatch number of times during the cooldown interval it will fire the trigger without waiting for cooldown to finish.
The trigger will fire upon first observed event. If another event arrives withing the cooldown interval the firing will be postponed until cooldown interval ends. I.e. trigger is guaranteed to fire, but may batch multiple events together into one flow run.
cooldownMaxBatch
integer
uint64
If an event is observed a cooldownMaxBatch number of times during the cooldown interval it will fire the trigger without waiting for cooldown to finish.
Top-level container for canonical representation of a resource that specifies the type and version of the resource, carries identity, ownership, and status information.
Property
Type
Required
Format
Description
$schema
string
✔️
type-uri
Identifies the controlling entity, a bounded context that this resource belongs to, and the version. Url should follow the pattern {base-url}/{context}/{version}/{name}.json e.g. https://opendatafabric.org/schemas/dataset/v1/Dataset.json.
Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. Unlike labels, annotations are not indexed and cannot be queried by.Map of string keys to arbitrary values.
Container of feneric contditions that can be added by contollers to provide additional information about the state of a resource. Keys uniquely identify the condition and should be in the form of URL to a schema describing this condition, e.g. https://opendatafabric.org/schemas/resource/ConditionReady.json.Map of string keys to arbitrary values.
Annotations is a key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. Unlike labels, annotations are not indexed and cannot be queried by.
References to resources that created this resource. Used for lineage tracking and cascading cleanup.
generation
integer
✔️
uint64
A sequential number that changes every time the resource header and spec are updated. Does not increment on status changes, thus signifying changes to the desired state. Populated by the system. Starts with 1.
Annotations is a key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. Unlike labels, annotations are not indexed and cannot be queried by.
Top-level container for user-authored representation of a resource that specifies the type and version of the resource and its desired state.
Property
Type
Required
Format
Description
$schema
string
✔️
type-uri
Identifies the controlling entity, a bounded context that this resource belongs to, and the version. Url should follow the pattern {base-url}/{context}/{version}/{name}.json e.g. https://opendatafabric.org/schemas/dataset/v1/Dataset.json.
Specifies how input files should be ordered before ingestion. Order is important as every file will be processed individually and will advance the dataset’s watermark.
Connects to an EVM (Ethereum) node to stream transaction logs.
Property
Type
Required
Format
Description
chainId
integer
uint64
Identifier of the chain to scan logs from. This parameter may be used for RPC endpoint lookup as well as asserting that provided nodeUrl corresponds to the expected chain.
nodeUrl
string
uri
Url of the node.
filter
string
An SQL WHERE clause that can be used to pre-filter the logs before fetching them from the ETH node.
signature
string
Solidity log event signature to use for decoding. Using this field adds event to the output containing decoded log as JSON.
Changelog stream merge strategy.This is the native stream format for ODF that accurately describes the evolution of all event records including appends, retractions, and corrections as per RFC-015. No pre-processing except for format validation is done.
Property
Type
Required
Format
Description
primaryKey
array(string)
✔️
Names of the columns that uniquely identify the record throughout its lifetime
Ledger merge strategy.This strategy should be used for data sources containing ledgers of events. Currently this strategy will only perform deduplication of events using user-specified primary key columns. This means that the source data can contain partially overlapping set of records and only those records that were not previously seen will be appended.
Property
Type
Required
Format
Description
primaryKey
array(string)
✔️
Names of the columns that uniquely identify the record throughout its lifetime
Snapshot merge strategy.This strategy can be used for data state snapshots that are taken periodically and contain only the latest state of the observed entity or system. Over time such snapshots can have new rows added, and old rows either removed or modified.This strategy transforms snapshot data into an append-only event stream where data already added is immutable. It does so by performing Change Data Capture - essentially diffing the current state of data against the reconstructed previous state and recording differences as retractions or corrections. The Operation Type “op” column will contain:
append (+A) when a row appears for the first time
retraction (-D) when row disappears
correction (-C, +C) when row data has changed, with -C event carrying the old value of the row and +C carrying the new value.
To correctly associate rows between old and new snapshots this strategy relies on user-specified primary key columns.To identify whether a row has changed this strategy will compare all other columns one by one. If the data contains a column that is guaranteed to change whenever any of the data columns changes (for example a last modification timestamp, an incremental version, or a data hash), then it can be specified in compareColumns property to speed up the detection of modified rows.
Property
Type
Required
Format
Description
primaryKey
array(string)
✔️
Names of the columns that uniquely identify the record throughout its lifetime.
compareColumns
array(string)
Names of the columns to compared to determine if a row has changed between two snapshots.
Upsert stream merge strategy.This strategy should be used for data sources containing ledgers of insert-or-update and delete events. Unlike ChangelogStream the insert-or-update events only carry the new values, so this strategy will use primary key to re-classify the events into an append or a correction from/to pair, looking up the previous values.
Property
Type
Required
Format
Description
primaryKey
array(string)
✔️
Names of the columns that uniquely identify the record throughout its lifetime
Reader for GeoJSON files. It expects one FeatureCollection object in the root and will create a record per each Feature inside it extracting the properties into individual columns and leaving the feature geometry in its own column.
Reader for Newline-delimited GeoJSON files. It is similar to GeoJson format but instead of FeatureCollection object in the root it expects every individual feature object to appear on its own line.
Reader for JSON files that contain an array of objects within them.
Property
Type
Required
Format
Description
subPath
string
Path in the form of a.b.c to a sub-element of the root JSON object that is an array or objects. If not specified it is assumed that the root element is an array.
ddlSchema
array(string)
DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types.
dateFormat
string
Sets the string that indicates a date format. The rfc3339 is the only required format, the other format strings are implementation-specific.
Default: rfc3339
encoding
string
Allows to forcibly set one of standard basic or extended encodings.
Default: utf8
timestampFormat
string
Sets the string that indicates a timestamp format. The rfc3339 is the only required format, the other format strings are implementation-specific.
Reader for GeoJSON files. It expects one FeatureCollection object in the root and will create a record per each Feature inside it extracting the properties into individual columns and leaving the feature geometry in its own column.
Property
Type
Required
Format
Description
ddlSchema
array(string)
DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types.
Reader for Newline-delimited GeoJSON files. It is similar to GeoJson format but instead of FeatureCollection object in the root it expects every individual feature object to appear on its own line.
Property
Type
Required
Format
Description
ddlSchema
array(string)
DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types.
DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types.
subPath
string
If the ZIP archive contains multiple shapefiles use this field to specify a sub-path to the desired .shp file. Can contain glob patterns to act as a filter.
