# FlightSQL
Source: https://docs.kamu.dev/api/flight-sql



[Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) is a protocol for interacting with SQL databases using the Arrow in-memory format.

It is column-oriented, compact, and extremely performant compared to legacy protocols such as [JDBC](https://en.wikipedia.org/wiki/Java_Database_Connectivity) and [ODBC](https://en.wikipedia.org/wiki/Open_Database_Connectivity).

FlightSQL endpoint is typically exposed as:

```
datafusion+flightsql://kamu:kamu@<api-server hostname>:50050?insecure=True
```

Please refer to [Integrations](/integrations) page for some examples of FlightSQL protocol use.


# GraphQL
Source: https://docs.kamu.dev/api/graphql



[![GraphQL API](https://img.shields.io/static/v1?logo=graphql\&logoColor=F133A8\&label=\&message=GraphQL%20API\&color=gray\&style=for-the-badge)](https://api.demo.kamu.dev/graphql)

[GraphQL](https://graphql.org/) is our primary API for interacting with the node when building rich applications. [Kamu Web Platform](/platform) uses it extensively.

Here you can access [interactive GraphQL playground](https://api.demo.kamu.dev/graphql) of our demo node.

The playground is also available when you run your local node or when you use `kamu system api-server` CLI command under the HTTP port.

Here you can find the latest version of static [GraphQL API schema](https://github.com/kamu-data/kamu-cli/blob/master/resources/schema.gql).


# JDBC
Source: https://docs.kamu.dev/api/jdbc



**🚧 This page is under construction 🚧**

Kamu Node supports JDBC protocol through FlightSQL adapter.

To use it you will need a special client-side driver JAR.

Please see instruction for connecting [DBeaver](/integrations/dbeaver) and [Tableau](/integrations/tableau) to Kamu for guidance on where to download the drive and how to configure the connection.


# Kafka
Source: https://docs.kamu.dev/api/kafka





# MQTT
Source: https://docs.kamu.dev/api/mqtt



**🚧 This page is under construction 🚧**

Kamu Node supports ingesting data using the [MQTT](https://mqtt.org/) protocol. See <Schema /> and this [example](https://github.com/kamu-data/kamu-cli/tree/master/examples/mqtt).

In current implementation Kamu Node acts as an MQTT **client** that subscribes to a given set of topics , meaning that an external broker is required to ingest MQTT data, and this broker needs to be accessible to wherever environment Kamu Node is running.

In future versions we are planning to add support for a simple publish-only **broker** in Kamu Node itself, allowing you to connect MQTT devices to it directly. If your use case requires this - let us know!

## Data Delivery Guarantees

Kamu Node may not have perfect connectivity with MQTT broker, so prolonged periods of disconnects should be accounted for. Additionally, MQTT sources can be used even with [kamu-cli](/cli) tool, and since the tool doesn't use any background processes it will connect to MQTT broker only when you run:

```sh theme={null}
kamu pull my-mqtt-device
```

What data `kamu` can get when it is connected to the broker depends on:

* [Retained](https://www.hivemq.com/blog/mqtt-essentials-part-8-retained-messages/) message mode
* [QoS level](https://www.hivemq.com/blog/mqtt-essentials-part-6-mqtt-quality-of-service-levels/) of the publisher (e.g. device) and the subscriber (`kamu`)
* Implementation of your MQTT broker - since delivery semantics is left under-specified in the core MQTT spec.

For example:

* `retained: true, qos: 0 (at most once)` - will ensure that broker remembers the last message from a device and always transmits it to `kamu` upon reconnect. This mode is sufficient for a lot of cases e.g. where device periodically reports temperature data and `kamu` periodically ingests the readings on its own schedule.
* `retained: false, qos: 1 (at least once)` - will *likely* make broker to buffer all messages from the device and deliver them to `kamu` upon reconnect. This mode is good for devices like door sensors, where the whole series of events matters. The exact behavior is very broker-specific, so make sure to read you broker's documentation.

<Note>
  You can use [merge strategies](/cli/ingest/merge-strategies) to avoid duplication of retained data in your dataset.
</Note>


# OData
Source: https://docs.kamu.dev/api/odata



Kamu Node has an experimental support for [OData](https://www.odata.org/) protocol.

The endpoints structured with one OData service root per every account:

```
/odata/{account}/
/odata/{account}/$metadata
```

Every dataset in the account has a corresponding collection endpoint:

```
/odata/{account}/{dataset}
```

Examples:

```
https://node.demo.kamu.dev/odata/kamu/
https://node.demo.kamu.dev/odata/kamu/$metadata
https://node.demo.kamu.dev/odata/kamu/co.alphavantage.tickers.daily.spy
https://node.demo.kamu.dev/odata/kamu/co.alphavantage.tickers.daily.spy?$select=offset,close&$orderby=offset+asc&$top=10
```

You can find more endpoint information in [OpenAPI documentation](/api/rest/).

See documentation of our [datafusion-odata](https://github.com/kamu-data/datafusion-odata/) library for the list of limitations.


# ODF
Source: https://docs.kamu.dev/api/odf



**🚧 This page is under construction 🚧**

Kamu Node implements both Simple and Smart Transfer Protocols according to the [ODF Specification](/odf/spec/overview).

[Simple Transfer Protocol](/odf/spec/exchange/transfer-protocols#simple-transfer-protocol) (HTTP / REST) can be used for data access via HTTP in constrained environments.

[Smart Transfer Protocol](/odf/spec/exchange/transfer-protocols#smart-transfer-protocol) (HTTP + Websocket) provides the most efficient way to download and upload datasets. Thanks to native support of pre-signed URLs they avoid proxying large chunks of data through the API server and instead can direct uploads and downloads directly to your storage system. Built-in metadata compression also ensures best performance and optimal use of bandwidth for datasets with very large histories.

You can see individual endpoints of ODF protocol in our [REST API documentation](/api/rest).


# Blockchain Oracle
Source: https://docs.kamu.dev/api/oracle



Oracle API allows you to perform SQL queries over massive volumes of data in the ODF network directly from smart contracts.

It's the first of a kind **optimistic oracle** that enables reproducibility and verifiability of requests without an expensive consensus protocol (more on this [below](#verifiability)).

## Example

Example of the Solidity contract using ODF oracle:

```solidity theme={null}
// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.25;

import { OdfRequest, OdfResponse, IOdfClient, CborReader } from "@opendatafabric/contracts/Odf.sol";

// This sample contract makes an ODF oracle query to calculate
// a Canadian province with the most recorded COVID-19 cases
contract TestConsumer {
    using OdfRequest for OdfRequest.Req;
    using OdfResponse for OdfResponse.Res;
    using CborReader for CborReader.CBOR;

    IOdfClient private oracle;
    string public province;
    int public totalCases;

    // Initialize contract with the oracle address
    constructor(address oracleAddr) {
        oracle = IOdfClient(oracleAddr);
    }

    function initiateQuery() public {
        OdfRequest.Req memory req = OdfRequest.init();

        // Specify ID of the dataset(s) we will be querying.
        // Repeat this call for multiple inputs.
        req.dataset(
            "covid19.canada.case-details",
            "did:odf:fed01c8788dc7825dc95dfaa6c67f989b758d3ebcb1efcb9f47ea914470bd1f7f2bbb"
        );

        // Specify an arbitrary complex SQL query.
        // Queries can include even JOINs
        req.sql(
            "with by_provice as ( "
            "  select "
            "    province, "
            "    count(*) as count "
            "  from 'covid19.canada.case-details' "
            "  group by 1 "
            "), "
            "ranked as ( "
            "  select "
            "    row_number() over (order by count desc) as rank, "
            "    province, "
            "    count "
            "  from by_provice "
            ") "
            "select "
            "  province, "
            "  count "
            "from ranked where rank = 1"
        );

        // Send request to the oracle and specify a callback
        oracle.sendRequest(req, this.onResult);
    }

    // This function will be called by the oracle when
    // request is fulfilled by some data provider
    function onResult(OdfResponse.Res memory result) external {
        // Important: Make sure only the ODF oracle can supply you data
        require(msg.sender == address(oracle), "Can only be called by oracle");

        // We expect only one row: [province, totalCases]
        require(result.numRecords() == 1, "Expected one record");

        CborReader.CBOR[] memory record = result.record(0);
        province = record[0].readString();
        totalCases = record[1].readInt();
    }
}
```

## Getting Started

Start by installing [`@opendatafabric/contracts`](https://www.npmjs.com/package/@opendatafabric/contracts) package that contains oracle interface and request/response types.

```sh theme={null}
npm i @opendatafabric/contracts
```

See the example above for the recommended way of forming requests and decoding results.

More examples, documentation, and tests can be found in the [contracts repository](https://github.com/kamu-data/kamu-contracts).

## Supported Networks

| Network | Contract Address | Oracle Fee |
| :- | :- | - |
| [Ethereum Sepolia Testnet](https://chainlist.org/chain/11155111) | [`0xf1960569d1b4a23c34109d9341af6496ed90c0c3`](https://sepolia.etherscan.io/address/0xf1960569d1b4A23c34109D9341Af6496Ed90C0c3#events) | 0 |
| [BNB Smart Chain Testnet](https://chainlist.org/chain/97) | [`0x83857865971e941933dd36ebbf9475a867f67ca6`](https://testnet.bscscan.com/address/0x83857865971e941933dd36ebbf9475a867f67ca6#events) | 0 |
| Ethereum Mainnet | *coming soon* | 0 |
| Optimism | *coming soon* | 0 |

ODF oracle can be deployed on any EVM-compatible chain - you can request network support by filing an issue on the [contracts repository](https://github.com/kamu-data/kamu-contracts).

Oracle contract and all providers currently operate with **zero fees**.

## Verifiability

Every response submitted to the oracle by data providers includes special data that allows the query to be reproduced later against the same exact version of input data.

In REST API semantics an oracle request that looks like this:

```json theme={null}
{
    "query": "select offset, city, population from populations order by offset desc",
    "queryDialect": "SqlDataFusion",
    "datasets": [{
        "id": "did:odf:fed0119d20360650afd3d412c6b11529778b784c697559c0107d37ee5da61465726c4",
        "alias": "populations"
    }],
    "dataFormat": "JsonAoA",
    "include": ["input"],
}
```

Will be given a result such as:

```json theme={null}
{
    "input": {
        "query": "select offset, city, population from populations order by offset desc",
        "queryDialect": "SqlDataFusion",
        "dataFormat": "JsonAoA",
        "include": ["Input"],
        "datasets": [{
            "id": "did:odf:fed0119d20360650afd3d412c6b11529778b784c697559c0107d37ee5da61465726c4",
            "alias": "populations",
            "blockHash": "f1620708557a44c88d23c83f2b915abc10a41cc38d2a278e851e5dc6bb02b7e1f9a1a"
        }]
    },
    "output": {
        "data": [
            [1, "B", 200], 
            [0, "A", 100]
        ],
        "dataFormat": "JsonAoa"
    }
}
```

The `input.datasets` block of the response contains state information that associates every input dataset with the hash of the last block that was considered during the query.

Whenever a provider supplies a response - this state information is recorded in the events emitted by the oracle contract on the blockchain.

This data acts as a **commitment** using which a data consumer can **dispute** results of a query. If another party repeats the same computations using the state data and arrives at different result - this means that the original provider has acted maliciously. See [commitments documentation](/node/commitments) for details.

The reproducibility and verifiability of queries therefore make ODF the first of a kind **optimistic oracle** where:

* Queries can be fulfilled fast and cheap just by one provider
* Without going through a complex consensus mechanism
* While validity of data can still be ensured through the dispute resolution and sampling mechanisms along with staking and slashing.

## Commitment Properties

Using the criteria discussed in the [commitments documentation](/node/commitments) the oracle commitment can be described as follows:

* **Hiding** - NO
  * The entire request and response data is publicly available, but this as an inherent limitation of non-ZK blockchains
* **Binding** - YES
  * All inputs and outputs are immutably stored on-chain
* **Succinct** - YES\*
  * All response data ends up stored which we cannot avoid, but the commitment itself does not add much overhead
* **Deterministic** - YES
* **Reproducible** - YES
  * Original query with all of its parameters is stored on-chain and can be repeated against the state information in response
* **Non-repudiable** - YES
  * Automatically achieved as the provider signs the response transaction with its private key
* **Composable** - YES
  * Through recursive nature of the commitment

## Troubleshooting

To better understand what oracle providers return find the incoming oracle transaction in the block explorer e.g. [0x0f3a..4ff6](https://sepolia.etherscan.io/tx/0x0f3af200610ee2a11b13afda14eea808b04cbe012197a352ec95b558bbce4ff6).

Find the `ProvideResults` log:

<Diagram alt="ProvideResults event data" />

Copy the data field and paste it in [https://cbor.me/](https://cbor.me/) to get something like this:

<Diagram alt="Decoded CBOR data" />

The first three fields are: `version`, `success`, `data`. Refer to [`OdfResponse`](https://github.com/kamu-data/kamu-contracts/blob/e25e896ede177fdac7d34e9a4a3330094d23cc6f/src/OdfResponse.sol#L9) to understand the layout of the other fields.

Once you have one oracle response - you can use it as input in your unit tests to fully debug the decoding and processing logic without constantly re-deploying the contract and spending a lot of gas.

Here's an example of one such tests in `foundry`:

```solidity theme={null}
function testResponseDecoding() public pure {
    bytes memory responseBytes = hex"8501F58A826..92AFE32";
    OdfResponse.Res memory res = OdfResponse.fromBytes(0, responseBytes);
    assertEq(res.numRecords(), 10);
    CborReader.CBOR[] memory record = res.record(0);
    string memory voiceId = record[0].readString();
    int256 playtimeTotal = record[1].readInt();
    assertEq(voiceId, "lana-del-rey");
    assertEq(playtimeTotal, 6449);
}
```

## Registering Your Own Provider

**🚧 Under construction 🚧** - contact us if you'd like to use ODF oracle contract along with your private ODF node.


# Overview
Source: https://docs.kamu.dev/api/overview



Kamu Node supports multiple network protocols for managing metadata, bulk uploads/downloads of datasets, and querying data.

Not sure where to start? Here are some suggestions:

* [REST](/api/rest) is best for interacting with the node from scripts, libraries, and system-to-system integration
* [GraphQL](/api/graphql) is best for interacting with the node when building rich applications
* [FlightSQL](/api/flight-sql) is the most modern way to connect SQL-based systems together including various BI and analytics tools
* [MQTT](/api/mqtt) is used to ingest real-time data from IoT devices
* [OData](/api/odata) can be used to integrate with some legacy systems
* [Blockchain Oracle](/api/oracle) can be used to query data directly from nodes into smart contracts


# PostgreSQL
Source: https://docs.kamu.dev/api/postgres





# REST
Source: https://docs.kamu.dev/api/rest



[![REST API](https://img.shields.io/static/v1?logo=openapiinitiative\&logoColor=6BFF39\&label=\&message=REST%20API\&color=gray\&style=for-the-badge)](https://api.demo.kamu.dev/openapi)

REST is our primary API for interacting with the node from scripts, libraries, and system-to-system integration.

Here you can find the latest [OpenAPI documentation](https://api.demo.kamu.dev/openapi) along with usage examples.

Interactive Swagger UI is also available on HTTP port of all running nodes.

If you are considering to build applications on top of Kamu Node - also consider using [GraphQL API](/api/graphql).

For accessing individual components of the datasets (blocks, references, data slices) you can use [ODF protocol](/api/odf).

## Reading Data

To get dataset information like description, schema, etc. use `/metadata` endpoint:

```
https://node.demo.kamu.dev/kamu/com.cryptocompare.ohlcv.eth-usd/metadata?include=attachments,info,license,refs,schema,seed,vocab
```

To get sample data from any dataset use `/tail` endpoint:

```
https://node.demo.kamu.dev/kamu/com.cryptocompare.ohlcv.eth-usd/tail?limit=10
```

To execute arbitrary SQL that can access multiple datasets at once use `/query` endpoint:

```
https://node.demo.kamu.dev/query?query=select+1&skip=0&limit=10&include=input,schema
```

The `/query` endpoint also supports `POST` requests for finer control:

Example request body:

```json theme={null}
{
    "query": "select event_time, from_symbol, to_symbol, close from \"kamu/com.cryptocompare.ohlcv.eth-usd\"",
    "limit": 3,
    "queryDialect": "SqlDataFusion",
    "dataFormat": "JsonAoa",
    "schemaFormat": "ArrowJson"
}
```

Example response:

```json theme={null}
{
    "output": {
        "data": [
            ["2024-09-02T21:50:00Z", "eth", "usd", 2537.07],
            ["2024-09-02T21:51:00Z", "eth", "usd", 2541.37],
            ["2024-09-02T21:52:00Z", "eth", "usd", 2542.66]
        ],
        "dataFormat": "JsonAoa",
        "schema": {"fields": ["..."]},
        "schemaFormat": "ArrowJson"
    }
}
```

## Cryptographic Proofs

[Cryptographic proofs](/node/commitments) can be also requested to hold the node **forever accountable** for the provided result.

Example request body:

```json theme={null}
{
    "query": "select event_time, from_symbol, to_symbol, close from \"kamu/com.cryptocompare.ohlcv.eth-usd\"",
    "limit": 3,
    "queryDialect": "SqlDataFusion",
    "dataFormat": "JsonAoA",
    "schemaFormat": "ArrowJson",
    "include": ["proof"]
}
```

Example response:

```json theme={null}
{
    "input": {
        "query": "select event_time, from_symbol, to_symbol, close from \"kamu/com.cryptocompare.ohlcv.eth-usd\"",
        "queryDialect": "SqlDataFusion",
        "dataFormat": "JsonAoa",
        "include": ["Input", "Proof", "Schema"],
        "schemaFormat": "ArrowJson",
        "datasets": [{
            "id": "did:odf:fed0119d20360650afd3d412c6b11529778b784c697559c0107d37ee5da61465726c4",
            "alias": "kamu/com.cryptocompare.ohlcv.eth-usd",
            "blockHash": "f1620708557a44c88d23c83f2b915abc10a41cc38d2a278e851e5dc6bb02b7e1f9a1a"
        }],
        "skip": 0,
        "limit": 3
    },
    "output": {
        "data": [
            ["2024-09-02T21:50:00Z", "eth", "usd", 2537.07],
            ["2024-09-02T21:51:00Z", "eth", "usd", 2541.37],
            ["2024-09-02T21:52:00Z", "eth", "usd", 2542.66]
        ],
        "dataFormat": "JsonAoa",
        "schema": {"fields": ["..."]},
        "schemaFormat": "ArrowJson"
    },
    "subQueries": [],
    "commitment": {
        "inputHash": "f1620e23f7d8cdde7504eadb86f3cdf34b3b1a7d71f10fe5b54b528dd803387422efc",
        "outputHash": "f1620e91f4d3fa26bc4ca0c49d681c8b630550239b64d3cbcfd7c6c2d6ff45998b088",
        "subQueriesHash": "f1620ca4510738395af1429224dd785675309c344b2b549632e20275c69b15ed1d210"
    },
    "proof": {
        "type": "Ed25519Signature2020",
        "verificationMethod": "did:key:z6MkkhJQPHpA41mTPLFgBeygnjeeADUSwuGDoF9pbGQsfwZp",
        "proofValue": "uJfY3_g03WbmqlQG8TL-WUxKYU8ZoJaP14MzOzbnJedNiu7jpoKnCTNnDI3TYuaXv89vKlirlGs-5AN06mBseCg"
    }
}
```

A client that gets a proof in response should perform [a few basic steps](/node/commitments#response-validation) to validate the proof integrity. For example making sure that the DID in `proof.verificationMethod` actually corresponds to the node you're querying data from and that the signature in `proof.proofValue` is actually valid. Only after this you can use this proof to hold the node accountable for the result.

A proof can be stored long-term and then disputed at a later point using your own node or a 3rd party node you can trust via the `/verify` endpoint.

Example request:

```json theme={null}
{
    "input": {
        "query": "select event_time, from_symbol, to_symbol, close from \"kamu/com.cryptocompare.ohlcv.eth-usd\"",
        "queryDialect": "SqlDataFusion",
        "dataFormat": "JsonAoa",
        "include": ["Input", "Proof", "Schema"],
        "schemaFormat": "ArrowJson",
        "datasets": [{
            "id": "did:odf:fed0119d20360650afd3d412c6b11529778b784c697559c0107d37ee5da61465726c4",
            "alias": "kamu/com.cryptocompare.ohlcv.eth-usd",
            "blockHash": "f1620708557a44c88d23c83f2b915abc10a41cc38d2a278e851e5dc6bb02b7e1f9a1a"
        }],
        "skip": 0,
        "limit": 3
    },
    "subQueries": [],
    "commitment": {
        "inputHash": "f1620e23f7d8cdde7504eadb86f3cdf34b3b1a7d71f10fe5b54b528dd803387422efc",
        "outputHash": "f1620e91f4d3fa26bc4ca0c49d681c8b630550239b64d3cbcfd7c6c2d6ff45998b088",
        "subQueriesHash": "f1620ca4510738395af1429224dd785675309c344b2b549632e20275c69b15ed1d210"
    },
    "proof": {
        "type": "Ed25519Signature2020",
        "verificationMethod": "did:key:z6MkkhJQPHpA41mTPLFgBeygnjeeADUSwuGDoF9pbGQsfwZp",
        "proofValue": "uJfY3_g03WbmqlQG8TL-WUxKYU8ZoJaP14MzOzbnJedNiu7jpoKnCTNnDI3TYuaXv89vKlirlGs-5AN06mBseCg"
    }
}
```

Example response:

```json theme={null}
{
    "ok": false,
    "error": {
        "kind": "VerificationFailed::OutputMismatch",
        "actual_hash": "f162..c12a",
        "expected_hash": "f162..2a2d",
        "message": "Query was reproduced but resulted in output hash different from expected.
                    This means that the output was either falsified, or the query
                    reproducibility was not guaranteed by the system.",
    }
}
```

See [commitments documentation](/node/commitments) for details.

## Ingesting Data

Example using `curl`:

```sh theme={null}
echo '[{"foo": "bar"}]' | curl -v -X POST \
    'https://api.demo.kamu.dev/account/my-dataset/ingest' \
    -H 'Authorization: Bearer <access-token>' \
    -H 'Content-Type: application/json'
```

Example using `xh`:

```sh theme={null}
echo '[{"foo": "bar"}]' | xh -v POST \
    'https://node.demo.kamu.dev/account/my-dataset/ingest' \
    'Authorization: Bearer <access-token>' \
    'Content-Type: application/json'
```

Content types correspond directly to [supported input formats](/cli/ingest/input-formats):

| Media Type | Example |
| - | - |
| `text/csv` | <code>2020-01-01T00:01:00Z,-123.12,49.28<br />2020-01-01T00:02:00Z,-123.13,49.29</code> |
| `application/json` | <code>\[\{<br />  "event\_time": "2020-01-01T00:01:00Z",<br />  "long": -123.12,<br />  "lat": 49.28<br />}, \{<br />  "event\_time": "2020-01-01T00:02:00Z",<br />  "long": -123.13,<br />  "lat": 49.29<br />}]</code> |
| `application/x-ndjson` | <code>\{"event\_time": "2020-01-01", "long": -123.12, "lat": 49.28}<br />\{"event\_time": "2020-01-01", "long": -123.13, "lat": 49.29}</code> |
| `application/geo+json` | <code>\{<br />  "type": "FeatureCollection",<br />  "features": \[<br />    \{"type": "Feature", "geometry": \{...}, "properties": \{...}},<br />    \{"type": "Feature", "geometry": \{...}, "properties": \{...}}<br />  ]<br />}</code> |
| `application/x-ndgeojson` | <code>\{"type": "Feature", "geometry": \{...}, "properties": \{...}}<br />\{"type": "Feature", "geometry": \{...}, "properties": \{...}}</code> |
| `application/vnd.apache.parquet` | <code>binary data</code> |
| `application/vnd.esri.shapefile` | <code>binary data</code> |


# Webhooks
Source: https://docs.kamu.dev/api/webhooks



**🚧 This page is under construction 🚧**

Kamu Node currently support the Webhook subsciptions to modifications of a dataset.

Subscriptions can be created via Web UI and GraphQL API and monitored via the "Flows" view.

Webhook mechanism support payload signing according to [IETF RFC-9421](https://datatracker.ietf.org/doc/html/rfc9421).


# WebSocket
Source: https://docs.kamu.dev/api/websocket





# About
Source: https://docs.kamu.dev/cli/about



<div>
  <img alt="Kamu logo" />

  [Repository] | [Demo] | [Tutorials] | [Examples] | [FAQ] | [Chat]

  <div>
    <a href="https://github.com/kamu-data/kamu-cli/releases/latest">
      <img />
    </a>

    <a href="https://github.com/kamu-data/kamu-cli/actions">
      <img />
    </a>

    <a href="https://discord.gg/nU6TXRQNXC">
      <img />
    </a>
  </div>
</div>

`kamu` *(pronounced [kæmˈuː](https://en.wikipedia.org/wiki/Albert_Camus))* is a command-line tool for management and verifiable processing of dynamic structured data. It's a green-field project that aims to enable **global collaboration on data** on the same scale as seen today in software.

## Intro

`kamu` is based on [Open Data Fabric](https://opendatafabric.org) protocol.

<div>
  <YouTube />

  <p><i>Introductory video</i></p>
</div>

You can think of `kamu` as:

* ***Local-first data lakehouse*** - a free alternative to Databricks / Snowflake / Microsoft Fabric that can run on your laptop without any accounts, and scale to a large on-prem cluster
* ***Kubernetes for data pipelines*** - an *infrastructure-as-code* framework for building ETL pipelines using [wide range of open-source SQL engines](https://docs.kamu.dev/cli/supported-engines/)
* ***Git for data*** - a tamper-proof ledger that handles data ownership and preserves full history of changes to source data
* ***Blockchain for data*** - a verifiable computing system for transforming data and recording fine-grain provenance and lineage
* ***Peer-to-peer data network*** - a set of [open data formats and protocols](https://github.com/open-data-fabric/open-data-fabric/) for:
  * Non-custodial data sharing
  * Federated querying of global data as if one giant database
  * Processing pipelines that can span across multiple organizations.

## Quick Start

Use the installer script *(Linux / MacOSX / WSL2)*:

```sh theme={null}
curl -s "https://get.kamu.dev" | sh
```

* Watch [introductory videos](https://www.youtube.com/watch?v=oUTiWW6W78A\&list=PLV91cS45lwVG20Hicztbv7hsjN6x69MJk) to see `kamu` in action
* Follow the ["First Steps"](/start/first-steps) guide through an online demo and installation instructions.

## How it Works

### Ingest from any source

`kamu` works well with popular data extractors like Debezium and provides [many built-in sources](/cli/ingest) ranging from polling data on the web to MQTT broker and blockchain logs.

<div>
  <img alt="Ingesting data" />
</div>

### Track tamper-proof history

Data is stored in [Open Data Fabric](/odf) (ODF) format - an open **Web3-native format** inspired by Apache Iceberg and Delta.

In addition to "table" abstraction on top of Parquet files, ODF provides:

* Cryptographic integrity and commitments
* Stable references
* Decentralized identity, ownership, attribution, and permissions (based on [W3C DIDs](https://www.w3.org/TR/did-core/))
* Rich extensible metadata (e.g. licenses, attachments, semantics)
* Compatibility with decentralized storages like [IPFS](https://ipfs.tech)

Unlike Iceberg and Delta that encourage continuous loss of history through Change-Data-Capture, ODF format is **history-preserving**. It encourages working with data in the [event form](https://www.kamu.dev/blog/a-brief-history-of-time-in-data-modelling-olap-systems/), and dealing with inaccuracies through [explicit retractions and corrections](/cli/transform/retractions-corrections).

### Explore, query, document

`kamu` offers a wide range of [integrations](/integrations), including:

* Embedded SQL shell for quick EDA
* Integrated Jupyter notebooks for ML/AI
* Embedded Web UI with SQL editor and metadata explorer
* Apache Superset and many other BI solutions

<div>
  <img alt="SQL Shell" />

  <img alt="Integrated Jupyter notebook" />
</div>

### Build enterprise-grade ETL pipelines

Data in `kamu` can only be [transformed through code](/cli/transform). An SQL query that cleans one dataset or combines two via JOIN can be used to create a **derivative dataset**.

`kamu` doesn't implement data processing itself - it integrates [many popular data engines](https://docs.kamu.dev/cli/supported-engines/) *(Flink, Spark, DataFusion...)* as plugins, so you can build an ETL flow that uses the strengths of different engines at different steps of the pipeline:

<div>
  <img alt="Complex ETL pipeline in Kamu Web UI" />
</div>

### Get near real-time consistent results

All derivative datasets use **stream processing** that results in some [revolutionary qualities](https://www.kamu.dev/blog/end-of-batch-era/):

* Input data is only read once, minimizing the traffic
* Configurable balance between low-latency and high-consistency
* High autonomy - once pipeline is written it can run and deliver fresh data forever with little to no maintenance.

### Share datasets with others

ODF datasets can be shared via any [conventional](https://docs.kamu.dev/cli/collab/repositories/) (S3, GCS, Azure) and [decentralized](/cli/collab/ipfs) (IPFS) storage and easily replicated. Sharing a large dataset is simple as:

```sh theme={null}
kamu push covid19.case-details "s3://datasets.example.com/covid19.case-details/"
```

Because dataset **identity is an inseparable part of the metadata** - dataset can be copied, but everyone on the network will know who the owner is.

### Reuse verifiable data

`kamu` will store the transformation code in the dataset metadata and ensure that it's **deterministic and reproducible**. This is a form of **verifiable computing**.

You can send a dataset to someone else and they can confirm that the data they see in fact corresponds to the inputs and code:

```sh theme={null}
# Download the dataset
kamu pull "s3://datasets.example.com/covid19.case-details/"

# Attempt to verify the transformations
kamu verify --recursive covid19.case-details
```

Verifiability allows you to [establish trust](/cli/collab/validity) in data processed by someone you don't even know and detect if they act maliciously.

Verifiable trust allows people to **reuse and collaborate** on data on a global scale, similarly to open-source software.

### Query world's data as one big database

Through federation, data in different locations can be queried as if it was in one big data lakehouse - `kamu` will take care of how to compute results most optimally, potentially delegating parts of the processing to other nodes.

Every query result is accompanied by a **cryptographic commitment** that you can use to reproduce the same query days or even months later.

### Start small and scale progressively

`kamu` offers unparalleled flexibility of deployment options:

* You can build, test, and debug your data projects and pipelines on a laptop
* Incorporate online storage for larger volumes, but keep processing it locally
* When you need real-time processing and 24/7 querying you can run the same pipelines with [`kamu-node`](/node) as a small server
* A node can be deployed in Kubernetes and scale to a large cluster.

### Get data to and from blockchains

Using `kamu` you can easily [read on-chain data](/cli/ingest/blockchain-source) to run analytics on smart contracts, and provide data to blockchains via novel [Open Data Fabric oracle](/api/oracle).

## Use Cases

In general, `kamu` is a great fit for cases where data is exchanged between several independent parties, and for (low to moderate frequency & volume) mission-critical data where high degree of trustworthiness and protection from malicious actors is required.

<Card>
  <Tabs>
    <Tab title="Open Data">
      To share data **outside of your organization** today you have limited options:

      * You can publish it on some open data portal, but lose ownership and control of your data
      * You can deploy and operate some open-source data portal (like CKAN or Dataverse), but you probably have neither time nor money to do so
      * You can self-host it as a CSV file on some simple HTTP/FTP server, but then you are making it extremely hard for others to discover and use your data

      Let's acknowledge that for organizations that produce the most valuable data (governments, hospitals, NGOs), publishing data is **not part of their business**. They typically don't have the incentives, expertise, and resources to be good publishers.

      This is why the goal of `kamu` is to make data publishing **cheap and effortless**:

      * It invisibly guides publishers towards best data management practices (preserving history, making data reproducible and verifiable)
      * Adds as little friction as exporting data to CSV
      * Lets you host your data on any storage (S3, IPFS, GCS, FTP etc.)
      * Maintain full control and ownership of your data

      As opposed to just the download counter you get on most data portals, `kamu` brings publishers closer to the communities allowing them to see who and how uses their data. You no longer send data into "the ether", but create a **closed feedback loop** with your consumers.
    </Tab>

    <Tab title="Science & Research">
      One of the driving forces behind `kamu`'s design was the [ongoing reproducibility crisis](https://www.nature.com/articles/533452a) in science, which we believe to a large extent is caused by our poor data management practices.

      After incidents like [The Surgisphere scandal](https://www.the-scientist.com/features/the-surgisphere-scandal-what-went-wrong--67955) the sentiment in research is changing from assuming that all research is done in good faith, to considering any research unreliable until proven otherwise.

      Data portals like Dataverse, Dryad, Figshare, and Zenodo are helping reproducibility by **archiving data**, but this approach:

      * Results in hundreds of millions of poorly systematized datasets
      * Tends to produce the research based on stale and long-outdated data
      * Creates lineage and provenance trail that is very manual and hard to trace (through published papers)

      In `kamu` we believe that the majority of valuable data (weather, census, health records, financial core data) **flows continuously**, and most of the interesting insights lie around the latest data, so we designed it to bring **reproducibility and verifiability to near real-time data**.

      When using `kamu`:

      * Your data projects are **100% reproducible** using a built-in stable references mechanism
      * Your results can be reproduced and **verified by others in minutes**
      * All the data prep work (that often accounts for [80% of time of a data scientist](https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/?sh=348d5f876f63)) can be shared and **reused** by others
      * Your data projects will **continue to function** long after you've moved on, so the work done years ago can continue to produce valuable insights with minimal maintenance on your part
      * Continuously flowing datasets are much **easier to systematize** than the exponentially growing number of snapshots
    </Tab>

    <Tab title="Data-driven Journalism">
      Data-driven journalism is on the rise and has proven to be extremely effective. In the world of misinformation and extremely polarized opinions data provides us an anchoring point to discuss complex problems and analyze cause and effect. Data itself is non-partisan and has no secret agenda, and arguments around different interpretations of data are infinitely more productive than ones based on gut feelings.

      Unfortunately, too often data has issues that undermine its trustworthiness. And even if the data is correct, it's very easy to pose a question about its sources that will take too long to answer - the data will be dismissed, and the gut feelings will step in.

      This is why `kamu`'s goal is to make data **verifiably trustworthy** and make answering **provenance** questions a **matter of seconds**. Only when data cannot be easily dismissed we will start to pay proper attention to it.

      And once we agree that source data can be trusted, we can build analyses and **real-time dashboards** that keep track of complex issues like corruption, inequality, climate, epidemics, refugee crises, etc.

      `kamu` prevents good research from going stale the moment it's published!
    </Tab>

    <Tab title="Business core data">
      `kamu` aims to be the most reliable data management solution that provides recent data while maintaining the **highest degree of accountability** and **tamper-proof provenance**, without you having to put all data in some central database.

      We're developing it with financial and pharmaceutical use cases in mind, where **audit and compliance could be fully automated** through our system.

      Note that we currently focus on mission-critical data and `kamu` is not well suited for IoT or other high-frequency and high-volume cases, but can be a good fit for insights produced from such data that influence your company's decisions and strategy.
    </Tab>

    <Tab title="Personal analytics">
      Being data geeks, we use `kamu` for data-driven decision-making even in our personal lives.

      Actually, our largest data pipelines so far were created for personal finance:

      * to collect and harmonize data from multiple bank accounts
      * convert currencies
      * analyze stocks trading data.

      We also scrape a lot of websites to make smarter purchasing decisions. `kamu` lets us keep all this data up-to-date with an **absolute minimal effort**.
    </Tab>
  </Tabs>
</Card>

## Features

`kamu` connects **publishers** and **consumers** of data through a decentralized network and lets people **collaborate** on extracting insight from data. It offers many perks for everyone who participates in this first-of-a-kind data supply chain:

<Card>
  <Tabs>
    <Tab title="For Data Publishers">
      * Easily **share your data** with the world **without moving it** anywhere
      * Retain full **ownership and control** of your data
      * Close the feedback loop and **see who and how uses your data**
      * Provide **real-time**, **verifiable and reproducible** data that follows the best data management practices
    </Tab>

    <Tab title="For Data Scientists">
      * **Ingest any existing dataset** from the web
      * Always **stay up-to-date** by pulling latest updates from the data sources with just one command
      * Use **stable data references** to make your data projects fully reproducible
      * **Collaborate** on cleaning and improving data of existing datasets
      * Create derivative datasets by transforming, enriching, and summarizing data others have published
      * **Write query once and run it forever** - our pipelines require nearly zero maintenance
      * Built-in support for **GIS data**
      * **Share** your results with others in a fully reproducible and reusable form
    </Tab>

    <Tab title="For Data Consumers">
      * **Download** a dataset from a shared repository
      * **Verify** that all data comes from trusted sources using 100% accurate **lineage**
      * **Audit** the chain of transformations this data went through
      * **Validate** that downloaded was not tampered with a single command
      * **Trust** your data by knowing where every single bit of information came from with our **fine grain provenance**
    </Tab>

    <Tab title="For Data Exploration">
      * Explore data and run **ad-hoc SQL queries**
      * Launch a **Jupyter notebook** with one command
      * Join, filter, and shape your data using SQL
      * Visualize the result using your favorite library
      * Explore complex pipelines in Web UI
    </Tab>
  </Tabs>
</Card>

## Community

If you like what we're doing - support us by [starring the repo](https://github.com/kamu-data/kamu-cli), this helps us a lot!

For the list of our community resource and guides on how to contribute [start here](/community).

[Repository]: https://github.com/kamu-data/kamu-cli

[Demo]: /cli/demo

[Tutorials]: /get-started/learning-materials

[Examples]: /examples

[FAQ]: /faq

[Chat]: https://discord.gg/nU6TXRQNXC


# Bad Data on Ingestion
Source: https://docs.kamu.dev/cli/bad-data/ingest-errors





# PII and Sensitive Data
Source: https://docs.kamu.dev/cli/bad-data/pii





# Retractions & Corrections
Source: https://docs.kamu.dev/cli/bad-data/retractions-corrections





# Bad Data in Upstream Datasets
Source: https://docs.kamu.dev/cli/bad-data/upstream





# Sharing & Collaboration
Source: https://docs.kamu.dev/cli/collab/index



While `kamu` is a very powerful tool for managing and processing data on your own computer, the real power of it becomes apparent only when you start exchanging data with other people. Thanks to its core properties it makes sharing data reliable and safe both within your organization and between multiple completely independent parties.


# Sharing datasets with IPFS
Source: https://docs.kamu.dev/cli/collab/ipfs

Describes integration with content-addressable decentralized storage

The `kamu` and [IPFS](https://ipfs.io/) (The Interplanetary File System) projects have overlapping goals - preserving and growing humanity's knowledge.

While IPFS focuses on storing files (objects), `kamu`'s focus is on structured data and dynamic data processing, but under the hood even a real-time dataset in `kamu` is just a set of files!

Below we show how `kamu` builds on top of many similar concepts as IPFS and how IPFS can be used as a very efficient **storage layer** for sharing your datasets and pipelines with the world.

## IPFS Basics

The IPFS is well-documented, so we will quickly review the basics, and if you find any of them confusing simply follow the links:

* IPFS is **decentralized**
  * You are fully responsible for your own data's persistence, replication, and availability ([more](https://docs.ipfs.io/concepts/persistence/))
  * To publish data you need to run a node ([more](https://docs.ipfs.io/concepts/nodes/))
  * To read data you can use your node or a public IPFS Gateway ([more](https://docs.ipfs.io/concepts/ipfs-gateway/))
* IPFS is a **content-addressable** storage ([more](https://docs.ipfs.io/concepts/content-addressing/))
  * The IPFS URL is based on a CID (content identifier) which is derived from a hash sum of the data it points to ([more](https://docs.ipfs.io/concepts/hashing/))
  * Every time the content of a file changes you get a **different URL**
  * Old and new file are different objects - the data in IPFS is **immutable** ([more](https://docs.ipfs.io/concepts/immutability/))
* **Mutability can be simulated** with special URLs that can point to different CIDs at different times
  * Currently supported naming mechanisms are [IPNS](https://docs.ipfs.io/concepts/ipns/) and [DNSLink](https://docs.ipfs.io/concepts/dnslink/)
  * IPNS uses a hashsum of a public key to create a unique name entry in a decentralized way ([more](https://docs.ipfs.io/concepts/ipns/)).

## Kamu and IPFS

Unlike any other data processing system, `kamu` was build from ground up with reproducibility in mind - it defaults to never losing history. As the [ODF spec](/odf/spec/overview) explains in detail - the best way to do this is to represent data as **append-only** event streams.

These event streams are represented by a linked list of metadata blocks (think git history or blockchain) that reference portions of raw data and checkpoint files:

<Diagram alt="ODF dataset structure" />

So, very similarly to IPFS, the composing parts of a dataset in `kamu` are:

* **Immutable** - dataset only changes by appending new blocks
* **Content-addressable** - a metadata block, data file, or a checkpoint are uniquely identified by their hashsum

Why it matters? Imagine you store a directory containing many large files in IPFS and it is assigned `CID1`. When you add a new file to it and do `ipfs add . -r` again IPFS will notice that it already has CIDs for all but one file and will reuse those objects:

<Diagram alt="CIDs in IPFS" />

If `CID1` remains "pinned" it basically represents the state of the same directory as `CID2` but at the previous point in time.

This also works for `kamu` datasets:

<Diagram alt="ODF dataset in IPFS" />

* Each time you push to IPFS you are only adding the blocks and objects that were not seen previously. There is no duplication.
* Previous CIDs remain valid - they simply point to an older subset of an event stream.
* Having just one "pointer" to a metadata block you can "walk" the entire metadata chain and discover all dataset components.

Unlike many other data processing systems that mutate data, the cheapness of an "append" operation in `kamu` allows it achieve near real-time latencies of data propagation even when handling massive datasets and using immutable storage like IPFS.

<Note>
  The above diagrams are good to build up some intuition, but this is not exactly how data looks like in IPFS on the [IPLD DAG](https://ipld.io/) level. An accurate diagram would need to consider:

  * Slicing of large files into multiple objects
  * Raw vs. wrapped nodes
  * Balanced vs. custom DAG structure
</Note>

## Pulling from IPFS

`kamu` supports pulling data from IPFS just like from any [other repository](/cli/collab/repositories).

You can use both `ipfs://` and `ipns://` URLs:

```bash theme={null}
kamu pull ipfs://bafybeietcz4lxovy3ejdhb67nt3lj43vaeuyhectkqfnmmlnatfug5vqhe --as my-dataset --no-alias
kamu pull ipns://k51qzi5uqu5dic6zu9i2f4afctxmsm298ypiuy3ijmob1w6m96c092qp4ev7mn --as my-dataset
```

<Tip>
  Since the content behind an `ipfs://` URL never changes we use `--no-alias` flag to skip creation of the pull alias.
</Tip>

### Configuring IPFS Gateway

Under the hood `kamu` will use the configured IPFS Gateway to fetch data converting the URL into something like `http(s)://{gateway}/ipfs/{cid}` or `http(s)://{gateway}/ipns/{key}`.

By default it will try to use your local IPFS daemon as a gateway. You can see this by running:

```bash theme={null}
kamu config get --with-defaults protocol.ipfs.httpGateway
> "http://localhost:8080/"
```

If you only read data from IPFS and not planning to write - you can avoid running an IPFS daemon by switching to one of the [public IPFS Gateways](https://ipfs.github.io/public-gateway-checker/):

```bash theme={null}
kamu config set --user protocol.ipfs.httpGateway "https://dweb.link/"
```

## Pushing to IPFS

Clearly, pushing to an `ipfs://` URL will not work as we don't know the CID upfront, not until all data has been hashed:

```bash theme={null}
kamu push my-dataset --to ipfs://<???>
```

We need a URL that remains stable when the underlying dataset is updated - we can use [IPNS](https://docs.ipfs.io/concepts/ipns/) for this.

IPNS is like a named pointer to a CID that can be re-pointed to a different CID only by its owner. To create unique collision-free names in a decentralized way and to have a way for you to prove ownership of this name at the same time IPNS is using cryptographic key pairs:

* A public/private key pair is generated
* Name is derived by hashing the public key
* Private key is used for signing publishing requests to prove that you are the owner of said name.

To push data to IPFS you will need a local IPFS daemon running.

We first generate a new key pair that we will use for our dataset:

```bash theme={null}
ipfs key gen my-dataset
> k51qzi5uqu5dgl4gf3uayepenee5tzix3r8oiwyimxsfuikr7alq928xxtwmew
```

We can now use this key to form a destination URL as `ipns://{key}` and push the dataset:

```bash theme={null}
kamu push my-dataset --to ipns://k51qzi5uqu5dgl4gf3uayepenee5tzix3r8oiwyimxsfuikr7alq928xxtwmew
```

<Warning>
  Currently IPNS keys have a default lifetime of 24h (see `ipfs name publish --help`). This means you'll need to run push command periodically (e.g. on a cron job) to prevent it from expiring. Otherwise your URL will become non-resolvable.
</Warning>

Since IPNS pointers are mutable the next time we update the dataset we can push it again using the push alias `kamu` created for us:

```bash theme={null}
# Update the dataset locally
kamu pull my-dataset 
> <Dataset is updated>

# Push updates via IPNS push alias
kamu push my-dataset
> Syncing dataset (my-dataset > ipns://k51q...wmew)
> Updated to <head> (1 block(s))
```

And that's all there is to it! You now have all the basic building blocks to create **fully decentralized data pipelines**, both on storage and compute levels!


# Privacy and Permissions
Source: https://docs.kamu.dev/cli/collab/privacy-and-permissions





# Provenance
Source: https://docs.kamu.dev/cli/collab/provenance





# Repositories
Source: https://docs.kamu.dev/cli/collab/repositories

Describes different approaches to sharing datasets

## Supported Repository Types

Data in `kamu` is shared via <Term />. There are multiple types of repositories that differ by the kinds of services they provide. The most basic repository allows you to upload ("push") and download ("pull") data, while more advanced can allow searching datasets, query data in-place, or subscribing for updates.

| Type | Description | Capabilities | URL Examples |
| :-: | - | :-: | - |
| Local FS | A basic repository that uses the local file system folder. Mainly used for examples and testing. | `pull`<br />`push` | `file:///home/me/example/repository`<br />`file:///c:/Users/me/example/repository` |
| HTTP(s) | A basic repository that provides read-only access to data. | `pull` | `https://example.org/dataset` |
| S3 | A basic repository that stores data in Amazon S3 bucket. Can be used with any S3-compatible storage API | `pull`<br />`push` | `s3://bucket.my-company.example`<br />`s3+http://my-minio-server:9000/bucket`<br />`s3+https://my-minio-server:9000/bucket` |
| IPFS | Uses IPFS HTTP Gateway for reading. Push is only possible via IPNS (see [details](/cli/collab/ipfs)). | `pull`<br />`push` | `ipfs://bafy...vqhe`<br />`ipns://k51q...v7mn` |
| ODF | ODF-native repositories types that support very fast data transfer and querying data in-place | `pull`<br />`push`<br />`query`<br />`search` | `odf+http://odf.acme.com`<br />`odf+https://kamu.dev` |

## Push / Pull Aliases

Starting out, you can always use explicit URLs when syncing data from and to repositories.

If some remote source contains a dataset you're interested in you can download it using the `pull` command:

```bash theme={null}
kamu pull https://example.org/dataset --as example-dataset
```

If you have a dataset you would like to share, or made some changes to a dataset you are sharing with your friends - you can upload the new data using the `push` command:

```bash theme={null}
kamu push example-dataset --to s3://my-bucket.example.org/example-dataset
```

When you run these commands, `kamu` will automatically create "pull" and "push aliases" for this dataset:

```bash theme={null}
kamu repo alias list example-dataset
```

```
┌─────────────────┬──────┬────────────────────────────────────────────┐
│     Dataset     │ Kind │                Alias                       │
├─────────────────┼──────┼────────────────────────────────────────────┤
│ example-dataset │ Pull │ https://example.org/dataset                │
│ example-dataset │ Push │ s3://my-bucket.example.org/example-dataset │
└─────────────────┴──────┴────────────────────────────────────────────┘
```

With aliases in place, next time you want to pull or push data you can simply do:

```bash theme={null}
kamu pull example-dataset  # Will pull from https://example.org/dataset
kamu push example-dataset  # Will push to s3://my-bucket.example.org/example-dataset
```

Both `pull` and `push` commands will analyze the state of the dataset in the <Term /> and at the repository and will only upload data and metadata that wasn't previously seen (a minimal update).

These commands are also "safe". They will detect all types of concurrent changes and history divergence and prevent you from overwriting someone else's changes.

<Note>
  When you pull a `Root` dataset from a remote source running `kamu pull` on it will attempt to sync it from the repository, and NOT execute a polling or other ingest action. In `kamu list` datasets with pull aliases will be showing up as `Remote(...)`.
</Note>

## Repositories

If you store multiple datasets side by side in some locations you can add it as a repository. Repositories are configured per <Term /> using the `kamu repo` command group.

To add new repo use:

```bash theme={null}
kamu repo add my-s3-bucket s3://my-bucket.example.org/
```

This repo will now be visible in `kamu repo list`.

To pull/push a dataset from/to this repo you can now use remote references like `<repo-name>/<dataset-name>`:

```bash theme={null}
kamu pull my-s3-bucket/example-dataset
kamu push example-dataset --to my-s3-bucket
```

## Searching for Data

To look for datasets in the repos use:

```bash theme={null}
kamu search covid19

# Searching with empty query lists (a limited number of) datasets in all repos
kamu search

# You can specify the search term and specific repo to look in
kamu search covid19 --repo my-s3-bucket
```

Search is delegated to the repository implementations and its capabilities depend on the type of the repo. Whereas smart repos may support advanced full-text search, simple storage-only repos may be limited to a substring search by dataset name.


# Validity of Data
Source: https://docs.kamu.dev/cli/collab/validity

Describes audit and verification procedures that can be used to ensure the data presented to you is trustworthy

This topic is also covered in detail in this video:

<YouTube />

With `kamu` sharing data becomes very easy, but with that problem out of the way you will soon start wondering "How can I trust the data I downloaded from someone else?". Let's first define what validity or trustworthiness of data means.

For a more technical description see <Term />.

### Validity of Root Data

Let's say you're about to use a root dataset containing historical weather information in your city. How can you be sure it's trustworthy?

Because source data is non-reproducible, its validity depends entirely on its publisher. Publisher is in full control of the data they present, which also means that measuring and processing errors and even malicious data can easily make its way into the root dataset. Make sure that your publishers have good reputation and prefer data that comes from well-established organizations (government or NGOs) that directly collect or operate the systems from which the data is gathered.

Aside from external audits, another way to improve confidence in data is to correlate it with data from other similar sources. In our example we could compare it with the data from a weather station in the neighboring city and look for anomalies.

### Validity of Derivative Data

Derivative data in `kamu` is created purely through transformations that are recorded in metadata. These transformations can still be malicious, but since they are usually small (e.g. a few SQL queries) we can easily audit them to ensure they are sound and done in good faith. Repeating this process for the entire transformation chain, starting with root datasets, will give you confidence in trustworthiness of derivative data.

### Verifying Validity

Based on the above, here are the steps needed to ensure a dataset is trustworthy:

* Inspect lineage and identify all root datasets it's composed of
* Ensure publishers of root datasets are reputable and credible
* Use lineage to audit all derivative transformations to see if they are sound and non-malicious
* Use `kamu verify` command to ensure that data you downloaded actually matches the declared transformations

Example:

```bash theme={null}
# Inspect the dependency graph of a dataset to find all root sources
kamu inspect lineage ca.vancouver.opendata.weather.aggregated-daily

# Inspect the transformations applied
# Repeat this for every derivative dataset in the dependency graph
kamu inspect query ca.vancouver.opendata.weather.aggregated-daily

# Verify the data matches the declared metadata and transformations
kamu verify --recursive ca.vancouver.opendata.weather.aggregated-daily
```

For every derivative dataset the `kamu verify` command does two things:

* Compares hashes of data you downloaded to the ones stored in metadata (ensures data is not tampered or corrupted)
* Executes the declared derivative transformations locally to compare the hash of the result to one stored in metadata (ensures that metadata was not spoofed to match the fake result)

<Info>
  Remember that in the fight for data validity you are not alone. Other people will also be verifying these datasets, so the Open Data Fabric network can quickly detect and exclude the participants who distribute malicious data.
</Info>


# Commands Reference
Source: https://docs.kamu.dev/cli/commands



## `kamu`

**Usage:** `kamu [OPTIONS] <COMMAND>`

**Subcommands:**

* `add` — Add a new dataset or modify an existing one
* `apply` — Create or update resources from manifest files
* `completions` — Generate tab-completion scripts for your shell
* `config` — Get or set configuration options
* `context [ctx]` — Manage resource contexts
* `delete [rm]` — Delete datasets or resources
* `export` — Exports a dataset
* `get` — Returns manifest representation of a resource
* `ingest` — Adds data to the root dataset according to its push source configuration
* `init` — Initialize an empty workspace in the current directory
* `inspect` — Group of commands for exploring dataset metadata
* `list [ls]` — List datasets or resources
* `log` — Shows dataset metadata history
* `login` — Authenticates with a remote ODF server interactively
* `logout` — Logs out from a remote Kamu server
* `new` — Creates a new dataset manifest from a template
* `notebook` — Starts the notebook server for exploring the data in the workspace
* `pull` — Pull new data into the datasets
* `push` — Push local data into a repository
* `rename [mv]` — Rename a dataset
* `reset` — Revert the dataset back to the specified state
* `repo` — Manage set of tracked repositories
* `search` — Searches for datasets in the registered repositories
* `sql` — Executes an SQL query or drops you into an SQL shell
* `summary` — Show resource summary for the active context
* `system` — Command group for system-level functionality
* `tail` — Displays a sample of most recent records in a dataset
* `ui` — Opens web interface
* `verify` — Verifies the validity of a dataset
* `version` — Outputs build information

**Options:**

* `-v` — Sets the level of verbosity (repeat for more)
* `--no-color` — Disable color output in the terminal
* `-q`, `--quiet` — Suppress all non-essential output
* `-y`, `--yes` — Do not ask for confirmation and assume the 'yes' answer
* `--trace` — Record and visualize the command execution as perfetto.dev trace
* `--show-error-stack-trace` — Show stack trace in case of a command execution error
* `--metrics` — Dump all metrics at the end of command execution

To get help for individual commands, use:
kamu \<command> -h
kamu \<command> \<sub-command> -h

## `kamu add`

Add a new dataset or modify an existing one

**Usage:** `kamu add [OPTIONS] [MANIFEST]...`

**Arguments:**

* `<MANIFEST>` — Dataset manifest reference(s) (path, or URL)

**Options:**

* `-r`, `--recursive` — Recursively search for all manifest in the specified directory
* `--replace` — Delete and re-add datasets that already exist
* `--stdin` — Read manifests from standard input
* `--name <N>` — Overrides the name in a loaded manifest
* `--visibility <VIS>` — Changing the visibility of the added dataset

  Possible values: `private`, `public`

This command creates a new dataset from the provided DatasetSnapshot manifest.

Note that after kamu creates a dataset the changes in the source file will not have any effect unless you run the add command again. When you are experimenting with adding new dataset you currently may need to delete and re-add it multiple times until you get your parameters and schema right.

In future versions the add command will allow you to modify the structure of already existing datasets (e.g. changing schema in a compatible way).

**Examples:**

Add a root/derivative dataset from local manifest:

kamu add org.example.data.yaml

Add datasets from all manifests found in the current directory:

kamu add --recursive .

Add a dataset from manifest hosted externally (e.g. on GihHub):

kamu add [https://raw.githubusercontent.com/kamu-data/kamu-contrib/master/ca.bankofcanada/ca.bankofcanada.exchange-rates.daily.yaml](https://raw.githubusercontent.com/kamu-data/kamu-contrib/master/ca.bankofcanada/ca.bankofcanada.exchange-rates.daily.yaml)

To add dataset from a repository see `kamu pull` command.

## `kamu apply`

Create or update resources from manifest files

**Usage:** `kamu apply [OPTIONS] [PATH]...`

**Arguments:**

* `<PATH>` — Manifest file or directory path(s)

**Options:**

* `-c`, `--context <CONTEXT_NAME>` — Override the current resource context for this invocation

* `--dry-run` — Preview the accepted changes without applying them

* `--format <FMT>` — Parse all selected files using the specified manifest format

  Possible values: `json`, `yaml`

* `-r`, `--recursive` — Recursively scan directories for manifests

* `--stdin` — Read manifest from standard input

* `--continue-on-error` — Apply each manifest independently instead of the batch as a single all-or-nothing transaction, so earlier successes survive a later manifest's rejection or failure

Applies one or more resource manifests to the active resource context.

If the active context is `local`, manifests are applied to the current
workspace. If the active context points to a remote server, manifests are
applied through the remote GraphQL API.

Use `--dry-run` to preview the accepted changes without applying them.

By default, a multi-manifest batch is applied atomically: if any manifest is
rejected or fails, the entire batch is rolled back, including manifests
earlier in the batch that would otherwise have succeeded. Use
`--continue-on-error` to instead apply each manifest independently, so that
earlier successes are kept even if a later manifest fails.

**Examples:**

Apply a single manifest:

kamu apply my-resource.yaml

Preview changes without applying them:

kamu apply my-resource.yaml --dry-run

Apply all manifests in a directory recursively:

kamu apply manifests/ --recursive

Apply multiple files in the given order:

kamu apply a.yaml b.json

Apply a manifest from standard input:

cat my-resource.yaml | kamu apply --stdin

Force JSON parsing regardless of file extension:

kamu apply generated.resource --format json

## `kamu completions`

Generate tab-completion scripts for your shell

**Usage:** `kamu completions <SHELL>`

**Arguments:**

* `<SHELL>`

  Possible values: `bash`, `elvish`, `fish`, `powershell`, `zsh`

The command outputs to STDOUT, allowing you to re-direct the output to the file of your choosing. Where you place the file will depend on which shell and which operating system you are using. Your particular configuration may also determine where these scripts need to be placed.

Here are some common set ups:

**Bash:**

Append the following to your `~/.bashrc`:

source \<(kamu completions bash)

You will need to reload your shell session (or execute the same command in your current one) for changes to take effect.

**Zsh:**

Append the following to your `~/.zshrc`:

autoload -U +X bashcompinit && bashcompinit
source \<(kamu completions bash)

Please contribute a guide for your favorite shell!

## `kamu config`

Get or set configuration options

**Usage:** `kamu config <COMMAND>`

**Subcommands:**

* `list [ls]` — Display current configuration combined from all config files
* `get` — Get current configuration value
* `set` — Set or unset configuration value

Configuration in `kamu` is managed very similarly to `git`. Starting with your current workspace and going up the directory tree you can have multiple `.kamuconfig` YAML files which are all merged together to get the resulting config.

Most commonly you will have a workspace-scoped config inside the `.kamu` directory and the user-scoped config residing in your home directory.

**Examples:**

List current configuration as combined view of config files:

kamu config list

Get current configuration value:

kamu config get engine.runtime

Set configuration value in workspace scope:

kamu config set engine.runtime podman

Set configuration value in user scope:

kamu config set --user engine.runtime podman

Unset or revert to default value:

kamu config set --user engine.runtime

## `kamu config list`

Display current configuration combined from all config files

**Usage:** `kamu config list [OPTIONS]`

**Options:**

* `--scope <SC>` — Which configs to use

  Default value: `combined`

  Possible values:

  * `user`:
    Includes only config in user home directory
  * `workspace`:
    Includes only current workspace config
  * `combined`:
    Includes configs in workspace, parent directories, and user home dir

* `--user` — Show only user scope configuration

* `--with-defaults` — Show configuration with all default values applied

* `-o`, `--output-format <FMT>` — Serialization format of the returned object

  Default value: `yaml`

  Possible values: `yaml`, `json`

## `kamu config get`

Get current configuration value

**Usage:** `kamu config get [OPTIONS] [CFGKEY]`

**Arguments:**

* `<CFGKEY>` — Path to the config option

**Options:**

* `--scope <SC>` — Which configs to use

  Default value: `combined`

  Possible values:

  * `user`:
    Includes only config in user home directory
  * `workspace`:
    Includes only current workspace config
  * `combined`:
    Includes configs in workspace, parent directories, and user home dir

* `--user` — Operate on the user scope configuration file

* `--with-defaults` — Get default value if config option is not explicitly set

* `-o`, `--output-format <FMT>` — Serialization format of the returned object

  Default value: `yaml`

  Possible values: `yaml`, `json`

## `kamu config set`

Set or unset configuration value

**Usage:** `kamu config set [OPTIONS] <CFGKEY> [VALUE]`

**Arguments:**

* `<CFGKEY>` — Path to the config option
* `<VALUE>` — New value to set

**Options:**

* `--scope <SC>` — Which configs to consider

  Default value: `combined`

  Possible values:

  * `user`:
    Includes only config in user home directory
  * `workspace`:
    Includes only current workspace config
  * `combined`:
    Includes configs in workspace, parent directories, and user home dir

* `--user` — Operate on the user scope configuration file

* `-i`, `--input-format <FMT>` — Serialization format of the provided object

  Default value: `yaml`

  Possible values: `yaml`, `json`

## `kamu context`

Manage resource contexts

**Usage:** `kamu context [NAME]
       context <COMMAND>`

**Subcommands:**

* `add` — Register a new remote resource context
* `list [ls]` — List configured resource contexts
* `delete [rm]` — Delete a remote resource context
* `check` — Check connectivity and authorization for a remote resource context
* `api-resources` — List supported resource types in the active context
* `use` — Switch the current resource context

**Arguments:**

* `<NAME>` — Context name to switch to

Contexts determine which workspace future resource commands will target.

When running inside a workspace, an implicit local context named `local` is
available automatically whenever no current context is selected. You can still
select `local` explicitly. Remote contexts can be registered either in the
workspace or in the user home scope.

**Examples:**

Show current context:

kamu context

List configured contexts:

kamu context ls

Check a remote context:

kamu context check demo

Refresh cached status for all remote contexts:

kamu context check --all

Switch to a context:

kamu context prod

Switch back to the local workspace context:

kamu context local

Register a workspace-scoped remote context:

kamu context add prod --url [https://api.kamu.dev](https://api.kamu.dev)

Register a user-scoped remote context:

kamu context add prod --url [https://api.kamu.dev](https://api.kamu.dev) --user

Register a remote context non-interactively with an existing token:

kamu context add prod --url [https://api.kamu.dev](https://api.kamu.dev) --access-token \<token>

List supported resource types in the active context:

kamu ctx api-resources

List supported resource types from a specific context:

kamu ctx api-resources --context prod

## `kamu context add`

Register a new remote resource context

**Usage:** `kamu context add [OPTIONS] --url <URL> <NEW_NAME>`

**Arguments:**

* `<NEW_NAME>` — Context name

**Options:**

* `--user` — Store context in the user home folder rather than in the workspace
* `--url <URL>` — Backend URL of the remote workspace
* `--access-token <TOKEN>` — Provide an existing access token instead of logging in
* `--password-login <LOGIN>` — User name for non-interactive password login
* `--password <PASSWORD>` — Password for non-interactive password login
* `--oauth-provider <PROVIDER>` — Name of the OAuth provider for non-interactive login, i.e. 'github'
* `--oauth-token <TOKEN>` — OAuth provider access token
* `--no-login` — Do not attempt to log in, only register the context
* `--repo-name <REPO_NAME>` — Repository name which will be used to store in repositories list
* `--skip-add-repo` — Don't automatically add a remote repository for this host

Registers a remote workspace context under a local name.

By default the context is stored in the current workspace. Use `--user` to
store it in the user home scope instead.

The name `local` is reserved for the implicit workspace context and cannot be
registered explicitly.

If no access token is stored for the given URL, the interactive browser login
flow is started automatically, provided the session is an interactive terminal.
In non-interactive sessions (CI, scripts) the context is only registered and a
warning is printed. Use the credential arguments below to authenticate
non-interactively, or `--no-login` to skip authentication entirely.

**Examples:**

Add a workspace-scoped remote context, logging in interactively if needed:

kamu context add prod --url [https://example.com](https://example.com)

Add a user-scoped remote context:

kamu context add prod --url [https://example.com](https://example.com) --user

Add a context using an existing access token:

kamu context add prod --url [https://example.com](https://example.com) --access-token \<token>

Add a context with non-interactive password login:

kamu context add prod --url [https://example.com](https://example.com) --password-login alice --password s3cret

Add a context with non-interactive OAuth login:

kamu context add prod --url [https://example.com](https://example.com) --oauth-provider github --oauth-token \<token>

Register a context without authenticating:

kamu context add prod --url [https://example.com](https://example.com) --no-login

## `kamu context list`

List configured resource contexts

**Usage:** `kamu context list [OPTIONS]`

**Options:**

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

Lists effective resource contexts configured in the workspace and user scopes.

When running inside a workspace, the implicit `local` context is also included.

**Examples:**

List contexts:

kamu context ls

List contexts in JSON:

kamu context ls -o json

## `kamu context delete`

Delete a remote resource context

**Usage:** `kamu context delete [OPTIONS] [NAME]`

**Arguments:**

* `<NAME>` — Context name

**Options:**

* `--user` — Delete context from the user home folder rather than in the workspace
* `--all` — Delete all remote contexts in the selected scope

Deletes a previously registered remote context from the selected scope.

By default deletion happens in the current workspace. Use `--user` to delete a
user-scoped context instead. Use `--all` to delete all remote contexts from
the selected scope.

The name `local` is reserved and cannot be deleted.

**Examples:**

Delete a workspace-scoped context:

kamu context delete prod

Delete a user-scoped context:

kamu context delete prod --user

Delete all workspace-scoped remote contexts:

kamu context rm --all

## `kamu context check`

Check connectivity and authorization for a remote resource context

**Usage:** `kamu context check [OPTIONS] [NAME]`

**Arguments:**

* `<NAME>` — Context name

**Options:**

* `--all` — Check all effective remote contexts

Checks backend reachability and access token validity for a context.

If no context name is provided, the effective current context is checked.
Use `--all` to refresh cached status for all configured remote contexts.

**Examples:**

Check a named context:

kamu context check demo

Check the current context:

kamu context check

Check all remote contexts:

kamu context check --all

## `kamu context api-resources`

List supported resource types in the active context

**Usage:** `kamu context api-resources [OPTIONS]`

**Options:**

* `-c`, `--context <CONTEXT_NAME>` — Override the current resource context for this invocation
* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

Lists resource types supported by the active context.

If the active context is `local`, this targets the current workspace. If the
active context points to a remote server, this targets that remote GraphQL API.

Use `--context` to override the current context for this invocation only.

**Examples:**

List supported resource types in the active context:

kamu ctx api-resources

List supported resource types from a specific context:

kamu ctx api-resources --context prod

List supported resource types in JSON:

kamu ctx api-resources -o json

## `kamu context use`

Switch the current resource context

**Usage:** `kamu context use <NAME>`

**Arguments:**

* `<NAME>` — Context name

Switches the current resource context to the specified named remote context.

This is the explicit form of `kamu context <name>`.

The special name `local` refers to the current workspace when one is available.

**Examples:**

Switch to a context:

kamu context use prod

Switch to the local workspace context:

kamu context use local

## `kamu delete`

Delete datasets or resources

**Usage:** `kamu delete [OPTIONS] [TARGET] [ARGS]...`

**Arguments:**

* `<TARGET>` — Target to delete: `datasets`, `%` (all resource types), or a resource selector such as `variablesets`, `vs`, `secretsets`, `ss`, `storages`, or `st`
* `<ARGS>` — Dataset selector(s) in dataset mode, or a single resource selector in resource mode

**Options:**

* `-c`, `--context <CONTEXT_NAME>` — Override the current resource context for this invocation
* `-a`, `--all` — Delete all matched datasets or all resources in the selected scope
* `-r`, `--recursive` — Also delete all transitive dependencies of specified datasets
* `-f`, `--force` — Do not ask for confirmation
* `--ignore-not-found` — Exit successfully when a selected resource does not exist
* `--dry-run` — Preview the resolved resource deletions without deleting anything
* `-l`, `--label <LABEL_SELECTOR>` — Narrow the selection to resources carrying all of the given labels, as `key=value` pairs (repeatable, or comma-separated)

This command deletes datasets using the legacy dataset path by default, or
resources when a resource target is specified explicitly.

**Examples:**

Delete a local dataset using the legacy default:

kamu delete my.dataset

Delete datasets explicitly:

kamu delete datasets my.dataset

Delete a dataset using the dataset pseudo-type selector:

kamu delete dataset/my.dataset

Delete local datasets matching a pattern:

kamu delete my.dataset.%

Delete a single resource:

kamu delete storages warehouse

Delete same-type resources by name pattern:

kamu delete vs app-%

Delete all resources of a type:

kamu delete storages --all

Delete all resources across types:

kamu delete '%/%'

Delete resources using slash selectors:

kamu delete vs/my-vars ss/my-secrets

Delete the same exact name across every resource type:

kamu delete % db-creds

Delete resources matching a name pattern across every resource type:

kamu delete %/db-%

Delete all resources of a type using a slash selector:

kamu delete 'storages/%'

Delete a dataset and a resource in one command:

kamu delete dataset/my.dataset vs/my-vars

Force dataset interpretation when a dataset account collides with a resource prefix:

kamu delete datasets vs/my.dataset

Preview deletion:

kamu delete dataset/my.dataset vs/my-vars --dry-run

Narrow a deletion to resources carrying a label:

kamu delete variablesets --all -l environment=stale --dry-run

## `kamu export`

Exports a dataset

**Usage:** `kamu export [OPTIONS] --output-format <OUTPUT_FORMAT> <DATASET>`

**Arguments:**

* `<DATASET>` — Local dataset reference

**Options:**

* `--output-path <OUTPUT_PATH>` — Export destination. Dafault is `<current workdir>/<dataset name>`
* `--output-format <OUTPUT_FORMAT>` — Output format
* `--records-per-file <RECORDS_PER_FILE>` — Number of records per file, if stored into a directory. It's a soft limit. For the sake of export performance the actual number of records may be slightly different

This command exports a dataset to a file or set of files of a given format.

Output path may be either file or directory.
When a path contains extention, and no trailing separator, it is considered as a file.
In all other cases a path is considered as a directory. Examples:

* `export/dataset.csv` is a file path
* `export/dataset.csv/` is a directory path
* `export/dataset/` is a directory path
* `export/dataset` is a directory path

## `kamu get`

Returns manifest representation of a resource

**Usage:** `kamu get [OPTIONS] [ARGS]...`

**Arguments:**

* `<ARGS>` — Resource selector(s): `type name...` or `type/name...`

**Options:**

* `-c`, `--context <CONTEXT_NAME>` — Override the current resource context for this invocation

* `-o`, `--output-format <FMT>` — Serialization format of the returned object

  Default value: `yaml`

  Possible values: `json`, `name`, `yaml`

* `--spec` — Return an apply-compatible spec manifest instead of the full resource view

* `--revealed` — Show actual secret values in the output (decrypts encrypted fields). Cannot be used with `-o name`

* `--ignore-not-found` — Exit successfully when the resource does not exist

* `--max-results <N>` — Maximum number of resources resolved by expanding selectors

  Default value: `100`

* `--unbounded` — Disable the limit for expanding selectors

* `-l`, `--label <LABEL_SELECTOR>` — Narrow the selection to resources carrying all of the given labels, as `key=value` pairs (repeatable, or comma-separated)

Returns the current state of one or more resources as YAML or JSON.

Only real resource types supported by the active context are accepted.
Datasets are intentionally not supported by this command.

By default this command returns the full resource view, including status.
Use `--spec` to return the apply-compatible spec manifest instead.

When multiple selectors are provided, the output is wrapped in an `items` list.

**Examples:**

Get a variable set manifest in YAML:

kamu get variablesets my-vars

Get the same resource in JSON:

kamu get vs my-vars -o json

Get the apply-compatible spec manifest:

kamu get vs my-vars --spec

Get multiple resources of the same type:

kamu get variablesets vars-a vars-b

Get same-type resources by name pattern:

kamu get vs app-%

Get multiple resources by slash-separated ref form:

kamu get vs/vars-a ss/db-creds

Get the same exact name across every resource type:

kamu get % db-creds

Get resources matching a name pattern across every resource type:

kamu get %/db-%

Get every resource of every type:

kamu get '%/%'

Get a resource by UUID:

kamu get variablesets 3d8d6d1c-6f7c-4c62-9f4e-7d8295e8fb69

Read a resource from a remote context:

kamu get storages warehouse --context prod

Ignore missing resources:

kamu get secretsets missing-a missing-b --ignore-not-found

Narrow matched resources to those carrying a label:

kamu get 'variablesets/%' -l environment=production

## `kamu ingest`

Adds data to the root dataset according to its push source configuration

**Usage:** `kamu ingest [OPTIONS] <DATASET> [FILE]...`

**Arguments:**

* `<DATASET>` — Local dataset reference
* `<FILE>` — Data file(s) to ingest

**Options:**

* `--source-name <SRC>` — Name of the push source to use for ingestion
* `--event-time <T>` — Event time to be used if data does not contain one
* `--stdin` — Read data from the standard input
* `-r`, `--recursive` — Recursively propagate the updates into all downstream datasets
* `--input-format <FMT>` — Overrides the media type of the data expected by the push source

  Possible values: `csv`, `json`, `ndjson`, `geojson`, `ndgeojson`, `parquet`, `esrishapefile`

**Examples:**

Ingest data from files:

kamu ingest org.example.data path/to/data.csv

Ingest data from standard input (assumes source is defined to use NDJSON):

echo '\{"key": "value1"}\n\{"key": "value2"}' | kamu ingest org.example.data --stdin

Ingest data with format conversion:

echo '\[\{"key": "value1"}, \{"key": "value2"}]' | kamu ingest org.example.data --stdin --input-format json

Ingest data with event time hint:

kamu ingest org.example.data data.json --event-time 2050-01-02T12:00:00Z

## `kamu init`

Initialize an empty workspace in the current directory

**Usage:** `kamu init [OPTIONS]`

**Options:**

* `--exists-ok` — Don't return an error if workspace already exists
* `--pull-images` — Only pull container images and exit

A workspace is where kamu stores all the important information about datasets (metadata) and in some cases raw data.

It is recommended to create one kamu workspace per data science project, grouping all related datasets together.

Initializing a workspace creates a `.kamu` directory contains dataset metadata, data, and all supporting files (configs, known repositories etc.).

## `kamu inspect`

Group of commands for exploring dataset metadata

**Usage:** `kamu inspect <COMMAND>`

**Subcommands:**

* `lineage` — Shows the dependency tree of a dataset
* `query` — Shows the transformations used by a derivative dataset
* `schema` — Shows the dataset schema

## `kamu inspect lineage`

Shows the dependency tree of a dataset

**Usage:** `kamu inspect lineage [OPTIONS] [DATASET]...`

**Arguments:**

* `<DATASET>` — Local dataset reference(s)

**Options:**

* `-o`, `--output-format <FMT>` — Format of the output

  Possible values: `shell`, `dot`, `csv`, `html`

* `-b`, `--browse` — Produce HTML and open it in a browser

Presents the dataset-level lineage that includes current and past dependencies.

**Examples:**

Show lineage of a single dataset:

kamu inspect lineage my.dataset

Show lineage graph of all datasets in a browser:

kamu inspect lineage --browse

Render the lineage graph into a png image (needs graphviz installed):

kamu inspect lineage -o dot | dot -Tpng > depgraph.png

## `kamu inspect query`

Shows the transformations used by a derivative dataset

**Usage:** `kamu inspect query <DATASET>`

**Arguments:**

* `<DATASET>` — Local dataset reference

This command allows you to audit the transformations performed by a derivative dataset and their evolution. Such audit is an important step in validating the trustworthiness of data (see `kamu verify` command).

## `kamu inspect schema`

Shows the dataset schema

**Usage:** `kamu inspect schema [OPTIONS] <DATASET>`

**Arguments:**

* `<DATASET>` — Local dataset reference

**Options:**

* `-o`, `--output-format <FMT>` — Format of the output

  Default value: `odf-yaml`

  Possible values: `arrow-json`, `ddl`, `odf-json`, `odf-yaml`, `parquet`, `parquet-json`

Displays the schema of the dataset. Note that dataset schemas can evolve over time and by default the latest schema will be shown.

**Examples:**

Show logical schema of a dataset in the ODF format:

kamu inspect schema my.dataset

Show physical schema of the underlying Parquet files:

kamu inspect schema my.dataset -o parquet

## `kamu list`

List datasets or resources

**Usage:** `kamu list [OPTIONS] [TARGETS]...`

**Arguments:**

* `<TARGETS>` — Targets to list: `datasets`, `%` (all resource types), a resource selector such as `variablesets`, `vs`, `secretsets`, `ss`, `storages`, or `st`, optionally narrowed as `type/name`, `type/pattern-%` or `type/<id>`. Several resource selectors may be given at once

**Options:**

* `-c`, `--context <CONTEXT_NAME>` — Override the current resource context for this invocation

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

* `-w`, `--wide` — Show more details (repeat for more)

* `--max-results <N>` — Maximum number of results to list

* `--unbounded` — Disable the result limit

* `-l`, `--label <LABEL_SELECTOR>` — Narrow the listing to resources carrying all of the given labels, as `key=value` pairs (repeatable, or comma-separated)

**Examples:**

To see a human-friendly list of datasets in your workspace:

kamu list

To list datasets explicitly:

kamu list datasets

To list all resources across all types:

kamu list '%'

To list variable sets:

kamu list variablesets

To list variable sets whose name matches a pattern:

kamu list 'vs/my-%'

To list one resource by name or ID:

kamu list 'vs/my-vars'
kamu list 3d8d6d1c-6f7c-4c62-9f4e-7d8295e8fb69

To list several resource types at once:

kamu list 'vs/app-%' 'ss/app-%'

To list storages from a specific context:

kamu list storages --context prod

To see more details:

kamu list -w

To list only resources carrying a label:

kamu list variablesets -l environment=production

To require several labels at once:

kamu list variablesets -l environment=production,tier=backend

To get a machine-readable list of all resources:

kamu list '%' -o csv

To get a machine-readable list of datasets:

kamu list -o csv

## `kamu log`

Shows dataset metadata history

**Usage:** `kamu log [OPTIONS] <DATASET>`

**Arguments:**

* `<DATASET>` — Local dataset reference

**Options:**

* `-o`, `--output-format <FMT>` — Format of the output

  Possible values: `shell`, `yaml`

* `-f`, `--filter <FLT>` — Types of events to include

* `--limit <LIMIT>` — Maximum number of blocks to display

  Default value: `500`

Metadata of a dataset contains historical record of everything that ever influenced how data currently looks like.

This includes events such as:

* Data ingestion / transformation
* Change of query
* Change of schema
* Change of source URL or other ingestion steps in a root dataset

Use this command to explore how dataset evolved over time.

**Examples:**

Show brief summaries of individual metadata blocks:

kamu log org.example.data

Show detailed content of all blocks:

kamu log -o yaml org.example.data

Using a filter to inspect blocks containing query changes of a derivative dataset:

kamu log -o yaml --filter source org.example.data

## `kamu login`

Authenticates with a remote ODF server interactively

**Usage:** `kamu login [OPTIONS] [SERVER] [COMMAND]`

**Subcommands:**

* `oauth` — Performs non-interactive login to a remote Kamu server via OAuth provider token
* `password` — Performs non-interactive login to a remote Kamu server via login and password

**Arguments:**

* `<SERVER>` — ODF server URL (defaults to kamu.dev)

**Options:**

* `--user` — Store access token in the user home folder rather than in the workspace
* `--check` — Check whether existing authorization is still valid without triggering a login flow
* `--access-token <ACCESS_TOKEN>` — Provide an existing access token
* `--repo-name <REPO_NAME>` — Repository name which will be used to store in repositories list
* `--skip-add-repo` — Don't automatically add a remote repository for this host

## `kamu login oauth`

Performs non-interactive login to a remote Kamu server via OAuth provider token

**Usage:** `kamu login oauth <PROVIDER> <ACCESS_TOKEN> [SERVER]`

**Arguments:**

* `<PROVIDER>` — Name of the OAuth provider, i.e. 'github'
* `<ACCESS_TOKEN>` — OAuth provider access token
* `<SERVER>` — ODF backend server URL (defaults to kamu.dev)

## `kamu login password`

Performs non-interactive login to a remote Kamu server via login and password

**Usage:** `kamu login password <LOGIN> <PASSWORD> [SERVER]`

**Arguments:**

* `<LOGIN>` — Specify user name
* `<PASSWORD>` — Specify password
* `<SERVER>` — ODF backend server URL (defaults to kamu.dev)

## `kamu logout`

Logs out from a remote Kamu server

**Usage:** `kamu logout [OPTIONS] [SERVER]`

**Arguments:**

* `<SERVER>` — ODF server URL (defaults to kamu.dev)

**Options:**

* `--user` — Drop access token stored in the user home folder rather than in the workspace
* `-a`, `--all` — Log out of all servers

## `kamu new`

Creates a new dataset manifest from a template

**Usage:** `kamu new [OPTIONS] <DATASET_NAME>`

**Arguments:**

* `<DATASET_NAME>` — Name of the new dataset

**Options:**

* `--root` — Create a root dataset
* `--derivative` — Create a derivative dataset

This command will create a dataset manifest from a template allowing you to customize the most relevant parts without having to remember the exact structure of the yaml file.

**Examples:**

Create `org.example.data.yaml` file from template in the current directory:

kamu new org.example.data --root

## `kamu notebook`

Starts the notebook server for exploring the data in the workspace

**Usage:** `kamu notebook [OPTIONS]`

**Options:**

* `--address <ADDRESS>` — Expose HTTP server on specific network interface

* `--http-port <HTTP_PORT>` — Expose HTTP server on specific port

* `--engine <ENG>` — Engine type to use for the notebook

  Possible values: `datafusion`, `spark`

* `-e`, `--env <VAR>` — Propagate or set an environment variable in the notebook (e.g. `-e VAR` or `-e VAR=foo`)

This command will run the Jupyter server and the Spark engine connected together, letting you query data with SQL before pulling it into the notebook for final processing and visualization.

For more information check out notebook examples at [https://github.com/kamu-data/kamu-cli](https://github.com/kamu-data/kamu-cli)

## `kamu pull`

Pull new data into the datasets

**Usage:** `kamu pull [OPTIONS] [DATASET]...`

**Arguments:**

* `<DATASET>` — Local or remote dataset reference(s)

**Options:**

* `-a`, `--all` — Pull all datasets in the workspace
* `-r`, `--recursive` — Also pull all transitive dependencies of specified datasets
* `--fetch-uncacheable` — Pull latest data from uncacheable data sources
* `--as <DATASET_NAME>` — Local name of a dataset to use when syncing from a repository
* `--no-alias` — Don't automatically add a remote push alias for this destination
* `--set-watermark <TIME>` — Injects a manual watermark into the dataset to signify that no data is expected to arrive with event time that precedes it
* `-f`, `--force` — Overwrite local version with remote, even if revisions have diverged
* `--reset-derivatives-on-diverged-input` — Run hard compaction of derivative dataset if transformation failed due to root dataset compaction
* `--visibility <VIS>` — Changing the visibility of the pulled dataset(s)

  Possible values: `private`, `public`

Pull is a multi-functional command that lets you update a local dataset. Depending on the parameters and the types of datasets involved it can be used to:

* Run polling ingest to pull data into a root dataset from an external source
* Run transformations on a derivative dataset to process previously unseen data
* Pull dataset from a remote repository into your workspace
* Update watermark on a dataset

**Examples:**

Fetch latest data in a specific dataset:

kamu pull org.example.data

Fetch latest data in datasets matching pattern:

kamu pull org.example.%

Fetch latest data for the entire dependency tree of a dataset:

kamu pull --recursive org.example.derivative

Refresh data of all datasets in the workspace:

kamu pull --all

Fetch dataset from a registered repository:

kamu pull kamu/org.example.data

Fetch dataset from a URL (see `kamu repo add -h` for supported sources):

kamu pull ipfs\://bafy...a0dx/data
kamu pull s3://my-bucket.example.org/odf/org.example.data
kamu pull s3+[https://example.org:5000/data](https://example.org:5000/data) --as org.example.data

Advance the watermark of a dataset:

kamu pull --set-watermark 2020-01-01 org.example.data

## `kamu push`

Push local data into a repository

**Usage:** `kamu push [OPTIONS] [DATASET]...`

**Arguments:**

* `<DATASET>` — Local or remote dataset reference(s)

**Options:**

* `-a`, `--all` — Push all datasets in the workspace
* `-r`, `--recursive` — Also push all transitive dependencies of specified datasets
* `--no-alias` — Don't automatically add a remote push alias for this destination
* `--to <REM>` — Remote alias or a URL to push to
* `-f`, `--force` — Overwrite remote version with local, even if revisions have diverged
* `--visibility <VIS>` — Changing the visibility of the initially pushed dataset(s)

  Possible values: `private`, `public`

Use this command to share your new dataset or new data with others. All changes performed by this command are atomic and non-destructive. This command will analyze the state of the dataset at the repository and will only upload data and metadata that wasn't previously seen.

Similarly to git, if someone else modified the dataset concurrently with you - your push will be rejected, and you will have to resolve the conflict.

**Examples:**

Sync dataset to a destination URL (see `kamu repo add -h` for supported protocols):

kamu push org.example.data --to s3://my-bucket.example.org/odf/org.example.data

Sync dataset to a named repository (see `kamu repo` command group):

kamu push org.example.data --to kamu-hub/org.example.data

Sync dataset that already has a push alias:

kamu push org.example.data

Sync datasets matching pattern that already have push aliases:

kamu push org.example.%

Add dataset to local IPFS node and update IPNS entry to the new CID:

kamu push org.example.data --to ipns\://k5..zy

## `kamu rename`

Rename a dataset

**Usage:** `kamu rename <DATASET> <NEW_NAME>`

**Arguments:**

* `<DATASET>` — Dataset reference
* `<NEW_NAME>` — The new name to give it

Use this command to rename a dataset in your local workspace. Renaming is safe in terms of downstream derivative datasets as they use stable dataset IDs to define their inputs.

**Examples:**

Renaming is often useful when you pull a remote dataset by URL, and it gets auto-assigned not the most convenient name:

kamu pull ipfs\://bafy...a0da
kamu rename bafy...a0da my.dataset

## `kamu reset`

Revert the dataset back to the specified state

**Usage:** `kamu reset <DATASET> <HASH>`

**Arguments:**

* `<DATASET>` — Dataset reference
* `<HASH>` — Hash of the block to reset to

Resetting a dataset to the specified block erases all metadata blocks that followed it and deletes all data added since that point. This can sometimes be useful to resolve conflicts, but otherwise should be used with care.

Keep in mind that blocks that were pushed to a repository could've been already observed by other people, so resetting the history will not let you take that data back and instead create conflicts for the downstream consumers of your data.

## `kamu repo`

Manage set of tracked repositories

**Usage:** `kamu repo <COMMAND>`

**Subcommands:**

* `add` — Adds a repository
* `delete [rm]` — Deletes a reference to repository
* `list [ls]` — Lists known repositories
* `alias` — Manage set of remote aliases associated with datasets

Repositories are nodes on the network that let users exchange datasets. In the most basic form, a repository can simply be a location where the dataset files are hosted over one of the supported file or object-based data transfer protocols. The owner of a dataset will have push privileges to this location, while other participants can pull data from it.

**Examples:**

Show available repositories:

kamu repo list

Add S3 bucket as a repository:

kamu repo add example-repo s3://bucket.my-company.example/

## `kamu repo add`

Adds a repository

**Usage:** `kamu repo add <REPO_NAME> <URL>`

**Arguments:**

* `<REPO_NAME>` — Local alias of the repository
* `<URL>` — URL of the repository

For local file system repositories use the following URL formats:

file:///home/me/example/repository/
file:///c:/Users/me/example/repository/

For S3-compatible basic repositories use:

s3://bucket.my-company.example/
s3+[http://my-server:9000/bucket/](http://my-server:9000/bucket/)
s3+[https://my-server:9000/bucket/](https://my-server:9000/bucket/)

For ODF-compatible smart repositories use:

odf+[http://odf-server/](http://odf-server/)
odf+[https://odf-server/](https://odf-server/)

## `kamu repo delete`

Deletes a reference to repository

**Usage:** `kamu repo delete [OPTIONS] [REPOSITORY]...`

**Arguments:**

* `<REPOSITORY>` — Repository name(s)

**Options:**

* `-a`, `--all` — Delete all known repositories

## `kamu repo list`

Lists known repositories

**Usage:** `kamu repo list [OPTIONS]`

**Options:**

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

## `kamu repo alias`

Manage set of remote aliases associated with datasets

**Usage:** `kamu repo alias <COMMAND>`

**Subcommands:**

* `add` — Adds a remote alias to a dataset
* `delete [rm]` — Deletes a remote alias associated with a dataset
* `list [ls]` — Lists remote aliases

When you pull and push datasets from repositories kamu uses aliases to let you avoid specifying the full remote reference each time. Aliases are usually created the first time you do a push or pull and saved for later. If you have an unusual setup (e.g. pushing to multiple repositories) you can use this command to manage the aliases.

**Examples:**

List all aliases:

kamu repo alias list

List all aliases of a specific dataset:

kamu repo alias list org.example.data

Add a new pull alias:

kamu repo alias add --pull org.example.data kamu.dev/me/org.example.data

## `kamu repo alias add`

Adds a remote alias to a dataset

**Usage:** `kamu repo alias add [OPTIONS] <DATASET> <ALIAS>`

**Arguments:**

* `<DATASET>` — Local dataset reference
* `<ALIAS>` — Remote dataset name

**Options:**

* `--push` — Add a push alias
* `--pull` — Add a pull alias

## `kamu repo alias delete`

Deletes a remote alias associated with a dataset

**Usage:** `kamu repo alias delete [OPTIONS] [DATASET] [ALIAS]`

**Arguments:**

* `<DATASET>` — Local dataset reference
* `<ALIAS>` — Remote dataset name

**Options:**

* `-a`, `--all` — Delete all aliases
* `--push` — Delete a push alias
* `--pull` — Delete a pull alias

## `kamu repo alias list`

Lists remote aliases

**Usage:** `kamu repo alias list [OPTIONS] [DATASET]`

**Arguments:**

* `<DATASET>` — Local dataset reference

**Options:**

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

## `kamu search`

Searches for datasets in the registered repositories

**Usage:** `kamu search [OPTIONS] [QUERY]`

**Arguments:**

* `<QUERY>` — Search terms

**Options:**

* `-l`, `--local` — Search local datasets instead of searching in remote repositories

* `-n`, `--max-results <MAX_RESULTS>` — Maximum results to fetch

  Default value: `10`

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

* `--repo <REPO>` — Repository name(s) to search in

Search is delegated to the repository implementations and its capabilities depend on the type of the repo. Whereas smart repos may support advanced full-text search, simple storage-only repos may be limited to a substring search by dataset name.

**Examples:**

Search all repositories:

kamu search covid19

Search only specific repositories:

kamu search covid19 --repo kamu --repo statcan.gc.ca

## `kamu sql`

Executes an SQL query or drops you into an SQL shell

**Usage:** `kamu sql [OPTIONS] [COMMAND]`

**Subcommands:**

* `server` — Runs an SQL engine in a server mode

**Options:**

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

* `--engine <ENG>` — Engine type to use for this SQL session

  Possible values: `datafusion`, `spark`

* `--url <URL>` — URL of a running JDBC server (e.g. jdbc:hive2://example.com:10000)

* `-c`, `--command <CMD>` — SQL command to run

* `--script <FILE>` — SQL script file to execute

* `--output-path <OUTPUT_PATH>` — When set, result will be stored to a given path instead of being printed to stdout

* `--records-per-file <RECORDS_PER_FILE>` — Number of records per file, if stored into a directory. It's a soft limit. For the sake of export performance the actual number records may be slightly different

SQL shell allows you to explore data of all dataset in your workspace using one of the supported data processing engines. This can be a great way to prepare and test a query that you cal later turn into derivative dataset.

Output path may be either file or directory.
When a path contains extention, and no trailing separator, it is considered as a file.
In all other cases a path is considered as a directory. Examples:

* `export/dataset.csv` is a file path
* `export/dataset.csv/` is a directory path
* `export/dataset/` is a directory path
* `export/dataset` is a directory path

**Examples:**

Drop into SQL shell:

kamu sql

Execute SQL command and return its output in CSV format:

kamu sql -c 'SELECT \* FROM `org.example.data` LIMIT 10' -o csv

Run SQL server to use with external data processing tools:

kamu sql server --address 0.0.0.0 --port 8080

Connect to a remote SQL server:

kamu sql --url jdbc:hive2://example.com:10000

Note: Currently when connecting to a remote SQL kamu server you will need to manually instruct it to load datasets from the data files. This can be done using the following command:

CREATE TEMP VIEW `my.dataset` AS (SELECT \* FROM parquet.`kamu_data/my.dataset`);

## `kamu sql server`

Runs an SQL engine in a server mode

**Usage:** `kamu sql server [OPTIONS]`

**Options:**

* `--address <ADDRESS>` — Expose server on specific network interface

* `--port <PORT>` — Expose server on specific port

* `--engine <ENG>` — Engine type to use for this server

  Possible values: `datafusion`, `spark`

* `--livy` — Run Livy server instead of JDBC

**Examples:**

By default runs the DataFusion engine exposing the FlightSQL protocol:

kamu sql server

To customize interface and port:

kamu sql server --address 0.0.0.0 --port 50050

To run with Spark engine:

kamu sql server --engine spark

By default Spark runs with JDBC protocol, to instead run with Livy HTTP gateway:

kamu sql server --engine spark --livy

## `kamu summary`

Show resource summary for the active context

**Usage:** `kamu summary [OPTIONS]`

**Options:**

* `-c`, `--context <CONTEXT_NAME>` — Override the current resource context for this invocation
* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values: `table`, `json`, `yaml`

Shows resource counts by type and reconciliation phase for the active context.

If the active context is `local`, commands target the current workspace. If the
active context points to a remote server, commands target that remote GraphQL
API.

Use `--context` to override the current context for this invocation only.

**Examples:**

Show summary from the active context:

kamu summary

Show summary from a specific context:

kamu summary --context prod

Show summary in YAML:

kamu summary -o yaml

## `kamu system`

Command group for system-level functionality

**Usage:** `kamu system <COMMAND>`

**Subcommands:**

* `api-server` — Run HTTP + GraphQL server
* `compact` — Compact a dataset
* `debug-token` — Validate a Kamu token
* `depgraph` — Validate a Kamu token
* `decode` — Decode a manifest file
* `diagnose` — Run basic system diagnose check
* `generate-token` — Generate a platform token from a known secret for debugging
* `gc` — Runs garbage collection to clean up cached and unreachable objects in the workspace
* `info` — Summary of the system information
* `ipfs` — IPFS helpers
* `upgrade-workspace` — Upgrade the layout of a local workspace to the latest version

## `kamu system api-server`

Run HTTP + GraphQL server

**Usage:** `kamu system api-server [OPTIONS] [COMMAND]`

**Subcommands:**

* `gql-query` — Executes the GraphQL query and prints out the result
* `gql-schema` — Prints the GraphQL schema

**Options:**

* `--address <ADDRESS>` — Bind to a specific network interface
* `--http-port <HTTP_PORT>` — Expose HTTP+GraphQL server on specific port
* `--get-token` — Output a JWT token you can use to authorize API queries
* `--external-address <EXTERNAL_ADDRESS>` — Allows changing the base URL used in the API. Can be handy when launching inside a container

**Examples:**

Run API server on a specified port:

kamu system api-server --http-port 12312

Execute a single GraphQL query and print result to stdout:

kamu system api-server gql-query '\{ apiVersion }'

Print out GraphQL API schema:

kamu system api-server gql-schema

## `kamu system api-server gql-query`

Executes the GraphQL query and prints out the result

**Usage:** `kamu system api-server gql-query [OPTIONS] <QUERY>`

**Arguments:**

* `<QUERY>` — GQL query

**Options:**

* `--full` — Display the full result including extensions

## `kamu system api-server gql-schema`

Prints the GraphQL schema

**Usage:** `kamu system api-server gql-schema`

## `kamu system compact`

Compact a dataset

**Usage:** `kamu system compact [OPTIONS] [DATASET]...`

**Arguments:**

* `<DATASET>` — Local dataset references

**Options:**

* `--max-slice-size <SIZE>` — Maximum size of a single data slice file in bytes

  Default value: `300000000`
* `--max-slice-records <RECORDS>` — Maximum amount of records in a single data slice file

  Default value: `10000`
* `--hard` — Perform 'hard' compaction that rewrites the history of a dataset
* `--keep-metadata-only` — Perform compaction without saving data blocks
* `--verify` — Perform verification of the dataset before running a compaction

For datasets that get frequent small appends the number of data slices can grow over time and affect the performance of querying. This command allows to merge multiple small data slices into a few large files, which can be beneficial in terms of size from more compact encoding, and in query performance, as data engines will have to scan through far fewer file headers.

There are two types of compactions: soft and hard.

Soft compactions produce new files while leaving the old blocks intact. This allows for faster queries, while still preserving the accurate history of how dataset evolved over time.

Hard compactions rewrite the history of the dataset as if data was originally written in big batches. They allow to shrink the history of a dataset to just a few blocks, reclaim the space used by old data files, but at the expense of history loss. Hard compactions will rewrite the metadata chain, changing block hashes. Therefore, they will **break all downstream datasets** that depend on them.

**Examples:**

Perform a history-altering hard compaction:

kamu system compact --hard my.dataset

## `kamu system debug-token`

Validate a Kamu token

**Usage:** `kamu system debug-token <TOKEN>`

**Arguments:**

* `<TOKEN>` — Access token

## `kamu system depgraph`

Validate a Kamu token

**Usage:** `kamu system depgraph`

## `kamu system decode`

Decode a manifest file

**Usage:** `kamu system decode [OPTIONS] [MANIFEST]`

**Arguments:**

* `<MANIFEST>` — Manifest reference (path, or URL)

**Options:**

* `--stdin` — Read manifests from standard input

## `kamu system diagnose`

Run basic system diagnose check

**Usage:** `kamu system diagnose`

## `kamu system generate-token`

Generate a platform token from a known secret for debugging

**Usage:** `kamu system generate-token [OPTIONS]`

**Options:**

* `--subject <SUBJECT>` — Account ID to generate token for
* `--login <LOGIN>` — Account name to derive ID from (for predefined accounts only)
* `--expiration-time-sec <EXPIRATION_TIME_SEC>` — Token expiration time in seconds

  Default value: `3600`

## `kamu system gc`

Runs garbage collection to clean up cached and unreachable objects in the workspace

**Usage:** `kamu system gc`

## `kamu system info`

Summary of the system information

**Usage:** `kamu system info [OPTIONS]`

**Options:**

* `-o`, `--output-format <FMT>` — Format of the output

  Possible values: `shell`, `json`, `yaml`

## `kamu system ipfs`

IPFS helpers

**Usage:** `kamu system ipfs <COMMAND>`

**Subcommands:**

* `add` — Adds the specified dataset to IPFS and returns the CID

## `kamu system ipfs add`

Adds the specified dataset to IPFS and returns the CID

**Usage:** `kamu system ipfs add <DATASET>`

**Arguments:**

* `<DATASET>` — Dataset reference

## `kamu system upgrade-workspace`

Upgrade the layout of a local workspace to the latest version

**Usage:** `kamu system upgrade-workspace`

## `kamu tail`

Displays a sample of most recent records in a dataset

**Usage:** `kamu tail [OPTIONS] <DATASET>`

**Arguments:**

* `<DATASET>` — Local dataset reference

**Options:**

* `-o`, `--output-format <FMT>` — Format to display the results in

  Possible values:

  * `csv`:
    Comma-separated values
  * `json`:
    Array of Structures format
  * `ndjson`:
    One Json object per line - easily splittable format
  * `json-soa`:
    Structure of arrays - more compact and efficient format for encoding entire dataframe
  * `json-aoa`:
    Array of arrays - compact and efficient and preserves column order
  * `table`:
    A pretty human-readable table
  * `parquet`:
    Parquet columnar storage. Only available when exporting to file(s)

* `-n`, `--num-records <NUM>` — Number of records to display

  Default value: `10`

* `-s`, `--skip-records <SKP>` — Number of initial records to skip before applying the limit

  Default value: `0`

This command can be thought of as a shortcut for:

kamu sql --engine datafusion --command 'select \* from "\{dataset}" order by \{offset\_col} desc limit \{num\_records}'

## `kamu ui`

Opens web interface

**Usage:** `kamu ui [OPTIONS]`

**Options:**

* `--address <ADDRESS>` — Expose HTTP server on specific network interface
* `--http-port <HTTP_PORT>` — Which port to run HTTP server on
* `--get-token` — Output a JWT token you can use to authorize API queries

Starts a built-in HTTP + GraphQL server and opens a pre-packaged Web UI application in your browser.

**Examples:**

Starts server and opens UI in your default browser:

kamu ui

Start server on a specific port:

kamu ui --http-port 12345

## `kamu verify`

Verifies the validity of a dataset

**Usage:** `kamu verify [OPTIONS] [DATASET]...`

**Arguments:**

* `<DATASET>` — Local dataset reference(s)

**Options:**

* `-r`, `--recursive` — Verify the entire transformation chain starting with root datasets
* `--integrity` — Check only the hashes of metadata and data without replaying transformations

Validity of derivative data is determined by:

* Trustworthiness of the source data that went into it
* Soundness of the derivative transformation chain that shaped it
* Guaranteeing that derivative data was in fact produced by declared transformations

For the first two, you can inspect the dataset lineage so see which root datasets the data is coming from and whether their publishers are credible. Then you can audit all derivative transformations to ensure they are sound and non-malicious.

This command can help you with the last stage. It uses the history of transformations stored in metadata to first compare the hashes of data with ones stored in metadata (i.e. verify that data corresponds to metadata). Then it repeats all declared transformations locally to ensure that what's declared in metadata actually produces the presented result.

The combination of the above steps can give you a high certainty that the data you're using is trustworthy.

When called on a root dataset the command will only perform the integrity check of comparing data hashes to metadata.

**Examples:**

Verify the data in a dataset starting from its immediate inputs:

kamu verify com.example.deriv

Verify the data in datasets matching pattern:

kamu verify com.example.%

Verify the entire transformation chain starting with root datasets (may download a lot of data):

kamu pull --recursive com.example.deriv

Verify only the hashes of metadata and data, without replaying the transformations. This is useful when you trust the peers performing transformations but want to ensure data was not tampered in storage or during the transmission:

kamu verify --integrity com.example.deriv

## `kamu version`

Outputs build information

**Usage:** `kamu version [OPTIONS]`

**Options:**

* `-o`, `--output-format <FMT>` — Format of the output

  Possible values: `shell`, `json`, `yaml`


# Config Reference
Source: https://docs.kamu.dev/cli/config



## `CLIConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>auth</code></td>
      <td><a href="#authconfig"><code>AuthConfig</code></a></td>
      <td><pre><code>\{
      "allowAnonymous": true,
      "users": \{
      "predefined": \[]
      }
      }</code></pre></td>
      <td>Auth configuration</td>
    </tr>

    <tr>
      <td><code>backgroundAgents</code></td>
      <td><a href="#backgroundagentsconfig"><code>BackgroundAgentsConfig</code></a></td>
      <td><pre><code>\{
      "batching": \{
      "flowActivations": 20,
      "flowSystemEvents": 20,
      "outboxMessages": 20
      },
      "concurrency": \{
      "outboxConsumers": 1
      },
      "maxListeningTimeout": "2s",
      "minDebounceInterval": "20ms"
      }</code></pre></td>
      <td>Background agents configuration (outbox, flow system events, tasks)</td>
    </tr>

    <tr>
      <td><code>database</code></td>
      <td><a href="#databaseconfig"><code>DatabaseConfig</code></a></td>
      <td><code>null</code></td>
      <td>Database connection configuration</td>
    </tr>

    <tr>
      <td><code>didEncryption</code></td>
      <td><a href="#didsecretencryptionconfig"><code>DidSecretEncryptionConfig</code></a></td>
      <td><pre><code>\{
      "enabled": false
      }</code></pre></td>
      <td>Did secret key encryption configuration</td>
    </tr>

    <tr>
      <td><code>engine</code></td>
      <td><a href="#engineconfig"><code>EngineConfig</code></a></td>
      <td><pre><code>\{
      "datafusionEmbedded": \{
      "base": \{
      "datafusion.catalog.default\_catalog": "kamu",
      "datafusion.catalog.default\_schema": "kamu",
      "datafusion.catalog.information\_schema": "true",
      "datafusion.sql\_parser.enable\_ident\_normalization": "false"
      },
      "batchQuery": \{},
      "compaction": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "ingest": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "useLegacyArrowBufferEncoding": false
      },
      "images": \{
      "datafusion": "ghcr.io/kamu-data/engine-datafusion:0.9.0",
      "flink": "ghcr.io/kamu-data/engine-flink:0.18.2-flink\_1.16.0-scala\_2.12-java8",
      "risingwave": "ghcr.io/kamu-data/engine-risingwave:0.3.0",
      "spark": "ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"
      },
      "networkNs": "Private",
      "runtime": "Docker",
      "shutdownTimeout": "5s",
      "startTimeout": "30s"
      }</code></pre></td>
      <td>Engine configuration</td>
    </tr>

    <tr>
      <td><code>extra</code></td>
      <td><a href="#extraconfig"><code>ExtraConfig</code></a></td>
      <td><pre><code>\{
      "graphql": \{}
      }</code></pre></td>
      <td>Experimental and temporary configuration options</td>
    </tr>

    <tr>
      <td><code>flowSystem</code></td>
      <td><a href="#flowsystemconfig"><code>FlowSystemConfig</code></a></td>
      <td><pre><code>\{
      "awaitingStepSecs": 1,
      "defaultRetryPolicies": \{},
      "mandatoryThrottlingPeriodSecs": 60
      }</code></pre></td>
      <td>Configuration for flow system</td>
    </tr>

    <tr>
      <td><code>frontend</code></td>
      <td><a href="#frontendconfig"><code>FrontendConfig</code></a></td>
      <td><pre><code>\{
      "jupyter": \{
      "image": "ghcr.io/kamu-data/jupyter:0.7.1",
      "livyImage": "ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"
      }
      }</code></pre></td>
      <td>Data access and visualization configuration</td>
    </tr>

    <tr>
      <td><code>identity</code></td>
      <td><a href="#identityconfig"><code>IdentityConfig</code></a></td>
      <td><code>\{}</code></td>
      <td>UNSTABLE: Identity configuration</td>
    </tr>

    <tr>
      <td><code>protocol</code></td>
      <td><a href="#protocolconfig"><code>ProtocolConfig</code></a></td>
      <td><pre><code>\{
      "flightSql": \{
      "allowAnonymous": true,
      "anonSessionExpirationTimeout": "30m",
      "anonSessionInactivityTimeout": "5s",
      "authedSessionExpirationTimeout": "30m",
      "authedSessionInactivityTimeout": "5s"
      },
      "ipfs": \{
      "httpGateway": "http\://localhost:8080/",
      "preResolveDnslink": true
      }
      }</code></pre></td>
      <td>Network protocols configuration</td>
    </tr>

    <tr>
      <td><code>quotaDefaults</code></td>
      <td><a href="#quotadefaults"><code>QuotaDefaults</code></a></td>
      <td><pre><code>\{
      "storage": 1000000000
      }</code></pre></td>
      <td>Default quotas configured by type</td>
    </tr>

    <tr>
      <td><code>search</code></td>
      <td><a href="#searchconfig"><code>SearchConfig</code></a></td>
      <td><pre><code>\{
      "embeddingsChunker": \{
      "kind": "Simple",
      "splitParagraphs": false,
      "splitSections": false
      },
      "embeddingsEncoder": \{
      "kind": "Dummy"
      },
      "indexer": \{
      "clearOnStart": false,
      "incrementalIndexing": true
      },
      "repo": \{
      "kind": "Dummy"
      }
      }</code></pre></td>
      <td>Search configuration</td>
    </tr>

    <tr>
      <td><code>secretsEncryption</code></td>
      <td><a href="#secretsencryptionconfig"><code>SecretsEncryptionConfig</code></a></td>
      <td><pre><code>\{
      "enabled": false
      }</code></pre></td>
      <td>Secrets encryption configuration</td>
    </tr>

    <tr>
      <td><code>source</code></td>
      <td><a href="#sourceconfig"><code>SourceConfig</code></a></td>
      <td><pre><code>\{
      "ethereum": \{
      "commitAfterBlocksScanned": 1000000,
      "getLogsBlockStride": 100000,
      "rpcEndpoints": \[],
      "useBlockTimestampFallback": false
      },
      "http": \{
      "connectTimeout": "30s",
      "maxRedirects": 10,
      "userAgent": "kamu-cli/0.270.0"
      },
      "mqtt": \{
      "brokerIdleTimeout": "1s"
      },
      "targetRecordsPerSlice": 10000
      }</code></pre></td>
      <td>Source configuration</td>
    </tr>

    <tr>
      <td><code>uploads</code></td>
      <td><a href="#uploadsconfig"><code>UploadsConfig</code></a></td>
      <td><pre><code>\{
      "maxFileSizeInMb": 50
      }</code></pre></td>
      <td>Uploads configuration</td>
    </tr>

    <tr>
      <td><code>webhooks</code></td>
      <td><a href="#webhooksconfig"><code>WebhooksConfig</code></a></td>
      <td><pre><code>\{
      "deliveryTimeout": 10,
      "maxConsecutiveFailures": 5,
      "secretEncryptionEnabled": false
      }</code></pre></td>
      <td>Configuration for webhooks</td>
    </tr>
  </tbody>
</table>

## `AccountConfig`

The declarative account configuration used to register an account if one
does not already exist.

To update an existing account, either `id` or `private_key` must be
specified.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>accountName</code></td>
      <td><a href="#accountname"><code>AccountName</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>accountType</code></td>
      <td><a href="#accounttype"><code>AccountType</code></a></td>
      <td><code>"User"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>avatarUrl</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>displayName</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Auto-derived from `account_name` if omitted
      </td>
    </tr>

    <tr>
      <td><code>email</code></td>
      <td><a href="#email"><code>Email</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>id</code></td>
      <td><a href="#accountid"><code>AccountID</code></a></td>
      <td><code>null</code></td>

      <td>
        May be omitted in favor of `private_key`.
      </td>
    </tr>

    <tr>
      <td><code>password</code></td>
      <td><a href="#password"><code>Password</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>privateKey</code></td>
      <td><a href="#privatekey"><code>PrivateKey</code></a></td>
      <td><code>null</code></td>

      <td>
        Optional ed25519 private key. When set, `id` is derived from it
        (and must match `id` if both are present).
      </td>
    </tr>

    <tr>
      <td><code>properties</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>

      <td />
    </tr>

    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>
      <td><code>"password"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>providerIdentityKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Auto-derived from `account_name` if omitted
      </td>
    </tr>

    <tr>
      <td><code>registeredAt</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>treatDatasetsAsPublic</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `AccountID`

Base type: `string`

## `AccountName`

Base type: `string`

## `AccountPropertyName`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>CanProvisionAccounts</code></td></tr>
    <tr><td><code>Admin</code></td></tr>
  </tbody>
</table>

## `AccountType`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>User</code></td></tr>
    <tr><td><code>Organization</code></td></tr>
  </tbody>
</table>

## `AuthConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>allowAnonymous</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>

      <td />
    </tr>

    <tr>
      <td><code>users</code></td>
      <td><a href="#predefinedaccountsconfig"><code>PredefinedAccountsConfig</code></a></td>
      <td><pre><code>\{
      "predefined": \[]
      }</code></pre></td>

      <td />
    </tr>
  </tbody>
</table>

## `BackgroundAgentsBatchingConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>flowActivations</code></td>
      <td><code>integer</code></td>
      <td><code>20</code></td>

      <td>
        Due flows the flow agent loads at once before activating them.
        0 is treated as 1
      </td>
    </tr>

    <tr>
      <td><code>flowSystemEvents</code></td>
      <td><code>integer</code></td>
      <td><code>20</code></td>

      <td>
        Flow system events applied to a projection per transaction. 0 is
        treated as 1
      </td>
    </tr>

    <tr>
      <td><code>outboxMessages</code></td>
      <td><code>integer</code></td>
      <td><code>20</code></td>
      <td>Outbox messages relayed per transaction. 0 is treated as 1</td>
    </tr>
  </tbody>
</table>

## `BackgroundAgentsConcurrencyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>outboxConsumers</code></td>
      <td><code>integer</code></td>
      <td><code>1</code></td>

      <td>
        Outbox consumers running at once, in order per producer, each holding a
        pooled connection: e.g. 8 with Postgres, 1 with `SQLite`. 0 is treated
        as 1
      </td>
    </tr>
  </tbody>
</table>

## `BackgroundAgentsConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>batching</code></td>
      <td><a href="#backgroundagentsbatchingconfig"><code>BackgroundAgentsBatchingConfig</code></a></td>
      <td><pre><code>\{
      "flowActivations": 20,
      "flowSystemEvents": 20,
      "outboxMessages": 20
      }</code></pre></td>
      <td>Batch sizes of agents processing records in batches</td>
    </tr>

    <tr>
      <td><code>concurrency</code></td>
      <td><a href="#backgroundagentsconcurrencyconfig"><code>BackgroundAgentsConcurrencyConfig</code></a></td>
      <td><pre><code>\{
      "outboxConsumers": 1
      }</code></pre></td>
      <td>Concurrency limits of agents processing records in parallel</td>
    </tr>

    <tr>
      <td><code>maxListeningTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"2s"</code></td>

      <td>
        Fallback period to re-check for work if a change signal is missed.
        With `SQLite` it also paces the polling. The flow agent wakes up at
        the next flow activation moment regardless of it.
      </td>
    </tr>

    <tr>
      <td><code>minDebounceInterval</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"20ms"</code></td>
      <td>How long agents absorb a burst of change signals before processing</td>
    </tr>
  </tbody>
</table>

## `ContainerRuntimeType`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>Docker</code></td></tr>
    <tr><td><code>Podman</code></td></tr>
  </tbody>
</table>

## `DatabaseConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#databaseconfigsqlite"><code>Sqlite</code></a></td></tr>
    <tr><td><a href="#databaseconfigpostgres"><code>Postgres</code></a></td></tr>
  </tbody>
</table>

## `DatabaseConfig::Sqlite`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>databasePath</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseConfig::Postgres`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>acquireTimeoutSecs</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>credentialsPolicy</code></td>
      <td><a href="#databasecredentialspolicyconfig"><code>DatabaseCredentialsPolicyConfig</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>databaseName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>host</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>maxConnections</code></td>
      <td><code>integer</code></td>
      <td><code>20</code></td>

      <td />
    </tr>

    <tr>
      <td><code>maxLifetimeSecs</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>port</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseCredentialSourceConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#databasecredentialsourceconfigrawpassword"><code>RawPassword</code></a></td></tr>
    <tr><td><a href="#databasecredentialsourceconfigawssecret"><code>AwsSecret</code></a></td></tr>
  </tbody>
</table>

## `DatabaseCredentialSourceConfig::RawPassword`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>rawPassword</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>userName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseCredentialSourceConfig::AwsSecret`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>secretName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseCredentialsPolicyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>rotationFrequencyInMinutes</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>source</code></td>
      <td><a href="#databasecredentialsourceconfig"><code>DatabaseCredentialSourceConfig</code></a></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DidSecretEncryptionConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>enabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>encryptionKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        The encryption key must be a 32-character alphanumeric string, which
        includes both uppercase and lowercase Latin letters (A-Z, a-z) and
        digits (0-9).

        To generate, use:

        ```sh theme={null}
        tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo
        ```
      </td>
    </tr>
  </tbody>
</table>

## `DurationString`

Base type: `string`

## `Email`

Base type: `string`

## `EmbeddingsChunkerConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#embeddingschunkerconfigsimple"><code>Simple</code></a></td></tr>
  </tbody>
</table>

## `EmbeddingsChunkerConfig::Simple`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>splitParagraphs</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>splitSections</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `EmbeddingsEncoderConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#embeddingsencoderconfigdummy"><code>Dummy</code></a></td></tr>
    <tr><td><a href="#embeddingsencoderconfigopenai"><code>OpenAi</code></a></td></tr>
  </tbody>
</table>

## `EmbeddingsEncoderConfig::Dummy`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `EmbeddingsEncoderConfig::OpenAi`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>apiKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>dimensions</code></td>
      <td><code>integer</code></td>
      <td><code>1536</code></td>

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>modelName</code></td>
      <td><code>string</code></td>
      <td><code>"text-embedding-ada-002"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>url</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `EngineConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>datafusionEmbedded</code></td>
      <td><a href="#engineconfigdatafusion"><code>EngineConfigDatafusion</code></a></td>
      <td><pre><code>\{
      "base": \{
      "datafusion.catalog.default\_catalog": "kamu",
      "datafusion.catalog.default\_schema": "kamu",
      "datafusion.catalog.information\_schema": "true",
      "datafusion.sql\_parser.enable\_ident\_normalization": "false"
      },
      "batchQuery": \{},
      "compaction": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "ingest": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "useLegacyArrowBufferEncoding": false
      }</code></pre></td>
      <td>Embedded Datafusion engine configuration</td>
    </tr>

    <tr>
      <td><code>images</code></td>
      <td><a href="#engineimagesconfig"><code>EngineImagesConfig</code></a></td>
      <td><pre><code>\{
      "datafusion": "ghcr.io/kamu-data/engine-datafusion:0.9.0",
      "flink": "ghcr.io/kamu-data/engine-flink:0.18.2-flink\_1.16.0-scala\_2.12-java8",
      "risingwave": "ghcr.io/kamu-data/engine-risingwave:0.3.0",
      "spark": "ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"
      }</code></pre></td>
      <td>UNSTABLE: Default engine images</td>
    </tr>

    <tr>
      <td><code>maxConcurrency</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>
      <td>Maximum number of engine operations that can be performed concurrently</td>
    </tr>

    <tr>
      <td><code>networkNs</code></td>
      <td><a href="#networknamespacetype"><code>NetworkNamespaceType</code></a></td>
      <td><code>"Private"</code></td>

      <td>
        Type of the networking namespace (relevant when running in container
        environments)
      </td>
    </tr>

    <tr>
      <td><code>runtime</code></td>
      <td><a href="#containerruntimetype"><code>ContainerRuntimeType</code></a></td>
      <td><code>"Docker"</code></td>
      <td>Type of the runtime to use when running the data processing engines</td>
    </tr>

    <tr>
      <td><code>shutdownTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5s"</code></td>
      <td>Timeout for waiting the engine container to stop gracefully</td>
    </tr>

    <tr>
      <td><code>startTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30s"</code></td>
      <td>Timeout for starting an engine container</td>
    </tr>
  </tbody>
</table>

## `EngineConfigDatafusion`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>base</code></td>
      <td><code>object</code></td>
      <td><pre><code>\{
      "datafusion.catalog.default\_catalog": "kamu",
      "datafusion.catalog.default\_schema": "kamu",
      "datafusion.catalog.information\_schema": "true",
      "datafusion.sql\_parser.enable\_ident\_normalization": "false"
      }</code></pre></td>

      <td>
        Base configuration options
        See: `<https://datafusion.apache.org/user-guide/configs.html>`
      </td>
    </tr>

    <tr>
      <td><code>batchQuery</code></td>
      <td><code>object</code></td>
      <td><code>\{}</code></td>
      <td>Batch query-specific overrides to the base config</td>
    </tr>

    <tr>
      <td><code>compaction</code></td>
      <td><code>object</code></td>
      <td><pre><code>\{
      "datafusion.execution.target\_partitions": "1"
      }</code></pre></td>
      <td>Compaction-specific overrides to the base config</td>
    </tr>

    <tr>
      <td><code>ingest</code></td>
      <td><code>object</code></td>
      <td><pre><code>\{
      "datafusion.execution.target\_partitions": "1"
      }</code></pre></td>
      <td>Ingest-specific overrides to the base config</td>
    </tr>

    <tr>
      <td><code>useLegacyArrowBufferEncoding</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td>
        Makes arrow batches use contiguous `Binary` and `Utf8` encodings instead
        of more modern `BinaryView` and `Utf8View`. This is only needed for
        compatibility with some older libraries that don't yet support them.

        See: [kamu-node#277](https://github.com/kamu-data/kamu-node/issues/277)
      </td>
    </tr>
  </tbody>
</table>

## `EngineImagesConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>datafusion</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-datafusion:0.9.0"</code></td>

      <td>
        UNSTABLE: `Datafusion` engine image
      </td>
    </tr>

    <tr>
      <td><code>flink</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-flink:0.18.2-flink\_1.16.0-scala\_2.12-java8"</code></td>

      <td>
        UNSTABLE: `Flink` engine image
      </td>
    </tr>

    <tr>
      <td><code>risingwave</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-risingwave:0.3.0"</code></td>

      <td>
        UNSTABLE: `RisingWave` engine image
      </td>
    </tr>

    <tr>
      <td><code>spark</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"</code></td>

      <td>
        UNSTABLE: `Spark` engine image
      </td>
    </tr>
  </tbody>
</table>

## `EthRpcEndpoint`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>chainId</code></td>
      <td><code>integer</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>chainName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>nodeUrl</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `EthereumSourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>commitAfterBlocksScanned</code></td>
      <td><code>integer</code></td>
      <td><code>1000000</code></td>

      <td>
        Forces iteration to stop after the specified number of blocks were
        scanned even if we didn't reach the target record number. This is useful
        to not lose a lot of scanning progress in case of an RPC error.
      </td>
    </tr>

    <tr>
      <td><code>getLogsBlockStride</code></td>
      <td><code>integer</code></td>
      <td><code>100000</code></td>

      <td>
        Default number of blocks to scan within one query to `eth_getLogs` RPC
        endpoint.
      </td>
    </tr>

    <tr>
      <td><code>rpcEndpoints</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>
      <td>Default RPC endpoints to use if source does not specify one explicitly.</td>
    </tr>

    <tr>
      <td><code>useBlockTimestampFallback</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td>
        Many providers don't yet return `blockTimestamp` from `eth_getLogs` RPC
        endpoint and in such cases `block_timestamp` column will be `null`.
        If you enable this fallback the library will perform additional call to
        `eth_getBlock` to populate the timestam, but this may result in
        significant performance penalty when fetching many log records.

        See: [ethereum/execution-apis#295](https://github.com/ethereum/execution-apis/issues/295)
      </td>
    </tr>
  </tbody>
</table>

## `ExtraConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>graphql</code></td>
      <td><a href="#gqlconfig"><code>GqlConfig</code></a></td>
      <td><code>\{}</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `FlightSqlConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>allowAnonymous</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>
      <td>Whether clients can authenticate as 'anonymous' user</td>
    </tr>

    <tr>
      <td><code>anonSessionExpirationTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30m"</code></td>

      <td>
        Time after which `FlightSQL` client session will be forgotten and client
        will have to re-authroize (for anonymous clients)
      </td>
    </tr>

    <tr>
      <td><code>anonSessionInactivityTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5s"</code></td>

      <td>
        Time after which `FlightSQL` session context will be released to free
        the resources (for anonymous clients)
      </td>
    </tr>

    <tr>
      <td><code>authedSessionExpirationTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30m"</code></td>

      <td>
        Time after which `FlightSQL` client session will be forgotten and client
        will have to re-authroize (for authenticated clients)
      </td>
    </tr>

    <tr>
      <td><code>authedSessionInactivityTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5s"</code></td>

      <td>
        Time after which `FlightSQL` session context will be released to free
        the resources (for authenticated clients)
      </td>
    </tr>
  </tbody>
</table>

## `FlowSystemConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>awaitingStepSecs</code></td>
      <td><code>integer</code></td>
      <td><code>1</code></td>

      <td>
        Scheduling granularity: activation times are rounded to it, and failed
        activations retried after it. Not a polling period
      </td>
    </tr>

    <tr>
      <td><code>defaultRetryPolicies</code></td>
      <td><code>object</code></td>
      <td><code>\{}</code></td>
      <td>Retry policies applied by default, by flow type</td>
    </tr>

    <tr>
      <td><code>mandatoryThrottlingPeriodSecs</code></td>
      <td><code>integer</code></td>
      <td><code>60</code></td>
      <td>Minimal time between two runs of the same flow</td>
    </tr>
  </tbody>
</table>

## `FrontendConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>jupyter</code></td>
      <td><a href="#jupyterconfig"><code>JupyterConfig</code></a></td>
      <td><pre><code>\{
      "image": "ghcr.io/kamu-data/jupyter:0.7.1",
      "livyImage": "ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"
      }</code></pre></td>
      <td>Integrated Jupyter notebook configuration</td>
    </tr>
  </tbody>
</table>

## `GqlConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody />
</table>

## `HttpSourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>connectTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30s"</code></td>
      <td>Timeout for the connect phase of the HTTP client</td>
    </tr>

    <tr>
      <td><code>maxRedirects</code></td>
      <td><code>integer</code></td>
      <td><code>10</code></td>
      <td>Maximum number of redirects to follow</td>
    </tr>

    <tr>
      <td><code>userAgent</code></td>
      <td><code>string</code></td>
      <td><code>"kamu-cli/0.270.0"</code></td>
      <td>Value to use for User-Agent header</td>
    </tr>
  </tbody>
</table>

## `IdentityConfig`

Private keys are used to sign API responses.
Supported algorithms: `ed25519`, `secp256k1`.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>ed25519PrivateKey</code></td>
      <td><a href="#privatekey"><code>PrivateKey</code></a></td>
      <td><code>null</code></td>

      <td>
        Root private key that corresponds to the `authority` and is used to sign
        responses.

        To generate, use:

        ```sh theme={null}
        od -vN 32 -An -tx1 /dev/urandom | tr -d ' \n' && echo
        ```

        or

        ```sh theme={null}
        openssl rand -hex 32
        ```
      </td>
    </tr>

    <tr>
      <td><code>secp256k1PrivateKey</code></td>
      <td><a href="#secp256k1signer"><code>Secp256k1Signer</code></a></td>
      <td><code>null</code></td>

      <td>
        Secp256k1 private key used to sign EIP-712 typed data.

        To generate, use:

        ```sh theme={null}
        od -vN 32 -An -tx1 /dev/urandom | tr -d ' \n' && echo
        ```

        or

        ```sh theme={null}
        openssl rand -hex 32
        ```

        or

        ```sh theme={null}
        cast wallet new
        ```
      </td>
    </tr>
  </tbody>
</table>

## `IpfsConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>httpGateway</code></td>
      <td><code>string</code></td>
      <td><code>"http\://localhost:8080/"</code></td>

      <td>
        HTTP Gateway URL to use for downloads.
        For safety, it defaults to `http://localhost:8080` - a local IPFS daemon.
        If you don't have IPFS installed, you can set this URL to
        one of the public gateways like `https://ipfs.io`.
        List of public gateways can be found here: `https://ipfs.github.io/public-gateway-checker/`
      </td>
    </tr>

    <tr>
      <td><code>preResolveDnslink</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>

      <td>
        Whether kamu should pre-resolve IPNS `DNSLink` names using DNS or leave
        it to the Gateway.
      </td>
    </tr>
  </tbody>
</table>

## `JupyterConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>image</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/jupyter:0.7.1"</code></td>
      <td>Jupyter notebook server image</td>
    </tr>

    <tr>
      <td><code>livyImage</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"</code></td>
      <td>UNSTABLE: Livy + Spark server image</td>
    </tr>
  </tbody>
</table>

## `MqttSourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>brokerIdleTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"1s"</code></td>

      <td>
        Time in milliseconds to wait for MQTT broker to send us some data after
        which we will consider that we have "caught up" and end the polling
        loop.
      </td>
    </tr>
  </tbody>
</table>

## `NetworkNamespaceType`

Corresponds to podman's `containers.conf::netns`
We podman is used inside containers (e.g. podman-in-docker or podman-in-k8s)
it usually runs uses host network namespace.

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>Private</code></td></tr>
    <tr><td><code>Host</code></td></tr>
  </tbody>
</table>

## `Password`

Base type: `string`

## `PredefinedAccountsConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>predefined</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `PrivateKey`

Base type: `string`

## `ProtocolConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>flightSql</code></td>
      <td><a href="#flightsqlconfig"><code>FlightSqlConfig</code></a></td>
      <td><pre><code>\{
      "allowAnonymous": true,
      "anonSessionExpirationTimeout": "30m",
      "anonSessionInactivityTimeout": "5s",
      "authedSessionExpirationTimeout": "30m",
      "authedSessionInactivityTimeout": "5s"
      }</code></pre></td>

      <td>
        `FlightSQL` configuration
      </td>
    </tr>

    <tr>
      <td><code>ipfs</code></td>
      <td><a href="#ipfsconfig"><code>IpfsConfig</code></a></td>
      <td><pre><code>\{
      "httpGateway": "http\://localhost:8080/",
      "preResolveDnslink": true
      }</code></pre></td>
      <td>IPFS configuration</td>
    </tr>
  </tbody>
</table>

## `QuotaDefaults`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>storage</code></td>
      <td><code>integer</code></td>
      <td><code>1000000000</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `RetryPolicyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>backoffType</code></td>
      <td><a href="#retrypolicyconfigbackofftype"><code>RetryPolicyConfigBackoffType</code></a></td>
      <td><code>"Fixed"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>maxAttempts</code></td>
      <td><code>integer</code></td>
      <td><code>0</code></td>

      <td />
    </tr>

    <tr>
      <td><code>minDelaySecs</code></td>
      <td><code>integer</code></td>
      <td><code>0</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `RetryPolicyConfigBackoffType`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>Fixed</code></td></tr>
    <tr><td><code>Linear</code></td></tr>
    <tr><td><code>Exponential</code></td></tr>
    <tr><td><code>ExponentialWithJitter</code></td></tr>
  </tbody>
</table>

## `SearchConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>embeddingsChunker</code></td>
      <td><a href="#embeddingschunkerconfig"><code>EmbeddingsChunkerConfig</code></a></td>
      <td><pre><code>\{
      "kind": "Simple",
      "splitParagraphs": false,
      "splitSections": false
      }</code></pre></td>
      <td>Embeddings chunker configuration</td>
    </tr>

    <tr>
      <td><code>embeddingsEncoder</code></td>
      <td><a href="#embeddingsencoderconfig"><code>EmbeddingsEncoderConfig</code></a></td>
      <td><pre><code>\{
      "kind": "Dummy"
      }</code></pre></td>
      <td>Embeddings encoder configuration</td>
    </tr>

    <tr>
      <td><code>indexer</code></td>
      <td><a href="#searchindexerconfig"><code>SearchIndexerConfig</code></a></td>
      <td><pre><code>\{
      "clearOnStart": false,
      "incrementalIndexing": true
      }</code></pre></td>
      <td>Indexer configuration</td>
    </tr>

    <tr>
      <td><code>repo</code></td>
      <td><a href="#searchrepositoryconfig"><code>SearchRepositoryConfig</code></a></td>
      <td><pre><code>\{
      "kind": "Dummy"
      }</code></pre></td>
      <td>Search repository configuration</td>
    </tr>
  </tbody>
</table>

## `SearchIndexerConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>clearOnStart</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>
      <td>Whether to clear and re-index on start or use existing vectors if any</td>
    </tr>

    <tr>
      <td><code>incrementalIndexing</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>
      <td>Whether incremental indexing is enabled</td>
    </tr>
  </tbody>
</table>

## `SearchRepositoryConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#searchrepositoryconfigdummy"><code>Dummy</code></a></td></tr>
    <tr><td><a href="#searchrepositoryconfigelasticsearch"><code>Elasticsearch</code></a></td></tr>
    <tr><td><a href="#searchrepositoryconfigelasticsearchcontainer"><code>ElasticsearchContainer</code></a></td></tr>
  </tbody>
</table>

## `SearchRepositoryConfig::Dummy`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `SearchRepositoryConfig::Elasticsearch`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>caCertPemPath</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>embeddingDimensions</code></td>
      <td><code>integer</code></td>
      <td><code>1536</code></td>

      <td />
    </tr>

    <tr>
      <td><code>enableCompression</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>indexPrefix</code></td>
      <td><code>string</code></td>
      <td><code>""</code></td>

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>password</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>timeoutSecs</code></td>
      <td><code>integer</code></td>
      <td><code>30</code></td>

      <td />
    </tr>

    <tr>
      <td><code>url</code></td>
      <td><code>string</code></td>
      <td><code>"http\://localhost:9200/"</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `SearchRepositoryConfig::ElasticsearchContainer`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>embeddingDimensions</code></td>
      <td><code>integer</code></td>
      <td><code>1536</code></td>

      <td />
    </tr>

    <tr>
      <td><code>image</code></td>
      <td><code>string</code></td>
      <td><code>"docker.io/elasticsearch:9.2.1"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>startTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30s"</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `Secp256k1Signer`

Base type: `string`

## `SecretsEncryptionConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>enabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>encryptionKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Represents the encryption key for secrets. This field is required if
        `enabled` is `true` or `None`.

        The encryption key must be a 32-character alphanumeric string, which
        includes both uppercase and lowercase Latin letters (A-Z, a-z) and
        digits (0-9).

        To generate use:

        ```sh theme={null}
        tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo
        ```
      </td>
    </tr>
  </tbody>
</table>

## `SourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>ethereum</code></td>
      <td><a href="#ethereumsourceconfig"><code>EthereumSourceConfig</code></a></td>
      <td><pre><code>\{
      "commitAfterBlocksScanned": 1000000,
      "getLogsBlockStride": 100000,
      "rpcEndpoints": \[],
      "useBlockTimestampFallback": false
      }</code></pre></td>
      <td>Ethereum-specific configuration</td>
    </tr>

    <tr>
      <td><code>http</code></td>
      <td><a href="#httpsourceconfig"><code>HttpSourceConfig</code></a></td>
      <td><pre><code>\{
      "connectTimeout": "30s",
      "maxRedirects": 10,
      "userAgent": "kamu-cli/0.270.0"
      }</code></pre></td>
      <td>HTTP-specific configuration</td>
    </tr>

    <tr>
      <td><code>mqtt</code></td>
      <td><a href="#mqttsourceconfig"><code>MqttSourceConfig</code></a></td>
      <td><pre><code>\{
      "brokerIdleTimeout": "1s"
      }</code></pre></td>
      <td>MQTT-specific configuration</td>
    </tr>

    <tr>
      <td><code>targetRecordsPerSlice</code></td>
      <td><code>integer</code></td>
      <td><code>10000</code></td>

      <td>
        Target number of records after which we will stop consuming from the
        resumable source and commit data, leaving the rest for the next
        iteration. This ensures that one data slice doesn't become too big.
      </td>
    </tr>
  </tbody>
</table>

## `UploadsConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>maxFileSizeInMb</code></td>
      <td><code>integer</code></td>
      <td><code>50</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `WebhooksConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>deliveryTimeout</code></td>
      <td><code>integer</code></td>
      <td><code>10</code></td>

      <td />
    </tr>

    <tr>
      <td><code>maxConsecutiveFailures</code></td>
      <td><code>integer</code></td>
      <td><code>5</code></td>

      <td />
    </tr>

    <tr>
      <td><code>secretEncryptionEnabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>secretEncryptionKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Represents the encryption key for the webhooks secret. This field is
        required if `secret_encryption_enabled` is `true` or `None`.

        The encryption key must be a 32-character alphanumeric string, which
        includes both uppercase and lowercase Latin letters (A-Z, a-z) and
        digits (0-9).

        # Example

        ```
        let config = WebhooksConfig \{
            ...
            secret_encryption_enabled: Some(true),
            encryption_key:
        Some(String::from("aBcDeFgHiJkLmNoPqRsTuVwXyZ012345")) };
        ```
      </td>
    </tr>
  </tbody>
</table>


# Derivative Dataset Evolution
Source: https://docs.kamu.dev/cli/evolve/deriv



## Derivative Dataset Evolution

## Handling upstream changes

## Evolving transformations


# Root Dataset Evolution
Source: https://docs.kamu.dev/cli/evolve/root



## Handling source URL changes

## Handling upstream format changes


# Schema Evolution
Source: https://docs.kamu.dev/cli/evolve/schema



## Adding / deprecating columns

## Upstream schema changes

## Backwards incompatible changes


# Exploring Data
Source: https://docs.kamu.dev/cli/explore

Viewing data and performing basic exploratory data analysis.

While `kamu` focuses primarily on the problem of data management, you often may want to do some basic data exploration before exporting data for further use in your data science projects, so we decided to provide a few simple exploration tools for you to assess the state of data without leaving the comfort of one tool.

## Tail Command

Use [`kamu tail`](/cli/commands#kamu-tail) command to quickly view a sample of last events in a dataset:

```bash theme={null}
kamu tail ca.bccdc.covid19.case-details
```

## Inspect Command Group

A set of [`kamu inspect *`](/cli/commands#kamu-inspect) commands allows you to explore metadata and lineage of datasets. For example to display the lineage of a certain dataset in a browser use:

```bash theme={null}
kamu inspect lineage ca.covid19.daily-cases -b
```

<Diagram alt="kamu sql" />

## SQL Console

[`kamu sql`](/cli/commands#kamu-sql) command group provides a simple way to run ad-hoc queries and explore data using SQL language.

<Diagram alt="kamu sql" />

Following command will drop you into the SQL shell:

```bash theme={null}
kamu sql
```

By default this command uses the [Apache Datafusion](https://arrow.apache.org/datafusion/) engine, so its [powerful SQL](https://arrow.apache.org/datafusion/user-guide/sql/index.html) is now available to you.

<Tip>
  You can also select other engines, e.g. [Apache Spark](https://spark.apache.org/)!
</Tip>

All datasets in your workspace should be available to you as tables:

```sql theme={null}
show tables;
```

You can use `describe` to inspect the dataset's schema:

```sql theme={null}
describe "us.cityofnewyork.data.zipcode-boundaries";
```

<Note>
  The extra quotes needed to treat the dataset name containing dots as a table name.
</Note>

And of course you can run queries against any dataset:

```sql theme={null}
select
  *
from "us.cityofnewyork.data.zipcode-boundaries"
order by pop_est desc
limit 5;
```

Use `Ctrl+D` to exit the SQL shell.

SQL is a widely supported language, so `kamu` can be used in conjunction with many other tools that support it, such as Tableau and Power BI. See [integrations](/integrations) for details.

The `kamu sql` is a very powerful command that you can use both interactively or for scripting. We encourage you to explore more of its options through `kamu sql --help`.

## Jupyter Notebooks

Kamu also connects the power of [Apache Spark](https://spark.apache.org/) with the [Jupyter Notebook](https://jupyter.org/) server. You can get started by running:

```bash theme={null}
kamu notebook
```

<Tip>
  You can use `-e ENV_VAR` option to pass additional environment variable into the notebook server. This can be very useful for different access and security tokens needed by different visualization APIs.
</Tip>

Executing this should open your default browser with a Jupyter running in it.

From here let's create a new notebook. We start our notebook by loading `kamu` extension:

```
%load_ext kamu
```

And creating `kamu` connection:

```python theme={null}
import kamu
con = kamu.connect()
```

The library will automatically connect to your local workspace node, but you can also provide a URL to mix local and remote connections (see the [library documentation](https://github.com/kamu-data/kamu-client-python) for details).

You can now query the datasets using the connection:

```python theme={null}
con.query("select * from 'my-dataset' limit 3")
```

Or, more conveniently, using the `%%sql` cell magic:

```sql theme={null}
%%sql
select * from 'my-dataset' limit 3
```

<Diagram alt="kamu notebook 002" />

<Note>
  By default the `notebook` command will use DataFusion engine. To start with Spark use:

  ```sh theme={null}
  kamu notebook --engine spark
  ```
</Note>

The queries return regular Pandas dataframe.

Thanks to the [autovizwidget](https://github.com/jupyter-incubator/sparkmagic) library you also get some simple instant visualizations for results of your queries.

<Diagram alt="kamu notebook 003" />

To assign the result of `%%sql` cell to a variable use:

```sql theme={null}
%%sql -o my_dataframe
select * from x
```

<Diagram alt="kamu notebook 004" />

You can use any of your favorite libraries to further process and visualize it.

## Web UI

And finally, `kamu` comes with embedded Web UI that you can use to explore your pipelines and run SQL queries on data from the comfort of your browser:

<Diagram alt="Kamu Web UI" />

You can launch it by running:

```bash theme={null}
kamu ui
```


# Blockchain Sources
Source: https://docs.kamu.dev/cli/ingest/blockchain-source



Blockchains, especially those that run smart contracts, have turned into real treasure troves of data and can be viewed as datasets that aggregate millions of smaller datasets within them. Kamu allows you to directly use data from any EVM-compatible blockchain.

Currently we support reading logs, but access to fully reconcilable transaction information is planned.

## Configuring RPC Node

Data is read directly from blockchain nodes, so to ingest data your CLI needs to have the RPC URL configured:

```yaml theme={null}
kind: CLIConfig
version: 1
content:
  source:
    ethereum:
      rpcEndpoints:
        - chainId: 1
          chainName: Ethereum Mainnet
          nodeUrl: wss://localhost:8545
```

You can also specify `nodeUrl` directly in the <Schema /> (see below).

## Accessing Specific Log

Logs can be accessed using <Schema /> ([full example](https://github.com/kamu-data/kamu-cli/blob/master/examples/reth-vs-snp500/net.rocketpool.reth.tokens-minted.yaml)):

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: net.rocketpool.reth.tokens-minted
  kind: Root
  metadata:
    - kind: SetPollingSource
      fetch:
        kind: EthereumLogs
        # Ethereum Mainnet
        # See: https://chainlist.org/
        chainId: 1
        # Optional signature can be used to decode
        # the event into a nested struct
        signature: |
          TokensMinted(
            address indexed to,
            uint256 amount,
            uint256 ethAmount,
            uint256 time
          )
        # Filter by contract address and the deployment
        # block to limit scanning. Syntax is the same as
        # in SQL WHERE clause.
        filter: |
          address = X'ae78736cd615f374d3085123a210448e74fc6393'
          and
          block_number > 13325304
      read:
        kind: Parquet
      preprocess:
        kind: Sql
        engine: datafusion
        query: |
          select
            to_timestamp_seconds(
              cast(event.time as bigint)
            ) as event_time,
            block_number,
            block_hash,
            transaction_index,
            transaction_hash,
            log_index,
            event.to as to,
            event.amount as amount,
            event."ethAmount" as eth_amount
          from input
      merge:
        kind: Append
```

The above source declaration will use `filter` to create the **most efficient RPC request** to the blockchain node and stream the events using .

The output of the `EthereumLogs` fetch step is a data structure that corresponds directly to the output of [`eth_getLogs`](https://ethereum.org/en/developers/docs/apis/json-rpc/#eth_getlogs) endpoint, with the addition of optional decoded `event` field:

| Name | Type | Nullable | Description |
| - | :-: | :-: | - |
| `block_number` | `uint64` | | |
| `block_hash` | `binary` | | |
| `block_timestamp` | `timestamp` | V | Many providers [don't yet return](https://github.com/ethereum/execution-apis/issues/295) `blockTimestamp` from `eth_getLogs` |
| `transaction_index` | `uint64` | | |
| `transaction_hash` | `binary` | | |
| `log_index` | `uint64` | | |
| `address` | `binary` | | |
| `topic0` | `binary` | V | |
| `topic1` | `binary` | V | |
| `topic2` | `binary` | V | |
| `topic3` | `binary` | V | |
| `data` | `binary` | | |
| **`event`** | `struct` | | If `signature` is specified, will contain a decoded version of the event as a nested struct field |

## Decoding Raw Logs in SQL

It's also possible to ingest raw logs without decoding them with `signature`.

In this example we read all logs of the specific contract and decode them later in SQL ([full example](https://github.com/kamu-data/kamu-contrib/blob/master/net.rocketpool/reth.mint-burn-2in1.yaml)):

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: net.rocketpool.reth.mint-burn
  kind: Root
  metadata:
    - kind: SetPollingSource
      fetch:
        kind: EthereumLogs
        # Ethereum Mainnet
        # See: https://chainlist.org/
        chainId: 1
        # Read raw logs, filtering by signatures we care about
        filter: |
          address = X'ae78736cd615f374d3085123a210448e74fc6393'
          and (
            topic0 = eth_event_selector(
              'TokensMinted(address indexed to, uint256 amount, uint256 ethAmount, uint256 time)'
            )
            or topic0 = eth_event_selector(
              'TokensBurned(address indexed from, uint256 amount, uint256 ethAmount, uint256 time)'
            )
          )
          and block_number > 13325304
      read:
        kind: Parquet
      preprocess:
        kind: Sql
        engine: datafusion
        query: |
          select
            to_timestamp_seconds(
              cast(json_get_str(event, 'time') as bigint
            ) as event_time,
            block_number,
            block_hash,
            transaction_index,
            transaction_hash,
            log_index,
            json_get_str(event, 'name') as event_name,
            decode(
              coalesce(
                json_get_str(event, 'to'),
                json_get_str(event, 'from')
              ),
              'hex'
            ) as holder_address,
            json_get_str(event, 'amount') as amount,
            json_get_str(event, 'ethAmount') as eth_amount
          from (
            select
              *,
              coalesce(
                eth_try_decode_event(
                  'TokensMinted(address indexed to, uint256 amount, uint256 ethAmount, uint256 time)',
                  topic0,
                  topic1,
                  topic2,
                  topic3,
                  data
                ),
                eth_try_decode_event(
                  'TokensBurned(address indexed from, uint256 amount, uint256 ethAmount, uint256 time)',
                  topic0,
                  topic1,
                  topic2,
                  topic3,
                  data
                )
              ) as event
            from input
          )
      merge:
        kind: Append
```

The above code is using `eth_try_decode_event` function provided by [`datafusion-ethers`](https://github.com/kamu-data/datafusion-ethers) extension to the SQL engine. Upon success this function returns a decoded event as JSON string, which we then take apart using [`datafusion-functions-json`](https://github.com/datafusion-contrib/datafusion-functions-json) set of functions.

## Future Work

Blockchain data satisfies all properties of an ODF dataset (history-preserving, reproducible, verifiable), with differences primarily in encoding. We therefore see Kamu NOT as an indexer (solutions that copy data from blockchains into queryable databases, leading to recentralization concerns). Instead we see **blockchains as a natural extension of the Open Data Fabric network**.

Integrating blockchains as data sources and consumers of data via [Kamu Oracle](/api/oracle) makes Kamu the first system where you can go through the whole cycle (reading on-chain data, merging it with off-chain data, providing data back on-chain to a smart contract) within one solution, with just SQL, with full verifiability.

Our future work will focus on further **erasing the boundary between on- and off-chain data**.


# Ingesting Data
Source: https://docs.kamu.dev/cli/ingest/index



Ingestion is the process by which external data is added into Open Data Fabric's <Term />. Kamu supports a wide variety of sources including resources on the web and [blockchains](/cli/ingest/blockchain-source). Below we will describe why this process is necessary and how it works.

## Motivation

When interacting with data on the web we usually cannot make any assumptions about guarantees that its publisher provides:

* Just by looking at the data we cannot tell if it is **mutable** or preserves **history of changes**
* We often can't tell where it **originates** from and who can be held **accountable** for its **validity**
* We can't be sure that it's sufficiently **replicated**, so that if the server that hosts it today disappears tomorrow - all our data science projects will not be left **non-reproducible**.

Data on the web is in a state of a *constant churn* where data is often updated in destructive ways and even important medical datasets disappear over night. Any guarantees do exist are hidden behind many layers of service agreements, yet still provide us no strong assurance that they won't be violated (on purpose or by accident). This is why step #1 of every data science project is to **copy and version** the data.

[Open Data Fabric](/odf/spec/overview) was created to **avoid excessive copying and harmful versioning** of data, and instead embed these guarantees into the data format itself, making them **impossible to violate**.

Ingestion step is about getting the external data from this *"churning world"* into strict ledgers of <Term />, where properties like clear ownership, reproducibility, accountability, and complete historical account can be guaranteed and made explicit.

## Sources

There are two types of ingestion sources:

* **Push sources** - for cases when external actor actively sends data into a dataset. This is suitable e.g. for IoT devices that periodically write data, business processes that can report events directly into ODF dataset, or for ingesting data from streaming data APIs and event queues like [Apache Kafka](https://kafka.apache.org/).
* **Polling source** - for cases when external data is stored somewhere in bulk and we want to synchronize its state with ODF dataset periodically. This is suitable e.g. for datasets that exist as files on the web or for reading data from bulk APIs.

## Phases

Ingestion process has several well-defined phases:

<div>
  <Diagram alt="Ingest flow" />
</div>

These phases are directly reflected in the <Schema /> event:

* `fetch` - specifies how to download the data from some external source (e.g. HTTP/FTP) and how to cache it efficiently
* `prepare` (optional) - specifies how to prepare raw binary data for reading (e.g. extracting an archive or converting between formats)
* `read` - specifies how to read the data into structured form (e.g. as CSV or Parquet)
* `preprocess` (optional) - allows to shape the structured data with queries (e.g. to parse and convert types into best suited form wit SQL)
* `merge` - specifies how to **combine the read data with the history of previously seen data** (this step is extremely important as it performs "ledgerization" / "historization" of the evolving state of data - see [Merge Strategies](/cli/ingest/merge-strategies) section for explanation).

<Tip>
  If you are confused about what `SetPollingSource` event is - please refer to [First Steps](/cli/quick-start) section that explains dataset creation.
</Tip>

The phases of push ingest are defined by the <Schema /> event and are very similar, except for omitting `fetch` and `prepare` steps.

For more information refer to [Polling Sources](/cli/ingest/polling-source) and [Push Sources](/cli/ingest/push-source) sections.

## Further Reading

* For more information about defining data sources refer to [Polling Sources](/cli/ingest/polling-source) and [Push Sources](/cli/ingest/push-source) sections.
* For examples of dealing with various types of data refer to [Input Formats](/cli/ingest/input-formats) section.
* For detailed explanation of "ledgerization" process see [Merge Strategies](/cli/ingest/merge-strategies) section.
* For more inspiration on creating <Term /> see [Examples](/examples) and [`kamu-contrib`](https://github.com/kamu-data/kamu-contrib/) repo.


# Input Formats
Source: https://docs.kamu.dev/cli/ingest/input-formats

Examples of handling tricky formats during ingestion

`kamu` supports multiple <Schema /> and <Schema /> that in combination with custom <Schema /> can be used ingest all kinds of data.

Note that `kamu` is not made to be or replace data workflow tools like [Apache Airflow](https://airflow.apache.org/) or [Apache Ni-Fi](https://nifi.apache.org/), or data extraction tool like [Debezium](https://debezium.io/). Utilities described below are here only to simplify the initial data ingestion step - the very first step in the data's journey through a web of structured stream processing pipelines.

# CSV Variants

You can customize a lot of formatting options for CSV format parser.

For example, a tab-separated file can be read as:

```yaml theme={null}
read:
  kind: Csv
  separator: "\t"
  quote: '"'
```

See: <Schema />

# JSON Document

If you have a JSON document such as this:

```json theme={null}
{
  "nested": {
    "values": [
        {"id": 1, "key": "value"},
        {"id": 2, "key": "value"},
    ]
  }
}
```

first consider if you can use [NDJSON](#ndjson) instead, as ordinary JSON documents cannot be split and read efficiently.

If not, use:

```yaml theme={null}
read:
  kind: Json
  subPath: nested.values
```

Where `subPath` points at the array of records withing the document.

See: <Schema />

# NDJSON

[NDJSON](https://ndjson.org/), aka newline-delimited JSON file such as:

```json theme={null}
{"id": 1, "key": "value"}
{"id": 2, "key": "value"}
```

Can be read using:

```yaml theme={null}
read:
  kind: NdJson
  schema:
    fields:
      - name: id
        type: Int64
      - name: key
        type: String
```

See: <Schema />

# GeoJSON Document

Simply use:

```yaml theme={null}
read:
  kind: GeoJson
```

The reader expects one `FeatureCollection` object in the root and will create a record per each `Feature` inside it, extracting the properties into individual columns and leaving the feature `geometry` in its own column.

See: <Schema />

# NDGeoJSON Document

Simply use:

```yaml theme={null}
read:
  kind: NdGeoJson
```

It is similar to [GeoJSON](#geojson-document) format but instead of `FeatureCollection` object in the root it expects every individual `Feature` to appear on its own line.

See: <Schema />

# Esri Shapefile

GIS data in ESRI format can be read as:

```yaml theme={null}
read:
  kind: EsriShapefile
  subPath: specific_data-*.shp
# Use preprocess to optionally convert between different projections
preprocess:
  kind: Sql
  engine: spark
  query: |
    SELECT
      ST_Transform(geometry, "epsg:3157", "epsg:4326") as geometry,
      ...
    FROM input
```

See: <Schema />

# Compressed Data & Archives

Use `decompress` preparation step to extract data from `gzip`, `zip` archives.

```yaml theme={null}
prepare:
- kind: Decompress
  format: Gzip
```

In case of a multi-file archive you can specify which file should be extracted:

```yaml theme={null}
prepare:
- kind: Decompress
  format: Zip
  subPath: specific-file-*.csv  # Note: can contain glob patterns
```

See: <Schema />

# Other Formats

If you have to work with formats that are not natively supported you'll need to transcode them.

Using the `pipe` preparation step you can specify a custom program or a script that will get data via STDIN and output result to STDOUT.

For example here's how transcoding a JSON document into CSV using `jq` may look like:

```yaml theme={null}
prepare:
- kind: Pipe
  command:
  - 'jq'
  - '-r'
  - '.values[] | [.id, .key] | @csv'
read:
  kind: Csv
  schema:
    fields:
      - name: id
        type: Int64
      - name: key
        type: String
```

See: <Schema />

# Directory of Timestamped Files

The <Schema /> is used in cases where directory contains a growing set of files. Files can be periodic snapshots of your database or represent batches of new data in a ledger. In either case file content should never change - once `kamu` processes a file it will not consider it again. It's OK for files to disappear - `kamu` will remember the name of the file it ingested last and will only consider files that are higher in order than that one (lexicographically based on file name, or based on event time as shown below).

In the example below data inside the files is in snapshot format, and to complicate things it does not itself contain an event time - the event time is written into the file's name.

Directory contents:

```bash theme={null}
db-table-dump-2020-01-01.csv
db-table-dump-2020-01-02.csv
db-table-dump-2020-01-03.csv
```

Fetch step:

```yaml theme={null}
fetch:
  kind: FilesGlob
  path: /home/username/data/db-table-dump-*.csv
  eventTime:
    kind: FromPath
    pattern: 'db-table-dump-(\d+-\d+-\d+)\.csv'
    timestampFormat: '%Y-%m-%d'
  cache:
    kind: Forever
```

See: <Schema />

# Dealing with API Keys

Sometimes you may want to parametrize the URL to include things like API keys and auth tokens. For this `kamu` supports basic variable substitution:

```yaml theme={null}
fetch:
  kind: Url
  url: "https://api.etherscan.io/api?apikey=${{ env.ETHERSCAN_API_KEY }}"
```

You can use default values for parameters:

```yaml theme={null}
fetch:
  kind: Container
  image: "ghcr.io/kamu-data/fetch-com.defillama:${{ env.IMAGE_VERSION || '0.1.5' }}"
  args:
    - --request-interval
    - '${{ env.request_interval || 2 }}'
```

Also, there's a possibility of specifying several possible options:

```yaml theme={null}
fetch:
  kind: Container
  image: "ghcr.io/kamu-data/fetch-com.github.stats:0.1.0 }}"
  args:
    - --access-token
    - ${{ env.GH_TOKEN || env.GITHUB_TOKEN || 'demo' }}
```

# Using Ingest Scripts

Sometimes you may need the power of a general purpose programming language to deal with particularly complex API, or when doing web scraping. For this `kamu` supports containerized ingestion tasks:

```yaml theme={null}
fetch:
  kind: Container
  image: "ghcr.io/kamu-data/fetch-my-dataset:0.1.0"
  env:
    - name: ETH_NODE_PROVIDER_URL
```

The specified container image is expected to conform to the following interface:

* Produce data to `stdout`
* Write warnings / errors to `sterr`
* Use following environment variables:
  * `ODF_LAST_MODIFIED` - last modified time of data from the previous ingest run, if any (in RFC3339 format)
  * `ODF_ETAG` - caching tag of data from the previous ingest run, if any
  * `ODF_BATCH_SIZE` - is the recommended number of records, for ingest scripts that provide continuous stream of data and can resume from previous state
    * default value is 10 000, can be overridden via `env`
  * `ODF_NEW_LAST_MODIFIED_PATH` - path to a text file where ingest script may write new `Last-Modified` timestamp
  * `ODF_NEW_ETAG_PATH` - path to a text file where ingest script may write new `eTag`
  * `ODF_NEW_HAS_MORE_DATA_PATH` - path to a text file which ingest script can create to indicate about having more data for the next batch
    * **⚠️ Please note:** if the file is created, one of the following output marks must also be present: `eTag` or `Last-Modified` timestamp

# Need More Examples?

To give you more examples on how to deal with **different ingest scenarios** we've created an experimental repository where we publish Root Dataset manifests for a variety of Open Data sources - check out [kamu-contrib repo](https://github.com/kamu-data/kamu-contrib).


# Merge Strategies
Source: https://docs.kamu.dev/cli/ingest/merge-strategies

Describes the process of historization used depending on the nature of source data

# Purpose

Open Data Fabric [by design](/odf/spec/overview#nature-of-data) stores all data in the append-only event log format, always preserving the entire history. Unfortunately, a lot of data in the world is not stored or exposed this way. Some organizations may share their data in the form of periodic database dumps, while some choose to provide it as a log of changes between current and the previous export.

When ingesting data from external sources, <Term /> can choose between different <Schema /> that define how to combine the newly-ingested data with the existing one.

# Types

## Ledger

<Schema /> strategy should be used for data sources containing append-only event streams. New data exports can have new rows added, but once data already made it into one export it should never change or disappear. A user-specified primary key is used to identify which events were already seen, not to duplicate them.

### Example

Imagine a data publisher that exposes dataset with **historical** populations of major cities around the world, for example:

```csv theme={null}
Year,Country,City,Population
2019,CA,Vancouver,2581000
2019,US,Seattle,3433000
```

This dataset is temporal (has a `year` column), so it's a good sign that we're dealing with a `ledger` data.

<Warning>
  Having a time column doesn't always mean that the dataset is a ledger - to be a true ledger all previous records should be immutable and never change.
</Warning>

So we use the following root dataset manifest to ingest it:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: cities-population
  kind: Root
  metadata:
    - kind: SetPollingSource
      ...
      merge:
        kind: Ledger
        primaryKey:
          - year
          - country
          - city
    - kind: SetVocab
      eventTimeColumn: year
```

Notice that we specify `ledger` merge strategy with composite primary key `(year, country, city)`, meaning that we expect only one record per city per year. We also set `eventTimeColumn` to use `year` as the source of event times.

The resulting dataset when ingested will look like this:

| op | system\_time | event\_time | country | city | population |
| :-: | :-: | :-: | :-: | :-: | :-: |
| +A | s1 | 2019 | CA | Vancouver | 2581000 |
| +A | s1 | 2019 | US | Seattle | 3433000 |

`+A` in `op` column identifies the <Term /> and in this case all we have is `Append` operations.

Now let's say after a census in Vancouver our source data changes to:

```csv theme={null}
Country,City,Population
2019,CA,Vancouver,2581000
2019,US,Seattle,3433000
2020,CA,Vancouver,2606000
```

So far this is a valid ledger data - history is preserved and changes are append-only.

Pulling the dataset will now result in the following history:

| op | system\_time | event\_time | country | city | population |
| :-: | :-: | :-: | :-: | :-: | :-: |
| +A | s1 | 2019 | CA | Vancouver | 2581000 |
| +A | s1 | 2019 | US | Seattle | 3433000 |
| **+A** | **s1** | **2020** | **CA** | **Vancouver** | **2606000** |

Note that the old events from 2019 were recognized as ones we already seen and were not added again.

## Snapshot

<Schema /> strategy can be used for data exports that are taken periodically and contain only the latest state snapshot of the observed entity or system. Over time such exports can have new rows added, and old rows either removed or modified.

It's important to understand that publishers that use such formats **routinely lose information**. When a record in the database is updated, or one DB dump is replaced with another we not only lose the history of previous values, but you also lose the context of why those changes happened. This is a really bad default!

The `snapshot` strategy transforms such data sources into a history of changes by performing the [change data capture](https://en.wikipedia.org/wiki/Change_data_capture). It relies on a user-specified primary key to correlate the rows between the two snapshots.

A new event is added into the output stream whenever:

* A row with a certain primary key appears for the first time
* A row with a certain key disappears from the snapshot
* A row data associated with a certain key has changed

In case of removal or changes in data this strategy will issue explicit **retraction** (`-R`) and **correction** (`-C` / `+C`) events.

The Snapshot strategy also requires special treatment in regards to the event time. Since snapshot-style data exports represent the state of some system at a certain time - it is important to know what that time was. This time is usually captured in some form of metadata (e.g. in the name of the file, in the URL, or the HTTP caching headers. It should be possible to extract and propagate this time into a data column.

### Example

Imagine a data publisher that exposes dataset with **current** populations of major cities around the world, for example:

```csv theme={null}
Country,City,Population
CA,Vancouver,2581000
US,Seattle,3433000
```

This dataset is non-temporal (doesn't have any date/time columns), so it's a clear sign we're dealing with a `snapshot` data.

So we use the following root dataset manifest to ingest it:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: cities-population
  kind: Root
  metadata:
    - kind: SetPollingSource
      fetch:
        kind: Url
        url: https://...
        eventTime:
          kind: FromMetadata
      ...
      merge:
        kind: Snapshot
        primaryKey:
          - country
          - city
```

Notice that we specify `snapshot` merge strategy with composite primary key `(country, city)`. We also specify the `eventTime` of kind `FromMetadata`, instructing the ingest to use time from the caching headers as the event time of new records.

The resulting dataset when ingested will look like this:

| op | system\_time | event\_time | country | city | population |
| :-: | :-: | :-: | :-: | :-: | :-: |
| +A | s1 | e1 | CA | Vancouver | 2581000 |
| +A | s1 | e1 | US | Seattle | 3433000 |

Notice that since it's the first time we ingested data both records have `op == '+A'` values, i.e. "append" operation.

Now let's say after a census in Vancouver our source data changes to:

```csv theme={null}
Country,City,Population
CA,Vancouver,2606000
US,Seattle,3433000
```

Pulling the dataset will now result in the following history:

| op | system\_time | event\_time | country | city | population |
| :-: | :-: | :-: | :-: | :-: | :-: |
| +A | s1 | e1 | CA | Vancouver | 2581000 |
| +A | s1 | e1 | US | Seattle | 3433000 |
| **-C** | **s2** | **e1** | **CA** | **Vancouver** | **2581000** |
| **+C** | **s2** | **e2** | **CA** | **Vancouver** | **2606000** |

The two added rows represent the `'-C' correct-from` event that carries the old value that was changed, and `'+C' correct-to` event that carries the new value that was observed. Similarly the `'-R' retract` event can be issued for a record that disappeared in the source data.

This model is extremely powerful, as it not only provides us the full history of how data was evolving over time, but allows all downstream computations to [automatically react to corrections and retractions](/cli/transform/retractions-corrections).

## Append

Under <Schema /> strategy all observed input data will be appended to the Dataset in its original form, without any modifications.

This strategy is rarely used in datasets that poll data from a remote source, but can be useful for datasets that get data pushed into them (e.g. from an IoT device).


# Polling Sources
Source: https://docs.kamu.dev/cli/ingest/polling-source



Polling sources are used for cases when external data is stored somewhere in bulk and we want to **periodically** synchronize its state with an ODF dataset.

Polling sources are suitable for ingesting data from:

* Periodic database dumps
* Data published as a set of files on the web
* Bulk data access APIs
* External systems using custom connector libraries.

## Source Metadata

Polling sources are defined via <Schema /> metadata event:

```yaml theme={null}
---
kind: DatasetSnapshot
version: 1
content:
  name: com.example.city-populations
  kind: Root
  metadata:
    - kind: SetPollingSource
      # Where to fetch the data from.
      # Supports multiple protocols and file layouts
      fetch:
        kind: Url
        url: https://example.com/city_populations_over_time.zip
      # OPTIONAL: How to prepare the binary data
      # Includes decompression, file filtering, format conversions
      prepare:
        - kind: Decompress
          format: Zip
      # How to interpret the data.
      # Includes data format, schema to apply, error handling
      read:
        kind: Csv
        header: true
        schema:
          fields:
            - name: date
              type: Date
            - name: city
              type: String
            - name: population
              type: String
      # OPTIONAL: Pre-processing query that shapes the data.
      # Useful for converting text data into strict types
      preprocess:
        kind: Sql
        # Use one of the supported engines and a query in its dialect
        # See: https://docs.kamu.dev/cli/transform/supported-engines/
        engine: datafusion
        query: |
          select
            date,
            city,
            -- remove commas between thousands
            cast(replace(population, ",", "") as bigint)
          from input
      # How to combine data ingested in the past with the new data.
      merge:
        kind: Ledger
        primaryKey:
          - date
          - city
      # Lets you manipulate names of the system columns to avoid conflicts
      # or use names better suited for your data.
    - kind: SetVocab
      eventTimeColumn: date
```

The structure of the <Schema /> event directly reflects all the ingestion phases:

* `fetch` - specifies how to download the data from some external source (e.g. HTTP/FTP) and how to cache it efficiently
* `prepare` (optional) - specifies how to prepare raw binary data for reading (e.g. extracting an archive or converting between formats)
* `read` - specifies how to read the data into structured form (e.g. as CSV or Parquet)
* `preprocess` (optional) - allows to shape the structured data with queries (e.g. to parse and convert types into best suited form wit SQL)
* `merge` - specifies how to **combine the read data with the history of previously seen data** (this step is extremely important as it performs "ledgerization" / "historization" of the evolving state of data - see [Merge Strategies](/cli/ingest/merge-strategies) section for explanation).

A polling source can later be deactivated using <Schema /> event.

For multiple example of handling tricky data sources see [input formats](/cli/ingest/input-formats) section.

## Polling Data via CLI

To poll data into dataset via `kamu` use the general-purpose [`kamu pull`](/cli/commands#kamu-ingest) command.

```sh theme={null}
kamu pull com.example.city-populations
```

## Polling Data via API

See [APIs documentation](/api) for various options of polling data programmatically via APIs.

## Event Time

The perfect scenario for `kamu` is when data records contain <Term /> within them as a column, but many data sources on the web are not like that.

If event time is not present in data - `kamu` will try to infer it. This can be:

* Modification time for files on local or remote file systems
* `Last-Modified` time for HTTP resources.

If no time can be extracted from the source - it will fall back to using <Term />.

Usually its best to be explicit about where your <Term /> is coming by defining <Schema /> in the `fetch` section of <Schema />. It's pretty flexible, allowing you to even extract time from timestamps that are part of file names:

```yaml theme={null}
fetch:
  kind: FilesGlob
  # Ingest all files matching the pattern (in lexicographic order)
  path: /data/database-dump-*.json.gz
  eventTime:
    kind: FromPath
    # Extract timestamp as the first regex group in the pattern
    pattern: database-dump-(\d+-\d+-\d+)\.json\.gz
    # Parse timestamp
    timestampFormat: '%Y-%m-%d'
```

## Source Caching

`kamu` does its best to avoid redundant work and not ingest data if source was not updated since the lass poll.

Exact mechanism of cache control depends on the <Schema /> and the protocol used. In case of HTTP, for example, it will rely on standard HTTP caching headers like `ETag` and `Last-Modified`.

The latest caching information is stored in dataset metadata in <Schema /> event in a special <Schema /> object. This means that it is possible for ingest to return no data and no new <Term />, but still write a <Term /> containing only the new source state.

You can control caching behavior via <Schema /> object in the `fetch` section of <Schema />.


# Push Sources
Source: https://docs.kamu.dev/cli/ingest/push-source



Push sources are used in cases when some external actor actively sends data into a dataset.

Push sources are suitable for:

* IoT devices that periodically send data
* Business processes that report events directly into ODF dataset
* For ingesting data from streaming data APIs and event queues like [Apache Kafka](https://kafka.apache.org/).

## Source Metadata

Push sources are added via <Schema /> metadata event:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: gps
  kind: Root
  metadata:
    - kind: AddPushSource
      sourceName: default
      read:
        kind: NdJson
        schema:
          fields:
            - name: t
              type: Timestamp
            - name: long
              type: float64
            - name: lat
              type: float64
      merge:
        kind: Append
    - kind: SetVocab
      eventTimeColumn: t
```

Note that push sources are **named**:

```yaml theme={null}
sourceName: default
```

This allows you to declare **multiple push sources per dataset**, which may be useful for situations where several devices from different manufacturers produce slightly different data, but you want them to be able to write directly to one dataset.

An existing push source can be deactivated using <Schema /> event.

Also note that, just like polling sources, push sources pass data through a [Merge Strategy](/cli/ingest/merge-strategies), so they too can benefit, for example, from **event deduplication** of the <Schema /> merge strategy.

## Pushing Data via CLI

To push data into dataset via `kamu` use [`kamu ingest`](/cli/commands#kamu-ingest) command.

You can push data from files:

```sh theme={null}
echo '{"t": "2020-01-01T00:00:00Z", "long": -123.12, "lat": 49.28}' > data.json
kamu ingest gps data.json
```

Or from directly from STDIN:

```sh theme={null}
echo '{"t": "2020-01-01T00:00:00Z", "long": -123.12, "lat": 49.28}' | kamu ingest gps --stdin
```

Result:

```sh theme={null}
kamu tail gps
```

```sh theme={null}
┌────────┬────┬──────────────────────────┬──────────────────────┬─────────┬───────┐
│ offset │ op │       system_time        │          t           │  long   │  lat  │
├────────┼────┼──────────────────────────┼──────────────────────┼─────────┼───────┤
│      0 │ +A │ 2024-01-31T03:51:03.618Z │ 2020-01-01T00:00:00Z │ -123.12 │ 49.28 │
└────────┴────┴──────────────────────────┴──────────────────────┴─────────┴───────┘
```

## Pushing Data via API

See [APIs documentation](/api) for various options of pushing data programmatically via APIs.

## Format Translation

While push sources define a specific <Schema /> for their data - there is some tolerance. For example you can push <Schema /> data to the source that declares <Schema /> format and `kamu` will attempt to read it using the same schema.

This behavior works both in CLI commands and in push APIs.


# Install
Source: https://docs.kamu.dev/cli/install

Covers installation steps to get kamu-cli running on your computer

<Info>
  Before using this product, please read the [project status disclaimer](/general/project-status).
</Info>

## General Information

`kamu` is a **single-binary** utility that comes bundled with most of its dependencies.

It relies on container-based virtualization (e.g. `docker` or `podman`) to run such heavyweight frameworks like Spark, Flink, and Jupyter in isolated environments without needing you to install thousands of libraries and bloating your beloved laptop with their dependencies.

The tool comes with very good **shell completions**, so make sure to configure them!

Please report any issues during the installation process [here](https://github.com/kamu-data/kamu-cli/issues/).

## Supported Platforms

### Linux

Linux is our primary target environment. We don't have packages for various Linux flavors yet, but since the tool is just a simple binary it's very easy to get started:

1. Install `docker` using your distro's package manager (alternatively we highly recommend trying [podman](/#using-podman-instead-of-docker))
2. Make sure you can launch containers without `sudo` by following [official documentation](https://docs.docker.com/engine/install/linux-postinstall/)
3. Install `kamu` via installer script by running:
   ```bash theme={null}
   curl -s "https://get.kamu.dev" | sh
   ```
4. Verify your setup by running:
   ```bash theme={null}
   kamu system diagnose
   ```

See also:

* [Installing shell completions](/#installing-shell-completions)
* [A Note on Docker Security](/#a-note-on-docker-security)
* [Using Podman instead of Docker](/#using-podman-instead-of-docker)

### MacOS X

We fully support Intel and M-series Macs, to install `kamu` please follow these steps:

1. Install [Docker for Mac](https://docs.docker.com/docker-for-mac/install/)
   1. Consider allocating more CPUs and memory to the Docker VM in the settings
   2. If you want to run `kamu` outside of your user home directory - you may need to add additional mounts to the Docker VM. For example if your workspace is in `/opt/myworkspace` you'll need to mount it under the same name into the VM in Docker settings.
2. Install `kamu` via installer script by running:
   ```bash theme={null}
   curl -s "https://get.kamu.dev" | sh
   ```
3. Verify your setup by running:
   ```bash theme={null}
   kamu system diagnose
   ```

See also:

* [Installing shell completions](/#installing-shell-completions)
* [Using Podman instead of Docker](/#using-podman-instead-of-docker)

### Windows (using WSL2)

1. Install WSL2 following [these steps](https://docs.microsoft.com/en-us/windows/wsl/install-win10)
   Make sure you can at least open Linux shell before proceeding
2. Install `docker`
   1. We recommend installing [Docker Desktop for Windows](https://docker.com) and [enable WSL2 backend](https://docs.docker.com/desktop/windows/wsl/)
   2. Ensure that from your linux distribution you can launch containers without `sudo` and the following works:
      ```sh theme={null}
      docker run -it hello-world
      ```
3. Inside WSL2 distribution install `kamu` via installer script by running:
   ```bash theme={null}
   curl -s "https://get.kamu.dev" | sh
   ```
4. Follow installer instructions to update `PATH` and install completions:
   ```sh theme={null}
   # Add to the end of your ~/.bashrc
   PATH="$PATH:/home/$USER/.local/bin
   source <(kamu completions bash)
   ```
5. After restarting the shell confirm that this works:
   ```sh theme={null}
   # Executable can be found
   kamu

   # Should auto-complete to `kamu version`
   kamu vers<press TAB>
   ```
6. Verify your setup by running:
   ```bash theme={null}
   kamu system diagnose
   ```

See also:

* [Installing shell completions](/#installing-shell-completions)
* [Using Podman instead of Docker](/#using-podman-instead-of-docker)

### Windows (using Docker Desktop)

<Warning>
  The native Windows binary is still highly experimental, so in most cases it's better to use the WSL
</Warning>

1. Install and run [Docker Desktop](https://docs.docker.com/docker-for-windows/install/).
   1. It's a good idea to give the Docker's VM more CPU and RAM - you can do so in `VirtualBox`.
   2. Make sure that you can run `docker ps` successfully.
2. We recommend using [Windows Terminal](https://apps.microsoft.com/store/detail/windows-terminal/9N0DX20HK701) that supports unicode symbols and full colors
3. Download the latest `kamu` binary for Windows
4. Add it to your `PATH` environment variable

Docker Toolbox runs Docker in a Virtual Machine. This means to mount a file from your host file system into a Docker container the file first needs to be mounted into VM, so make sure all paths that `kamu` will need are mapped in VirtualBox VM settings.

<Tip>
  **Example:** Lets assume your workspace directory is `C:\Users\me\kamu`. When `kamu` runs it will detect that Docker runs in a VM it will convert it to `/c/Users/me/kamu`. So in your VM settings you may need to add a mapping from `C:\Users\me` to `/c/Users/me`.
</Tip>

## Upgrading

On most platforms a new version of `kamu` can be installed by simply re-running the installer script:

```bash theme={null}
curl -s "https://get.kamu.dev" | sh
```

## Installing shell completions

To be able to auto-complete the `kamu` commands please install completion scripts for the shell of your choosing. You can find detailed instructions by running `kamu completions --help`.

If you use `bash` add the following to your `~/.bashrc` file:

```bash theme={null}
source <(kamu completions bash)
```

If you use `zsh` add the following to your `~/.zshrc` file:

```bash theme={null}
autoload -U +X bashcompinit && bashcompinit
source <(kamu completions bash)
```

## A Note on Docker Security

We take your security very seriously. Unfortunately the execution model of `docker` that involves running the daemon process under `root` violates the Unix user permission model. Combined with the step of making `docker` command [sudo-less](https://docs.docker.com/engine/install/linux-postinstall/) this means that any process you run under your user can potentially access the entire file system with root privileges. Until `docker` changes its runtime model, sudo-less access to Docker will remain a security threat.

On our side we are taking following measures to gain your trust:

* `kamu` and all of its components are open-source and [available for review](https://github.com/kamu-data/kamu-cli/)
* All of our `docker` images are based on reputable source images and are [available for review](/general/supported-engines)
* When `kamu` starts `docker` containers it limits the scope of volumes it's mounting to a minimum. You can review the volume mounts by running `kamu` with `-v` flag or using `docker ps`.

To avoid all these issues please consider using [`podman`](/#using-podman-instead-of-docker) - this container runtime operates in **daemon-less** and **root-less** mode, so it's fully compliant with the standard Unix permission model.

## Using Podman instead of Docker

[`podman`](https://podman.io/) is an alternative container runtime that fixes the [security shortcomings of `docker`](#a-note-on-docker-security). We highly recommend you to give it a try, as we are planning to make it a default runtime in the near future.

<Warning>
  Make sure to follow [Podman's Rootless Setup Guide](https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md) to allow `kamu` to run containers without `sudo`.
</Warning>

In order to instruct `kamu` to use `podman` run:

```bash theme={null}
kamu config set --user engine.runtime podman
```

<Info>
  On some systems you need to separately install `podman-dnsname` package to allow containers to communicate with one another via hostnames. To check whether you have it run:

  ```bash theme={null}
  podman network create test
  podman network ls
  # NETWORK ID    NAME    VERSION  PLUGINS
  # 9f86d081884c  test    0.4.0    bridge,portmap,firewall,tuning,dnsname
  #                                                               ^^^ plugin installed
  podman network prune
  ```
</Info>

## Development Images

It is sometimes convenient to get `kamu-cli` in a Docker/Podman image. For this we have few options:

* [`ghcr.io/kamu-data/kamu-base:latest`](https://github.com/kamu-data/kamu-cli/tree/master/images/kamu-base) - comes with just `podman` and `kamu-cli` pre-installed
* [`ghcr.io/kamu-data/kamu-base:latest-with-data`](https://github.com/kamu-data/kamu-cli/tree/master/images/kamu-base-with-data) - comes with a sample data pipeline that you can use to test different features with
* [`ghcr.io/kamu-data/kamu-base:latest-with-data-mt`](https://github.com/kamu-data/kamu-cli/tree/master/images/kamu-base-with-data-mt) - comes with a sample multi-tenant dataset repository

For example, try running:

```bash theme={null}
docker run -it --rm ghcr.io/kamu-data/kamu-base:latest-with-data kamu list
```

<Info>
  These images have `podman` installed inside the container, so when `kamu` runs an engine `podman` will be trying to pull an image and start a container from within another container. For such container-in-container setup to work you may need to pass `--privileged` flag when running this image.
</Info>


# Quick Start
Source: https://docs.kamu.dev/cli/quick-start

A quick rundown of the key features to get a feel for the tool

For a quick overview of key functionality you can also view this tutorial:

<YouTube />

This tutorial will give you a high-level tour of `kamu` and show you how it works through examples.

We assume that you have already followed the [installation steps](/cli/install) and have `kamu` tool ready.

<Info>
  **Not ready to install just yet?**

  Try `kamu` in this [online tutorial](/start/tutorial) without needing to install anything.
</Info>

Don't forget to set up **shell completions** - they make using `kamu` a lot more fun!

## The help command

When you execute `kamu` or `kamu -h` - the help message about all top-level commands will be displayed.

To get help on individual commands type:

```bash theme={null}
kamu <command> -h
```

This will usually contain a detailed description of what command does along with usage examples.

Note that some command also have sub-commands, e.g. `kamu repo {add,list,...}`, same help pattern applies to those as well:

```bash theme={null}
kamu repo add -h
```

Command help is also available online on [CLI Commands Reference](/cli/commands) page.

## Ingesting data

Throughout this tutorial we will be using the [Modified Zip Code Areas](https://data.cityofnewyork.us/Health/Modified-Zip-Code-Tabulation-Areas-MODZCTA-/pri4-ifjk) dataset from New York Open Data Portal.

### Initializing the workspace

To work with `kamu` you first need a **workspace**. <Term /> is where kamu will store important information about <Term /> and cached data. Let's create one:

<Diagram alt="kamu init" />

```bash theme={null}
mkdir my_workspace
cd my_workspace
kamu init
```

A <Term /> is just a directory with `.kamu` folder where all sorts of <Term /> and <Term /> are stored. It behaves very similarly to `.git` directory version-controlled repositories.

As you'd expect the <Term /> is currently empty:

```bash theme={null}
kamu list
```

### Creating a dataset

One of the design principles of `kamu` is to know exactly where any piece of data came from. So it never blindly copies data around - instead we establish ownership and links to external sources.

We'll get into the details of that later, but for now let's create such link.

<Diagram alt="kamu pull" />

<Term /> are created from <Term /> - special files that describe the **desired state** of the metadata upon creation.

We will use a <Schema /> file from [kamu-contrib repo](https://github.com/kamu-data/kamu-contrib/blob/master/us.cityofnewyork.data/zipcode-boundaries.yaml) that looks like this:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: us.cityofnewyork.data.zipcode-boundaries
  kind: Root
  metadata:
    - kind: SetPollingSource
      fetch:
        kind: Url
        # Dataset home: https://data.cityofnewyork.us/Health/Modified-Zip-Code-Tabulation-Areas-MODZCTA-/pri4-ifjk
        url: https://data.cityofnewyork.us/api/geospatial/pri4-ifjk?date=20240115&accessType=DOWNLOAD&method=export&format=Shapefile
      read:
        kind: EsriShapefile
      merge:
        kind: Snapshot
        primaryKey:
          # Modified ZIP Code Tabulation Area (ZCTA)
          # See for explanation: https://nychealth.github.io/covid-maps/modzcta-geo/about.html
          - modzcta
```

You can either copy the above into a `example.yaml` file and run:

```bash theme={null}
kamu add example.yaml
```

Or add it directly from URL like so:

```bash theme={null}
kamu add https://raw.githubusercontent.com/kamu-data/kamu-contrib/master/us.cityofnewyork.data/zipcode-boundaries.yaml
```

Such YAML files are called <Term />. First two lines specify that the file contains <Schema /> object and then specify the version of the schema, for upgradeability:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content: ...
```

Next we give dataset a name and declare its kind:

```yaml theme={null}
name: us.cityofnewyork.data.zipcode-boundaries
kind: Root
```

<Term /> that ingest external data in `kamu` are called <Term /> datasets.

Next we have:

```yaml theme={null}
metadata:
  - kind: ...
    ...
  - kind: ...
    ...
```

This section contains one or many <Term /> that can describe different aspects of a dataset, like:

* where data comes from
* its schema
* license
* relevant documentation
* query examples
* data quality checks
* and much more.

<Tip>
  To create your own snapshot file use `kamu new` command - it outputs a well-annotated template that you can customize for your needs.
</Tip>

### Dataset Identity

During the dataset creation it is assigned a very special identifier.

You can see it by running:

```sh theme={null}
kamu list -w
```

```
┌───────────────────────────────────────────────────────────────────────────────┬──────────────────────────────────────────┬─────┐
│                                      ID                                       │                   Name                   │ ... │
├───────────────────────────────────────────────────────────────────────────────┼──────────────────────────────────────────┼─────┤
│ did:odf:fed01057c94fb0378e3222704bb70a261d3ebeaa0d1b38c056a0bdd476360b8548db1 │ us.cityofnewyork.data.zipcode-boundaries │ ... │
└───────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────┴─────┘
```

Or:

```sh theme={null}
kamu log us.cityofnewyork.data.zipcode-boundaries
```

```
Block #0: ...
SystemTime: ...
Kind: Seed
DatasetKind: Root
DatasetID: did:odf:fed01057c94fb0378e3222704bb70a261d3ebeaa0d1b38c056a0bdd476360b8548db1
```

This is a [globally unique identity](/odf/spec/datasets/identity) which is based on a cryptographic key-pair that only you control.

Thus by creating an ODF dataset you get both:

* a way to **uniquely identify** it on the web
* and a way to **prove ownership** over it.

This will be extremely useful when we get to [sharing data with others](/cli/collab).

### Fetching data

At this point our new dataset is still empty:

```bash theme={null}
kamu list
```

```
┌──────────────────────────────────────────┬──────┬────────┬─────────┬──────┐
│                   Name                   │ Kind │ Pulled │ Records │ Size │
├──────────────────────────────────────────┼──────┼────────┼─────────┼──────┤
│ us.cityofnewyork.data.zipcode-boundaries │ Root │   -    │       - │    - │
└──────────────────────────────────────────┴──────┴────────┴─────────┴──────┘
```

But the <Schema /> event that we specified in the snapshot describes where from and how `kamu` can ingest external data.

Polling sources perform following steps:

* `fetch` - downloading the data from some external source (e.g. HTTP/FTP)
* `prepare` (optional) - preparing raw binary data for ingestion (e.g. extracting an archive or converting between formats)
* `read` - reading the data into a structured form (e.g. from CSV or Parquet)
* `preprocess` (optional) - shaping the structured data with queries (e.g. to convert types into best suited form)
* `merge` - merging the new data from the source with the **history of previously seen data**

You can find more information about data sources and ingestion stages in [this section](/cli/ingest).

Note that the data file we are ingesting is in [ESRI Shapefile](https://en.wikipedia.org/wiki/Shapefile) format, which is a widespread format for geo-spatial data, so we are using a special <Schema /> reader.

To fetch data from the source run:

```bash theme={null}
kamu pull --all
```

At this point the source data will be downloaded, decompressed, parsed into the structured form, preprocessed and saved locally.

```bash theme={null}
kamu list
```

```
┌──────────────────────────────────────────┬──────┬───────────────┬─────────┬──────────┐
│                   Name                   │ Kind │    Pulled     │ Records │   Size   │
├──────────────────────────────────────────┼──────┼───────────────┼─────────┼──────────┤
│ us.cityofnewyork.data.zipcode-boundaries │ Root │ X seconds ago │     178 │ 1.87 MiB │
└──────────────────────────────────────────┴──────┴───────────────┴─────────┴──────────┘
```

Note that when you `pull` a dataset, only the new records that `kamu` hasn't previously seen will be added. In fact `kamu` preserves the complete history of all data - this is what enables you to have stable references to data, lets you "time travel", and establish from where and how certain data was obtained (provenance). We will discuss this in depth in further tutorials.

For now it suffices to say that all data is tracked by `kamu` in a series of blocks. The `Committed new block X` message you've seen during the `pull` tells us that the new data block was appended. You can inspect these blocks using `log` command:

```bash theme={null}
$ kamu log us.cityofnewyork.data.zipcode-boundaries
```

## Exploring data

Since you might not have worked with this dataset before you'd want to explore it first.

For this `kamu` provides many tools (from basic to advanced):

* `tail` command
* SQL shell
* Jupyter notebooks integration
* Web UI

### Tail command

To quickly preview few last <Term /> of any dataset use `tail` command:

```bash theme={null}
$ kamu tail us.cityofnewyork.data.zipcode-boundaries
```

### SQL shell

SQL is the *lingua franca* of the data science and `kamu` uses it extensively. So naturally it provides you a simple way to run ad-hoc queries on data.

<Diagram alt="kamu sql" />

Following command will drop you into the SQL shell:

```bash theme={null}
kamu sql
```

By default this command uses the [Apache Datafusion](https://arrow.apache.org/datafusion/) <Term />, so its [powerful SQL](https://arrow.apache.org/datafusion/user-guide/sql/index.html) is now available to you.

<Tip>
  You can also select other engines, e.g. [Apache Spark](https://spark.apache.org/)!
</Tip>

All datasets in your <Term /> should be available to you as tables:

```sql theme={null}
show tables;
```

You can use `describe` to inspect the dataset schema:

```sql theme={null}
describe "us.cityofnewyork.data.zipcode-boundaries";
```

<Note>
  The extra quotes are needed to treat the dataset name containing dots as a table name.
</Note>

And of course you can run queries against any dataset:

```sql theme={null}
select
  *
from "us.cityofnewyork.data.zipcode-boundaries"
order by pop_est desc
limit 5;
```

Use `Ctrl+D` to exit the SQL shell.

SQL is a widely supported language, so `kamu` can be used in conjunction with many other tools that support it, such as Tableau and Power BI. See [integrations](/integrations) for details.

The `kamu sql` is a very powerful command that you can use both interactively or for scripting. We encourage you to explore more of its options through `kamu sql --help`.

### Notebooks

Kamu lets you access the power of multiple data engines through a convenient interface of [Jupyter Notebooks](https://jupyter.org/).

Get started by running:

```bash theme={null}
kamu notebook -e MAPBOX_ACCESS_TOKEN=<your mapbox token>
```

<Note>
  Above we also tell `kamu` to pass the [MapBox](https://www.mapbox.com/) access token as `MAPBOX_ACCESS_TOKEN` environment variable into Jupyter, which we will use for plotting. You can get the token for free or skip this step and simply run `kamu notebook`.
</Note>

Executing this should open your default browser with a Jupyter running in it.

From here let's create a notebook and start it by loading `kamu` extension:

```
%load_ext kamu
```

We then need to create a connection:

```python theme={null}
import kamu
con = kamu.connect()
```

The `kamu` Python library will automatically connect to your local workspace node.

After this the `import_dataset` command becomes available and we can load the dataset and alias it by doing:

You can now query the datasets using the connection:

```python theme={null}
con.query("select * from 'us.cityofnewyork.data.zipcode-boundaries' limit 3")
```

Or, more conveniently, using the `%%sql` cell magic:

```sql theme={null}
%%sql
select * from 'us.cityofnewyork.data.zipcode-boundaries' limit 3
```

The queries return regular [Pandas](https://pandas.pydata.org/) dataframe.

<Diagram alt="kamu notebook 002" />

Thanks to the [autovizwidget](https://github.com/jupyter-incubator/sparkmagic) library you also get some simple instant visualizations for results of your queries.

<Diagram alt="kamu notebook 003" />

To assign the result of `%%sql` cell to a variable use:

```sql theme={null}
%%sql -o population_density
select ...
```

<Diagram alt="kamu notebook 004" />

You can use any of your favorite libraries to further process and visualize it:

Example of visualizing population density data as a *choropleth* chart using [mapboxgl](https://github.com/mapbox/mapboxgl-jupyter) library:

<Diagram alt="kamu notebook 005" />

You can find this as well as many other notebooks in [kamu-contrib](https://github.com/kamu-data/kamu-contrib) repo.

### Web UI

All the above and more is also available to you via embedded Web UI which you can launch by running:

```bash theme={null}
kamu ui
```

Web UI is especially useful once you start developing complex stream processing pipelines, to explore them more visually:

<Diagram alt="Kamu Web UI" />

## Conclusion

We hope this quick overview inspires you to give `kamu` a try!

Don't get distracted by the pretty notebooks and UIs though - we covered only the tip of the iceberg. The true power of `kamu` lies in how it manages data, letting you to reliably track it, transform it, and share results with your peers in an easily **<Term />** way.

Please continue to the [online tutorial](/start/tutorial) for some hands-on walkthroughs and tutorials, and check out our other [learning materials](/start/learning-materials).


# Streaming Aggregations
Source: https://docs.kamu.dev/cli/transform/aggregations





# Geospatial Data
Source: https://docs.kamu.dev/cli/transform/gis-data





# Transforming Data
Source: https://docs.kamu.dev/cli/transform/index

Cleaning, enriching, aggregating, and otherwise transforming data in the system. Properties of derivative datasets and stream processing.

## Conceptual Model

Open Data Fabric clearly separates two types of data:

* **Source** - represented by <Term /> this data comes directly from external systems, with party that owns the dataset having complete authority over the data and fully accountable for its veracity
* **Derivative** - represented by <Term /> this data is produced by transforming and combining other datasets (root or derivative).

In simple terms, given two root datasets `A` and `B` the derivative dataset `C` can be though of as a [pure function](https://en.wikipedia.org/wiki/Pure_function):

```
C = f(A, B)
```

Unlike databases, or most data processing libraries you might've used ODF is not "batch", but **stream-oriented**. So `A`, `B` here are both potentially infinite [data streams](/odf/spec/overview#nature-of-data), and function `f()` is a [streaming (temporal) processing](/odf/spec/overview#stream-processing-model) operation that produces another stream as an output.

Stream processing composition works just like normal functional composition:

```
D = h(C) = h(f(A, B))
F = g(D, E) = g(h(f(A, B)), E)
```

This allows us to build **infinitely complex data processing pipelines** and supply chains in the form of a Directed Acyclic Graph (DAG):

<Diagram alt="Web3 data pipeline" />

Functions being "pure" means that <Term /> don't depend on any information outside of ODF - you cannot call arbitrary APIs or download files as part of such processing.

While this may seem restrictive - this unlocks highly desirable properties like <Term /> of all computations and ability to automatically derive <Term />.

<Info>
  Imagine being able to erase all derivative data in the world and [reconstruct it in identical form](/odf/spec/transformations/processing-model#derivative-data-transience) just from source data. Derivative data in ODF sometimes feels like a superpower.
</Info>

## Creating Derivative Datasets

Derivative datasets are created by specifying the <Schema /> metadata event.

Example:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: my.trading.holdings
  kind: Derivative
  metadata:
    - kind: SetTransform
      # Using one dataset as an input
      inputs:
        - datasetRef: my.trading.transactions
      transform:
        # Using Apache Flink SQL engine to perform cumulative sum streaming aggregation
        kind: Sql
        engine: flink
        query: |
          SELECT
            event_time,
            symbol,
            quantity,
            price,
            settlement,
            sum(quantity) over(partition by symbol order by event_time rows unbounded preceding) as cum_quantity,
            sum(settlement) over(partition by symbol order by event_time rows unbounded preceding) as cum_balance
          FROM `my.trading.transactions`
```

This event is as simple as defining two things:

* `inputs` - data from which datasets will be used as input to this pipeline step
* `transform` - the transformation being performed.

Technically ODF supports [any data processing framework](/odf/spec/transformations/engine-contract). You will see a lot of Streaming SQL in our examples as we believe it currently offers best user experience when defining streaming transformations.

## Execution Model

When you use [`kamu pull`](/cli/commands#kamu-pull) on a derivative dataset the tool will:

1. Start the appropriate <Term />
2. Restore the state of computation from previous <Term /> (if any)
3. Feed the **previously unseen** data from all inputs
4. Write results into a new <Term />
5. Suspend the computation state into a new <Term />
6. Write a new <Term />

Note that query engines **see input data only once**. When performing things like windowed aggregations the intermediate state of computations between the suspends is stored in <Term />.

## Supported Operations

Some typical operations you can perform on datasets (in the order of increasing complexity) are:

* **Map** - performing computation on one record at a time
* **Filter** - deciding whether to include a record in the output based on some condition that considers only one record at a time
* **Aggregation** - combining values of multiple records together based on some grouping function (e.g. a time window)
* **Projection** - an aggregation aimed at reducing dimensionality of data
* **Stream-to-Stream Join** - combining several streams of data based on some predicate and a time window
* **Temporal Table Join** - joining a stream to a projection of another stream that represents "current state" of some evolving data based on the timestamp of individual records.

Please see [Examples](/examples) for detailed explanation of each of these classes of operations.

## Consistency

Modern stream processing frameworks should be more appropriately called *"temporal processing frameworks"* as they account for time of records and all kinds of time-related problems during the computations.

Transparently to the user streaming can handle situations like:

* Late arrivals and backfills
* Out-of-order arrivals
* Phase shift and differences in data arrival cadences when joining several streams

Unlike batch that has no notion of time, streaming is a **more complete processing model** that provides you complete **control over the tradeoff between consistency and latency**.

<Info>
  Stream processing to batch is like what calculus is to algebra!
</Info>

See [Kamu Blog: The End of Batch Era](https://www.kamu.dev/blog/end-of-batch-era/) to learn more.

## Self-correcting Nature

While we mostly talk about data streams from perspective of observing new events and adding new records - there are many situations when events that were already published may be deemed incorrect and need to be **retracted or corrected**.

We [previously discussed](/cli/ingest/merge-strategies#snapshot) retractions and corrections in the context of <Term />, but the most amazing thing about stream processing is that it **can automatically react to retractions and corrections in inputs** to issue appropriate retractions or corrections in the output.

This means that whenever a major issue is detected and fixed in some <Term /> - these events can **automatically and rapidly propagate** through hundreds of data pipeline stages with no human involvement.

<Tip>
  Imagine you publish a research paper based on some medical dataset. Few months passes and the hospital that published the dataset notices and corrects a major inconsistency in data. By linking your paper to a certain range within the source dataset you will be able to get an instant **notification** that your results may be no longer valid. And if your analysis is also stored as an ODF dataset - it can be **instantly updated** accounting for all corrections.
</Tip>

See [Retractions & Corrections](/cli/transform/retractions-corrections) section for a deeper look.


# Stream-to-Stream Joins
Source: https://docs.kamu.dev/cli/transform/joins-s2s





# Temporal Table Joins
Source: https://docs.kamu.dev/cli/transform/joins-tt





# Projections
Source: https://docs.kamu.dev/cli/transform/projections





# Retractions & Corrections
Source: https://docs.kamu.dev/cli/transform/retractions-corrections



## Motivation

Errors in source data are inevitable and require a mechanism for correcting them post factum. Unlike databases, where one could issue `DELETE` or `UPDATE` queries, our core data model is an [immutable append-only stream](/odf/spec/overview#nature-of-data), and thus requires a different mechanism to issue **retractions and corrections** for past events.

In cases when stream processing operations encounter late data (beyond the current <Term />), or retractions and corrections in the input, they may also need to issue corrections or retractions for previously produced results that were influenced by these events.

We consider the correction / retraction model that works across all stages of data pipelines essential to making processing **maximally autonomous**. In turn, only by making data processing autonomous can collaborative data pipelines be **sustainable at global scale**.

## Scenario

Consider a video game that writes events to a `match_scores` dataset as players complete the matches:

| match\_time | match\_id | player\_name | score |
| -: | -: | -: | -: |
| t1 | 1 | Alice | 100 |
| t1 | 1 | Bob | 80 |
| t2 | 2 | Alice | 70 |
| t2 | 2 | Charlie | 90 |
| t3 | 3 | Bob | 60 |
| t3 | 3 | Charlie | 110 |

Let's say we want to create a dataset that continuously tracks the **top two** (for simplicity) highest-scoring players - a **leader board**.

This is the case of [Top-N query](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/topn/) and [Window Top-N](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-topn/), which can be written in Apache Flink SQL as:

```sql theme={null}
select
  *
from (
  select
    ROW_NUMBER() OVER (ORDER BY score desc) AS place,
    match_time,
    player_name,
    score,
  from match_scores
) where place <= 2
```

## Representing changes in streams

There are several ways to represent how this query behaves when applied to the above input stream.

### Retract Stream

The most generic way is a "retract stream" that only uses append `+A` and retract `-R` operations (note the new `op` column):

| op | place | match\_time | player\_name | score |
| -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| -R | 2 | t1 | Bob | 80 |
| +A | 2 | t2 | Charlie | 90 |
| -R | 1 | t1 | Alice | 100 |
| -R | 2 | t2 | Charlie | 90 |
| +A | 1 | t3 | Charlie | 110 |
| +A | 2 | t3 | Alice | 100 |

In this model the updated state of the leader board is compared with the previous state after each new event, and necessary records first get retracted before being replaced with appends.

### Upsert Stream

Using the knowledge that the `place` column plays the role of a unique key of the resulting state we could also represent the above as an "upsert stream" using only upsert `+A` and retract `-R` operations:

| op | place | match\_time | player\_name | score |
| -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| +A | 2 | t2 | Charlie | 90 |
| +A | 1 | t3 | Charlie | 110 |
| +A | 2 | t3 | Alice | 100 |

This additional knowledge allows us to significantly compact the stream.

Although the retract operation does not appear in our example, it is needed for completeness, e.g. imagine if the source stream retracted the result of the first match right after `t1` - this would require us to empty the leader board with retractions too.

### Changelog Stream (single event)

Some systems produce "changelog streams" containing append `+A`, retract `-R`, and correct `+C` operations with update carrying both the new values and the old values of the record being changed:

| op | place | match\_time | player\_name | score | match\_time\_old | old\_player\_name | old\_score |
| -: | -: | -: | -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 | | | |
| +A | 2 | t1 | Bob | 80 | | | |
| +C | 2 | t2 | Charlie | 90 | t1 | Bob | 80 |
| +C | 1 | t3 | Charlie | 110 | t1 | Alice | 100 |
| +C | 2 | t1 | Alice | 100 | t2 | Charlie | 90 |

This format is also used by CDC systems like [Debezium](https://debezium.io/) and as an internal data representation in [Arroyo](https://github.com/ArroyoSystems/arroyo).

This format is the most "informative" one, as it differentiates retractions from corrections, and provides access to both the new and the previous state within one event. The drawback is that it significantly impacts the schema to essentially allow carrying two events (old and new) in one record.

### Changelog Stream (two events)

The Apache Flink's "changelog streams" variant is using append `+A`, retract `-R`, correct-from `-C`, and correct-to `+C` operations. Here, the "correct-from" events carry the previous values of the record about to be updated and "correct-to" events carry the new values, with the restriction that these events must **always appear side by side** and in order.

| op | place | match\_time | player\_name | score |
| -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| -C | 2 | t1 | Bob | 80 |
| +C | 2 | t2 | Charlie | 90 |
| -C | 1 | t1 | Alice | 100 |
| +C | 1 | t3 | Charlie | 110 |
| -C | 2 | t2 | Charlie | 90 |
| +C | 2 | t1 | Alice | 100 |

By splitting the update operation in two events this format does not require extending the schema with multiple columns.

Due to the restriction that correct from/to events appear side by side, this format can be easily converted into single-event changelog stream form upon reading.

## ODF Model

ODF uses the [two-event changelog stream](#changelog-stream-two-events) as its base model.

Just like in our example above a special `op` column carries the <Term /> of each record.

Retractions and corrections are supported on <Term /> level via [merge strategies](/cli/ingest/merge-strategies).

We are currently working to integrate their support into all processing engines for seamless propagation across <Term />.


# Watermarks
Source: https://docs.kamu.dev/cli/transform/watermarks





# Troubleshooting
Source: https://docs.kamu.dev/cli/troubleshooting

Diagnosing the errors and reporting issues.

Here are some tips on diagnosing the errors:

## Common Environment Issues

This command can be used to verify your environment setup:

```sh theme={null}
kamu system diagnose
```

```sh Example response theme={null}
docker installed .. ok
docker can pull images .. ok
docker can run without sudo .. ok
docker can mount volumes .. ok
workspace consistency check .. ok
...
```

## Standard Logging

When `kamu` runs, it logs into `.kamu/run` directory. This directory is **cleaned up on every run**, so if you encounter an error make sure to copy important logs before running next `kamu` command.

## Verbose Logging

When encountering an error that is not descriptive enough you may want to see more debugging information by using `kamu -v <command>` flag. This flag will redirect `kamu` log output into the terminal (while removing all UI widgets for clarity) and will enable backtrace information on errors. Please use this mode when submitting bugs on this repo.

## Tracing

Using `kamu --trace` flag allows you to record the execution of the program and open [Perfetto UI](https://perfetto.dev/) in a browser, allowing to easily analyze async code execution and task performance.

> Note: If you are using Brave or a similar high-security browser and get an error from Perfetto when loading the trace - try disabling the security features to allow the UI app fetch data from `http://localhost:9001`.

<Diagram alt="Perfetto UI displaying a trace" />

## Engine Errors

In this early stages of development `kamu` does a minimal amount of error reporting from engines, so you might definitely encounter some issues that will require an ability to read exception information logged by the engines.

When `kamu` runs an engine it redirects its logs into `.kamu/run` directory. If you get a cryptic "engine operation failed" error you will need to inspect the logs in that directory for some clues on why your query failed.

Bear with us while we are working on improving the error reporting and introducing more guard rails to prevent errors in the first place.

## Reporting Issues

If you could not resolve the error yourself - please file an issue in our [GitHub repository](https://github.com/kamu-data/kamu-cli).


# COVID-19 Daily Cases
Source: https://docs.kamu.dev/examples/covid19



**Topics covered:**

* Harmonization
* Unions
* Aggregations
* Notebooks

## Summary

This example currently exists as part of the [online tutorial](/start/tutorial).

While we're working on the text version try the demo and watch Kamu 101 playlist:

<YouTubeList />

The dataset files can be found in the [examples/covid](https://github.com/kamu-data/kamu-cli/tree/master/examples/covid) directory.


# Currency Conversion
Source: https://docs.kamu.dev/examples/currency-conversion



**Topics covered:**

* Temporal table joins

## Summary

Currency conversion is a great example of a problem that cannot be solved without considering the time dimension. As exchange rates vary significantly every day using an average rate for a certain period is a gross oversimplification of a problem that can give very inaccurate results. This example shows how `kamu` and streaming SQL can be used to join your overseas trading account data with an exchange rate stream to bring your calculations to the common currency denominator.

## Steps

### Getting Started

To follow this example checkout `kamu-cli` repository and navigate into [examples/currency\_conversion](https://github.com/kamu-data/kamu-cli/tree/master/examples/currency_conversion) sub-directory.

Create a temporary kamu <Term /> in that folder using:

```sh theme={null}
kamu init
```

You can either follow the example steps below or fast-track through it by running:

```sh theme={null}
kamu add --recursive .
kamu pull --all
```

### Root Datasets

We will be using two root datasets:

* `ca.bankofcanada.exchange-rates.daily` - contains daily exchange rates between `USD` and `CAD`.
* `my.trading.transactions` - contains a transaction ledger from a fake trading account similar to what you'd get in data export from your trading platform.

The `my.trading.transactions` is sourcing its data from a file located in `data/` sub-directory, while the exchange rates are obtained from an external source - the [Bank of Canada historical exchange rates](https://www.bankofcanada.ca/rates/exchange/) dataset.

Let's add the <Term /> and ingest data:

```sh theme={null}
kamu add ca.bankofcanada.exchange-rates.daily.yaml my.trading.transactions.yaml
kamu pull --all
```

You can verify the result using `tail` command or the SQL shell:

```bash theme={null}
kamu tail ca.bankofcanada.exchange-rates.daily
```

Or explore the dataset via SQL shell:

```bash theme={null}
kamu sql
```

```sql theme={null}
select * from "ca.bankofcanada.exchange-rates.daily" limit 5;
```

```sql theme={null}
+-------------+------------+---------------+-----------------+--------------+
| system_time |    date    | currency_base | currency_target |     rate     |
+-------------+------------+---------------+-----------------+--------------+
|     ...     | 1999-01-04 |      USD      |       CAD       | 1.5271863602 |
|     ...     | 1999-01-05 |      USD      |       CAD       | 1.5237489398 |
|     ...     | 1999-01-06 |      USD      |       CAD       | 1.5082176616 |
|     ...     | 1999-01-07 |      USD      |       CAD       | 1.5132393398 |
|     ...     | 1999-01-08 |      USD      |       CAD       | 1.5132515653 |
+-------------+------------+---------------+-----------------+--------------+
```

Remember, that after adding a dataset from a `yaml` file `kamu` creates an internal representation of it in the <Term /> `.kamu` directory, so **to make any changes** to the dataset you will need to **re-add** the dataset again after changing the `yaml` file.

```sh theme={null}
kamu delete my.trading.transactions
# <Make changes to the yaml file>
kamu add my.trading.transactions.yaml

# Alternatively
kamu add --replace my.trading.transactions.yaml
```

### Converting transaction values into CAD

If you look at `my.trading.transactions` dataset - it contains the trading activity of a US account, so all prices and settlement amounts there are listed in US Dollars (`USD`). But let's imagine that you live in Canada and all of your other accounts and your credit cards are all in Canadian Dollars (`CAD`). It would be a lot easier for you to monitor your financials if they were in one currency.

If you analyze your financials on a quarterly or yearly basis you might've dealt with this problem by looking up an **average exchange rate** for a certain time period and used it in your calculations. This is of course an **oversimplification** that may give **misleading** results - your transactions are most likely not distributed uniformly throughout the time period, and if you live in a country with volatile economy or trading crypto - the exchange rate may fluctuate significantly.

Ideally we would like to take **every single transaction** and convert the price using the exchange rate for the corresponding date. Thankfully, `kamu` can make this process very easy using the **temporal table joins**.

Temporal table joins (see this [great explanation in Apache Flink's blog](https://flink.apache.org/2019/05/14/temporal-tables.html)) take one of the streams and represent it as a three-dimensional table. When joining to such a table you need to pass in the time argument to tell join to consider the version of the table that would exist at that point in time.

Let's have a look at the `my.trading.transactions.cad` dataset:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: my.trading.transactions.cad
  kind: Derivative
  metadata:
    - kind: SetTransform
      inputs:
        - datasetRef: ca.bankofcanada.exchange-rates.daily
        - datasetRef: my.trading.transactions
      transform:
        kind: Sql
        engine: flink
        temporalTables:
          - name: ca.bankofcanada.exchange-rates.daily
            primaryKey:
              - currency_base
        query: |
          SELECT
            tr.`event_time`,
            tr.`symbol`,
            tr.`quantity`,
            tr.`price` as `price_usd`,
            tr.`price` * exc.`rate` as `price_cad`,
            tr.`settlement` as `settlement_usd`,
            tr.`settlement` * exc.`rate` as `settlement_cad`
          FROM `my.trading.transactions` as tr
          LEFT JOIN `ca.bankofcanada.exchange-rates.daily` FOR SYSTEM_TIME AS OF tr.`event_time` as exc
          ON tr.`currency` = exc.`currency_base` AND exc.`currency_target` = 'CAD'
```

<Note>
  The excessive use of back ticks is currently caused by the SQL parser used by Apache Flink which is overly sensitive to reserved words - this should improve in future versions.
</Note>

Using the `temporalTables` section we instruct the Flink engine to use `ca.bankofcanada.exchange-rates.daily` event stream to create a temporal table of the same name.

Here's how this three-dimensional table would look like when sliced at different points in time:

```sql theme={null}
ca.bankofcanada.exchange-rates.daily:

@ 1999-01-04
+---------------+-----------------+--------------+
| currency_base | currency_target |     rate     |
+---------------+-----------------+--------------+
|      USD      |       CAD       | 1.5271863602 |
+---------------+-----------------+--------------+

@ 1999-01-05
+---------------+-----------------+--------------+
| currency_base | currency_target |     rate     |
+---------------+-----------------+--------------+
|      USD      |       CAD       | 1.5237489398 |
+---------------+-----------------+--------------+

@ 1999-01-06
+---------------+-----------------+--------------+
| currency_base | currency_target |     rate     |
+---------------+-----------------+--------------+
|      USD      |       CAD       | 1.5082176616 |
+---------------+-----------------+--------------+
```

It basically remembers the last observed value of every column grouped by the provided `primaryKey` (`currency_base` in our case).

The `LEFT JOIN ca.bankofcanada.exchange-rates.daily FOR SYSTEM_TIME AS OF tr.event_time` part can be interpreted as us taking every transaction event from `my.trading.transactions`, indexing the temporal table `ca.bankofcanada.exchange-rates.daily` at this event's `event_time` and then joining the same event with the resulting (now ordinary two-dimensional) table.

<Note>
  Note that `SYSTEM_TIME AS OF` syntax is a relic of SQL:2011 standard and should not be confused with Kamu's `system_time` column. The actual join is performed in the **<Term />**. You can learn more about temporal event time joins in [Flink documentation](https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/table/sql/queries/joins/#temporal-joins).
</Note>

With theory out of the way, it's time to give this a try:

```sh theme={null}
kamu add my.trading.transactions.cad.yaml
kamu pull my.trading.transactions.cad
```

The results should be:

```bash theme={null}
kamu tail my.trading.transactions.cad
```

```sql theme={null}
+-------------+------------+--------+----------+-----------+---------------------+
| system_time | event_time | symbol | quantity | price_usd |      price_cad      |
+-------------+------------+--------+----------+-----------+---------------------+
|     ...     | 2016-01-04 | SPY    | 1        | 201.0200  | 279.85643605283600  |
|     ...     | 2016-02-01 | SPY    | 1        | 193.6500  | 271.38399945165000  |
|     ...     | 2016-03-01 | SPY    | 1        | 198.1100  | 267.84574042498800  |
|     ...     | 2016-04-01 | SPY    | 1        | 206.9200  | 269.58244227661600  |
|     ...     | 2016-05-02 | SPY    | 1        | 207.9700  | 260.35607413671100  |
+-------------+------------+--------+----------+-----------+---------------------+
```

The best thing about this is that *you may never have to touch this SQL query again*. Each time you run `kamu pull` in the future the latest transaction data will be ingested along with latest exchange rates, producing new transactions with converted prices. This is the "write once - run forever" philosophy of `kamu` that combines best propagation times with the accuracy of solving temporal problems without taking any shortcuts.

Once you master this example - make sure to check out the [Stock Market Trading example](/examples/stock-trading) that introduces another really important mechanism related to temporal joins - the **<Term />**.


# Housing prices
Source: https://docs.kamu.dev/examples/housing-prices



**Topics covered:**

* GIS functions
* Spatial joins
* Notebooks

## Summary

Geospatial data is so widespread that no data science software would be complete without ability to manipulate it efficiently. In this example we will take a look how GIS data can be used in `kamu` using `Apache Spark` engine that comes with [Apache Sedona](http://sedona.apache.org/) extension.

## Steps

### Getting Started

To follow this example checkout `kamu-cli` repository and navigate into [examples/housing\_prices](https://github.com/kamu-data/kamu-cli/tree/master/examples/housing_prices) sub-directory.

### Root Datasets

We will be using following root datasets:

* `ca.vancouver.opendata.property.tax-reports` - contains property assessment reports from the city of Vancouver
* `ca.vancouver.opendata.property.parcel-polygons` - contains geographical outlines of properties
* `ca.vancouver.opendata.property.block-outlines` - contains geographical block outlines
* `ca.vancouver.opendata.property.local-area-boundaries` - contains outlines and names of Vancouver's districts

For geospatial data `kamu` supports several input formats including [ESRI Shapefile](https://en.wikipedia.org/wiki/Shapefile) and [GeoJson](https://geojson.org/). Take a look at dataset definitions to understand how data is being ingested.

Quite often when ingesting GIS data you will need to deal with different projections. Example below shows how you can use a pre-processing step to convert data into another projection:

```yaml theme={null}
preprocess:
  kind: Sql
  engine: spark
  query: |
    SELECT
      ST_Transform(geometry, "epsg:3157", "epsg:4326") as geometry
    FROM input
```

> Note: [epsg:4326](https://epsg.io/4326) is one of the more commonly used projections which is expected by many visualization tools.

The `ST_Transform` function here comes from [Apache Sedona](http://sedona.apache.org/) extension for the Apache Spark engine for working with GIS data. We will be relying on many of its function throughout this example.

Lets add all these datasets to our workspace and ingest them:

```sh theme={null}
# Create a kamu workspace in the example folder
kamu init

# Add all dataset manifests found in the current directory
kamu add --recursive .
kamu pull --all

# Alternatively: Pull datasets in ODF format from Kamu's repository
./init-s3.sh
```

<Warning>
  The source data for `tax-reports` dataset is quite large (\~400MB in CSV format) so if you don't want to wait too long - use the `init-s3.sh` script.
</Warning>

This may take 10+ minutes as the tax report dataset is quite large. You can leave the ingest running and move on to the next step meanwhile.

### Creating MapBox Account

For visualizing the GIS data we will be using the awesome [Mapbox SDK](https://www.mapbox.com/). To follow the example you will need to create a free account and get your own access token. The token should look something like `pk.eyJ...T6Q`.

### Visualizing Housing Prices

Once you have downloaded all datasets we can start with a simple notebook to get a feel for GIS visualization.

Start the notebook server and pass it your Mapbox token via environment variable:

```sh theme={null}
kamu notebook -e MAPBOX_ACCESS_TOKEN=<your mapbox token>
```

Now you can open the `heatmap.ipynb` and run through it executing each step. The notebook is pretty straightforward - it joins together the `tax-reports` dataset containing assessment values with `parcel-polygons` dataset containing land geometries in order to produce a heatmap.

```sql theme={null}
%%sql
CREATE OR REPLACE TEMP VIEW lot_tax AS (
  SELECT
      t.*,
      l.geometry
  FROM lots as l
  JOIN tax as t
  ON l.tax_coord = t.land_coordinate
  WHERE
      t.legal_type = 'LAND'
      AND t.tax_assessment_year = 2019
      AND t.current_land_value is not null
)
```

It then pulls the result from Spark into the notebook itself using this query:

```sql theme={null}
%%sql -o df -n 100000
SELECT
    land_coordinate,
    ST_AsGeoJSON(geometry) as geometry,
    current_land_value + current_improvement_value as current_total_value
FROM lot_tax
```

The `-n 100000` parameter here overrides the default limit on the dataframe size, while `ST_AsGeoJSON(geometry)` also converts the internal binary `geometry` type into the GeoJson format. We later use custom-written `df_to_geojson` to combine individual geometries into `GeoJson`'s `FeatureCollection` objectm understood by the `mapboxgl` library.

We can then use the resulting GeoJson data and render the heatmap using the [mapboxgl-jupyter](https://github.com/mapbox/mapboxgl-jupyter) library which comes pre-installed with `kamu`'s notebook server image.

<Diagram alt="House prices heatmap" />

### Spatial Joins

Let's take a look how datasets can be joined using the geometries. The `local-area-boundaries` contains the names of the Vancouver's neighbourhoods along with their geographical boundaries. We also have the `block-outlines` dataset containing boundaries of individual blocks, but there's an issue - there is no identifier that would link a block to a specific neighbourhood. For us to associate a block with the neighbourhood we have to see if block's boundaries are contained or intersect the boundaries of some neighbourhood. `Apache Sedona` used by `kamu` optimizes such spatial joins.

You can open up the `spatial_join.ipynb` notebook and run through it step by step. The most important part ther is the join itself:

```sql theme={null}
CREATE OR REPLACE TEMP VIEW blocks_by_hood AS (
    SELECT h.name, b.geometry
    FROM blocks as b
    INNER JOIN hoods as h
    ON ST_Intersects(b.geometry, h.geometry)
)
```

The rest of the visualization steps should be familiar to you from the previous example.

As an exercise - see if you can replace the `median_value` column that we have populated with `RAND()` for every block with an actual median property price. Just like before, there is unfortunately no identifier that would tie the property to a specific block, but see if `Sedona` can handle a massive spatial join between blocks and individual property outlines.


# Examples
Source: https://docs.kamu.dev/examples/index

Sample data projects executed with kamu

## Guided Examples

| Name | Level | Topics |
| - | :-: | :-: |
| [Self-serve Tutorial](/start/tutorial) | Beginner | basic commands |
| [Currency Conversion](/examples/currency-conversion) | Beginner | temporal-table joins |
| [COVID-19 Daily Cases](/examples/covid19) | Beginner | harmonization, unions, tumbling window aggregations, notebooks |
| [Leaderboard](/examples/leaderboard) | Beginner | top-n aggregations, push ingest |
| [Housing Prices](/examples/housing-prices) | Beginner | GIS data, GIS functions, GIS joins, notebooks |
| [Stock Market Trading](/examples/stock-trading) | Intermediate | aggregations, temporal-table joins, watermarks, notebooks |
| [Overdue Order Shipments](/examples/overdue-shipments) | Intermediate | stream-to-stream joins, watermarks |
| [Web3 Data](/examples/web3-data) | Intermediate | repositories, temporal-table joins, stream processing, watermarks |

To work with examples we recommend you to clone the [kamu-cli repo](https://github.com/kamu-data/kamu-cli) and use directories in `examples/` as your workspaces.

For instance:

```bash theme={null}
git clone https://github.com/kamu-data/kamu-cli.git
cd kamu-cli/examples/currency_conversion

kamu init
kamu add . -r
kamu pull --all
```

## Miscellaneous

| Name | Description |
| - | - |
| [MQTT ingest](https://github.com/kamu-data/kamu-cli/tree/master/examples/mqtt) | Shows how to ingest MQTT data using publicly available Mosquitto broker |
| [FlightSQL API](https://github.com/kamu-data/kamu-cli/tree/master/examples/flight-sql) | Examples of connecting to Kamu via FlightSQL protocol using a variety of different libraries |

## Need More Examples?

The wild variety of formats and ways to model data used by the data publishers today often makes ingesting data a challenge. When you work with the data that is already in the ODF network you feel like a productive data scientist, but until publishers start using this format directly we'll have to continue wasting time on dull tasks like parsing CSVs, inferring schemas, and cleaning data.

To give you more examples on how to deal with **different ingest scenarios** we've created an [experimental repository](https://github.com/kamu-data/kamu-contrib) where we publish dataset manifests for a variety of Open Data sources.

<Tip>
  Stuck on parsing a yet another variety of timestamps? Confused on how to change projection of the GIS data when reading ESRI file? Then check out [`kamu-contrib`](https://github.com/kamu-data/kamu-contrib) repo.
</Tip>


# Leaderboard
Source: https://docs.kamu.dev/examples/leaderboard



**Topics covered:**

* Top-N aggregation
* Retractions / corrections
* Push ingest

## Summary

Suppose we have a game where players compete in matches against one another and get assigned an overall match score. In this example we will see:

* How to create a dataset that represents a leaderboard of top-scoring players in the game using "Top-N" class of queries
* How such dataset tracks the entire history of changes
* And how "Top-N" queries manifest into a mechanism called retractions and corrections.

## Steps

### Getting Started

To follow this example checkout `kamu-cli` repository and navigate into [examples/leaderboard](https://github.com/kamu-data/kamu-cli/tree/master/examples/leaderboard) sub-directory.

Create a temporary kamu <Term /> in that folder using:

```sh theme={null}
kamu init
```

You can either follow the example steps below or fast-track through it by running:

```sh theme={null}
./init.sh
```

### Scores Dataset

We start with the `player_scores` dataset that will ingest the scores of all players who played a match. It will have a very simple schema:

* `match_time: Timestamp`
* `match_id: Int64`
* `player_id: String`
* `score: Int64`

Our game will be sending scores data directly to the <Term /> hosting this dataset, so we will use <Schema /> metadata event to describe the format that data will come in.

We'll use <Schema />:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: player-scores
  kind: Root
  metadata:
    - kind: AddPushSource
      sourceName: default
      read:
        kind: NdJson
        schema:
          fields:
            - name: match_time
              type: Timestamp
            - name: match_id
              type: Int64
            - name: player_id
              type: String
            - name: score
              type: Int64
      merge:
        kind: Ledger
        primaryKey:
          - match_id
          - player_id
    - kind: SetVocab
      eventTimeColumn: match_time
```

Create the dataset now:

```sh theme={null}
kamu add player-scores.yaml
```

To get data in we will ingest it from a file, but note that it could similarly be any other ["push" protocol](/cli/ingest/push-source) like REST, MQTT, Kafka...

Let's add the first match scores in:

```sh theme={null}
kamu ingest player-scores ./data/1.ndjson
```

You can see the new data via:

```sh theme={null}
kamu tail player-scores
```

```
┌────────┬────┬─────┬──────────┬───────────┬───────┐
│ offset │ op │ ... │ match_id │ player_id │ score │
├────────┼────┼─────┼──────────┼───────────┼───────┤
│      0 │ +A │ ... │        1 │     Alice │   100 │
│      1 │ +A │ ... │        1 │       Bob │    80 │
└────────┴────┴─────┴──────────┴───────────┴───────┘
```

### Leaderboard Dataset

Having all the historical scores we could already build a leaderboard using batch computations as simple as:

```sh theme={null}
kamu sql
```

```sql theme={null}
select
  *
from (
  select
    row_number() over (order by score desc) as place,
    match_time,
    match_id,
    player_id,
    score
  from "player-scores";
) where place <= 2
```

But there are **many benefits** to expressing it as a streaming pipeline, which we will discuss shortly.

For now let's create a streaming leaderboard dataset. Luckily the query is almost identical to the batch variant:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: leaderboard
  kind: Derivative
  metadata:
    - kind: SetTransform
      inputs:
        - datasetRef: player-scores
          alias: player_scores
      transform:
        kind: Sql
        engine: risingwave
        query: |
          select
            *
          from (
            select
              row_number() over (partition by 1 order by score desc) as place,
              match_time,
              match_id,
              player_id,
              score
            from player_scores
          )
          where place <= 2
    - kind: SetVocab
      eventTimeColumn: match_time
```

Create this dataset:

```sh theme={null}
kamu add leaderboard.yaml
```

Let's "pull" the `leaderboard` dataset to process all the data that is already in `player-scores` and check the results:

```sh theme={null}
kamu pull leaderboard
kamu tail leaderboard
```

```
┌────────┬────┬─────┬───────┬──────────┬───────────┬───────┐
│ offset │ op │ ... │ place │ match_id │ player_id │ score │
├────────┼────┼─────┼───────┼──────────┼───────────┼───────┤
│      0 │ +A │ ... │     1 │        1 │     Alice │   100 │
│      1 │ +A │ ... │     2 │        1 │       Bob │    80 │
└────────┴────┴─────┴───────┴──────────┴───────────┴───────┘
```

### Introducing Retractions

Suppose we played another match with scores like these:

```
Alice: 70
Charlie: 90
```

Let's see what happens to the `leaderboard` dataset:

```sh theme={null}
kamu ingest player-scores ./data/2.ndjson
kamu pull leaderboard
kamu tail leaderboard
```

Pay attention to the `op` column:

```
┌────────┬────┬─────┬───────┬──────────┬───────────┬───────┐
│ offset │ op │ ... │ place │ match_id │ player_id │ score │
├────────┼────┼─────┼───────┼──────────┼───────────┼───────┤
│      0 │ +A │ ... │     1 │        1 │     Alice │   100 │
│      1 │ +A │ ... │     2 │        1 │       Bob │    80 │
│      2 │ -R │ ... │     2 │        1 │       Bob │    80 │
│      3 │ +A │ ... │     2 │        2 │   Charlie │    90 │
└────────┴────┴─────┴───────┴──────────┴───────────┴───────┘
```

Now this is very interesting:

* We still see the history of Alice and Bob being ranked 1 and 2 initially
* Then we see a "retraction" (`-R`) record that says "Bob is no longer ranked 2nd"
* And an "append" (`+A`) record that says "Charlie is now ranked 2nd"

If we repeat this again with scores:

```
Bob: 60
Charlie: 110
```

As usual:

```sh theme={null}
kamu ingest player-scores ./data/3.ndjson
kamu pull leaderboard
kamu tail leaderboard
```

As Alice was displaced by Charlie from the 1st place we see (starting from row `offset: 4`) how both records are retracted and replaced by new values:

```
┌────────┬────┬─────┬───────┬──────────┬───────────┬───────┐
│ offset │ op │ ... │ place │ match_id │ player_id │ score │
├────────┼────┼─────┼───────┼──────────┼───────────┼───────┤
│      0 │ +A │ ... │     1 │        1 │     Alice │   100 │
│      1 │ +A │ ... │     2 │        1 │       Bob │    80 │
│      2 │ -R │ ... │     2 │        1 │       Bob │    80 │
│      3 │ +A │ ... │     2 │        2 │   Charlie │    90 │
│      4 │ -R │ ... │     1 │        1 │     Alice │   100 │
│      5 │ -R │ ... │     2 │        2 │   Charlie │    90 │
│      6 │ +A │ ... │     1 │        3 │   Charlie │   110 │
│      7 │ +A │ ... │     2 │        1 │     Alice │   100 │
└────────┴────┴─────┴───────┴──────────┴───────────┴───────┘
```

You are witnessing the <Term /> of ODF datasets!

## So, why not Batch?

So what do we gain from having a changelog stream with retractions? Why not run a batch query every once in a while to get the current leaderboard?

The answer is: ***automation and infinite composability***.

As your game evolves, you might want to start rewarding people at top spots with prizes. You may also want to reset the leaderboard each month to make people prove themselves again. All this logic can be expressed as additional steps in the streaming data pipeline.

If you write services that batch-query data at specific points in time and distribute rewards imagine what happens:

* if some player is caught cheating and has to be **retroactively** stripped of its match scores (input retraction)
* if score ingestion pipeline goes down for a day and people who did in fact get top scores end up not accounted for during the reward distribution (backfill)
* if your game is very high-stakes and your have to prove during an audit how the rewards were issued (provenance).

These situations that we call "abnormal" are actually very frequent and in batch world require **highly error-prone manual intervention**.

In the world of ODF streaming - late data, backfills, and input retractions are just normal occurrences. ODF pipelines know how to react to these events and can **automatically reconcile their state** without human intervention, and maintaining a **verifiable provenance**.

***

If you haven't already - make sure to check out the [Stock Market Trading example](/examples/stock-trading) that introduces the **<Term />** - another really important mechanism related to streams.


# Overdue Order Shipments
Source: https://docs.kamu.dev/examples/overdue-shipments



**Topics covered:**

* Stream-to-Stream joins
* Temporal aggregations
* Watermarks

## Summary

In this example we will take a look at an imaginary e-commerce company `acme.com` committed to shipping their products to customers as soon as possible. To maintain their level of service they have decided to create an alerting system using `kamu` that would notify them if any order did not ship within a certain time window, so that they could investigate the root cause of the delay.

## Steps

### Getting Started

To follow this example checkout `kamu-cli` repository and navigate into [examples/overdue\_shipments](https://github.com/kamu-data/kamu-cli/tree/master/examples/overdue_shipments) sub-directory.

Create a temporary kamu workspace in that folder using:

```sh theme={null}
kamu init
```

You can either follow the example steps below or fast-track through it by running:

```sh theme={null}
# Add all dataset manifests found in the current directory
kamu add -recursive .
kamu pull --all
```

### Root Datasets

We will be using two root datasets:

* `com.acme.orders` - contains a log of orders as they are being placed in the system. To make it more interesting we will assume that each order can request different quantity of "the product" that can be shipped to the customer in one or several shipments.
* `com.acme.shipments` - contains a log of shipments as they are being dispatched to the customers. Every shipment links to the original order and specifies the quantity of "the product" that has been shipped.

Both datasets are sourcing their data from files located in `data/` sub-directory. Once comfortable with this example you can try adding more data in `orders-xxx.csv` and `shipments-xxx.csv` to see how `fileGlob` data sources work and test the queries with any edge cases you can come up with.

Let's add the root datasets and ingest the data:

```sh theme={null}
kamu add com.acme.orders.yaml com.acme.shipments.yaml
kamu pull --all
```

You can verify the result using `tail` command or the SQL shell:

```bash theme={null}
kamu sql
```

```sql theme={null}
0: kamu> select * from `com.acme.orders`;
+-------------+------------+-----------+-----------+
| system_time | event_time | order_id  | quantity  |
+-------------+------------+-----------+-----------+
|     ...     | 2020-01-01 | 1         | 10        |
|     ...     | 2020-01-01 | 2         | 120       |
|     ...     | 2020-01-05 | 3         | 9         |
|     ...     | 2020-01-10 | 4         | 110       |
+-------------+------------+-----------+-----------+
4 rows selected (0.11 seconds)

0: kamu> select * from `com.acme.shipments`;
+-------------+------------+--------------+-----------+-----------+
| system_time | event_time | shipment_id  | order_id  | quantity  |
+-------------+------------+--------------+-----------+-----------+
|     ...     | 2020-01-01 | 1            | 1         | 4         |
|     ...     | 2020-01-02 | 3            | 2         | 120       |
|     ...     | 2020-01-02 | 2            | 1         | 6         |
|     ...     | 2020-01-08 | 4            | 3         | 9         |
|     ...     | 2020-01-11 | 5            | 4         | 50        |
|     ...     | 2020-01-13 | 6            | 4         | 60        |
+-------------+------------+--------------+-----------+-----------+
```

### Joining Shipments with Orders

The goal of the `com.acme.shipments.overdue` dataset that we will be creating is to detect orders that have not been fulfilled at all or were fulfilled only partially within a specified time interval - let's say `2 DAYS`. We want the alert to go off as soon such condition occurs - there's no use investigating the root cause of the late order when few weeks have already passed since then.

We can define our goal as: an order placed in the system is considered "late" if during the interval of `2 DAYS` after the order has been placed the total quantity of product shipped doesn't equal the quantity that has been ordered.

First we need to get these two quantities somehow in one dataset, so let's join the orders and shipments data using `order_id` as the key:

```sql theme={null}
-- Using Flink SQL dialect
SELECT
  o.event_time as order_time,
  o.order_id,
  o.quantity as order_quantity,
  CAST(s.event_time as TIMESTAMP) as shipped_time,
  COALESCE(s.quantity, 0) as shipped_quantity
FROM `com.acme.orders` as o
LEFT JOIN `com.acme.shipments` as s
ON
  o.order_id = s.order_id
  AND s.event_time BETWEEN o.event_time AND o.event_time + INTERVAL '2' DAY
```

This is a Stream-to-Stream join which uses two join conditions.

Firstly, we match orders and shipments on `order_id` - this is pretty standard and something you would do in a regular SQL with non-temporal data. If we only have this condition the query would already work, but since `kamu` doesn't know how far apart in time the orders and shipments can be it will expect that order `1` can match a shipment that happens a century or even a billion years from now, so it will have to keep track of all the orders forever.

To prevent that we add another condition that restricts the `shipment.event_time` to a time interval `[order.event_time, order.event_time + 2 DAYS)`. This tells `kamu` that all orders that are older than two days can be forgotten since they will never match this condition.

Also note that we are especially interested in orders that had no shipments at all, so we are using `LEFT JOIN` that will produce a record even if nothing on the right side has matched our criteria. In this case all `s.*` fields will be `NULL`, so we are using `COALESCE` to replace `NULL` values with a zero when computing `shipped_quantity`.

Also note that we need to `CAST` the `s.event_time` to `TIMESTAMP`. This is a specific of Flink SQL which treats event time of stream as a special type. Every stream is allowed to have only one event time so when two streams are joined we need to cast one of the times to a plain `TIMESTAMP` type.

### Computing total quantity of shipped product

The query above will produce a new event for every shipment. Since one order can have multiple shipment events and we're only interested in total quantity shipped - we need to aggregate them.

```sql theme={null}
SELECT
  TUMBLE_START(order_time, INTERVAL '1' DAY) as order_time,
  order_id,
  count(*) as num_shipments,
  min(shipped_time) as first_shipment,
  max(shipped_time) as last_shipment,
  max(order_quantity) as order_quantity,
  sum(shipped_quantity) as shipped_quantity_total
FROM order_shipments
GROUP BY TUMBLE(order_time, INTERVAL '1' DAY), order_id
```

Since in our previous query (which we aliased as `order_shipments`) we emit results using `order.event_time` as the event time - all records for the same order will have the same event time. Even when event times match exactly Flink requires us to use some form of windowing in aggregations so we use a 1-day interval and a tumbling window. As long as our interval is less than the time we chose for considering the shipment "late" - we will not delay the emission of the results.

### Detecting late shipments

The final step is very straightforward - we want to emit only records for orders that were not fully fulfilled:

```sql theme={null}
SELECT *
FROM order_shipments_agg
WHERE order_quantity <> shipped_quantity_total
```

Putting all of this together, our final dataset will look like this:

```yaml theme={null}
version: 1
kind: DatasetSnapshot
content:
  name: com.acme.shipments.overdue
  kind: Derivative
  metadata:
    - kind: SetTransform
      inputs:
        - datasetRef: com.acme.orders
        - datasetRef: com.acme.shipments
      transform:
        kind: Sql
        engine: flink
        queries:
          # This is a Stream-to-Stream join that matches orders and shipments on order_id, 
          # restricting the shipment.event_time to a time interval [order.event_time, order.event_time + 2 DAYS).
          # Note that the cast of s.event_time is necessary to tell Flink to no longer treat it as `rowtime`.
          - alias: order_shipments
            query: |
              SELECT
                o.event_time as order_time,
                o.order_id,
                o.quantity as order_quantity,
                CAST(s.event_time as TIMESTAMP(3)) as shipped_time,
                COALESCE(s.quantity, 0) as shipped_quantity
              FROM `com.acme.orders` as o
              LEFT JOIN `com.acme.shipments` as s
              ON
                o.order_id = s.order_id
                AND s.event_time BETWEEN o.event_time AND o.event_time + INTERVAL '2' DAY
          # This is a windowed aggregation that computes the total quantity of shipped product
          - alias: order_shipments_agg
            query: |
              SELECT
                TUMBLE_START(order_time, INTERVAL '1' DAY) as order_time,
                order_id,
                count(*) as num_shipments,
                min(shipped_time) as first_shipment,
                max(shipped_time) as last_shipment,
                max(order_quantity) as order_quantity,
                sum(shipped_quantity) as shipped_quantity_total
              FROM order_shipments
              GROUP BY TUMBLE(order_time, INTERVAL '1' DAY), order_id
          # Return only records for orders that were not fully fulfilled
          - query: |
              SELECT *
              FROM order_shipments_agg
              WHERE order_quantity <> shipped_quantity_total
    - kind: SetVocab
      eventTimeColumn: order_time
```

Add this dataset and let's compute the result:

```sh theme={null}
kamu add com.acme.shipments.overdue.yaml
kamu pull com.acme.shipments.overdue
```

### Reminder about Watermarks

If you look at the resulting data you will notice that we didn't get the alert for `order_id == 4` which shipped in two batches, last of which was late by one day. This is because the watermark of the `orders` dataset did not advance far enough to cover that period. Try setting the watermark manually and then running the processing again:

```sh theme={null}
kamu pull com.acme.orders --set-watermark 2020-01-13T00:00:00Z
kamu pull --all
```

More detailed explanation of watermarks is given in the [Stock Market Trading example](/examples/stock-trading).


# Stock trading
Source: https://docs.kamu.dev/examples/stock-trading



**Topics covered:**

* Temporal aggregations (cumulative sum)
* Temporal table joins
* Watermarks

## Summary

Stock market data analysis is one of the problems where time dimension plays a crucial importance and cannot be left out. This example shows how `kamu` and streaming SQL can be used to build your personal trading performance dashboard which is more powerful than what most trading platforms provide. You will learn how to perform aggregations and join temporal stream and the role of watermarks in this process.

## Steps

### Getting Started

To follow this example checkout `kamu-cli` repository and navigate into [examples/trading](https://github.com/kamu-data/kamu-cli/tree/master/examples/trading) sub-directory.

Create a temporary kamu workspace in that folder using:

```sh theme={null}
kamu init
```

You can either follow the example steps below or fast-track through it by running:

```sh theme={null}
# Add all dataset manifests found in the current directory
kamu add --recursive .
kamu pull --all
```

### Root Datasets

We will be using two root datasets:

* `com.yahoo.finance.tickers.daily` - contains daily price summaries (candles) of several major tickers.
* `my.trading.transactions` - contains a log of transaction from a fake trading account similar to what you'd get in data export from your trading platform.

Both datasets are sourcing their data from files located in `data/` sub-directory. Once comfortable with this example you can explore sourcing ticker data from external financial APIs (e.g. as name suggests from `finance.yahoo.com`) and using your own trading transaction history.

Let's add the root datasets and ingest data:

```sh theme={null}
kamu add com.yahoo.finance.tickers.daily.yaml my.trading.transactions.yaml
kamu pull --all
```

You can verify the result using `tail` command or the SQL shell:

```sh theme={null}
kamu sql
```

```sql theme={null}
0: kamu> select * from `com.yahoo.finance.tickers.daily` limit 5;
+-------------+------------+---------+------------+-----------+-----------+-----------+-----------+-----------------+
| system_time | event_time | symbol  | close_adj  |   close   |   high    |    low    |   open    |     volume      |
+-------------+------------+---------+------------+-----------+-----------+-----------+-----------+-----------------+
|     ...     | 2016-01-04 | IPO     | 19.8662    | 20.2800   | 20.6000   | 20.1700   | 20.6000   | 3800.0000       |
|     ...     | 2016-01-04 | SPY     | 183.2916   | 201.0200  | 201.0300  | 198.5900  | 200.4900  | 222353500.0000  |
|     ...     | 2016-01-05 | SPY     | 183.6017   | 201.3600  | 201.9000  | 200.0500  | 201.4000  | 110845800.0000  |
|     ...     | 2016-01-05 | IPO     | 19.8173    | 20.2300   | 20.2800   | 20.2000   | 20.2800   | 4000.0000       |
|     ...     | 2016-01-06 | IPO     | 19.4156    | 19.8200   | 20.0600   | 19.8200   | 19.9700   | 2600.0000       |
+-------------+------------+---------+------------+-----------+-----------+-----------+-----------+-----------------+
```

Remember that after adding a dataset from a `yaml` file - `kamu` creates an internal representation of it in the workspace `.kamu` directory, so if you want to make any changes to the dataset you will need to re-add the dataset again after changing the `yaml` file.

```sh theme={null}
kamu delete com.yahoo.finance.tickers.daily
# ... Make changes to the yaml file ...
kamu add com.yahoo.finance.tickers.daily.yaml

# Or alternatively
kamu add --replace com.yahoo.finance.tickers.daily.yaml
```

### Deriving current holdings from the transaction log

Our first goal will be to visualize the market value of our account. The account's market value can be estimated by multiplying the number of stocks held by their last known ticker price. Of course we aren't as interested in *current* market value as in *dynamics* of how it was changing over time, so we'd want to produce a stream of events telling us the estimated market value at different points in time.

If you look at the `my.trading.transactions` dataset it essentially contains deltas: how many stocks were bought/sold in each transaction, and the settlement amounts going in/out of our cache account. Our first step, therefore, should be obtaining a dataset that can tell us how many stocks *in total* (per symbol) we own. This can be easily achieved by calculating a running cumulative sum over stock quantities bought and sold.

Since this data might be something we can reuse in other calculations - let's create a dedicated derivative dataset for it.

The following listing shows the use of Flink engine and the SQL window functions for enriching all transaction events with cumulative sums of stock quantities and cash put into buying them:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: my.trading.holdings
  kind: Derivative
  metadata:
    - kind: SetTransform
      inputs:
        - datasetRef: my.trading.transactions
      transform:
        kind: Sql
        engine: flink
        query: |
          SELECT
            event_time,
            symbol,
            quantity,
            price,
            settlement,
            sum(quantity) over(partition by symbol order by event_time rows unbounded preceding) as cum_quantity,
            sum(settlement) over(partition by symbol order by event_time rows unbounded preceding) as cum_balance
          FROM `my.trading.transactions`
```

Let's create this dataset:

```sh theme={null}
kamu add my.trading.holdings.yaml
kamu pull my.trading.holdings
```

The result should look something like:

```sh theme={null}
kamu tail my.trading.holdings
```

```sql theme={null}
+-------------+------------+---------+-----------+-----------+-------------+---------------+--------------+
| system_time | event_time | symbol  | quantity  |   price   | settlement  | cum_quantity  | cum_balance  |
+-------------+------------+---------+-----------+-----------+-------------+---------------+--------------+
|     ...     | 2016-01-04 | SPY     | 1         | 201.0200  | -201.0200   | 1             | -201.0200    |
|     ...     | 2016-02-01 | SPY     | 1         | 193.6500  | -193.6500   | 2             | -394.6700    |
|     ...     | 2016-03-01 | SPY     | 1         | 198.1100  | -198.1100   | 3             | -592.7800    |
|     ...     | 2016-04-01 | SPY     | 1         | 206.9200  | -206.9200   | 4             | -799.7000    |
|     ...     | 2016-05-02 | SPY     | 1         | 207.9700  | -207.9700   | 5             | -1007.6700   |
+-------------+------------+---------+-----------+-----------+-------------+---------------+--------------+
```

### Calculating current market value of held positions

Now that we have enriched our transaction data with a running total of holding calculating market value should be a matter of joining these value with the ticker price data.

In the non-temporal world we could solve this problem as:

```sql theme={null}
holdings:              tickers:
+--------+----------+  +--------+----------+
| symbol | quantity |  | symbol | price    |
+--------+----------+  +--------+----------+
| SPY    | 50       |  | SPY    | 257.9700 |
+--------+----------+  +--------+----------+

SELECT
    h.symbol,
    h.quantity,
    h.price as price_unit,
    h.quantity * h.price as price_total
FROM tickers as t
LEFT JOIN holdings as h ON t.symbol = h.symbol
```

In our case, however, we are joining two streams of events, which might be new to many people.

Consider, for example, the following event in `tickers`:

```sql theme={null}
+-------------+------------+---------+------------+-----------+-----------+-----------+-----------+-----------------+
| system_time | event_time | symbol  | close_adj  |   close   |   high    |    low    |   open    |     volume      |
+-------------+------------+---------+------------+-----------+-----------+-----------+-----------+-----------------+
|     ...     | 2016-01-06 | SPY     | 181.2856   | 198.8200  | 200.0599  | 197.600   | 198.3399  | 152112600.0000  |
+-------------+------------+---------+------------+-----------+-----------+-----------+-----------+-----------------+
```

When we join it with `holdings` data we would like it to be joined with the holdings event that directly precedes it in the event time space - current price joined to the value of portfolio at that time. In our example it would be:

```sql theme={null}
+-------------+------------+---------+-----------+-----------+-------------+---------------+--------------+
| system_time | event_time | symbol  | quantity  |   price   | settlement  | cum_quantity  | cum_balance  |
+-------------+------------+---------+-----------+-----------+-------------+---------------+--------------+
|     ...     | 2016-01-04 | SPY     | 1         | 201.0200  | -201.0200   | 1             | -201.0200    |
+-------------+------------+---------+-----------+-----------+-------------+---------------+--------------+
```

There are two ways to achieve this. We could use **stream-to-stream joins** and describe the tolerance intervals (how far two events can be apart from each other in time) and the "precedes" condition in our join statement. While stream-to-stream joins have valuable applications using them in this case would be too cumbersome. Instead, we can use the **temporal table joins**.

Temporal table joins (see this [great explanation in Apache Flink's blog](https://flink.apache.org/2019/05/14/temporal-tables.html)) take one of the streams and represent it as a three-dimensional table. You can think of it as `holdings` table from the non-temporal example above that has a third "depth" axis which is `event_time`. When joining to such a table you need to pass in the time argument to tell the join which version of the table to consider.

Let's have a look at the `my.trading.holdings.market-value` dataset:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: my.trading.holdings.market-value
  kind: Derivative
  metadata:
    - kind: SetTransform
      inputs:
        - datasetRef: com.yahoo.finance.tickers.daily
        - datasetRef: my.trading.holdings
      transform:
        kind: Sql
        engine: flink
        temporalTables:
          - name: my.trading.holdings
            primaryKey:
              - symbol
        query: |
          SELECT
            tickers.`event_time`,
            holdings.`symbol`,
            holdings.`cum_quantity`,
            holdings.`quantity` as `quantity`,
            tickers.`close_adj` * holdings.`cum_quantity` as `market_value`
          FROM `com.yahoo.finance.tickers.daily` as tickers
          JOIN `my.trading.holdings` FOR SYSTEM_TIME AS OF tickers.`event_time` AS holdings
          ON tickers.`symbol` = holdings.`symbol`
```

Using `temporalTables` section we instruct the Flink engine to use `my.trading.holdings` event stream to create a virtual temporal table of the same name.

Here's how this three-dimensional table would look like when sliced at different points in time:

```sql theme={null}
my.trading.holdings:

@ 2016-01-04
+---------+---+---------------+--------------+
| symbol  |...| cum_quantity  | cum_balance  |
+---------+---+---------------+--------------+
| SPY     |   | 1             | -201.0200    |
+---------+---+---------------+--------------+

@ 2016-02-01
+---------+---+---------------+--------------+
| symbol  |...| cum_quantity  | cum_balance  |
+---------+---+---------------+--------------+
| SPY     |   | 2             | -394.6700    |
+---------+---+---------------+--------------+

@ 2016-03-01
+---------+---+---------------+--------------+
| symbol  |...| cum_quantity  | cum_balance  |
+---------+---+---------------+--------------+
| SPY     |   | 3             | -592.7800    |
+---------+---+---------------+--------------+
```

It basically remembers the last observed value of every column grouped by the provided `primaryKey` (`symbol` in our case).

The `JOIN my.trading.holdings FOR SYSTEM_TIME AS OF tickers.event_time` part can be interpreted as us taking every ticker event from `tickers`, indexing the temporal table `my.trading.holdings` at this event's `event_time` and then joining the same event with the resulting (now ordinary two-dimensional) table.

With theory out of the way, time to give this a try:

```sh theme={null}
kamu add my.trading.holdings.market-value.yaml
kamu pull my.trading.holdings.market-value
```

To view the results you can use the provided `trading.ipynb` notebook and see the actual graph:

```sh theme={null}
kamu notebook
```

<Diagram alt="Market value graph as viewed in the notebook" />

### The role of watermarks

If you look closely at the previous graph you will notice that the last event there is dated `2019-12-02`. This might be surprising because even though our account has stopped actively trading on that date, the ticker data still keeps coming, so we should be seeing the market value change over the course of 2020 and onwards...

Congratulations, you have just stumbled onto one of the most important features that distinguishes stream processing queries from batch processing!

When joining two streams `kamu` and its engines can't make any guesses about how complete the data in your dataset is. The streams you are joining can have vastly different intervals with which data is added. Ticker data might be added daily, while you, as the owner of transactions dataset, might be exporting your trading data semi-regularly, once in a month or two. Therefore when the transaction data stops coming on `Dec 2 2019` it means that `kamu` can't tell whether you are not actively trading any more or you simply forgot to import the recent transaction history. It therefore **pauses the processing** until this ambiguity is resolved (effectively by starting to buffer the ticker data).

The property behind this mechanism is called **watermark**. The watermark is a simple timestamp `T` that tells that with a high probability all events before `T` have already been ingested into the system. You can take a look at your dataset's watermarks using the `log` command:

```sh theme={null}
kamu log my.trading.transactions

Block #3: f16201cfd3c1822add90232665e78e94092d027ec87f93dd7410fc59104262cfbf378
SystemTime: 2024-01-16T00:42:57.722238098Z
Kind: AddData
NewData:
  NumRecords: 48
  Size: 3564
  ...
NewSourceState: ...
NewWatermark: 2019-12-02T00:00:00Z
```

There are two ways to resolve the "no data vs. stale data" ambiguity and advance the watermark:

* **Adding data** - by default `kamu` advances the watermark to the latest observed event time of the stream. This is not always what you want - if your data often arrives late and out of order it might be wise to delay the watermark to maximize the chances that further processing will not have to deal with late data.
* **Advancing watermark manually** - doing this tells `kamu` that we don't expect any data prior to some time `T` to arrive any more.

### Using manual watermarks

Let's put this into practice and tell `kamu` that transaction data is complete as of `2020-09-01`:

```sh theme={null}
kamu pull my.trading.transactions --set-watermark 2020-09-01T00:00:00Z
```

As you can see, this commits a watermark-only metadata block:

```sh theme={null}
kamu log my.trading.transactions

Block #4: f1620c4951a20fccd02e9125497b7c1e032f4d22de8c71b1cade3824c15bf61258161
SystemTime: 2024-01-16T00:45:22.998784847Z
Kind: AddData
PrevOffset: 47
NewWatermark: 2020-09-01T00:00:00Z
```

You can also use the short-hand forms like:

```sh theme={null}
kamu pull my.trading.transactions --set-watermark '1 month ago'
```

If we now pull the market value dataset we will see new data emitted:

```sh theme={null}
kamu pull my.trading.holdings.market-value --recursive
```

You can now reload the notebook and see that the graph extends to our watermark date.

<Note>
  Watermarks and many other stream processing concepts make temporal data processing much simpler. Take a minute to imagine how the queries we wrote above would look like in the batch processing form, how many edge cases associated with varying cadences, late and out-of-order arrivals the equivalent batch processing code would have to handle, and how **error-prone** it would be to write.
</Note>


# Web3 data
Source: https://docs.kamu.dev/examples/web3-data



**Topics covered:**

* Blockchain data
* Decentralized Storage
* Temporal-Table Joins
* Repositories

## Summary

This example currently exists as part of the [online tutorial](/start/tutorial).

The dataset files can be found in the [examples/reth-vs-snp500](https://github.com/kamu-data/kamu-cli/tree/master/examples/reth-vs-snp500) directory and creates a pretty complex streaming pipeline:

<Diagram alt="Web3 data pipeline" />


# Community
Source: https://docs.kamu.dev/general/community



We welcome participation from everyone - join, ask questions, and get involved!

## Discord

Stop by and say "Hi" in our [Discord Server](https://discord.gg/nU6TXRQNXC) - we're always happy to chat about data, and value your feedback.

[![Chat](https://shields.io/discord/898726370199359498?style=for-the-badge\&logo=discord\&label=Discord)](https://discord.gg/nU6TXRQNXC)

## YouTube

Subscribe to our [YouTube channel](https://www.youtube.com/channel/UCWciDIWI_HsJ6Md_DdyJPIQ) to get fresh tech talks and deep dives.

## GitHub

We do most project tracking on GitHub, and vast majority of communication occurs there.

The top-level repositories are:

* [open-data-fabric/open-data-fabric](https://github.com/open-data-fabric/open-data-fabric/)
* [kamu-data/kamu-cli](https://github.com/kamu-data/kamu-cli)
* [kamu-data/kamu-node](https://github.com/kamu-data/kamu-node)
* [kamu-data/kamu-web-ui](https://github.com/kamu-data/kamu-web-ui)

## Roadmap

[Open Data Fabric](https://github.com/orgs/open-data-fabric/projects/2/views/1) has its own roadmap.

We are preparing to make the existing multi-year Kamu roadmap public, but most of it can already be seen as "Epics" in top-level repositories ([example in kamu-cli](https://github.com/kamu-data/kamu-cli/issues?q=is%3Aissue+is%3Aopen+label%3Aepic)).

## Contributing

See [Contribution Guidelines](/general/contrib) to get started.


# Contributing
Source: https://docs.kamu.dev/general/contrib



Thanks for considering to contribute! This project is a massive undertaking and developed just by a few people, so every little bit helps.

We welcome contributors of any skill level and background. You [don't have to be a software engineer](https://dev.to/navendu/how-to-make-non-code-contributions-to-open-source-projects-35nj) to make a large impact in an open community project!

Besides code you can contribute in many different ways:

* Improving the [documentation](https://github.com/kamu-data/kamu-docs)
* Graphic design
* Adding more interesting [data science examples](/examples)
* Improving CI pipelines
* Working with the [community](https://discord.gg/aSpVjWwu)
* Mentoring others
* and much more...

Before you start working on anything:

1. Consider joining our [Discord Server](https://discord.gg/aSpVjWwu) to discuss your ideas
2. Make sure your work has an associated **GitHub issue**, it's assigned to you, and the scope of work was agreed upon with project maintainers (we value your time and really don't like rejecting PRs!)
3. Technical contributors should read the [DEVELOPER.md](https://github.com/kamu-data/kamu-cli/blob/master/DEVELOPER.md) for instructions on correctly setting up their environment (similar files exist in all top-level repositories)
4. Understand the [Licensing Terms](/general/licensing)


# Glossary
Source: https://docs.kamu.dev/general/glossary

Definitions of commonly used terms in Open Data Fabric and Kamu projects

# Open Data Fabric

## Account

A named identity within a <Term /> that can own <Term /> or act as a <Term />. Accounts can be users, organizations, service accounts, or the node itself.

Every resource belongs to an account, making accounts the unit of multi-tenancy - resources of the same type and name can exist under different accounts.

See also:

* [Authorization: Accounts](/odf/spec/auth-and-privacy/authorization#accounts)
* [Resource Framework: Ownership](/odf/spec/automation/resource-framework#ownership)

## Checkpoint

Checkpoints are used by the <Term /> to store the computation state between the different invocations of a <Term />. They are fully engine-specific and opaque to the system. They are however an essential durable part of a <Term /> as they are necessary to be able to pause and resume the streaming queries, and are essential in implementing "exactly-once" processing semantics.

## Coordinator

Coordinator is an application that implements the common <Term /> management logic.

Core responsibilities:

* Handles all <Term /> operations
* Splits the transformation work into batches based on the dataset's evolution timeline
* Collects relevant data slices of the input datasets
* Delegates data processing to the <Term />, making all transformations look to them as conventional stream processing
* Commits the resulting data slices and new metadata blocks

See also:

* [Components: Coordinator](/odf/spec/overview#coordinator)

## Data

Data is a set of <Term /> stored in the system. Since events are immutable data can only grow over time. Conceptually it's best to think of data as a full log of a potentially infinite event stream.

Data never appears in the system alone as we would not be able to tell where it came from and whether it can be trusted. Data always appears as part of a <Term />.

<Diagram alt="Diagram: Dataset/Data" />

See also:

* [Data Model](/odf/spec/datasets/data-model)
* [Common Data Schema](/odf/spec/datasets/data-model#common-data-schema)

## Data Slice

<Term /> arrives into the system as the arbitrary large sets of events. We refer to them as "slices".

More formally, a slice is a:

* Continuous part of <Term />
* That has the same <Term />
* Defined by its `[start; end]` <Term /> interval

<Diagram alt="Diagram: Data Slices and Metadata" />

## Dataset

Dataset is the main unit of data exchange in the system. It's simply a combination of:

* [Identity](/odf/spec/datasets/identity)
* <Term />
* <Term />
* <Term />

Depending on where the data comes from datasets can be of these kinds:

* <Term />
* <Term />

<Diagram alt="Diagram: Dataset Graph" />

See also:

* [Logical Structure](/odf/spec/datasets/logical-structure)

### Root Dataset

Root datasets are the points of entry of external data into the system. They are usually owned by the organization that has full authority and responsibility over that data, i.e. a trusted source.

Data gets into root datasets via <Term /> from sources that are defined separately from the dataset as `Source` <Term />. A source definition includes:

* Where to fetch the data from - e.g. source URL, a protocol to use, cache control
* How to prepare the binary data - e.g. decompression, file filtering, format conversions
* How to interpret the data - e.g. data format, schema to apply, error handling
* How to combine data ingested in the past with the new data - e.g. append as log or diff as a snapshot of the current state

Source definitions are not part of the <Term />, as they don't contribute to provenance or validity of data, and may reveal sensitive information about the infrastructure of a publisher. The metadata chain records the outcome of every ingestion, including the data, its schema, and the state of the source needed to resume ingestion.

See also:

* [Ingestion](/odf/spec/datasets/ingestion)
* [Merge Strategy](/odf/spec/datasets/ingestion#merge-strategies)

### Derivative Dataset

Derivative datasets are created by transforming/combining one or multiple existing datasets.

They are defined by the combination of:

* Input datasets
* A <Term /> to apply to those
* An <Term /> used to execute the query

This information is stored in the <Term /> and can change over time as the dataset evolves.

See also:

* [Processing Model](/odf/spec/transformations/processing-model)
* [Derivative Data Transience](/odf/spec/transformations/processing-model#derivative-data-transience)

## Engine

Engine is an interface shared by all specific implementations of a <Term /> dialect. Engine implementations are responsible for applying defined queries to input data and returning the result. For example, some engines allows you to query data using a series of streaming SQL statements.

Engines run in a sandboxed environments and are not permitted to use any external resources to guarantee the reproducibility of all operations.

<Diagram alt="Diagram: Derivative Transformation" />

As Engines are in the full control of all data transformations, they are also responsible for answering the <Term /> queries.

See also:

* [Engine Contract](/odf/spec/transformations/engine-contract)

## Entity

A data plane object materialized from a <Term /> to fulfill its desired state, e.g. an ODF <Term /> stored in an S3 bucket, a secret, or an encryption key.

Entities are separate from the resources that describe them. For example, when a dataset is replicated between two nodes, each node has its own `Dataset` resource with a different ID, while both describe the same dataset entity with the same DID.

See also:

* [Resource Framework: Resources and Entities](/odf/spec/automation/resource-framework#resources-and-entities)

## Event

As described in the [Nature of Data](/odf/spec/overview#nature-of-data) section, the system operates only on data expressed as past events, observations, or propositions believed to be true at a certain time. For simplicity, we will use the term "event" throughout to refer to such data tuples.

## Hash

[Cryptographic hash functions](https://en.wikipedia.org/wiki/Cryptographic_hash_function) are used by the system in these three scenarios:

* Computing a logical hash sum of a <Term />.
* Computing a physical hash sum of a <Term />.
* Computing a hash sum of a <Term />.

Whenever new events are appended to the <Term /> the <Term /> will also be extended with a block containing a hash sum of the new data slice. The hash sum provides a very quick and reliable way to later validate that the data matches the one that has been written earlier.

The new <Term /> will also be cryptographically signed to guarantee its integrity - this excludes any malicious or accidental alterations to the block.

Usage examples:

* If the <Term /> of a certain dataset is reliably known (e.g. available from many independent peers) a peer can then download the <Term /> from any untrusted source and use the hash function to validate the authenticity of every data slice that composes it.
* The trustworthiness of any <Term /> can be established by reviewing the transformations it claims to be performing on data (contained in the <Term />), re-applying those transformations in a trusted environment, and then comparing the hash sums of the result slices.

See also:

* [Data Hashing](/odf/spec/datasets/data-and-checkpoint-hashing#physical-vs-logical-hashes)
* [Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing#checkpoint-hashing)
* [Metadata Block Hashing](/odf/spec/datasets/metadata-chain#metadata-block-hashing)

## Ingestion

Ingestion is the process by which external data gets into the system. Typical ingestion steps that describe how data is obtained and read (e.g. fetching data from some URL on the web, decompressing it, and reading it as CSV) are defined by `Source` <Term />, and the ingested data is added to <Term />.

See also:

* [Ingestion](/odf/spec/datasets/ingestion)

## Manifest

A declarative configuration (e.g. a YAML file) that describes the desired state of a <Term />. Applying a manifest to a <Term /> creates or updates the corresponding resource, after which controllers reconcile the actual state of the system with the desired state.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
spec: {}
```

> **Note:** Earlier versions of the spec used the term for the legacy [`Manifest`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/Manifest.json) wrapper of metadata objects (e.g. `DatasetSnapshot`) that is now deprecated.

See also:

* [Resource Framework: Resource Manifests](/odf/spec/automation/resource-framework#resource-manifests)

## Merge Strategy

By [design](/odf/spec/overview#nature-of-data), the system only stores data in the append-only event log format to preserve the entire history. Unfortunately, a lot of data in the world is not stored or exposed this way. Some organizations may expose their data in the form of periodic database dumps, while some choose to provide it as a log of changes between current and the previous export.

When <Term /> from external sources, the <Term /> can choose between different [Merge Strategies](/odf/spec/datasets/ingestion#merge-strategies) that define how to combine the newly-ingested data with the existing one.

For example, when dealing with the daily database dumps, a user can choose the merge strategy that performs [change data capture](https://en.wikipedia.org/wiki/Change_data_capture), transforming dumps into a set of events that signify record creation, update, or deletion.

See also:

* [Merge Strategies](/odf/spec/datasets/ingestion#merge-strategies)

## Metadata

Refers to information about a <Term /> stored in its <Term />.

## Metadata Chain

Metadata Chain is an append-only log of **Metadata Blocks** that captures the entire history of a <Term />: its identity, <Term />, what data was added and when, how it was processed (see <Term />), and other information like licenses and descriptions.

Every block carries one metadata event and refers to the <Term /> of the previous block, so having a reference to the last block one can traverse the entire chain and discover and verify all components of the dataset.

See also:

* [Metadata Chain](/odf/spec/datasets/metadata-chain)

## Node

An entire ODF deployment acting under a single identity. A node hosts <Term /> and other <Term /> belonging to one or many <Term />, manages them using the <Term /> logic, and exposes them to other participants via <Term /> and query protocols.

A node also acts as a <Term /> in system tasks and node-to-node communications.

See also:

* [Authorization: Node](/odf/spec/auth-and-privacy/authorization#node)

## Offset

Offset is a monotonically increasing sequential numeric identifier that is assigned to every record and represents its position relative to the beginning of the dataset. Offsets are used to uniquely identify any record in the dataset. Offset of the first record in a dataset is `0`.

## Operation Type

Since past <Term /> are immutable, if some event is deemed incorrect later on it can only be rectified by issuing an explicit <Term />. Retraction and corrections are also represented as <Term /> in the same stream of <Term /> and differentiated by a special "operation type" field.

See also:

* [Common data schema](/odf/spec/datasets/data-model#common-data-schema)
* [Representation of retractions and corrections](/odf/spec/datasets/data-model#representation-of-retractions-and-corrections)

## Principal

The identity performing an action, established through authentication: a user, a service account, a <Term />, or an anonymous caller. Principals are the parties whose permissions are checked during authorization, and are recorded as initiators of operations (e.g. the identity a scheduled flow runs as).

See also:

* [Authorization: Principals](/odf/spec/auth-and-privacy/authorization#principals)

## Projection

In relational algebra, a [projection](https://en.wikipedia.org/wiki/Projection_\(relational_algebra\)) is an operation that removes one or many dimensions from a data tuple. In the context of our system the most common projections are *temporal projections* involving the <Term /> and <Term /> dimensions.

Depending on the time axis, we arrive at two most important types of projections in [bitemporal data modelling](https://en.wikipedia.org/wiki/Bitemporal_Modeling):

* **AS AT** or *"As we knew at that time"*. This projection collapses the system time dimension and shows us what the state of the data was at that time to the best knowledge of the system.
* **AS OF** or *“As we should've known at the time”*. This projection collapses the event time dimension and shows us what **should've happened** at that time if we knew about all potential corrections and compensating events that were added since then.

Understanding the difference between these projections is essential when working with time series data and for achieving the reproducibility of results.

See also:

* [A Brief History of Time in Data Modelling: OLAP Systems](https://www.kamu.dev/blog/a-brief-history-of-time-in-data-modelling-olap-systems/)

## Provenance

Data provenance describes the origins and the history of data and adds value to data by explaining how it was obtained.

<Term /> alone can already significantly narrow down the search space when you want to explain how a certain piece of data came to be, as it keeps track of all the inputs and queries used to create a dataset. But the goal of the provenance system is to make this type of inquiry effortless.

We differentiate the following kinds of provenance:

* **Why-provenance** - tells us which input data elements were inspected to decide that an element should be present in the output at all - i.e. defines a sufficient set of elements needed to produce the output.
* **How-provenance** - tells us the process by which the elements of *why-provenance* caused the output to appear
* **Where-provenance** - narrows down *why-provenance* to input data elements that were copied or transformed to determine the output element value.

Since the <Term /> are responsible for all data transformations, it's also the Engine's responsibility to answer provenance queries.

There are many different ways to implement provenance:

* By statically analyzing the queries
* By inverting transformations
* By repeating the computations and logging the data used at every step
* By propagating provenance data through all computations

Depending on the language used by an <Term /> one approach may work better in one situation than the other, so we avoid prescribing the exact method to use but instead standardize the language used for provenance-related queries and responses.

See also:

* [Provenance in Databases: Why, How, and Where](http://homepages.inf.ed.ac.uk/jcheney/publications/provdbsurvey.pdf)
* [Engine Contract: Derive Provenance](/odf/spec/transformations/engine-contract#derive-provenance)

## Query

Queries define how input data is combined, modified, and re-shaped to produce new data.

Queries are used in two contexts:

* When defining new <Term />
* When analyzing and extracting data from an existing <Term /> (locally or from a <Term />)

The system is agnostic to the exact language used to define the query and the set of supported dialects can be extended by implementing a new <Term />.

All queries, however, must have the following properties:

* Deterministic
* Pure
* Stream/Batch agnostic

In other words, they should be guaranteed to always produce the same result for the same input data, without side effects.

Example windowed aggregation query in streaming SQL:

```sql theme={null}
SELECT
  TUMBLE_ROWTIME(event_time, INTERVAL '1' MONTH) as event_time,
  sku_id,
  min(price) as min_monthly_price,
  max(price) as max_monthly_price,
  avg(price) as avg_monthly_price
FROM sku_prices
GROUP BY TUMBLE(event_time, INTERVAL '1' MONTH), sku_id
```

See also:

* [Stream Processing Model](/odf/spec/overview#stream-processing-model)
* [Derivative Data Transience](/odf/spec/transformations/processing-model#derivative-data-transience)
* [Engine Contract](/odf/spec/transformations/engine-contract)

## Repository

Repositories let participants of the system exchange <Term /> with one another.

Repository definition includes:

* Location where the repository can be reached (URL)
* Protocols that it supports
* Credentials needed to access it
* Any necessary protocol-specific configuration

In the most basic form, a <Term /> can simply be a location where the dataset files are hosted over one of the file or object-based data [transfer protocols](/odf/spec/exchange/transfer-protocols). The owner of a dataset will have push privileges to this location, while other participants can pull data from it.

An advanced repository can support more functionality like:

* Push data API for publishers
* Subscription API for consumers
* Query API for making use of repository's compute resources and reducing the amount of transferred data

See also:

* [Transfer Protocols](/odf/spec/exchange/transfer-protocols)

## Resource

A managed object of a <Term /> that has an identity and whose desired state is described declaratively in a <Term />, e.g. `Dataset`, `SecretSet`, `Flow`, or `User`. Resources form the control plane of the system - controllers continuously reconcile the actual state with the desired state, materializing resources into <Term /> in the data plane.

A resource manifest consists of:

* `$schema` - URI of the JSON schema that identifies the resource type
* `headers` - identity and ownership information, labels, and annotations
* `spec` - the desired state
* `status` - the current state and progress of reconciliation, maintained by the node

A resource is identified by its type, owning <Term />, and name, and is assigned a stable ID by the node upon creation.

See also:

* [Resource Framework](/odf/spec/automation/resource-framework)

## Retractions and Corrections

Errors in source data are inevitable and require a mechanism for correcting them. Unlike databases, where one could issue `DELETE` or `UPDATE` queries, ODF's core data model is an immutable append-only stream, and thus requires a different mechanism.

Retractions and corrections are explicit events that can appear in <Term /> datasets to signify that some previous event was published in error, or to correct some of its fields. They are differentiated from regular events by the special <Term /> field.

Retractions and corrections can also naturally occur in <Term /> datasets in cases when a stream processing operation encounters late data (data arriving past the current <Term />). In such cases streaming transformation may publish corrections or retractions for previously produced result records that were influenced by the late events.

Retractions and corrections model is fundamental to making data processing **maximally autonomous**.

See also:

* [Common data schema](/odf/spec/datasets/data-model#common-data-schema)
* [Representation of retractions and corrections](/odf/spec/datasets/data-model#representation-of-retractions-and-corrections)

## Schema

Schema describes the shape of the <Term /> by associating names and data types to columns that data is composed of. Schema can change over time and its changes are tracked in the <Term />.

Example:

```
registration_time TIMESTAMP(3),
registration_id UUID,
email STRING,
first_name STRING,
last_name STRING,
date_of_birth DATE,
```

See also:

* [Schema Format](/odf/spec/datasets/data-schema#schema-format)
* [Schema Evolution](/odf/spec/datasets/data-model#schema-evolution)

## Time

The system applies the idea of [bitemporal data modelling](https://en.wikipedia.org/wiki/Bitemporal_Modeling) to the event streams. It differentiates two kinds of time:

* <Term /> - tells us when some event was observed by the system
* <Term /> - tells when some event occurred from the perspective of the outside world

Every record in the system has exactly one system time associated with it upon the ingestion but can have zero to many event times.

### System Time

System time gives us a reference point for when something has occurred from the perspective of the system.

<Term /> the data onto the system time axis answers the question: *"what did the system know at the time T?"*, meaning that such projections effectively freeze data in time, providing the natural way to achieve **reproducibility**.

For all intents and purposes system time is treated as **ordered monotonically non-decreasing value** that lets us establish a *before-after* relationship between certain events. Note, however, that *before-after* relationship is only meaningful for data within one <Term /> and its upstream dependencies. System time cannot be used to establish an exact *before-after* relationship between the events of the independent datasets.

### Event Time

Event time tells us when something happened from the outside world's perspective. This time, therefore, is usually the most useful one for querying and joining data.

There are no restrictions on the event time in the system - there can be many event times associated with any record, and unlike system time, event times don't have to be monotonic. This allows the system to support many kinds and varieties of event time use, like:

* Post-dated events and predictions - with event time set into the future
* Back-dated events and corrections - with event time set into the past

Depending on the type of transformations these restrictions may be more strict, e.g. joining datasets based on event time may require it to be quasi-monotonic increasing to know when the join operation can be considered complete (see <Term />).

See also:

* <Term />
* <Term />
* [Streaming 101 by Tyler Akidau](https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/)
* [The Dataflow Model  by Tyler Akidau et al](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43864.pdf)

## Validation

Validation is the process of checking that an object conforms to the rules it must follow - e.g. that a <Term /> satisfies its consistency rules, that a file matches the <Term /> under which it is referenced, or that data follows its <Term />.

Validation answers the question *"is this object intact and well-formed?"*. It is cheap, performed locally, and does not depend on any other data, so it is usually done every time an object is received or read.

Not to be confused with <Term />, which establishes whether a claim about how data was produced is true.

See also:

* [Metadata Chain: Validation](/odf/spec/datasets/metadata-chain#validation)
* [Data & Checkpoint Hashing: Validation and Verification](/odf/spec/datasets/data-and-checkpoint-hashing#validation-and-verification)
* [Transfer Protocols: Validation of Transferred Data](/odf/spec/exchange/transfer-protocols#validation-of-transferred-data)

## Verifiability

In the scope of this specification, verifiability of data means the ability to establish:

* The ultimate source(s) of data:
  * Which <Term /> the data is coming from
  * Who these datasets belong to (ownership)
  * And which actor has added the specific records (accountability)
* The transformations performed to create this data:
  * The graph of <Term /> upstream to the one being verified
  * Authorship of those datasets (accountability)
* And finally, that the data in fact corresponds to performing declared transformations on the source data.

In other words, having root datasets `A`, `B` and a derivative dataset `C = f(A, B)`:

* The data in `A` (and similarly in `B`) is verifiable if:
  * <Term /> of `A` is valid
    * Metadata block hashes are valid, forming a valid chain
    * Blocks point to <Term /> and <Term /> with valid hashes
* The data in `C` is verifiable if:
  * <Term /> of `C` is valid
  * Data in `C` corresponds to applying `f(A, B)` according to all transformation steps declared in the <Term />.

The last step of ensuring that `f(A, B) = C` can be achieved by several means:

* Reproducibility - by applying same transformations and comparing the results
* Verifiable computing - different types of proofs that can attest to validity of results without redoing the computations.

Examples of verifiable computing can include:

* [Trusted Execution Environments](https://en.wikipedia.org/wiki/Trusted_execution_environment) (TEEs)
* [Non-interactive Proofs](https://en.wikipedia.org/wiki/Non-interactive_zero-knowledge_proof) (including "zero-knowledge").

Verifiability should not be confused with *trustworthiness* or *reality* of data. Verifying a dataset doesn't prove that the data in it is either truthful or more "real" than other data. The value of verifiability comes from establishing the provenance of data so that:

* One could understand whether data is coming from reputable sources - sources they can trust (attribution)
* One could review all derivative transformations applied to the data by intermediate actors (auditability).

Verifiability provides the foundation upon which *trust* in data can be built:

* First in the form **authority** - organizations putting their name behind the data they publish
* Secondly in the form of **reputation** - trusting the sources or pipelines used by large parts of the community
* Thirdly in the form of **cross-validation** - e.g. performing outlier detection on data from several similar publishers to establish common truth.

## Verification

Verification is the process of confirming that a claim about how data came to be is true - e.g. that data of a <Term /> is in fact the result of applying the declared transformations to its inputs, or that a query result corresponds to specific states of the queried datasets.

Verification answers the question *"is this claim true?"*. It can be performed by reproducing the computation or by checking proofs of verifiable computing, may require significant resources and access to other data, and can happen asynchronously and be delegated to other parties. Verification relies on <Term /> of all objects involved.

Verification is the means of achieving <Term />.

See also:

* [Overview: Verification](/odf/spec/overview#verification)
* [Processing Model: Verification](/odf/spec/transformations/processing-model#verification)
* [Verifiable Batch Queries](/odf/spec/exchange/verifiable-batch-queries)

## Watermark

A watermark defines the point in <Term /> for which with a high probability we've already observed all preceding events.

<Diagram alt="Diagram: Watermarks in the Stream" />

When performing time-based windowed operation, aggregations, or joins it is important to know when a certain time window can be considered closed. Watermark tells the system "You most likely will not get event with time less than `T` anymore".

In the <Term /> events can still arrive even after their time interval has been already been closed by the watermark. Such events are considered "late" and it's up to the individual <Term /> to decide how to handle them. They can be simply ignored, emitted into a side output, or still considered by emitting the "correction" event into the output.

Watermarks in the system are defined per every <Term />. By default, the <Term /> will assign the watermark to the maximum observed <Term /> in the <Term />. You can and should override this behavior if you expect events to arrive out-of-order to some degree, e.g. offsetting the watermark by `1 day` prior to last seen event.

<Diagram alt="Diagram: Watermarks in Time Domains" />

Watermarks can also be set based on the <Term /> manually or semi-automatically. This is valuable for the slow moving <Term /> where it's normal not to see any events in days or even months. Setting the watermark explicitly allows all computations based on such stream to proceed, knowing that there were no events for that time period, where otherwise the output would be stalled assuming the <Term /> was not updated for a while and old data can still arrive.

# Kamu

## Dataset Snapshot

A <Term /> with <Schema /> schema.

## Workspace

A workspace in the context of [Kamu CLI](/cli) is a directory that contains a hidden `.kamu` sub-directory where `kamu` stores local copies of datasets and configuration, including connection parameters to remote ODF repositories.

## Kamu Node

[Kamu Node](/node) is a set of [Kubernetes](https://kubernetes.io/) applications that can be installed in a distributed environment to operate data pipelines, verify computations done by other parties, and execute queries on co-located data. It is a horizontally-scalable **back-end implementation** of ODF protocol. Operating a node is the primary way to contribute compute resources to the ODF network.

## Oracle

[Blockchain Oracles](https://en.wikipedia.org/wiki/Blockchain_oracle) provide information from the outside-world sources to the on-chain [smart contracts](https://en.wikipedia.org/wiki/Smart_contract). The need for oracles arises because in typical blockchain implementations nodes cannot call any external APIs, therefore are limited to operate only on the information that is stored on chain.

Working in combination with ODF smart contracts, [Kamu Node](/node) implements an ODF oracle provider component that allows requesting data from the ODF network and supplying it to the smart contracts through a blockchain transaction.

Using ODF network oracle, smart contracts gain the ability to:

* access outside-world data through ODF nodes
* execute very complex and resource-intensive data queries off-chain, for significant cost savings
* get results along with the <Term /> and reproducibility information.

The last property makes ODF oracle really stand compared to the alternatives. Typical oracles act as intermediaries between the calling smart contract and some external API and thus have to employ complex consensus schemes to agree on what the actual result looks like and exclude MITM attacks. The more independent parties vote one the result - the more reliable it will be. But this also balloons the costs, as consensus needs to be established per every request. Therefore, users have to chose between reliability and costs and unfortunately often chose the latter. ODF, by design, is itself reproducible and verifiable - when requesting data from one ODF oracle provider you get full provenance information that can be used to **dispute and verify** the result at any time after it has been given. This allows ODF to function as the first of a kind "optimistic" oracle - one that can supply results very fast and cheaply, while allowing anyone to verify them asynchronously and engage a dispute resolution mechanism if foul play is suspected.


# Licensing Terms
Source: https://docs.kamu.dev/general/licensing



## Brief explanation of Kamu's license

Kamu is using [Business Source License](https://mariadb.com/bsl-faq-adopting/) (BSL).

We chose it as the closest thing to a fully Open Source license that allows us to build a sustainable business and continue to pursue our ambitious technology goals.

Under BSL:

* All source **code is provided in the open**
* All your **contributions benefit the community** in the same way as with any Open Source project
* Most people and organizations can still **use the product for free** in any environment and for any purpose **as long as**:
  * You're not providing product's functionality as a service to third parties
  * You're not distributing the product under a different name

If you clearly don't fall into these categories - you can stop reading here - it's that simple :)

Refer to "Additional Use Grant" section of the license for the exact terms.

## Understanding the Change Date

If you look at the [LICENSE.txt](https://github.com/kamu-data/kamu-cli/blob/master/LICENSE.txt) you will see the "Change Date" that is roughly 4 years from now. It represents the date on which a *specific version* of this software will automatically **convert into permissive Apache 2.0 license** ("Change License").

We increment this date during every release, so earlier versions of our software are closer to becoming pure Open Source.

## Motivation behind using BSL

The BSL license was **created by MariaDB** to strike a balance between being able to run a financially viable software company while still supporting the original tenets of Open Source, such as empowering all software developers to be part of the innovation cycle – giving them open access to the code so they can modify or distribute the software by making the entire source code available from the start. Ultimately, hoping that BSL will create more Open Source software.

BSL allows us to fund the project, while allowing virtually anyone to use our software for free. Because our solution is decentralized and encourages alternative implementations by being developed as an open standard we are hoping to create an environment where companies profit from data by providing **great value-added services at a fair price**, not by holding your data hostage.

As opposed to the "open-core" model, we believe BSL license also avoids wasting effort and splitting up the community when an open-source clone has to be written from scratch to be used instead of some proprietary component that does not satisfy some use cases.

## Who else is using BSL?

Unfortunately, there are many precedents where the work of Open Source companies was bluntly exploited. This not only prevents the developers of the technology from reaping the well-deserved rewards, but also prevents them from **re-investing the resources into improving the tech**, which leads to stagnation. Many such companies have successfully adopted the BSL license as a way to restore the balance:

* [MariaDB](https://mariadb.com/bsl-faq-adopting/)
* [CockroachDB](https://www.cockroachlabs.com/blog/oss-relicensing-cockroachdb/)
* [Couchbase](https://blog.couchbase.com/couchbase-adopts-bsl-license/)
* [Sentry](https://blog.sentry.io/2019/11/06/relicensing-sentry)

<Info>
  Same reasons have also lead to creation of the [SSPL license](https://www.mongodb.com/licensing/server-side-public-license/faq) by MongoDB, which is also adopted by many companies, but we chose BSL for being **simpler and less ambiguous**.
</Info>

## FAQ

***Can a company use Kamu for free to provide their data to customers?***

Yes, the way we look at it is:

* Your company uses `kamu` internally to publish datasets
* Datasets are available online in [ODF format](/odf), which is an open protocol
* Your clients independently can use `kamu` (or any other tool compatible with ODF) to download data

Since you're not operating `kamu` as a *service* for your customers - you're not violating the "Additional Use Grant".


# Project Status
Source: https://docs.kamu.dev/general/project-status



## Disclaimer

`kamu` is at the MVP stage of maturity, but is still **actively evolving**. Before `v1.0` we reserve the right to break compatibility between the releases.

That said `kamu` is already used in production.

When you use it for projects, make sure to **maintain your source data separately** and not rely on `kamu` for durable long-term data storage. This way any time a new version comes out that breaks some compatibility you can simply delete your workspace and re-create it from scratch in a matter of seconds.

### Note on Performance

Please be patient with current performance and resource usage. Stream processing technologies is a relatively new area, and some of the data processing <Term /> used by `kamu` (e.g. Apache Spark and Flink) are tailored to run in large clusters, and not on a laptop. They take a lot of resources to just boot up, so the *start-stop* nature of `kamu`'s transformations is at odds with their design. We are hoping that the industry will recognize our use-case and expect to see a better support for it in future. We are committed to continuously improving the performance and user experience.

## Feature Coverage

| Feature | ODF | kamu |
| - | :-: | :-: |
| Root datasets | ✔️ | ✔️ |
| Ingest merge strategies | ✔️ | ✔️ |
| Derivative datasets | ✔️ | ✔️ |
| Verifiability - Metadata integrity | ✔️ | ✔️ |
| Verifiability - Data integrity | ✔️ | ✔️ |
| Verifiability - Transformation replay | ✔️ | ✔️ |
| Engines - Determinism enforcement | ❌ | ❌ |
| Source evolution | ❌ | ❌ |
| Schema evolution | ❌ | ❌ |
| Query migrations | ❌ | ❌ |
| Engine versioning | ❌ | ❌ |
| Engine migrations | ❌ | ❌ |
| Dataset identity | ✔️ | ✔️ |
| Dataset ownership | ❌ | ❌ |
| Transaction signing | ❌ | ❌ |
| Data encryption | ❌ | ❌ |

## Stability

| Component | Stability |
| - | :-: |
| Dataset on-disk layout | Unstable |
| CLI interface | Unstable |
| Engine interface | Unstable |

***

See also:

* [Supported engines](/general/supported-engines)


# Supported Data Engines
Source: https://docs.kamu.dev/general/supported-engines

Describes strengths and development state of different engines supported by kamu

All data processing in `kamu` is done by a set of plug-in <Term />. This allows us to integrate many mature data processing frameworks, use them to transform data, while `kamu` coordinates all the advanced aspects of processing, tracks <Term />, ensures <Term />, etc.

<Note>
  The opinions below relate to ODF adapters implemented using the described engine, not the engines themselves. Engines featured here all have very different designs, making them more suitable for some tasks than others. Information below is intended as a rough guidance for engine choice within ODF and should be taken with a big grain of salt.
</Note>

## Known Engine Implementations

| Name | Technology | Query Dialect | Links | Notes |
| - | :-: | :-: | :-: | - |
| `spark` | [Apache Spark](https://spark.apache.org/) | [Spark Streaming SQL](https://spark.apache.org/docs/latest/sql-ref.html) with [Sedona GIS Extensions](https://sedona.apache.org/1.5.0/api/sql/Overview/) | [Repository](https://github.com/kamu-data/kamu-engine-spark)<br />[Image](https://ghcr.io/kamu-data/engine-spark) | Spark is used in `kamu-cli` for all data ingestion and is default (but not only) engine for SQL shell. Spark is also used in combination with Livy to query data from embedded Jupyter Notebooks. It's currently the only engine that supports **GIS data** via Apache Sedona integration. |
| `flink` | [Apache Flink](https://flink.apache.org/) | [Flink Streaming SQL](https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/gettingstarted/) | [Repository](https://github.com/kamu-data/kamu-engine-flink)<br />[Image](https://ghcr.io/kamu-data/engine-flink) | Flink has most mature support for stream processing, like stream-to-stream and stream-to-table joins, windowed aggregations, watermarks etc. It's thus the recommended engine for most derivative datasets. |
| `datafusion` | [Apache Arrow DataFusion](https://github.com/apache/arrow-datafusion) | [DataFusion SQL](https://arrow.apache.org/datafusion/user-guide/sql/index.html) | [Repository](https://github.com/kamu-data/kamu-engine-datafusion)<br />[Image](https://ghcr.io/kamu-data/engine-datafusion) | An extremely fast and low-footprint **batch processing** engine. DataFusion is also embedded into `kamu-cli` and is used for data ingestion, ad-hoc SQL queries, and SQL console. Despite being a batch-only engine it can still be used in pipelines for simple map/filter/union operations where temporal semantics is not needed. |
| `risingwave` | [RisingWave](https://github.com/risingwavelabs/risingwave) | [RisingWave SQL](https://docs.risingwave.com/docs/current/sql-references/) | [Repository](https://github.com/kamu-data/kamu-engine-risingwave)<br />[Image](https://ghcr.io/kamu-data/engine-risingwave) | Experimental stream process engine. There are [ongoing attempts](https://github.com/apache/arrow-datafusion/issues/4285) to add stream processing functionality. DataFusion is also embedded into `kamu-cli` and is used for executing ad-hoc SQL queries. We are aiming to make data ingest functionality use DataFusion for most cases instead of Spark. |

### Schema Support

| Feature | kamu | Spark | Flink | DataFusion | RisingWave |
| - | :-: | :-: | :-: | :-: | :-: |
| Basic types | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
| Decimal type | ✔️ | ✔️ | ✔️\*\* | ✔️ | ✔️ |
| Nested types | ✔️\* | ✔️ | ❌ | ❔\*\*\* | ❔\*\*\* |
| GIS types | ✔️\* | ✔️ | ❌ | ❌ | ❌ |

✔️\* - There is currently no way to express nested and GIS data types when declaring root dataset schemas, but you still can use them through pre-processing queries

✔️\*\* - Apache Flink has known issues with Decimal type and currently relies on our patches that have not been upstreamed yet, so stability is not guaranteed [FLINK-17804](https://issues.apache.org/jira/browse/FLINK-17804).

❔ - Engine capability exists but requires more integration testing

### Operation Types

Note that ODF always operates in <Term />, this all temporal aggregations and joins have to be supported by the engine in event-time processing mode.

| Feature | Spark | Flink | DataFusion | RisingWave |
| - | :-: | :-: | :-: | :-: |
| Filter | ✔️ | ✔️ | ✅ | ✅ |
| Map | ✔️ | ✔️ | ✅ | ✅ |
| Aggregation: Window functions | ❌ | ✔️ | ❌ | ✅ |
| Aggregation: Tumbling windows | ❌ | ✔️ | ❌ | ✅ |
| Aggregation: Top-N | ❌ | ❔ | ❌ | ✅ |
| Join: Windowed | ❌ | ✔️ | ❌ | ❔ |
| Join: Temporal Table | ❌ | ✔️ | ❌ | ❔ |
| GIS extensions | ✅ | ❌ | ❌ | ❌ |

✔️ - supported<br />
✅ - supported and recommended<br />
❌ - not supported<br />
❔ - engine capability exists but requires more integration testing


# Terms of Service
Source: https://docs.kamu.dev/general/terms



## 1. Definitions

"Kamu", "We", and "Us" refer to Kamu, as well as our affiliates, directors, subsidiaries, contractors, licensors, officers, agents, and employees. "Customer", "You", and "Your" refer to the Customer who is entering into this agreement.

"Products" refer to the software code, applications, libraries, tools, and deployment artifacts developed or distributed by Kamu (collectively, "Software"), programing interfaces ("APIs"), public and private online platform instances (each, a "Platform"), and websites such as ***kamu.dev*** and all of its subdomains (each, a "Site").

"Services" refer to Products and the underlying infrastructure like storage, compute, data processing, monitoring, backups, and replication provided and operated by Kamu, and related technical support concerning correct functionality of Products and the infrastructure.

"User" means the users of the Kamu Services. "End Users" means the individuals or users who are permitted by Customer to use the Services. For clarity, End Users may include employees of Customer, affiliates, and other authorized third parties.

"Content" means any content made available or enabled through the Services by Kamu, Customer, or other Users or End Users, and includes, without limitation, any (i) information, data, documents, images, photographs, graphics, audio, videos, or webcasts, and (ii) products, applications or projects. “Kamu Content” means those content that are made available to you through the Services or otherwise by Kamu or its licensors, while Content made available or enabled by you as “Customer Content” or other Users are referred to herein as “User Content.” Customer Content may include End User Content.

"Customer Application" means a software program or application that Customer creates or hosts using the Services. "Customer Data" means data provided to Kamu by Customer or its End Users through the Services, and data that Customer or End Users derive from that data through their use of the Services. For clarity, Customer Content includes Customer Applications and Customer Data.

"Brand Features" means the trade names, trademarks, service marks, logos, domain names, and other distinctive brand features of each Party, respectively, as secured by such Parties from time to time.

The "Agreement" refers, collectively, to all the terms, conditions, notices contained or referenced in this document.

"Including" means including but not limited to.

## 2. Use of Services and Products

By using Kamu’s Products and Services the Customer agrees to comply with and be bound by this Agreement. If you do not agree to this Agreement, you may not use the Products and Services. Your continued use of the Products or Services constitutes your acceptance of this Agreement. Customer agrees to use the Kamu Products and Services solely for the purposes permitted by this Agreement. Customer is solely responsible for its use of the Products and Services

Kamu grants you the right to access and use the Services in accordance with this Agreement. We reserve all other rights.

You are responsible for End Users’ use of Your Content and the Services, and for their compliance with your obligations under this Agreement. If you become aware of any violation of your obligations under this Agreement caused by an End User, you will immediately suspend access to Your Content and the Services by such End User.

Kamu does not provide any support or services to End Users unless we have a separate agreement with you or an End User obligating us to provide such support or services.

**2.1. Additional Terms**

Any Software, Product, or Service made available by Kamu (including the Kamu Node, Kamu CLI, or Kamu Web UI software, or private or public Kamu Platforms), may also be subject to other agreements, licenses, posted guidelines, rules, or additional terms of service ("Additional Terms"), which you agree to adhere to if you access or use this Software, Product, or Service. If there is any conflict between this Agreement and the Additional Terms, the Additional Terms take precedence in relation to that Software, Platform, or any other Service. The Additional Terms might be modified from time to time, and it is the responsibility of the Customer to check the corresponding information pages regularly for modifications. Terms of this Agreement and any applicable Additional Terms and all other documents incorporated by reference in the Agreement are referred to herein as the "Terms".

**2.2. Access to Services**

Kamu makes certain Services available only if you have created a Kamu account and password or other log-in credentials and private API keys (collectively, “Account Information”).

**(a)** It is your responsibility to keep your Account Information confidential at all times, and you are solely responsible for all activity that occurs under your account with Kamu Services (“Account”) or to your Account Information.

**(b)** You may not use another person’s Account Information, create an account for anyone other than yourself, or sell, rent, sublicense, transfer, lease, share or provide access to your Account Information to anyone else, except that you may disclose your private key to your agents and subcontractors performing work on your behalf.

**(c)** Kamu may require that you change your Account Information or certain parts of your Account Information at any time for any reason.

**(d)** Customer is responsible for all activities that occur under the Customer Account, regardless of whether the activities are authorized by or undertaken by the Customer, its employees or a third party (including your contractors, agents or End Users). Kamu and its affiliates are not responsible for unauthorized access to the Customer’s Account.

**(e)** Customer (i) is responsible for properly configuring and using the Services and otherwise taking appropriate action to secure, protect and backup your Accounts, Account Information and Customer Content in a manner that will provide appropriate security and protection, and (ii) will promptly notify Kamu of any unauthorized use of, or access to, the Services, Customer Account, or Customer's Account Information of which Customer becomes aware.

**(f)** Kamu reserves the right to investigate any potential violation of the Agreement or Terms by Customer, which may include reviewing Customer Content.

**(g)** You agree that Kamu retains the right to create limits on the access and use of the Services, including your Content, such as limits on file size, storage space, processing capacity, API use limitations, and similar limits described in the web pages accompanying the Services and as otherwise determined by Kamu in its sole discretion.

**2.3. Restrictions**

The source code, design, and structure of all or any part of the Kamu Services and Products are trade secrets. Customer must not (and must not permit anyone else, including End Users, to):

**(a)** rent, lease, sell, distribute or sublicense the Services include them in a service bureau or outsourcing offering, or otherwise transfer any part of your rights to use the Services;

**(b)** provide access to the Services to a third party;

**(c)** charge its customers a specific fee for use of the Kamu Services, but Customer may charge an overall fee for its own offerings (of which the Products are ancillary);

**(d)** use the Services to develop, commercialize, license or sell a similar or competing product or service;

**(e)** reverse engineer, decompile, disassemble, translate, in whole or in part, or distribute the Services;

**(f)** copy, modify or create derivative works of the Services;

**(g)** interfere with, avoid or circumvent Service usage limits, quotas or incurring fees;

**(h)** remove, obscure or modify in any way any proprietary or other notices, copyright or attributions in the Services;

**(i)** use the Services for benchmarking or competitive analysis with respect to competitive or related products or services;

**(j)** use Services to engage in cryptocurrency mining without Kamu's prior written approval;

**(k)** download data or Content for spamming or scraping purposes.

Customer and its End Users will not use the Services for illegal purposes. You will comply with the terms of this Agreement and other Terms and all laws, rules and regulations (including any laws or regulations regarding the use and export of data or software to and from the Canada, United States or other applicable countries) (collectively, "Law"), as well as generally accepted practices or guidelines in any applicable jurisdiction, applicable to your use of the Services.

Customer is solely responsible for all Customer Content, including its End User’s use of the content or the Services. Customer will ensure that Customer Content, and the use of Customer Content or Services by you or your End Users will not violate any of the Policies or any applicable Law.

**2.4. Customer Obligations**

Customer will secure and maintain all rights in Customer Content necessary for Kamu to provide the Services without violating the rights of any third party or otherwise obligating Kamu to you or to any third party. Kamu does not and will not assume any obligations with respect to Customer Content or to your use of the Services other than as expressly set forth in this agreement or as required by applicable law. Customer is responsible for any consents and notices required to permit:

**(a)** Customer's use and receipt of the Services;

**(b)** Kamu’s accessing, storing, and processing of data provided by Customer or other Customer Content (including End User Data, if applicable) under the Agreement.

**2.5. Modifications to Agreement, to Services**

Kamu may at any time modify or discontinue, temporarily or permanently, the Services, or any portion thereof, with or without notice. You agree that Kamu shall not be liable to you or any third party for any modification, suspension or discontinuance of the Services or Agreement, to the maximum extent permitted under applicable law.

**(a)** Modifications to the Services. Kamu may automatically provide updates or changes to the Services from time to time, without notice. These updates are designed to improve, enhance and further develop the Services and may take the form of bug fixes, enhanced functions, new Software modules and completely new versions. You agree to receive such updates (and permit Kamu to deliver these to you with or without your knowledge) as part of your use of the Services, and all such Software (as updated from time to time) will remain governed by the Agreement and other Terms.

**(b)** Discontinuation of the Services. We may discontinue any or part of the Services from time to time. We will provide you at least 30 days’ prior notice before discontinuing a material functionality of a Service that has a material impact on the Services the Customer is using and ensure to provide you means to retrieve your data until the discontinuation date. Kamu will not be obligated to provide such notice if the discontinuation is necessary to (a) address an emergency, or risk of harm to the Services or Kamu, (b) respond to claims, litigation, or loss of license rights related to third party intellectual property rights, or (c) comply with law, but should any of the preceding occur Kamu will provide you with as much prior notice as is reasonably practicable under the circumstances.

**(c)** Modifications to the Agreement. Kamu may modify this Agreement. Kamu will notify the Customer of material changes to this Agreement, such as price increases, at least 30 days prior to the change taking effect by sending email to the primary contact address specified in the Agreement, except to the extent the changes apply to new functionality or are required by applicable law, in which case they will be effective immediately. Customer's continued use of the Service constitutes agreement to those revisions of this Agreement. Customer may also terminate this Agreement for convenience under Section 4.1 (Termination for Convenience).

## 3. Fees and Payment

**3.1 Fees**
Exact fees are governed by individual Customer agreements.

**3.2. Payments**

Starting from the Effective Date of the Agreement every month will constitute a Billing Period. Kamu will issue an invoice to Customer at the end of each billing period, until termination or modification of the Agreement. Customer agrees to pay all Fees in the currency stated in the invoice within thirty (30) days of the invoice date. Unless required by law, Customer's obligation to pay all Fees is non-cancellable and non-refundable. Payments made via wire transfer must include the bank information provided by Kamu. Customer is responsible for all the banking fees associated with the payment.

**3.3. Taxes**

Fees are exclusive of any sales, use, GST, value-added, withholding or similar taxes unless otherwise specified on the invoice as tax inclusive. Customer is responsible for and must pay all such taxes that are owed under this agreement and which we are permitted to collect from you under applicable law. Customer will provide Kamu with any applicable tax identification information that Kamu may require under applicable law to ensure its compliance with applicable tax regulations and authorities in applicable jurisdictions. Customer will be liable to pay (or reimburse Kamu for) any taxes, interest, penalties, or fines arising out of any mis-declaration by Customer.

**(a)** *Withholding Taxes*. To the extent Customer is required to withhold tax from payment to Kamu in certain jurisdictions, Customer must provide valid documentation it receives from the taxing authority in such jurisdictions confirming remittance of withholding. This documentation must be provided at the time of payment of the applicable invoice to Kamu.

**(b)** *Exemptions*. If Customer claims exemption from any sales tax, VAT, GST or similar taxes under this Agreement, Customer must provide Kamu a valid tax exemption certificate or tax ID at the time of Order, and after receipt of valid evidence of exemption, Kamu will not include applicable taxes on the relevant Customer invoice.

**3.4. Suspension for Non-payment**

Kamu may suspend Customer’s rights to use Services if payment is overdue, and Kamu has given Customer no fewer than thirty (30) days’ written notice.

## 4. Term and Termination

This Agreement commences on the date Customer accepts it (the "Effective Date") and expires when all Subscription Terms have ended.

**4.1. Termination for Convenience**

Customer may terminate this Agreement at any time on prior written notice and, upon termination, must cease use of the Services. Kamu may terminate this agreement upon 30 days written prior notice, for any reason.

**4.2. Termination for Cause**

Either party may terminate this Agreement if the other party:

**(a)** fails to cure a material breach of this Agreement including a failure to pay fees (or have acted in a manner that indicates or suggests you do not intend to, or are unable to, comply with any of the Agreement terms or other Terms);

**(b)** ceases operation without a successor;

**(c)** become the subject of any bankruptcy, reorganization, liquidation, dissolution or similar proceeding.

Kamu may terminate this Agreement immediately without written notice in order to comply with the law or requests of governmental entities, or if Kamu reasonably believes that:

**(a)** continued provision of any Service used by Customer would violate applicable Law(s);

**(b)** Customer has violated or caused Kamu to violate any Law(s).

**4.3. Effect of Termination**

Upon termination of this Agreement:

**(a)** all Customer’s rights and access to the Services will cease (including access to Customer Content);

**(b)** Customer must immediately cease accessing or using the Services;

**(c)** Customer must delete (or, on request, return) all Account Information (including license keys, access keys and any Product copies).

All Fees owed by Customer to Kamu are immediately due upon Customer’s receipt of the final electronic bill or as stated in the final invoice.

Termination of your Account may include:

**(i)** removal of access to Services;

**(ii)** deletion of Customer Content and Account Information, and all related information, including Customer and End User data and associated with or inside your Account (or any part thereof).

You agree that Kamu shall not be liable to you or any third party for any termination of your Account and accompanying effects.

**4.4. Survival**

All provisions of this Agreement which, by their nature, should survive termination will survive termination — including, without limitation: Section 1 Definitions, Section 4 Term and Termination, Section 3 Fees and Payment, Section 5 IP Rights, Section 6 Indemnification, Section 7 Limitations of Liability, Section 8 Disclaimers, and Section 9 Governing Law, Jurisdiction and Venue.

**4.5. Temporary Suspension**

Kamu may immediately suspend all or part of Customer's use of the Services if:

**(a)** Kamu reasonably believes Suspension is needed to protect the Services, Kamu’s infrastructure supporting the Services, or any other customer of the Services (or their end users);

**(b)** there is suspected unauthorized third-party access to the Services;

**(c)** Kamu reasonably believes that immediate Suspension is required to comply with any applicable law;

**(d)** Customer or its End User’s use of the Services could subject Kamu, its affiliates, or any third party to liability, or could be fraudulent;

**(e)** Kamu determines that the Customer or its End Users abuse or excessively frequent requests to the Services via the API;

**(f)** Customer or its End Users is in material breach of any of the Agreement terms or other Terms.

Kamu will lift any such Suspension when the circumstances giving rise to the Suspension have been resolved. At Customer's request, Kamu will, unless prohibited by applicable law, notify Customer of the basis for the Suspension as soon as is reasonably possible. If one or more of these conditions occurs, then:

**(i)** You will be responsible for all fees and charges you incur during the period of suspension that we bill to you;

**(ii)** Suspension will be in effect only while the condition or need exists. We will give notice before we suspend, except where we reasonably believe we need to suspend immediately. We will give at least 30 days' notice before suspending for non-payment. If you do not fully address the reasons for the suspension within 60 days after we suspend, we may terminate your Subscription and delete your Customer Content without any retention period.

## 5. Intellectual Property Rights, Protection of Customer Content, Feedback

**5.1. No Transfer of Rights**

Except as expressly stated in this Agreement, this Agreement does not grant either party any rights, implied or otherwise, to the other's content or any of the other's intellectual property. As between the parties, Customer retains all Intellectual Property Rights in Customer Content and Customer Applications, and Kamu retains all Intellectual Property Rights in the Services and Software. Customer grants Kamu the right to store, copy, modify, process, perform, archive, parse, display or otherwise use Customer Content, as necessary to provide or improve the Services.

**5.2. Protection of Customer Content**

Kamu will only access, use, and otherwise process Customer Content as necessary to maintain, provide or improve the Services or as necessary to comply with the law or a binding order of a governmental body, and will not access, use, or process Customer Content for any other purpose, unless authorized by the Customer.

**(a)** Some Services involve collaboration and file-sharing services among other Users or a specific group in conjunction with such collaboration and data-sharing. The User Content that you or other Users share with other Users through the Services shall be referred to as “Shared Content”. While some Services offer functionality to limit another User’s use of your Shared Content, such limitations are not guaranteed and it is your sole responsibility to determine what limitations, if any, are placed on Shared Content that you distribute. You agree that Kamu has no liability of any kind for or in connection with Shared Content, including if other Users use, modify, destroy, corrupt, copy or distribute your Shared Content in violation of the limitations that you may impose on its use. Further, it is your sole responsibility to determine what limitations are placed on your use of another User’s Shared Content, including by looking at a given Service’s functionality and Additional Terms along with any additional restrictions placed by such User on your use of their Shared Content.

**(b)** For any Content that you made publicly available to End Users or as Shared Content (such as data, photos, audio, software and videos and/or comments that might or might not be covered by Intellectual Property rights), you hereby grant Kamu, its Users or Customer’s End Users with a non-exclusive, transferable, worldwide license to use, share, copy, modify, distribute, publicly display, perform and reproduce your User Content as permitted through the functionality of Kamu Services and Products. If you are sharing Content you did not create or own, you are responsible for ensuring that the Content you upload is licensed under terms that grant these permissions to other Kamu Users.

**(c)** In some cases, you may remove your Content by deleting it within the Services. However, you understand that removed content may persist in backup copies for a reasonable period of time, or otherwise pursuant to our data retention and archival policies. Additionally, you acknowledge that prior to your deletion on the Services, if previous copies of such deleted Content was made available as Shared Content, other Kamu Users may still own a copy or part of such Content.

**5.3. Customer Feedback and Suggestions**

At its option, Customer may provide feedback or suggestions about the Services to Kamu ("Feedback"). If Customer provides Feedback, then Kamu and its Affiliates may use that Feedback without restriction and without obligation to Customer. You hereby irrevocably assign to us all right, title, and interest in and to the Feedback and agree to provide us any assistance we require to document, perfect, and maintain our rights in the Feedback.

**5.4. Publicity**

Customer may state publicly that it is a Kamu customer and display Kamu Brand Features in accordance with the Trademark Guidelines and IP rights. Kamu may use Customer's name and Brand Features in online or offline promotional materials of the Services, with prior written approval by the Customer. Any use of a party's Brand Features will inure to the benefit of the party holding Intellectual Property Rights to those Brand Features.

**5.5. Adequate Rights**

Customer represents and warrants to Kamu that:

**(a)** Customer or its End Users own all right, title, and interest in and to Customer Content and Feedback;

**(b)** you have all rights in Customer Content and Feedback necessary to grant the rights contemplated by this Agreement;

**(c)** you have the rights necessary to grant the licenses and sublicenses described in the Agreement;

**(d)** you have received consent from any and all persons depicted in your Content to use your Content as set forth in the Agreement and other Terms, including distribution, public display, public performance and reproduction of your User Content;

**(e)** none of Customer Content or End Users’ use of Customer Content or the Services will violate the Agreement, a third party’s property rights or rights of publicity or privacy or result in the violation of any applicable law or regulation.

## 6. Indemnification

You will defend, indemnify, and hold Kamu, its affiliates and licensors, and each of their respective employees, officers, directors, and representatives harmless from and against any losses, claims, liabilities, demand, and expenses (including legal and accounting fees), arising out of or relating to any third-party disputes concerning:

**(a)** Customer or its End Users’ use of the Services (including any Data, Content, Application or activities under your Kamu account, and use by your employees and personnel);

**(b)** breach of this Agreement or violation of applicable law by you, End Users or Your Content;

**(c)** your sharing of any Content (including User Content or Shared Content) that infringe or misappropriate a third party’s Intellectual Property Rights or violate Law;

**(d)** any agreements, transactions, losses, or liability arising between you and any third parties advertising or promoting via the Services;

**(e)** a dispute between you and any User (including End Users), including any Intellectual Property Rights;

And will pay the amount of any adverse final judgment or settlement.

**6.1. Procedure of Defending**

Each party must notify the other promptly of a claim, under this Section so as to not prejudice the defense of the claim; and reasonably cooperates with the other party in the defense and settlement of the claim. Any indemnified party must tender sole control of the indemnified portion of the Third-Party Legal Proceeding to the indemnifying party, subject to the following: **(i)** the indemnified party may appoint its own non-controlling counsel, at its own expense; and **(ii)** any settlement requiring the indemnified party to admit liability, pay money, or take (or refrain from taking) any action, will require the indemnified party's prior written consent, not to be unreasonably withheld, conditioned, or delayed.

**6.2. Remedies**

If Kamu reasonably believes the Services might infringe a third party's Intellectual Property Rights, or subject to any claim, then Kamu may **(i)** procure the right for Customer to continue using the Services; **(ii)** modify the Services to make them non-infringing without materially reducing their functionality; or **(iii)** replace the Services with a non-infringing, functionally equivalent alternative. If Kamu does not believe the remedies are commercially reasonable, then Kamu may Suspend or terminate Customer's use of the impacted Services.

## 7. Limitations of Liability

You understand and agree that Kamu, its officers, directors, employees, agents, partners or suppliers will not be liable to the Customer, its End Users or any third party for any loss of profits, revenue, savings, use, business opportunities, goodwill, content, or data; or for any incidental, indirect, special, punitive, reliance, consequential or exemplary damages of any kind; or for damages for business interruption or loss of business information, even if informed of their possibility in advance, which might arise from:

**(a)** the use, disclosure, or display of your Content;

**(b)** your use or inability to use the Services;

**(c)** any modification, price change, suspension or discontinuance of the Services;

**(d)** the Services generally or the software or systems that make the Services available;

**(e)** unauthorized access to or alterations of your transmissions or data;

**(f)** statements or conduct of any third party on the Services;

**(g)** any other user interactions that you input or receive through your use of the Services, including your use, access and/or download of Shared Content;

**(h)** any other matter relating to the Services.

**7.1. Unlimited Liabilities**

Nothing in the Agreement excludes or limits either party's Liability for:

**(a)** its fraud or fraudulent misrepresentation;

**(b)** its obligations under Section 6 (Indemnification);

**(c)** its infringement of the other party's Intellectual Property Rights;

**(d)** its payment obligations under the Agreement;

**(e)** matters for which liability cannot be excluded or limited under applicable law.

**7.2. Limitation on Amount of Liability**

Each party's total aggregate Liability for damages arising out of or relating to the Agreement is limited to the Fees Customer paid for the Service giving rise to the liability during the 12 month period before the event giving rise to Liability. Kamu’s total aggregate Liability for damages arising out of or related to Services or Software provided free of charge is limited to \$1,000 USD.

## 8. Disclaimer

The Services and Kamu Content are provided “as is” and “as available,” without warranty of any kind. Except to the extent prohibited by law, or to the extent any statutory rights apply that cannot be excluded, limited or waived, Kamu and its affiliates or licensors **(a)** make no representations or warranties of any kind, whether express, implied, statutory or otherwise, regarding the Services, Kamu Content or the third-party Content, and **(b)** disclaim all warranties, including any implied, statutory or express warranties **(i)** of title, merchantability, satisfactory quality, fitness for a particular purpose, accuracy or non-infringement, **(ii)** arising out of any course of dealing or usage of trade, **(iii)** that the Services or Kamu Content or third-party Content will be uninterrupted, timely, error-free or free of harmful components, and **(iv)** that any content will be secure or not otherwise lost or altered, **(v)** the results that may be obtained from the use of the Services will be effective, accurate or reliable, **(vi)** the quality of the Services will meet your expectations, or that **(vii)** any errors or defects in the Services will be corrected. No advice or information, whether oral or written, obtained by you from Kamu, or through or from use of the Services shall create any warranty not expressly stated in the terms of this Agreement or other Terms.

Kamu does not control, endorse, or accept responsibility for any Content (including Shared Content), software, data or service offered by third parties accessible through its Services. Kamu makes no representations or warranties whatsoever about, and shall not be liable for, any such third parties, their Content, or services. Any dealings that you may have with such third parties are at your own risk, and you will be solely responsible for any damage or loss that results from the download or use of any such Content. Additionally, to the maximum extent permitted under applicable law, Kamu disclaims all, and assumes no, liability for any illegal or damaging Shared Content that you access, download, use and/or obtain via the Services.

## 9. Governing Law, Jurisdiction and Venue

All issues and questions concerning the application, construction, validity, interpretation and enforcement of this Agreement shall be governed by and construed in accordance with the laws of the Province of British Columbia and the federal laws of Canada applicable therein. Any dispute, controversy or claim arising out of, relating to, or in connection with, this Agreement or any breach, termination or validity thereof (a "Dispute") shall be finally settled by arbitration. The arbitration shall be conducted in accordance with the Arbitration Act (British Columbia) in effect at the time of the arbitration, except as they may be modified herein or by agreement of the parties. The seat of the arbitration shall be the City of Vancouver.

## 10. Notices

Any notice required or permitted to be given hereunder shall be in writing and sent by the means of recorded electronic communications to addresses herein:

E-mail: [legal@kamu.dev](mailto:legal@kamu.dev)

Any notice sent by electronic communications shall be deemed to have been delivered on the date of receipt.


# Typography
Source: https://docs.kamu.dev/general/typography

Common UI elements used in documentation

This section provides examples of all style elements available to documentation authors and the guidelines on using them.

## Links

We have special types of links to refer readers to:

* definitions of terms in the [Glossary](/general/glossary)
* schemas in the [ODF Schemas Reference](/odf/schemas)

**Input:**

```
import {Term, Schema} from '/components/common.jsx'

Raw data is stored in <Term t="slices" id="data-slice"/> that
are linked from <Schema t="MetadataBlock"/>s.
```

**Result:**

Raw data is stored in <Term /> that are linked from <Schema />s.

## Code Blocks

**Input:**

````mdx theme={null} theme={null}
```js
const client = OdfClient::new("odf+https://localhost:8080");
```
````

**Result:**

```js theme={null}
const client = OdfClient::new("odf+https://localhost:8080");
```

## Text Blocks

**Input:**

```markdown theme={null}
<Tip>
Tip block
</Tip>

<Note>
Note block
</Note>

<Info>
Info block
</Info>

<Warning>
Warning block
</Warning>

<Danger>
Danger block
</Danger>
```

**Result:**

<Tip>
  Tip block
</Tip>

<Note>
  Note block
</Note>

<Info>
  Info block
</Info>

<Warning>
  Warning block
</Warning>

<Danger>
  Danger block
</Danger>

## Tables

Input:

```markdown theme={null}
| Name | Description |
| ---- | :---------: |
| Foo  |    Test     |
| Bar  |    Test     |
```

Result:

| Name | Description |
| - | :-: |
| Foo | Test |
| Bar | Test |

## Tabs

**Input:**

```markdown theme={null}
<Tabs>
  <Tab title="First tab">
    Tab 1 content with plain text
  </Tab>

  <Tab title="Second tab">
    Tab 2 with markdown
    - as
    - list
  </Tab>

  <Tab title="Third tab">
    <Info>Tab 3 content with rich elements</Info>
  </Tab>
</Tabs>
```

**Result:**

<Tabs>
  <Tab title="First tab">
    Tab 1 content with plain text
  </Tab>

  <Tab title="Second tab">
    Tab 2 with markdown

    * as
    * list
  </Tab>

  <Tab title="Third tab">
    <Info>Tab 3 content with rich elements</Info>
  </Tab>
</Tabs>

## Static Image

**Input:**

```markdown theme={null}
<img 
  src="/images/kamu-logo-dark.svg" 
  alt="Kamu logo"
  style={{width: "30%"}}
  className="rounded-lg"
/>
```

Result:

<img alt="Kamu logo" />

## YouTube Video

**Input:**

```markdown theme={null}
import {YouTube} from '/components/common.jsx'

<YouTube id="hN_vpHYmwi0"/>
```

**Result:**

<YouTube />

## YouTube Playlist

**Input:**

```markdown theme={null}
import {YouTubeList} from '/components/common.jsx'

<YouTubeList id="PLV91cS45lwVG20Hicztbv7hsjN6x69MJk"/>
```

**Result:**

<YouTubeList />

## Mermaid Diagrams

**Input:**

````mdx theme={null} theme={null}
```mermaid
sequenceDiagram
    Alice ->> Bob: Hello Bob, how are you?
    Bob-->>John: How about you John?
    Bob--x Alice: I am good thanks!
    Bob-x John: I am good thanks!
    Note right of John: Bob thinks a long<br/>long time, so long<br/>that the text does<br/>not fit on a row.

    Bob-->Alice: Checking with John...
    Alice->John: Yes... John, how are you?
```
````

**Result:**

```mermaid theme={null}
sequenceDiagram
    Alice ->> Bob: Hello Bob, how are you?
    Bob-->>John: How about you John?
    Bob--x Alice: I am good thanks!
    Bob-x John: I am good thanks!
    Note right of John: Bob thinks a long<br/>long time, so long<br/>that the text does<br/>not fit on a row.

    Bob-->Alice: Checking with John...
    Alice->John: Yes... John, how are you?

```

See more examples [here](https://mermaid.js.org/syntax/examples.html).


# Generic ADBC Client
Source: https://docs.kamu.dev/integrations/adbc

Connecting client applications via ADBC protocol.

To connect any client that supports [Apache Arrow ADBC protocol](https://arrow.apache.org/adbc/current/index.html) follow these simple steps:

1. Run `kamu` Flight SQL server in a desired workspace:
   ```bash theme={null}
   kamu sql server --address 127.0.0.1 --port 50050
   ```
2. [Follow the steps](https://arrow.apache.org/adbc/current/index.html) for your specific language and environment to connect your ADBC client to FlightSQL server
3. Use `anonymous` as login and password for anonymous connection (if allowed by the node) or authenticate using a personal access token

An example is [available here](https://github.com/kamu-data/kamu-cli/blob/master/examples/flight-sql/python/client_flightsql_adbc.py) using Python.


# Apache Superset
Source: https://docs.kamu.dev/integrations/apache-superset

Connecting Apache Superset to Kamu via Flight SQL.

[Apache Superset](https://superset.apache.org/) is an modern open-source data exploration and visualization platform. It can be connected to `kamu` via native [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) protocol using Python client.

In this setup:

* `kamu` runs a Flight SQL server - a high-performance protocol for data transfer
* In the `Superset` environment we install an additional `flightsql-dbapi` Python package
* `Superset` uses generic database API provided by `SQLAlchemy` framework
* `flightsql-dbapi` package provides custom engine implementation for `SQLAlchemy` that translates all `Superset`'s queries into Flight SQL protocol

To connect `Superset` to `kamu` follow these simple steps:

1. Start with being able to run `Superset` locally using `docker-compose` (see [this official guide](https://superset.apache.org/docs/installation/installing-superset-using-docker-compose/))
   * Rest of the guide assumes that you are launching superset in "non-dev" mode using:
     ```bash theme={null}
     docker-compose -f docker-compose-non-dev.yml up
     ```
2. Install [flightsql-dbapi](https://github.com/influxdata/flightsql-dbapi) Python package into `Superset` container:
   * Stop and clean up the environment:
     ```bash theme={null}
     docker-compose -f docker-compose-non-dev.yml down
     ```
   * Create `<superset repo>/docker/requirements-local.txt` file (as per [this guide](https://superset.apache.org/docs/databases/docker-add-drivers/)) with the following contents:
     ```
     # At the time of this writing Superset used arrow version with a critical to us bug
     pyarrow==13.0.0
     flightsql-dbapi==0.2.1
     ```
3. (Optional) Specify your MabBox API Token in `<superset repo>/docker/.env-non-dev`
4. Run `kamu` Flight SQL server in a desired workspace:
   ```bash theme={null}
   kamu sql server --address 0.0.0.0 --port 50050
   ```
5. Start `Superset` via `docker-compose` again
6. Create a new database connection in `Superset`
   * Use `"Other"` database kind
   * As URL specify:
     ```bash theme={null}
     datafusion+flightsql://anonymous:anonymous@<hostname or IP>:50050?insecure=True
     ```

* Skip `insecure=True` when node is set up with TLS
* To authenticate via access token use:
  ```bash theme={null}
  datafusion+flightsql://<hostname or IP>:50050?token=<TOKEN>
  ```


# DBeaver
Source: https://docs.kamu.dev/integrations/dbeaver

Connecting DBeaver via JDBC protocol.

[DBeaver](https://dbeaver.io/) is a great free database exploration tool. It can be connected to `kamu` using a JDBC driver.

In this setup:

* `kamu` runs an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) server - a high-performance protocol for data transfer
* `DBeaver` loads a special JDBC driver - a Java software module that implements an abstract database interface
* JDBC driver translates `DBeaver`'s commands into efficient Flight SQL calls in `kamu`

To connect `DBeaver` to `kamu` follow these simple steps:

1. Downlaod the latest [Flight SQL JDBC Driver](https://central.sonatype.com/artifact/org.apache.arrow/flight-sql-jdbc-driver)
2. Add a new database driver in `DBeaver` by following [these steps](https://dbeaver.com/docs/dbeaver/Database-drivers/)
   * For class name use:
     ```
     org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver
     ```
   * For URL template use:
     ```
     jdbc:arrow-flight-sql://{host}:{port}
     ```
   * In libraries add the driver's `.jar` file you just downloaded
3. Run `kamu` Flight SQL server in a desired workspace:
   ```bash theme={null}
   kamu sql server --flight-sql --address 127.0.0.1 --port 50050
   ```
4. Create a new connection in `DBeaver` using the driver you created:
   * Specify host and port, e.g. `127.0.0.1:50050`
   * Use `"anonymous"` as username and password or specify a token for authentication
   * In `Driver properties` tab add user property:
     ```bash theme={null}
     useEncryption=false
     ```


# Generic JDBC Client
Source: https://docs.kamu.dev/integrations/jdbc

Connecting client applications via JDBC protocol.

To connect any client that supports JDBC protocol follow these simple steps:

1. Downlaod the latest [Flight SQL JDBC Driver](https://central.sonatype.com/artifact/org.apache.arrow/flight-sql-jdbc-driver)
2. Install the driver following the instructions of your specific client application
3. Run `kamu` Flight SQL server in a desired workspace:
   ```bash theme={null}
   kamu sql server --engine datafusion --address 127.0.0.1 --port 50050
   # or
   kamu sql server --engine spark --address 127.0.0.1 --port 50050
   ```
4. Connect the client using URL:
   ```bash theme={null}
   jdbc:arrow-flight-sql://127.0.0.1:50050?useEncryption=false
   ```
5. Use `anonymous` as login and password or provide an access token to authenticate

An example is [available here](https://github.com/kamu-data/kamu-cli/blob/master/examples/flight-sql/python/client_jdbc.py) using Python.


# Generic ODBC Client
Source: https://docs.kamu.dev/integrations/odbc

Connecting client applications via ODBC protocol.

We do not currently support ODBC protocol natively. If your use case requires it - [let us know](https://github.com/kamu-data/kamu-cli/issues).

As a workaround you may consider using a 3rd party ODBC-to-JDBC bridge.


# Overview
Source: https://docs.kamu.dev/integrations/overview



`kamu` is a data supply chain that bringing fresh and reliable data to the tools you work with. It is designed to integrate with a wide range of modern analytics, business inteligence, and AI/ML apps and frameworks, letting you put the data from `kamu` to good use.

Don't see an integration you need? Please [file an issue](https://github.com/kamu-data/kamu-cli/issues) in our repository.


# PowerBI
Source: https://docs.kamu.dev/integrations/powerbi

Connecting PowerBI via ODBC protocol.

We do not currently support ODBC protocol and in the process of developing a PowerBI connector. If yor use case requires it - [let us know](https://github.com/kamu-data/kamu-cli/issues).

If you want to connect PowerBI to a Kamu Node consider trying the [OData connector](/api/odata).


# Python Scripts & Notebooks
Source: https://docs.kamu.dev/integrations/python

Connecting Python applications to Kamu.

This guide explains how to connect to Kamu from Python scripts, applications, and Notebook environments like Jupyter.

## Official Client Library

The easiest way to get started is the official [`kamu`](https://github.com/kamu-data/kamu-client-python) Python library available on [PyPi](https://pypi.org/project/kamu/).

Insall:

```sh theme={null}
pip install kamu
```

Connect and query data via SQL:

```python theme={null}
import kamu

# Create connection
con = kamu.connect("grpc+tls://node.demo.kamu.dev:50050")

# Returns a Pandas dataframe
con.query("selec 1 as value")
```

Take a look at the [library documentation](https://github.com/kamu-data/kamu-client-python) for examples of authenticating, using different engines, and connecting to local workspaces.

## Jupyter Notebooks

The official [`kamu`](https://github.com/kamu-data/kamu-client-python) Python library offers a few optional Jupyter integrations that you can install as:

```sh theme={null}
pip install[jupyter-autoviz,jupyter-sql]
```

Import the extension at the beginning of your notebook:

```python theme={null}
%load_ext kamu
```

Create a connection:

```python theme={null}
import kamu
con = kamu.connect("grpc+tls://node.demo.kamu.dev:50050")
```

You can now use `%%sql` cell magic:

```sql theme={null}
%%sql
select 1 as value
```

This extension is pre-installed in the integrated notebook environment you can run via `kamu notebook` command line command.

See [Exploring Data](/cli/explore#jupyter-notebooks) section for a tutorial on using notebooks.

For more information on extensions see the [library documentation](https://github.com/kamu-data/kamu-client-python).

## Google Colab

To install `kamu` in Colab environment use:

```
!pip install kamu
```

See also [example Google Colab notebook](https://colab.research.google.com/drive/1WQqZJsPQpipU4kW6SPea9H2qmHH4rF8k).

## Alternative Options

### SQL Using ADBC

Kamu is based on the [ADBC](https://arrow.apache.org/docs/format/ADBC.html) client API standard and the [Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) protocol, so any library that can work with those should be able to connect.

Example using the official ADBC library and Pandas:

```python theme={null}
import adbc_driver_manager
import adbc_driver_flightsql.dbapi
import pandas

con = adbc_driver_flightsql.dbapi.connect(
    "grpc+tls://node.demo.kamu.dev:50050",
    db_kwargs={
        # Anonymous users have to authenticate using basic auth so they could be assigned a session token
        adbc_driver_manager.DatabaseOptions.USERNAME.value: "anonymous",
        adbc_driver_manager.DatabaseOptions.PASSWORD.value: "anonymous",
        # Registered users can provide a bearer token directy
        # adbc_driver_flightsql.DatabaseOptions.AUTHORIZATION_HEADER.value: "Bearer <token>",
    },
    autocommit=True,
)

with con:
    # Pandas natively supports ADBC connections!
    df = pandas.read_sql("select 1 as value", con)
    print(df)
```

### SQL Using SQLAlchemy

[SQLAlchemy](https://www.sqlalchemy.org/) is a popular ORM library for Python that you can use to access data in Kamu in a same way as dozens of other data sources. It is row-oriented, so may be less efficient than using ADBC client for certain use cases. To use it you'll need to install [flightsql-dbapi](https://github.com/influxdata/flightsql-dbapi) package.

Example:

```python theme={null}
import flightsql.sqlalchemy
import sqlalchemy
import pandas as pd

# Secure remote connection
engine = sqlalchemy.create_engine(
    # Anonymous users have to authenticate using basic auth so they could be assigned a session token
    "datafusion+flightsql://anonymous:anonymous@node.demo.kamu.dev:50050"
    # Registered users can provide a bearer token directy
    # "datafusion+flightsql://node.demo.kamu.dev:50050?token=kamu-token"
)

with engine.connect() as con:
    df = pd.read_sql(sql="select 1 as value", con=con.connection)
    print(df)
```

### SQL Using DBAPI2

DBAPI2 (aka [PEP-249](https://peps.python.org/pep-0249/)) is a standard DB access API in Python, but its row-oriented access to data may be less efficient than using ADBC client. To use it you'll need to install [flightsql-dbapi](https://github.com/influxdata/flightsql-dbapi) package.

Example:

```python theme={null}
from flightsql import connect, FlightSQLClient
import pandas

# Secure remote connection
client = FlightSQLClient(
    host="node.demo.kamu.dev",
    port=50050,
    # Anonymous users have to authenticate using basic auth so they could be assigned a session token
    user="anonymous",
    password="anonymous",
    # Registered users can provide a bearer token
    # token="<kamu-token>",
)

df = pandas.read_sql("select 1 as value", con)
print(df)
```


# Tableau
Source: https://docs.kamu.dev/integrations/tableau

Connecting Tableau via JDBC protocol.

[Tableau](https://www.tableau.com/) is a professional business intelligence tool. It can be connected to `kamu` using a JDBC driver.

In this setup:

* `kamu` runs an [Arrow Flight SQL](https://arrow.apache.org/docs/format/FlightSql.html) server - a high-performance protocol for data transfer
* `Tableau` loads a special JDBC driver - a Java software module that implements an abstract database interface
* JDBC driver translates `Tableau`'s commands into efficient Flight SQL calls in `kamu`

To connect `Tableau` to `kamu` follow these simple steps:

1. Downlaod the latest [Flight SQL JDBC Driver](https://central.sonatype.com/artifact/org.apache.arrow/flight-sql-jdbc-driver)
2. Install the driver into `Tableau` (see [official instructions](https://help.tableau.com/current/pro/desktop/en-us/examples_otherdatabases_jdbc.htm)) by copying the `.jar` file to:
   * Windows: `C:\Program Files\Tableau\Drivers`
   * Mac: `~/Library/Tableau/Drivers`
   * Linux: `/opt/tableau/tableau_driver/jdbc`
3. Run `kamu` Flight SQL server in a desired workspace:
   ```bash theme={null}
   kamu sql server --flight-sql --address 127.0.0.1 --port 50050
   ```
4. Create a new connection in `Tableau` using `Other Databases (JDBC)` as a source:
   * Select `PostgreSQL` in the dialect selector
   * As URL specify:
     ```bash theme={null}
     jdbc:arrow-flight-sql://127.0.0.1:50050?useEncryption=false
     ```
   * Use `"kamu"` as username and password


# About
Source: https://docs.kamu.dev/node/about



<div>
  <img alt="Kamu logo" />

  <div>
    <a href="https://github.com/kamu-data/kamu-node/">
      <img />
    </a>

    <a href="https://github.com/kamu-data/kamu-node/releases/latest">
      <img />
    </a>

    <a href="https://github.com/kamu-data/kamu-node/actions">
      <img />
    </a>

    <a href="https://discord.gg/nU6TXRQNXC">
      <img />
    </a>

    <a href="/api/rest">
      <img />
    </a>

    <a href="/api/graphql">
      <img />
    </a>
  </div>
</div>

## About

Kamu Compute Node is a set of [Kubernetes](https://kubernetes.io/)-native applications that can be deployed in any cloud or on-prem to:

* Operate the stream processing pipelines for a certain set of data flows
* Continuously ingest external data into signed root datasets
* Continuously verify a configured set of pipelines to catch malicious behavior
* Execute queries on co-located data and provide data via variety of protocols
* Provide data to smart contracts via multi-chain ODF Oracle mechanism

Nodes are the building pieces of the [Open Data Fabric](/odf) and the primary way of contributing resources to the network. Unlike blockchain nodes that maintain a single ledger, Kamu nodes can form loosely connected clusters based on vested interests of their operators in certain data pipelines.

If you are new to ODF - we recommend you to start with [Kamu CLI](/cli) for a gradual introduction.

You should consider Kamu Node when you want to:

* Build a horizontally-scalable lakehouse for your data
* Need a decentralized infrastructure for sharing data with your partners or globally without intermediaries
* Want to continuously operate ODF data pipelines or verify data
* Need a rich set of [data APIs](/api)
* Want to provide data to [ODF blockchain oracle](/api/oracle)


# Architecture
Source: https://docs.kamu.dev/node/architecture



[Kamu Node](/node) is a Kubernetes-native set of applications designed to be easily installable on-premise and in private cloud.

This diagram shows a complete deployment of [Kamu Node](/node) and [Kamu Web Platform](/platform):

<Diagram alt="Kamu Node and Platform deployment components" />

## Dataset Storage

ODF datasets are always safely stored in the storage of your choice, separate from the rest of the system. Multiple [storage systems](/node/deploy/storage) are supported.

<Term /> and <Term /> datasets can be stored separately, so that only root datasets were (geo-)replicated to safeguard them from catastrophic loss of data. The property that derivative datasets [can be fully reconstructed](/odf/spec/transformations/processing-model#derivative-data-transience) from metadata can enable significant cost savings.

## Operational Storage

This set of storage systems is needed for operating the node and web platform.

**Node data:**

* **Metadata** - used for efficient access to metadata chains of datasets without making too many queries to dataset storage
* **Flows & Tasks** - stores the configuration, execution state, and history of various operations performed by the node (ingesting data, derivative transforms, data queries)
* **Monitoring** - stores traces and metrics for monitoring the health of the deployment

**Web platform data:**

* **Auth** - stores user and organization accounts, permissions, and linked cryptographic keys
* **Governance** - stores issues, discussions, comments, attachments and other information associated with datasets

## Node

### Data Processing Layer

To perform data-intensive operations on datasets the node maintains a **pool of engines** of needed type and version (see also [Supported Engines](/general/supported-engines)). Acting as a [Kubernetes controller](https://kubernetes.io/docs/concepts/architecture/controller/) the **Engine Provisioner** decides when to start and stop individual **engine pods** based on demand, configuration, and available capacity in the cluster.

Unlike many streaming data processing engines Kamu Node does not keep pipelines in memory, running all the time. Slow-moving pipelines get suspended into checkpoints and spawned only when needed. This allows Kamu Node to use even very limited resources to switch between many thousands of concurrent pipelines based on defined priorities.

The current demand for data processing is managed by the **Task Scheduler** that understands which datasets need to be updated, prioritizes individual tasks, and hands them over to executors.

Another interesting aspect is that multiple different versions of the same engine type can be used at the same time to ensure reproducibility of derivative data.

### API Layer

These services are the "brain" of the node. They serve API requests as well as decide when to perform various processing and maintenance tasks on datasets and pipelines.

Data-intensive operations like push ingest and SQL queries are handled by the **Data Gateway** component that exposes data under a wide variety of protocols for reading and writing.

All other kinds of requests are handled by the pool of **API Server** components that provide access to all functionality of the node via GraphQL and REST APIs.

Services communicate with one-another via **Event Bus** component.

The **Oracle Provider** is an optional component of the API layer that can supply data to various blockchains. See <Term /> for explanation of its function. Provider has to be configured with the address(es) of the blockchain nodes to get requests from, the address of the API server to use for executing queries, and the wallet key for executing transactions that carry data into smart contracts.

## Web Platform

[Kamu Web Platform](/platform) is a browser application that is served to the browser as a set of static files. The **Web Server** pods are stateless and linearly scalable. Since application files change infrequently, they can be fronted by a CDN to optimize loading times and decrease the traffic to the deployment.

Web platform application running in a browser communicates with **API Server** via GraphQL protocol.

Optionally, other front-end applications like [Jupyter Hub](https://z2jh.jupyter.org/) and [Apache Superset](https://superset.apache.org/) can be deployed to provide data science and business intelligence capabilities on top of Kamu Node. We often include them in our deployment examples to show how to connect them to Kamu.


# Verifiable Queries
Source: https://docs.kamu.dev/node/commitments



## Motivation

When you query data in `kamu`, we create an unbreakable **cryptographic commitment** that allows you to hold every party in the data supply chain accountable for veracity of the result.

This accountability should persist long after the response is provided, so that participants can process data fast in “optimistic” fashion, and can **verify and dispute the results later**.

This mechanism is foundational to **automatic dispute resolution** mechanism of the ODF network that can financially penalize malicious activity.

## Commitment Properties

* **Hiding** - Commitment should not reveal any information about the operation.
* **Binding** - It should be impossible to change any inputs or outputs of the operation without invalidating the commitment.
* **Succinct** - Commitment itself should be small enough to cheaply store on-chain. The process of disputing the operation results should not require storing and transferring the actual data.
* **Deterministic** - Same query should result in the same commitment as long as the query itself is deterministic.
* **Reproducible** - Given all the same inputs of the operation a third party should be able to repeat the query, arrive at the same results, and be able to successfully validate them against the original commitment.
* **Non-repudiable** - A prover cannot dispute performing the operation. There exists a mechanism that ties its identity to the provided commitment. This mechanism may be separate from the commitment itself to keep the latter pure zero-knowledge
* **Composable** - It should be possible to split computation into several steps and compose their commitments into one final commitment. When joining two datasets from different owners the entire execution may be split into sub-plans that are executed by different nodes belonging to different parties. In the event of a dispute, it should be possible to assign responsibility to the specific party and not just stop at the party that performed the final result aggregation.
* **Multi-protocol** - Ideally the commitment structure can be reused without significant alterations across multiple protocols like REST, Blockchain Oracle, FlightSQL, Kafka egress, WebSockets etc.

## Structured Recursive Commitments

Let's explore how the commitment scheme works using a [REST batch query API](/api/rest):

```python theme={null}
POST /query
Host: node.demo.kamu.dev

{
  "query": "select * from foo",
  "queryDialect": "SqlDataFusion",
  "dataFormat": "JsonAoS",
  "schemaFormat": "ArrowJson",
  "include": ["input", "schema", "proof"]
}
```

```python theme={null}
200 OK

{
  # Input section that can be used to fully reproduce the result of operation
  "input": {
    "query": "select * from foo",
    "queryDialect": "SqlDataFusion",
    "dataFormat": "JsonAoS",
    "schemaFormat": "ArrowJson",
    "datasets": [{
      "id": "did:odf:fed0..ee02",
      "alias": "foo",
      "blockHash": "f162..3e0b",
    }]
  },

  # The result
  "output": {
    "data": [
      {"foo": 1, "bar": "A"},
      {"foo": 2, "bar": "B"}
    ],
    "dataFormat": "JsonAoS",
    "schema": {...},
    "schemaFormat": "ArrowJson",
  },

  # Information about processing performed by other nodes as part of this operation
  # that are identical in structure to the top-level response (recursive)
  "subQueries": [{
    "input": {...},
    "subQueries": [...],
    "commitment": {...},
    "proof": {...},
  }]

  # Succinct commitment
  "commitment": {
    # Input hash: multihash(canonicalize(input))
    "inputHash": "f162..3e0b",

    # Output hash: multihash(canonicalize(output))
    "outputHash": "f162..baaf",

    # Sub-queries hash: multihash(canonicalize(subQueries))
    "subQueriesHash": "f162..8dae",
  }

  # Signature block
  "proof": {
    "type": "Ed25519Signature2020",

    # DID (public key) of the node performing the computation
    "verificationMethod": "did:key:fed0..3e41",

    # Signature: multibase(sign(canonicalize(commitment)))
    "proofValue": "u3MvG...ouQk"
  }
}
```

**Note:** Technically the commitment block is redundant and the caller could compute all hashes themselves. In fact they have to do so to validate the response (see below). We include this block purely to:

* Protect clients from implementing hashing in a way that is not consistent with the specification
* Provide more information during the validation and dispute stages to isolate inconsistencies to specific parts.

### JSON Hashing

This scheme involves hashing JSON objects. To exclude variations like whitespace characters and property order differences between JSON library implementations we need to convert it into canonical form as specified in [RFC 8785: JSON Canonicalization Scheme](https://datatracker.ietf.org/doc/html/rfc8785).

### Response Validation

Immediately upon receiving the response the caller must validate it to ensure that the commitment is not bogus and is disputable:

1. Verify that `proof.verificationMethod` DID belongs to the node you are requesting data from
2. Verify `commitment.outputHash` by re-computing the hash
3. Verify `commitment.inputHash` by re-computing the hash
4. Verify `commitment.subQueriesHash` by re-computing the hash
5. Verify the signature in proof.proofValue usings
   1. Message: `canonicalize(commitment)`
   2. Public key: `key_from_did(proof.verificationMethod)`
6. Repeat the steps 3-5 recursively for every entry in subQueries

### Storing Commitments

To be able to dispute the result in future, consumer should store the following fields:

* `input`
* `subQueries`
* `commitment`
* `proof`

These different parts can be also used separately for different purposes as they all provide different “layers” of verifiability:

* `input` - provides reproducibility
* `commitment` - seals the response against tampering
* `proof` - adds non-repudiation via signature
* `subQueries` - provide the above for multi-party computations

### Dispute Resolution

The procedure of resolving a dispute using a commitment via reproducibility is as follows:

1. Consumer submits a claim to the verifier by providing:
   * input
   * commitment
   * subQueries
   * proof
2. Verifier obtains the relevant input.datasets
3. Verifier validates the claim according to the same rules as described in Response Validation
   * If any check fails the claim is rejected as invalid
4. Verifier checks that all authority specified in subQueries are registered nodes
   * In an unregistered node is found the claim is accepted and the blame is assigned to the root authority for using a bogus node for computing part of the response
5. Verifier generates the logical query plan and checks that the plan is identical or equivalent to the plan composed from the subQueries
   * If plan does not correspond to the original query the claim is accepted and the blame is assigned to the root authority for providing a malicious query plan
6. In a bottom-up fashion Verifier repeats all the computation steps using input and compares hashes of outputs to those captured in commitment.outputHash
   * If a hash doesn’t match the claim is accepted and the blame is assigned to the corresponding authority for providing a malicious output data

Properties:

* **Hiding** - YES\*
  * The hashes in the commitment block can be used as zero-knowledge commitments that do not reveal any information
  * The public part of the proof reveals only the identity of the node that performed the computation
* **Biding** - YES
* **Deterministic** - YES
  * Assuming canonical response format
* **Reproducible** - YES
  * The `input` field contains all the necessary info to reproduce the request
* **Non-repudiable** - YES
  * Through the identity and signature in the `proof`
* **Composable** - YES
  * Proof is recursive via `subQueries` field
* **Multi-protocol** - YES
  * A similar scheme can be implemented in other protocols.

## Commitments in ODF Oracle

For the explanation of oracle commitments see [ODF Oracle protocol documentation](/api/oracle).

## Zero-Knowledge Proofs

Currently ZKs are very limited in terms of data types and functions they can support. Supporting JOINs is especially problematic. We are working on introducing ZK proofs progressively, where possible for simpler queries, as an alternative to verification through signing and reproducibility.

## Alternatives

### HTTP Message Signatures

[RFC 9421: HTTP Message Signatures](https://datatracker.ietf.org/doc/rfc9421/) proposes a standard for signing HTTP requests and responses.
Under this scheme a query operation could look like this:

```python theme={null}
POST /query
Host: node.demo.kamu.dev
Accept-Signature: sig=(@status, ...)

{
  "query": "select * from foo",
  "queryFormat": "SqlDataFusion",
  "aliases": [{"alias": "foo", "id": "did:odf:fed0..ee02"}],
  "dataFormat": "JsonAoS"
}
```

```python theme={null}
200 OK
Signature-Input: (@status, ...)
Signature: sig=:...:

{
  "input": {
    "query": "select * from foo",
    "queryDialect": "SqlDataFusion",
    "datasets": [{"id": "did:odf:fed0..ee02", "alias": "foo", "blockHash": "f162..3e0b"}],
  },
  "output": {
    "dataFormat": "JsonAoS",
    "data": [
      {"foo": 1, "bar": "A"},
      {"foo": 2, "bar": "B"}
    ],
  }
}
```

In this case the commitment would consist of the entire HTTP response (excluding only some headers that are not part of the Signature-Input).

* **Hiding** - YES / NO
  * Depending on the algorithm, signature may reveal the public key of the signer and therefore the identity of a node that provided the response.
* **Biding** - YES
* **Succinct** - **NO**
  * The signature itself is small and constant in size, however to dispute the operation a client will have to hold on to the entire response data to show the validity of a signature before verifier attempts to reproduce the request - this would be impractical at scale
  * RFC specifies `Content-Digest` header that may be included into the signature instead of the entire payload, thus allowing not to store data, but this is insufficient due to other issues
* **Deterministic** - YES
  * Assuming stable / canonical response format
* **Reproducible** - YES
  * Note that the response would include the query and all other input parameters, so removing the output fields like data and schema gives us all the necessary inputs to reproduce the request
* **Non-repudiable** - YES
  * Through the signature
* **Composable** - **NO**
  * Including commitments from other nodes that performed parts of the request will require some different mechanism where HTTP signatures are included as parts of the JSON response
    Nodes may not even use HTTP when communicating with one another and use FlightSQL instead.
* **Multi-protocol** - **NO**
  * This scheme is HTTP specific - other protocols will have to use other methods.


# Config Reference
Source: https://docs.kamu.dev/node/config



## `ApiServerConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>auth</code></td>
      <td><a href="#authconfig"><code>AuthConfig</code></a></td>
      <td><pre><code>\{
      "allowAnonymous": true,
      "didEncryption": \{
      "enabled": false
      },
      "jwtSecret": "",
      "passwordPolicy": \{
      "minNewPasswordLength": 8
      },
      "providers": \[]
      }</code></pre></td>
      <td>Authentication & authorization</td>
    </tr>

    <tr>
      <td><code>backgroundAgents</code></td>
      <td><a href="#backgroundagentsconfig"><code>BackgroundAgentsConfig</code></a></td>
      <td><pre><code>\{
      "batching": \{
      "flowActivations": 100,
      "flowSystemEvents": 100,
      "outboxMessages": 100
      },
      "concurrency": \{
      "outboxConsumers": 8
      },
      "maxListeningTimeout": "2m",
      "minDebounceInterval": "20ms"
      }</code></pre></td>
      <td>Background agents configuration (outbox, flow system events, tasks)</td>
    </tr>

    <tr>
      <td><code>database</code></td>
      <td><a href="#databaseconfig"><code>DatabaseConfig</code></a></td>
      <td><pre><code>\{
      "provider": "InMemory"
      }</code></pre></td>
      <td>Database</td>
    </tr>

    <tr>
      <td><code>email</code></td>
      <td><a href="#emailconfig"><code>EmailConfig</code></a></td>
      <td><pre><code>\{
      "gateway": \{
      "kind": "Dummy"
      },
      "senderAddress": ""
      }</code></pre></td>
      <td>Email gateway configuration</td>
    </tr>

    <tr>
      <td><code>engine</code></td>
      <td><a href="#engineconfig"><code>EngineConfig</code></a></td>
      <td><pre><code>\{
      "datafusionEmbedded": \{
      "base": \{
      "datafusion.catalog.default\_catalog": "kamu",
      "datafusion.catalog.default\_schema": "kamu",
      "datafusion.catalog.information\_schema": "true",
      "datafusion.sql\_parser.enable\_ident\_normalization": "false"
      },
      "batchQuery": \{},
      "compaction": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "ingest": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "useLegacyArrowBufferEncoding": false
      },
      "images": \{
      "datafusion": "ghcr.io/kamu-data/engine-datafusion:0.9.0",
      "flink": "ghcr.io/kamu-data/engine-flink:0.18.2-flink\_1.16.0-scala\_2.12-java8",
      "risingwave": "ghcr.io/kamu-data/engine-risingwave:0.3.0",
      "spark": "ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"
      },
      "networkNs": "Private",
      "runtime": "Podman",
      "shutdownTimeout": "5s",
      "startTimeout": "30s"
      }</code></pre></td>
      <td>Ingest and transform engines</td>
    </tr>

    <tr>
      <td><code>extra</code></td>
      <td><a href="#extraconfig"><code>ExtraConfig</code></a></td>
      <td><pre><code>\{
      "graphql": \{}
      }</code></pre></td>
      <td>Experimental and temporary module configuration</td>
    </tr>

    <tr>
      <td><code>flowSystem</code></td>
      <td><a href="#flowsystemconfig"><code>FlowSystemConfig</code></a></td>
      <td><pre><code>\{
      "awaitingStepSecs": 1,
      "defaultRetryPolicies": \{},
      "mandatoryThrottlingPeriodSecs": 60
      }</code></pre></td>
      <td>Configuration for the flow system</td>
    </tr>

    <tr>
      <td><code>identity</code></td>
      <td><a href="#identityconfig"><code>IdentityConfig</code></a></td>
      <td><code>\{}</code></td>
      <td>UNSTABLE: Identity configuration</td>
    </tr>

    <tr>
      <td><code>protocol</code></td>
      <td><a href="#protocolconfig"><code>ProtocolConfig</code></a></td>
      <td><pre><code>\{
      "flightSql": \{
      "allowAnonymous": true,
      "anonSessionExpirationTimeout": "5m",
      "anonSessionInactivityTimeout": "5s",
      "authedSessionExpirationTimeout": "30m",
      "authedSessionInactivityTimeout": "5s"
      },
      "ipfs": \{
      "httpGateway": "http\://localhost:8080/",
      "preResolveDnslink": true
      }
      }</code></pre></td>
      <td>Protocols</td>
    </tr>

    <tr>
      <td><code>quota</code></td>
      <td><a href="#quotaconfig"><code>QuotaConfig</code></a></td>
      <td><pre><code>\{
      "account": \{}
      }</code></pre></td>
      <td>Default quotas configured by type</td>
    </tr>

    <tr>
      <td><code>repo</code></td>
      <td><a href="#repoconfig"><code>RepoConfig</code></a></td>
      <td><pre><code>\{
      "caching": \{
      "registryCacheEnabled": false
      }
      }</code></pre></td>
      <td>Dataset repository</td>
    </tr>

    <tr>
      <td><code>runtime</code></td>
      <td><a href="#runtimeconfig"><code>RuntimeConfig</code></a></td>
      <td><code>\{}</code></td>
      <td>Tokio runtime</td>
    </tr>

    <tr>
      <td><code>search</code></td>
      <td><a href="#searchconfig"><code>SearchConfig</code></a></td>
      <td><pre><code>\{
      "embeddingsChunker": \{
      "kind": "Simple",
      "splitParagraphs": false,
      "splitSections": false
      },
      "embeddingsEncoder": \{
      "dimensions": 1536,
      "kind": "OpenAi",
      "modelName": "text-embedding-ada-002"
      },
      "indexer": \{
      "clearOnStart": false,
      "incrementalIndexing": false
      },
      "repo": \{
      "kind": "Dummy"
      },
      "semanticSearchThresholdScore": 0.0
      }</code></pre></td>
      <td>Search configuration</td>
    </tr>

    <tr>
      <td><code>secretsEncryption</code></td>
      <td><a href="#secretsencryptionconfig"><code>SecretsEncryptionConfig</code></a></td>
      <td><pre><code>\{
      "enabled": false
      }</code></pre></td>
      <td>Secrets encryption configuration</td>
    </tr>

    <tr>
      <td><code>source</code></td>
      <td><a href="#sourceconfig"><code>SourceConfig</code></a></td>
      <td><pre><code>\{
      "ethereum": \{
      "commitAfterBlocksScanned": 1000000,
      "getLogsBlockStride": 100000,
      "rpcEndpoints": \[],
      "useBlockTimestampFallback": false
      },
      "mqtt": \{
      "brokerIdleTimeoutMs": 1000
      },
      "targetRecordsPerSlice": 10000
      }</code></pre></td>
      <td>Ingestion's sources</td>
    </tr>

    <tr>
      <td><code>uploadRepo</code></td>
      <td><a href="#uploadrepoconfig"><code>UploadRepoConfig</code></a></td>
      <td><pre><code>\{
      "maxFileSizeMb": 50,
      "storage": \{
      "kind": "Local"
      }
      }</code></pre></td>
      <td>File upload repository</td>
    </tr>

    <tr>
      <td><code>url</code></td>
      <td><a href="#urlconfig"><code>UrlConfig</code></a></td>
      <td><pre><code>\{
      "baseUrlFlightsql": "grpc://localhost:50050",
      "baseUrlPlatform": "http\://localhost:4200/",
      "baseUrlRest": "http\://localhost:8080/"
      }</code></pre></td>
      <td>External URLs</td>
    </tr>

    <tr>
      <td><code>webhooks</code></td>
      <td><a href="#webhooksconfig"><code>WebhooksConfig</code></a></td>
      <td><pre><code>\{
      "deliveryTimeoutSecs": 10,
      "maxConsecutiveFailures": 5,
      "secretEncryptionEnabled": false
      }</code></pre></td>
      <td>Configuration for webhooks</td>
    </tr>
  </tbody>
</table>

## `AccountConfig`

The declarative account configuration used to register an account if one
does not already exist.

To update an existing account, either `id` or `private_key` must be
specified.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>accountName</code></td>
      <td><a href="#accountname"><code>AccountName</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>accountType</code></td>
      <td><a href="#accounttype"><code>AccountType</code></a></td>
      <td><code>"User"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>avatarUrl</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>displayName</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Auto-derived from `account_name` if omitted
      </td>
    </tr>

    <tr>
      <td><code>email</code></td>
      <td><a href="#email"><code>Email</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>id</code></td>
      <td><a href="#accountid"><code>AccountID</code></a></td>
      <td><code>null</code></td>

      <td>
        May be omitted in favor of `private_key`.
      </td>
    </tr>

    <tr>
      <td><code>password</code></td>
      <td><a href="#password"><code>Password</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>privateKey</code></td>
      <td><a href="#privatekey"><code>PrivateKey</code></a></td>
      <td><code>null</code></td>

      <td>
        Optional ed25519 private key. When set, `id` is derived from it
        (and must match `id` if both are present).
      </td>
    </tr>

    <tr>
      <td><code>properties</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>

      <td />
    </tr>

    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>
      <td><code>"password"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>providerIdentityKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Auto-derived from `account_name` if omitted
      </td>
    </tr>

    <tr>
      <td><code>registeredAt</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>treatDatasetsAsPublic</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `AccountID`

Base type: `string`

## `AccountName`

Base type: `string`

## `AccountPropertyName`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>CanProvisionAccounts</code></td></tr>
    <tr><td><code>Admin</code></td></tr>
  </tbody>
</table>

## `AccountType`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>User</code></td></tr>
    <tr><td><code>Organization</code></td></tr>
  </tbody>
</table>

## `AuthConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>allowAnonymous</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>

      <td />
    </tr>

    <tr>
      <td><code>didEncryption</code></td>
      <td><a href="#didsecretencryptionconfig"><code>DidSecretEncryptionConfig</code></a></td>
      <td><pre><code>\{
      "enabled": false
      }</code></pre></td>

      <td />
    </tr>

    <tr>
      <td><code>jwtSecret</code></td>
      <td><code>string</code></td>
      <td><code>""</code></td>

      <td />
    </tr>

    <tr>
      <td><code>passwordPolicy</code></td>
      <td><a href="#passwordpolicyconfig"><code>PasswordPolicyConfig</code></a></td>
      <td><pre><code>\{
      "minNewPasswordLength": 8
      }</code></pre></td>

      <td />
    </tr>

    <tr>
      <td><code>providers</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `AuthProviderConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#authproviderconfiggithub"><code>Github</code></a></td></tr>
    <tr><td><a href="#authproviderconfigpassword"><code>Password</code></a></td></tr>
  </tbody>
</table>

## `AuthProviderConfig::Github`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>clientId</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>clientSecret</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `AuthProviderConfig::Password`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>accounts</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `BackgroundAgentsBatchingConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>flowActivations</code></td>
      <td><code>integer</code></td>
      <td><code>100</code></td>

      <td>
        Due flows the flow agent loads at once before activating them.
        0 is treated as 1
      </td>
    </tr>

    <tr>
      <td><code>flowSystemEvents</code></td>
      <td><code>integer</code></td>
      <td><code>100</code></td>

      <td>
        Flow system events applied to a projection per transaction. 0 is
        treated as 1
      </td>
    </tr>

    <tr>
      <td><code>outboxMessages</code></td>
      <td><code>integer</code></td>
      <td><code>100</code></td>
      <td>Outbox messages relayed per transaction. 0 is treated as 1</td>
    </tr>
  </tbody>
</table>

## `BackgroundAgentsConcurrencyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>outboxConsumers</code></td>
      <td><code>integer</code></td>
      <td><code>8</code></td>

      <td>
        Outbox consumers running at once, in order per producer, each holding a
        pooled connection: e.g. 8 with Postgres, 1 with `SQLite`. 0 is treated
        as 1
      </td>
    </tr>
  </tbody>
</table>

## `BackgroundAgentsConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>batching</code></td>
      <td><a href="#backgroundagentsbatchingconfig"><code>BackgroundAgentsBatchingConfig</code></a></td>
      <td><pre><code>\{
      "flowActivations": 100,
      "flowSystemEvents": 100,
      "outboxMessages": 100
      }</code></pre></td>
      <td>Batch sizes of agents processing records in batches</td>
    </tr>

    <tr>
      <td><code>concurrency</code></td>
      <td><a href="#backgroundagentsconcurrencyconfig"><code>BackgroundAgentsConcurrencyConfig</code></a></td>
      <td><pre><code>\{
      "outboxConsumers": 8
      }</code></pre></td>
      <td>Concurrency limits of agents processing records in parallel</td>
    </tr>

    <tr>
      <td><code>maxListeningTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"2m"</code></td>

      <td>
        Fallback period to re-check for work if a change signal is missed.
        With `SQLite` it also paces the polling. The flow agent wakes up at
        the next flow activation moment regardless of it.
      </td>
    </tr>

    <tr>
      <td><code>minDebounceInterval</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"20ms"</code></td>
      <td>How long agents absorb a burst of change signals before processing</td>
    </tr>
  </tbody>
</table>

## `ContainerRuntimeType`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>Docker</code></td></tr>
    <tr><td><code>Podman</code></td></tr>
  </tbody>
</table>

## `DatabaseConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#databaseconfiginmemory"><code>InMemory</code></a></td></tr>
    <tr><td><a href="#databaseconfigsqlite"><code>Sqlite</code></a></td></tr>
    <tr><td><a href="#databaseconfigpostgres"><code>Postgres</code></a></td></tr>
  </tbody>
</table>

## `DatabaseConfig::InMemory`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseConfig::Sqlite`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>databasePath</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseConfig::Postgres`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>acquireTimeoutSecs</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>credentialsPolicy</code></td>
      <td><a href="#databasecredentialspolicyconfig"><code>DatabaseCredentialsPolicyConfig</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>databaseName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>host</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>maxConnections</code></td>
      <td><code>integer</code></td>
      <td><code>20</code></td>

      <td />
    </tr>

    <tr>
      <td><code>maxLifetimeSecs</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>port</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>provider</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseCredentialSourceConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#databasecredentialsourceconfigrawpassword"><code>RawPassword</code></a></td></tr>
    <tr><td><a href="#databasecredentialsourceconfigawssecret"><code>AwsSecret</code></a></td></tr>
  </tbody>
</table>

## `DatabaseCredentialSourceConfig::RawPassword`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>rawPassword</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>userName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseCredentialSourceConfig::AwsSecret`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>secretName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DatabaseCredentialsPolicyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>rotationFrequencyInMinutes</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>source</code></td>
      <td><a href="#databasecredentialsourceconfig"><code>DatabaseCredentialSourceConfig</code></a></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `DidSecretEncryptionConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>enabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>encryptionKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        The encryption key must be a 32-character alphanumeric string, which
        includes both uppercase and lowercase Latin letters (A-Z, a-z) and
        digits (0-9).

        To generate, use:

        ```sh theme={null}
        tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo
        ```
      </td>
    </tr>
  </tbody>
</table>

## `DurationString`

Base type: `string`

## `Email`

Base type: `string`

## `EmailConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>gateway</code></td>
      <td><a href="#emailconfiggateway"><code>EmailConfigGateway</code></a></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>senderAddress</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>senderName</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `EmailConfigGateway`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#emailconfiggatewaydummy"><code>Dummy</code></a></td></tr>
    <tr><td><a href="#emailconfiggatewaypostmark"><code>Postmark</code></a></td></tr>
  </tbody>
</table>

## `EmailConfigGateway::Dummy`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `EmailConfigGateway::Postmark`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>apiKey</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `EmbeddingsChunkerConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#embeddingschunkerconfigsimple"><code>Simple</code></a></td></tr>
  </tbody>
</table>

## `EmbeddingsChunkerConfig::Simple`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>splitParagraphs</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>splitSections</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `EmbeddingsEncoderConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#embeddingsencoderconfigopenai"><code>OpenAi</code></a></td></tr>
    <tr><td><a href="#embeddingsencoderconfigdummy"><code>Dummy</code></a></td></tr>
  </tbody>
</table>

## `EmbeddingsEncoderConfig::OpenAi`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>apiKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>dimensions</code></td>
      <td><code>integer</code></td>
      <td><code>1536</code></td>

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>modelName</code></td>
      <td><code>string</code></td>
      <td><code>"text-embedding-ada-002"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>url</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `EmbeddingsEncoderConfig::Dummy`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `EngineConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>datafusionEmbedded</code></td>
      <td><a href="#engineconfigdatafusion"><code>EngineConfigDatafusion</code></a></td>
      <td><pre><code>\{
      "base": \{
      "datafusion.catalog.default\_catalog": "kamu",
      "datafusion.catalog.default\_schema": "kamu",
      "datafusion.catalog.information\_schema": "true",
      "datafusion.sql\_parser.enable\_ident\_normalization": "false"
      },
      "batchQuery": \{},
      "compaction": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "ingest": \{
      "datafusion.execution.target\_partitions": "1"
      },
      "useLegacyArrowBufferEncoding": false
      }</code></pre></td>
      <td>Embedded Datafusion engine configuration</td>
    </tr>

    <tr>
      <td><code>images</code></td>
      <td><a href="#engineimagesconfig"><code>EngineImagesConfig</code></a></td>
      <td><pre><code>\{
      "datafusion": "ghcr.io/kamu-data/engine-datafusion:0.9.0",
      "flink": "ghcr.io/kamu-data/engine-flink:0.18.2-flink\_1.16.0-scala\_2.12-java8",
      "risingwave": "ghcr.io/kamu-data/engine-risingwave:0.3.0",
      "spark": "ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"
      }</code></pre></td>
      <td>UNSTABLE: Default engine images</td>
    </tr>

    <tr>
      <td><code>maxConcurrency</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>
      <td>Maximum number of engine operations that can be performed concurrently</td>
    </tr>

    <tr>
      <td><code>networkNs</code></td>
      <td><a href="#networknamespacetype"><code>NetworkNamespaceType</code></a></td>
      <td><code>"Private"</code></td>

      <td>
        Type of the networking namespace (relevant when running in container
        environments)
      </td>
    </tr>

    <tr>
      <td><code>runtime</code></td>
      <td><a href="#containerruntimetype"><code>ContainerRuntimeType</code></a></td>
      <td><code>"Podman"</code></td>
      <td>Type of the runtime to use when running the data processing engines</td>
    </tr>

    <tr>
      <td><code>shutdownTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5s"</code></td>
      <td>Timeout for waiting the engine container to stop gracefully</td>
    </tr>

    <tr>
      <td><code>startTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30s"</code></td>
      <td>Timeout for starting an engine container</td>
    </tr>
  </tbody>
</table>

## `EngineConfigDatafusion`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>base</code></td>
      <td><code>object</code></td>
      <td><pre><code>\{
      "datafusion.catalog.default\_catalog": "kamu",
      "datafusion.catalog.default\_schema": "kamu",
      "datafusion.catalog.information\_schema": "true",
      "datafusion.sql\_parser.enable\_ident\_normalization": "false"
      }</code></pre></td>

      <td>
        Base configuration options
        See: `<https://datafusion.apache.org/user-guide/configs.html>`
      </td>
    </tr>

    <tr>
      <td><code>batchQuery</code></td>
      <td><code>object</code></td>
      <td><code>\{}</code></td>
      <td>Batch query-specific overrides to the base config</td>
    </tr>

    <tr>
      <td><code>compaction</code></td>
      <td><code>object</code></td>
      <td><pre><code>\{
      "datafusion.execution.target\_partitions": "1"
      }</code></pre></td>
      <td>Compaction-specific overrides to the base config</td>
    </tr>

    <tr>
      <td><code>ingest</code></td>
      <td><code>object</code></td>
      <td><pre><code>\{
      "datafusion.execution.target\_partitions": "1"
      }</code></pre></td>
      <td>Ingest-specific overrides to the base config</td>
    </tr>

    <tr>
      <td><code>useLegacyArrowBufferEncoding</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td>
        Makes arrow batches use contiguous `Binary` and `Utf8` encodings instead
        of more modern `BinaryView` and `Utf8View`. This is only needed for
        compatibility with some older libraries that don't yet support them.

        See: [kamu-node#277](https://github.com/kamu-data/kamu-node/issues/277)
      </td>
    </tr>
  </tbody>
</table>

## `EngineImagesConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>datafusion</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-datafusion:0.9.0"</code></td>
      <td>UNSTABLE: Datafusion engine image</td>
    </tr>

    <tr>
      <td><code>flink</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-flink:0.18.2-flink\_1.16.0-scala\_2.12-java8"</code></td>
      <td>UNSTABLE: Flink engine image</td>
    </tr>

    <tr>
      <td><code>risingwave</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-risingwave:0.3.0"</code></td>

      <td>
        UNSTABLE: `RisingWave` engine image
      </td>
    </tr>

    <tr>
      <td><code>spark</code></td>
      <td><code>string</code></td>
      <td><code>"ghcr.io/kamu-data/engine-spark:0.23.1-spark\_3.5.0"</code></td>
      <td>UNSTABLE: Spark engine image</td>
    </tr>
  </tbody>
</table>

## `EthRpcEndpoint`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>chainId</code></td>
      <td><code>integer</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>chainName</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>nodeUrl</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `EthereumSourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>commitAfterBlocksScanned</code></td>
      <td><code>integer</code></td>
      <td><code>1000000</code></td>

      <td>
        Forces iteration to stop after the specified number of blocks were
        scanned even if we didn't reach the target record number. This is useful
        to not lose a lot of scanning progress in case of an RPC error.
      </td>
    </tr>

    <tr>
      <td><code>getLogsBlockStride</code></td>
      <td><code>integer</code></td>
      <td><code>100000</code></td>

      <td>
        Default number of blocks to scan within one query to `eth_getLogs` RPC
        endpoint.
      </td>
    </tr>

    <tr>
      <td><code>rpcEndpoints</code></td>
      <td><code>array</code></td>
      <td><code>\[]</code></td>
      <td>Default RPC endpoints to use if source does not specify one explicitly.</td>
    </tr>

    <tr>
      <td><code>useBlockTimestampFallback</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td>
        Many providers don't yet return `blockTimestamp` from `eth_getLogs` RPC
        endpoint and in such cases `block_timestamp` column will be `null`.
        If you enable this fallback the library will perform additional call to
        `eth_getBlock` to populate the timestam, but this may result in
        significant performance penalty when fetching many log records.

        See: [ethereum/execution-apis#295](https://github.com/ethereum/execution-apis/issues/295)
      </td>
    </tr>
  </tbody>
</table>

## `ExtraConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>graphql</code></td>
      <td><a href="#gqlconfig"><code>GqlConfig</code></a></td>
      <td><code>\{}</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `FlightSqlConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>allowAnonymous</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>
      <td>Whether clients can authenticate as 'anonymous' user</td>
    </tr>

    <tr>
      <td><code>anonSessionExpirationTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5m"</code></td>

      <td>
        Time after which `FlightSQL` client session will be forgotten and client
        will have to re-authroize (for anonymous clients)
      </td>
    </tr>

    <tr>
      <td><code>anonSessionInactivityTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5s"</code></td>

      <td>
        Time after which `FlightSQL` session context will be released to free
        the resources (for anonymous clients)
      </td>
    </tr>

    <tr>
      <td><code>authedSessionExpirationTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"30m"</code></td>

      <td>
        Time after which `FlightSQL` client session will be forgotten and client
        will have to re-authroize (for authenticated clients)
      </td>
    </tr>

    <tr>
      <td><code>authedSessionInactivityTimeout</code></td>
      <td><a href="#durationstring"><code>DurationString</code></a></td>
      <td><code>"5s"</code></td>

      <td>
        Time after which `FlightSQL` session context will be released to free
        the resources (for authenticated clients)
      </td>
    </tr>
  </tbody>
</table>

## `FlowSystemConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>awaitingStepSecs</code></td>
      <td><code>integer</code></td>
      <td><code>1</code></td>

      <td>
        Scheduling granularity: activation times are rounded to it, and failed
        activations retried after it. Not a polling period
      </td>
    </tr>

    <tr>
      <td><code>defaultRetryPolicies</code></td>
      <td><code>object</code></td>
      <td><code>\{}</code></td>
      <td>Retry policies applied by default, by flow type</td>
    </tr>

    <tr>
      <td><code>mandatoryThrottlingPeriodSecs</code></td>
      <td><code>integer</code></td>
      <td><code>60</code></td>
      <td>Minimal time between two runs of the same flow</td>
    </tr>
  </tbody>
</table>

## `GqlConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody />
</table>

## `IdentityConfig`

Private keys are used to sign API responses.
Supported algorithms: `ed25519`, `secp256k1`.

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>ed25519PrivateKey</code></td>
      <td><a href="#privatekey"><code>PrivateKey</code></a></td>
      <td><code>null</code></td>

      <td>
        Root private key that corresponds to the `authority` and is used to sign
        responses.

        To generate, use:

        ```sh theme={null}
        od -vN 32 -An -tx1 /dev/urandom | tr -d ' \n' && echo
        ```

        or

        ```sh theme={null}
        openssl rand -hex 32
        ```
      </td>
    </tr>

    <tr>
      <td><code>secp256k1PrivateKey</code></td>
      <td><a href="#secp256k1signer"><code>Secp256k1Signer</code></a></td>
      <td><code>null</code></td>

      <td>
        Secp256k1 private key used to sign EIP-712 typed data.

        To generate, use:

        ```sh theme={null}
        od -vN 32 -An -tx1 /dev/urandom | tr -d ' \n' && echo
        ```

        or

        ```sh theme={null}
        openssl rand -hex 32
        ```

        or

        ```sh theme={null}
        cast wallet new
        ```
      </td>
    </tr>
  </tbody>
</table>

## `IpfsConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>httpGateway</code></td>
      <td><code>string</code></td>
      <td><code>"http\://localhost:8080/"</code></td>

      <td>
        HTTP Gateway URL to use for downloads.
        For safety, it defaults to `http://localhost:8080` - a local IPFS daemon.
        If you don't have IPFS installed, you can set this URL to
        one of the public gateways like `https://ipfs.io`.
        List of public gateways can be found here: `https://ipfs.github.io/public-gateway-checker/`
      </td>
    </tr>

    <tr>
      <td><code>preResolveDnslink</code></td>
      <td><code>boolean</code></td>
      <td><code>true</code></td>

      <td>
        Whether kamu should pre-resolve IPNS `DNSLink` names using DNS or leave
        it to the Gateway.
      </td>
    </tr>
  </tbody>
</table>

## `MqttSourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>brokerIdleTimeoutMs</code></td>
      <td><code>integer</code></td>
      <td><code>1000</code></td>

      <td>
        Time in milliseconds to wait for MQTT broker to send us some data after
        which we will consider that we have "caught up" and end the polling
        loop.
      </td>
    </tr>
  </tbody>
</table>

## `NetworkNamespaceType`

Corresponds to podman's `containers.conf::netns`
We podman is used inside containers (e.g. podman-in-docker or podman-in-k8s)
it usually runs uses host network namespace.

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>Private</code></td></tr>
    <tr><td><code>Host</code></td></tr>
  </tbody>
</table>

## `Password`

Base type: `string`

## `PasswordPolicyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>minNewPasswordLength</code></td>
      <td><code>integer</code></td>
      <td><code>8</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `PrivateKey`

Base type: `string`

## `ProtocolConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>flightSql</code></td>
      <td><a href="#flightsqlconfig"><code>FlightSqlConfig</code></a></td>
      <td><pre><code>\{
      "allowAnonymous": true,
      "anonSessionExpirationTimeout": "5m",
      "anonSessionInactivityTimeout": "5s",
      "authedSessionExpirationTimeout": "30m",
      "authedSessionInactivityTimeout": "5s"
      }</code></pre></td>

      <td>
        `FlightSQL` configuration
      </td>
    </tr>

    <tr>
      <td><code>ipfs</code></td>
      <td><a href="#ipfsconfig"><code>IpfsConfig</code></a></td>
      <td><pre><code>\{
      "httpGateway": "http\://localhost:8080/",
      "preResolveDnslink": true
      }</code></pre></td>
      <td>IPFS configuration</td>
    </tr>
  </tbody>
</table>

## `QuotaAccountConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>defaultStorageLimitInBytes</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `QuotaConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>account</code></td>
      <td><a href="#quotaaccountconfig"><code>QuotaAccountConfig</code></a></td>
      <td><code>\{}</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `RepoCachingConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>metadataLocalFsCachePath</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>registryCacheEnabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `RepoConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>caching</code></td>
      <td><a href="#repocachingconfig"><code>RepoCachingConfig</code></a></td>
      <td><pre><code>\{
      "registryCacheEnabled": false
      }</code></pre></td>

      <td />
    </tr>

    <tr>
      <td><code>dataBlocksPageSize</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>repoUrl</code></td>
      <td><a href="#urlorpath"><code>UrlOrPath</code></a></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `RetryPolicyConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>backoffType</code></td>
      <td><a href="#retrypolicyconfigbackofftype"><code>RetryPolicyConfigBackoffType</code></a></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>maxAttempts</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>minDelaySecs</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `RetryPolicyConfigBackoffType`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><code>Fixed</code></td></tr>
    <tr><td><code>Linear</code></td></tr>
    <tr><td><code>Exponential</code></td></tr>
    <tr><td><code>ExponentialWithJitter</code></td></tr>
  </tbody>
</table>

## `RuntimeConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>maxBlockingThreads</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>threadStackSize</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>workerThreads</code></td>
      <td><code>integer</code></td>
      <td><code>null</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `SearchConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>embeddingsChunker</code></td>
      <td><a href="#embeddingschunkerconfig"><code>EmbeddingsChunkerConfig</code></a></td>
      <td><pre><code>\{
      "kind": "Simple",
      "splitParagraphs": false,
      "splitSections": false
      }</code></pre></td>
      <td>Embeddings chunker configuration</td>
    </tr>

    <tr>
      <td><code>embeddingsEncoder</code></td>
      <td><a href="#embeddingsencoderconfig"><code>EmbeddingsEncoderConfig</code></a></td>
      <td><pre><code>\{
      "dimensions": 1536,
      "kind": "OpenAi",
      "modelName": "text-embedding-ada-002"
      }</code></pre></td>
      <td>Embeddings encoder configuration</td>
    </tr>

    <tr>
      <td><code>indexer</code></td>
      <td><a href="#searchindexerconfig"><code>SearchIndexerConfig</code></a></td>
      <td><pre><code>\{
      "clearOnStart": false,
      "incrementalIndexing": false
      }</code></pre></td>
      <td>Indexer configuration</td>
    </tr>

    <tr>
      <td><code>repo</code></td>
      <td><a href="#searchrepositoryconfig"><code>SearchRepositoryConfig</code></a></td>
      <td><pre><code>\{
      "kind": "Dummy"
      }</code></pre></td>
      <td>Search repository configuration</td>
    </tr>

    <tr>
      <td><code>semanticSearchThresholdScore</code></td>
      <td><code>number</code></td>
      <td><code>0.0</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `SearchIndexerConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>clearOnStart</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>incrementalIndexing</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>
      <td>Whether incremental indexing is enabled</td>
    </tr>
  </tbody>
</table>

## `SearchRepositoryConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#searchrepositoryconfigdummy"><code>Dummy</code></a></td></tr>
    <tr><td><a href="#searchrepositoryconfigelasticsearch"><code>Elasticsearch</code></a></td></tr>
  </tbody>
</table>

## `SearchRepositoryConfig::Dummy`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `SearchRepositoryConfig::Elasticsearch`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>caCertPemPath</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>embeddingDimensions</code></td>
      <td><code>integer</code></td>
      <td><code>1536</code></td>

      <td />
    </tr>

    <tr>
      <td><code>enableCompression</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>indexPrefix</code></td>
      <td><code>string</code></td>
      <td><code>"kamu-node"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>password</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td />
    </tr>

    <tr>
      <td><code>timeoutSecs</code></td>
      <td><code>integer</code></td>
      <td><code>30</code></td>

      <td />
    </tr>

    <tr>
      <td><code>url</code></td>
      <td><code>string</code></td>
      <td><code>"http\://localhost:9200"</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `Secp256k1Signer`

Base type: `string`

## `SecretsEncryptionConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>enabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>encryptionKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Represents the encryption key for secrets. This field is required if
        `enabled` is `true` or `None`.

        The encryption key must be a 32-character alphanumeric string, which
        includes both uppercase and lowercase Latin letters (A-Z, a-z) and
        digits (0-9).

        To generate use:

        ```sh theme={null}
        tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo
        ```
      </td>
    </tr>
  </tbody>
</table>

## `SourceConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>ethereum</code></td>
      <td><a href="#ethereumsourceconfig"><code>EthereumSourceConfig</code></a></td>
      <td><pre><code>\{
      "commitAfterBlocksScanned": 1000000,
      "getLogsBlockStride": 100000,
      "rpcEndpoints": \[],
      "useBlockTimestampFallback": false
      }</code></pre></td>
      <td>Ethereum-specific configuration</td>
    </tr>

    <tr>
      <td><code>mqtt</code></td>
      <td><a href="#mqttsourceconfig"><code>MqttSourceConfig</code></a></td>
      <td><pre><code>\{
      "brokerIdleTimeoutMs": 1000
      }</code></pre></td>
      <td>MQTT-specific configuration</td>
    </tr>

    <tr>
      <td><code>targetRecordsPerSlice</code></td>
      <td><code>integer</code></td>
      <td><code>10000</code></td>

      <td>
        Target number of records after which we will stop consuming from the
        resumable source and commit data, leaving the rest for the next
        iteration. This ensures that one data slice doesn't become too big.
      </td>
    </tr>
  </tbody>
</table>

## `UploadRepoConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>maxFileSizeMb</code></td>
      <td><code>integer</code></td>
      <td><code>50</code></td>

      <td />
    </tr>

    <tr>
      <td><code>storage</code></td>
      <td><a href="#uploadrepostorageconfig"><code>UploadRepoStorageConfig</code></a></td>
      <td><pre><code>\{
      "kind": "Local"
      }</code></pre></td>

      <td />
    </tr>
  </tbody>
</table>

## `UploadRepoStorageConfig`

<table>
  <thead><tr><th>Variants</th></tr></thead>

  <tbody>
    <tr><td><a href="#uploadrepostorageconfigs3"><code>S3</code></a></td></tr>
    <tr><td><a href="#uploadrepostorageconfiglocal"><code>Local</code></a></td></tr>
  </tbody>
</table>

## `UploadRepoStorageConfig::S3`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>bucketS3Url</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>

    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `UploadRepoStorageConfig::Local`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>kind</code></td>
      <td><code>string</code></td>

      <td />

      <td />
    </tr>
  </tbody>
</table>

## `UrlConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>baseUrlFlightsql</code></td>
      <td><a href="#urlorpath"><code>UrlOrPath</code></a></td>
      <td><code>"grpc://localhost:50050"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>baseUrlPlatform</code></td>
      <td><a href="#urlorpath"><code>UrlOrPath</code></a></td>
      <td><code>"http\://localhost:4200/"</code></td>

      <td />
    </tr>

    <tr>
      <td><code>baseUrlRest</code></td>
      <td><a href="#urlorpath"><code>UrlOrPath</code></a></td>
      <td><code>"http\://localhost:8080/"</code></td>

      <td />
    </tr>
  </tbody>
</table>

## `UrlOrPath`

Base type: `string`

## `WebhooksConfig`

<table>
  <thead><tr><th>Field</th><th>Type</th><th>Default</th><th>Description</th></tr></thead>

  <tbody>
    <tr>
      <td><code>deliveryTimeoutSecs</code></td>
      <td><code>integer</code></td>
      <td><code>10</code></td>

      <td />
    </tr>

    <tr>
      <td><code>maxConsecutiveFailures</code></td>
      <td><code>integer</code></td>
      <td><code>5</code></td>

      <td />
    </tr>

    <tr>
      <td><code>secretEncryptionEnabled</code></td>
      <td><code>boolean</code></td>
      <td><code>false</code></td>

      <td />
    </tr>

    <tr>
      <td><code>secretEncryptionKey</code></td>
      <td><code>string</code></td>
      <td><code>null</code></td>

      <td>
        Represents the encryption key for the webhooks secret. This field is
        required if `secret_encryption_enabled` is `true` or `None`.

        The encryption key must be a 32-character alphanumeric string, which
        includes both uppercase and lowercase Latin letters (A-Z, a-z) and
        digits (0-9).

        # Example

        ```ignore theme={null}
        let config = WebhooksConfig \{
            secret_encryption_enabled: true,
            secret_encryption_key: Some(String::from("aBcDeFgHiJkLmNoPqRsTuVwXyZ012345")),
            ..
        };
        ```
      </td>
    </tr>
  </tbody>
</table>


# Artifacts
Source: https://docs.kamu.dev/node/deploy/artifacts



All components of Kamu Node are provided as [OCI container images](https://opencontainers.org/) and [Helm charts](https://helm.sh/).

We are limiting the **official support** to these types of artifacts as the most widespread and interoperable distribution models for Kubernetes applications. Other types of distributions may be supported by the community.

## Helm Charts

Latest Helm charts can be found in the [ArtifactHub](https://artifacthub.io/packages/search?org=kamu):

* [**`kamu-api-server`**](https://artifacthub.io/packages/helm/kamu/kamu-api-server)
* [**`kamu-oracle-provider`**](https://artifacthub.io/packages/helm/kamu/kamu-oracle-provider)
* [**`kamu-web-ui`**](https://artifacthub.io/packages/helm/kamu/kamu-web-ui)

You can use ArtifactHub to explore the configuration parameters schema for each chart as well as the default values.

## Helm Repository

To use our charts with Helm you will need to add our repository:

```sh theme={null}
helm repo add kamu https://kamu-data.github.io/helm-charts
helm repo update
```

## Source Code

Latest source code of all charts can be found in [`kamu-data/helm-charts`](https://github.com/kamu-data/helm-charts) GitHub repository.


# Auth
Source: https://docs.kamu.dev/node/deploy/auth





# Capacity Planning
Source: https://docs.kamu.dev/node/deploy/capacity





# Customization
Source: https://docs.kamu.dev/node/deploy/customization





# Kubernetes Setup
Source: https://docs.kamu.dev/node/deploy/k8s



* Persistent volumes
* Ingress controller


# Deployment Manual
Source: https://docs.kamu.dev/node/deploy/manual



<Info>
  Want to try out Kamu Node in a **local environment**?

  Then check out the [Quick Start](/node/quick-start) guide.
</Info>

## Recommended Steps

Deployment and customization of a production environment is an iterative process, but you may want to roughly follow these steps:

### 1. Prepare your Kubernetes cluster

All Kubernetes clusters are unique and have different sets of capabilities. See the [Kubernetes](/node/deploy/k8s) page to understand the basic requirements for Kubernetes cluster before you start deploying.

### 2. Prepare artifacts & configuration

The [Artifacts](/node/deploy/artifacts) page explains Helm charts as our distribution model, which is compatible with many deployment workflows and CI/CD tools. On this stage you should register necessary Helm repositories and prepare the initial configurations for the releases.

If you are starting from scratch and don't have a strong tooling preference - we recommend you to try the "infrastructure-as-code" approach that is described in our [Quick Start](/node/quick-start) guide.

### 3. Deploy a minimal environment

We recommend you to deploy to get things running early. It's always easier to progressively customize a working deployment than to plan out everything ahead.

On this stage we recommend you to deploy the Node using in-memory transient storage and dummy auth which we will subsequently replace with real implementations.

If you are using our deployment template, once again follow the [Quick Start](/node/quick-start) guide.

### 4. Customize authentication & authorization

See [Authentication & Authorization](/node/deploy/auth) page for available options of providing own or integrating with 3rd party auth providers.

### 5. Customize dataset and operational storage

Most companies will have their own preferences for storage, databases, security, and disaster recovery workflows. For this reason in a production deployment we expect you to provide storage to Kamu Node. All storage solutions we provide with our deployment examples are meant for demonstrative purposes only and must not be used for data of any value.

See [Storage](/node/deploy/storage) page for the list of supported storage systems for data lake and operational data.

### 6. Create a capacity plan and configure resources

Kamu Node is designed to provide optimal performance of all workloads according to the available cluster capacity, applying back pressure when needed. If you are deploying the node in a Kubernetes cluster shared with other applications you may want to restrict the resource usage of the node to prevent applications from "starving" one another. Limiting capacity can also be done for cost management.

See [Capacity](/node/deploy/capacity) page to understand available configuration options and the back pressure strategies the node uses when resources are insufficient.

### 7. Security checklist

Security of the deployment is your responsibility, but our [Security](/node/deploy/security) page provides a list of things to watch out for.

### 8. Further customization

Once you have a fully working secure deployment you can further customize it to your liking. See [Customization](/node/deploy/customization) page for the list of available options.

## What's Next?

Now that you have a working deployment, see [Operator Manual](/node/operator) to learn how to keep it up-to-date and troubleshoot issues.


# Security
Source: https://docs.kamu.dev/node/deploy/security





# Storage
Source: https://docs.kamu.dev/node/deploy/storage



## Supported Storage Systems

[Kamu Node](/node) is designed to be storage-agnostic and supports wide variety of storage systems.

### Managed cloud

* [Amazon S3](https://aws.amazon.com/s3/)
* [Google Cloud Storage (GCS)](https://cloud.google.com/storage/)
* [Azure Blob Storage (ABS)](https://azure.microsoft.com/en-gb/services/storage/blobs/)

### On-premise

* [Minio](https://min.io/) or **any S3-compatible** storage
* [Dell ObjectScale](https://www.dell.com/en-us/dt/storage/objectscale.htm)
* [Hadoop HDFS](https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html) <Badge>WIP</Badge>

### Decentralized

* [IPFS](https://ipfs.tech/)
* [Arweave](https://www.arweave.org/) <Badge>WIP</Badge>

If there is a storage not this list that you'd like to be supported - [let us know](https://kamu.dev/contact/).

## Safeguarding data

<Warning>
  This section is under construction, please check back soon!
</Warning>


# Monitoring
Source: https://docs.kamu.dev/node/operator/monitoring





# Performance Tips
Source: https://docs.kamu.dev/node/operator/performance





# Upgrades
Source: https://docs.kamu.dev/node/operator/upgrades





# Quick Start
Source: https://docs.kamu.dev/node/quick-start



<Tip>
  If you are looking to try out some [APIs](/api) and [integrations](/integrations) without deploying an full node - you can start API server directly from the CLI tool as:

  ```sh theme={null}
  kamu system api-server
  ```
</Tip>

**Ready to give Kamu Node a try?** This guide will help you deploy all components of Kamu Node and [Kamu Web Platform](/platform) on your local machine in a few simple steps.

Use it to familiarize yourself with the process of deploying all components in a minimalistic configuration. Once you are ready to deploy a production-ready node instance please refer to the [Deployment Manual](/node/deploy).

## Prerequisites

Kamu Node is a **Kubernetes-native application** and this guide assumes basic familiarity with Kubernetes. If you worked with `docker-compose` before - you will see a few new tools, but shouldn't encounter any problems.

Before we begin, you will need the following tools installed:

1. [**`git`**](https://git-scm.com/) - to clone the [`kamu-deploy-example`](https://github.com/kamu-data/kamu-deploy-example) repo
2. [**`minikube`**](https://minikube.sigs.k8s.io/docs/) - to create a local Kubernetes cluster
3. [**`kubectl`**](https://kubernetes.io/docs/reference/kubectl/) - client for interacting with Kubernetes cluster
4. [**`helm`**](https://helm.sh/) - software management tool for Kubernetes
5. [**`helmfile`**](https://helmfile.readthedocs.io) - "infrastructure-as-code" wrapper for `helm`
6. [**`aws-cli`**](https://aws.amazon.com/cli/) (optional) - needed to download example datasets

## Start your cluster

First we will start by creating a local Kubernetes cluster using `minikube`:

```sh theme={null}
minikube start
```

<Info>
  You can also specify how much resources to allocate to the cluster. For example, we can let it use all available resources as:

  ```sh theme={null}
  minikube start --cpus no-limit --memory no-limit
  ```
</Info>

Once command finishes you should have a fully functional Kubernetes cluster running on your local machine!

Ensure that `kubectl` client can communicate with your new cluster:

```sh theme={null}
kubectl get pods --all-namespaces
```

```
NAMESPACE     NAME                               READY   STATUS    RESTARTS  AGE
kube-system   kube-apiserver-minikube            1/1     Running   0         2m8s
kube-system   etcd-minikube                      1/1     Running   0         2m7s
...
```

## Clone the example repo

While we could deploy all components by running `helm install` commands, we will use "infrastructure-as-code" approach where the entire state of our deployment is described by files in a `git` repository.

Go ahead and clone the [`kamu-deploy-example`](https://github.com/kamu-data/kamu-deploy-example) repo:

```sh theme={null}
git clone https://github.com/kamu-data/kamu-deploy-example.git
cd kamu-deploy-example
```

## Select components

By default all components are enabled, but if, for example, you don't need JupyterHub open:

```
environments/minikube/environment.yaml
```

and modify it to:

```yaml theme={null}
jupyterHub:
  enabled: false
```

## Deploy Kamu Node

### Using convenience script

To get all components installed and running you can use a convenience script:

```sh theme={null}
./sync.sh
```

<Warning>
  Multiple container images will be downloaded in the process, so if script times-out it's safe to re-run again.
</Warning>

Once it completes it's just a matter of exposing a few ports:

```sh theme={null}
./port-forward.sh
```

```
=============== Kamu Web UI ===============
Web UI (fwd):     http://localhost:4200
Web UI:           http://192.168.X.X:30211

=============== JupyterHub ================
Web UI (fwd):     http://localhost:4300
Web UI:           http://192.168.X.X:30230

================ Kamu Node ================
GraphQL / REST:   http://192.168.X.X:30201
FlightSQL:        http://192.168.X.X:31402

================== Minio ==================
Minio API:        http://192.168.X.X:30401
Minio Console:    http://192.168.X.X:30402
Username:         admin
Password:         password123

Forwarding from 127.0.0.1:4200 -> 8080
Forwarding from 127.0.0.1:4300 -> 8000
Press Ctrl+C to exit
```

You can now open the provided `Kamu Web UI` URL in your browser and see the [Kamu Web Platform](/platform) interface.

### Using helmfile

Instead of the convenience script you can use `helmfile` directly to **synchronize** the current state the cluster with the desired state of our deployment defined in configuration files:

```sh theme={null}
export MINIKUBE_HOST=$(minikube ip)
helmfile -e minikube sync
```

Either way you will now have multiple `kamu` component pods running:

```sh theme={null}
kubectl get pods
```

```
NAME                               READY   STATUS      RESTARTS   AGE
kamu-api-server-79974ff49c-5bb7s   1/1     Running     0          1h
kamu-web-ui-85fb7b7957-vb5kf       1/1     Running     0          1h
minio-58c59dd784-4jkfk             1/1     Running     0          1h
minio-provisioning-zfz5n           0/1     Completed   0          1h
...
```

## Example Content

If you open the `Kamu Web UI` URL in your browser you will see a functional but very empty platform.

To load some example data to play with use:

```sh theme={null}
./bootstrap-examples.sh
```

Refresh the browser tab and you should see a lot of new pipelines to explore.

### Superset example

We are still working on automated Superset examples setup, so to see some data follow these simple steps:

1. Run `port-forward.sh` script
2. Open Superset Web UI and login
3. Go to "Settings" > "Database connections"
4. Select "+ Database" using type "Other"
5. Display name: `kamu`
6. For Sqlalchemy URL use:
   ```
   datafusion+flightsql://kamu:kamu@192.168.49.2:30241?insecure=True
   ```
   replacing the IP/port with the output of `port-forward.sh` script for `FlightSQL` protocol
7. Now go to `Datasets` tab and create one for `kamu/co.alphavantage.tickers.daily.spy`
8. Create a `Line Chart` by:
   1. dropping `event_time` onto `X-Axis`
   2. dropping `close` onto `Metrics`
   3. hitting `Create chart` at the bottom

## Making changes

After the initial deployment, if you modify any configuration you can easily **apply** your changes to the deployment using a convenience script:

```sh theme={null}
./apply.sh
```

Or manually:

```sh theme={null}
helmfile -e minikube apply
```

## Under the hood

Main entry point of this repository is `helmfile.yaml` which:

* Configures some basic `helm` parameters (e.g. deployment timeouts)
* Connects [Kamu's helm chart repository](https://artifacthub.io/packages/search?org=kamu)
* Defines the `minikube` environment
* And defines a multi-layer structure for supplying configuration values to individual helm chart releases

The skeleton `helmfile.yaml` provided in this example is simple yet very capable. It can be a good foundation for operating multiple different Kamu Node environments in production.

## What's Next?

See [Architecture](/node/architecture) section for the description of all components we have installed and how they interoperate.

Once you get familiar and comfortable with this local setup - proceed to the [Deployment Manual](/node/deploy) for instructions on setting up a production-grade environment.

Oh, and don't forget to clean up your `minikube` environment:

```sh theme={null}
minikube delete
```


# Troubleshooting
Source: https://docs.kamu.dev/node/troubleshoot





# About
Source: https://docs.kamu.dev/odf/about



<div>
  <img alt="Open Data Fabric logo" />

  <p><strong>Open protocol for decentralized exchange and transformation of data</strong></p>

  [Website](https://opendatafabric.org) |
  [Repository](https://github.com/kamu-data/open-data-fabric) |
  [Reference Implementation](/cli) |
  [Original Whitepaper](https://arxiv.org/abs/2111.06364) |
  [Chat](https://discord.gg/nU6TXRQNXC)

  <div>
    <a href="/odf/spec/overview">
      <img />
    </a>

    <a href="/odf/schemas">
      <img />
    </a>

    <a href="https://discord.gg/nU6TXRQNXC">
      <img />
    </a>
  </div>
</div>

## Introduction

[Open Data Fabric](https://opendatafabric.org) (ODF) is an open protocol specification for decentralized exchange and transformation of semi-structured data, that aims to holistically address many shortcomings of the modern data management systems and workflows.

<div>
  <YouTube />

  <p><i>Introductory video</i></p>
</div>

The goal of this specification is to develop a method of data exchange that would:

* Enable worldwide collaboration around data cleaning, enrichment, and derivation
* Create an environment of verifiable trust between participants without the need for a central authority
* Enable high degree of data reuse, making quality data more readily available
* Improve liquidity of data by speeding up the data propagation times from publishers to consumers
* Create a feedback loop between data consumers and publishers, allowing them to collaborate on better data availability, recency, and design

ODF protocol combines four decades of evolution in enterprise analytical data architectures with innovations in Web3 cryptography and trustless networks to create a decentralized data supply chain that can provide timely, high-quality, and verifiable data for data science, AI, smart contracts, and applications.

## Quick Summary

At a bird's eye view, ODF specifies the following aspects of data management, with every subsequent layer building tightly upon previous:

### Data Format

* **Parquet** columnar format is used for efficient storage of raw data
* **Event streams** (not tables) as logical model of a dataset, with focus on supporting high-frequency real-time data
* All data forms an **immutable ledger** - streams evolve only by appending new events
* Retractions & corrections are explicit - full history of evolution of data is preserved by default
* **Storage agnostic** layout based on blob hashes adds support for decentralized content-addressable file systems

### Metadata Format

* Metadata format is extensible and records all aspects of dataset lifecycle: from creation and identity, to adding raw data, and to licensing, stewardship, and governance
* Metadata is organized into cryptographic ledger (similar to git history or a blockchain) to keep it tamper-proof
* Block hash of the metadata ledger serves as a stable reference to how dataset looked like at a specific point in time (Merkel root)
* Maintenance operations on metadata include compactions, history pruning, indexing etc.

### Identity, Ownership, Accountability

* W3C [DIDs](https://www.w3.org/TR/did-1.0/) are used to uniquely identify datasets on the network - they are an irrevocable part of the metadata chain
* DIDs likewise identify actors: data owners, processors, validators, replicators...
* DID key signing chains are used to **prove ownership** over datasets in a decentralized network
* Metadata blocks support actor signatures to assign **accountability**

<img alt="ODF Dataset Format" />

### Data Transfer Protocols

* ODF "nodes" that range from CLI tools to analytical clusters can exchange data using a set of protocols
* "Simple transfer protocol" is provided for maximal interoperability
* "Smart transfer protocol" is provided for secure yet highly efficient direct-to-storage data replication

<img alt="ODF Transfer Protocols" />

### Processing & query model

* **Derivative data** *(i.e. data created by transforming other data)* is a first-class citizen in ODF
* **Stateful temporal/stream processing** is used as the core primitive in ODF to create **derivative data pipelines** (Kappa architecture) that are highly autonomous and near-zero maintenance
* All derivative data is **traceable, verifiable, and auditable**
  * No matter how many hands and transformation steps data went through - we could always tell where every bit came from, who processed it, and how
* **Verifiable computing techniques** are extended to work with temporal processing via:
  * Determinism + Reproducibility
  * Zero-Knowledge (TEEs, FHE, MPC, Zk proofs)
* **Verifiable querying** - allows to accompany every query result with a reliable proof that it was executed correctly, even when processing is done by an unreliable 3rd party
* **Abstract engine interface** spec allows easily adding new engine implementations as plug-ins.

<img alt="Dataset Graph" />

### Dataset Discovery

* Global, permissionless, censorship-proof dataset registry is proposed to advertise datasets (blockchain-based)

### Privacy & Access Control

* Node-level ReBAC permission management
* Dataset **encryption** - allows exchanging private data via open non-secure storage systems
* Key exchange system (smart contract) is proposed to facilitate negotiating access to encrypted private data
* Advanced encryption schemes can be used to reveal only part of the full dataset history, enabling e.g. latency-based monetization schemes.

### Federated Querying

* Thanks to DIDs and content-addressability, the queries and transformations in ODF can express *"what needs to be done"* without depending on *"where data is located"*
* ODF nodes can form ad hoc peer-to-peer networks to:
  * Execute queries that touch data located in different organizations
  * Continuously run cross-organizational streaming data processing pipelines
* The verifiability scheme is designed to swiftly expose malicious actors.

## Introductory materials

* [Original Whitepaper (July 2020)](https://arxiv.org/abs/2111.06364)
* [Kamu Blog: Introducing Open Data Fabric](https://www.kamu.dev/blog/introducing-odf/)
* [Talk: Open Data Fabric for Research Data Management](https://www.youtube.com/watch?v=Ivh-YDDmRf8)
* [PyData Global 2021 Talk: Time: The most misunderstood dimension in data modelling](https://www.youtube.com/watch?v=XxKnTusccUM)
* [Data+AI Summit 2020 Talk: Building a Distributed Collaborative Data Pipeline](https://databricks.com/session_eu20/building-a-distributed-collaborative-data-pipeline-with-apache-spark)

More tutorials and articles can be found in [kamu-cli documentation](/start/learning-materials).

## Current State

The specification is currently in actively evolving and welcomes feedback.

See also our [Roadmap](https://github.com/kamu-data/open-data-fabric/projects/1) for future direction and [RFC archive](/odf/rfcs) for the record of changes.

## Known Implementations

`Coordinator` implementations:

* [kamu-cli](https://github.com/kamu-data/kamu-cli/) - data management tool that serves as the reference implementation.

`Engine` implementations:

* [kamu-engine-spark](https://github.com/kamu-data/kamu-engine-spark) - engine based on Apache Spark.
* [kamu-engine-flink](https://github.com/kamu-data/kamu-engine-flink) - engine based on Apache Flink.
* [kamu-engine-datafusion](https://github.com/kamu-data/kamu-engine-datafusion) - engine based on Apache DataFusion.
* [kamu-engine-risingwave](https://github.com/kamu-data/kamu-engine-risingwave) - engine based on RisingWave.

## History

The specification was originally developed by [Kamu](https://kamu.dev) as part of the [kamu-cli](https://github.com/kamu-data/kamu-cli/) data management tool. While developing it, we quickly realized that the very essence of what we're trying to build - a collaborative open data processing pipeline based on verifiable trust - requires full transparency and openness on our part. We strongly believe in the potential of our ideas to bring data management to the next level, to provide better quality data faster to the people who need it to innovate, fight diseases, build better businesses, and make informed political decisions. Therefore, we saw it as our duty to share these ideas with the community and make the system as inclusive as possible for the existing technologies and future innovations, and work together to build momentum needed to achieve such radical change.

## Contributing

If you like what we're doing - support us by [starring the repo](https://github.com/kamu-data/kamu-cli), this helps us a lot!

For the list of our community resources and guides on how to contribute [start here](/community).


# RFC-001: Record Offsets
Source: https://docs.kamu.dev/odf/rfcs/001-record-offsets



**Start Date**: 2021-06-01

**Specs**:

* [Data Model](/odf/spec/datasets/data-model)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/10?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/10)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/13?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/13)

## Summary

This RFC introduces new system column `offset` that represents the sequential number of the row from the beginning of the dataset.

This is a **backwards incompatible change**.

## Motivation

We need a way to **uniquely identify** a record in a dataset.

This should simplify referring to data slices (we currently use `system_time` intervals in metadata).

And in future should help with features like fine-grain provenance, corrections, and retractions.

## Guide-level explanation

Offset is a monotonically increasing sequential numeric identifier that is assigned to every record and represents its position relative to the beginning of the dataset (first record's offset is `0`).

## Reference-level explanation

Common schema will be extended with `offset` column, with data type: `int64` (Parquet/Arrow), `BIGINT` (DDL), non-null. Offset of the first record is `0`.

`DataSclice` schema will be updated to use offsets instead of system time intervals.

`InputSlice` schema will be introduced that contains data offsets and metadata blocks intervals. Previously we relied on system time interval to define both data and metadata slices, but this proved to be not flexible and complex.

`ExecuteQueryRequest` will be extended with `offset` field telling the engine the starting offset to use for new data.

`DatasetVocab` will be extended with `offsetColumn` field.

All intervals (for offsets and metadata) will be closed/inclusive `[start; end]`. Empty intervals should be expressed by the absence of the interval field.

`date-time-interval` format will be removed.

## Drawbacks

* Backwards incompatible change
* Increases the number of system columns the users are exposed to
* Storage overhead
* Sequential identifiers are bad for data-parallel systems
  * But so far our streams are not going to be processed fully in parallel

## Rationale and alternatives

* We could rely on individual data engines to materialize offset column on the fly, but this
  * Would need to be handled in every engine, query console etc.
  * May interfere with storage formats if rows are reordered during writing or reading

## Prior art

* Many SQL databases have `ROWID` pseudo-column and `ROW_NUMBER()` function
* Kafka relies heavily on stream offsets

## Unresolved questions

## Future possibilities

* Offsets should make it easier to implement features like
  * Fine-grain provenance
  * Corrections & retractions
* This feature should coexist fine with data purging and decay (if those will be needed for high volume/rate use cases like IoT)


# RFC-002: Logical Data Digests
Source: https://docs.kamu.dev/odf/rfcs/002-logical-data-hashes



**Start Date**: 2021-11-15

**Specs**:

* [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/1?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/1)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/16?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/16)

## Summary

This RFC standardizes how equality, equivalence, and integrity checks are performed on data part files.

## Motivation

Use cases we want to cover are:

* **Integrity Checks** - ensuring that data is not tampered after downloading it from untrusted source
* **Equivalence Checks** - after replaying the transformation ensure result is the same as one presented by the dataset owner (transformation result is not spoofed)
* **Content Addressability** - being able to store data in a content-addressable system (e.g. IPFS)

A standard practice for these is through use of **cryptographic digests** (hash sums). Git, blockchain, Docker, IPFS are few of many examples.

The one big problem with this approach is **reproducibility**. ODF's core guarantee that repeating the same transformation always produces identical result. Reproducibility requires perfect determinism. While ODF goes a long way in improving likelihood of determinism in the engines, some non-determinism may also come from the coordinator.

The task of writing output data is delegated to the coordinator. We are currently using Parquet format for on-disk representation. Parquet format is **non-reproducible** (non-deterministic) - writing logically identical data with different implementations or different versions of the libraries may produce files that are different on the binary level.

Things that contribute to this are: heuristics around encodings (e.g. RLE vs bitpack), heuristics around batch sizes, different settings for statistics, etc.

Therefore, if coordinators would use a hash sum of a Parquet file for integrity/equivalence checks they would be likely to break with any changes to Parquet libraries and produce different hashes between different coordinator implementations.

## Guide-level explanation

### Logical Hashes

To isolate ourselves from non-determinism of Parquet storage format we will use a combination of two types of data hashes:

* **Physical hash** - is a (non-reproducible) digest of an entire binary file as originally produced by the owner of the dataset
* **Logical hash** - is a stable (reproducible) digest computed on data records, resistant to encoding variations

**Physical hash** will be used only in the context of uploading and downloading data files to/from content-addressable systems.

**Logical hash** will be used for integrity and equivalence checks. It should maximize the chances that logically identical data results in an identical hash sum.

Luckily, we already use a data format that specifies binary layouts of data in the memory - **Apache Arrow**. It has a lot of similar sources of non-determinism as Parquet (e.g. data cell marked as `null` by validity mask might have uninitialized memory in it), but can be used as a foundation for **efficiently** implementing a stable hash algorithm.

Due to absence of existing solutions it was decided to implement our own hashing scheme on top of Apache Arrow format as described in detail in [arrow-digest](https://github.com/sergiimk/arrow-digest) repository.

### Future-Proofing

This is likely not the last time we change the hashing algorithm, but we should aim for such changes not to be breaking in the future.

As algorithms may have vulnerabilities, or as hash lengths become less secure with advances in computing power, we need an ability to evolve the algorithms we use over time. To future-proof metadata we will be storing the hashing algorithm identifier alongside the hash value, and coordinator implementations will be able to differentiate between versions and decide how to deal with them.

This problem is already addressed by the [multiformats](https://github.com/multiformats/multiformats) project, specifically:

* [multihash](https://github.com/multiformats/multihash) - describes how to encode algorithm ID alongside the hash value.
* [multibase](https://github.com/multiformats/multibase) - describes how to represent binary value in a string (e.g. for YAML format) without ambiguity of which encoding was used.

In the future, we will apply this scheme to all hashes, but for now only limiting them to data-related.

## Reference-level explanation

New schema format `multihash` is introduced:

* In binary form it is using [multihash](https://github.com/multiformats/multihash) specification
* In string form it is encoded using [multibase](https://github.com/multiformats/multibase) specification, using `base58` codec

The official `multicodec` table will be extended with the following codes in the "private use area":

| Codec | Code |
| - | - |
| `arrow0-sha3-256` | `0x300016` |

Because `multihash` format has a static prefix the short (8-character) hash representation should use tail bytes of multihash instead of the head.

`OutputSlice` schema is updated:

* `dataLogicalHash` updated to use `multihash` format
* new field `dataPhysicalHash` is added with `multihash` format

`dataLogicalHash` is updated to use [arrow-digest](https://github.com/sergiimk/arrow-digest) with `sha3-256` base algorithm.

`dataPhysicalHash` will contain a `sha3-256` hash of the data part file.

`ExecuteQueryResponse` schema is updated not to carry hash information - this was a temporary solution before coordinator hashing support is available.

## Drawbacks

* Logical hash equality doesn't guarantee that data file was not tampered with (see [unresolved questions](#unresolved-questions))
* Maintaining two hashes add a lot of complexity - it would be much nicer to have just physical hash and a reproducible file format (see [future possibilities](#future-possibilities))

## Rationale and alternatives

* Maintain separate logical and physical hashes?
* Hash Arrow in-memory representation for stability and performance?
* Achieve deterministic layout of Parquet and only have physical hash that is stable?

## Prior art

Hashing of structured data is mostly used in the context of buckets for hash-based join operations etc. We haven't seen Arrow/Parquet hashing used in cryptographic context.

We did not consider any other formats as we have strong reasons to continue using Arrow and Parquet and encoding in a third format will be wasteful.

* [Our Apache Arrow User mailing list thread on format reproducibility](https://lists.apache.org/thread/fndxck757vfkxd8wx5smspmov9nrzvft)
* Relevant issues:
  * [https://issues.apache.org/jira/browse/ARROW-3978](https://issues.apache.org/jira/browse/ARROW-3978)
  * [https://issues.apache.org/jira/browse/ARROW-8991](https://issues.apache.org/jira/browse/ARROW-8991)
  * [https://issues.apache.org/jira/browse/ARROW-6024](https://issues.apache.org/jira/browse/ARROW-6024)
  * [https://issues.apache.org/jira/browse/ARROW-6030](https://issues.apache.org/jira/browse/ARROW-6030)
  * [https://issues.apache.org/jira/browse/ARROW-11266](https://issues.apache.org/jira/browse/ARROW-11266)

## Unresolved questions

* Checking only the logical hash to verify the part file may expose us to attacks that tamper non-data blocks like statistics. Tampering the Parquet statistics may result in exclusion of some records from query results that use predicate pushdown
  * To mitigate this we may in future incorporate statistics sanity checks into data validation process

## Future possibilities

* To avoid maintaining two hashes we might want instead to implement a structured data format that is fully deterministic
  * It can be a complete subset of Parquet, to stick with a widespread format
  * Coordinator implementations will have to use a special writer that will have to be implemented in different languages
    * This is already the case with `arrow-digest`, but hashing is a smaller problem to tackle
* We may want to remove the existing limitation that data part files correspond 1-to-1 with metadata blocks (e.g. for compactions) - this is left out for future RFCs


# RFC-003: Content Addressability
Source: https://docs.kamu.dev/odf/rfcs/003-content-addressability



**Start Date**: 2021-11-20

**Specs**:

* [Storage Layout](/odf/spec/datasets/storage-layout)
* [Identity](/odf/spec/datasets/identity)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/6?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/6)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/17?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/17)

## Summary

This RFC specifies how Datasets are uniquely identified across the ODF network, decoupling identity from the aliases.

## Motivation

Datasets are currently identified using their symbolic names. These names are then used, for example, when defining inputs of a derivative dataset or when referring to a dataset in a repository.

Two parties, however, can independently create datasets with conflicting names resulting in high likelihood of name collisions. Allowing users to rename datasets to resolve collisions will inevitably result in breaking the link between derivative dataset and its inputs.

There needs to be a **way to uniquely identify a dataset** on the network, with this identifier being immutable throughout dataset's lifetime and resolvable to find the location(s) of data.

## Guide-level explanation

Per [rationale](#rationale-and-alternatives), we have established that:

* A hash of the last Metadata Block of the dataset can be sufficient to download an entire (subset of a) dataset from a content-addressable storage
* A named reference that points to the last Metadata Block in the chain is needed for identifying dataset as a whole

This RFC therefore suggests some tweaks to the `MetadataBlock` schema to **align it with content-addressability** (see [reference section](#reference-level-explanation)).

Additionally, it will **introduce a globally unique dataset identifier** that can be used to refer to a dataset as a whole. This identifier will follow the [W3C DID Identity Scheme](https://w3c.github.io/did-core/). It will be created by hashing a public key of a `ed25519` key pair, practically guaranteeing its uniqueness.

Symbolic names will become aliases for such identities, meaning that:

* Same dataset can have different names in different repositories (e.g. mirrors) while still sharing same identity
* Inputs of the derivative datasets will also be independent of renaming

Conforming to the `DID` specification will allow us in future to expand related mechanisms like proof of control and ownership.

## Reference-level explanation

To make `MetadataBlock` content-addressable we will remove `blockHash` from it to avoid chicken-egg problem of hashing. The metadata hashing procedure will be updated accordingly.

We will also expand the use of `multihash` format proposed in [RFC-002](/odf/rfcs/002-logical-data-hashes) to all hashes, removing the use of `sha3-256` schema format.

New `dataset-name` format will be introduced for dataset names (symbolic aliases) in the same manner as the existing `dataset-id` format.

Existing `dataset-id` schema format will be changed to expect a DID, following these rules:

* ODF datasets will use a custom DID method: `did:odf:...`
* The method-specific identifier will replicate the structure of [`did:key` method](https://w3c-ccg.github.io/did-method-key/) as a self-describing and upgradeable way to store dataset identity represented by a public key.

Dataset identity will be stored in the new `seed` fields in the `MetadataBlock` schema. Seed will be present (only) in the first block of every dataset.

Dataset creation procedure will involve:

* Generating a new cryptographic key pair (defaulting to `ed25519` algorithm)
* Prefixing it with and appropriate `multicodec` identifier (like `ed25519-pub`)
* Storing this data in the first Metadata Block's `seed` field.

When representing dataset ID as a string the DID format `did:odf:<multibase>` will be used, where the binary data will use `multibase` format and `base58btc` encoding just like in `did:key` method.

The `DatasetSource::Derivative` schema will be updated so that inputs specify:

* `id` - unique identity of a dataset
* `name` - symbolic name of an input to be used in queries only.

With the separation of Dataset IDs from Names we will update our [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) grammar to:

```
DatasetRefLocal = DatasetID / DatasetName
DatasetRefRemote = DatasetID / RemoteDatasetName
DatasetRefAny = DatasetRemoteRef / DatasetLocalRef

RemoteDatasetName = RepositoryName "/" (AccountName "/")? DatasetName
AccountName = Subdomain
RepositoryName = Hostname

DatasetName = Hostname
DatasetID = "did:odf:" Multibase

Hostname = Subdomain ("." Subdomain)*
Subdomain = [a-zA-Z0-9]+ ("-" [a-zA-Z0-9]+)*

Multibase = [a-zA-Z0-9+/=]+
```

## Drawbacks

* We are adding more fields to `MetadataBlock` which is already anemic - this will be addressed separately in a follow-up RFC.
* A seemingly unavoidable break in abstraction layers exists where a named reference (higher-level concept) is used from within derivative dataset inputs from metadata (lower-level concept). This is, however, similar to having an HTML page that contains a relative URL of another page.

## Rationale and alternatives

### Identity of static data in content-addressable systems

The most widespread form of resource identity in decentralized systems today is **content addressability**. Git, Docker & OCI image registries, DHTs, Blockchain, IPFS - resources in these systems are uniquely identifies by (hashes of) their content.

Assuming no hash collisions, this approach allows creating an identity for a resource without any central authority and a risk of collisions. If hashes collide this means resources are in fact identical, leading to a very natural de-duplication of data in the system.

This form of identity, however, is applicable only to static data. When you share a file via IPFS - it's identity is a hash of file's contents. When you modify and share the file again - you get a new identity.

Such form of identity is already perfectly suited for many components of ODF:

* Data part files
* Checkpoints
* Metadata blocks

If we align ODF's hashing with the content-addressable system like IPFS we can get a cool effect:

* A hash of data part file stored in a Metadata Block could be directly used to find and download that data file from IPFS.
* Same goes for the previous metadata block identified by its hash - you could "walk" the metadata chain stored in IPFS in the same way you do on the local drive.

Entire ODF dataset can be stored inside IPFS as its structure maps onto content-addressable storage seamlessly.

### Identity of dynamic data

Unlike all individual components of datasets that are immutable, ODF datasets themselves are dynamic.

In a content-addressable system dataset can be thought of as a "pointer" to the last Metadata Block in the chain. Every time new block is appended - dataset is updated to "point" to the new block. How to represent such "pointer" is ofter referred to as the "naming" problem.

This idea of named pointers is similar to:

* References (e.g. `HEAD`), branches, and tags in `git`
* Image names in Docker / OCI
* IPNS naming service in IPFS

In case of Docker / OCI the naming is centralized - first person to create an organization and push an image with a certain name into the registry wins.

IPNS takes a much more decentralized and conflict-free approach:

* For every "name" a new key pair is generated
* Hash of the public key serves as the name of the pointer
* Private key is used to sign the value to later prove the ownership over it
* Owner writes an entry into DHT with signed value pointing to the hash of IPFS resource
* Entry can be updated when needed

Note that this is just one implementation. Alternative naming schemes using services like DNS and Blockchain exist, but they all have these parts in common:

* A single global-scale system that performs the name resolution (e.g. DNS)
* A mechanism to prove the ownership over a record (e.g. DNS registrar)

Out of this commonality a [W3C Decentralized Identifiers (DIDs) specification](https://w3c.github.io/did-core/) have emerged that provides a common naming scheme, a data model of objects that names resolve into, and mechanisms of proving the control over decentralized identity. IPNS, Blockchains, etc. act as specific implementations of the "verifiable data registries" conforming to DID spec.

## Prior art

* Content addressability in Git, Docker / OCI
* [IPFS](https://ipfs.io/)
  * [naming via IPNS](https://docs.ipfs.io/concepts/ipns/)
  * [naming via DNSLink](https://docs.ipfs.io/concepts/dnslink/)
* [W3C Decentralized Identifiers (DIDs) Specification](https://w3c.github.io/did-core/)
  * [DID Specification Registries](https://www.w3.org/TR/did-spec-registries/)
  * [DID Method Rubric](https://w3c.github.io/did-rubric/)
  * Use of DIDs in the [Ocean Protocol](https://docs.oceanprotocol.com/concepts/did-ddo/)
  * [IPID DID Method](https://did-ipid.github.io/ipid-did-method/)

These are covered in [rationale](#rationale-and-alternatives) above for better flow.

## Unresolved questions

## Future possibilities

* A key pair generated during creation of the dataset identity in future can be used to implement access control and proof of control schemes described in the [W3C DID-Core](https://w3c.github.io/did-core/) specification.

* [IPLD Format](https://ipld.io/) looks a lot similar to what ODF language-independent schemas are trying to accomplish, so we might consider using it in future for better interoperability.


# RFC-004: Metadata Extensibility
Source: https://docs.kamu.dev/odf/rfcs/004-metadata-extensibility



**Start Date**: 2020-12-29

**Specs**:

* [Metadata Chain](/odf/spec/datasets/metadata-chain)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/8?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/8)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/19?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/19)

## Summary

This RFC proposes a new `MetadataBlock` schema that makes metadata less ambiguous and easier to extend.

## Motivation

`MetadataBlock` schema today consists of multiple optional fields that appear in metadata under different conditions. For example, the `outputSlice` field is present when a transformation resulted in some new data added to the datasets, the `outputWatermark` tells us that transformation had bumped up the watermark of a dataset - it can be present even when no new output data was produced. So when you want to process only metadata blocks that correspond to transformations - you get into a position of wondering what's the "definitive sign" of transformation taking place, is it the presence of `outputWatermark`, `ouputSlice`, or `inputSclices`?

These are the clear signs of "anemic data" - the problem ODF was designed to prevent in data, as compared to approaches like Change Data Capture. As the set of features ODF supports grows, such anemic ever-growing schema will become unsustainable.

This RFC will explore how to transition metadata into a descriptive event-based format that can be easily extended by different parties to implement higher-level features.

## Guide-level explanation

This RFC proposes to replace the anemic fields of the `MetadataBlock` schema with an extensible set of `MetadataEvent`s, with every `MetadataBlock` containing a single `MetadataEvent` of a certain type. `MetadataEvent` will correspond to a certain transaction on the dataset (e.g. executing query, specifying polling source, updating transformation query). Every event will contain only fields that are relevant to the transaction.

A similar transformation will be done to the `DatasetSnapshot` schema. Instead of directly representing the first `MetadataBlock` it will contain an array of `MetadataEvent`s that bring the dataset into a desired state. This means that in most cases a dataset newly-created from a `DatasetSnapshot` will have more than one block.

For example, creating dataset from the following snapshot:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: ca.bccdc.covid19.case-details
  kind: root
  metadata:
    - kind: setPollingSource
      fetch: ...
      read: ...
      merge: ...
    - kind: setVocab
      eventTimeColumn: reported_date
```

will produce metadata chain with three blocks:

```yaml theme={null}
- Seed
- SetPollingSource
- SetVocab
```

## Reference-level explanation

Core events:

| Event Type | Description |
| - | - |
| `Seed` | Contains identity information of a dataset and always appears as the first metadata block. |
| `AddData` | Signifies that data was added into the `root` dataset. |
| `SetWatermark` | Signifies that watermark of the dataset has been advanced. |
| `SetTransform` | Defines transformation of the `derivative` dataset. |
| `ExecuteQuery` | Signifies data processing step on the `derivative` dataset. |

Extension events:

| Event Type | Description |
| - | - |
| `SetPollingSource` | (Optional extension) Defines how externally-hosted data can be ingested into the `root` dataset. |

## Drawbacks

[drawbacks]: #drawbacks

* Makes `DatasetSnapshot` schema a little more verbose
* There will be some events that contain a similar set of fields (e.g. `AddData` and `ExecuteQuery` both contain data and watermark info) but will now need to be processed separately. This is a small price to pay for a robust domain model though.

## Rationale and alternatives

* As in case of dataset creation, we can see that it's no longer possible to move dataset into desired state with just one metadata block. Does this hurt consistency?
  * One considered alternative was to allow multiple `MetadataEvent`s per `MetadataBlock` - this would ensure per-block consistency but at the expense of higher complexity
  * It was decided that this complexity is not warranted. Just like with `git`, code is not guaranteed to compile when looking at it on per-commit basis - as long as a group of commits that constitutes a consistent state is pushed atomically consistency overall can be achieved.

## Prior art

## Unresolved questions

* **Forward-compatibility** requires a way to differentiate between events that coordinator has to understand and events that can be safely ignored if coordinator does not support them.
  * Currently, we don't have a good way to achieve this without running into too many issues with `flatbuffers`.
  * We postpone this issue until we get better clarity on the long-term serialization format - we're getting more indications that `flatbuffers` is not the best way forward and may consider alternatives. This also ties in with considering `IPLD`.

## Future possibilities

* This RFC opens up the path for introducing many new `MetadataEvent` types such as for:
  * Storing description / readme
  * Linking to additional content (e.g. notebooks)
  * Embedding a license


# RFC-005: New Annotation Metadata Events
Source: https://docs.kamu.dev/odf/rfcs/005-dataset-annotations



**Start Date**: 2022-04-08

**Specs**:

* [Metadata Chain](/odf/spec/datasets/metadata-chain)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/20?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/20)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/21?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/21)

## Summary

This RFC proposes to extend the set of metadata events with new types used for annotating datasets, such as adding description, licenses, and attachments.

## Motivation

Existing metadata events focus primarily on data pipeline functionality like ingestion and transformation of data, but we need to start expanding into governance/stewardship aspects. This first step will add event types that will let dataset authors provide basic human-oriented information about datasets.

## Guide-level explanation

Example of annotating a dataset:

```yaml theme={null}
kind: DatasetSnapshot
version: 1
content:
  name: ca.ontario.covid19.case-details
  kind: root
  metadata:
    - kind: setPollingSource
      ...
    - kind: setInfo
      description: Confirmed positive cases of COVID-19 in Ontario
      keywords:
        - Healthcare
        - Epidemiology
        - COVID-19
        - SARS-CoV-2
        - Disaggregated
        - Anonymized
        - Ontario
        - Canada
    - kind: setLicense
      name: Open Government Licence – Ontario
      url: https://www.ontario.ca/page/open-government-licence-ontario
    - kind: setAttachments
      attachments:
        kind: embedded
        items:
          - path: README.md
            body: |
              # Confirmed positive cases of COVID-19 in Ontario
              Detailed description follows...
```

## Reference-level explanation

Following extension events will be added to the specification:

| Event Type | Description |
| - | - |
| `SetInfo` | (Optional extension, unstable) Provides basic human-readable description of a dataset |
| `SetLicense` | (Optional extension, unstable) Defines the dataset license. |
| `SetAttachments` | (Optional extension, unstable) Associates a set of files with this dataset (readme, notebooks, additional metadata, etc.). |

## Drawbacks

* More event types to deal with

## Rationale and alternatives

We are entering the territory of metadata where a lot of existing standards exist: `schema.org`, Dublin Core, and many others dealing with dataset descriptions.

This presents a few options:

* Define our own schemas
  * Easy for development, but will result in poor interoperability
* Define our own schemas, but align them with existing standards
  * Better interop, but puts burden on choosing the right standards to follow and keeping up with their evolution
* Start with an existing standard
  * It's likely that some aspects we need will not be covered, and we'll have to extend and customize
  * We may not be able to express some standard schemas in our strongly-typed data model

We decide to delay the decision for now until we do more research on standards and investigate replacing data model with `IPLD`.

In addition to basic annotative metadata we will also have files associated with datasets. These could be:

* A simple README file in different text formats
* Images used in the README
* A set of notebooks that demonstrate how to use datasets

We do not want to re-invent version control systems in ODF metadata chain - therefore we will associate files with datasets by referencing external systems such as `git` repositories. For simple cases we will allow embedding files directly into metadata (e.g. ability to provide README as in example above).

## Prior art

## Unresolved questions

* **Balance between schema and flexibility** - it should be easy for users to extend the metadata for their own governance needs. We should either allow some free-form component in the existing metadata events, or create separate event types.

## Future possibilities

* Information from new events can be indexed and used for full-text search
* License change events should be one of those that trigger notifications for downstream consumers
* All textual fields should be extended with internationalization support


# RFC-006: Store Checkpoints as Files
Source: https://docs.kamu.dev/odf/rfcs/006-checkpoints-as-files



**Start Date**: 2022-04-19

**Specs**:

* [Storage Layout](/odf/spec/datasets/storage-layout)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/24?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/24)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/25?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/25)

## Summary

This RFC proposes to no longer allow arbitrary files and directory structures as engine checkpoints and require that a checkpoint was always a single file.

## Motivation

Currently, engine can produce a checkpoint that includes arbitrary structure of files and directories. This presents few problems:

1. Metadata blocks need to refer to checkpoints by hash, but there is no standard approach to computing a hash of a directory. We would need to create a stable directory hashing algorithm ourselves and this complexity will spread to all implementations.

2. Allowing arbitrary file structures is also a security concern, e.g. need to make sure engines don't create weird symlinks.

3. When downloading a dataset from repository, many transfer protocols don't have a standard way to list a directory. One such example is HTTP - there is no standard content format for `GET` on a directory - most web servers will return a styled HTML. Similarly to how Git can clone a repo using the ["dumb protocol"](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols) we'd like to be able to walk and download entire dataset, and this means having a fixed directory structure and only referencing files.

## Guide-level explanation

## Reference-level explanation

Specification will be updated to no longer refer to checkpoints as opaque directories.

The temporary `ExecuteQueryRequest` schema that relies on file mounting will be updated.

## Drawbacks

* Engines that produce multiple files as checkpoints will need extra `tar` / `untar` steps

## Rationale and alternatives

* Since ODF implements its own content-addressable storage we could support "tree" structures in it just like git does. This would however come at a high cost in complexity and is not justified at the current stage.

## Prior art

* Git's bare repository format and  ["dumb sync protocol"](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols) don't rely on directory listing

## Unresolved questions

* What's the most efficient checkpoint management scheme for long-term solution?
  * Can it be mmap'ed files as in our goal for data slices?

## Future possibilities

* This is a necessary step for implementing a sync protocol for pulling datasets from IPFS and other storage systems that provide HTTP gateway.


# RFC-007: Simple Transfer Protocol
Source: https://docs.kamu.dev/odf/rfcs/007-simple-transfer-protocol



**Start Date**: 2022-04-19

**Specs**:

* [Transfer Protocols](/odf/spec/exchange/transfer-protocols)
* [Identity](/odf/spec/datasets/identity)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/26?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/26)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/27?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/27)

## Summary

A minimalistic read-only transfer protocol that can be used to explore and transfer datasets between repositories.

## Motivation

ODF needs a bare-minimum transfer protocol for synchronizing datasets between two repositories. The protocol needs to specify how, knowing the URL of a dataset, one can discover ("walk") the dataset's Metadata Chain and find all associated objects like refs, checkpoints, and data files.

Protocol is aimed to achieve maximal interoperability and ease of repository setup and less concerned with efficiency of transfer. More advanced protocols supporting efficient traversal of metadata, highly-parallel downloads, and data querying will be proposed separately.

Introducing this protocol will allow people to easily operate basic read-only ODF repositories by serving a directory under an HTTP server, or sync ODF datasets from IPFS.

## Guide-level explanation

The protocol proposed here is inspired by [Git's "Dumb Protocol"](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols) that allows you to clone a Git repository backed by many application-level (L7) protocols (like HTTP and FTP) and works over any block or file-based storage that allows Unix path-like names for objects.

The only operation a server needs to support is **reading an object by key**. The protocol purposefully does not rely on any advanced features like ability to list a directory.

## Reference-level explanation

The following section describing the protocol will be introduced into the spec:

> Simple Transfer Protocol is a bare-minimum read-only protocol used for synchronizing a dataset between repositories. It requires no ODF-specific logic on the server side and can be easily implemented, for example, by serving a dataset directory under an HTTP server.
>
> To describe the protocol we will use HTTP `GET {object-key}` notation below, but note that this protocol can be implemented on top of any block or file-based protocol that supports Unix path-like object keys.
>
> 1. Process begins with `GET /refs/head` to get the hash of the last Metadata Block
> 2. The "metadata walking" process starts with `GET /blocks/{blockHash}` and continues following the `prevBlockHash` links
> 3. Data part files can be downloaded by using `DataSlice::physicalHash` links with `GET /data/{physicalHash}`
> 4. Checkpoints are similarly downloaded using `Checkpoint::physicalHash` links with `GET /checkpoints/{physicalHash}`
> 5. The process continues until reaching the first (seed) block of the dataset or other termination condition (e.g. reaching the block that has already been synced previously)

Additionally, the dataset identity grammar will be updated to allow referencing remote datasets by URL:

```peg theme={null}
DatasetRefRemote = DatasetID / RemoteDatasetName / Url

Url = Scheme "://" [^\n]+
Scheme = [a-z0-9]+ ("+" [a-z0-9]+)*
```

The `DataSlice` and `Checkpoint` objects will be extended to carry the `size` of the files to provide size hints when transferring files between repositories in a streaming fashion.

## Drawbacks

* Walking the metadata chain one object at a time limits parallelism, but as previously stated - performance is not the goal here, and some parallelism is still possible when downloading data and checkpoint files.

## Rationale and alternatives

## Prior art

* [Git's "Dumb Protocol"](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols)

## Unresolved questions

* Git's traversal begins with `info/refs` file that is updated by `git update-server-info` command. Having this file as a starting point allows client to discover all branches and tags in the repo. For the time being while ODF branching model remains underspecified we will begin traversal with the `/refs/head` path.
* This protocol is scoped to a single dataset - how multi-dataset repositories and mirrors will function is left out of scope of this RFC.

## Future possibilities

* This is a necessary step for implementing pulling of datasets from IPFS and other storage systems that provide HTTP gateway.
* In the future, we can specify protocols that allow more efficient and highly-parallel transfer in expense of having slightly more logic on the server side.


# RFC-008: Smart Transfer Protocol
Source: https://docs.kamu.dev/odf/rfcs/008-smart-transfer-protocol



**Start Date**: 2022-12-23

**Specs**:

* [Transfer Protocols](/odf/spec/exchange/transfer-protocols)

[![RFC Status](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/35?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/35)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/38?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/38)

## Summary

An enhanced read/write transfer protocol that can be used to efficiently synchronize remote dataset repositories.

## Motivation

The performance limitations of the [Simple Transfer Protocol](/odf/rfcs/007-simple-transfer-protocol) are significant:

* the exchange becomes noticeably slow on datasets with >200 files, there is no use of batching, compression
* proxied push/pull are not allowed (being able to read/write directly from/to S3 instead of overloading the servers)

Another limitation this protocol is trying to address is that HTTP-based dataset sharing should not be read-only.

## Guide-level explanation

The protocol assumes establishing a duplex asynchronous communication channel between the parties.

A separate session of message exchanges is open for a single pull or push operation,
and it is closed automatically after the intended operation is complete or fails with an error.

For a `pull` operation:

* Client initiates the request to the Server, sending the ID of the dataset and a desired block range
* Server responds by sending a tarball of metadata that matches the requested block range
* Client analyzes the received blocks and identifiers missing object files (such as data slices, checkpoints)
* Client negotiates with the Server on the download method for a subset or full set of missing object files:
  * for each object file, Server creates a time-limited pre-signed download URL with read-only access the storage system
  * Client directly reads the objects files from the storage system via pre-signed URLs
  * the negotiation can repeat for another subset of missing object files
* Client commits the new synchronized blocks, and closes the operation

For a `push` operation:

* Client initiates the request to the Server, sending the ID of the dataset and a tarball of the new metadata blocks
* If Server does not detect a divergence in the metadata, it confirms the Client may proceed synchronizing objects
* Client negotiates with the Server on the upload method for a subset or full set of new object files:
  * for each object file, Server creates a time-limited pre-signed upload URL with write-only access to the storage system
  * Client directly uploads the object files to the storage system via pre-signed URLs
  * the negotiation can repeat for another subset of new object files
* Client informs the Server about completion of the objects synchronization
* Server verifies the required objects exist and attempts to commit the new metadata blocks
* In case of success, the Server confirms the commit succeeded, and closes the operation

The exchange of object files should be performed with delays and object sizes taken into account.
As the expiration time of pre-signed URLs is limited, in case of large bulks have to be exchanged,
the pre-signed URLs might expire before the actual transfer starts - in this case they need to be re-requested.

The parallelism of the object download/upload operations to/from storage system is not limited by the protocol,
neither is to be constrained by the server. It's considered to be an implementation detail of the Client.

In case of errors, timeouts, the Client needs to re-initiate the operation with the Server by opening another session.
New operation may skip the synchronization of the already processed object files from the previous sessions,
even if the changes have not been finally committed to the metadata. Cleaning orphan files that accumulate on transfer errors
is a subject for a separate Garbage Collector mechanism within the dataset storage, that is out of the scope of this RFC.

The protocol can be implemented using [WebSockets](https://websockets.spec.whatwg.org//) or any other
alternative type of channel that supports bi-directional non-blocking message exchanges.

## Reference-level explanation

Smart Transfer Protocol defines 2 kinds of APIs:

* `OpenAPI` [description of the initial REST endpoints](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/transfer/smart-transfer-protocol.openapi.yaml).
* `AsyncAPI` [description of the protocol messages](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/transfer/smart-transfer-protocol.asyncapi.yaml).

`OpenAPI` is a superset of REST API endpoints as defined in the
[Simple Transfer Protocol](/odf/rfcs/007-simple-transfer-protocol), with extensions to establish
asynchronous duplex messaging channel as defined by `AsyncAPI`.

### Push flow

<Diagram alt="Diagram: ODF Smart TransferProtocol Push" />

### Pull flow

<Diagram alt="Diagram: ODF Smart TransferProtocol Pull" />

## Drawbacks

* Ended up with a custom protocol rather than reusing some standard
* Asynchronous duplex messaging APIs are harder to test

## Alternatives

* Considered implementing the protocol in the form of synchronous request-response REST API:
  * Good parts:
    * Better portability (although, not really relevant in 2023 by now)
    * Easier to test the implementations
  * Bad parts:
    * Requires modeling push flows as a long-running operation with the state stored in the DB
    * More complex state recovery logic in case of resume/cancellation
    * Potential re-iteration over the same metadata chain, as messages get split between independent stateless calls
  * Decision:
    * Refused in favor of asynchronous bi-directional messaging API

## Prior art

* [Git's "Smart Protocol"](https://git-scm.com/book/en/v2/Git-Internals-Transfer-Protocols)

* [DeltaSharing](https://github.com/delta-io/delta-sharing/blob/main/PROTOCOL.md):
  * Good parts:
    * Simple - can easily support this protocol if needed
    * Choice of \[application/x-ndjson] ([https://github.com/ndjson/ndjson-spec](https://github.com/ndjson/ndjson-spec)) encoding
    * human-readable
    * easy to parse
    * self-delimiting
    * efficient for streaming / iterative production and consumption
  * Bad parts:
    * Very "egocentric" - designed mostly just with Databricks' own case in mind
    * Uses Spark-specific schema format - not even Parquet, which would be a much better lower-level abstraction
    * Using "SQL subset" for predicate hints (pre-filters)
    * Change Data Feed API is Delta Lake specific
    * Leaks the format of CDF files that show INSERT/UPDATE/DELETE/MERGE operations
    * Forces share/schema/table hierarchy upon server which is unnecessary

## Unresolved questions

* Garbage Collector mechanism within the dataset storage, as orphaned files may occur:
  * when client failed to upload the objects and did not commit the push
  * when late collision happens (parallel pushes)
* Should this protocol support live-streaming of datasets?
* Should it support in-band data transfers?
  * And if yes, should it be used for query results or for getting data from projections?

## Future possibilities

* DeltaSharing can be implemented as a separate protocol reusing a lot of the parts from this
* A detection of unfinished pushes can be added to clean up orphaned files without doing a full GC run on a dataset


# RFC-009: Ingestion Source State
Source: https://docs.kamu.dev/odf/rfcs/009-ingest-source-state



**Start Date**: 2023-04-28

**Specs**:

* [Ingestion](/odf/spec/datasets/ingestion)

[![RFC](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/50?label=RFC%20Status)](https://github.com/kamu-data/open-data-fabric/issues/50)

[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/51?label=Spec%20PR)](https://github.com/kamu-data/open-data-fabric/pull/51)

**Compatibility**: Backwards-compatible, forwards-incompatible

## Summary

Introduces a `sourceState` field into `AddData` event to store the state of the generic data source from which the data is being added, allowing for incremental ingestion.

## Motivation

ODF implementations often need to store the state of the data source from which data is ingested to avoid expensive recomputations.

For example when using `SetPollingSource` source state can be in the form of:

* ETag of Last-Modified header for HTTP sources
* filename of the last file matched by glob pattern
* or a height of the block of indexed blockchain.

Currently, we don't have anywhere to store such data.

## Guide-level explanation

When ingesting data, an ODF implementation will be able to attach opaque state data to the `AddData` event in order to resume ingestion most efficiently on the next iteration. The source state data will allow differentiating the kind of state that is preserved (similarly to MIME type) and will specify the identity of the source.

## Reference-level explanation

New `SourceState` metadata fragment will be introduced.

The schema of `AddData` metadata event will:

* be extended with an optional `sourceState` field
* have `outputData` field made optional (as it's possible for ingest to produce no new data but update source state, watermark, or checkpoint)

Two predefined source state kinds will be added:

* `odf/etag` - for state identifiers similar to `ETag` HTTP header
* `odf/last-modified` - for RFC3338 timestamps with meaning similar to `Last-Modified` HTTP header

One predefined source ID will be added:

* `odf/polling` - referring to the source specified in the `SetPollingSource` metadata event

Both fields will be plain strings and not enums allowing different implementations to define their own extensions.

It should be **safe to ignore** the source state that an implementation does not understand.

Example `SourceState`:

```yaml theme={null}
sourceState:
  kind: odf/etag
  source: odf/polling
  value: W/"95c5fde3918cba7c33eaac7ff9d02b22"
```

## Compatibility

This change will be fully backwards-compatible. It will be forwards-incompatible due to making of `AddData::outputData` field optional.

## Drawbacks

* More complexity in metadata

## Alternatives

* Store source state as part of the ingest checkpoint:
  * Would require storing source state alongside engine checkpoints within one file (e.g. by `tar`'ing them together)
  * Would complicate and slows down engine checkpoint extraction
  * Would only work for a single source

## Prior art

N/A

## Unresolved questions

N/A

## Future possibilities

* The source identity (`source` field) should allow us to support multiple concurrent ingestion sources if needed


# RFC-010: Data Schema in Metadata
Source: https://docs.kamu.dev/odf/rfcs/010-data-schema-in-metadata



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/62?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/55)
[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/63?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/63)

**Start Date**: 2023-10-25

**Specs**:

* [Metadata Chain](/odf/spec/datasets/metadata-chain)

**Authors**:

* [Sergiy Zaychenko](mailto:sergiy.zaychenko@kamu.dev), [Kamu](https://kamu.dev)
* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* \[V] Backwards-compatible
* [ ] Forwards-compatible

## Summary

Introduces a new metadata event `SetDataSchema` that specifies the Arrow schema of the Data Slices added to the dataset following this event.

## Motivation

We currently don't store the schema of Data Slice files in metadata.

The closest thing we have is the `read`-stage schemas defined in `SetPollingSource` event, but those schemas are only applied to external data when reading it, can be reshaped by the `preprocess` queries, and also get extended with system columns.

This creates several problems:

* One cannot understand the data schema from metadata alone and needs to inspect the last Parquet file, which can be costly
* The data schema is effectively *undefined* until the first data file is written
* It's not possible to pre-define the dataset schema before writing data into it (which poses a problem for "push"-style ingestion, see RFC-011)

## Guide-level explanation

A new metadata event `SetDataSchema` is introduced that will contain full schema of the Data Slice files that are written following it.

Multiple `SetDataSchema` events can appear in the metadata chain, allowing for schema evolution.

`SetDataSchema` event will not normally be specified in Dataset Snapshots, but instead will be generated by ODF implementations by inferring the final schema based on:

* First root ingestion / derivative transformation to take place
* Column names defined in `SetVocab` events

## Reference-level explanation

`SetDataSchema` event schema:

```json theme={null}
{
  "$id": "http://open-data-fabric.github.com/schemas/SetDataSchema",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Specifies the complete schema of Data Slices added to the Dataset following this event.",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "schema"
  ],
  "properties": {
    "schema": {
      "type": "object",
      "format": "flatbuffers",
      "description": "Apache Arrow schema encoded in its native flatbuffers representation."
    }
  }
}
```

Note the introduction of new format `flatbuffers`. It will handle the data as a plain byte array. Libraries will be able to add special accessors that provide typed access to the underlying object - in this case Arrow Schema type. Due to the nature of flatbuffers this can be done without extra allocations.

## Compatibility

This change is **backwards-compatible**, as existing implementations can continue to fall back to reading schema from Data Slice files if `SetDataSchema` event is not found.

This change is **forwards-incompatible** as older implementation will fail to parse the new core event (we still lack the mechanism to mark some events as "OK to disregard").

## Drawbacks

* More complexity in metadata

## Alternatives

### Event order rules

It was debated whether to require the existence of `SetDataSchema` prior to definition of any ingest sources or transformations.

This would require:

* Either making users specify schema explicitly - deemed too complex and impractical.
* Or implementations to infer the schema from the defined sources - deemed as something that would slow down the dataset creation from snapshots, since inferring the schema would require instantiating an Engine.

The compromise was reached in requiring `SetDataSchema` before any Data Slices are added, but allowing it to follow the `SetPollingSource` and `SetTransform` events from which it will be **inferred during the first processing**.

### Schema format

We chose to use Arrow schema instead of Parquet because:

* Arrow is our cross-system data interchange format
* Arrow schema is more expressive and can represent types more accurately (e.g. timezones)
* We care about more about logical schema rather than how data appears on disk, and this will allow us in future to support other on-disk formats as long as their schemas can be mapped to Arrow.

### Schema representation

Arrow Schema currently does not have a standard human-readable representation - it is specified only by its `flatbuffers` schema.

We therefore store the schema as flatbuffer-in-flatbuffer in the metadata chain, which gives us optimal encoding, and leave it up to implementations to present in human-readable format of their choice to the users.

For now, we don't expect users ever having to manually specify the schema, but if needed we could explore flatbuffers own standard way to serialize data to JSON.

See also:

* [https://github.com/apache/arrow/issues/25078](https://github.com/apache/arrow/issues/25078)
* [https://stackoverflow.com/questions/48215929/can-i-serialize-dserialize-flatbuffers-to-from-json](https://stackoverflow.com/questions/48215929/can-i-serialize-dserialize-flatbuffers-to-from-json)

## Prior art

N/A

## Unresolved questions

N/A

## Future possibilities

N/A


# RFC-011: Push Ingest Sources
Source: https://docs.kamu.dev/odf/rfcs/011-push-ingest-sources



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/62?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/62)
[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/63?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/63)

**Start Date**: 2023-10-25

**Specs**:

* [Ingestion](/odf/spec/datasets/ingestion)

**Authors**:

* [Sergiy Zaychenko](mailto:sergiy.zaychenko@kamu.dev), [Kamu](https://kamu.dev)
* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [ ] Forwards-compatible

## Summary

Introduces a set of new metadata events (`AddPushSource`, `DisablePushSource`) for defining "push"-style data ingestion sources for root datasets.

## Motivation

Using `SetPollingSource` event users of ODF can already define "poll"-style data ingestion sources that describe how to read and historize external data to bring it into a root dataset.

We would also like to support "push"-style sources where an external agent periodically adds data into a root dataset. Such sources can range from IoT devices to scripts and all kinds of business process automation.

Similarly to polling sources, push sources also will require defining:

* What format the data is expected to arrive in
* Preprocessing queries (e.g. if the data format of the source like IoT device cannot be changed upstream)
* Merge strategy

## Guide-level explanation

Users can define push sources on root datasets using `AddPushSource`, `DisablePushSource` events.

Multiple push sources can be active simultaneously per one dataset for cases where multiple actors are writing to the same dataset simultaneously in slightly varying formats.

Push and polling sources are mutually exclusive. However, it must be possible to switch dataset from "push" to "pull" ingest and vice versa - thus we also introduce `DisablePollingSource` event that allows to turn off the polling source before switching to push model.

The state of push sources can be stored in existing `sourceState` section of `AddData` event.

The implementations will verify that all push sources result in the same final data schema, as captured in the `SetDataSchema` event.

## Reference-level explanation

`AddPushSource` event schema:

```json theme={null}
{
  "$id": "http://open-data-fabric.github.com/schemas/AddPushSource",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Describes how to ingest data into a root dataset from a certain logical source.",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "source",
    "read",
    "merge"
  ],
  "properties": {
    "source": {
      "type": "string",
      "description": "Name that identifies this source within this dataset."
    },
    "read": {
      "$ref": "/schemas/ReadStep",
      "description": "Defines how data is read into structured format."
    },
    "preprocess": {
      "$ref": "/schemas/Transform",
      "description": "Pre-processing query that shapes the data."
    },
    "merge": {
      "$ref": "/schemas/MergeStrategy",
      "description": "Determines how newly-ingested data should be merged with existing history."
    }
  }
}
```

`DisablePushSource` event schema:

```json theme={null}
{
  "$id": "http://open-data-fabric.github.com/schemas/DisablePushSource",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Disables the previously defined source.",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "source"
  ],
  "properties": {
    "source": {
      "type": "string",
      "description": "Identifier of the source to be disabled."
    }
  }
}
```

`DisablePollingSource` event schema:

```json theme={null}
{
  "$id": "http://open-data-fabric.github.com/schemas/DisablePollingSource",
  "$schema": "http://json-schema.org/draft-07/schema#",
  "description": "Disables the previously defined polling source.",
  "type": "object",
  "additionalProperties": false,
  "required": [],
  "properties": {}
}
```

The `source` identifier is the same one as used in `SourceState::source`, meaning that `AddData::sourceState` can be used by the implementations to store the state of the push source for resuming the consumption and implementing "exactly-once" semantics.

## Compatibility

This change is **backwards-compatible**.

This change is **forwards-incompatible** as older implementation will fail to parse the new core events (we still lack the mechanism to mark some events as "OK to disregard").

## Drawbacks

* More complexity in metadata

## Alternatives

N/A

## Prior art

N/A

## Unresolved questions

* Non-trivial ingestion merge strategies (snapshot, ledger) currently require access to past stream data to perform CDC and deduplication. We were considering to avoid the need to read past data eventually by storing all necessary information in the checkpoints. However, allowing for multiple push sources would mean that we need separate checkpoints per source as sources can have different merge strategies. Given that it's still not clear if storing state in checkpoints for things like CDC is even practical for large datasets - we decided to not let this block this RFC, as the need for several push sources per dataset does seem like a practical necessity.

## Future possibilities

N/A


# RFC-012: Recommend `base16` encoding for textual representation of hashes and DIDs
Source: https://docs.kamu.dev/odf/rfcs/012-recommend-base16-encoding



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/62?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/23)
[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/63?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/63)

**Start Date**: 2023-12-14

**Specs**:

* [Metadata Chain](/odf/spec/datasets/metadata-chain)
* [Identity](/odf/spec/datasets/identity)

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [ ] Backwards-compatible
* [ ] Forwards-compatible

## Summary

Proposes to expand the spec's support to all `multibase` encodings but recommend `base16` as default choice for representing hashes and DIDs.

## Motivation

Very early on ODF adopted `base58btc` encoding, simply mimicking other projects like Bitcoin, IPFS, and `did:key` W3C spec. But this encoding is problematic:

* It can result in text that resembles words, potentially resulting in swear words in hashes and DIDs
* It's hard to write down or read out as many letters and numbers can look similar
* Its lexicographic order does not match the sort order of binary data
* It doesn't work with subdomains ([RFC1035](https://datatracker.ietf.org/doc/html/rfc1035), [RFC1123](https://datatracker.ietf.org/doc/html/rfc1123)) that are restricted to: case-insensitive, `a-b0-9` and less than 63 bytes.

The latter issue is especially important to allow exposing datasets and their attachments in a similar way to IPFS gateways, where subdomains are needed to ensure HTTP origin security.

## Reference-level explanation

The proposal is to:

1. Specify that ODF implementations must support all [multibase](https://github.com/multiformats/multibase) formats in `final` status of approval process, performing transcoding where needed.

2. Recommend new implementations to use `base16` encoding ([RFC4648](https://datatracker.ietf.org/doc/html/rfc4648)) where possible to avoid the pitfalls described in motivation section.

3. Update `did:odf` method specification to use `base16` encoding too, while allowing to transcode to `base58btc` for compatibility with `did:key` method.

## Compatibility

This change will be executed as part of the backwards compatibility breaking changes.

## Drawbacks

1. **Compatibility**: Declaring that we support multiple `multibase` encodings instead of just one may slightly complicate compatibility between implementations, but since `multibase` is a self-describing format these issues should be easily addressable. This approach also has additional benefit of upgradability.

2. **Increase of length**: For an implementation choosing to stick with `base16` recommendation the main drawback is the increase of length of hashes in:

* file and directory names
* human-readable formats
* presentation layer

Because the binary representation is unaffected we don't see this as an issue.

3. **`did:key` compatibility**: `did:key` spec currently only allows `base58btc` encoding. By using `base16` in `did:odf`, we will have to transcode the DIDs when passing them to frameworks that work with `did:key`. The main drawback is that developers will not be able to visually compare or search by a `did:key` that, for example, appears in an auth token, without performing conversion first.

We believe accept this issue while expecting that:

* Most places that accept `did:odf` (e.g. API endpoints) can perform transcoding automatically
* The choice of `base58btc` in `did:key` method will be reconsidered as per this [open issue](https://github.com/w3c-ccg/did-method-key/issues/21), potentially opening `did:key` up to supporting multiple `multibase` encodings.

## Alternatives

* `base32hex` encoding is case-insensitive and maintains sort order - dismissed as it can still form obscene words
* [Crackford's base32 encoding](https://www.crockford.com/base32.html) avoids letter/number similarity and excludes 'U' to reduce likelihood of accidental obscenity - dismissed as not yet supported by the [multibase](https://github.com/multiformats/multibase) spec

## Prior art

* Related IPFS tickets:
  * [https://github.com/ipfs/kubo/issues/4143](https://github.com/ipfs/kubo/issues/4143)
  * [https://github.com/ipfs/specs/issues/247](https://github.com/ipfs/specs/issues/247)

## Unresolved questions

N/A

## Future possibilities

N/A


# RFC-013: Enum representation in YAML encoding
Source: https://docs.kamu.dev/odf/rfcs/013-yaml-enum-representation



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/62?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/70)
[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/63?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/63)

**Start Date**: 2023-12-14

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* \[V] Backwards-compatible
* [ ] Forwards-compatible

## Summary

Proposes to use `kind: PascalCaseVariantName` convention to standardize how enum variants are differentiated in YAML / JSON serialization format.

## Motivation

Currently, the spec does not specify how enum variants must be differentiated between when ODF structures are serialized in YAML / JSON form.

This part was left to implementations and Kamu just ended up using `kind: camelCaseVariantName`.

## Reference-level explanation

The proposal is to use internally-tagged representation:

```yaml theme={null}
enumProperty:
  kind: PascalCaseVariantName
  foo: bar
```

The internal tagging provides the most concise representation, while remaining non-ambiguous. The slight drawback is that it gives `kind` property a special function, and we should be mindful of name conflicts.

Using the `PascalCase` for variant names makes names appear identical in YAML / JSON to how they appear in ODF schemas.

While `PascalCase` is the representation that should be used by all implementations when outputting YAML/JSON, in the interest of being more forgiving to human input implementations should also tolerate `camelCase` and `lowercase` names when reading YAML/JSON metadata.

Same rules should apply to string enums. For example a property with schema such as:

```json theme={null}
{
  "type": "string",
  "enumName": "SourceOrdering",
  "enum": [
    "ByEventTime",
    "ByName"
  ]
}
```

When serialized to YAML should output `ByEventTime`, and be able to deserialize from `ByEventTime`, `byEventTime`, and `byeventtime`.

## Compatibility

This change is backwards compatible as all existing `DatasetSnapshot`s that exist in YAML format and use `camelCase` naming will still be accepted under the new scheme.

## Drawbacks

N/A

## Alternatives

* [Supported enum representations in `serde`](https://serde.rs/enum-representations.html) (a de facto Rust's standard serialization library)

## Prior art

N/A

## Unresolved questions

N/A

## Future possibilities

N/A


# RFC-014: Minimizing scanning for last offset and block
Source: https://docs.kamu.dev/odf/rfcs/014-minimize-offset-scanning



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/62?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/69)
[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/63?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/63)

**Start Date**: 2023-12-17

**Specs**:

* [Ingestion](/odf/spec/datasets/ingestion)
* [Metadata Chain](/odf/spec/datasets/metadata-chain)

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [ ] Backwards-compatible
* [ ] Forwards-compatible

## Summary

Removes `SetWatermark` and modifies `AddData` and `ExecuteQuery` events so that they always carried enough information about the output offset and offsets of the inputs to prepare the next transaction without deep scanning of the metadata.

## Motivation

Currently, `AddData` and `ExecuteQuery` events carry  `[start, end]` intervals for blocks and offsets to describe inputs and outputs of a transaction. To represent empty input/output they are **made optional**.

This is problematic, because to understand what offsets or blocks need to be used for inputs / output of the next transaction it's sometimes required to scan metadata chain until a **non-empty interval** is encountered.

Extending metadata to carry this information regardless of whether any input data was used or any output data was written would allow us to prepare the next transaction using the last `AddData` or `ExecuteQuery` block, avoiding deep scanning of the chain.

## Guide-level explanation

Imagine a derivative dataset that is aggregating data from an IoT source:

* The source writes 1M batches over the year, resulting in \~1M `ExecuteQuery` events
* The dataset is configured to aggregate data per month, thus only \~12 of `ExecuteQuery` events will have data in them
* Because when transaction does not produce data we don't write any offset information, to find out the last offset to use for the next transaction the system needs to routinely **traverse thousands of empty blocks** to find a non-empty one.

Similar problem exists in multi-input derivative datasets where one input updates significantly more frequently than the other, making us scan many `ExecuteQuery` blocks to understand the last input offset interval that was already processed.

The proposed change avoids this by always carrying enough information in the events to understand which offsets and blocks were already processed.

It is also proposed to remove the `SetWatermark` event, treating the advancement of the watermark the same as in the case when `AddData` / `ExecuteQuery` blocks don't contain any state change except for the watermark, reducing the complexity and interplay of events that implementations have to deal with.

## Reference-level explanation

The `AddData` and `ExecuteQuery` events will be extended with the new `prevOffset` property to represent last offset of the previous data slice. It must be equal to the last non-empty `outputData.offsetInterval.end`.

For example, if the first `AddData` event of a Root dataset looks like this (using `null` to represent the missing properties explicitly):

```yaml theme={null}
prevCheckpoint: null
prevOffset: null
newData:
  logicalHash: <hash>
  physicalHash: <hash>
  offsetInterval:
    start: 0
    end: 9
  size: 100
newCheckpoint: null
newWatermark: "2023-12-31T00:00:00"
newSourceState: null
```

A correctly chained `AddData` event with data will look like so:

```yaml theme={null}
prevCheckpoint: null
prevOffset: 9
newData:
  logicalHash: <hash>
  physicalHash: <hash>
  offsetInterval:
    start: 10
    end: 19
  size: 100
newCheckpoint: null
newWatermark: "2023-12-31T01:00:00"
newSourceState: null
```

While the next chained `AddData` that only advances the watermark will look like so:

```yaml theme={null}
prevCheckpoint: null
prevOffset: 19
newData: null
newCheckpoint: null
newWatermark: "2023-12-31T03:00:00"
newSourceState: null
```

The `SetWatermark` event, while advancing the watermark, does not carry over any other things like previous offset, checkpoint, or source state, which unnecessarily complicates the processing. We, therefore, propose removing this event in favor of "no-data" variant of `AddData` and `ExecuteQuery` events, as shown in the example above.

Similarly, the `InputSlice` will no longer use closed `[start, end]` intervals for `blockInterval` and `offsetInterval`. The new schema will use what are essentially half-open intervals where starting point will always be carried across all transaction, even if the interval itself is empty.

Proposed `InputSlice` schema:

```json theme={null}
{
  "description": "Describes a slice of the input dataset used during a transformation",
  "type": "object",
  "additionalProperties": false,
  "required": [
    "datasetId"
  ],
  "properties": {
    "datasetId": {
      "type": "string",
      "format": "dataset-id",
      "description": "Input dataset identifier."
    },
    "prevBlockHash": {
      "type": "string",
      "format": "multihash",
      "description": "Last block of the input dataset that was previously incorporated into the derivative transformation, if any. Must be equal to the last non-empty `newBlockHash`. Together with `newBlockHash` defines a half-open `(prevBlockHash, newBlockHash]` interval of blocks that will be considered in this transaction."
    },
    "newBlockHash": {
      "type": "string",
      "format": "multihash",
      "description": "Hash of the last block that will be incorporated into the derivative transformation. When present, defines a half-open `(prevBlockHash, newBlockHash]` interval of blocks that will be considered in this transaction."
    },
    "prevOffset": {
      "type": "integer",
      "format": "uint64",
      "description": "Last data record offset in the input dataset that was previously incorporated into the derivative transformation, if any. Must be equal to the last non-empty `newOffset`. Together with `newOffset` defines a half-open `(prevOffset, newOffset]` interval of data records that will be considered in this transaction."
    },
    "newOffset": {
      "type": "integer",
      "format": "uint64",
      "description": "Offset of the last data record that will be incorporated into the derivative transformation, if any. When present, defines a half-open `(prevOffset, newOffset]` interval of data records that will be considered in this transaction."
    }
  }
}
```

## Compatibility

This change will be executed as part of the backwards compatibility breaking changes.

## Drawbacks

* Half-open intervals are slightly harder to understand
* Will require stricter validation in implementations to catch propagation errors
* Replacing `SetWatermark` with "no-data" version of `AddData` event makes the event slightly bigger, but with an added bonus of minimizing the chain scanning and simplifying resuming the processing

## Alternatives

* We could rely on "skip list" data structure to find blocks with non-empty intervals
  * While this may work for output data, it would bring too much complexity in case of input datasets

## Prior art

N/A

## Unresolved questions

N/A

## Future possibilities

We are still considering to implement the "skip list" data structure, but primarily to allow skipping all *"routine"* data processing events like `AddData` and `ExecuteQuery` (high-volume) and let us quickly iterate over the *"out-of-ordinary"* (low-volume) metadata events.


# RFC-015: Unified changelog stream schema
Source: https://docs.kamu.dev/odf/rfcs/015-unified-changelog-stream-schema



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/47?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/47)
[![Spec PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/72?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/72)

**Start Date**: 2023-12-17

**Specs**:

* [Data Model](/odf/spec/datasets/data-model)
* [Processing Model](/odf/spec/transformations/processing-model)
* [Engine Contract](/odf/spec/transformations/engine-contract)

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [ ] Backwards-compatible
* [ ] Forwards-compatible

## Summary

Introduces new `op` schema column that will be used across all datasets to differentiate regular appends, corrections, and retractions. To represent corrections a two-event "changelog stream" data model similar to Apache Flink's will be used.

## Motivation

Errors in source data are inevitable and require a mechanism for correcting them post factum. Unlike databases, where one could issue `DELETE` or `UPDATE` queries, our core data model is an immutable append-only stream, and thus requires a different mechanism to issue **retractions and corrections** for past events.

Additionally, in cases when a stream processing operation encounters late data (beyond the current watermark), or late upstream retractions and corrections, it may also need to issue corrections or retractions for previously produced result records that were influenced by the late events.

We consider the correction/retraction model fundamental and essential to making data processing **maximally autonomous**. In turn, only by making data processing autonomous can collaborative data pipelines exist at the **global scale**. This RFC proposes necessary schema changes to make this model uniform across all ODF datasets.

## Guide-level explanation

### Scenario

Consider a video game that writes a `match_scores` dataset as players complete the matches:

| match\_time | match\_id | player\_name | score |
| -: | -: | -: | -: |
| t1 | 1 | Alice | 100 |
| t1 | 1 | Bob | 80 |
| t2 | 2 | Alice | 70 |
| t2 | 2 | Charlie | 90 |
| t3 | 3 | Bob | 60 |
| t3 | 3 | Charlie | 110 |

Let's say we want to create a dataset that continuously tracks the **top two** (for simplicity) highest-scoring players - a leader board.

This is the case of Top-N query, which can be written in Apache Flink SQL as:

```sql theme={null}
select
  *
from (
  select
    ROW_NUMBER() OVER (ORDER BY score desc) AS place,
    match_time,
    player_name,
    score,
  from match_scores
) where place <= 2
```

### Representing changes in streams

There are several ways to represent how this query behaves when applied to the above input stream.

#### Retract Stream

The most generic way is a "retract stream" that only uses append `+A` and retract `-R` operations (note the new `op` column):

| op | place | match\_time | player\_name | score |
| -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| -R | 2 | t1 | Bob | 80 |
| +A | 2 | t2 | Charlie | 90 |
| -R | 1 | t1 | Alice | 100 |
| -R | 2 | t2 | Charlie | 90 |
| +A | 1 | t3 | Charlie | 110 |
| +A | 2 | t3 | Alice | 100 |

In this model the updated state of the leader board is compared with the previous state after each new event, and necessary records first get retracted before being replaced with appends.

#### Upsert Stream

Using the knowledge that the `place` column plays the role of a unique key of the resulting state we could also represent the above as an "upsert stream" using only upsert `+A` and retract `-R` operations:

| op | place | match\_time | player\_name | score |
| -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| +A | 2 | t2 | Charlie | 90 |
| +A | 1 | t3 | Charlie | 110 |
| +A | 2 | t3 | Alice | 100 |

This additional knowledge allows us to significantly compact the stream.

Although the retract operation does not appear in our example, it is needed for completeness, e.g. imagine if the source stream retracted the result of the first match right after `t1` - this would require us to empty the leader board with retractions too.

#### Changelog Stream (single event)

Some systems produce "changelog streams" containing append `+A`, retract `-R`, and update `+U` operations with update carrying both the new values and the old values of the record being changed:

| op | place | match\_time | player\_name | score | match\_time\_old | old\_player\_name | old\_score |
| -: | -: | -: | -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 | | | |
| +A | 2 | t1 | Bob | 80 | | | |
| +U | 2 | t2 | Charlie | 90 | t1 | Bob | 80 |
| +U | 1 | t3 | Charlie | 110 | t1 | Alice | 100 |
| +U | 2 | t1 | Alice | 100 | t2 | Charlie | 90 |

This format is also used by CDC systems like [Debezium](https://debezium.io/) and as an internal data representation in [Arroyo](https://github.com/ArroyoSystems/arroyo).

This format is the most "informative" one, as it differentiates retractions from corrections, and provides access to both the new and the previous state within one event. The drawback is that it significantly impacts the schema to essentially allow carrying two events (old and new) in one record.

#### Changelog Stream (two events)

The Apache Flink's "changelog streams" variant is using append `+A`, retract `-R`, update-before `-U`, and update-after `+U` operations. Here, the "update-before" events carry the previous values of the record about to be updated and "update-after" events carry the new values, with the restriction that these events must always appear side by side and in order.

| op | place | match\_time | player\_name | score |
| -: | -: | -: | -: | -: |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| -U | 2 | t1 | Bob | 80 |
| +U | 2 | t2 | Charlie | 90 |
| -U | 1 | t1 | Alice | 100 |
| +U | 1 | t3 | Charlie | 110 |
| -U | 2 | t2 | Charlie | 90 |
| +U | 2 | t1 | Alice | 100 |

By splitting the update operation in two events this format does not require extending the schema with multiple columns.

Due to the restriction that update before/after events appear side by side, this format can be easily converted into single-event changelog stream form upon reading.

### Conclusion

Considering all the above we decide to:

* Use **Two-event Changelog Stream format** as the most complete and the least intrusive format for representing corrections and retractions
  * Retract and Upsert streams can still be supported as subsets of the Changelog Stream model
* Extend the set of standard columns with `op` column to define the operation
* Avoid the use of the word `update` and favor the word `correction` to further distant ourselves from the CDC and CRUD mentality
* Generalize `MergeSchema::Snapshot` to use this column.

## Reference-level explanation

The set of system columns will be extended with `op` column:

* Arrow type: `uint8`
* Parquet type: `INT32`
* Recommended Parquet encoding: `RLE_DICTIONARY`

| value | operation | code |
| :-: | :-: | :-: |
| 0 | `append` | `+A` |
| 1 | `retract` | `-R` |
| 2 | `correct-from` | `-C` |
| 3 | `correct-to` | `+C` |

`DatasetVocabulary` schema will be updated to include `operationTypeColumn`.

`MergeStrategy::Snapshot` behavior will be modified as follows:

* The `obsv` column will be removed in favor of `op` column
* Users will no longer be able to specify custom vocabulary for observations
* Strategy will emit changelog-compliant stream, i.e. will emit two correction events (`-C/+C`) instead of one (`U`) as previously

## Compatibility

This change will be executed as part of the backwards compatibility breaking changes.

## Drawbacks

### More columns

Additional system column may be hard on user eyes. In datasets where retractions/corrections don't appear or rare UI layer may optimize the presentation to exclude them, or find different approaches to visualize them.

### More complex CDC

Previously `MergeStrategy::Snapshot` produced an [upsert stream](#upsert-stream). This resulted in a short diff, but to reconstruct state from an upsert stream one needs to know the primary key of the events, which is currently known only to the merge strategy.

Migration to [changelog stream](#changelog-stream-two-events) format will produce more records and slow down the CDC operation, but make stream more versatile, as it will be possible to feed it directly into engines without propagating the knowledge of primary keys. We therefore decide to accept the associated costs.

### Affect on current batch engines

Some ODF engine implementations, like Kamu Spark and DataFusion engines, are operating in batch mode. This is a transitional measure, and they explicitly warn users that they are not complete ODF engine implementations and should be used only for `map` / `filter` style queries. Those engines often reorder rows due to their parallel nature and resort to re-sorting records by `event_time` after processing. While this was mostly fine before, with these changes the event order becomes very important, and in the presence of corrections and retractions order cannot be easily restored by re-sorting, as that would require some kind of primary key which not every dataset has.

We accept this added complexity and recommend that transitional batch-mode engines:

* preserve the order of records during processing (e.g. by coalescing inputs into one partition)
* propagate operation type column as-is, allowing `map` / `filter` operations to act identically on both append and retract/correct records, resulting in a valid changelog stream.

## Alternatives

* Treat retractions and corrections as completely separate data blocks.
  * Given how frequent they can occur in some situations - it's best to keep them as part of the core data model.

## Prior art

* **Apache Flink**:
  * [RowKind type](https://github.com/apache/flink/blob/f7ababa8f0bab87267b101efe2ecc77b700c2222/flink-core/src/main/java/org/apache/flink/types/RowKind.java) that is used to represent a changelog stream uses:
    * `+I INSERT` - for inserts
    * `-U UPDATE_BEFORE` - for updates to carry the previous value that needs to be retracted first (must occur together with `UPDATE_AFTER`)
    * `+U UPDATE_AFTER` - for updates to carry the new value
    * `-D DELETE` - for retractions
  * [Versioned Tables](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/concepts/versioned_tables/)
  * [Top-N](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/topn/) and [Window Top-N](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-topn/) examples in Flink's documentation
  * [Table to Stream Conversion](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/concepts/dynamic_tables/#table-to-stream-conversion) docs explain three types of streams :
    * append-only
    * retract
    * upsert (requires unique key)
  * See also [ChangelogMode type in table connector](https://github.com/apache/flink/blob/1c67cccd2fdd6c674a38e0c26fe990e1dd7b62ae/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/ChangelogMode.java#L36)
* **Debezium**:
  * [Format description](https://debezium.io/documentation/reference/stable/tutorial.html) uses:
    * `c` - for create / insert
    * `u` - for update
    * `d` - for delete
    * `r` - for read (in the case of a snapshot)
  * Flink docs [call out](https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/formats/debezium/) that they currently don't support merging their `UPDATE_BEFORE` and `UPDATE_AFTER` messages into a single Debezium update message
* **Arroyo**:
  * [UpdatingData type](https://github.com/ArroyoSystems/arroyo/blob/master/arroyo-types/src/lib.rs#L469)
    * `Retract(T)`
    * `Update { old: T, new: T }`
    * `Append(T)`
* **Apache Spark**:
  * [Structured Streaming](https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html) only supports append and update modes and does not support retractions
* **Differential Dataflow / Materialize**:
  * Seems to only support retraction model in the form of `(data, timestamp, diff)` records, where `diff` is a record count difference (`1` for append, and `-1` for retraction) [\[1\]](https://github.com/MaterializeInc/materialize/blob/ea9338a1d007bf594bfb1a8d068095cd64090666/src/catalog/src/durable/impls/persist/state_update.rs#L53C52-L53C67) [\[2\]](https://materialize.com/docs/integrations/python/#stream) [\[3\]](https://materialize.com/blog/power-user/)

## Unresolved questions

Ideally, every retraction/correction in a dataset can be easily associated back with the original event that is being retracted or corrected. One could imagine this working via `offset` system column, that uniquely identifies all records in a dataset.

Such association would allow:

* Fast navigation between retractions/corrections and original records
* Fast lookups of whether a particular range of records was affected by any retractions/corrections in the future
  * e.g. for generating warnings during `AS OF` queries or about stale notebooks

In practice this may be hard to implement. A record identifier such as `offset` would need to be propagated through all streaming queries. We wouldn't want to leave such a delicate and error-prone detail to the user, so we would need to analyze and dynamically rewrite queries to add offset propagation. And since the only thing that knows how to interpret engine's dialect is the engine itself - this rewrite would need to be implemented individually by each engine.

At this stage of ODF development we decided that introducing query rewrite would be too costly, but we will consider it in the future.

Additionally, query rewrites and associations between records are already a part of the vision for ODF's **fine-grain provenance**, and must be designed together, holistically.

## Future possibilities

* `MergeStrategy::Snapshot` may allow omitting the `primaryKey`, in which case we would default to **retract stream** CDC scheme
* Retractions/corrections can be associated with the records they affect via `offset` that would allow fast navigation and affected range queries (see [Unresolved questions](#unresolved-questions) above)
* `Top-N` and similar queries may be a better fit for "Projections" feature where a state is maintained in some separate data structure that is optimized for querying
* We may want to support [upsert](#upsert-stream) streams as they can significantly reduce the amount of events needed to reconstruct the state, but they require knowledge of primary key. We should consider whether upsert streams are useful in our core model or more suitable for projection upkeep.


# RFC-016: ODF Schema Format
Source: https://docs.kamu.dev/odf/rfcs/016-odf-schema



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/109?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/109)
[![PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/116?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/116)

**Start Date**: 2025-06-14\
**Published Date**: 2025-08-25

**Specs**:

* [Data Schema](/odf/spec/datasets/data-schema)

**Authors**:

* [Sergii Mikhtoniuk](mailto:smikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [ ] Forwards-compatible

## Summary

This RFC introduces a new human-friendly, extensible schema format for ODF datasets. It replaces the use of Arrow flatbuffer schemas in the `SetDataSchema` event with an explicitly defined logical schema that can include rich metadata, such as column descriptions, logical types, and custom annotations. The new format balances human readability with future extensibility, making it easier to evolve schema definitions and metadata together, and adding support for custom attributes / annotations.

## Motivation

We need an ability to attach extra information to data columns (such as description, ontology, logical types) and dataset as a whole (such as archetypes, visualization option etc.).

We also lack an ability to predefine schema of the dataset in the manifest. We currently only have control over the "read" schema and queries/transformations. The final schema of the dataset always ends up inferred. Ability to define schema is a prerequisite in order to attach any extra information to it.

## Guide-level explanation

Currently in ODF the schema of the actual dataset is never defined in `DatasetSnapshot` manifests - it is **inferred** upon first processing and written in the `SetDataSchema` event using Arrow schema flatbuffer format.

In order for us to enrich the schema with extra information we have two choices:

* Attach extra information to columns via external annotations ([rejected](#rejected-external-annotations))
* Provide ability to explicitly define the schema as part of the manifest and make this definition extensible.

We pick the latter option as it:

* Avoids duplication of column names in several places
* More convenient to see everything related to a column in one place
* Evolution of schema happens along with evolution of annotations - there is no risk of schema and annotations diverging.

To explicitly define the schema we need a format that can be expressed within our manifests.

The first obvious candidate was Apache Arrow which we already use. The only well-defined layout for Arrow schema currently is its [flatbuffers schema](https://github.com/apache/arrow/blob/main/format/Schema.fbs). The [prior art](#prior-art) section links a few tickets that discuss common JSON representation from which we draw inspiration.

We [chose not to use Arrow schema](#rejected-compatibility-with-arrow-flatbuffer-schema) and define our own schema format instead, because:

* The purpose of our schema is different - we want it to describe **logical types** and semantics of column values, rather than **physical layout** of data in files or in memory
* We want schema to be simple, expressive, and easy to define manually in a manifest
* We want to provide extensibility that Arrow schema lacks

## Reference-level explanation

### Flatbuffer schema

The old `SetDataSchema` flatbuffer definition:

```fbs theme={null}
table SetDataSchema {
  // Apache Arrow schema encoded in its native flatbuffers representation.
  schema: [ubyte];
}
```

Will be modified as follows:

```fbs theme={null}
table SetDataSchema {
  // DEPRECATED: Apache Arrow schema encoded in its native flatbuffers representation.
  raw_arrow_schema: [ubyte] (id: 0);
  // Defines the logical schema of the data files that follow this event. Will become a required field after migration.
  schema: DataSchema (id: 1);
}
```

This format is compatible with old event format because flatbuffers only care about field IDs and not their names, so we are free to rename the old `schema` field to `raw_arrow_schema` and introduce new `schema` field. Explicit tags are added as a reminder about the ongoing schema migration.

### Manifest Schema Format

```yaml theme={null}
- kind: SetDataSchema
  schema:  # [1]
    fields:
      - name: offset
        type:
          kind: UInt64
      - name: op
        type:
          kind: UInt8
      - name: system_time
        type:
          kind: Timestamp
          unit: Millisecond
          timezone: UTC
      - name: event_time
        type:
          kind: Timestamp
          unit: Millisecond
          timezone: UTC
        extra:  # [2]
          a.com/a: foo
          b.com/x: bar
      - name: mri_content_hash
        type:
          kind: String
        extra:  # [2]
          opendatafabric.org/description: References the MRI scan data linked to the dataset by its hash  # [3]
          opendatafabric.org/type:  # [4]
            kind: Multihash
      - name: subject
        type:
          kind: Struct
          fields:  # [5]
            - name: id
              type:
                kind: String
              extra:
                opendatafabric.org/description: Subject's unique identity
                opendatafabric.org/type:
                  kind: Did
            - name: gender
              type:
                kind: Option  # [6]
                inner:
                  kind: String
              extra:
                opendatafabric.org/description: Subject's gender
        extra:
          opendatafabric.org/description: Information about the subject
      - name: area_codes
        type:
          kind: List
          itemType:
            kind: String
        extra:
          opendatafabric.org/description: List of body area codes covered by this MRI scan
    extra: # [2]
      c.com/z: baz
```

`[1]` New `schema` field replaces the old `schema` field now containing the ODF schema. Note that schema includes the system columns like `offset`, `op`, and `system_time`.

`[2]` The `extra` field is a container for custom attributes (see [format explanation](#extra-attributes-format) below). Extra attributes will be allowed on the schema level too and work the same as on field (column) level.

`[3]` We introduce `opendatafabric.org/description` attribute to contain field (column) description.

`[4]` The `opendatafabric.org/type` here is used only as an example of possible advanced attribute used to extend logical type beyond core schema - see [RFC-017](/odf/rfcs/017-large-files-linking) for more details.

`[5]` Types recursively work for struct fields.

`[6]` The nullability of fields will be represented by the special `Option` type (unlike on field level in Arrow).

### Extra Attributes Format

Custom attributes is a key-value map.

Keys in the map must be in the format `<domain>/<attribute-name>`. The domain prefix is used to disambiguate and avoid name collisions between different extensions.

Attribute value can be a `string` or any JSON value icluding nested objects.

It will be represented in Flatbuffers as a `string` containing JSON, and natively in YAML.

### Logical Types vs. Encoding

Further we decide that `SetDataSchema` event will be restricted to carry **logical type** information, deprived of encoding details. Currently it is stuck in between - capturing Arrow encoding details, without capturing Parquet encoding. This means we cannot use it as a reliable source of information to enforce similar encoding across all data chunks. At the same time by capturing Arrow encoding details we impose the encoding selection on the read phase.

> Example issue: Old datasets created with earlier versions of Datafusion have `Utf8` and `Binary` types in their schemas. Later versions of Datafusion have switch to use `Utf8View` and `BinaryView` to achieve zero-copy reads. But because when reading the files for querying we use schema from metadata to avoid expensive schema unification step - the encoding details from old schema seep into the query phase and invalidate the view optimization forcing Datafusion to use old types that imply contiguous buffer encoding.

This separation ensures that:

* Logical types remain stable across data chunk formats and engines.
* Encoding strategies (compression, dictionary usage, encoding hints) can evolve independently, both at file level and in-memory level.
* Producers and consumers can negotiate encoding during runtime while relying on consistent logical schema contracts.

### Encoding Hints

The custom attributes mechanism can be used in cases when dataset author wants to provide hints about optimal file and in-memory representation of certain columns. These are left out of scope of this RFC.

### ODF Schema from/to Arrow Conversion

It is desirable for us to be able to convert between ODF and Arrow schema back and forth without information loss. Since ODF focuses on logical types and Arrow on in-memory layout - we will again use custom attriibutes mechanism to preserve encoding information in ODF schema. The following ecoding hints were identified at this initial stage:

**Arrow type:** `LargeUtf8`

**ODF schema:**

```yaml theme={null}
type:
  kind: String
extra:
  arrow.apache.org/bufferEncoding:
    kind: Contiguous
    offsetBitWidth: 64
```

**Arrow type:** `Utf8View` (analogously for `BinaryView`, `ListView`, `LargeListView`)

**ODF schema:**

```yaml theme={null}
type:
  kind: String
extra:
  arrow.apache.org/bufferEncoding:
    kind: View
    offsetBitWidth: 32
```

**Arrow type:** `Date64` (millisecond-based representation)

**ODF schema:**

```yaml theme={null}
type:
  kind: Date
extra:
  arrow.apache.org/dateEncoding:
    unit: Millisecond
```

**Arrow type:** `Decimal128` (analogously for `Decimal256`)

**ODF schema:**

```yaml theme={null}
type:
  kind: Decimal
extra:
  arrow.apache.org/decimalEncoding:
    bitWidth: 128
```

### Schema in APIs

We decide that when data schema is returned in the APIs it should carry only logical type information and not the encoding, unless the encoding represents the encoding of the data that this API transmits.

## Compatibility

Flatbuffer level:

* Flatbuffer schema is made backwards-compatible.
* New datasets will be created **only** with new `schema` field populated, leaving `raw_arrow_schema` field empty - as at the current stage of evolution we can sacrifice forwards-compatibility
* Full removal of `raw_arrow_schema` will be considered in later versions.

Manifest level:

* YAML manifests are backwards-compatible as `SetDataSchema` event never appeared in them before.
* When new `schema` will appear in dataset manifests - only the new tooling will be able to handle it.

### Field optionality during schema upgrades

Field optionality (nullability) deserves a special note. Arrow currently has very poor control over nullability of fields, and implementations had to ignore nullability when checking for compatibility between schemas. With explicit ODF schemas we would like to pave the path towards enforcing field optionality and making schemas that differ in optionality be considered incompatible. To achieve this we special-case the metadata validation logic: the first time a schema is converted from `raw_arrow_schema` to ODF `schema` the schema equivalence check should ignore differences in field optionality. This will allow dataset authors to define schemas they actually expect, rather than propagate often incorrect nullability inference from Arrow.

## Drawbacks

* It requires changing `SetDataSchema` event to store ODF schemas instead of nested [Arrow Flatbuffers](https://github.com/apache/arrow/blob/main/format/Schema.fbs) (see [RFC-010](/odf/rfcs/010-data-schema-in-metadata))

* At present users would have to define the schema for **system fields** (`offset, op, system_time, event_time`) which is verbose and error-prone. We believe that this extra burden can be resolved by providing better automatic schema inference in the tooling.

* This approach is **all-or-nothing**. If user wants to annotate just a few columns - they will need to specify the entire schema. We believe that this extra burden can be resolved by providing better automatic schema inference in the tooling.

## Rationale and alternatives

### Rejected: External annotations

**Idea:** We could continue using `SetDataSchema` event to store serialized schema, while using some other event to add annotations to schema and columns.

Example:

```yaml theme={null}
- kind: SetInfo
  description: Open dataset of MRI scans performed by the X medical lab at Y university
  keywords:
    - MRI
    - Healthcare
  columns:
    - name: event_time
      description: Time when the MRI scan was taken
      a.com/a: foo
    - name: content_hash
      type: Utf8
      description: References the MRI scan data linked to the dataset by its hash
      logicalType:
        kind: ObjectLink
        inner:
          kind: Multihash
    - name: subject
      type: Struct
      fileds:
        - name: id
          logicalType: DID
          description: Subject identifier
        - name: gender
          description: Subject's gender
  a.com/a: foo
  b.com/b:
    $value: baz
    mustUnderstand: true
```

**Reasoning:**

* External annotations allow to only annotate the desired parts of schema without necessary declaring all columns

* This however introduces a chance of them going out of sync and the need for validation

* We already saw the need for the ability to define output schema upfront in datasets and need the human-friendly representation regardless

* Pros:
  * Annotation can be partial
  * Keeps annotation process separate from schema changes

* Cons:
  * Duplication if user has to both define a schema and annotate it
  * Introduces two potentially conflicting sources of truth
    * Will need to validate that annotations refer to columns that exist
    * Need to handle potential desync when schema evolves
  * Does not address inability of user to define schema manually

### Rejected: Compatibility with Arrow Flatbuffer schema

**Idea:**

* Create ODF schemas for Arrow Schema that match the existing flatbuffer-serialized schema in `SetDataSchema` event
* Introduce new extension fields for annotations as part of the Arrow schema object, offsetting their serialization tags to allow for Arrow schema evolution without interfering with our extensions
* New schema will both be compatible with Arrow in flatbuffers and flatbuffer-JSON serialized form

**Reasoning:**
Binary compatibility is difficult to achieve and brittle:

* Due to flatbuffer design `nested: [ubyte]` where data is a separately-serialized `Nested` table is serialized differently from `nested: Nested`, making it not possible to easily match the old binary layout
* Due to Arrow flatbuffers still actively evolving and sometimes breaking compatibility we would have to closely monitoring the upstream changes
* And due to flatbuffers requiring serialization IDs to be sequential, offsetting our extensions would require ugly dummy fields to pad extensions to be sufficiently far enough from base schema

While flatbuffer-JSON is the first candidate for an official "human-friendly" representation, the schema itself makes it frequently overly verbose and cumbersome to use due to multiple design issues. These issues seem severe enough that Rust implementation defined types that significantly differ from the Arrow flatbuffer layout already.

Few prominent design issues:

* The `children` field appears on the `Field` table, instead of being part of a `Type::Struct` variant
* A single-item `children: [Field]` is used to define the type of elements in a `List` elements, instead of item type being a part of `Type::List` variant
* Because of this multi-purpose use the `name` and `nullable` fields are also made optional
* Integer types in Arrow flatbuffer schema are structured as `Int { bitWidth: 64, is_signed: false }` which is much more verbose than `UInt64`
* There are multiple factors that contribute to combinatorial explosion of types:
  * `Large*` types only differ by using 64-bit offset encoding
  * `View*` types (prefix or ["german-style" strings](https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/)) are fundamentally an encoding concern, not a logical type
  * `RunEndEncoded` is similarly an encoding concern orthogonal to the type
  * Even the type `Utf8` mixes the fact that it's semantically a "unicode string" and how it is represented

Prior works like Vortex format also [pointed out](https://docs.vortex.dev/concepts/dtypes) the Arrow's lack of distrinction between logical and physical types.

We could introduce some hacks to make schemas easier to define manually, but this would sill mean that the verbose low-level structure is what users would see in our APIs.

We anticipate many more evolutions to the schema:

* Supporting different column-wise compressions schemes
* Supporting file, column, and page-wise encryption

## Prior art

* [Arrow Flatbuffers Schema](https://github.com/apache/arrow/blob/56c0e2f508fdc5137d6734b406634386f9284a52/format/Schema.fbs)
* Arrow issues related to standard JSON representation
  * [https://github.com/apache/arrow/issues/13803](https://github.com/apache/arrow/issues/13803)
  * [https://github.com/apache/arrow/issues/25078](https://github.com/apache/arrow/issues/25078)
  * [https://github.com/apache/arrow/issues/34898](https://github.com/apache/arrow/issues/34898)
* [Logical vs. Physical data types in Vortex](https://docs.vortex.dev/concepts/dtypes)
* [Arrow View types introduction blog post](https://datafusion.apache.org/blog/2024/09/13/string-view-german-style-strings-part-1/)
* [Apache Iceberg Schema](https://iceberg.apache.org/spec/#schemas-and-data-types)
* [JSON-LD](https://www.w3.org/TR/json-ld11/)

## Unresolved questions

### System Columns

The `SetDataSchema` event was meant to represent the full dataset schema, including the system columns.

Requiring system columns when defining schema manually adds a lot of boilerplate to manifest-based schema definitions.

It's unclear yet what is the best way to resolve this issue:

* Should system fields like `offset`, `op`, `system_time` and `event_time` be auto-populated?
* What about `event_time` that often comes from ingested data, can vary in type and renamed?

### Logical Types Evolution

It's not yet fully clear what approach we should take when it comes to expanding the set of logical types:

* Will types like `Multihash` and `Did` ever make it into core schema?
* When should this happen - when every implementation and every engine supports them via custom attributes?
* Can we incorporate new types into the core as long as they provide a fallback? e.g. allowing `Multihash` and `Did` to fall back to `String` type?
* Will semantics of new logical type be governed by ODF schema?

## Future possibilities

### ODF schema in ingest sources

Curretly our push and polling source definitions rely on DDL-style schema. For consistency we should allow defining source schemas in ODF format as well. This is left out of scope of this RFC.

### Compressed enum format

Basic types in new schema will be quite verbose to spell out, e.g.:

```yaml theme={null}
- name: offset
  type:
    kind: UInt64
```

We could allow enum variants to be shortened to just a string when they don't specify any additional fields, shortening the above to:

```yaml theme={null}
- name: offset
  type: UInt64
```

This would be similar to `@value` contracted/expanded syntax in JSON-LD.

**NOTE:** Compressed enum format should be **input-only features**, i.e. they would only be applied when reading YAML manifests. The full expanded form would be used in flatbuffers.

### Replacing `SetVocabulary` event

The `SetVocabulary` event can be potentially replaced with annotations on system columns (e.g. `opendatafabric.org/systemColumn: offset`). This could be a way to allow user skip defining system columns in the schema - unless they are specified and tagged explicitly tooling can insert them automatically.

### JSON-LD extensibility

We want the set of schema and column annotations to be fully extensible. Core ODF annotations will be a part of Flatbuffer schema and will be efficient to store, but the reset can be essentially treated as `additionalProperties: true` and store them as arbintrary data. Using JSON-LD's expansion and compation mechanisms we can blur the difference between these for library users.


# RFC-017: Large Files Linking
Source: https://docs.kamu.dev/odf/rfcs/017-large-files-linking



[![Issue](https://img.shields.io/github/issues/detail/state/kamu-data/open-data-fabric/108?label=Issue)](https://github.com/kamu-data/open-data-fabric/issues/108)
[![PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/116?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/116)

**Start Date**: 2025-06-14\
**Published Date**: 2025-08-25

**Specs**:

* [Storage Layout](/odf/spec/datasets/storage-layout)

**Authors**:

* [Sergii Mikhtoniuk](mailto:smikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [x] Forwards-compatible

## Summary

This RFC introduces a new an experimental way to link large binary objects like images, videos, and documents to the dataset.

## Motivation

While Parquet format supports large binary columns, sometimes it's sub-efficient to keep binary data within the structured records of a dataset. Resons may include:

* Downloading the core dataset much faster to query and filter it before deciding which of the large binaries to download
* Storing large binaries in a separate storage from the dataset while maintaining **tamper-proof** properties *(e.g. fast storage for dataset, glacier/tape for infrequently accessed binaries)*
* **Resumable uploads / downloads** of large binaries
* **Streaming** binary data directly from storage without proxying data through a server *(e.g. for image, video, and document previews)*.

> NOTE: For streaming, while it is technically possible to obtain a byte range of a binary value withing a Parquet file and use `Range: bytes=start-end` header to access it - many preview systems take only a URL and don't support streaming within a specified byte range, or the value in Parquet may be in the middle of a compressed chunk requiring special handling.

## Guide-level explanation

Considering the above, we decided to provide an ability to attach external binaries to ODF datasets.

In order to preserve tamper-proof properties of the ODF datasets we refer to external binaries only by their hashes. This ensures that file cannot be modified without a corresponding change in the dataset and that dataset's `head` block hash acts as a Merkle root of the dataset and linked objects.

The links (hashes) of the external binaries will be specified in data columns (see [rejected alternatives](#rejected-linking-binaries-via-metadata-chain)). Metadata chain will only contain summary information that allows to count and calculate the size of linked objects.

To mark a column that stores links to external objects we will introduce a new `ObjectLink` logical type that initially will be in the `multihash` format. During this early stage we don't have enough confidence in this approach to make `ObjectLink` and `Multihash` part of the core ODF schema, so we will first introduce them via extra attributes mechanism.

Initially we will only support linked objects that are stored alongside (embedded) in a dataset, but may expand it in future towards sharing objects between multiple datasets and use of separate storage.

## Reference-level explanation

Example of defining a column using extended `ObjectLink` type:

```yaml theme={null}
- name: content_hash
  type:
    kind: String
  extra:
    opendatafabric.org/type:
      kind: ObjectLink
      linkType:
        kind: Multihash
```

Linked object data will be stored in `/data/<multihash>` section of the dataset, together with the data chunks.

Example schema of a dataset that links to externally-stored MRI images:

```yaml theme={null}
fields:
- name: offset
  type:
    kind: UInt64
- name: op
  type:
    kind: UInt8
- name: system_time
  type:
    kind: Timestamp
    unit: Millisecond
    timezone: UTC
- name: event_time
  type:
    kind: Timestamp
    unit: Millisecond
    timezone: UTC
- name: mri_content_hash
  type:
    kind: String
  extra:
    opendatafabric.org/type:  # [1]
      kind: ObjectLink  # [2]
      linkType:
        kind: Multihash  # [3]
- name: patient_id
  type:
    kind: String
  extra:
    opendatafabric.org/type:
      kind: Did  # [4]
```

`[1]` New experimental attribute `opendatafabric.org/type` that defines an extended set of field `Type`s
`[2]` New extended logical type `ObjectLink` signifies that the value references an external object. The mandatory `linkType` property defines the type of the link.
`[3]` New extended logical type `Multihash` signifies a `String` in a self-describing [multihash](https://github.com/multiformats/multihash) format.
`[4]` Same extended type mechanism is used for `Did` type.

The `AddData` event will be extended with `linkedObjects` summary:

```yaml theme={null}
event:
  kind: AddData
  prevOffset: 2165
  newData:
    logicalHash: f9680c001200456651f4be2a8c1a2404c89c6356ce875969d69d93333f5ae97155ff996c4a7
    physicalHash: f162014678f832e823c1429eae7d178e0af1957291372b15171414976c04223026581
    offsetInterval:
      start: 2166
      end: 6644
    size: 554005
  newWatermark: 2022-08-12T03:41:37Z
  extra:
    opendatafabric.org/linkedObjects:  # [5]
      numObjectsNaive: 1123  # [6]
      sizeNaive: 100500  # [6]
```

`[5]` New `opendatafabric.org/linkedObjects` custom attribute contains the summary section to understand how many external objects were associated with a certain `AddData` event as well as their total size from metadata only, without querying the individual Parquet data chunks.
`[6]` The object count and total size are labeled "naive" as they do NOT account for possible duplicates (see [notes](#accounting-for-duplicates-in-size-calculation)).

### Impact on existing functionality

To properly support this extension, implementations will need to:

* Allow upload/download of opaque binary data in the `/data` section of the dataset
* Update ingest procedure to ensure referrential integrity - that hashes that appear in such columns have corresponding data objects uploaded
* Update compaction procedures to combine linked object summaries
* Update smart and simple transfer protocols with an additional stage that download/uploads the external data chunks.
* Update dataset data size summaries to separately mention the number of linked objects and their total size.

## Compatibility

The change is backwards and forwards compatible in a sense that introduction of the new logical type and extending the `AddData` event will not break old clients - they will be able to work with updated datasets, although they will not transport the external objects along with them.

## Drawbacks

## Rationale and alternatives

### Rejected: Linking binaries via metadata chain

Initial idea was to introduce a special event or to extend `AddData` to allow linking the binaries. This would allow to identify which binaries are linked to the dataset using only the metadata chain.

This however introduces many problems:

* Duplication of information between metadata and data chunks
* Having two "sources of truth" that opens up avenue for dangling references
* Issue of compact storage - metadata was not meant to store a lot of information, but this opens up possibility for it to have millions of linked object entries.

We rejected this path in favor of storing links only in data chunks, even though it also has drawbacks:

* We now need to read the data chunks in order to see all linked objects complicates the process of scanning

## Prior art

## Unresolved questions

### Linked objects in derivative datasets

It's not clear how linked objects should be treated in derivative datasets. We may want to at least support plain propagation of `ObjectLink` columns. In this case we likely want to reuse the data objects from the input datasets instead of duplicating it to derivative dataset. This entails cross-dataset content links which may complicate size computation and grabage collection.

## Future possibilities

### External storage support

While the initial implementation aims to store large files as part of the dataset itself, we can extend this mechanism in future to support external storages for large objects.

Ideally the choice of storage could vary and be selected on per-node basis, e.g. one node may keep all data together, while another keep objects in a separate cheaper bucket. This means configuration of where liked objects are stored should be separate from `ObjectLink` column definition.

IPFS case is tricky as CID != hash of data - it can vary due to slicing settings. We likely want to store reference that represent content hash independent of the physical file layout of the object, which will require additional layer of mapping between hash references and CIDs.

### Accounting for duplicates in size calculation

We chose to populate metadata with `sizeNaive` as we don't want to require deduplication of references. Deduplication across the whole dataset would involve complex queries that span multiple data chunks which is impractical during ingest stage.

We cannot rely on checking if object store previously contained referenced object during the upload, as this would require complex accumulation of state during the uploads and passing it to ingest. It still would likely to be incorrect in presence of different merge strategies, retractions, compactions, and purging of history.

Even if we performed full deduplication within a dataset during every ingest, depending on the storage used by a node additional deduplication may happen on the storage level (e.g. across multiple datasets), so the size calculation may still over-estimate the footprint.

We therefore decide to keep "naive" size counting as a very rough estimate and leave it up to individual node implementations to perform exact size calculations, e.g. in a heavy background process, if desired.


# RFC-018: Infrastructure-as-Code Resource Framework
Source: https://docs.kamu.dev/odf/rfcs/018-iac-resource-framework



[![PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/126?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/126)

**Start Date**: 2025-06-14\
**Published Date**: 2026-06-28

**Specs**:

* [Resource Framework](/odf/spec/automation/resource-framework)

**Authors**:

* [Sergiy Zaychenko](mailto:sergiy.zaychenko@kamu.dev), [Kamu](https://kamu.dev)
* [Sergii Mikhtoniuk](mailto:smikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [ ] Forwards-compatible

## Summary

This RFC proposes a new Open Data Fabric manifest format and a set of resource types aiming to re-center ODF around the declarative approach for defining the desired state of the system, achieve better functional decomposition of the system, and provide a better framework for extensibility.

## Table of Contents

* [Motivation](#motivation)
* [General Direction](#general-direction)
* [Terminology](#terminology)
* [Proposal](#proposal)
  * [Resource Manifests](#resource-manifests)
    * [Type Identification](#type-identification)
    * [Versioning](#versioning)
    * [Multi-tenancy](#multi-tenancy)
    * [Identity](#identity)
      * [IDs vs. DIDs](#ids-vs-dids)
    * [Labels & Annotations](#labels--annotations)
    * [References](#references)
    * [Typed References](#typed-references)
    * [Reference resolution](#reference-resolution)
    * [Selectors](#selectors)
    * [Ownership](#ownership)
    * [Generations](#generations)
    * [Status](#status)
  * [Authorization](#authorization)
  * [Resource Application](#resource-application)
  * [APIs](#apis)
    * [Current state of ODF APIs](#current-state-of-odf-apis)
    * [REST API strategy](#rest-api-strategy)
    * [GraphQL strategy](#graphql-strategy)
* [Compatibility](#compatibility)
* [Prior art](#prior-art)
  * [Kubernetes Design Notes](#kubernetes-design-notes)
    * [Kubernetes API Overview](#kubernetes-api-overview)
* [Rationale and alternatives](#rationale-and-alternatives)
  * [ODF objects as Kubernetes CRDs](#odf-objects-as-kubernetes-crds)
  * [Terraform modules for ODF API](#terraform-modules-for-odf-api)
* [Unresolved questions](#unresolved-questions)
* [Future possibilities](#future-possibilities)
  * [JSON-LD](#json-ld)
  * [Extensibility of union types](#extensibility-of-union-types)
* [Appendix A: Example Manifests](#appendix-a-example-manifests)

# Motivation

1. Open Data Fabric spec started and revolved around the single `DatasetSnapshot` manifest that defines the desired state of dataset metadata. We have used `MetadataEvent` types in the snapshot type to define quite complex features like polling data ingestion and push sources.

   With gained experience, we started to realize that some of these features don’t belong in dataset metadata as they not only don’t contribute to dataset provenance or validity, but may **reveal sensitive information about publisher’s infrastructure** that must remain private. This RFC proposes to move some of this information out of the dataset metadata chain to be kept private and at the same time **make core ODF format spec more lean** and easier to adopt.

2. Reference implementations like Kamu have been adding a lot of new features that are strongly related to reusable data pipelines but clearly don’t belong in the dataset metadata:

   * Flow schedules and configuration
   * Dataset variables and secrets
   * ReBAC permissions

   Because it didn’t make sense to encode these into metadata events, configuration started relying on the growing amount of custom GQL and REST APIs **sidetracking us into imperative configuration**, while ODF’s intention was always to be maximally declarative. This makes configuring the state of the system significantly more challenging, and makes configuration process **non-portable** between ODF implementations.

   With this RFC we want to provide all ODF implementations a single resource configuration framework with a clear way to upstream established resources later into ODF, and correct the course back towards the **declarative Infrastructure-as-Code approach**.

# General Direction

ODF is quickly becoming a ***“Kubernetes for Data”*** - a high-level integration framework for different data formats, compute engines, privacy technologies, and data access APIs.

We are embracing **Infrastructure-as-Code** philosophy:

* Manifest files will be the primary way to instantiate and update all resources in data pipelines
* Configurations can be managed in git
* Configuration will describe "desired state" of the system
* Reconciliation mechanism will be responsible for matching actual state with the desired state

We introduce a **unified model of resource, identity, references, ownership**. We want to provide similar API experience whether you are configuring datasets, flows, variables, or access policies. We want representation to work similarly across different flavors of APIs (REST / GQL) to reduce maintenance burden - adding a new resource type should not require extending the API surface.

# Terminology

We will stick to the following terminology:

* **Resource** - is a managed back-end entity that has an identity and consumes or allocates some compute or storage resources
* **Manifest** - is a declarative configuration (e.g. a YAML file) that describes the desired resource state.
* **Controller** - a process that keeps track of certain manifest types and reconciles them with the state of objects under its control. The purpose of controllers is to constantly move the current resource state towards the desired state described by manifests.

Examples:

* `Dataset` YAML file is a manifest applying which will create a corresponding `Dataset` resource
* `VariableSet` is a high-level manifest applying which will create a set of `Variable` resources
* `Ingress` is a manifest application of which will create a high-level `Ingress` resource. The controller of `Ingress` will then generate and apply lower-level manifests for `ApiEndpoint`, a `Buffer` (e.g. Kafka topic), and a `Source`, which will in turn result in provisioning of corresponding resources.

# Proposal

The proposal will often reference Kubernetes as one of the best examples of the declarative IaC approach. We will draw many parallels with its design and build on its learnings.

## Resource Manifests

In Kubernetes manifests are structured as:

```yaml theme={null}
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: {}
spec: {}
status: {}
```

In ODF we propose a format like this:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers: {}
spec: {}
status: {}
```

Here:

* `$schema` identifies the type of the resource using a resolvable URL that points to the JSON Schema file
  * This replaces separate `apiVersion`/`kind` fields (as in K8s) with a single self-describing identifier
* `headers` contains identity and ownership information
  * `headers` is used instead of `metadata` because the latter is already an overloaded term in ODF and is generic to the point of being meaningless
* `spec` defines the desired state of the resource
* `status` contains information about the current state and the reconciliation process

### Type Identification

We use `$schema` URL to identify resource types:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
```

The `$schema` URL is formatted as `{base-url}/{context}/{version}/{Name}` and carries:

* Controlling organization domain (e.g. `opendatafabric.org`)
* Bounded context (e.g. `config`)
* Version (e.g. `v1alpha1`)
* Resource name (e.g. `SecretSet`)

Many IDEs recognize `$schema` field and automatically fetch the associated JSON schema to provide validation and auto-completion.

Resource schemas will be registered within ODF node and, similarly to Kubernetes CRDs, assigned a **short type name** (e.g. `SecretSet`) that can be used instead of the schema. This concept is very similar to JSON-LD expansion.

### Versioning

Version is part of the `$schema` URL in the form of `v1` or `v1alpha1`.

* Version should NOT be thought of only as manifest schema. It captures both how resource is defined and the semantics of how it behaves, i.e. version may be incremented if resource behavior changes significantly even when the schema stays the same.
* Versions apply to the level of entire bounded context, not an individual resource, so if one domain contains multiple related resources a version bump would apply to all of them.

### Multi-tenancy

Resources can explicitly define which `account` they belong to:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  account:
    name: alice
spec: {}
```

Or using a short form:

```yaml theme={null}
account: alice  # Short form can parse UID, DID, or name
```

See [resource references](#references) for details about referring to accounts.

Unlike Kubernetes that uses RBAC and `namespace`-based isolation - ODF is based on an **account-centric model** that allows complex ownership and access control hierarchies, e.g. teams, organizations, flexible permissions for accounts outside of organizations (see [RFC-020](/odf/rfcs/020-auth)).

### Identity

A manifest file will usually only define the `name`:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
spec: {}
```

Note that resource **names are mutable** (unlike in Kubernetes).

At one point in time the tuple `(Account, ResourceType, ResourceName)` uniquely identifies a resource within an ODF node (see also [references](#references)). There may be multiple resources of different types under one account with the same name.

Upon resource creation ODF node will additionally assign resource a unique `id` (UUID v4):

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  id: 6767a4ee-d74d-436e-84f9-709407869a26
  name: my-secrets
spec: {}
status: {}
```

The `id` is a stable identifier of a resource **within one node**.

Including `id` in the manifest can be used to ensure the manifest applies to the exact needed resource, but sacrifices portability of the manifest across ODF nodes.

When including both `id` and `name` the identity matching will be performed by `id` while the `name` will be synchronized with the manifest, thus allowing you to **rename** resources.

#### IDs vs. DIDs

As resources describe the **desired state** of the system - resource instances are separate from objects that are created to fulfill that state. Thus e.g. a `Dataset` resource is not the same thing as an ODF dataset stored in some S3 bucket. `Dataset` resource can be used to create or configure an ODF dataset, but they are separate objects.

This is why resources like `Dataset` or `User` have both:

* `id` (UUIDv4) that identifies the resource within a node
* and a `did` (W3C DID) that identifies an ODF dataset or a person on a global network

When one ODF node populates an ODF dataset with `did:odf:xxyy` and another node replicates this dataset - they will have two `Dataset` resources with the same `did` but different `id`.

Since DIDs are unique, reference types support **identifying target resources by DIDs of the corresponding objects**, purely for convenience.

### Labels & Annotations

Resources can specify custom labels and annotations. Both are maps of string keys to any JSON values, but only labels get indexed and can be used for querying:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
  labels:
    env: prod
  annotations:
    owner: https://github.com/open-data-fabric
    repo: https://github.com/open-data-fabric/spec
spec: {}
```

Labels and annotations are fully **mutable**.

Label and annotation keys can be either short type names or full URIs:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/Dataset
headers:
  name: my-dataset
  labels:
    # Resolves to https://opendatafabric.org/schemas/datasets/v1alpha1/DatasetKind
    # Anything but Root or Derivative will fail validation
    datasetKind: Root
    did: did:odf:aa..bb
  annotations:
    # Expanded form
    https://opendatafabric.org/schemas/labels/v1/Repo: https://github.com/open-data-fabric/spec
spec:
  kind: Root
  metadata: []
```

Thus every label and annotation has a schema and can be type-checked.

Controllers may contribute their own labels to simplify common filtering scenarios. For example a `Dataset` resource above will automatically get the `datasetKind: Root` label without you needing to specify it manually because it's very common to filter datasets by `datasetKind`.

### References

Resource manifests can link to other resources using **references**, forming a DAG.

Resources can be referenced by:

* ID (unique within a node)
* DID (unique globally)
* Type, name, and the (optional) owning account
  * When account is not specified the name is resolved within the account of the current principal

Example of referencing a `PersistentVolume` by name and owning account:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset
headers:
  name: my-dataset
spec:
  metadata: []
  volume:
    type: PersistentVolume
    name: my-s3-bucket
    account:
      name: my-org
```

Or in short form:

```yaml theme={null}
volume: PersistentVolume:my-org/my-s3-bucket
```

Example of referencing by ID:

```yaml theme={null}
volume:
  type: PersistentVolume
  id: 6767a4ee-d74d-436e-84f9-709407869a26
```

Example of referencing by DID:

```yaml theme={null}
account:
  type: User
  did: did:odf:0xfa..bc
```

Cyclical references are not allowed - this can be enforced by implementations via linters.

Unlike Kubernetes that doesn't specify a common reference format - in ODF all references and selectors will have a common schema and thus can be easily picked up by linters and other automation uniformly, without knowing the specifics of individual resource schemas.

### Typed References

Many contexts may choose to provide extended versions of resource references that:

* Restrict target resources to a specific type (e.g. `DatasetRef`, `VolumeRef`)
* Provide additional features (e.g. ability to reference a sub-path of a secret via `Secret:postgres#password`)

### Reference resolution

When setting up complex resource graphs like ingestion pipelines it's very convenient to reference all components by names, especially when target resources are not created yet and were not assigned an `id`.

When operating complex graphs long-term, however, as resources get renamed and deleted referencing by name may not only become a nuisance, but cause security risks if old names get reused.

This is why ODF nodes will resolve all references to stable IDs upon first apply, while account and name information will remain for annotational purposes:

```yaml theme={null}
volume:
  id: 6767a4ee-d74d-436e-84f9-709407869a26  # Stable reference
  type: PersistentVolume
  name: my-s3-bucket  # Remains only for human readability
  account:
    name: my-org
```

**Dangling references are allowed** - it's up to implementations to warn users about such situations, but:

* It should be possible to create e.g. a `Source` that references a non-existing `Secret` and have it created later
* It should be possible to delete a resource referenced by others

### Selectors

Multiple resources can be referenced at once with **selectors**.

Resources can be referenced in bulk by shared properties like:

* Owning account
* Resource type
* Name pattern
* Label filters

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  name: periodic-compaction
spec:
  target:
    type: Dataset
    name: org.opendatafabric.%  # A SQL LIKE-style wildcard that matches names starting with 'org.opendatafabric.'
    labels:
      datasetKind: Root
      env: prod
  triggers:
    - kind: Cron
      cron: "@daily"
  steps:
    - kind: Compaction
      maxSliceSize: 100MiB
      maxSliceRecords: 10_000
```

Being a superset of reference type, selectors can match singular resources by:

* ID and DID
* Type and a name (without wildcards)

Note however that, unlike references, selectors are not resolved to specific IDs during the apply process. Whether a resource matches a selector is determined repeatedly during controller operations, so resources can start and stop matching selector criteria after the initial application of a resource containing the selector.

### Ownership

When an object is created by another higher-level object it can write the association into the header as `ownerReference`. This creation provenance trail can be used for automatic cascading deletion and garbage collection.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/sources/v1alpha1/Buffer
headers:
  name: buffer-aabbcc
  ownerReferences:
  - id: c27331ce-ce88-4ff9-8c5a-4ce8107cc03f
    name: ingest-f76666445
spec: {}
```

See also:

* [Kubernetes Owners and Dependents](https://kubernetes.io/docs/concepts/overview/working-with-objects/owners-dependents/)

### Generations

Resource reconciliation is an **eventually-consistent** process. While a controller is working to reconcile one version of a resource the desired state may be changed by the user. To reflect this lag, a sequential `generation` number is incremented every time the resource header and spec are updated.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
  generation: 4
  createdAt: 2026-01-01T00:00:00Z
  updatedAt: 2026-01-04T00:00:00Z
spec: {}
status: {}
```

In the `status` section `observedGeneration` can be used to see what generation the controller had a chance to process.

Note that `generation` does not increment on status changes as it is intended to signify changes to the desired state.

Changes to the status are counted separately by `statusGeneration`, which increments every time the status is updated:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
  generation: 4
  statusGeneration: 9
spec: {}
status: {}
```

Together the two headers tell whether anything changed in a resource since it was last read, without comparing its contents.

Status may be updated by several controllers concurrently. A controller can pass the `statusGeneration` it read as a **precondition** of a status update, which is then rejected as a conflict if the status was modified in the meantime. This allows **compare-and-swap** updates, e.g. for a controller that must verify some condition of the status before updating it, without the check and the update racing with other controllers.

### Status

The `status` section of the resource manifest never appears in user-defined manifests. It is maintained by the ODF nodes and writeable only by resource controllers. It is used to provide detailed information about the reconciliation status of the resource.

Controllers may act asynchronously, so a combination of `observedGeneration` and `reconciledGeneration` fields tells which generation of the resource spec main controller have last seen, and which it was able to successfully reconcile.

Example of `Source` resource status where reconciliation attempt for `generation: 2` failed because it links to non-existing secret, leaving the previous generation running:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/source/v1/Source
headers:
  name: api-polling-source
  generation: 2
spec: {}
status:
  phase: Degraded
  observedGeneration: 2
  observedAt: 2026-01-02T00:00:00Z
  reconciledGeneration: 1
  reconciledAt: 2026-01-01T00:00:00Z
  conditions:
    https://opendatafabric.org/schemas/resources/v1/ReconciliationError:
      code: unresolved-reference
      message: "Secret `new-api-key` not found"
```

The `phase` field is populated as follows:

| `generation` | `observedGeneration` | `reconciledGeneration` | `phase` | Meaning |
| :-: | :-: | :-: | :-: | - |
| 1 | — | — | `Pending` | Controller has not seen the resource yet |
| 3 | 2 | 2 | `Pending` | Controller has not seen gen 3 yet |
| 3 | 3 | 2 | `Reconciling` | Controller is actively applying gen 3 |
| 3 | 3 | 3 | `Ready` | Fully reconciled, running current spec |
| 3 | 3 | 2 | `Degraded` | Reconcile of gen 3 failed, but gen 2 is still running |
| 3 | 3 | — | `Failed` | Controller saw gen 3, but never successfully reconciled anything |
| ? | ? | ? | `Deleting` | When `deletionRequestedAt` header is set and controller is expected to clean up |
| ? | \* | \* | `Deleted` | Controller cleaned up the resource and set the `deletedAt` header |

Note that deletion takes precedence over the other phases. Deleting a resource is a two-step process, so that controllers can clean up the objects they created or finish work in progress before the resource disappears.

The `conditions` field provides extended information about the state of a specific resource type. Conditions can be contributed by multiple controllers. They are keyed by schema IDs to disambiguate, avoid name collisions, and provide schema checking.

Conditions contributed by controllers other than the main resource controller should carry their own `observedGeneration` and `reconciledGeneration` fields to reflect that they may lag behind or be ahead of the main controller's `status.observedGeneration`.

## Authorization

The authorization model is defined in [RFC-020](/odf/rfcs/020-auth).

## Resource Application

When applying manifests the following steps take place:

1. Load each manifest in apply batch and validate against its schema
2. Lookup if target resources already exist (by `id` or `name`)
3. Assign `id`s to resources that do not yet exist
4. Resolve all resource references to `id`s of existing resources or those about to be created
5. Reorder resources in depth-first dependency order
6. Pass every resource into `pre-apply` step of its corresponding main controller to:
   1. Perform additional validation
   2. Contribute additional labels (e.g. `datasetKind`)
   3. Re-write parts of the spec (e.g. to encrypt raw secrets into `jwe` before the spec is stored)
7. Resource `generation` is incremented
8. Resource specs are saved into the event store
9. Reconciliation process is initiated asynchronously

## APIs

### Current state of ODF APIs

Current API of the Node evolved as several groups of functionality:

* REST API is composed of semi-overlapping groups like:
  * Simple transfer protocol
  * Smart transfer protocol
  * Data query and commitments
* Standalone APIs
  * GraphQL
  * FlightSQL

<Diagram alt="" />

Our REST API is currently missing the functionality for listing datasets, inspecting metadata, and manipulating other objects like accounts, flows, variables etc - this role is only filled by GraphQL.

As we evolve our APIs we would like:

* REST API to become a superset of GraphQL API
* Minimize the burden of maintaining two APIs (reuse object schemas as much as possible)
* Establish a good versioning strategy

### REST API strategy

We will introduce another core REST API protocol group: **Resource protocol**.

<Diagram alt="" />

Resource protocol will define:

* How to list, create, update, delete, and get the state of various resources in an ODF system
* Top-level **resource addressing scheme** and serve as a **nesting point** for other protocols like simple/smart transfer and data querying

Proposed protocol endpoint scheme:

* The account acted upon by the principal will be reflected in `account` query argument:
  * no argument - same as `?account=`
  * `?account=` - use account from the auth token
  * `?account=opendatafabric.org` - specifies account by name
  * `?account=did:key:...` - specifies account by DID
* Listing: `/<context>/<version>/<type>`
  * `/datasets/v1/dataset`
  * `/auth/v1/group`
* By object ref: `/<context>/<version>/<type>/<id-did-name>`
  * `/config/v1/secret/c27331ce-ce88-4ff9-8c5a-4ce8107cc03f`
  * `/datasets/v1/dataset/did:odf:123..321`
  * `/datasets/v1/dataset/my-dataset` (searches current account only)
* Other HTTP-based protocols: `/<protocol>/...`
  * `/graphql`

### GraphQL strategy

Initially GQL will cover only generic resource operations (apply, rename, delete etc.) while representing the resources as plain JSON scalars.

This approach avoids having a dynamic GQL schema for types that will be registered in ODF node in runtime.

In the future we will consider the benefits of representing ODF resource manifests as fully-featured types and reference graph navigation.

For example a manifest like this one:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset
headers:
  did: did:odf:123..321
  name: foo
  account: sergiimk
spec:
  volume: PersistentVolume:sergiimk/my-s3-bucket
```

Could allow navigation of references as:

```gql theme={null}
datasets {
  byId(id: "did:odf:123..321") {
    spec {
      volume {  # reference becomes navigable
        id
        type
        name
        account {
          name
        }
      }
    }
  }
}
```

# Compatibility

Proposed changes can be introduced in implementations in parallel with existing mechanics, gradually adopting `Dataset` and `Source` resources as replacements for `DatasetSnapshot` and in-metadata polling and push sources.

# Prior art

## Kubernetes Design Notes

* [Kubernetes Architectural Principles](https://github.com/kubernetes/design-proposals-archive/blob/main/architecture/principles.md)
* [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/main/contributors/devel/sig-architecture/api-conventions.md)
* [Generated Object API Reference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#api-overview)
* OpenAPI Spec
  * [Schema link](https://raw.githubusercontent.com/kubernetes/kubernetes/refs/heads/master/api/openapi-spec/swagger.json)
  * [OpenAPI Editor](https://editor-next.swagger.io/)
* [apimachinery](https://github.com/kubernetes/apimachinery/blob/master/pkg/apis/meta/v1/types.go)
* [Kubernetes API concepts](https://kubernetes.io/docs/reference/using-api/api-concepts/)
* [kube.rs](http://kube.rs)
* [https://github.com/Arnavion/k8s-openapi](https://github.com/Arnavion/k8s-openapi)
* [Kubernetes API Groups](https://github.com/kubernetes/design-proposals-archive/blob/main/api-machinery/api-group.md) (api-based versioning instead of resource-based)

### Kubernetes API Overview

Kubernetes provides a unified API that is very extensible and provides a uniform way to work with all object resources.

<Diagram alt="" />

Basic Kubernetes API scheme is:

* `/apis/<group>/<version>/<kind>/<name>`
  * `/apis/apps/v1/deployments`
  * `/apis/apps/v1/deployments/my-deployment`
* `/apis/<group>/<version>/namespaces/<namespace>/<kind>/<name>`
  * `/apis/apps/v1/namespaces/my-namespace/deployments/my-deployment`

Benefits of Kubernetes REST API:

* Proven, mature model
* API-based versioning
* API groups allowing different subsystems to evolve independently

Problems with Kubernetes REST API:

* Namespace is the only unit of multi-tenancy
  * In ODF we aim for more flexible multi-tenancy model based on accounts
* Objects are addressed by names - it’s not possible to use `uid`
  * The names in k8s are immutable, but in ODF they can not only be changed within a node, but also be different across nodes

# Rationale and alternatives

## ODF objects as Kubernetes CRDs

If we were to piggy-back on Kubernetes:

* Pros:
  * Mature system
  * Reusing tools and integrations
* Cons:
  * We would inherit many legacy design choices
  * Namespaces and RBAC don’t provide real multi-tenancy
    * We would like to think multi-tenant / multi-region /multi-cloud
    * See [https://www.kcp.io/](https://www.kcp.io/)
  * Performance concerns in case of millions of objects
  * Excessive coupling that would be hard to undo

Building our own resource system:

* Pros:
  * More flexibility
  * Can natively support fine-grained access control and our vision of multi-tenancy
  * Can freely evolve manifest formats to make them easier to author
* Cons:
  * A lot more work
  * Templating would require re-inventing something like helm + helmfile

## Terraform modules for ODF API

* Pros:
  * Can wrap existing APIs in TF modules without modifications
  * Existing tool for complex dependency management and templating
* Cons:
  * TF feels like a hack to add declarativeness onto services that don’t support it
  * Very few data engineers have TF experience
  * High potential for desync between TF state and actual pipeline state
  * TF state needs to be stored somewhere, requiring more infrastructure on the user’s side
  * Harder to validate and to report useful errors

# Unresolved questions

* DatasetID vs ResourceID
  * should we have both?
  * two types of selectors then?

# Future possibilities

## JSON-LD

We are [considering using JSON-LD](https://github.com/open-data-fabric/open-data-fabric/issues/123) for manifests in future. JSON-LD manifest could look like this:

```yaml theme={null}
"@context": https://opendatafabric.org/core/v1/context.jsonld
"@type": config:SecretSet
"@id": https://cluster.example.com/resources/8f3b2c1a-4d5e-6f7a-8b9c-0d1e2f3a4b5c
headers: {}
spec: {}
status: {}
```

They would provide a standard URI expansion/collapsing mechanism for types and may help with flexible versioning of schema fields.

## Extensibility of union types

Some union types like `TaskSpec` and `Ingress` currently provide a fixed set of variants. This is very convenient for manifest auto-completion, but will restrict implementation from experimenting with other types of tasks and sources.

We should in the future extend those types with an `Ext` variant that allows specifying fully generic values, e.g.:

```yaml theme={null}
tasks:
- kind: Ext
  $schema: https://kamu.dev/schemas/v1/AwesomeTask
  makeEverythingAwesome: true
```

# Appendix A: Example Manifests

Example resource manifests are provided with this RFC in the [`/examples`](https://github.com/open-data-fabric/open-data-fabric/tree/master/examples) directory.

An [entity-relationship diagram](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/erd.svg) is also available to see the interplay of core prototype types.

Note that many resource types are still in **early prototype stage** of maturity and will be refined during the reference implementation.


# RFC-019: IaC Task and Flow System
Source: https://docs.kamu.dev/odf/rfcs/019-iac-task-flow-system



[![PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/131?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/131)

**Start Date**: 2026-09-10\
**Published Date**: 2026-09-22

**Specs**:

* [Tasks and Flows](/odf/spec/automation/tasks-and-flows)

**Authors**:

* [Sergii Mikhtoniuk](mailto:smikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)
* [Sergiy Zaychenko](mailto:sergiy.zaychenko@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [ ] Forwards-compatible

## Summary

This RFC proposes a new set of resources to define and control workload scheduling and execution within ODF nodes.

## Table of Contents

* [Current Prototype Implementation in Kamu](#current-prototype-implementation-in-kamu)
* [Proposed IaC-based System](#proposed-iac-based-system)
  * [Task](#task)
    * [Task Types](#task-types)
  * [`FlowRun`](#flowrun)
  * [`Flow`](#flow)
    * [Target Selector](#target-selector)
  * [`FlowTrigger`](#flowtrigger)
  * [Authorization](#authorization)
* [Appendix A: Flow System Prototype in Kamu](#appendix-a-flow-system-prototype-in-kamu)

## Current Prototype Implementation in Kamu

Kamu has been the testing bed for the initial implementation of the task and flow systems. Current design is presented in [Appendix A](#appendix-a-flow-system-prototype-in-kamu).

The current prototype system has a few design issues:

* Flows are closely coupled with datasets - we'll need them to work with many other types of resources
* It exposes an ever growing GQL API surface that has to be updated when adding new flow types
* It is not extensible and would not allow custom plug-in flow types
* Flows consist of one workload only and cannot chain multiple steps (e.g. compact then GC)

## Proposed IaC-based System

We propose to define a new Tasks & Flows system on the core ODF level that builds on the [Resource Framework](/odf/rfcs/018-iac-resource-framework) and allows defining and scheduling workloads in a generic, extensible way.

### Task

Building in a bottom-up order, we define the `Task` resource. `Task` represents a single unit of work, from intent through execution to its outcome.

`Task` resource `spec` captures the **intent**: the target resource and the operation kind with high-level parameters. This is what a human operator or a `FlowRun` controller writes when creating a task. Spec is stable, human-readable, and never rewritten.

Tasks are executed by **workers**. Every worker specializes in one task type and pulls the next task of that type from the node's **queue**, which schedules and prioritizes tasks. The worker then advances the task through the steps its type requires - e.g. fetching data for an ingest, or elaborating the execution plan of a transform, requesting an engine and dispatching the plan to it - and commits the result. As it progresses, the worker records the state of the task in type-specific status conditions. For example, the **execution plan** - with fully resolved paths, offsets, schemas, and all other inputs - is written into the `TaskPlan` condition.

The `TaskStatus` lifecycle proceeds through the following phases:

* **Pending** — task created; node is validating its inputs and resolving what is needed to schedule it (e.g. the engine a transform uses)
* **Queued** — task is waiting in the queue for a worker
* **Running** — a worker has claimed the task and is executing it, including the commit
* **Finished** — terminal; `TaskOutcome` condition is either `Success`, `Failed`, `NoOp` or `Cancelled`, and the task resource is deleted (see below).

The resource `phase` reflects the work of the **task controller**, whose job is to admit the task: validate its inputs, resolve what is needed to schedule it, and put it into the queue. `phase: Ready` means the task was admitted, while `phase: Failed` means it was rejected - in which case the controller also sets an unrecoverable `Failed` outcome. Execution is reported separately by workers in the `TaskStatus` condition, so an admitted task stays in `phase: Ready` whatever its `TaskStatus` is, until it is deleted.

Example of a task as created by a `FlowRun` controller (spec describes only the intent, no plan yet):

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/tasks/v1alpha1/Task
headers:
  ownerReferences:
    - FlowRun:c27331ce-ce88-4ff9-8c5a-4ce8107cc03f  # ResourceRef of the FlowRun that spawned this task
spec:
  kind: Transform
  target: Dataset:sergiimk/foo  # ResourceRef
status:
  phase: Pending
```

Task resources are **deleted as soon as they finish**, so the existing tasks are exactly the work that is pending, queued, or running. Deleted tasks **stay readable** for a configurable retention period (e.g. 10 days) before being purged. This allows the UI and operators to inspect recent runs.

Example of the same task after planning and execution:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/tasks/v1alpha1/Task
headers:
  ownerReferences:
    - FlowRun:c27331ce-ce88-4ff9-8c5a-4ce8107cc03f  # ResourceRef of the FlowRun that spawned this task
  deletedAt: 2026-09-11T02:23:41Z  # Deleted upon completion, readable until the retention period ends
spec:
  kind: Transform
  target: Dataset:sergiimk/foo  # ResourceRef
status:
  phase: Deleted  # Deleted upon completion; execution is reported in `TaskStatus`
  observedGeneration: 1
  conditions:
    https://opendatafabric.org/schemas/tasks/v1alpha1/TaskStatus: Finished  # Pending / Queued / Running / Finished
    https://opendatafabric.org/schemas/tasks/v1alpha1/TaskPlan:
      kind: TransformPlan
      datasetId: did:odf:fed0..17bf
      systemTime: 2026-09-11T02:22:52Z
      queryInputs:
        - datasetId: did:odf:fed0..685b
          offsetInterval:
            start: 0
            end: 399493
          dataPaths:
            - s3://repo/f162..8a9f/data/0a..fa
      transform:
        kind: Sql
        query: SELECT ... FROM ...
      newDataPath: s3://repo/f162..8a9f/data/...
      newCheckpointPath: s3://repo/f162..8a9f/checkpoint/...
    https://opendatafabric.org/schemas/tasks/v1alpha1/TaskOutcome:
      kind: Success  # Success / Failed / NoOp / Cancelled
      result:
        kind: TransformResult
        newBlockHash: f162..f008
        newOffsetInterval:
          start: 0
          end: 1050
        newWatermark: 2023-04-15T00:00:00Z
```

Note that a task may finish with a `NoOp` outcome and no `TaskPlan` if the worker realizes there is nothing to do (e.g. no new data to process).

A `Failed` outcome describes the error and tells whether it is **recoverable**, i.e. whether retrying the task could succeed. Recoverable failures (e.g. a network error) can be retried by the flow according to its `retryPolicy`, while unrecoverable ones (e.g. an invalid query) would fail the same way again. Type-specific details of the error are carried in `error`:

```yaml theme={null}
https://opendatafabric.org/schemas/tasks/v1alpha1/TaskOutcome:
  kind: Failed
  message: Input dataset was compacted since the last transformation
  recoverable: false
  error:
    kind: InputDatasetCompacted
    inputDataset: Dataset:sergiimk/sensor-temp
```

Deleting a task that has not finished yet **cancels** it when possible. Until its outcome is recorded, a task being deleted remains visible in `phase: Deleting`.

#### Task Types

The following task types will be initially supported.

`Ingest` - fetches data from a source and appends it to a dataset.

```yaml theme={null}
kind: Ingest
target: Dataset:sergiimk/sensor-temp  # optional: defaults to the flow-level target
source: Source:sergiimk/sensor-temp-http  # ResourceRef to the Source resource
targetRecordsPerSlice: 10000  # optional: target number of records to ingest per slice
```

`Transform` - executes transformation of data defined in a derivative dataset.

```yaml theme={null}
kind: Transform
target: Dataset:sergiimk/sensor-temp-daily  # ResourceRef to the derivative dataset
```

`SyncFrom` - pulls new blocks from a remote dataset into a local dataset.

```yaml theme={null}
kind: SyncFrom
target: Dataset:sergiimk/sensor-temp  # optional: defaults to the flow-level target
source:
  url: odf+https://node.example.com/acme/sensor-temp  # scheme selects the transfer protocol
  auth:  # optional
    kind: Bearer
    token: SecretSet:acme-node#accessToken  # ValueRef
force: false  # optional: overwrite the local dataset even if histories have diverged
```

`SyncTo` - pushes new blocks of a local dataset to a remote dataset.

```yaml theme={null}
kind: SyncTo
target: Dataset:sergiimk/sensor-temp  # optional: defaults to the flow-level target
destination:
  url: s3://my-bucket/datasets/sensor-temp/
  auth:
    kind: Aws
    region: us-west-2
    accessKey: SecretSet:my-aws-secrets#accessKey  # ValueRef
    secretKey: SecretSet:my-aws-secrets#secretKey  # ValueRef
force: false  # optional: overwrite the remote dataset even if histories have diverged
createIfNotExists: true  # optional: create the remote dataset if it does not exist
```

Sync endpoints support the following `auth` kinds:

* `Bearer` - passes a token, e.g. an access token of a remote ODF node
* `Aws` - credentials for AWS or an AWS-compatible object storage
* `Headers` - custom request headers with values referencing `VariableSet`s and `SecretSet`s

Note that the execution plan of a sync task carries references to secrets and never their values - these are resolved only during execution.

`Compaction` - compacts data files in matching datasets to improve query performance.

```yaml theme={null}
kind: Compaction
target: Dataset:sergiimk/sensor-temp  # ResourceRef to the dataset to compact
maxSliceSize: 100MiB     # optional: target maximum size of each compacted data slice
maxSliceRecords: 10000   # optional: target maximum number of records per slice
```

`Reset` - moves a block reference of a dataset to an earlier block, discarding the history that follows it.

```yaml theme={null}
kind: Reset
target: Dataset:sergiimk/sensor-temp  # ResourceRef to the dataset to reset
ref: head  # optional: block reference to reset, defaults to `head`
newBlockHash: f162..8a9f  # optional: block the reference will point to, defaults to the `Seed` block
oldBlockHash: f162..f008  # optional: expected current block, task fails if the reference has moved
```

`GarbageCollection` - removes unreferenced data files from matching datasets.

```yaml theme={null}
kind: GarbageCollection
target: Dataset:sergiimk/sensor-temp  # ResourceRef to the dataset to collect garbage from
```

`Verify` - checks dataset metadata for integrity, optionally replaying transformations.

```yaml theme={null}
kind: Verify
target: Dataset:sergiimk/sensor-temp  # optional: defaults to the flow-level target
replayTransform: true  # optional: re-executes transformations to verify reproducibility
```

`WebhookCall` - dispatches a payload to a `WebhookEndpoint` resource.

```yaml theme={null}
kind: WebhookCall
endpoint: WebhookEndpoint:sergiimk/notify-slack  # ResourceRef to the WebhookEndpoint
payload: '{"event": "ingest-complete", "dataset": "{{task.source}}"}'  # optional, supports templating
retryPolicy:                     # optional: overrides the flow-level retry policy for this task
  maxAttempts: 5
  minDelay: 30s
  backoff: Exponential
```

Custom task types can also be specified using a full schema URI as the kind:

```yaml theme={null}
kind: https://acme.com/schemas/tasks/v1/SomeTask
someCustomParam: value
```

### `FlowRun`

`FlowRun` resources are a set of tasks to be executed in a sequence.

Example:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/FlowRun
headers:
  ownerReferences:
    - Flow:f47ac10b-58cc-4372-a567-0e02b2c3d479  # ResourceRef of the Flow that spawned this run
spec:
  target: Dataset:sergiimk/foo
  tasks:
    - kind: Transform   # Implicit name: task-0-transform
    - kind: GarbageCollection  # Implicit name: task-1-garbage-collect
status:
  phase: Ready  # NOTE: FlowRun resource is deleted when the run finishes, like a Task
  conditions:
    # Tracks the overall status and the tasks that were spawned during the execution
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowRunStatus:
      status: Running  # Waiting / Running / Finished
      tasks:
        - name: task-0-transform  # Corresponds to spec.tasks[0]
          task: Task:a1b2c3d4-1111-2222-3333-444444444444  # ResourceHandle
          status: Finished
          outcome:
            kind: Success
          lastUpdatedAt: 2026-09-11T02:00:00Z
        - name: task-1-garbage-collect   # Corresponds to spec.tasks[1]
          task: Task:e5f6a7b8-5555-6666-7777-888888888888  # ResourceHandle
          status: Finished
          outcome:
            kind: Success
          lastUpdatedAt: 2026-09-11T02:02:00Z
    # Explains what led to execution of this flow
    # In case of a retry - the causes of the original run are preserved
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowRunActivationCauses:
      activationCauses:
        - activationTime: 2026-09-11T02:00:00Z
          initiator: system  # AccountHandle
          trigger:  # Copy of the trigger's configuration in the parent Flow
            kind: Manual
      lateActivationCauses: []
    # Links to the previous FlowRun, if this is a retry
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowRunRetry:
      retryOf: FlowRun:9b2e4f1a-3c7d-4e8b-a1f2-6d5e7c8b9a0d  # ResourceHandle
      retryNumber: 2  # Second time retrying (i.e. third run total)
```

The `spec.target` on the `FlowRun` level is used as the default `target` for tasks in `spec.tasks` list to avoid duplication.

Like tasks, `FlowRun` resources are **deleted as soon as the run finishes** and stay readable for the retention period. Deleting a run that has not finished yet cancels it, together with its unfinished tasks, which reference the run in `ownerReferences`.

A run created by the flow controller copies the `serviceAccount` of its `Flow` into its own spec, so changes to the flow affect only future runs, never runs in progress.

> Note: Although `retryOf` and `activationCauses` are immutable and known at `FlowRun` creation, they are part of `status` rather than `spec` because they carry information that can only be reliably set by the controller, not by a user.

### `Flow`

`Flow` resources act as templates for instantiating `FlowRun`s and define triggers that decide when to instantiate them.

Example:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  name: compact-and-gc-roots
spec:
  target:  # Selector matching all Root datasets under `sergiimk`
    type: Dataset
    account: sergiimk
    name: %
    labels:
      datasetKind: Root
  triggers:
    - kind: Event
      events:
        type: dataset.ref.updated  # TODO: Spec for event types / groups
      cooldown: 10min
  tasks:
    - kind: Compaction
      maxSliceSize: 100MiB
      maxSliceRecords: 10000
      onNoOp: Break         # Break / Continue: Break finishes the flow run early without error
      onFailure: Fail       # Fail / Continue
    - kind: GarbageCollection
  retryPolicy:
    maxAttempts: 3
    minDelay: 1min
    backoff: Exponential    # optional: Linear / Exponential, constant delay when omitted
  recentBindingsRetention:  # Controls retention for `FlowStatus.recentBindings`
    maxBindings: 10
  recentRunsRetention:      # Controls retention for `FlowStatus.recentRuns`
    maxRuns: 100            # Also capped by the controller settings
    maxAge: 30d
status:
  phase: Ready
  conditions:
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowStatus:
      status: Active  # Active / Paused
      recentBindings:
        - target: Dataset:sergiimk/foo
          boundAt: 2026-01-01T00:00:00Z
      bindingsTotal: 1
      recentRuns:
        - flowRun: FlowRun:f47ac10b-1111-2222-3333-444444444444  # ResourceHandle
          status: Finished
          outcome: Success
          finishedAt: 2026-09-11T02:00:00Z
          retryOf: FlowRun:9b2e4f1a-5555-6666-7777-888888888888  # ResourceHandle
          retryNumber: 1  # First time retrying (i.e. second run total)
        - flowRun: FlowRun:9b2e4f1a-5555-6666-7777-888888888888  # ResourceHandle
          status: Finished
          outcome: Failed
          finishedAt: 2026-09-10T02:00:00Z
```

The `spec.target` on the `Flow` level is used as the default `target` for triggers in `spec.triggers` list and tasks in `spec.tasks` list to avoid duplication while still allowing overrides (e.g. triggering a flow on dataset `A` based on events in dataset `B`).

#### Target Selector

When a `Flow`'s `target` selector matches a resource, the flow controller sets up triggers for that resource automatically. As resources are created or deleted, the controller subscribes to those events and updates its set of active targets accordingly.

The association between a `Flow` and its matched resources is purely a derived state that the controller maintains. The `FlowStatus.recentBindigs` may reflect last N bindings that were created for debugging purposes. To show the full paginated list of bindings, the flow controller may provide a dedicated API for querying the index, e.g. in REST:

* `GET /flow/v1alpha1/flow/_/inverseSearch?target={ResourceRef}` - list flows by target
* `GET /flow/v1alpha1/flow/{id}/targets` - list targets by flow

### `FlowTrigger`

Flows specify a set of triggers that decide when to instantiate a `FlowRun`.

`FlowRun` stores the trigger configuration that lead to its creation in `activationCauses`.

**Core trigger types**:

`Manual` trigger - reacts to an API call or a UI button press.

```yaml theme={null}
kind: Manual
```

`Schedule` trigger - fires on specified Cron schedule. Guaranteed to fire if node was down when the next tick was supposed to happen. Fires only once for all missed ticks.

```yaml theme={null}
kind: Schedule
cron: "@daily"
```

`Interval` trigger - fires at regular intervals. Guaranteed to fire if node was down when the next tick was supposed to happen. Fires only once for all missed ticks.

```yaml theme={null}
kind: Interval
interval: 15m
```

`Event` trigger - reacts to events on the event bus. This is a very low-level trigger that should be used sparingly.

```yaml theme={null}
kind: Event
target: Dataset:sergiimk/foo
events:
  type: dataset.ref.updated  # Domain event filter (TODO: Spec for event types)
```

`InputsUpdated` trigger - fires when inputs of a derivative dataset have updates.

```yaml theme={null}
kind: InputsUpdated
target: Dataset:sergiimk/bar
minRecordsToAwait: 100  # optional batching
maxAwaitInterval: 1h  # run at least once an hour if minRecordsToAwait have not been reached
```

`Source` trigger - fires when the specified source receives new data.

```yaml theme={null}
kind: Source
source: Source:sensor.temp.http
minRecordsToAwait: 100  # optional batching
maxAwaitInterval: 1h  # run at least once an hour if minRecordsToAwait have not been reached
```

Besides triggers defined in ODF spec custom triggers can also be specified.

```yaml theme={null}
kind: https://acme.com/schemas/tasks/v1/SpecialTrigger
someCustomParam: value
```

The following properties are common across all trigger types:

```yaml theme={null}
enabled: true  # Allows to pause an individual trigger
cooldown: 10m  # Don't fire more often than every 10 minutes (batches multiple activations into one)
cooldownMaxBatch: 100  # Fire without waiting for the cooldown to end after this many activations
```

### Authorization

A task executes with the permissions of a user **principal** who created it.

When a task, flow run, or a flow is defined for an organization they must specify `serviceAccount` property which defines a non-human principal that gets permissions only through explicit policy bindings.

A task created from a `FlowRun` inherits the `serviceAccount` of the run, and `FlowRun` inherits one from `Flow`.

Example:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  account: acme
  name: ingest-sensors
spec:
  serviceAccount: acme/ingest-bot  # AccountRef to an account of service account type
  target: Dataset:acme/sensor-temp
  # ...
```

Details of the authorization mechanism are outside of the scope of this RFC.

## Appendix A: Flow System Prototype in Kamu

The design is roughly captured by the following GQL types:

```rust theme={null}
type Task {
	taskId: TaskID!
	status: TaskStatus!
	cancellationRequested: Boolean!
	outcome: TaskOutcome
	createdAt: DateTime!
	ranAt: DateTime
	cancellationRequestedAt: DateTime
	finishedAt: DateTime
}
enum TaskStatus {
	QUEUED
	RUNNING
	FINISHED
}
union TaskOutcome = TaskOutcomeSuccess | TaskOutcomeFailed | TaskOutcomeCancelled
union TaskFailureReason = TaskFailureReasonGeneral | TaskFailureReasonInputDatasetCompacted | TaskFailureReasonWebhookDeliveryProblem

// When to run the flow
type FlowTrigger {
	paused: Boolean!
	schedule: FlowTriggerScheduleRule
	reactive: FlowTriggerReactiveRule
	stopPolicy: FlowTriggerStopPolicy!
}
union FlowTriggerScheduleRule = TimeDelta | Cron5ComponentExpression
type FlowTriggerReactiveRule {
	forNewData: FlowTriggerBatchingRule!
	forBreakingChange: FlowTriggerBreakingChangeRule!
}

// NEW: Flow
type FlowConfiguration {
	rule: FlowConfigRule!
	retryPolicy: FlowRetryPolicy
}

// Parameters used to plan the task
// NEW: Flow.spec.tasks[].params
type FlowConfigRule = FlowConfigRuleIngest | FlowConfigRuleCompaction | FlowConfigRuleReset
type FlowConfigRuleCompaction {
	maxSliceSize: Int!
	maxSliceRecords: Int!
}
type FlowConfigRuleIngest {
	fetchUncacheable: Boolean!
	fetchNextIteration: Boolean!
}
type FlowConfigRuleReset {
	mode: FlowConfigResetPropagationMode!
	oldHeadHash: Multihash
}

// Whether to retry on failures
// NOTE: Old flows could only execute one thing
// NEW: Flow.spec.retryPolicy
type FlowRetryPolicy {
	maxAttempts: Int!
	minDelay: TimeDelta!
	backoffType: FlowRetryBackoffType!
}

// A single execution (run) of the flow
// NEW: FlowRun
type Flow {
  flowId: FlowID!
  datasetId: DatasetID
  description: FlowDescription!
  status: FlowStatus!
  outcome: FlowOutcome
  timing: FlowTimingRecords!
  taskIds: [TaskID!]!
  history: [FlowEvent!]!
  initiator: Account
  primaryActivationCause: FlowActivationCause!
  startCondition: FlowStartCondition
  configSnapshot: FlowConfigRule
  retryPolicy: FlowRetryPolicy
  relatedTrigger: FlowTrigger
}

enum FlowStatus {
	WAITING
	RUNNING
	RETRYING
	FINISHED
}

// What led to the execution of the flow
// NEW: Flow.spec.triggers
type FlowStartCondition = FlowStartConditionSchedule | 
                          FlowStartConditionThrottling | 
                          FlowStartConditionReactive | 
                          FlowStartConditionExecutor

union FlowActivationCause = FlowActivationCauseManual | 
                            FlowActivationCauseAutoPolling | 
                            FlowActivationCauseDatasetUpdate | 
                            FlowActivationCauseIterationFinished

type DatasetFlowProcess {
	flowType: DatasetFlowType!
	dataset: Dataset!
	summary: FlowProcessSummary!
}

// Summary of last flow runs (for UI cards)
type FlowProcessSummary {
	effectiveState: FlowProcessEffectiveState!
	consecutiveFailures: Int!
	stopPolicy: FlowTriggerStopPolicy!
	lastSuccessAt: DateTime
	lastAttemptAt: DateTime
	lastFailureAt: DateTime
	nextPlannedAt: DateTime
	runningSince: DateTime
	pausedAt: DateTime
	autoStoppedReason: FlowProcessAutoStopReason
	autoStoppedAt: DateTime
}
```


# RFC-020: Authorization Model
Source: https://docs.kamu.dev/odf/rfcs/020-auth



[![PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/133?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/133)

**Start Date**: 2026-09-23\
**Published Date**: 2026-10-12

**Specs**:

* [Authorization](/odf/spec/auth-and-privacy/authorization)

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)
* [Sergiy Zaychenko](mailto:sergiy.zaychenko@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [ ] Forwards-compatible

## Summary

This RFC defines the authorization model for ODF nodes. It specifies how accounts, groups, auth attributes, and policies are expressed within the [Resource Framework](/odf/rfcs/018-iac-resource-framework). The data model is designed to be engine-agnostic. A minimal node can implement simple ownership checks, while a full node can plug in a RBAC / ABAC / ReBAC policy engine.

## Table of Contents

* [Motivation](#motivation)
* [Proposed Solution](#proposed-solution)
  * [Terminology](#terminology)
  * [Auth Maturity Levels](#auth-maturity-levels)
  * [Accounts](#accounts)
  * [Principals](#principals)
  * [Subjects](#subjects)
  * [Groups](#groups)
  * [Attributes](#attributes)
  * [Actions](#actions)
    * [`AddLabel` action](#addlabel-action)
  * [Policies and Policy Bindings](#policies-and-policy-bindings)
    * [Generic policies](#generic-policies)
    * [Engine-specific policies](#engine-specific-policies)
* [Compatibility](#compatibility)
* [Drawbacks](#drawbacks)
* [Rationale and alternatives](#rationale-and-alternatives)
* [Prior art](#prior-art)
* [Unresolved questions](#unresolved-questions)
* [Future possibilities](#future-possibilities)
* [Appendix A: Cedar - Example Schema and Policies](#appendix-a-cedar---example-schema-and-policies)
* [Appendix B: OpenFGA - Example Schema](#appendix-b-openfga---example-schema)
* [Appendix C: Reserved Names](#appendix-c-reserved-names)

## Motivation

ODF nodes host datasets and other resources that belong to different accounts. A flexible, standards-aligned access control model is needed to govern:

* Who can view and manage resources under a given account (control plane)
* Who can decrypt secrets and use signing keys
* Who can read or write the data of a dataset (data plane)
* Whether a dataset is public and who can apply labels to make dataset publicly readable
* Which accounts belong to privileged groups (e.g. node administrators)

The model must be expressive enough to support GitHub-style organization permissions while remaining implementable by minimal nodes that may not have a full policy engine available.

## Proposed Solution

### Terminology

* **Node** - an entire ODF deployment acting under a single identity
* **Resource** - a control plane manifest that describes the desired state and status of a data plane entity
* **Entity** - a data plane object that is materialized from a resource (e.g. an ODF dataset, secret, encryption key)
* **User** - an authenticated account of an individual person
* **Organization** - an account that owns resources on behalf of a group of users but can't authenticate or act itself, so its resources are always accessed through its members
* **Service account** - a non-human identity used by automation (scheduled flows, CI, installed applications); it can act as a principal but doesn't own resources
* **Account** - a general term for a named identity (node, user, organization, service account)
* **Anonymous** - the principal of an unauthenticated request, which has no identity and owns nothing
* **Group** - a named, possibly nested set of principals that policies can be bound to, so that permissions are granted to all members at once
* **Principal** - the identity performing an action in an authorization request, established through authentication (a user, service account, node, or anonymous)
* **Subject** - the party a grant is given to in a policy binding or relation: a principal or a group
* **Action** - an operation the principal attempts to perform, identified by a schema URI (e.g. `View`, `Read`, `Decrypt`)
* **Object** - the target of an action: a resource, an entity, or a resource type (`TypeUri`) for actions like `Create` and `List` where no instance exists yet

### Auth Maturity Levels

We loosely define several authorization maturity levels that ODF implementations may choose to support:

| Maturity Level | Description |
| - | - |
| **0 - No Auth** | All resources and actions are available to the caller |
| **1 — Personal Ownership** | Support for `User` accounts and access based on ownership only |
| **2 — Policies** | Support for generic `Policy`, per-account `PolicyBinding`, and resource attributes |
| **3 — Groups and Organizations** | Support for `Group` membership, binding policies to groups, `Organization` and `ServiceAccount` accounts |

### Accounts

ODF recognizes the following kinds of accounts.

**Node** — an ODF node:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Node
headers:
  name: node
spec:
  did: did:odf:0xfa..bc
  displayName: Current Node
```

**User** — an individual person who can authenticate and act as a principal:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/User
headers:
  name: alice
spec:
  email: alice@example.com
  displayName: Alice
```

**Organization** — a group identity that owns resources but cannot itself perform actions:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Organization
headers:
  name: acme
spec:
  displayName: Acme Corp
```

Upon creation of an organization, the node automatically creates two groups under its account:

* `Group:acme/admins` — organization administrators, initially containing the principal that created the organization. Members of this group have the same rights over resources owned by the organization as an owner would
* `Group:acme/members` — organization members

These are ordinary groups that can be edited to customize membership (e.g. by nesting team groups). Their names are reserved, so they can't be created or deleted independently of the organization. Whether admins are implicitly considered members is left to implementations.

**ServiceAccount** — a non-human identity used by automation (scheduled flows, CI, installed applications) that can act as a principal but doesn't own resources:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/ServiceAccount
headers:
  account: acme
  name: ingest-prod
spec:
  displayName: Ingest (prod)
```

> **Note:** To avoid any ambiguity or impersonation, node implementations should ensure that names of nodes, users, and organizations are unique regardless of resource type, making it possible to refer to any account only by its name. Service account names don't follow this rule as they are always scoped under the owning account.

### Principals

A principal is the identity performing an action in an authorization request. It is established through authentication and is always a single concrete identity.

| Type | Can act as principal | Can own resources |
| - | - | - |
| `User` | ✅ | ✅ |
| `ServiceAccount` | ✅ | ❌ |
| `Node` | ✅ | ✅ |
| `Organization` | ❌ | ✅ |
| `Anonymous` | ✅ | ❌ |

* `Organization` cannot authenticate - its resources are always accessed through its members.
* `Anonymous` represents unauthenticated requests. It is not a resource and carries no identity.
* `Node` acts as a principal in system tasks and node-to-node communications. When a node executes work that belongs to a user or service account (e.g. a scheduled flow run) it should do so as that principal rather than as itself.

Principals are referenced using `PrincipalRef`, e.g. to specify the service account a flow runs as, or to record the initiator of an operation:

```yaml theme={null}
serviceAccount: ServiceAccount:acme/ingest-prod  # PrincipalRef
```

### Subjects

A subject is the party a grant is given to in a `PolicyBinding`. Unlike a principal, a subject may stand for many principals at once.

Subjects are referenced using `SubjectRef`, which is a union of:

| Kind | Matches | Short form |
| - | - | - |
| `Resource` | A specific `User`, `ServiceAccount`, `Node`, or members of a `Group` | `User:alice`, `Group:acme/engineering` |
| `Authenticated` | Any authenticated principal | `Authenticated` |
| `Anonymous` | Any unauthenticated request | `Anonymous` |
| `Everyone` | Any principal, authenticated or not | `Everyone` |

Short forms expand as follows:

```yaml theme={null}
subject: Group:acme/engineering
# Expands to:
subject:
  kind: Resource
  type: Group
  account: acme
  name: engineering

subject: Authenticated
# Expands to:
subject:
  kind: Authenticated
```

`Authenticated`, `Anonymous`, and `Everyone` can be used in policy bindings, but not as group members - otherwise nesting such a subject into a group would extend the group's permissions to all principals.

### Groups

A `Group` resource represents a named set of subjects. Membership is declared inline in the spec. Groups support nesting — a group may include other groups as members.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Group
headers:
  name: engineering
  account: acme
spec:
  members:
    - User:alice  # SubjectRef
    - User:bob
    - Group:acme/engineering-contractors  # nested group
```

Groups owned by the system `Node` account like `Group:node/admins` can be used to represent node-wide predefined roles.

### Attributes

Some properties of a resource directly affect access control — for example, whether a dataset allows public reads. We expresses these properties using the existing [labels](/odf/rfcs/018-iac-resource-framework#labels--annotations) mechanism.

A label schema that declares `labelProperties.isAuthAttribute: true` signals that any resource carrying that label should have its value materialized into an auth attribute.

Example auth attribute label:

```json theme={null}
{
  "$id": "https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead",
  "$schema": "https://opendatafabric.org/schemas/resources/v1alpha1/meta/ResourceLabel",
  "description": "Controls whether the dataset is readable by any authenticated user.",
  "type": "boolean",
  "labelProperties": {
    "isAuthAttribute": true,
    "resourceTypes": [
      "https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset"
    ]
  }
}
```

It now can be defined like any other resource label:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset
headers:
  name: my-dataset
  labels:
    # Short form - resolved to https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead
    AllowPublicRead: true
    # Full URI form
    https://opendatafabric.org/schemas/datasets/v1alpha1/AllowAnonymousRead: false
spec:
  kind: Root
  metadata: []
```

Key properties:

* **Single authoring surface** — attributes live on the resource they describe, eliminating cross-ownership ambiguity
* **Typed and validated** — the label schema's `type` field is enforced at apply time
* **Indexed** — because auth attributes are labels, they are also queryable in the resource listing API
* **Schema-discoverable** — implementations enumerate label schemas with `isAuthAttribute: true` at startup to know which labels to materialize, without hardcoding a list
* **Auth attributes must be labels, not annotations** — validators reject a resource that places an `isAuthAttribute` label under `annotations`
* **Admission control** — changing a label with `isAuthAttribute: true` may require elevated permissions beyond those needed to update the rest of the resource spec.

We propose to define this initial set of auth attributes:

| Label | Type | Description |
| - | - | - |
| `datasets/v1alpha1/AllowPublicRead` | `boolean` | Readable by any authenticated principal |
| `datasets/v1alpha1/AllowAnonymousRead` | `boolean` | Readable without authentication |

In resource manifest:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset
headers:
  name: bobs-dataset
  account: bob
  labels:
    AllowPublicRead: true
    AllowAnonymousRead: false
spec:
  kind: Root
  metadata: []
```

Labels like `AllowPublicRead: true` may require special permissions to set, as not every maintainer of the dataset should have the power to make data public. Per-label access control is discussed in [`AddLabel` action](#addlabel-action) section.

Auth attributes are propagated from resources to entities they define, e.g. an `AllowPublicRead` attribute on the dataset resource (control plane) will also appear as attribute on ODF dataset entity (data plane).

> **Note:** The effects of `AllowPublicRead` and  `AllowAnonymousRead` can be achieved by defining policies for `Authenticated` or `Anonymous` [subjects](#subjects). Attributes, however, remain the primary way to make datasets public, as they are atomic with the resource manifest and can be efficiently filtered on when listing resources. Bindings to these subjects are better suited for granting other actions publicly, e.g. making a resource manifest viewable by everyone. Since the `BindPolicy` action receives the `subject` in its context, policies can restrict who is allowed to create such bindings.

### Actions

Authorization decisions are made against a set of well-defined actions. Like label schemas, each action is defined by a JSON schema using the `Action` metaschema. The schema URI is the canonical action identifier used in authorization requests and policies, for example `https://opendatafabric.org/schemas/resources/v1alpha1/actions/View`. The schema itself describes the context values needed to authorize the action (e.g. the label being added for `AddLabel`), while `actionProperties.objectTypes` lists the types of objects the action applies to. Every action lists the object types it applies to.

Below is a set of initial actions ODF will specify.

**Resource actions** — operate on resource instances. Exceptions are `Create` and `List`, which operate on a resource type (`TypeUri`) and supply the account under which resources are created or listed in the request context (`targetOwner`):

| Action | Object | Description |
| - | - | - |
| `Create` | `TypeUri` | Create a new resource instance under an account |
| `List` | `TypeUri` | List resource instances of this type under an account |
| `View` | `Resource` | Read the resource manifest (metadata) |
| `Edit` | `Resource` | Modify the resource manifest |
| `Delete` | `Resource` | Delete the resource instance |
| `AddLabel` | `Resource` | Apply a specific label to the resource |
| `RemoveLabel` | `Resource` | Remove a specific label from the resource |
| `BindPolicy` | `Resource` | Create `PolicyBinding` resources that reference this resource as `object` |

**Dataset actions** — operate on dataset instances in the data plane, separate from dataset resources:

| Action | Object | Description |
| - | - | - |
| `Read` | `Dataset` | Read dataset data (query, download) |
| `Write` | `Dataset` | Append new data to the dataset |
| `AlterHistory` | `Dataset` | Rewrite or truncate dataset history |

**Config actions** — operate on configuration entities:

| Action | Object | Description |
| - | - | - |
| `Decrypt` | `SecretSet` | Decrypt a secret from a `SecretSet` |

#### `AddLabel` action

`AddLabel` is an example action that operates on the sub-resource level. It authorizes applying a specific label to a resource. It is distinct from `edit` to allow fine-grained control. A user may have `edit` permission but be restricted from changing auth-sensitive labels like `AllowPublicRead`.

An example of Cedar policy using this action may look like this:

```cedar theme={null}
// Only "acme" organization admins may mark a dataset public
permit(
  principal in odf::auth::Group::"acme/admins",
  action in [odf::resources::Action::"https://opendatafabric.org/schemas/resources/v1alpha1/actions/AddLabel"],
  resource is odf::resources::Resource
) when {
  context.labelType == odf::resources::TypeUri::"https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead" &&
  resource.owner in odf::auth::ResourceOwner::"acme"
};
```

### Policies and Policy Bindings

Beyond resource ownership, `Policy` and `PolicyBinding` resources are the primary mechanism for granting permissions.

A `Policy` resource is a reusable, potentially parameterized grant template. It declares what actions are permitted but may leave `subject` and `object` as slots to be filled in later.

A `PolicyBinding` instantiates a `Policy` by supplying the concrete `subject` and `object`. This separation mirrors the Kubernetes `Role`/`RoleBinding` pattern: a node admin or org admin defines policy templates once; a resource owner creates bindings to apply them to specific resources.

Engine-specific policies that don't declare `subject` and `object` slots are not templates — they take effect on their own without a binding (e.g. a node-wide policy granting all actions to members of `Group:node/admins`).

Creating a `PolicyBinding` is itself an authorized operation — the principal creating it must hold the `BindPolicy` action on the referenced `object`. Resource owners implicitly hold `BindPolicy` on their own resources and can grant it to others, enabling delegated access management without transferring ownership.

A `Policy` resource's `kind` field selects the policy language. ODF spec only defines the `Generic` policy type — an engine-agnostic way to express grants directly in terms of ODF action URIs. Additional kinds like `Cedar` and `Rego` may be added by implementations.

#### Generic policies

A `Generic` policy grants a set of ODF actions on a resource or resource type. It is the portable building block — any node that implements Level 2+ can evaluate it without a specialized policy engine:

```yaml theme={null}
# A reusable "dataset reader" policy template
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Policy
headers:
  name: dataset-reader
  account: node
spec:
  kind: Generic
  actions:
    - https://opendatafabric.org/schemas/resources/v1alpha1/actions/View
    - https://opendatafabric.org/schemas/datasets/v1alpha1/actions/Read
  # subject and object are filled in by PolicyBinding
```

```yaml theme={null}
# Bob grants Alice reader access to his dataset
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/PolicyBinding
headers:
  account: bob
  name: alice-foo-reader
spec:
  policy: Policy:node/dataset-reader
  subject: User:alice  # SubjectRef
  object: Dataset:bob/dataset-foo  # ResourceRef
```

The `object` refers to both the resource and the entity materialized from it, so the policy above grants `View` on the dataset resource and `Read` on the dataset entity.

Higher-level features like dataset roles (`Reader`, `Editor`, `Maintainer`) are not part of the ODF spec — implementations can build them using groups, `Generic` policy templates, and `PolicyBinding` resources.

#### Engine-specific policies

Nodes with a full policy engine may support richer `Policy` kinds that can express attribute-based conditions, org-scoped grants, and custom logic that `Generic` cannot.

Here is an example of a custom Cedar policy template of `acme` organization:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Policy
headers:
  name: dataset-reader-acme
  account: node
spec:
  kind: Cedar
  body: |
    permit(
      principal == ?principal,
      action in [
        odf::resources::Action::"https://opendatafabric.org/schemas/resources/v1alpha1/actions/View",
        odf::datasets::Action::"https://opendatafabric.org/schemas/datasets/v1alpha1/actions/Read"
      ],
      resource == ?resource
    ) when {
      resource.owner in odf::auth::ResourceOwner::"acme"
    };
```

The `?principal` and `?resource` slots above are filled in by the `subject` and `object` of the `PolicyBinding`:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/PolicyBinding
headers:
  account: bob
  name: alice-reader-binding
spec:
  policy: Policy:node/dataset-reader-acme
  subject: User:alice
  object: Dataset:acme/weather
```

Nodes that do not support a given `Policy` kind MUST reject the resource with a clear error rather than silently ignoring it.

## Compatibility

This RFC is backwards-compatible. No existing metadata event schemas are changed. All new resource types are additive, so existing resources remain valid. Nodes that do not implement auth enforcement will reject the new resources as unknown types.

## Drawbacks

* `Policy` bodies for engine-specific kinds are not portable — a Cedar policy means nothing to an OpenFGA node. Interoperability at the policy layer is not a goal. Interoperability at the data layer (accounts, groups, `Generic` policies) is.

* Inline group membership (vs. separate `Member` relation triples) creates write contention if many members are being added concurrently. Acceptable at ODF node scale - will revisit if needed.

## Rationale and alternatives

**Why a separate `ServiceAccount` type instead of a `User` with a flag?**
Service accounts differ from users in ways that matter for authorization: they are scoped under an owning account, cannot own resources, and have no email or interactive login. A separate type lets policies distinguish them explicitly (e.g. `principal is ServiceAccount`) and keeps user-only fields out of their manifests. Installing an application creates a service account under the installing account, so applications don't need a principal type of their own.

**Why are access token scopes not part of the authorization model?**
A token may be restricted to a set of scopes (e.g. read-only access to specific datasets), but scopes can only narrow what the principal is allowed to do. Checking whether an action on an object falls within the token's scopes is a pure function of the token, action, and object that needs no policies or hierarchies, so implementations perform it before policy evaluation. Policies then decide based on the principal's own permissions, so a token can never grant more than its owner has.

**Why are organizations part of the spec while teams and roles are not?**
Teams and roles can be fully expressed with groups, `Generic` policy templates, and bindings, so implementations can build them on top of the spec. Organizations can't: they are a type of resource owner that cannot act on its own, and granting their admins owner rights over all organization resources is not expressible with single-object `Generic` bindings. Since ownership is part of every resource header, owner types must also be consistent between implementations for manifests to be portable.

**Why inline `spec.members` on `Group` instead of separate `Member` relations?**
Simpler to reason about, easier to diff in git, maps directly to auth engine entity parents. The `Member` relation approach carries Zanzibar-scale complexity that ODF nodes don't need. A group's membership is a property of the group, not a separate lifecycle object.

**Why no `RoleAssignment` resource?**
Named dataset roles (Reader, Editor, Maintainer) are a useful UX abstraction but are not universal — different implementations may define different roles for different resource types. Rather than standardizing a role vocabulary in the ODF spec, implementations like Kamu build role UX on top of `Group`, `Generic` policy templates, and `PolicyBinding` resources. The spec stays minimal; the implementation provides the ergonomics.

**Why a `Generic` policy kind instead of just engine-specific kinds?**
Engine-specific policy bodies (Cedar, Rego) are not portable across implementations. `Generic` policies — expressed directly in terms of ODF actions — can be evaluated by any Level 2+ node without a specialized engine, preserving interoperability for the common cases (grant action X to principal Y on resource Z).

**Why `Policy`/`PolicyBinding` instead of a flat ACL?**
Policies and bindings have different lifecycles and different authors. A system admin writes `Policy` templates once; a data owner creates `PolicyBinding` resources to instantiate them per resource. Separating them mirrors the Cedar template/link model and the Kubernetes `Role`/`RoleBinding` pattern.

**Why engine-agnostic instead of mandating Cedar?**
ODF is a specification, not an implementation. Mandating a specific policy engine would exclude implementations that use Rego, custom evaluators, or pure ReBAC engines. The layered model lets each implementation choose what fits its stack while remaining interoperable at the data level.

## Prior art

* [Google Zanzibar](https://research.google/pubs/zanzibar-googles-consistent-global-authorization-system/) — the canonical ReBAC system.
* [OpenFGA](https://openfga.dev/) — open-source ReBAC engine implementing the Zanzibar model.
* [Cedar](https://cedarpolicy.com/en) — open-source policy engine by AWS; used by the reference implementation for engine-specific policies.
* [Open Policy Agent (OPA)](https://www.openpolicyagent.org/) / [Rego](https://www.openpolicyagent.org/docs/latest/policy-language/) — general-purpose policy engine and its declarative language; an alternative engine for nodes that prefer Rego's flexibility over Cedar's formal verification guarantees.
* [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) — `Role` + `RoleBinding` pattern; directly analogous to `Policy` + `PolicyBinding`.
* [GitHub repository permissions](https://docs.github.com/en/get-started/learning-about-github/access-permissions-on-github) — teams (groups) + per-repo role assignments; the UX model ODF targets.
* Other projects: [Auth0 FGA](https://auth0.com/fine-grained-authorization), [Topaz](https://www.topaz.sh/), [StrongDM](https://www.strongdm.com/), [SpiceDB](https://authzed.com/spicedb), [Amazon Verified Permissions](https://aws.amazon.com/verified-permissions/)

## Unresolved questions

* Interplay between this model and decentralized capabilities-based auth (UCAN).
* How should the admission layer enforce that `headers.account` on a `PolicyBinding` resource matches the owner of all referenced objects — reject at admission or flag as a validation error in status?
* Who can author `Policy` resources? Initially only node admins should be allowed to create policies, as an engine-specific policy (e.g. `Cedar`) can grant arbitrary permissions across the node. Delegating policy authoring to e.g. organization admins will be addressed in future RFCs.

## Future possibilities

* **Cross-node relations:** A subject on one ODF node granting access to an object on another, using DIDs for stable cross-node identity.
* **Audit log:** Surface all auth resource changes as events in the event sourcing store for compliance and forensics.
* **Transfer ownership:** A dedicated action and policy mechanism for transferring resource ownership between accounts.
* **Bulk bindings:** Allowing `PolicyBinding` objects to target a resource type within an account (e.g. all `Dataset`s of `acme`) so that organization admins could express org-wide grants with `Generic` policies, without resorting to engine-specific policies.

## Appendix A: Cedar - Example Schema and Policies

Full prototype is available [here](https://github.com/kamu-data/tmp-cedar-prototype).

A schema can be code-generated based on types registered in the system allowing policies to be validated against known resource types and attributes:

> **Note:** For brevity, actions here use short names (e.g. `"read"`). A generated schema would use full action schema URIs (e.g. `"https://opendatafabric.org/schemas/datasets/v1alpha1/actions/Read"`) as in the examples above.

```cedar theme={null}
namespace impl {
    // Base context is implmentation-specific and not part of ODF,
    // but it will be present in all actions as `context.base`.
    // It's a workaround for missing type inheritance in Cedar.
    type Context = {
        ip?: String,
        country_code_a2?: String,
    };
}

namespace odf {
    // Common context type for ODF actions that don't need anything extra
    type Context = {
        base: impl::Context,
    };
}

namespace odf::auth {
    // Abstract type that can own resources
    // User and Organization entities specify this type as parent
    entity ResourceOwner = {
        // Set for organizations to their reserved `{org}/admins` group,
        // whose members get the same rights as the owner
        admins?: odf::auth::Group,
    };

    // Marker for any authenticated principal: User, ServiceAccount, or Node
    // Allows checking `principal in Authenticated::""` instead of creating
    // separate policies for each principal type.
    entity Authenticated;

    // Unauthenticated users are represented as separate type
    // to ensure they are handled in policies explicitly
    entity Anonymous;

    // The node is a principal used for local system tasks and node-to-node communications.
    // When node performs actions on behalf of users and service accounts
    // (e.g. scheduled flow runs) the principal will be the user or service account
    // that task belongs to, i.e. node acts on behalf of the account.
    // The local node may own a few system-wide resources, e.g. the `admin` group.
    entity Node in [ResourceOwner, Authenticated];

    // Authenticated users
    entity User in [ResourceOwner, Group, Authenticated];

    // Organizations can own resources but can't perform actions (not principals)
    entity Organization in [ResourceOwner];

    // Principal for automation that lives under a user or organization
    // used to bind permissions to org-scoped flows and application installations.
    // Can act but does not own any resources.
    entity ServiceAccount in [Group, Authenticated] = {
        owner: odf::auth::ResourceOwner,
    };

    // Groups are a set of users, service accounts or other groups (nesting)
    // used to manage permissions in bulk.
    entity Group in [Group] = {
        owner: odf::auth::ResourceOwner,
    };
}

namespace odf::resources {
    entity TypeUri;

    entity Resource in [TypeUri] = {
        owner: odf::auth::ResourceOwner,

        // All possible auth attributes are accessible via full URIs
        "https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead"?: Bool,
        "https://opendatafabric.org/schemas/datasets/v1alpha1/AllowAnonymousRead"?: Bool,
    };

    action create, list appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::Anonymous,
            odf::auth::ServiceAccount
        ],
        resource: [odf::resources::TypeUri],
        context: odf::resources::ContextCreateList,
    };

    type ContextCreateList = {
        // Owner under which resources are being created / listed
        targetOwner: odf::auth::ResourceOwner,
        base: impl::Context,
    };

    action view, edit, delete, bindPolicy appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::resources::Resource],
        context: odf::Context,
    };

    action addLabel, removeLabel appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::resources::Resource],
        context: odf::resources::ContextAddRemoveLabel,
    };

    type ContextAddRemoveLabel = {
        // Type of the label being added / removed
        labelType: odf::resources::TypeUri,
        // New value of the label when adding or existing value when removing (as stringified json)
        labelValue: String,
        base: impl::Context,
    };
}

namespace odf::datasets {
    // Represents a data plane entity, not the control plane resource
    entity Dataset = {
        owner: odf::auth::ResourceOwner,

        // Relevant attributes are propagated into the data plane
        // as short names allowing for type-checked access
        allowPublicRead?: Bool,
        allowAnonymousRead?: Bool,
    };

    action read, write, alterHistory appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::datasets::Dataset],
        context: odf::Context,
    };
}

namespace odf::config {
    // Represents a data plane entity, not the control plane resource
    entity SecretSet = {
        owner: odf::auth::ResourceOwner,
    };

    action decrypt appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::config::SecretSet],
        context: odf::Context,
    };
}
```

Example policies:

```cedar theme={null}
// System admins can do anything
permit(
    principal in odf::auth::Group::"node/admins",
    action,
    resource,
);

// The local node can do anything. Implementations drop privileges
// by executing work as a user or service account.
permit(
    principal == odf::auth::Node::"${local_node_id}",
    action,
    resource,
);

// Users can create and list resources under their own account
permit(
    principal is odf::auth::User,
    action in [
        odf::resources::Action::"create",
        odf::resources::Action::"list",
    ],
    resource is odf::resources::TypeUri,
) when {
    principal in context.targetOwner
};

// Users can do anything to resources they own
permit(
    principal is odf::auth::User,
    action,
    resource is odf::resources::Resource
) when {
    principal in resource.owner
};

// Users can do anything to data plane entities they own
permit(
    principal is odf::auth::User,
    action,
    resource is ${entity_type}
) when {
    principal in resource.owner
};

// Org admins can do anything to resources owned by the organization
permit(
    principal is odf::auth::User,
    action,
    resource is odf::resources::Resource
) when {
    resource.owner has admins && principal in resource.owner.admins
};

// Public datasets can be read by any authenticated principal (users and service accounts)
permit(
    principal in odf::auth::Authenticated::"",
    action in [odf::datasets::Action::"read"],
    resource is odf::datasets::Dataset
) when {
    resource has allowPublicRead && resource.allowPublicRead
};

// Anonymous users can read datasets that explicitly allow it
permit(
    principal is odf::auth::Anonymous,
    action in [odf::datasets::Action::"read"],
    resource is odf::datasets::Dataset
) when {
    resource has allowPublicRead && resource.allowPublicRead &&
    resource has allowAnonymousRead && resource.allowAnonymousRead
};
```

`PolicyBinding` resources become template-linked policies, with the `?principal` slot filled based on the `SubjectRef` variant:

| `SubjectRef` | Cedar |
| - | - |
| `Resource` (e.g. `Group:acme/engineering`) | Entity UID, e.g. `odf::auth::Group::"acme/engineering"` |
| `Authenticated` | `odf::auth::Authenticated::""` |
| `Anonymous` | `odf::auth::Anonymous::""` |
| `Everyone` | No slot — compiled into a static policy with an unconstrained `principal` |

## Appendix B: OpenFGA - Example Schema

Full prototype is available [here](https://github.com/kamu-data/tmp-openfga-prototype).

> **Note:** This model predates the `ServiceAccount` and `Node` principals and is kept as an illustration of how the model maps onto a ReBAC engine.

Example schema:

```openfga theme={null}
model
  schema 1.1

# Singleton representing the kamu node. Carries the node-wide admin group.
# Users and organizations link to the node.
type node
  relations
    define admin: [group#member]

# An authenticated human user.
type user
  relations
    define node: [node]
    define node_admin: admin from node

# An unauthenticated (anonymous) caller. Kept separate from user so that
# user:* wildcards never accidentally grant anonymous access.
type anonymous

# An organization can own resources but cannot act.
type organization
  relations
    define node: [node]
    define node_admin: admin from node

# A set of users or other groups (supports nesting).
type group
  relations
    define direct_member: [user, group]
    define member: direct_member or member from direct_member

# Resource stores exist under every account per every resource type
# (e.g. `alice/dataset`, `bob/dataset`, `alice/secretset`, `bob/secretset`).
# They allow to express permissions for an entire resource type within
# an account or the whole node.
type resource_store
  relations
    define node_store: [resource_store]
    define owner: [user, organization]
    define lister: [user, group#member]
    define creator: [user, group#member]
    define deleter: [user, group#member]
    define viewer: [user, group#member]
    define editor: [user, group#member]
    define label_assignee: [user with label_type_matches, group#member with label_type_matches]

    define can_create: owner or node_admin from owner or creator or can_create from node_store
    define can_delete: owner or node_admin from owner or deleter or can_delete from node_store
    define can_list: owner or node_admin from owner or lister or can_list from node_store
    define can_view: owner or node_admin from owner or viewer or editor or can_view from node_store
    define can_edit: owner or node_admin from owner or editor or can_edit from node_store
    define can_set_label: owner or node_admin from owner or label_assignee or can_set_label from node_store

# A control-plane resource record (metadata / lifecycle object).
# Linked to a resource_store for org-wide viewer/editor grants and node_admin
# traversal. Per-resource viewer/editor tuples are also supported for individual grants.
type resource
  relations
    define resource_store: [resource_store]
    define owner: [user, organization]
    define viewer: [user, group#member]
    define editor: [user, group#member]

    define can_view: owner or viewer or editor or can_view from resource_store
    define can_edit: owner or editor or can_edit from resource_store
    define can_delete: owner or can_delete from resource_store
    define can_set_label: owner or can_set_label from resource_store

# Permission container for datasets, scoped to node, org, or user.
# Stores link to a parent node_store to inherit node-wide grants.
type dataset_store
  relations
    define node_store: [dataset_store]
    define owner: [user, organization]
    define reader: [user, group#member]
    define writer: [user, group#member]
    define maintainer: [user, group#member]

    define can_read: owner or node_admin from owner or reader or writer or maintainer or can_read from node_store
    define can_write: owner or node_admin from owner or writer or maintainer or can_write from node_store
    define can_alter_history: owner or node_admin from owner or maintainer or can_alter_history from node_store

# A data-plane dataset entity. Permissions are independent of the control-plane
# resource record — a reader role here grants data access only.
type dataset
  relations
    define dataset_store: [dataset_store]
    define owner: [user, organization]
    define reader: [user, group#member]
    define writer: [user, group#member]
    define maintainer: [user, group#member]
    define public_reader: [user:*]       # any authenticated user can read
    define anonymous_reader: [anonymous:*]  # unauthenticated callers can read

    define store_can_read: can_read from dataset_store
    define store_can_write: can_write from dataset_store
    define store_can_alter_history: can_alter_history from dataset_store

    define can_read: owner or reader or writer or maintainer or public_reader or anonymous_reader or store_can_read
    define can_write: owner or writer or maintainer or store_can_write
    define can_alter_history: owner or maintainer or store_can_alter_history

# Permission container for secret sets, scoped to node, org, or user.
# Stores link to a parent node_store to inherit node-wide grants.
type secret_set_store
  relations
    define node_store: [secret_set_store]
    define owner: [user, organization]
    define decryptor: [user, group#member]

    define can_decrypt: node_admin from owner or owner or decryptor or can_decrypt from node_store

# A data-plane secret set entity (e.g. encryption keys).
# Linked to a secret_set_store so that store-level grants (node admin,
# org group) propagate without needing per-secret-set tuples.
type secret_set
  relations
    define owner: [user, organization]
    define decryptor: [user, group#member]
    define secret_set_store: [secret_set_store]

    define can_decrypt: owner or decryptor or can_decrypt from secret_set_store

condition label_type_matches(requested: string, allowed: list<string>) {
  requested in allowed
}
```

## Appendix C: Reserved Names

Names that implementations must reserve and treat consistently.

**Accounts**

The following names cannot be used by `User` or `Organization` accounts, to prevent impersonation of the node or its administrators:

| Name | Reason |
| - | - |
| `node` | Conventional name of the local `Node` account |
| `admin`, `admins`, `root`, `system` | Suggest administrative privileges |
| `authenticated`, `anonymous`, `everyone` | Match subject keywords |

Matching is case-insensitive. Implementations may reserve additional names (e.g. their own product name).

**Groups**

| Name | Created with | Description |
| - | - | - |
| `{node}/admins` | `Node` | Node administrators |
| `{org}/admins` | `Organization` | Organization administrators, have owner rights over all resources of the organization |
| `{org}/members` | `Organization` | Organization members |

Reserved groups are created together with the account that owns them and cannot be created or deleted independently.

**Subject keywords**

| Keyword | Description |
| - | - |
| `Authenticated` | `SubjectRef` matching any authenticated principal |
| `Anonymous` | `SubjectRef` matching any unauthenticated principal |
| `Everyone` | `SubjectRef` matching any principal |

Subject keywords cannot collide with resource names, as resource references in short form always include a type prefix, a DID, or an ID.


# RFC-021: ODF Spec Restructuring
Source: https://docs.kamu.dev/odf/rfcs/021-spec-restructuring



[![PR](https://img.shields.io/github/pulls/detail/state/kamu-data/open-data-fabric/135?label=PR)](https://github.com/kamu-data/open-data-fabric/pull/)

**Start Date**: 2026-10-13\
**Published Date**: 2026-10-14

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [x] Forwards-compatible

## Summary

Proposes a reorganization of the ODF specification from a single document into a set of interconnected standards.

## Table of Contents

* [Motivation](#motivation)
* [Proposed Solution](#proposed-solution)
  * [Top-level Documents](#top-level-documents)
  * [Domains](#domains)
    * [Datasets](#datasets)
    * [Transformations](#transformations)
    * [Exchange](#exchange)
    * [Auth & Privacy](#auth--privacy)
    * [Automation](#automation)
    * [Reference](#reference)
  * [Repository Structure](#repository-structure)
  * [Naming and Versioning](#naming-and-versioning)
  * [Common Spec Template](#common-spec-template)
* [Drawbacks](#drawbacks)
* [Rationale and alternatives](#rationale-and-alternatives)
* [Prior art](#prior-art)
* [Unresolved questions](#unresolved-questions)
* [Future possibilities](#future-possibilities)

## Motivation

The current spec has grown very large and covers an extensive breadth of topics starting from common terminology, metadata structure, and data layouts, and to transformations, checkpointing, engine protocols and many more. This is too much information in a single document already.

We also see several more areas developing that are currently not reflected in the spec. From low-level things like encryption, privacy-preserving computations, and logical data hashing, to high-level things like IaC resource framework, node-level and decentralized authorization etc. These areas also need to be covered by a spec, but represent their own domains that would be better covered in standalone documents that reference other domains rather than being mixed with the rest.

## Proposed Solution

We propose to split the current ODF specification into a set of interconnected standards, similarly to [W3C Credentials Community Group](https://w3c-ccg.org/specifications/standards/), [Open Container Initiative](https://opencontainers.org/), and others.

### Top-level Documents

Domain-independent documents that serve as an entry point into the specs:

| Document | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Overview** | Problem statement, requirements, design decisions, map of all specs and their dependencies, conformance profiles. Mostly non-normative. | Abstract, Problem Statement, Requirements, Scope, Design Decisions | Existing |
| ✅ **Glossary** | Definitions of core concepts and terms shared by all specs | Concepts and Components | Existing |

### Domains

Specs are grouped into domains that roughly follow the protocol stack, from the dataset structure up to network-level and node-level concerns. Rather than defining general-purpose primitives like hashing and identity in standalone specs, each spec explains how it applies established standards like [Multiformats](https://multiformats.io/) and [W3C DID](https://www.w3.org/TR/did-core/) where it uses them. Each spec is a standalone document that can reference specs in its own or lower domains.

Status legend:

* **Existing** - content already exists in the current spec and will be moved
* **RFC** - content exists in accepted RFCs but was never fully integrated into the spec
* **New** - area not yet covered, the spec will be created as it matures

#### Datasets

| Spec | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Logical Structure** | Overview of the dataset logical structure and its components | Nature of Data, Evolution of Data, Dataset, Data, Data Slice, Metadata | Existing |
| ✅ **Identity** | `did:odf` DID method, key pairs and proof of ownership, aliases and references | Dataset Identity, [RFC-003], [RFC-007], [RFC-012] | Existing |
| ✅ **Data Model** | Bitemporal event model, common data schema (offset, operation type, system / event time), changelog representation of retractions and corrections, Parquet layout, schema evolution rules | Data Format, Common Data Schema, Retractions and Corrections, Schema Evolution, [RFC-001], [RFC-015] | Existing |
| ✅ **Data Schema** | ODF logical schema format for describing data independently of its physical format (Parquet, Avro, etc.), logical types and their mapping to physical types | Schema Format, [RFC-010], [RFC-016] | Existing |
| ✅ **Metadata Chain** | Blocks, events, semantics and consistency rules, binary and human-readable encodings, block hashing, hash representation (multihash / multibase), extensibility and annotations | Metadata, Metadata Chain, Metadata Format, Metadata Block Hashing, Hash Representation, [RFC-004], [RFC-005], [RFC-010], [RFC-012], [RFC-014] | Existing |
| ✅ **Data & Checkpoint Hashing** | Physical hashing of data and checkpoint files, stable hashing of data records independent of physical encoding, canonical representation of logical types, hashing algorithm (currently defined in [arrow-digest](https://github.com/sergiimk/arrow-digest)) and its multicodec code | Physical and Logical Hashes, Checkpoint Hashing, [RFC-002] | Existing |
| ✅ **Storage Layout** | Dataset structure, refs, tags, branches, named vs. content-addressable layouts, linked large files | Dataset Layout, [RFC-003], [RFC-006], [RFC-017] | Existing |
| **Compactions** | Soft compactions (merging data files while preserving history) and hard compactions (permanent removal of history for compliance or storage optimization) | | New |
| ✅ **Ingestion** | Root datasets, polling and push sources, source state, merge strategies | Root Dataset, Ingestion, Merge Strategy, Data Ingestion, [RFC-009], [RFC-011], [RFC-014] | Existing |

#### Transformations

| Spec | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Processing Model** | Derivative datasets, queries, event time, watermarks, checkpoints, query evolution and migrations, provenance | Derivative Dataset, Query, Time, Watermark, Checkpoint, Provenance, Derivative Transformations, Query Evolution | Existing |
| ✅ **Engine Contract** | Engine execution model, communication interface, operations, sandboxing, engine versioning and deprecation | Engine, Engine Contract, Engine Deprecation | Existing |
| **Projections** | Temporal projections (AS AT / AS OF), collapsing changelog streams into state | Projection | Existing |

#### Exchange

| Spec | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Transfer Protocols** | Repositories, simple and smart transfer protocols, direct-to-storage transfers, replication | Repository, Data Sharing, Dataset Sharing, Repository Contract, [RFC-007], [RFC-008] | Existing |
| **Query Protocols** | Standard query interfaces (Flight SQL / ADBC), federated query planning | | New |
| ✅ **Verifiable Batch Queries** | Query commitments and proofs that allow consumers to verify that query results correspond to specific dataset states | [RFC-022] | RFC |
| **Discovery** | Dataset registries, anchoring of dataset state to external ledgers | | New |

Verification is not a standalone spec. Different kinds of verification (metadata chain integrity, data and checkpoint hashes, validity of transferred data, reproducibility of transformations, validity of query results) depend on the mechanics of the parts of the system they apply to, so each is specified in the corresponding spec. The **Overview** maps all verification layers to these specs.

#### Auth & Privacy

| Spec | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Authorization** | Accounts and their identifiers, groups, auth attributes, policies and policy bindings within a node | [RFC-020] | RFC |
| **Decentralized Permissions** | Access control spanning multiple nodes and organizations: delegated authorization (UCAN), threshold cryptography | | New |
| **Layered Encryption** | Encryption of data and metadata at rest and in transit, key management and rotation, interaction with content hashing | | New |
| **Confidential Computing** | Integration of TEEs, ZK proofs, FHE as alternative verification mechanisms; anonymization and other privacy-preserving transformations (e.g. query gateways, portal datasets that expose any dataset as root without disclosing details) | Future Topics / Anonymization | New |

#### Automation

| Spec | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Resource Framework** | Manifest format, resource model, desired vs. observed state, extensibility of resource types | Manifest, [RFC-018] | RFC |
| **Event Bus** | The delivery mechanism and common types of events flowing in the system | | New |
| ✅ **Tasks and Flows** | Workload scheduling and execution resources within ODF nodes | [RFC-019] | RFC |
| **Webhooks** | Delivery of dataset events to external HTTP endpoints | `sinks/v1alpha1` schemas | New |

#### Reference

Schemas and API definitions that domain specs refer to, largely generated from machine-readable sources in the repository.

| Reference | Scope | Sources | Status |
| - | - | - | - |
| ✅ **Metadata Schemas** | Metadata blocks, events, and their nested types | Metadata Reference, `MetadataBlock` / `MetadataEvent` and related types in `schemas/{datasets,data}` | Existing |
| ✅ **Resource Schemas** | Resource types and their specs, grouped by API group and version | `schemas/*` | RFC |
| **Engine API** | Engine communication interface IDL and messages | Engine API Reference, `schemas/engines` | Existing |
| **Transfer Protocol API** | Simple and smart transfer protocol endpoints and messages | Repository API Reference, `contracts/transfer/smart-transfer-protocol.{openapi,asyncapi}.yaml` | Existing |

### Repository Structure

Current layout of the repository:

```
.
├── open-data-fabric.md      # Generated spec (from src/open-data-fabric.md + build/metadata-reference.md)
├── src/
│   ├── open-data-fabric.md  # Spec source
│   └── images/              # Diagram sources (.puml, .svg, .png)
├── images/                  # Generated diagrams referenced by the spec
├── rfcs/
│   ├── {xxx}-{name}/        # Images of {xxx}-{name}.md
│   └── {xxx}-{name}.md
├── schemas/                 # JSON Schemas for resources and metadata
│   └── {context}/{version}/
│       └── {schema}.json
├── schemas-generated/       # Generated artifacts
│   ├── flatbuffers/opendatafabric.fbs
│   └── mermaid/erd.{mmd,svg}
├── protocols/               # API definitions
│   ├── smart-transfer-protocol.asyncapi.yaml
│   └── smart-transfer-protocol.openapi.yaml
├── examples/                # Example manifests, one directory per scenario
│   └── {scenario}/{name}.yaml
├── tools/
├── authoring_schemas.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── Makefile
└── README.md
```

Proposed layout of the repository:

```
.
├── specs/                           # Spec sources split by domain
│   ├── {domain}/
│   │   ├── {spec}/                  # Images of {spec}.md
│   │   └── {spec}.md
│   ├── overview.md
│   ├── glossary.md
│   └── reference.md
├── rfcs/
│   ├── {xxx}-{name}/                # Images of {xxx}-{name}.md
│   └── {xxx}-{name}.md
├── schemas/
│   └── {context}/{version}/
│       └── {schema}.json
├── contracts/                       # Serde and API contracts
│   ├── erd.{mmd,svg}                # Generated from schemas
│   ├── metadata/opendatafabric.fbs  # Generated from schemas
│   └── transfer/
│       ├── smart-transfer-protocol.asyncapi.yaml
│       └── smart-transfer-protocol.openapi.yaml
├── examples/
│   └── {scenario}/{name}.yaml
├── tools/
├── authoring_schemas.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── Makefile
└── README.md
```

### Naming and Versioning

Each spec is stored as `/specs/{domain}/{spec}.md`, where both domain and spec names are the kebab-cased titles from the [Domains](#domains) tables, e.g. `/specs/auth-and-privacy/authorization.md`. Domain-independent documents (`overview.md`, `glossary.md`, `reference.md`) are stored at the root of `/specs`.

All specs are versioned together as a single set, i.e. a release of ODF includes all specs at their current state. Releases are versioned by tagging the repository - the version is not written into the spec documents, neither as a whole nor per individual spec.

### Common Spec Template

All spec files follow a common structure:

```markdown theme={null}
# {Spec Title}

**Domain**: {Domain}\
**Depends on**: [{Spec}](../{domain}/{spec}.md), ...\
**RFCs**: [RFC-{NNN}](../../rfcs/{NNN}-{rfc}.md), ...

## Abstract
One paragraph describing what this spec covers.

## Table of Contents

## ...
Spec-specific sections.
```

The **RFCs** field lists the RFCs that introduced or changed the spec, mirroring the **Specs** field in RFC headers, so the two are linked in both directions.

## Drawbacks

* Existing links to sections of the monolithic spec (from RFCs, docs, websites, and implementations) will break unless redirects or a stub document with pointers to new locations is maintained.
* Readers lose a single linear narrative and need to navigate between multiple documents to understand the system as a whole. The **Overview** and **Glossary** specs are meant to mitigate this.
* More documents means more maintenance overhead and a higher risk of duplication or inconsistency between specs. This increases the need for tooling that checks cross-references.
* Publishing a set of specs requires more elaborate build and website tooling than rendering a single markdown file.

## Rationale and alternatives

Splitting by domain aligns specs with their audiences and their rate of change: an engine implementer, a repository operator, and a node administrator care about different subsets of ODF, and the data format evolves much more slowly than node-level resources. It also allows new areas to start as standalone drafts without destabilizing mature parts of the spec, and makes it clear which parts of the spec an RFC affects.

Alternatives considered:

* **Single document split into files at build time** - would reduce the size of individual source files, but still presents one large document to readers and does not allow different parts of the spec to have different maturity levels.
* **Split by implementer role** (e.g. engine, coordinator, repository contracts, as partially done in the current spec) - roles cut across many domains (e.g. a coordinator touches ingestion, hashing, metadata, and transfer), which would lead to duplication. Instead, roles are expected to be expressed as conformance profiles in the **Overview** spec that reference relevant domain specs.
* **Few coarse-grained specs** (e.g. format, processing, and protocols, similarly to OCI) - simpler to navigate, but new areas like privacy, authorization, and automation would again bloat these documents.

## Prior art

* [W3C Credentials Community Group](https://w3c-ccg.org/specifications/standards/)
* [Open Container Initiative](https://opencontainers.org/) - separate, independently versioned specs for [image format](https://github.com/opencontainers/image-spec), [runtime](https://github.com/opencontainers/runtime-spec), and [distribution](https://github.com/opencontainers/distribution-spec)
* [Apache Arrow](https://arrow.apache.org/docs/format/index.html) - format specs (Columnar, IPC, C Data Interface, Flight, Flight SQL, ADBC) versioned separately from libraries
* [Apache Iceberg](https://iceberg.apache.org/spec/) - table spec alongside [view spec](https://iceberg.apache.org/view-spec/), [Puffin spec](https://iceberg.apache.org/puffin-spec/), and [REST catalog API](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml)
* [AT Protocol](https://atproto.com/specs/atp) - content-addressed repositories and DID-based identity split into small specs (data model, cryptography, DID, repository, sync, etc.)
* [Eclipse Dataspace Protocol](https://eclipse-dataspace-protocol-base.github.io/DataspaceProtocol/) - abstract protocol specs (catalog, contract negotiation, transfer process) with separate protocol bindings

## Unresolved questions

## Future possibilities

* Versioning specs independently, each with its own maturity level (similarly to W3C CSS modules), once some domains stabilize while others continue to evolve rapidly
* Rust linter to ensure referential integrity

[RFC-001]: ./001-record-offsets.md

[RFC-002]: ./002-logical-data-hashes.md

[RFC-003]: ./003-content-addressability.md

[RFC-004]: ./004-metadata-extensibility.md

[RFC-005]: ./005-dataset-annotations.md

[RFC-006]: ./006-checkpoints-as-files.md

[RFC-007]: ./007-simple-transfer-protocol.md

[RFC-008]: ./008-smart-transfer-protocol.md

[RFC-009]: ./009-ingest-source-state.md

[RFC-010]: ./010-data-schema-in-metadata.md

[RFC-011]: ./011-push-ingest-sources.md

[RFC-012]: ./012-recommend-base16-encoding.md

[RFC-014]: ./014-minimize-offset-scanning.md

[RFC-015]: ./015-unified-changelog-stream-schema.md

[RFC-016]: ./016-odf-schema.md

[RFC-017]: ./017-large-files-linking.md

[RFC-018]: ./018-iac-resource-framework.md

[RFC-019]: ./019-iac-task-flow-system.md

[RFC-020]: ./020-auth.md

[RFC-022]: ./022-batch-query-commitments.md


# RFC-022: Batch Query Commitments
Source: https://docs.kamu.dev/odf/rfcs/022-batch-query-commitments



**Start Date**: 2024-07-26\
**Published Date**: 2024-10-01

**Specs**:

* [Verifiable Batch Queries](/odf/spec/exchange/verifiable-batch-queries)

**Authors**:

* [Sergii Mikhtoniuk](mailto:sergii.mikhtoniuk@kamu.dev), [Kamu](https://kamu.dev)

**Compatibility**:

* [x] Backwards-compatible
* [ ] Forwards-compatible

## Table of Contents

* [Summary](#summary)
* [Motivation](#motivation)
  * [Requirements](#requirements)
  * [Current API](#current-api)
  * [Commitments in the ODF Oracle](#commitments-in-the-odf-oracle)
  * [Need for REST Commitments](#need-for-rest-commitments)
* [Reference-level explanation](#reference-level-explanation)
  * [Structured Recursive Commitments](#structured-recursive-commitments)
  * [JSON Hashing](#json-hashing)
  * [Response Validation](#response-validation)
  * [Storing Commitments](#storing-commitments)
  * [Dispute Resolution](#dispute-resolution)
  * [Requirements Checklist](#requirements-checklist)
  * [Effects on ODF Oracle](#effects-on-odf-oracle)
* [Compatibility](#compatibility)
* [Drawbacks](#drawbacks)
* [Rationale and alternatives](#rationale-and-alternatives)
  * [HTTP Message Signatures](#http-message-signatures)
* [Prior art](#prior-art)
* [Unresolved questions](#unresolved-questions)
* [Future possibilities](#future-possibilities)

## Summary

Proposes a new commitment scheme for REST Batch Query API results and the process of opening (verifying) such commitments.

## Motivation

When a user queries data, we want to establish an unbreakable cryptographic commitment that allows the user to **hold every party in the supply chain accountable** for veracity of the result.

This accountability should persist long after the response is provided, so that participants can process data fast in "**optimistic**" fashion, and can **verify and dispute the results later**.

This mechanism in future will allow us to build **automatic dispute resolution** mechanisms within the ODF network that financially penalize malicious actors.

### Requirements

* **Hiding** - Commitment should not reveal any information about the operation.
* **Binding** - It should be impossible to change any inputs or outputs of the operation without invalidating the commitment.
* **Succinct** - Commitment itself should be small enough to e.g. store on-chain, and the process of disputing the operation results should not require storing and transferring the resulting data.
* **Deterministic** - Same query should result in the same commitment as long as the query itself is deterministic.
* **Reproducible** - Given all the same inputs of the operation a third party should be able to repeat the query, arrive at the same results, and be able to successfully validate them against the original commitment.
  * Reproducibility will be the *primary way of verifying the commitments* for now, until we introduce zero-knowledge computation proofs.
* **Non-repudiable** - A prover cannot dispute performing the operation. There exists a mechanism that ties its identity to the provided commitment. This mechanism may be separate from the commitment itself to keep the latter pure zero-knowledge.
* **Composable** - It should be possible to split computation into several steps and compose their commitments into one final commitment. When joining two datasets from different owners the entire execution may be split into sub-plans that are executed by different nodes belonging to different parties. In the event of a dispute, we should be able to assign responsibility to the specific party and not just stop at the party that performed the final result aggregation.
* **Multi-protocol** - Ideally the commitment structure can be reused without significant alterations in other protocols like Blockchain Oracle, Flight SQL, Kafka egress, WebSockets etc.

### Current API

Our initial attempt to add a resemblance of cryptographic commitments to the REST Batch Query API looked like this:

```
POST /query
Host: node.demo.kamu.dev

{
  "query": "select offset, city, population from populations order by offset desc",
  "dataFormat": "JsonAoA",
  "aliases": [{
    "alias": "populations",
    "id": "did:odf:fed0..ee02"
  }]
}
```

```
200 OK

{
  "data": [
    [1, "B", 200],
    [0, "A", 100]
  ],
  "schema": "...",
  "resultHash": "f968..656e",
  "state": {
    "inputs": [{
      "id": "did:odf:fed0..ee02",
      "blockHash": "f162..3e0b"
    }]
  }
}
```

In this case the commitment consists of `resultHash` and `state` properties.

* **Hiding** - **NO**
  * `state` reveals in plain text what datasets were queried.
* **Binding** - **NO**
  * Commitment does not cover the inputs like `query`, `aliases` and other parameters like `dataFormat`, row limits etc.
  * Commitment does not cover outputs like `schema`
* **Succinct** - **YES**
* **Deterministic** - **YES**
* **Reproducible** - **NO**
  * The `state` information allows the user to run the query on the same exact input data
  * But the query itself is not part of the commitment. This creates a loophole: when the result is disputed, a node operator can claim that this result was provided for a slightly different query than what the user is showing.
* **Non-repudiable** - **NO**
  * There is no mechanism that ties the result to a node that executed it.
* **Composable** - **NO**
* **Multi-protocol** - **NO**

### Commitments in the ODF Oracle

Although purely in REST API context this attempt looked like a failure, it actually provided a good starting point for ODF Oracle commitments on the blockchain.

* **Hiding** - **NO**
  * The entire request and response data is publicly available, but this is an inherent limitation of non-ZK blockchains
* **Binding** - **YES**
  * All inputs and outputs are immutably stored on-chain
* **Succinct** - **YES** / **NO**
  * All response data ends up stored which we cannot avoid, but the commitment itself does not add much overhead
* **Deterministic** - **YES**
* **Reproducible** - **YES**
  * Original query with all of its parameters is stored on-chain and can be repeated against the `state` information in response
* **Non-repudiable** - **YES**
  * Automatically achieved as the provider signs the response transaction with its private key
* **Composable** - **NO**
* **Multi-protocol** - **NO**

### Need for REST Commitments

As you can see, in combination with properties added by blockchain, the existing API fulfills the oracle use case pretty well.

However, now we are starting to work with clients that require commitments from REST API alone. This is, for example, important in the scenario where ODF nodes are used to provide factual data to LLM chatbots - the calling party will interact only with REST API and needs to have an ability to store the commitment in the conversation history.

## Reference-level explanation

### Structured Recursive Commitments

```
POST /query
Host: node.demo.kamu.dev

{
  "query": "select * from foo",
  "queryDialect": "SqlDataFusion",
  "dataFormat": "JsonAoS",
  "schemaFormat": "ArrowJson",
  "signed": true
}
```

```
200 OK

{
  # Input section that can be used to fully reproduce the result of operation
  "input": {
    "query": "select * from foo",
    "queryDialect": "SqlDataFusion",
    "dataFormat": "JsonAoS",
    "schemaFormat": "ArrowJson",
    "datasets": [{
      "id": "did:odf:fed0..ee02",
      "alias": "foo",
      "blockHash": "f162..3e0b",
    }]
  },

  # The result
  "output": {
    "data": [
      {"foo": 1, "bar": "A"},
      {"foo": 2, "bar": "B"}
    ],
    "schema": "{\"fields\": [...]}",
  },

  # Information about processing performed by other nodes as part of this operation
  # that are identical in structure to the top-level response (recursive)
  "subQueries": [{
    "input": {...},
    "subQueries": [...],
    "commitment": {...},
    "proof": {...},
  }]

  # Succinct commitment
  "commitment": {
    # Input hash: multihash(canonicalize(input))
    "inputHash": "f162..3e0b",

    # Output hash: multihash(canonicalize(output))
    "outputHash": "f162..baaf",

    # Sub-queries hash: multihash(canonicalize(subQueries))
    "subQueriesHash": "f162..8dae",
  }

  # Signature block
  "proof": {
    "type": "Ed25519Signature2020",

    # DID (public key) of the node performing the computation
    "verificationMethod": "did:key:fed0..3e41",

    # Signature: multibase(sign(canonicalize(commitment)))
    "proofValue": "u3MvG...ouQk"
  }
}
```

Note: Technically the `commitment` block is redundant and the caller could compute all hashes themselves. In fact they have to do so to validate the response (see below). We include this block purely to:

* Protect clients from implementing hashing in a way that is not consistent with the specification
* Provide more information during the validation and dispute stages to isolate inconsistencies to specific parts.

### JSON Hashing

Note that the proposed scheme involves hashing JSON objects. To exclude variations like whitespace characters and property order differences between JSON library implementations we need to convert it into canonical form as specified in [RFC 8785: JSON Canonicalization Scheme](https://datatracker.ietf.org/doc/html/rfc8785).

### Response Validation

Immediately upon receiving the response the caller must validate it to ensure that the commitment is not bogus and is disputable:

1. Verify that `proof.verificationMethod` DID belongs to the node you are requesting data from
2. Verify `commitment.outputHash` by re-computing the hash
3. Verify `commitment.inputHash` by re-computing the hash
4. Verify `commitment.subQueriesHash` by re-computing the hash
5. Verify the signature in `proof.proofValue` using
   1. Message: `canonicalize(commitment)`
   2. Public key: `key_from_did(proof.verificationMethod)`
6. **Repeat the steps 3-5 recursively** for every entry in `subQueries`

### Storing Commitments

To be able to dispute the result in future, consumer should store the following fields:

* `input`
* `subQueries`
* `commitment`
* `proof`

These different parts can be also used separately for different purposes as they all provide different "layers" of verifiability:

* `input` - provides reproducibility
* `commitment` - seals the response against tampering
* `proof` - adds non-repudiation via signature
* `subQueries` - provide the above for multi-party computations

### Dispute Resolution

The procedure of resolving a dispute using a commitment via reproducibility is as follows:

1. Consumer submits a claim to the verifier by providing:
   1. `input`
   2. `commitment`
   3. `subQueries`
   4. `proof`
2. Verifier obtains the relevant `input.datasets`
3. Verifier validates the claim according to the same rules as described in [Response Validation](#response-validation)
   1. If any check fails the claim is **rejected** as invalid
4. Verifier checks that all `authority` specified in `subQueries` are registered nodes
   1. If an unregistered node is found the claim is **accepted** and the blame is assigned to the root `authority` for using a bogus node for computing part of the response
5. Verifier generates the logical query plan and checks that the plan is identical or equivalent to the plan composed from the `subQueries`
   1. If plan does not correspond to the original query the claim is **accepted** and the blame is assigned to the root `authority` for providing a malicious query plan
6. In a bottom-up fashion Verifier repeats all the computation steps using `input` and compares hashes of outputs to those captured in `commitment.outputHash`
   1. If a hash doesn't match the claim is **accepted** and the blame is assigned to the corresponding `authority` for providing a malicious output data

### Requirements Checklist

* **Hiding** - **YES** / **NO**
  * The hashes in the `commitment` block can be used as zero-knowledge commitments that do not reveal any information
  * The `proof` reveals the identity of the node performing the computation
* **Binding** - **YES**
* **Deterministic** - **YES**
  * Assuming canonical response format
* **Reproducible** - **YES**
  * Using the `input` field gives us all the necessary info to reproduce the request
* **Non-repudiable** - **YES**
  * Through the identity and signature in `proof`
* **Composable** - **YES**
  * Through `subQueries`
* **Multi-protocol** - **YES**
  * A similar scheme can be implemented in other protocols e.g. Flight SQL using canonical structs in FlatBuffers format.

### Effects on ODF Oracle

Oracle response structure will be modified to include all the elements of the new REST API response.

Some of them may be a little redundant: e.g. root-level authority and signature are not needed, because we have the address and transaction signature of the provider.

But storing these fields will allow us to use the same dispute resolution mechanism for oracle responses as for other protocols, and avoid the need to map provider addresses to node DIDs on multiple blockchains.

## Compatibility

The new response structure replaces the `resultHash` and `state` properties of the existing REST Batch Query API response, so clients relying on them will need to be updated.

## Drawbacks

* Responses become larger and more nested, as they now include the full `input` section, sub-queries, and the signature block.
* Validating a response requires clients to implement JSON canonicalization, hashing, and signature verification.

## Rationale and alternatives

### HTTP Message Signatures

[RFC 9421: HTTP Message Signatures](https://datatracker.ietf.org/doc/rfc9421/) proposes a standard for signing HTTP requests and responses.

Under this scheme a query operation could look like this:

```
POST /query
Host: node.demo.kamu.dev
Accept-Signature: sig=(@status, ...)

{
  "query": "select * from foo",
  "queryFormat": "SqlDataFusion",
  "aliases": [{"alias": "foo", "id": "did:odf:fed0..ee02"}],
  "dataFormat": "JsonAoS"
}
```

```
200 OK
Signature-Input: (@status, ...)
Signature: sig=:...:

{
  "data": [
    {"foo": 1, "bar": "A"},
    {"foo": 2, "bar": "B"}
  ],
  "dataFormat": "JsonAoS",
  "query": "select * from foo",
  "queryFormat": "SqlDataFusion",
  "aliases": [{"alias": "foo", "id": "did:odf:fed0..ee02"}],
  "state": [{"id": "did:odf:fed0..ee02", "blockHash": "f162..3e0b"}]
}
```

In this case the commitment would consist of the entire HTTP response (excluding only some headers that are not part of the `Signature-Input`).

* **Hiding** - **YES** / **NO**
  * Depending on the algorithm, signature may reveal the public key of the signer and therefore the identity of a node that provided the response.
  * RFC specifies `Content-Digest` header that may be included into the signature instead of the entire payload, thus hiding the response data
* **Binding** - **YES**
* **Succinct** - **NO**
  * The signature itself is small and constant in size, however to dispute the operation a client will have to hold on to the entire response data to show the validity of a signature before verifier attempts to reproduce the request - this would be impractical at scale
* **Deterministic** - **YES**
  * Assuming stable / canonical response format
* **Reproducible** - **YES**
  * Note that the response would include the query and all other input parameters, so removing the output fields like `data` and `schema` gives us all the necessary inputs to reproduce the request
* **Non-repudiable** - **YES**
  * Through the signature
* **Composable** - **NO**
  * Including commitments from other nodes that performed parts of the request will require some different mechanism where HTTP signatures are included as parts of the JSON response
  * Nodes may not even use HTTP when communicating with one another and use Flight SQL instead.
* **Multi-protocol** - **NO**
  * This scheme is HTTP specific - other protocols will have to use other methods.

## Prior art

* [RFC 9421: HTTP Message Signatures](https://datatracker.ietf.org/doc/rfc9421/) - see [Rationale and alternatives](#http-message-signatures)
* [RFC 8785: JSON Canonicalization Scheme](https://datatracker.ietf.org/doc/html/rfc8785) - used for hashing JSON objects
* [W3C Verifiable Credentials Data Integrity](https://www.w3.org/TR/vc-data-integrity/) - structure of the `proof` block (`type`, `verificationMethod`, `proofValue`)

## Unresolved questions

* Do we need to include a proof that a specific `blockHash` actually exists, e.g. by including a signature? Otherwise would it be too easy for a malicious prover to say that the dataset altered its history and they cannot be held accountable for a non-reproducible result?
* Checking that the root SQL query corresponds to the `subQueries` plans will be a difficult task, but there doesn't seem to be an alternative. Even with ZK proofs we will likely hit this problem of computation equivalence.

## Future possibilities

* Zero-knowledge proofs are currently very limited in terms of data types and functions they can support, with JOINs being especially problematic. They can be introduced progressively, where possible for simpler queries, as an alternative to verification through signing and reproducibility.
* Commitments can serve as a foundation for automatic dispute resolution in the ODF network that financially penalizes malicious actors.


# RFCs
Source: https://docs.kamu.dev/odf/rfcs/index



Protocol design evolution proposals:

* [RFC-001: Record Offsets](/odf/rfcs/001-record-offsets)
* [RFC-002: Logical Data Digests](/odf/rfcs/002-logical-data-hashes)
* [RFC-003: Content Addressability](/odf/rfcs/003-content-addressability)
* [RFC-004: Metadata Extensibility](/odf/rfcs/004-metadata-extensibility)
* [RFC-005: New Annotation Metadata Events](/odf/rfcs/005-dataset-annotations)
* [RFC-006: Store Checkpoints as Files](/odf/rfcs/006-checkpoints-as-files)
* [RFC-007: Simple Transfer Protocol](/odf/rfcs/007-simple-transfer-protocol)
* [RFC-008: Smart Transfer Protocol](/odf/rfcs/008-smart-transfer-protocol)
* [RFC-009: Ingestion Source State](/odf/rfcs/009-ingest-source-state)
* [RFC-010: Data Schema in Metadata](/odf/rfcs/010-data-schema-in-metadata)
* [RFC-011: Push Ingest Sources](/odf/rfcs/011-push-ingest-sources)
* [RFC-012: Recommend `base16` encoding for textual representation of hashes and DIDs](/odf/rfcs/012-recommend-base16-encoding)
* [RFC-013: Enum representation in YAML encoding](/odf/rfcs/013-yaml-enum-representation)
* [RFC-014: Minimizing scanning for last offset and block](/odf/rfcs/014-minimize-offset-scanning)
* [RFC-015: Unified changelog stream schema](/odf/rfcs/015-unified-changelog-stream-schema)
* [RFC-016: ODF Schema Format](/odf/rfcs/016-odf-schema)
* [RFC-017: Large Files Linking](/odf/rfcs/017-large-files-linking)
* [RFC-018: Infrastructure-as-Code Resource Framework](/odf/rfcs/018-iac-resource-framework)
* [RFC-019: IaC Task and Flow System](/odf/rfcs/019-iac-task-flow-system)
* [RFC-020: Authorization Model](/odf/rfcs/020-auth)
* [RFC-021: ODF Spec Restructuring](/odf/rfcs/021-spec-restructuring)
* [RFC-022: Batch Query Commitments](/odf/rfcs/022-batch-query-commitments)


# Schemas
Source: https://docs.kamu.dev/odf/schemas

Schemas of common metadata objects

## Auth

### `Group`

A named, possibly nested set of principals. Policies can be bound to groups so that permissions are granted to all members at once. Some group names are reserved, e.g. `{org}/admins` and `{org}/members` are created together with an organization.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`GroupSpecInput`](#groupspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/Group.json)

### `GroupSpec`

Group specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `members` | `array(`[`SubjectHandle`](#subjecthandle)`)` | | | Members of the group. Only the `Resource` subject kind is allowed, so that well-known subjects like `Everyone` cannot be nested into a group. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/GroupSpec.json)

### `GroupSpecInput`

Group specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `members` | `array(`[`SubjectRef`](#subjectref)`)` | | | Members of the group. Only the `Resource` subject kind is allowed, so that well-known subjects like `Everyone` cannot be nested into a group. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/GroupSpecInput.json)

### `Node`

An entire ODF deployment acting under a single identity. The node acts as a principal in system tasks and node-to-node communications and owns node-wide resources like the reserved `{node}/admins` group.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`NodeSpecInput`](#nodespecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/Node.json)

### `NodeSpec`

Node account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `did` | `string` | | `account-id` | DID associated with the account by ODF or an external system. |
| `displayName` | `string` | | | Human-friendly display name. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/NodeSpec.json)

### `NodeSpecInput`

Node account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `did` | `string` | | `account-id` | DID associated with the account by ODF or an external system. |
| `displayName` | `string` | | | Human-friendly display name. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/NodeSpecInput.json)

### `Organization`

An account that owns resources on behalf of a group of users but cannot authenticate or act itself. Upon creation the node creates reserved groups `{org}/admins`, whose members have owner rights over all resources of the organization, and `{org}/members`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`OrganizationSpecInput`](#organizationspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/Organization.json)

### `OrganizationSpec`

Organization account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `displayName` | `string` | | | Human-friendly display name. |
| `avatarUrl` | `string` | | `uri` | URL of the account's avatar image. |
| `disabled` | `boolean` | | | Suspends the account. Effects of suspension are implementation-specific. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/OrganizationSpec.json)

### `OrganizationSpecInput`

Organization account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `displayName` | `string` | | | Human-friendly display name. |
| `avatarUrl` | `string` | | `uri` | URL of the account's avatar image. |
| `disabled` | `boolean` | | | Suspends the account. Effects of suspension are implementation-specific. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/OrganizationSpecInput.json)

### `Policy`

A reusable grant of permissions. Policies with subject and object slots are templates instantiated via `PolicyBinding`, while policies without slots take effect on their own.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`PolicySpecInput`](#policyspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/Policy.json)

### `PolicyBinding`

Instantiates a `Policy` template for a concrete subject and object.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`PolicyBindingSpecInput`](#policybindingspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicyBinding.json)

### `PolicyBindingSpec`

Policy binding specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `policy` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Policy template to instantiate. |
| `subject` | [`SubjectHandle`](#subjecthandle) | ✔️ | | Party the permissions are granted to. |
| `object` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Resource the permissions are granted on. Refers to both the resource and the entity materialized from it. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicyBindingSpec.json)

### `PolicyBindingSpecInput`

Policy binding specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `policy` | [`PolicyRef`](#policyref) | ✔️ | | Policy template to instantiate. |
| `subject` | [`SubjectRef`](#subjectref) | ✔️ | | Party the permissions are granted to. |
| `object` | [`ResourceRef`](#resourceref) | ✔️ | | Resource the permissions are granted on. Refers to both the resource and the entity materialized from it. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicyBindingSpecInput.json)

### `PolicyRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the `Policy`. |
| `id` | `string` | | `resource-id` | ID of the resource. |
| `name` | `string` | | `resource-name` | Name of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicyRef.json)

### `PolicySpec`

Policy specification. The `kind` selects the policy language.

| Union Type | Description |
| - | - |
| [`PolicySpec::Generic`](#policyspecgeneric) | Engine-agnostic policy that grants a set of actions. Subject and object are always supplied by a `PolicyBinding`. |
| [`PolicySpec::Cedar`](#policyspeccedar) | Policy expressed in the Cedar policy language. Policies that declare `?principal` and `?resource` slots are templates instantiated via `PolicyBinding`, otherwise they take effect on their own. Only supported by nodes that use Cedar. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicySpec.json)

### `PolicySpec::Generic`

Engine-agnostic policy that grants a set of actions. Subject and object are always supplied by a `PolicyBinding`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `actions` | `array(string)` | ✔️ | | Schema URIs of actions granted by the policy. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicySpec.json)

### `PolicySpec::Cedar`

Policy expressed in the Cedar policy language. Policies that declare `?principal` and `?resource` slots are templates instantiated via `PolicyBinding`, otherwise they take effect on their own. Only supported by nodes that use Cedar.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `body` | `string` | ✔️ | | Cedar policy source. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicySpec.json)

### `PolicySpecInput`

Policy specification. The `kind` selects the policy language.

| Union Type | Description |
| - | - |
| [`PolicySpecInput::Generic`](#policyspecinputgeneric) | Engine-agnostic policy that grants a set of actions. Subject and object are always supplied by a `PolicyBinding`. |
| [`PolicySpecInput::Cedar`](#policyspecinputcedar) | Policy expressed in the Cedar policy language. Policies that declare `?principal` and `?resource` slots are templates instantiated via `PolicyBinding`, otherwise they take effect on their own. Only supported by nodes that use Cedar. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicySpecInput.json)

### `PolicySpecInput::Generic`

Engine-agnostic policy that grants a set of actions. Subject and object are always supplied by a `PolicyBinding`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `actions` | `array(string)` | ✔️ | | Schema URIs of actions granted by the policy. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicySpecInput.json)

### `PolicySpecInput::Cedar`

Policy expressed in the Cedar policy language. Policies that declare `?principal` and `?resource` slots are templates instantiated via `PolicyBinding`, otherwise they take effect on their own. Only supported by nodes that use Cedar.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `body` | `string` | ✔️ | | Cedar policy source. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PolicySpecInput.json)

### `PrincipalHandle`

Link to a resolved identity that can act as a principal: `User`, `ServiceAccount`, or `Node`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `type` | `string` | ✔️ | `type-uri` | Type URI of the principal. |
| `account` | [`ResourceOwnerHandle`](#resourceownerhandle) | | | Account that owns the service account. |
| `id` | `string` | ✔️ | `resource-id` | ID of the principal resource. |
| `did` | `string` | | `did` | DID of the principal, if applicable. |
| `name` | `string` | ✔️ | `resource-name` | Name of the principal resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PrincipalHandle.json)

### `PrincipalRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `type` | `string` | | `type-ref` | Short type name or full type URI of the principal. |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the service account. |
| `id` | `string` | | `resource-id` | ID of the principal resource. |
| `did` | `string` | | `did` | DID of the principal. |
| `name` | `string` | | `resource-name` | Name of the principal resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/PrincipalRef.json)

### `ResourceOwnerHandle`

Link to a resolved account that can own resources: `User`, `Organization`, or `Node`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `id` | `string` | ✔️ | `resource-id` | ID of the account resource. |
| `did` | `string` | ✔️ | `account-id` | DID of the account. |
| `name` | `string` | ✔️ | `account-name` | Name of the account. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/ResourceOwnerHandle.json)

### `ResourceOwnerRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `id` | `string` | | `resource-id` | UUID of the account resource. |
| `did` | `string` | | `account-id` | DID of the account. |
| `name` | `string` | | `account-name` | Name of the account. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/ResourceOwnerRef.json)

### `ServiceAccount`

A non-human identity used by automation (scheduled flows, CI, installed applications). Service accounts are scoped under an owning account and can act as principals, but cannot own resources.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`ServiceAccountSpecInput`](#serviceaccountspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/ServiceAccount.json)

### `ServiceAccountSpec`

Service account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `displayName` | `string` | | | Human-friendly display name. |
| `disabled` | `boolean` | | | Suspends the account. Effects of suspension are implementation-specific. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/ServiceAccountSpec.json)

### `ServiceAccountSpecInput`

Service account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `displayName` | `string` | | | Human-friendly display name. |
| `disabled` | `boolean` | | | Suspends the account. Effects of suspension are implementation-specific. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/ServiceAccountSpecInput.json)

### `SubjectHandle`

Resolved reference to a party a grant is given to.

| Union Type | Description |
| - | - |
| [`SubjectHandle::Resource`](#subjecthandleresource) | A specific `User`, `ServiceAccount`, `Node`, or members of a `Group`. |
| [`SubjectHandle::Authenticated`](#subjecthandleauthenticated) | Matches any authenticated principal. |
| [`SubjectHandle::Anonymous`](#subjecthandleanonymous) | Matches any unauthenticated principal. |
| [`SubjectHandle::Everyone`](#subjecthandleeveryone) | Matches any principal, authenticated or not. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectHandle.json)

### `SubjectHandle::Resource`

A specific `User`, `ServiceAccount`, `Node`, or members of a `Group`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `type` | `string` | ✔️ | `type-uri` | Type URI of the subject resource. |
| `account` | [`ResourceOwnerHandle`](#resourceownerhandle) | | | Account that owns the subject resource (for service accounts and groups). |
| `id` | `string` | ✔️ | `resource-id` | ID of the subject resource. |
| `did` | `string` | | `did` | DID of the subject resource, if applicable. |
| `name` | `string` | ✔️ | `resource-name` | Name of the subject resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectHandle.json)

### `SubjectHandle::Authenticated`

Matches any authenticated principal.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectHandle.json)

### `SubjectHandle::Anonymous`

Matches any unauthenticated principal.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectHandle.json)

### `SubjectHandle::Everyone`

Matches any principal, authenticated or not.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectHandle.json)

### `SubjectRef`

Reference to a party a grant is given to. Unlike a principal, a subject may stand for many principals at once. Short form is either a well-known subject name (e.g. `Authenticated`) or a resource reference with a type prefix, DID, or ID (e.g. `Group:acme/engineering`).

| Union Type | Description |
| - | - |
| [`SubjectRef::Resource`](#subjectrefresource) | A specific `User`, `ServiceAccount`, `Node`, or members of a `Group`. |
| [`SubjectRef::Authenticated`](#subjectrefauthenticated) | Matches any authenticated principal. |
| [`SubjectRef::Anonymous`](#subjectrefanonymous) | Matches any unauthenticated principal. |
| [`SubjectRef::Everyone`](#subjectrefeveryone) | Matches any principal, authenticated or not. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectRef.json)

### `SubjectRef::Resource`

A specific `User`, `ServiceAccount`, `Node`, or members of a `Group`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `type` | `string` | | `type-ref` | Short type name or full type URI of the subject resource. |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the subject resource (for service accounts and groups). |
| `id` | `string` | | `resource-id` | ID of the subject resource. |
| `did` | `string` | | `did` | DID of the subject resource. |
| `name` | `string` | | `resource-name` | Name of the subject resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectRef.json)

### `SubjectRef::Authenticated`

Matches any authenticated principal.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectRef.json)

### `SubjectRef::Anonymous`

Matches any unauthenticated principal.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectRef.json)

### `SubjectRef::Everyone`

Matches any principal, authenticated or not.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/SubjectRef.json)

### `User`

An individual person who can authenticate and act as a principal.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`UserSpecInput`](#userspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/User.json)

### `UserSpec`

User account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `did` | `string` | | `account-id` | DID associated with the account by ODF or an external system. |
| `displayName` | `string` | | | Human-friendly display name. |
| `email` | `string` | | `email` | Email address of the user. |
| `avatarUrl` | `string` | | `uri` | URL of the account's avatar image. |
| `password` | [`Secret`](#secret) | | | Password for local authentication, stored as a hash upon apply. Absent for SSO or DID-based accounts. |
| `disabled` | `boolean` | | | Suspends the account. Effects of suspension are implementation-specific. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/UserSpec.json)

### `UserSpecInput`

User account specification.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `did` | `string` | | `account-id` | DID associated with the account by ODF or an external system. |
| `displayName` | `string` | | | Human-friendly display name. |
| `email` | `string` | | `email` | Email address of the user. |
| `avatarUrl` | `string` | | `uri` | URL of the account's avatar image. |
| `password` | [`Secret`](#secret) | | | Password for local authentication, stored as a hash upon apply. Absent for SSO or DID-based accounts. |
| `disabled` | `boolean` | | | Suspends the account. Effects of suspension are implementation-specific. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/UserSpecInput.json)

### `BindPolicy`

Create `PolicyBinding` resources that reference this resource as `object`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `subject` | [`SubjectHandle`](#subjecthandle) | ✔️ | | Subject of the binding being created, allowing policies to restrict granting access to e.g. `Everyone`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/auth/v1alpha1/actions/BindPolicy.json)

## Config

### `Secret`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `value` | `string` | ✔️ | | A secret value in raw or encoded form. |
| `contentEncoding` | `string` | | | Represents the encoding of the value. Typically will be `jwe` after a raw secret gets encrypted. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/Secret.json)

### `SecretSet`

Defines a set of secrets stored and managed by the ODF node.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`SecretSetSpecInput`](#secretsetspecinput) | ✔️ | | Specifies the desired state of the secret set. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/SecretSet.json)

### `SecretSetSpec`

Defines a set of secrets stored and managed by the ODF node and accessible via embedded sercets provider.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `secrets` | [`Secrets`](#secrets) | ✔️ | | Key value pairs of secrets. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/SecretSetSpec.json)

### `SecretSetSpecInput`

Defines a set of secrets stored and managed by the ODF node and accessible via embedded sercets provider.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `secrets` | [`Secrets`](#secrets) | ✔️ | | Key value pairs of secrets. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/SecretSetSpecInput.json)

### `Secrets`

Container for key-value secrets. Every key must be a string. Values may be strings with raw unencrypted data or objects that signify the encoding.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/Secrets.json)

### `ValueHandle`

Reference to a value within a `VariableSet` or a `SecretSet`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerHandle`](#resourceownerhandle) | ✔️ | | Account that owns the target resource. |
| `type` | `string` | ✔️ | `type-uri` | Type URI of the target resource. |
| `id` | `string` | ✔️ | `resource-id` | ID of the resource within a node. |
| `name` | `string` | ✔️ | `resource-name` | Name of a resource. |
| `path` | `string` | | | JSON path to a value within a `VariableSet` or a `SecretSet`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/ValueHandle.json)

### `ValueRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the `VariableSet` or the `SecretSet`. |
| `id` | `string` | | `resource-id` | ID of a resource. |
| `type` | `string` | | `type-ref` | Short type name or full type URI of the target resource. |
| `name` | `string` | | `resource-name` | Name of a resource. |
| `path` | `string` | | | JSON path to a value within a `VariableSet` or a `SecretSet`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/ValueRef.json)

### `ValueRefs`

Container for key-value variables. Every key must be a string. Values shoud reference fields in `SecretSet`s and `VariableSet`s.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/ValueRefs.json)

### `Variable`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `value` | `string` | ✔️ | | A value in raw or encoded form. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/Variable.json)

### `VariableSet`

Defines a set of variables stored and managed by the ODF node.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`VariableSetSpecInput`](#variablesetspecinput) | ✔️ | | Specifies the desired state of the variable set. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/VariableSet.json)

### `VariableSetSpec`

Defines a set of variables stored and managed by the ODF node and accessible via embedded variables provider.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `variables` | [`Variables`](#variables) | ✔️ | | Key value pairs of variables. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/VariableSetSpec.json)

### `VariableSetSpecInput`

Defines a set of variables stored and managed by the ODF node and accessible via embedded variables provider.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `variables` | [`Variables`](#variables) | ✔️ | | Key value pairs of variables. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/VariableSetSpecInput.json)

### `Variables`

Container for key-value variables. Every key must be a string. Values may be raw strings or objects that incorporate the encoding.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/Variables.json)

### `Decrypt`

Decrypt a secret from a `SecretSet`.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/config/v1alpha1/actions/Decrypt.json)

## Data

### `DataField`

Represents a named field (column) in a root or nested struct schema

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | ✔️ | | Name of the field |
| `type` | [`DataType`](#datatype) | ✔️ | | Logical type of the field that defines its semantic behavior and value ranges |
| `extra` | [`ExtraAttributes`](#extraattributes) | | | ODF extensions |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataField.json)

### `DataSchema`

This schema aims to be a human-friendly variant of Arrow. Arrow currently specifies only the [flatbuffer format](https://github.com/apache/arrow/blob/f9301c0ba8a7ed1b0b63275cfdd4c44c26b04675/format/Schema.fbs) which has many legacy to it and is not suited to be defined by humans, so we had to define our own schema format. While inspired by Arrow - this format makes a clear separation between logical data types and encoding (physical layout) of data in the chunks.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `fields` | `array(`[`DataField`](#datafield)`)` | ✔️ | | Top-level fields (columns) of the schema. |
| `extra` | [`ExtraAttributes`](#extraattributes) | | | ODF extensions |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataSchema.json)

### `DataType`

Defines a logical type of the field. Logical type determines the semantics and boudaries of a type and how it can be operated on, without a concern about encoding and physical layout of the data in chunks.

| Union Type | Description |
| - | - |
| [`DataType::Binary`](#datatypebinary) | A sequence of bytes. Used for arbitrary binary data. |
| [`DataType::Bool`](#datatypebool) | A boolean value representing true or false. |
| [`DataType::Date`](#datatypedate) | A calendar date. |
| [`DataType::Decimal`](#datatypedecimal) | A fixed-point decimal number with a specified precision and scale. |
| [`DataType::Duration`](#datatypeduration) | An elapsed time interval with a specified time unit. |
| [`DataType::Float16`](#datatypefloat16) | A floating-point number. |
| [`DataType::Float32`](#datatypefloat32) | A floating-point number. |
| [`DataType::Float64`](#datatypefloat64) | A floating-point number. |
| [`DataType::Int8`](#datatypeint8) | An integer value. |
| [`DataType::Int16`](#datatypeint16) | An integer value. |
| [`DataType::Int32`](#datatypeint32) | An integer value. |
| [`DataType::Int64`](#datatypeint64) | An integer value. |
| [`DataType::UInt8`](#datatypeuint8) | An integer value. |
| [`DataType::UInt16`](#datatypeuint16) | An integer value. |
| [`DataType::UInt32`](#datatypeuint32) | An integer value. |
| [`DataType::UInt64`](#datatypeuint64) | An integer value. |
| [`DataType::List`](#datatypelist) | A list of values, all having the same data type. |
| [`DataType::Map`](#datatypemap) | A map of key-value pairs, represented as a list of entries (structs with key and value fields). |
| [`DataType::Null`](#datatypenull) | A type representing the absence of a value (null). |
| [`DataType::Option`](#datatypeoption) | A type representing an optional (nullable) value of another data type. |
| [`DataType::Struct`](#datatypestruct) | A collection of named fields, each with its own data type. |
| [`DataType::Time`](#datatypetime) | A time of day value, without a date, with a specified unit of granularity. |
| [`DataType::Timestamp`](#datatypetimestamp) | A point in time, represented as an offset from the Unix epoch in a specific timezone. |
| [`DataType::String`](#datatypestring) | A Unicode string. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Binary`

A sequence of bytes. Used for arbitrary binary data.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `fixedLength` | `integer` | | `uint64` | Number of bytes per value for fixed-size binary. If omitted, the binary is variable-length. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Bool`

A boolean value representing true or false.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Date`

A calendar date.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Decimal`

A fixed-point decimal number with a specified precision and scale.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `precision` | `integer` | ✔️ | `uint32` | Total number of decimal digits that can be stored. |
| `scale` | `integer` | ✔️ | `int32` | Number of digits after the decimal point. In certain situations, scale could be negative number. For negative scale, it is the number of padding 0 to the right of the digits.<br /><br />For example the number 12300 could be treated as a decimal has precision 3 and scale -2. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Duration`

An elapsed time interval with a specified time unit.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `unit` | [`TimeUnit`](#timeunit) | | | The unit of the duration measurement.<br /><br />Default: `Millisecond` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Float16`

A floating-point number.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Float32`

A floating-point number.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Float64`

A floating-point number.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Int8`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Int16`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Int32`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Int64`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::UInt8`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::UInt16`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::UInt32`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::UInt64`

An integer value.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::List`

A list of values, all having the same data type.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `itemType` | [`DataType`](#datatype) | ✔️ | | Data type of list items. |
| `fixedLength` | `integer` | | `uint64` | Number of list items per value for fixed-size lists. If omitted, the list is variable-length. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Map`

A map of key-value pairs, represented as a list of entries (structs with key and value fields).

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `keyType` | [`DataType`](#datatype) | ✔️ | | Data type of the map's keys. |
| `valueType` | [`DataType`](#datatype) | ✔️ | | Data type of the map's values. |
| `keysSorted` | `boolean` | | | Set to true if the keys within each value are sorted. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Null`

A type representing the absence of a value (null).

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Option`

A type representing an optional (nullable) value of another data type.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `inner` | [`DataType`](#datatype) | ✔️ | | Inner data type for the optional value. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Struct`

A collection of named fields, each with its own data type.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `fields` | `array(`[`DataField`](#datafield)`)` | ✔️ | | Fields that make up the struct. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Time`

A time of day value, without a date, with a specified unit of granularity.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `unit` | [`TimeUnit`](#timeunit) | | | The unit of the time value.<br /><br />Default: `Millisecond` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::Timestamp`

A point in time, represented as an offset from the Unix epoch in a specific timezone.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `unit` | [`TimeUnit`](#timeunit) | | | The unit of the timestamp value that determines its precision.<br /><br />Default: `Millisecond` |
| `timezone` | `string` | | | The timezone is an optional string indicating the name of a timezone<br />one of<br /><br />\* As used in the Olson timezone database (the "tz database" or<br />  "tzdata"), such as "America/New\_York".<br />\* An absolute timezone offset of the form "+XX:XX" or "-XX:XX",<br />  such as "+07:30".<br /><br />Default: `UTC` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `DataType::String`

A Unicode string.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/DataType.json)

### `ExtraAttributes`

Container for custom key-value extension attributes. Every key must be in the form of `<domain>/<path>` (e.g. `kamu.dev/archetype`) in order to fully disambiguate the value in the face of multiple extensions. Values may be any valid JSON including nested objects.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/ExtraAttributes.json)

### `OperationType`

Defines an operation in a changelog stream.

| Enum Value |
| :-: |
| `Append` |
| `Retract` |
| `CorrectFrom` |
| `CorrectTo` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/OperationType.json)

### `TimeUnit`

Defines the unit of measurement of time

| Enum Value |
| :-: |
| `Second` |
| `Millisecond` |
| `Microsecond` |
| `Nanosecond` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/data/v1alpha1/TimeUnit.json)

## Datasets

### `AddData`

Indicates that data has been ingested into a root dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `prevCheckpoint` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Hash of the checkpoint file used to restore ingestion state, if any. |
| `prevOffset` | `integer` | | `uint64` | Last offset of the previous data slice, if any. Must be equal to the last non-empty `newData.offsetInterval.end`. |
| `newData` | [`DataSlice`](#dataslice) | | | Describes output data written during this transaction, if any. |
| `newCheckpoint` | [`Checkpoint`](#checkpoint) | | | Describes checkpoint written during this transaction, if any. If an engine operation resulted in no updates to the checkpoint, but checkpoint is still relevant for subsequent runs - a hash of the previous checkpoint should be specified. |
| `newWatermark` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Last watermark of the output data stream, if any. Initial blocks may not have watermarks, but once watermark is set - all subsequent blocks should either carry the same watermark or specify a new (greater) one. Thus, watermarks are monotonically non-decreasing. |
| `newSourceState` | [`SourceState`](#sourcestate) | | | The state of the source the data was added from to allow fast resuming. If the state did not change but is still relevant for subsequent runs it should be carried, i.e. only the last state per source is considered when resuming. |
| `extra` | [`ExtraAttributes`](#extraattributes) | | | ODF extensions. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/AddData.json)

### `AllowAnonymousRead`

Controls whether the dataset is readable by users who did not authenticate. Materialized into the auth engine by the Dataset controller.

*Type: `boolean`*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/AllowAnonymousRead.json)

### `AllowPublicRead`

Controls whether the dataset is readable by any authenticated principal. Materialized into the auth engine by the Dataset controller.

*Type: `boolean`*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/AllowPublicRead.json)

### `AttachmentEmbedded`

Embedded attachment item.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `path` | `string` | ✔️ | | Path to an attachment if it was materialized into a file. |
| `content` | `string` | ✔️ | | Content of the attachment. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/AttachmentEmbedded.json)

### `Attachments`

Defines the source of attachment files.

| Union Type | Description |
| - | - |
| [`Attachments::Embedded`](#attachmentsembedded) | For attachments that are specified inline and are embedded in the metadata. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Attachments.json)

### `Attachments::Embedded`

For attachments that are specified inline and are embedded in the metadata.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `items` | `array(`[`AttachmentEmbedded`](#attachmentembedded)`)` | ✔️ | | List of embedded items. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Attachments.json)

### `Checkpoint`

Describes a checkpoint produced by an engine

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `physicalHash` | `string` | ✔️ | [`multihash`](https://github.com/multiformats/multihash) | Hash sum of the checkpoint file. |
| `size` | `integer` | ✔️ | `uint64` | Size of checkpoint file in bytes. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Checkpoint.json)

### `DataSlice`

Describes a slice of data added to a dataset or produced via transformation

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `logicalHash` | `string` | ✔️ | [`multihash`](https://github.com/multiformats/multihash) | Logical hash sum of the data in this slice. |
| `physicalHash` | `string` | ✔️ | [`multihash`](https://github.com/multiformats/multihash) | Hash sum of the data part file. |
| `offsetInterval` | [`OffsetInterval`](#offsetinterval) | ✔️ | | Data slice produced by the transaction. |
| `size` | `integer` | ✔️ | `uint64` | Size of data file in bytes. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DataSlice.json)

### `Dataset`

Represents a desired state of a dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`DatasetSpecInput`](#datasetspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Dataset.json)

### `DatasetHandle`

Link to a dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerHandle`](#resourceownerhandle) | | | Reference to an account that owns the dataset. |
| `id` | `string` | ✔️ | `resource-id` | ID of the dataset resource. |
| `did` | `string` | ✔️ | `dataset-id` | DID of the dataset. |
| `name` | `string` | ✔️ | `resource-name` | Name of the dataset. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetHandle.json)

### `DatasetKind`

Represents type of the dataset.

| Enum Value |
| :-: |
| `Root` |
| `Derivative` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetKind.json)

### `DatasetRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the dataset. |
| `id` | `string` | | `resource-id` | UUID of the dataset resource. |
| `did` | `string` | | `dataset-id` | DID of the dataset. |
| `name` | `string` | | `resource-name` | Name of the dataset. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetRef.json)

### `DatasetSelector`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the target resources. |
| `id` | `string` | | `resource-id` | ID of the singular resource. |
| `name` | `string` | | | Name pattern in SQL `LIKE` format. |
| `labels` | [`LabelFilter`](#labelfilter) | | | Filter by resource labels. |
| `kind` | [`DatasetKind`](#datasetkind) | | | Restricts the selector to datasets of a specific kind. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetSelector.json)

### `DatasetSpec`

Represents a desired state of the dataset metadata.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `did` | `string` | ✔️ | `dataset-id` | DID of the dataset in global ODF network |
| `kind` | [`DatasetKind`](#datasetkind) | ✔️ | | Type of the dataset. |
| `metadata` | `array(`[`MetadataEvent`](#metadataevent)`)` | ✔️ | | An array of metadata events that will be used to populate the chain, e.g. to set info, licenses, and attachments. As the dataset evolves, it maintains a snapshot of the key events that describe the dataset in its latest state, excluding intermediate transitions (e.g. previous schema changes) and events related to ingestions and transformations. |
| `volume` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Reference to a storage volume where dataset data will be stored. If omitted, the node's default storage is used. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetSpec.json)

### `DatasetSpecInput`

Represents a desired state of the dataset metadata.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `did` | `string` | | `dataset-id` | DID of the dataset in global ODF network |
| `kind` | [`DatasetKind`](#datasetkind) | ✔️ | | Type of the dataset. |
| `metadata` | `array(`[`MetadataEvent`](#metadataevent)`)` | ✔️ | | An array of metadata events that will be used to populate the chain, e.g. to set info, licenses, and attachments. As the dataset evolves, it maintains a snapshot of the key events that describe the dataset in its latest state, excluding intermediate transitions (e.g. previous schema changes) and events related to ingestions and transformations. |
| `volume` | [`PersistentVolumeRef`](#persistentvolumeref) | | | Reference to a storage volume where dataset data will be stored. If omitted, the node's default storage is used. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetSpecInput.json)

### `DatasetVocabulary`

Specifies the mapping of system columns onto dataset schema.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `offsetColumn` | `string` | | | Name of the offset column.<br /><br />Default: `offset` |
| `operationTypeColumn` | `string` | | | Name of the operation type column.<br /><br />Default: `op` |
| `systemTimeColumn` | `string` | | | Name of the system time column.<br /><br />Default: `system_time` |
| `eventTimeColumn` | `string` | | | Name of the event time column.<br /><br />Default: `event_time` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/DatasetVocabulary.json)

### `ExecuteTransform`

Indicates that derivative transformation has been performed.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `queryInputs` | `array(`[`ExecuteTransformInput`](#executetransforminput)`)` | ✔️ | | Defines inputs used in this transaction. Slices corresponding to every input dataset must be present. |
| `prevCheckpoint` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Hash of the checkpoint file used to restore transformation state, if any. |
| `prevOffset` | `integer` | | `uint64` | Last offset of the previous data slice, if any. Must be equal to the last non-empty `newData.offsetInterval.end`. |
| `newData` | [`DataSlice`](#dataslice) | | | Describes output data written during this transaction, if any. |
| `newCheckpoint` | [`Checkpoint`](#checkpoint) | | | Describes checkpoint written during this transaction, if any. If an engine operation resulted in no updates to the checkpoint, but checkpoint is still relevant for subsequent runs - a hash of the previous checkpoint should be specified. |
| `newWatermark` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Last watermark of the output data stream, if any. Initial blocks may not have watermarks, but once watermark is set - all subsequent blocks should either carry the same watermark or specify a new (greater) one. Thus, watermarks are monotonically non-decreasing. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/ExecuteTransform.json)

### `ExecuteTransformInput`

Describes a slice of the input dataset used during a transformation

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `datasetId` | `string` | ✔️ | `dataset-id` | Input dataset identifier. |
| `prevBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Last block of the input dataset that was previously incorporated into the derivative transformation, if any. Must be equal to the last non-empty `newBlockHash`. Together with `newBlockHash` defines a half-open `(prevBlockHash, newBlockHash]` interval of blocks that will be considered in this transaction. |
| `newBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Hash of the last block that will be incorporated into the derivative transformation. When present, defines a half-open `(prevBlockHash, newBlockHash]` interval of blocks that will be considered in this transaction. |
| `prevOffset` | `integer` | | `uint64` | Last data record offset in the input dataset that was previously incorporated into the derivative transformation, if any. Must be equal to the last non-empty `newOffset`. Together with `newOffset` defines a half-open `(prevOffset, newOffset]` interval of data records that will be considered in this transaction. |
| `newOffset` | `integer` | | `uint64` | Offset of the last data record that will be incorporated into the derivative transformation, if any. When present, defines a half-open `(prevOffset, newOffset]` interval of data records that will be considered in this transaction. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/ExecuteTransformInput.json)

### `MetadataBlock`

An individual block in the metadata chain that captures the history of modifications of a dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `systemTime` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | System time when this block was written. |
| `prevBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Hash sum of the preceding block. |
| `sequenceNumber` | `integer` | ✔️ | `uint64` | Block sequence number, starting from zero at the seed block. |
| `event` | [`MetadataEvent`](#metadataevent) | ✔️ | | Event data. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/MetadataBlock.json)

### `MetadataEvent`

Represents a transaction that occurred on a dataset.

| Union Type | Description |
| - | - |
| [`AddData`](#adddata) | Indicates that data has been ingested into a root dataset. |
| [`ExecuteTransform`](#executetransform) | Indicates that derivative transformation has been performed. |
| [`Seed`](#seed) | Establishes the identity of the dataset. Always the first metadata event in the chain. |
| [`SetPollingSource`](#setpollingsource) | |
| [`SetTransform`](#settransform) | Defines a transformation that produces data in a derivative dataset. |
| [`SetVocab`](#setvocab) | Lets you manipulate names of the system columns to avoid conflicts. |
| [`SetAttachments`](#setattachments) | Associates a set of files with this dataset. |
| [`SetInfo`](#setinfo) | Provides basic human-readable information about a dataset. |
| [`SetLicense`](#setlicense) | Defines a license that applies to this dataset. |
| [`SetDataSchema`](#setdataschema) | Specifies the complete schema of Data Slices added to the Dataset following this event. |
| [`AddPushSource`](#addpushsource) | |
| [`DisablePushSource`](#disablepushsource) | |
| [`DisablePollingSource`](#disablepollingsource) | |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/MetadataEvent.json)

### `OffsetInterval`

Describes a range of data as a closed arithmetic interval of offsets

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `start` | `integer` | ✔️ | `uint64` | Start of the closed interval \[start; end]. |
| `end` | `integer` | ✔️ | `uint64` | End of the closed interval \[start; end]. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/OffsetInterval.json)

### `Projection`

Represents a projection of a dataaset history into a state for fast lookups.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`ProjectionSpecInput`](#projectionspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Projection.json)

### `ProjectionSpec`

Represents a projection of a dataaset history into a state for fast lookups.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `inputs` | `array(`[`TransformInput`](#transforminput)`)` | ✔️ | | Datasets that will be used as sources. |
| `project` | [`Transform`](#transform) | ✔️ | | Transformation that will be applied to produce new data. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/ProjectionSpec.json)

### `ProjectionSpecInput`

Represents a projection of a dataaset history into a state for fast lookups.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `inputs` | `array(`[`TransformInput`](#transforminput)`)` | ✔️ | | Datasets that will be used as sources. |
| `project` | [`Transform`](#transform) | ✔️ | | Transformation that will be applied to produce new data. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/ProjectionSpecInput.json)

### `Seed`

Establishes the identity of the dataset. Always the first metadata event in the chain.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `datasetId` | `string` | ✔️ | `dataset-id` | Unique identity of the dataset. |
| `datasetKind` | [`DatasetKind`](#datasetkind) | ✔️ | | Type of the dataset. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Seed.json)

### `SetAttachments`

Associates a set of files with this dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `attachments` | [`Attachments`](#attachments) | ✔️ | | One of the supported attachment sources. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SetAttachments.json)

### `SetDataSchema`

Specifies the complete schema of Data Slices added to the Dataset following this event.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `rawArrowSchema` | `string` | | [`flatbuffers`](https://flatbuffers.dev/) | DEPRECATED: Apache Arrow schema encoded in its native flatbuffers representation. |
| `schema` | [`DataSchema`](#dataschema) | | | Defines the logical schema of the data files that follow this event. Will become a required field after migration. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SetDataSchema.json)

### `SetInfo`

Provides basic human-readable information about a dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `description` | `string` | | | Brief single-sentence summary of a dataset. |
| `keywords` | `array(string)` | | | Keywords, search terms, or tags used to describe the dataset. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SetInfo.json)

### `SetLicense`

Defines a license that applies to this dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `shortName` | `string` | ✔️ | | Abbreviated name of the license. |
| `name` | `string` | ✔️ | | Full name of the license. |
| `spdxId` | `string` | | | License identifier from the SPDX License List. |
| `websiteUrl` | `string` | ✔️ | `uri` | URL where licensing terms can be found. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SetLicense.json)

### `SetTransform`

Defines a transformation that produces data in a derivative dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `inputs` | `array(`[`TransformInput`](#transforminput)`)` | ✔️ | | Datasets that will be used as sources. |
| `transform` | [`Transform`](#transform) | ✔️ | | Transformation that will be applied to produce new data. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SetTransform.json)

### `SetVocab`

Lets you manipulate names of the system columns to avoid conflicts.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `offsetColumn` | `string` | | | Name of the offset column. |
| `operationTypeColumn` | `string` | | | Name of the operation type column. |
| `systemTimeColumn` | `string` | | | Name of the system time column. |
| `eventTimeColumn` | `string` | | | Name of the event time column. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SetVocab.json)

### `SqlQueryStep`

Defines a query in a multi-step SQL transformation.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `alias` | `string` | | | Name of the temporary view that will be created from result of the query. Step without this alias will be treated as an output of the transformation. |
| `query` | `string` | ✔️ | | SQL query the result of which will be exposed under the alias. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/SqlQueryStep.json)

### `TemporalTable`

Temporary Flink-specific extension for creating temporal tables from streams.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | ✔️ | | Name of the dataset to be converted into a temporal table. |
| `primaryKey` | `array(string)` | ✔️ | | Column names used as the primary key for creating a table. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/TemporalTable.json)

### `Transform`

Engine-specific processing queries that shape the resulting data.

| Union Type | Description |
| - | - |
| [`Transform::Sql`](#transformsql) | Transform using one of the SQL dialects. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Transform.json)

### `Transform::Sql`

Transform using one of the SQL dialects.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `engine` | `string` | ✔️ | | Identifier of the engine used for this transformation. |
| `version` | `string` | | | Version of the engine to use. |
| `query` | `string` | | | SQL query the result of which will be used as an output. This is a convenience property meant only for defining queries by hand. When stored in the metadata this property will never be set and instead will be converted into a single-iter `queries` array. |
| `queries` | `array(`[`SqlQueryStep`](#sqlquerystep)`)` | | | Specifies multi-step SQL transformations. Each step acts as a shorthand for `CREATE TEMPORARY VIEW <alias> AS (<query>)`. Last query in the array should have no alias and will be treated as an output. |
| `temporalTables` | `array(`[`TemporalTable`](#temporaltable)`)` | | | Temporary Flink-specific extension for creating temporal tables from streams. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Transform.json)

### `TransformInput`

Describes a derivative transformation input

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `datasetRef` | `string` | ✔️ | `dataset-ref` | A local or remote dataset reference. When block is accepted this MUST be in the form of a DatasetId to guarantee reproducibility, as aliases can change over time. |
| `alias` | `string` | | | An alias under which this input will be available in queries. Will be populated from `datasetRef` if not provided before resolving it to DatasetId. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/TransformInput.json)

### `Watermark`

Represents a watermark in the event stream.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `systemTime` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Moment in processing time when watermark was emitted. |
| `eventTime` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Moment in event time which watermark has reached. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/Watermark.json)

### `AlterHistory`

Rewrite or truncate dataset history.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/actions/AlterHistory.json)

### `Read`

Read dataset data (query, download).

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/actions/Read.json)

### `Write`

Append new data to the dataset.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/datasets/v1alpha1/actions/Write.json)

## Engines

### `RawQueryRequest`

Sent by the coordinator to an engine to perform query on raw input data, usually as part of ingest preprocessing step

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `inputDataPaths` | `array(string)` | ✔️ | | Paths to input data files to perform query over. Must all have identical schema. |
| `transform` | [`Transform`](#transform) | ✔️ | | Transformation that will be applied to produce new data. |
| `outputDataPath` | `string` | ✔️ | `path` | Path where query result will be written. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/RawQueryRequest.json)

### `RawQueryResponse`

Sent by an engine to coordinator when performing the raw query operation

| Union Type | Description |
| - | - |
| [`RawQueryResponse::Progress`](#rawqueryresponseprogress) | Reports query progress |
| [`RawQueryResponse::Success`](#rawqueryresponsesuccess) | Query executed successfully |
| [`RawQueryResponse::InvalidQuery`](#rawqueryresponseinvalidquery) | Query did not pass validation |
| [`RawQueryResponse::InternalError`](#rawqueryresponseinternalerror) | Internal error during query execution |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/RawQueryResponse.json)

### `RawQueryResponse::Progress`

Reports query progress

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/RawQueryResponse.json)

### `RawQueryResponse::Success`

Query executed successfully

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `numRecords` | `integer` | ✔️ | `uint64` | Number of records produced by the query |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/RawQueryResponse.json)

### `RawQueryResponse::InvalidQuery`

Query did not pass validation

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `message` | `string` | ✔️ | | Explanation of an error |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/RawQueryResponse.json)

### `RawQueryResponse::InternalError`

Internal error during query execution

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `message` | `string` | ✔️ | | Brief description of an error |
| `backtrace` | `string` | | | Details of an error (e.g. a backtrace) |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/RawQueryResponse.json)

### `TransformRequest`

Sent by the coordinator to an engine to perform the next step of data transformation

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `datasetId` | `string` | ✔️ | `dataset-id` | Unique identifier of the output dataset. |
| `datasetAlias` | `string` | ✔️ | `dataset-alias` | Alias of the output dataset, for logging purposes only. |
| `systemTime` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | System time to use for new records. |
| `vocab` | [`DatasetVocabulary`](#datasetvocabulary) | ✔️ | | Vocabulary of the output dataset. |
| `transform` | [`Transform`](#transform) | ✔️ | | Transformation that will be applied to produce new data. |
| `queryInputs` | `array(`[`TransformRequestInput`](#transformrequestinput)`)` | ✔️ | | Defines inputs used in this transaction. Slices corresponding to every input dataset must be present. |
| `nextOffset` | `integer` | ✔️ | `uint64` | Starting offset to use for new data records. |
| `prevCheckpointPath` | `string` | | `path` | TODO: This will be removed when coordinator will be speaking to engines purely through Arrow. |
| `newCheckpointPath` | `string` | ✔️ | `path` | TODO: This will be removed when coordinator will be speaking to engines purely through Arrow. |
| `newDataPath` | `string` | ✔️ | `path` | TODO: This will be removed when coordinator will be speaking to engines purely through Arrow. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformRequest.json)

### `TransformRequestInput`

Sent as part of the engine transform request operation to describe the input

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `datasetId` | `string` | ✔️ | `dataset-id` | Unique identifier of the dataset. |
| `datasetAlias` | `string` | ✔️ | `dataset-alias` | Alias of the output dataset, for logging purposes only. |
| `queryAlias` | `string` | ✔️ | | An alias of this input to be used in queries. |
| `vocab` | [`DatasetVocabulary`](#datasetvocabulary) | ✔️ | | Vocabulary of the input dataset. |
| `offsetInterval` | [`OffsetInterval`](#offsetinterval) | | | Subset of data that goes into this transaction. |
| `dataPaths` | `array(string)` | ✔️ | | TODO: This will be removed when coordinator will be slicing data for the engine. |
| `schemaFile` | `string` | ✔️ | `path` | TODO: replace with actual DDL or Parquet schema. |
| `explicitWatermarks` | `array(`[`Watermark`](#watermark)`)` | ✔️ | | Watermarks that should be injected into the stream to separate micro batches for reproducibility. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformRequestInput.json)

### `TransformResponse`

Sent by an engine to coordinator when performing the data transformation

| Union Type | Description |
| - | - |
| [`TransformResponse::Progress`](#transformresponseprogress) | Reports query progress |
| [`TransformResponse::Success`](#transformresponsesuccess) | Query executed successfully |
| [`TransformResponse::InvalidQuery`](#transformresponseinvalidquery) | Query did not pass validation |
| [`TransformResponse::InternalError`](#transformresponseinternalerror) | Internal error during query execution |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformResponse.json)

### `TransformResponse::Progress`

Reports query progress

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformResponse.json)

### `TransformResponse::Success`

Query executed successfully

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `newOffsetInterval` | [`OffsetInterval`](#offsetinterval) | | | Data slice produced by the transaction, if any. |
| `newWatermark` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Watermark advanced by the transaction, if any. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformResponse.json)

### `TransformResponse::InvalidQuery`

Query did not pass validation

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `message` | `string` | ✔️ | | Explanation of an error |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformResponse.json)

### `TransformResponse::InternalError`

Internal error during query execution

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `message` | `string` | ✔️ | | Brief description of an error |
| `backtrace` | `string` | | | Details of an error (e.g. a backtrace) |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/engines/v1alpha1/TransformResponse.json)

## Events

### `EventFilter`

Filters that work on domain event types and fields.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/events/v1alpha1/EventFilter.json)

## Flows

### `Flow`

Defines a sequence of tasks to be executed upon certain trigger conditions.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`FlowSpecInput`](#flowspecinput) | ✔️ | | Specifies the desired state of the flow. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/Flow.json)

### `FlowRun`

Defines a set of tasks to be executed in a sequence.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`FlowRunSpecInput`](#flowrunspecinput) | ✔️ | | Specifies the desired state of the flow run. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRun.json)

### `FlowRunActivationCause`

Cause of the flow run activation

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `activationTime` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time at which the trigger fired. |
| `initiator` | [`PrincipalHandle`](#principalhandle) | | | Principal that initiated the run, if applicable. |
| `trigger` | [`FlowTrigger`](#flowtrigger) | ✔️ | | Copy of the trigger configuration from the parent Flow that fired. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRunActivationCause.json)

### `FlowRunActivationCauses`

Condition capturing what caused this FlowRun to be scheduled. Set by the controller at creation time; never written by users. In case of a retry, the causes of the original run are preserved.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `activationCauses` | `array(`[`FlowRunActivationCause`](#flowrunactivationcause)`)` | ✔️ | | Triggers that caused this run to be scheduled. |
| `lateActivationCauses` | `array(`[`FlowRunActivationCause`](#flowrunactivationcause)`)` | | | Additional triggers that fired while this run was already queued or executing. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRunActivationCauses.json)

### `FlowRunRetry`

Condition linking this FlowRun to the previous FlowRun it is retrying. Set by the controller; never written by users.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `retryOf` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Reference to the FlowRun this run is retrying. |
| `retryNumber` | `integer` | ✔️ | `uint32` | Retry attempt number (1 = first retry, i.e. second run total). |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRunRetry.json)

### `FlowRunSpec`

Defines a set of tasks to be executed in a sequence.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `target` | [`ResourceHandle`](#resourcehandle) | | | Defines the default target resources on which tasks will be performed. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the tasks of this run execute with. Must be an account of service account type. A run created by the flow controller copies it from the `Flow`. |
| `tasks` | `array(`[`TaskSpec`](#taskspec)`)` | ✔️ | | List of tasks to run consecutively. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRunSpec.json)

### `FlowRunSpecInput`

Defines a set of tasks to be executed in a sequence.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `target` | [`ResourceRef`](#resourceref) | | | Defines the default target resources on which tasks will be performed. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the tasks of this run execute with. Must be an account of service account type. A run created by the flow controller copies it from the `Flow`. |
| `tasks` | `array(`[`TaskSpecInput`](#taskspecinput)`)` | ✔️ | | List of tasks to run consecutively. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRunSpecInput.json)

### `FlowRunStatus`

Condition tracking the overall execution status of a FlowRun and its spawned tasks.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `status` | [`Value`](#value) | ✔️ | | Overall execution status of the FlowRun. |
| `tasks` | `array(`[`TaskEntry`](#taskentry)`)` | | | Tasks spawned by this FlowRun, in execution order. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowRunStatus.json)

### `FlowSpec`

Defines a sequence of tasks to be executed upon certain trigger conditions.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `target` | [`ResourceSelector`](#resourceselector) | ✔️ | | Defines resources for which this flow will be instantiated. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the tasks of this flow execute with. Must be an account of service account type. Required for flows owned by an organization. When omitted, tasks execute with the permissions of the user account that owns the flow. |
| `triggers` | `array(`[`FlowTrigger`](#flowtrigger)`)` | ✔️ | | Conditions that cause this flow to execute. |
| `tasks` | `array(`[`TaskSpec`](#taskspec)`)` | ✔️ | | List of tasks to run consecutively. |
| `retryPolicy` | [`RetryPolicy`](#retrypolicy) | | | Defines how a flow should react to failures. |
| `recentBindingsRetention` | [`RecentBindingsRetention`](#recentbindingsretention) | | | Controls retention of `FlowStatus.recentBindings`. |
| `recentRunsRetention` | [`RecentRunsRetention`](#recentrunsretention) | | | Controls retention of `FlowStatus.recentRuns`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowSpec.json)

### `FlowSpecInput`

Defines a sequence of tasks to be executed upon certain trigger conditions.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `target` | [`ResourceSelector`](#resourceselector) | ✔️ | | Defines resources for which this flow will be instantiated. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the tasks of this flow execute with. Must be an account of service account type. Required for flows owned by an organization. When omitted, tasks execute with the permissions of the user account that owns the flow. |
| `triggers` | `array(`[`FlowTriggerInput`](#flowtriggerinput)`)` | ✔️ | | Conditions that cause this flow to execute. |
| `tasks` | `array(`[`TaskSpecInput`](#taskspecinput)`)` | ✔️ | | List of tasks to run consecutively. |
| `retryPolicy` | [`RetryPolicy`](#retrypolicy) | | | Defines how a flow should react to failures. |
| `recentBindingsRetention` | [`RecentBindingsRetention`](#recentbindingsretention) | | | Controls retention of `FlowStatus.recentBindings`. |
| `recentRunsRetention` | [`RecentRunsRetention`](#recentrunsretention) | | | Controls retention of `FlowStatus.recentRuns`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowSpecInput.json)

### `FlowStatus`

Condition tracking the overall status of a Flow and a summary of recent runs.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `status` | [`Value`](#value) | ✔️ | | Overall operational status of the Flow. |
| `recentBindings` | `array(`[`BindingEntry`](#bindingentry)`)` | | | Last N resources matched by the flow's target selector. Bounded by spec.recentBindingsRetention. |
| `bindingsTotal` | `integer` | | `uint64` | Total number of resources currently matched by the flow's target selector. |
| `recentRuns` | `array(`[`RunEntry`](#runentry)`)` | | | Summaries of recent FlowRuns. Bounded by spec.recentRunsRetention. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowStatus.json)

### `FlowTrigger`

Condition that causes a flow to be executed.

| Union Type | Description |
| - | - |
| [`FlowTrigger::Manual`](#flowtriggermanual) | Triggers the flow via an API call or UI action. |
| [`FlowTrigger::Schedule`](#flowtriggerschedule) | Triggers the flow on a cron schedule. |
| [`FlowTrigger::Interval`](#flowtriggerinterval) | Triggers the flow at regular intervals. |
| [`FlowTrigger::Event`](#flowtriggerevent) | Triggers the flow when an event bus event matching one of the filters is observed. |
| [`FlowTrigger::Source`](#flowtriggersource) | Triggers the flow when a source receives new data, with optional batching controls. |
| [`FlowTrigger::InputsUpdated`](#flowtriggerinputsupdated) | Triggers the flow when inputs of a derivative dataset receive new data, with optional batching controls. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTrigger::Manual`

Triggers the flow via an API call or UI action.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTrigger::Schedule`

Triggers the flow on a cron schedule.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `cron` | `string` | ✔️ | | Cron5 expression defining the schedule e.g. `@daily` or `*/30 * * * *`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTrigger::Interval`

Triggers the flow at regular intervals.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `interval` | `string` | ✔️ | `duration` | Interval between activations e.g. `15m`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTrigger::Event`

Triggers the flow when an event bus event matching one of the filters is observed.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `events` | [`EventFilter`](#eventfilter) | ✔️ | | Filters the event by type and fields. |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTrigger::Source`

Triggers the flow when a source receives new data, with optional batching controls.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `source` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Reference to the source resource that drives this trigger. |
| `minRecordsToAwait` | `integer` | | `uint64` | Minimum number of new records to accumulate before triggering. |
| `maxAwaitInterval` | `string` | | `duration` | Maximum time to wait for `minRecordsToAwait` before triggering anyway e.g. `1h`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTrigger::InputsUpdated`

Triggers the flow when inputs of a derivative dataset receive new data, with optional batching controls.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `target` | [`DatasetSelector`](#datasetselector) | | | Selector of derivative datasets whose inputs are watched. Defaults to the flow-level target when omitted. |
| `minRecordsToAwait` | `integer` | | `uint64` | Minimum number of new input records to accumulate before triggering. |
| `maxAwaitInterval` | `string` | | `duration` | Maximum time to wait for `minRecordsToAwait` before triggering anyway e.g. `1h`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTrigger.json)

### `FlowTriggerInput`

Condition that causes a flow to be executed.

| Union Type | Description |
| - | - |
| [`FlowTriggerInput::Manual`](#flowtriggerinputmanual) | Triggers the flow via an API call or UI action. |
| [`FlowTriggerInput::Schedule`](#flowtriggerinputschedule) | Triggers the flow on a cron schedule. |
| [`FlowTriggerInput::Interval`](#flowtriggerinputinterval) | Triggers the flow at regular intervals. |
| [`FlowTriggerInput::Event`](#flowtriggerinputevent) | Triggers the flow when an event bus event matching one of the filters is observed. |
| [`FlowTriggerInput::Source`](#flowtriggerinputsource) | Triggers the flow when a source receives new data, with optional batching controls. |
| [`FlowTriggerInput::InputsUpdated`](#flowtriggerinputinputsupdated) | Triggers the flow when inputs of a derivative dataset receive new data, with optional batching controls. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `FlowTriggerInput::Manual`

Triggers the flow via an API call or UI action.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `FlowTriggerInput::Schedule`

Triggers the flow on a cron schedule.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `cron` | `string` | ✔️ | | Cron5 expression defining the schedule e.g. `@daily` or `*/30 * * * *`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `FlowTriggerInput::Interval`

Triggers the flow at regular intervals.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `interval` | `string` | ✔️ | `duration` | Interval between activations e.g. `15m`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `FlowTriggerInput::Event`

Triggers the flow when an event bus event matching one of the filters is observed.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `events` | [`EventFilter`](#eventfilter) | ✔️ | | Filters the event by type and fields. |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `FlowTriggerInput::Source`

Triggers the flow when a source receives new data, with optional batching controls.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `source` | [`ResourceRef`](#resourceref) | ✔️ | | Reference to the source resource that drives this trigger. |
| `minRecordsToAwait` | `integer` | | `uint64` | Minimum number of new records to accumulate before triggering. |
| `maxAwaitInterval` | `string` | | `duration` | Maximum time to wait for `minRecordsToAwait` before triggering anyway e.g. `1h`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `FlowTriggerInput::InputsUpdated`

Triggers the flow when inputs of a derivative dataset receive new data, with optional batching controls.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `enabled` | `boolean` | | | Allows to pause an individual trigger.<br /><br />Default: `True` |
| `cooldown` | `string` | | `duration` | Minimum interval between activations of the trigger. If the trigger fires again within the interval, the activation is postponed until the interval ends, batching multiple activations into one flow run. |
| `cooldownMaxBatch` | `integer` | | `uint64` | If the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end. |
| `target` | [`DatasetSelector`](#datasetselector) | | | Selector of derivative datasets whose inputs are watched. Defaults to the flow-level target when omitted. |
| `minRecordsToAwait` | `integer` | | `uint64` | Minimum number of new input records to accumulate before triggering. |
| `maxAwaitInterval` | `string` | | `duration` | Maximum time to wait for `minRecordsToAwait` before triggering anyway e.g. `1h`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/FlowTriggerInput.json)

### `RecentBindingsRetention`

Controls how many recent bindings are reported in `FlowStatus.recentBindings`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `maxBindings` | `integer` | | `uint32` | Maximum number of recent bindings to report. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/RecentBindingsRetention.json)

### `RecentRunsRetention`

Controls how many recent runs are reported in `FlowStatus.recentRuns`. Implementations may additionally cap these values.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `maxRuns` | `integer` | | `uint32` | Maximum number of recent runs to report. |
| `maxAge` | `string` | | `duration` | Maximum age of a finished run to be reported e.g. `30d`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/RecentRunsRetention.json)

### `RetryBackoff`

Type of the backoff scaling.

| Enum Value |
| :-: |
| `Linear` |
| `Exponential` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/RetryBackoff.json)

### `RetryPolicy`

Defines how a flow should react to failures.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `maxAttempts` | `integer` | | `uint32` | Number of attempts before flow auto-scheduling will be disabled. |
| `minDelay` | `string` | | `duration` | How long to wait until the first retry. |
| `backoff` | [`RetryBackoff`](#retrybackoff) | | | Type of the backoff scaling. When omitted, retries use a constant `minDelay`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/flows/v1alpha1/RetryPolicy.json)

## Resources

### `LabelFilter`

Filters that work on resource labels and identity headers.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/LabelFilter.json)

### `ReconciliationError`

Condition describing why the controller failed to reconcile the latest generation of a resource. Set by the controller; never written by users.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `code` | `string` | ✔️ | | Machine-readable code identifying the type of the error. |
| `message` | `string` | ✔️ | | Human-readable description of the error. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ReconciliationError.json)

### `Resource`

Top-level container for canonical representation of a resource that specifies the type and version of the resource, carries identity, ownership, and status information.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies the controlling entity, a bounded context that this resource belongs to, and the version. Url should follow the pattern `{base-url}/{context}/{version}/{name}` e.g. `https://opendatafabric.org/schemas/datasets/v1/Dataset`. |
| `headers` | [`ResourceHeaders`](#resourceheaders) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | `object` | ✔️ | `fragment` | Specifies the desired state of a resource. |
| `status` | [`ResourceStatus`](#resourcestatus) | ✔️ | | Resource lifecycle and reconciliation information. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/Resource.json)

### `ResourceAnnotations`

Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. Unlike labels, annotations are not indexed and cannot be queried by.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceAnnotations.json)

### `ResourceConditions`

Container of generic contditions that can be added by contollers to provide additional information about the state of a resource. Keys uniquely identify the condition and should be in the form of a schema URL describing this condition, e.g. `https://opendatafabric.org/schemas/tasks/v1/TaskStatus`.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceConditions.json)

### `ResourceHandle`

Link to another resolved resource.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerHandle`](#resourceownerhandle) | ✔️ | | Account that owns the target resource. |
| `type` | `string` | ✔️ | `type-uri` | Type URI of the target resource. |
| `id` | `string` | ✔️ | `resource-id` | ID of the resource within a node. |
| `did` | `string` | | `did` | DID of the resource, if applicable. |
| `name` | `string` | ✔️ | `resource-name` | Name of a resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceHandle.json)

### `ResourceHeaders`

Container for identity and ownership information of a resource.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `id` | `string` | ✔️ | `resource-id` | Unique identifier of a resource within entire ODF node. Automatically assigned upon resource creation. |
| `name` | `string` | ✔️ | `resource-name` | Symbolic name of a resource that identifies it within a scope of an onwing account. |
| `account` | [`ResourceOwnerHandle`](#resourceownerhandle) | ✔️ | | Link to the account that owns the resource. |
| `labels` | [`ResourceLabels`](#resourcelabels) | ✔️ | | Map of string keys and values that can be used to organize, categorize, and query resources. |
| `annotations` | [`ResourceAnnotations`](#resourceannotations) | ✔️ | | Annotations is a key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. Unlike labels, annotations are not indexed and cannot be queried by. |
| `ownerReferences` | `array(`[`ResourceHandle`](#resourcehandle)`)` | | | References to resources that created this resource. Used for lineage tracking and cascading cleanup. |
| `generation` | `integer` | ✔️ | `uint64` | A sequential number that changes every time the resource header and spec are updated. Does not increment on status changes, thus signifying changes to the desired state. Populated by the system. Starts with `1`. |
| `statusGeneration` | `integer` | ✔️ | `uint64` | A sequential number that changes every time the resource status is updated. Can be passed back as a precondition of a status update to make sure the status was not modified since it was read. Populated by the system. Starts with `1`. |
| `createdAt` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time when the resource was first applied and assigned an identity. |
| `updatedAt` | `string` | ✔️ | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time when the resource was last updated, including header, spec, and status updates. |
| `deletionRequestedAt` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time when deletion of the resource was requested. The resource remains visible until its controllers finish cleaning up and the deletion completes. |
| `deletedAt` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time when the resource was deleted. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceHeaders.json)

### `ResourceHeadersInput`

Container for identity and ownership information of a resource.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `id` | `string` | | `resource-id` | Unique identifier of a resource within entire ODF node. Automatically assigned upon resource creation. |
| `name` | `string` | ✔️ | `resource-name` | Symbolic name of a resource that identifies it within a scope of an onwing account. |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to the account that owns the resource. |
| `labels` | [`ResourceLabels`](#resourcelabels) | | | Map of string keys and values that can be used to organize, categorize, and query resources. |
| `annotations` | [`ResourceAnnotations`](#resourceannotations) | | | Annotations is a key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. Unlike labels, annotations are not indexed and cannot be queried by. |
| `ownerReferences` | `array(`[`ResourceRef`](#resourceref)`)` | | | References to resources that created this resource. Used for lineage tracking and cascading cleanup. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceHeadersInput.json)

### `ResourceInput`

Top-level container for user-authored representation of a resource that specifies the type and version of the resource and its desired state.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies the controlling entity, a bounded context that this resource belongs to, and the version. Url should follow the pattern `{base-url}/{context}/{version}/{name}` e.g. `https://opendatafabric.org/schemas/datasets/v1/Dataset`. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | `object` | ✔️ | `fragment` | Specifies the desired state of a resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceInput.json)

### `ResourceLabels`

Map of string keys and values that can be used to organize, categorize, and query resources.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceLabels.json)

### `ResourcePhase`

Represents the reconciliation or deletion phase of a resource.

| Enum Value |
| :-: |
| `Pending` |
| `Reconciling` |
| `Ready` |
| `Degraded` |
| `Failed` |
| `Deleting` |
| `Deleted` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourcePhase.json)

### `ResourceRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the target resource. |
| `id` | `string` | | `resource-id` | ID of the resource within a node. |
| `did` | `string` | | `did` | DID of the resource. |
| `type` | `string` | | `type-ref` | Short type name or full type URI of the target resource. |
| `name` | `string` | | `resource-name` | Name of a resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceRef.json)

### `ResourceSelector`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the target resources. |
| `id` | `string` | | `resource-id` | ID of the singular resource. |
| `did` | `string` | | `did` | DID of the resource. |
| `type` | `string` | | `type-ref` | Short type name of the target resource e.g. `SecretSet` or a full schema URI e.g. `https://opendatafabric.org/schemas/config/v1alpha1/SecretSet`. |
| `name` | `string` | | | Name pattern in SQL `LIKE` format. |
| `labels` | [`LabelFilter`](#labelfilter) | | | Filter by resource labels. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceSelector.json)

### `ResourceStatus`

Resource lifecycle and reconciliation information.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `phase` | [`ResourcePhase`](#resourcephase) | ✔️ | | Represents the reconciliation phase of a resource as seen by the main resource controller. |
| `observedGeneration` | `integer` | | `uint64` | Resource generation that was last seen by the main resource controller. |
| `observedAt` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time when the controller seen the resource state as defined in `observedGeneration`. |
| `reconciledGeneration` | `integer` | | `uint64` | Resource generation that was last successfully reconciled by the main resource controller. |
| `reconciledAt` | `string` | | [`date-time`](https://json-schema.org/draft/2019-09/json-schema-validation.html#rfc.section.7.3.1) | Time when the controller last reconciled the desired resource state as defined in `reconciledGeneration`. |
| `conditions` | [`ResourceConditions`](#resourceconditions) | ✔️ | | Detailed conditions describing the state of the resource that are added by controllers. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/ResourceStatus.json)

### `AddLabel`

Apply a specific label to the resource.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `labelType` | `string` | ✔️ | `type-uri` | Schema URI of the label being added. |
| `labelValue` | `object` | ✔️ | | Value of the label being added. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/AddLabel.json)

### `Create`

Create a new resource instance of a type (object is a type URI) under an account.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `targetOwner` | [`ResourceOwnerHandle`](#resourceownerhandle) | ✔️ | | Account under which the resource is being created. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/Create.json)

### `Delete`

Delete the resource instance.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/Delete.json)

### `Edit`

Modify the resource manifest.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/Edit.json)

### `List`

List resource instances of a type (object is a type URI) under an account.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `targetOwner` | [`ResourceOwnerHandle`](#resourceownerhandle) | ✔️ | | Account under which resources are being listed. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/List.json)

### `RemoveLabel`

Remove a specific label from the resource.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `labelType` | `string` | ✔️ | `type-uri` | Schema URI of the label being removed. |
| `labelValue` | `object` | ✔️ | | Existing value of the label being removed. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/RemoveLabel.json)

### `View`

Read the resource manifest.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/actions/View.json)

## Sinks

### `WebhookEndpoint`

Defines a webhook endpoint that can receive event notifications and data.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`WebhookEndpointSpecInput`](#webhookendpointspecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sinks/v1alpha1/WebhookEndpoint.json)

### `WebhookEndpointSpec`

Defines a webhook endpoint that can receive event notifications and data.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `url` | `string` | ✔️ | `uri` | URL of the webhook endpoint. |
| `secret` | [`Secret`](#secret) | | | Shared secret used for HMAC signature of the request payload for authentication. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sinks/v1alpha1/WebhookEndpointSpec.json)

### `WebhookEndpointSpecInput`

Defines a webhook endpoint that can receive event notifications and data.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `url` | `string` | ✔️ | `uri` | URL of the webhook endpoint. |
| `secret` | [`Secret`](#secret) | | | Shared secret used for HMAC signature of the request payload for authentication. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sinks/v1alpha1/WebhookEndpointSpecInput.json)

## Sources

### `CompressionFormat`

Defines a compression algorithm.

| Enum Value |
| :-: |
| `Gzip` |
| `Zip` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/CompressionFormat.json)

### `EnvVar`

Defines an environment variable passed into some job.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | ✔️ | | Name of the variable. |
| `value` | `string` | | | Value of the variable. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/EnvVar.json)

### `EventTimeSource`

Defines the external source of data.

| Union Type | Description |
| - | - |
| [`EventTimeSource::FromMetadata`](#eventtimesourcefrommetadata) | Extracts event time from the source's metadata. |
| [`EventTimeSource::FromPath`](#eventtimesourcefrompath) | Extracts event time from the path component of the source. |
| [`EventTimeSource::FromSystemTime`](#eventtimesourcefromsystemtime) | Assigns event time from the system time source. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/EventTimeSource.json)

### `EventTimeSource::FromMetadata`

Extracts event time from the source's metadata.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/EventTimeSource.json)

### `EventTimeSource::FromSystemTime`

Assigns event time from the system time source.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/EventTimeSource.json)

### `EventTimeSource::FromPath`

Extracts event time from the path component of the source.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `pattern` | `string` | ✔️ | `regex` | Regular expression where first group contains the timestamp string. |
| `timestampFormat` | `string` | | | Format of the expected timestamp in java.text.SimpleDateFormat form. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/EventTimeSource.json)

### `Ingress`

Defines the point where data enters the system.

| Union Type | Description |
| - | - |
| [`Ingress::Url`](#ingressurl) | Pulls data from one of the supported sources by its URL. |
| [`Ingress::FilesGlob`](#ingressfilesglob) | Uses glob operator to match files on the local file system. |
| [`Ingress::Container`](#ingresscontainer) | Runs the specified OCI container to fetch data from an arbitrary source. |
| [`Ingress::Mqtt`](#ingressmqtt) | Connects to an MQTT broker to fetch events from the specified topic. |
| [`Ingress::EvmLogs`](#ingressevmlogs) | Connects to an EVM (Ethereum) node to stream transaction logs. |
| [`Ingress::RestEndpoint`](#ingressrestendpoint) | Exposes a REST HTTP endpoint that accepts pushed data records. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `Ingress::Url`

Pulls data from one of the supported sources by its URL.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `url` | `string` | ✔️ | `uri` | URL of the data source |
| `eventTime` | [`EventTimeSource`](#eventtimesource) | | | Describes how event time is extracted from the source metadata. |
| `cache` | [`SourceCaching`](#sourcecaching) | | | Describes the caching settings used for this source. |
| `headers` | `array(`[`RequestHeader`](#requestheader)`)` | | | Headers to pass during the request (e.g. HTTP Authorization) |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `Ingress::FilesGlob`

Uses glob operator to match files on the local file system.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `path` | `string` | ✔️ | | Path with a glob pattern. |
| `eventTime` | [`EventTimeSource`](#eventtimesource) | | | Describes how event time is extracted from the source metadata. |
| `cache` | [`SourceCaching`](#sourcecaching) | | | Describes the caching settings used for this source. |
| `order` | [`SourceOrdering`](#sourceordering) | | | Specifies how input files should be ordered before ingestion.<br />Order is important as every file will be processed individually<br />and will advance the dataset's watermark. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `Ingress::Container`

Runs the specified OCI container to fetch data from an arbitrary source.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `image` | `string` | ✔️ | | Image name and and an optional tag. |
| `command` | `array(string)` | | | Specifies the entrypoint. Not executed within a shell. The default OCI image's ENTRYPOINT is used if this is not provided. |
| `args` | `array(string)` | | | Arguments to the entrypoint. The OCI image's CMD is used if this is not provided. |
| `env` | `array(`[`EnvVar`](#envvar)`)` | | | Environment variables to propagate into or set in the container. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `Ingress::Mqtt`

Connects to an MQTT broker to fetch events from the specified topic.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `host` | `string` | ✔️ | | Hostname of the MQTT broker. |
| `port` | `integer` | ✔️ | `int32` | Port of the MQTT broker. |
| `username` | `string` | | | Username to use for auth with the broker. |
| `password` | `string` | | | Password to use for auth with the broker (can be templated). |
| `topics` | `array(`[`MqttTopicSubscription`](#mqtttopicsubscription)`)` | ✔️ | | List of topic subscription parameters. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `Ingress::RestEndpoint`

Exposes a REST HTTP endpoint that accepts pushed data records.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `buffer` | [`IngressBuffer`](#ingressbuffer) | | | Buffer configuration for holding records until they are ingested. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `Ingress::EvmLogs`

Connects to an EVM (Ethereum) node to stream transaction logs.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `chainId` | `integer` | | `uint64` | Identifier of the chain to scan logs from. This parameter may be used for RPC endpoint lookup as well as asserting that provided `nodeUrl` corresponds to the expected chain. |
| `nodeUrl` | `string` | | `uri` | Url of the node. |
| `filter` | `string` | | | An SQL WHERE clause that can be used to pre-filter the logs before fetching them from the ETH node. |
| `signature` | `string` | | | Solidity log event signature to use for decoding. Using this field adds `event` to the output containing decoded log as JSON. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Ingress.json)

### `IngressBuffer`

Buffer configuration for holding pushed records until they are ingested.

| Union Type | Description |
| - | - |
| [`IngressBuffer::Memory`](#ingressbuffermemory) | An in-memory buffer. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/IngressBuffer.json)

### `IngressBuffer::Memory`

An in-memory buffer.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `bufferSize` | `integer` | | `uint64` | Maximum number of records to hold in the buffer. |
| `overflowPolicy` | `string` | | | Policy applied when the buffer is full. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/IngressBuffer.json)

### `MergeStrategy`

Merge strategy determines how newly ingested data should be combined with the data that already exists in the dataset.

| Union Type | Description |
| - | - |
| [`MergeStrategy::Append`](#mergestrategyappend) | Append merge strategy. |
| [`MergeStrategy::Ledger`](#mergestrategyledger) | Ledger merge strategy. |
| [`MergeStrategy::Snapshot`](#mergestrategysnapshot) | Snapshot merge strategy. |
| [`MergeStrategy::ChangelogStream`](#mergestrategychangelogstream) | Changelog stream merge strategy. |
| [`MergeStrategy::UpsertStream`](#mergestrategyupsertstream) | Upsert stream merge strategy. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MergeStrategy.json)

### `MergeStrategy::Append`

Append merge strategy.

Under this strategy new data will be appended to the dataset in its entirety, without any deduplication.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MergeStrategy.json)

### `MergeStrategy::ChangelogStream`

Changelog stream merge strategy.

This is the native stream format for ODF that accurately describes the evolution of all event records including appends, retractions, and corrections as per RFC-015. No pre-processing except for format validation is done.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `primaryKey` | `array(string)` | ✔️ | | Names of the columns that uniquely identify the record throughout its lifetime |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MergeStrategy.json)

### `MergeStrategy::Ledger`

Ledger merge strategy.

This strategy should be used for data sources containing ledgers of events. Currently this strategy will only perform deduplication of events using user-specified primary key columns. This means that the source data can contain partially overlapping set of records and only those records that were not previously seen will be appended.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `primaryKey` | `array(string)` | ✔️ | | Names of the columns that uniquely identify the record throughout its lifetime |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MergeStrategy.json)

### `MergeStrategy::Snapshot`

Snapshot merge strategy.

This strategy can be used for data state snapshots that are taken periodically and contain only the latest state of the observed entity or system. Over time such snapshots can have new rows added, and old rows either removed or modified.

This strategy transforms snapshot data into an append-only event stream where data already added is immutable. It does so by performing Change Data Capture - essentially diffing the current state of data against the reconstructed previous state and recording differences as retractions or corrections. The Operation Type "op" column will contain:

* append (`+A`) when a row appears for the first time
* retraction (`-R`) when row disappears
* correction (`-C`, `+C`) when row data has changed, with `-C` event carrying the old value of the row and `+C` carrying the new value.

To correctly associate rows between old and new snapshots this strategy relies on user-specified primary key columns.

To identify whether a row has changed this strategy will compare all other columns one by one. If the data contains a column that is guaranteed to change whenever any of the data columns changes (for example a last modification timestamp, an incremental version, or a data hash), then it can be specified in `compareColumns` property to speed up the detection of modified rows.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `primaryKey` | `array(string)` | ✔️ | | Names of the columns that uniquely identify the record throughout its lifetime. |
| `compareColumns` | `array(string)` | | | Names of the columns to compared to determine if a row has changed between two snapshots. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MergeStrategy.json)

### `MergeStrategy::UpsertStream`

Upsert stream merge strategy.

This strategy should be used for data sources containing ledgers of insert-or-update and delete events. Unlike ChangelogStream the insert-or-update events only carry the new values, so this strategy will use primary key to re-classify the events into an append or a correction from/to pair, looking up the previous values.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `primaryKey` | `array(string)` | ✔️ | | Names of the columns that uniquely identify the record throughout its lifetime |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MergeStrategy.json)

### `MqttQos`

MQTT quality of service class.

| Enum Value |
| :-: |
| `AtMostOnce` |
| `AtLeastOnce` |
| `ExactlyOnce` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MqttQos.json)

### `MqttTopicSubscription`

MQTT topic subscription parameters.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `path` | `string` | ✔️ | | Name of the topic (may include patterns). |
| `qos` | [`MqttQos`](#mqttqos) | | | Quality of service class.<br /><br />Default: `AtMostOnce` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/MqttTopicSubscription.json)

### `PrepStep`

Defines the steps to prepare raw data for ingestion.

| Union Type | Description |
| - | - |
| [`PrepStep::Decompress`](#prepstepdecompress) | Pulls data from one of the supported sources by its URL. |
| [`PrepStep::Pipe`](#prepsteppipe) | Executes external command to process the data using piped input/output. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/PrepStep.json)

### `PrepStep::Decompress`

Pulls data from one of the supported sources by its URL.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `format` | [`CompressionFormat`](#compressionformat) | ✔️ | | Name of a compression algorithm used on data. |
| `subPath` | `string` | | | Path to a data file within a multi-file archive. Can contain glob patterns. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/PrepStep.json)

### `PrepStep::Pipe`

Executes external command to process the data using piped input/output.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `command` | `array(string)` | ✔️ | | Command to execute and its arguments. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/PrepStep.json)

### `ReadStep`

Defines how raw data should be read into the structured form.

| Union Type | Description |
| - | - |
| [`ReadStep::Csv`](#readstepcsv) | Reader for comma-separated files. |
| [`ReadStep::GeoJson`](#readstepgeojson) | Reader for GeoJSON files. It expects one `FeatureCollection` object in the root and will create a record per each `Feature` inside it extracting the properties into individual columns and leaving the feature geometry in its own column. |
| [`ReadStep::EsriShapefile`](#readstepesrishapefile) | Reader for ESRI Shapefile format. |
| [`ReadStep::Parquet`](#readstepparquet) | Reader for Apache Parquet format. |
| [`ReadStep::Json`](#readstepjson) | Reader for JSON files that contain an array of objects within them. |
| [`ReadStep::NdJson`](#readstepndjson) | Reader for files containing multiple newline-delimited JSON objects with the same schema. |
| [`ReadStep::NdGeoJson`](#readstepndgeojson) | Reader for Newline-delimited GeoJSON files. It is similar to `GeoJson` format but instead of `FeatureCollection` object in the root it expects every individual feature object to appear on its own line. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::Csv`

Reader for comma-separated files.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `separator` | `string` | | | Sets a single character as a separator for each field and value.<br /><br />Default: `,` |
| `encoding` | `string` | | | Decodes the CSV files by the given encoding type.<br /><br />Default: `utf8` |
| `quote` | `string` | | | Sets a single character used for escaping quoted values where the separator can be part of the value. Set an empty string to turn off quotations.<br /><br />Default: `"` |
| `escape` | `string` | | | Sets a single character used for escaping quotes inside an already quoted value.<br /><br />Default: `\` |
| `header` | `boolean` | | | Use the first line as names of columns. |
| `inferSchema` | `boolean` | | | Infers the input schema automatically from data. It requires one extra pass over the data. |
| `nullValue` | `string` | | | Sets the string representation of a null value. |
| `dateFormat` | `string` | | | Sets the string that indicates a date format. The `rfc3339` is the only required format, the other format strings are implementation-specific.<br /><br />Default: `rfc3339` |
| `timestampFormat` | `string` | | | Sets the string that indicates a timestamp format. The `rfc3339` is the only required format, the other format strings are implementation-specific.<br /><br />Default: `rfc3339` |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::Json`

Reader for JSON files that contain an array of objects within them.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `subPath` | `string` | | | Path in the form of `a.b.c` to a sub-element of the root JSON object that is an array or objects. If not specified it is assumed that the root element is an array. |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `dateFormat` | `string` | | | Sets the string that indicates a date format. The `rfc3339` is the only required format, the other format strings are implementation-specific.<br /><br />Default: `rfc3339` |
| `encoding` | `string` | | | Allows to forcibly set one of standard basic or extended encodings.<br /><br />Default: `utf8` |
| `timestampFormat` | `string` | | | Sets the string that indicates a timestamp format. The `rfc3339` is the only required format, the other format strings are implementation-specific.<br /><br />Default: `rfc3339` |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::NdJson`

Reader for files containing multiple newline-delimited JSON objects with the same schema.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `dateFormat` | `string` | | | Sets the string that indicates a date format. The `rfc3339` is the only required format, the other format strings are implementation-specific.<br /><br />Default: `rfc3339` |
| `encoding` | `string` | | | Allows to forcibly set one of standard basic or extended encodings.<br /><br />Default: `utf8` |
| `timestampFormat` | `string` | | | Sets the string that indicates a timestamp format. The `rfc3339` is the only required format, the other format strings are implementation-specific.<br /><br />Default: `rfc3339` |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::GeoJson`

Reader for GeoJSON files. It expects one `FeatureCollection` object in the root and will create a record per each `Feature` inside it extracting the properties into individual columns and leaving the feature geometry in its own column.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::NdGeoJson`

Reader for Newline-delimited GeoJSON files. It is similar to `GeoJson` format but instead of `FeatureCollection` object in the root it expects every individual feature object to appear on its own line.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::EsriShapefile`

Reader for ESRI Shapefile format.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `subPath` | `string` | | | If the ZIP archive contains multiple shapefiles use this field to specify a sub-path to the desired `.shp` file. Can contain glob patterns to act as a filter. |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `ReadStep::Parquet`

Reader for Apache Parquet format.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `ddlSchema` | `array(string)` | | | DEPRECATED: A DDL-formatted schema. Schema can be used to coerce values into more appropriate data types. |
| `schema` | [`DataSchema`](#dataschema) | | | Schema used to coerce values into more appropriate data types. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/ReadStep.json)

### `RequestHeader`

Defines a header (e.g. HTTP) to be passed into some request.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | ✔️ | | Name of the header. |
| `value` | `string` | ✔️ | | Value of the header. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/RequestHeader.json)

### `Source`

Defines an external source of data for ingestion.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`SourceSpecInput`](#sourcespecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/Source.json)

### `SourceCaching`

Defines how external data should be cached.

| Union Type | Description |
| - | - |
| [`SourceCaching::Forever`](#sourcecachingforever) | After source was processed once it will never be ingested again. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/SourceCaching.json)

### `SourceCaching::Forever`

After source was processed once it will never be ingested again.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/SourceCaching.json)

### `SourceOrdering`

Specifies how input files should be ordered before ingestion.

| Enum Value |
| :-: |
| `ByEventTime` |
| `ByName` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/SourceOrdering.json)

### `SourceSpec`

Specifies an external source of data for ingestion.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `config` | [`ValueRefs`](#valuerefs) | | | Brings the configuration values into the local `config` context. |
| `ingress` | [`Ingress`](#ingress) | | | Determines where data is sourced from. |
| `prepare` | `array(`[`PrepStep`](#prepstep)`)` | | | Defines how raw data is prepared before reading. |
| `read` | [`ReadStep`](#readstep) | ✔️ | | Defines how data is read into structured format. |
| `preprocess` | [`Transform`](#transform) | | | Pre-processing query that shapes the data. |
| `merge` | [`MergeStrategy`](#mergestrategy) | | | Determines how newly-ingested data should be merged with existing history. |
| `vocab` | [`DatasetVocabulary`](#datasetvocabulary) | | | Defines the mapping of system fields to dataset column names. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/SourceSpec.json)

### `SourceSpecInput`

Specifies an external source of data for ingestion.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `config` | [`ValueRefs`](#valuerefs) | | | Brings the configuration values into the local `config` context. |
| `ingress` | [`Ingress`](#ingress) | | | Determines where data is sourced from. |
| `prepare` | `array(`[`PrepStep`](#prepstep)`)` | | | Defines how raw data is prepared before reading. |
| `read` | [`ReadStep`](#readstep) | ✔️ | | Defines how data is read into structured format. |
| `preprocess` | [`Transform`](#transform) | | | Pre-processing query that shapes the data. |
| `merge` | [`MergeStrategy`](#mergestrategy) | | | Determines how newly-ingested data should be merged with existing history. |
| `vocab` | [`DatasetVocabulary`](#datasetvocabulary) | | | Defines the mapping of system fields to dataset column names. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/SourceSpecInput.json)

### `SourceState`

The state of the source the data was added from to allow fast resuming.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `sourceName` | `string` | ✔️ | | Identifies the source that the state corresponds to. |
| `kind` | `string` | ✔️ | | Identifies the type of the state. Standard types include: `odf/etag`, `odf/last-modified`. |
| `value` | `string` | ✔️ | | Opaque value representing the state. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/sources/v1alpha1/SourceState.json)

## Storage

### `AwsCredentials`

Access credentials for AWS or an AWS-compatible service.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `accessKey` | [`ValueHandle`](#valuehandle) | | | Reference to a secret containing the AWS access key ID. |
| `secretKey` | [`ValueHandle`](#valuehandle) | | | Reference to a secret containing the AWS secret access key. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/AwsCredentials.json)

### `AwsCredentialsInput`

Access credentials for AWS or an AWS-compatible service.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `accessKey` | [`ValueRef`](#valueref) | | | Reference to a secret containing the AWS access key ID. |
| `secretKey` | [`ValueRef`](#valueref) | | | Reference to a secret containing the AWS secret access key. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/AwsCredentialsInput.json)

### `PersistentVolume`

Defines a storage volume where data can be stored and its access credentials.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`PersistentVolumeSpecInput`](#persistentvolumespecinput) | ✔️ | | Specifies the desired state of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/PersistentVolume.json)

### `PersistentVolumeRef`

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `account` | [`ResourceOwnerRef`](#resourceownerref) | | | Reference to an account that owns the `PersistentVolume`. |
| `id` | `string` | | `resource-id` | ID of the resource. |
| `name` | `string` | | `resource-name` | Name of the resource. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/PersistentVolumeRef.json)

### `PersistentVolumeSpec`

Defines a storage volume where data can be stored and its access credentials.

| Union Type | Description |
| - | - |
| [`PersistentVolumeSpec::S3`](#persistentvolumespecs3) | An Amazon S3 or S3-compatible object storage bucket. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/PersistentVolumeSpec.json)

### `PersistentVolumeSpec::S3`

An Amazon S3 or S3-compatible object storage bucket.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `endpoint` | `string` | | `uri` | S3 endpoint URL. If omitted, defaults to AWS S3. Use for S3-compatible stores e.g. `https://s3.amazonaws.com`. |
| `region` | `string` | | | AWS region where the bucket is located e.g. `us-west-2`. |
| `bucket` | `string` | ✔️ | | Name of the S3 bucket. |
| `prefix` | `string` | | | Optional path prefix within the bucket. |
| `capacity` | [`VolumeCapacity`](#volumecapacity) | | | Storage capacity allocation. |
| `credentials` | [`AwsCredentials`](#awscredentials) | | | Access credentials for the bucket. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/PersistentVolumeSpec.json)

### `PersistentVolumeSpecInput`

Defines a storage volume where data can be stored and its access credentials.

| Union Type | Description |
| - | - |
| [`PersistentVolumeSpecInput::S3`](#persistentvolumespecinputs3) | An Amazon S3 or S3-compatible object storage bucket. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/PersistentVolumeSpecInput.json)

### `PersistentVolumeSpecInput::S3`

An Amazon S3 or S3-compatible object storage bucket.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `endpoint` | `string` | | `uri` | S3 endpoint URL. If omitted, defaults to AWS S3. Use for S3-compatible stores e.g. `https://s3.amazonaws.com`. |
| `region` | `string` | | | AWS region where the bucket is located e.g. `us-west-2`. |
| `bucket` | `string` | ✔️ | | Name of the S3 bucket. |
| `prefix` | `string` | | | Optional path prefix within the bucket. |
| `capacity` | [`VolumeCapacity`](#volumecapacity) | | | Storage capacity allocation. |
| `credentials` | [`AwsCredentialsInput`](#awscredentialsinput) | | | Access credentials for the bucket. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/PersistentVolumeSpecInput.json)

### `VolumeCapacity`

Storage capacity allocation.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `storage` | `string` | | `byte-size` | Maximum storage size e.g. `10Gi`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/storage/v1alpha1/VolumeCapacity.json)

## Tasks

### `FailureBehavior`

Determines how a flow run proceeds when a task finishes with a `Failed` outcome.

| Enum Value |
| :-: |
| `Fail` |
| `Continue` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/FailureBehavior.json)

### `NoOpBehavior`

Determines how a flow run proceeds when a task finishes with a `NoOp` outcome.

| Enum Value |
| :-: |
| `Continue` |
| `Break` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/NoOpBehavior.json)

### `SyncAuth`

Credentials used to access a remote location during a sync.

| Union Type | Description |
| - | - |
| [`SyncAuth::Bearer`](#syncauthbearer) | Authenticates with a bearer token, e.g. an access token of a remote ODF node. |
| [`SyncAuth::Aws`](#syncauthaws) | Authenticates with AWS or an AWS-compatible object storage. |
| [`SyncAuth::Headers`](#syncauthheaders) | Authenticates by passing custom request headers. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuth.json)

### `SyncAuth::Bearer`

Authenticates with a bearer token, e.g. an access token of a remote ODF node.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `token` | [`ValueHandle`](#valuehandle) | ✔️ | | Reference to a secret containing the token. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuth.json)

### `SyncAuth::Aws`

Authenticates with AWS or an AWS-compatible object storage.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `endpoint` | `string` | | `uri` | S3 endpoint URL. If omitted, defaults to AWS S3. Use for S3-compatible stores. |
| `region` | `string` | | | AWS region where the bucket is located e.g. `us-west-2`. |
| `accessKey` | [`ValueHandle`](#valuehandle) | | | Reference to a secret containing the AWS access key ID. |
| `secretKey` | [`ValueHandle`](#valuehandle) | | | Reference to a secret containing the AWS secret access key. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuth.json)

### `SyncAuth::Headers`

Authenticates by passing custom request headers.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `headers` | `array(`[`SyncAuthHeader`](#syncauthheader)`)` | ✔️ | | Headers to pass with every request. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuth.json)

### `SyncAuthHeader`

A request header whose value is taken from a `VariableSet` or a `SecretSet`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | ✔️ | | Name of the header. |
| `value` | [`ValueHandle`](#valuehandle) | ✔️ | | Reference to the value of the header. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuthHeader.json)

### `SyncAuthHeaderInput`

A request header whose value is taken from a `VariableSet` or a `SecretSet`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | ✔️ | | Name of the header. |
| `value` | [`ValueRef`](#valueref) | ✔️ | | Reference to the value of the header. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuthHeaderInput.json)

### `SyncAuthInput`

Credentials used to access a remote location during a sync.

| Union Type | Description |
| - | - |
| [`SyncAuthInput::Bearer`](#syncauthinputbearer) | Authenticates with a bearer token, e.g. an access token of a remote ODF node. |
| [`SyncAuthInput::Aws`](#syncauthinputaws) | Authenticates with AWS or an AWS-compatible object storage. |
| [`SyncAuthInput::Headers`](#syncauthinputheaders) | Authenticates by passing custom request headers. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuthInput.json)

### `SyncAuthInput::Bearer`

Authenticates with a bearer token, e.g. an access token of a remote ODF node.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `token` | [`ValueRef`](#valueref) | ✔️ | | Reference to a secret containing the token. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuthInput.json)

### `SyncAuthInput::Aws`

Authenticates with AWS or an AWS-compatible object storage.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `endpoint` | `string` | | `uri` | S3 endpoint URL. If omitted, defaults to AWS S3. Use for S3-compatible stores. |
| `region` | `string` | | | AWS region where the bucket is located e.g. `us-west-2`. |
| `accessKey` | [`ValueRef`](#valueref) | | | Reference to a secret containing the AWS access key ID. |
| `secretKey` | [`ValueRef`](#valueref) | | | Reference to a secret containing the AWS secret access key. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuthInput.json)

### `SyncAuthInput::Headers`

Authenticates by passing custom request headers.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `headers` | `array(`[`SyncAuthHeaderInput`](#syncauthheaderinput)`)` | ✔️ | | Headers to pass with every request. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncAuthInput.json)

### `SyncEndpoint`

A remote location a dataset is synced with.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `url` | `string` | ✔️ | `uri` | URL of the remote dataset. The scheme selects the transfer protocol e.g. `odf+https://`, `s3://`, `file://`. |
| `auth` | [`SyncAuth`](#syncauth) | | | Credentials used to access the remote location. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncEndpoint.json)

### `SyncEndpointInput`

A remote location a dataset is synced with.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `url` | `string` | ✔️ | `uri` | URL of the remote dataset. The scheme selects the transfer protocol e.g. `odf+https://`, `s3://`, `file://`. |
| `auth` | [`SyncAuthInput`](#syncauthinput) | | | Credentials used to access the remote location. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/SyncEndpointInput.json)

### `Task`

An individual work item to be executed.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `$schema` | `string` | ✔️ | `type-uri` | Identifies this resource type. |
| `headers` | [`ResourceHeadersInput`](#resourceheadersinput) | ✔️ | | Container for identity and ownership information of a resource. |
| `spec` | [`TaskSpecInput`](#taskspecinput) | | | Specifies the desired state of the task. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/Task.json)

### `TaskError`

Type-specific details of a task failure. The `kind` property identifies the error, e.g. `InputDatasetCompacted`, and the remaining properties depend on it.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskError.json)

### `TaskOutcome`

Result of the execution of a task.

| Union Type | Description |
| - | - |
| [`TaskOutcome::Success`](#taskoutcomesuccess) | Task completed successfully. |
| [`TaskOutcome::Failed`](#taskoutcomefailed) | Task failed. |
| [`TaskOutcome::NoOp`](#taskoutcomenoop) | Task completed with no work done (e.g. no new data to process). |
| [`TaskOutcome::Cancelled`](#taskoutcomecancelled) | Task was cancelled before completion. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskOutcome.json)

### `TaskOutcome::Success`

Task completed successfully.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskOutcome.json)

### `TaskOutcome::Failed`

Task failed.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `message` | `string` | ✔️ | | Human-readable description of the failure. |
| `recoverable` | `boolean` | ✔️ | | Whether retrying the task could succeed, e.g. after a network error. An unrecoverable failure, such as an invalid query, would fail the same way again. |
| `error` | [`TaskError`](#taskerror) | | | Type-specific details of the failure. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskOutcome.json)

### `TaskOutcome::NoOp`

Task completed with no work done (e.g. no new data to process).

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskOutcome.json)

### `TaskOutcome::Cancelled`

Task was cancelled before completion.

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskOutcome.json)

### `TaskPlan`

A self-contained logical execution plan of a task.

*Map of string keys to arbitrary values.*

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskPlan.json)

### `TaskSpec`

An individual work item to be executed as part of a flow.

| Union Type | Description |
| - | - |
| [`TaskSpec::Ingest`](#taskspecingest) | Fetches data from a source and appends it to a dataset. |
| [`TaskSpec::Transform`](#taskspectransform) | Executes transformation of data defined in a derivative dataset. |
| [`TaskSpec::SyncFrom`](#taskspecsyncfrom) | Pulls new blocks from a remote dataset into a local dataset. |
| [`TaskSpec::SyncTo`](#taskspecsyncto) | Pushes new blocks of a local dataset to a remote dataset. |
| [`TaskSpec::Compaction`](#taskspeccompaction) | Compacts data files in matching datasets to improve query performance. |
| [`TaskSpec::Reset`](#taskspecreset) | Moves a block reference of a dataset to an earlier block, discarding the history that follows it. |
| [`TaskSpec::GarbageCollection`](#taskspecgarbagecollection) | Removes unreferenced data files from matching datasets. |
| [`TaskSpec::Verify`](#taskspecverify) | Checks dataset metadata for integrity. |
| [`TaskSpec::WebhookCall`](#taskspecwebhookcall) | Dispatches a certain payload to a specific `WebhookEndpoint`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::Ingest`

Fetches data from a source and appends it to a dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the dataset to ingest into. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `source` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Reference to the source resource that defines how to fetch data. |
| `targetRecordsPerSlice` | `integer` | | `uint64` | Target number of records to ingest per data slice. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::Transform`

Executes transformation of data defined in a derivative dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the derivative dataset that defines how to transform data. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::SyncFrom`

Pulls new blocks from a remote dataset into a local dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the local dataset to pull into. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `source` | [`SyncEndpoint`](#syncendpoint) | ✔️ | | Remote dataset to pull from. |
| `force` | `boolean` | | | Overwrite the local dataset even if its history has diverged from the source. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::SyncTo`

Pushes new blocks of a local dataset to a remote dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the local dataset to push. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `destination` | [`SyncEndpoint`](#syncendpoint) | ✔️ | | Remote dataset to push to. |
| `force` | `boolean` | | | Overwrite the remote dataset even if its history has diverged from the local one. |
| `createIfNotExists` | `boolean` | | | Create the remote dataset if it does not exist.<br /><br />Default: `True` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::Compaction`

Compacts data files in matching datasets to improve query performance.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the dataset to compact. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `maxSliceSize` | `string` | | `byte-size` | Target maximum size of each compacted data slice e.g. `100MiB`. |
| `maxSliceRecords` | `integer` | | `uint64` | Target maximum number of records per compacted data slice. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::Reset`

Moves a block reference of a dataset to an earlier block, discarding the history that follows it.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the dataset to reset. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `ref` | `string` | | | Name of the block reference to reset.<br /><br />Default: `head` |
| `newBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Hash of the block the reference will point to. Defaults to the `Seed` block when omitted. |
| `oldBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Expected hash of the block the reference currently points to. When specified, the task fails if the reference has moved, protecting against concurrent updates. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::GarbageCollection`

Removes unreferenced data files from matching datasets.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the dataset to collect garbage from. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::Verify`

Checks dataset metadata for integrity.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetHandle`](#datasethandle) | | | Reference to the dataset to verify. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `replayTransform` | `boolean` | | | If true, re-executes transformations on derivative datasets to verify reproducibility. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpec::WebhookCall`

Dispatches a certain payload to a specific `WebhookEndpoint`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `serviceAccount` | [`PrincipalHandle`](#principalhandle) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `endpoint` | [`ResourceHandle`](#resourcehandle) | ✔️ | | Reference to the `WebhookEndpoint`. |
| `payload` | `string` | | | The payload to send. May include templating. |
| `retryPolicy` | [`RetryPolicy`](#retrypolicy) | | | Defines how a webhook should react to failures. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpec.json)

### `TaskSpecInput`

An individual work item to be executed as part of a flow.

| Union Type | Description |
| - | - |
| [`TaskSpecInput::Ingest`](#taskspecinputingest) | Fetches data from a source and appends it to a dataset. |
| [`TaskSpecInput::Transform`](#taskspecinputtransform) | Executes transformation of data defined in a derivative dataset. |
| [`TaskSpecInput::SyncFrom`](#taskspecinputsyncfrom) | Pulls new blocks from a remote dataset into a local dataset. |
| [`TaskSpecInput::SyncTo`](#taskspecinputsyncto) | Pushes new blocks of a local dataset to a remote dataset. |
| [`TaskSpecInput::Compaction`](#taskspecinputcompaction) | Compacts data files in matching datasets to improve query performance. |
| [`TaskSpecInput::Reset`](#taskspecinputreset) | Moves a block reference of a dataset to an earlier block, discarding the history that follows it. |
| [`TaskSpecInput::GarbageCollection`](#taskspecinputgarbagecollection) | Removes unreferenced data files from matching datasets. |
| [`TaskSpecInput::Verify`](#taskspecinputverify) | Checks dataset metadata for integrity. |
| [`TaskSpecInput::WebhookCall`](#taskspecinputwebhookcall) | Dispatches a certain payload to a specific `WebhookEndpoint`. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::Ingest`

Fetches data from a source and appends it to a dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the dataset to ingest into. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `source` | [`ResourceRef`](#resourceref) | ✔️ | | Reference to the source resource that defines how to fetch data. |
| `targetRecordsPerSlice` | `integer` | | `uint64` | Target number of records to ingest per data slice. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::Transform`

Executes transformation of data defined in a derivative dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the derivative dataset that defines how to transform data. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::SyncFrom`

Pulls new blocks from a remote dataset into a local dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the local dataset to pull into. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `source` | [`SyncEndpointInput`](#syncendpointinput) | ✔️ | | Remote dataset to pull from. |
| `force` | `boolean` | | | Overwrite the local dataset even if its history has diverged from the source. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::SyncTo`

Pushes new blocks of a local dataset to a remote dataset.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the local dataset to push. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `destination` | [`SyncEndpointInput`](#syncendpointinput) | ✔️ | | Remote dataset to push to. |
| `force` | `boolean` | | | Overwrite the remote dataset even if its history has diverged from the local one. |
| `createIfNotExists` | `boolean` | | | Create the remote dataset if it does not exist.<br /><br />Default: `True` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::Compaction`

Compacts data files in matching datasets to improve query performance.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the dataset to compact. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `maxSliceSize` | `string` | | `byte-size` | Target maximum size of each compacted data slice e.g. `100MiB`. |
| `maxSliceRecords` | `integer` | | `uint64` | Target maximum number of records per compacted data slice. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::Reset`

Moves a block reference of a dataset to an earlier block, discarding the history that follows it.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the dataset to reset. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `ref` | `string` | | | Name of the block reference to reset.<br /><br />Default: `head` |
| `newBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Hash of the block the reference will point to. Defaults to the `Seed` block when omitted. |
| `oldBlockHash` | `string` | | [`multihash`](https://github.com/multiformats/multihash) | Expected hash of the block the reference currently points to. When specified, the task fails if the reference has moved, protecting against concurrent updates. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::GarbageCollection`

Removes unreferenced data files from matching datasets.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the dataset to collect garbage from. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::Verify`

Checks dataset metadata for integrity.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `target` | [`DatasetRef`](#datasetref) | | | Reference to the dataset to verify. Defaults to the flow-level target when omitted. |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `replayTransform` | `boolean` | | | If true, re-executes transformations on derivative datasets to verify reproducibility. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskSpecInput::WebhookCall`

Dispatches a certain payload to a specific `WebhookEndpoint`.

| Property | Type | Required | Format | Description |
| - | - | :-: | :-: | - |
| `name` | `string` | | | An alias for the task used to refer to it in flows and access the results |
| `serviceAccount` | [`PrincipalRef`](#principalref) | | | Service account whose permissions the task executes with. Must be an account of service account type. Defaults to the service account of the run when omitted. |
| `onNoOp` | [`NoOpBehavior`](#noopbehavior) | | | How the flow run proceeds when this task finishes with a `NoOp` outcome: `Continue` executes the next task, while `Break` finishes the flow run early without an error.<br /><br />Default: `Continue` |
| `onFailure` | [`FailureBehavior`](#failurebehavior) | | | How the flow run proceeds when this task finishes with a `Failed` outcome: `Fail` finishes the flow run with an error, while `Continue` executes the next task.<br /><br />Default: `Fail` |
| `endpoint` | [`ResourceRef`](#resourceref) | ✔️ | | Reference to the `WebhookEndpoint`. |
| `payload` | `string` | | | The payload to send. May include templating. |
| `retryPolicy` | [`RetryPolicy`](#retrypolicy) | | | Defines how a webhook should react to failures. |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskSpecInput.json)

### `TaskStatus`

Execution phase of a task.

| Enum Value |
| :-: |
| `Pending` |
| `Queued` |
| `Running` |
| `Finished` |

[![JSON Schema](https://img.shields.io/badge/schema-JSON-orange)](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/tasks/v1alpha1/TaskStatus.json)


# Authorization
Source: https://docs.kamu.dev/odf/spec/auth-and-privacy/authorization



**Domain**: Auth & Privacy\
**Depends on**: [Resource Framework](/odf/spec/automation/resource-framework)\
**RFCs**: [RFC-020](/odf/rfcs/020-auth)

## Abstract

This spec defines the authorization model of ODF nodes - how accounts, groups, auth attributes, actions, and policies are expressed as resources of the [Resource Framework](/odf/spec/automation/resource-framework). The model is engine-agnostic: a minimal node can implement simple ownership checks, while a full node can plug in an RBAC / ABAC / ReBAC policy engine, with both remaining interoperable at the level of resources.

## Terminology

This spec builds on the <Term />, <Term />, <Term />, <Term />, and <Term /> concepts defined in the glossary, and introduces the following terms:

* **Subject** - the party a grant is given to in a policy binding: a principal, a group, or a well-known set of principals
* **Group** - a named, possibly nested set of subjects that policies can be bound to, so that permissions are granted to all members at once
* **Action** - an operation the principal attempts to perform, identified by a schema URI (e.g. `View`, `Read`, `Decrypt`)
* **Object** - the target of an action: a resource, an entity, or a resource type (for actions like `Create` and `List` where no instance exists yet)

## Maturity Levels

Implementations may support authorization at one of the following maturity levels:

| Level | Description |
| - | - |
| **0 - No Auth** | All resources and actions are available to the caller |
| **1 - Personal Ownership** | Support for `User` accounts and access based on [ownership](#ownership) only |
| **2 - Policies** | Support for `Generic` [policies](#policies-and-policy-bindings), per-account `PolicyBinding`, and [auth attributes](#auth-attributes) |
| **3 - Groups and Organizations** | Support for [`Group`](#groups) membership, binding policies to groups, [`Organization`](#organization) and [`ServiceAccount`](#service-account) accounts |

## Accounts

### Node

An ODF node. Acts as a principal in system tasks and node-to-node communications, and may own a few node-wide resources (e.g. the `node/admins` group).

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Node
headers:
  name: node
spec:
  did: did:odf:fed0..bc
  displayName: Current Node
```

### User

An individual person who can authenticate and act as a principal.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/User
headers:
  name: alice
spec:
  email: alice@example.com
  displayName: Alice
```

A user may also specify a `did` associated with the account by ODF or an external system, and a `password` for local authentication, which is stored as a hash upon apply and is absent for SSO or DID-based accounts. A user can be suspended by setting `disabled: true` - the effects of suspension are implementation-specific.

### Organization

An account that owns resources on behalf of a group of users, but cannot authenticate or act itself, so its resources are always accessed through its members.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Organization
headers:
  name: acme
spec:
  displayName: Acme Corp
```

Upon creation of an organization, the node automatically creates two groups under its account:

* `Group:acme/admins` - organization administrators, initially containing the principal that created the organization. Members of this group have the same rights over resources owned by the organization as an owner would.
* `Group:acme/members` - organization members

These are ordinary [groups](#groups) that can be edited to customize membership (e.g. by nesting team groups). Their names are [reserved](#reserved-names), so they can't be created or deleted independently of the organization. Whether admins are implicitly considered members is left to implementations.

### Service Account

A non-human identity used by automation (scheduled flows, CI, installed applications) that can act as a principal but doesn't own resources. Service accounts are always scoped under the owning account.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/ServiceAccount
headers:
  account: acme
  name: ingest-prod
spec:
  displayName: Ingest (prod)
```

### Account Names

To avoid ambiguity and impersonation, nodes must ensure that names of nodes, users, and organizations are unique regardless of their resource type, making it possible to refer to any of these accounts by name alone. Service account names are exempt from this rule, as they are always scoped under the owning account.

Some names cannot be used by accounts - see [Reserved Names](#reserved-names).

## Ownership

Every resource is owned by the account specified in its `headers.account` (see [Resource Framework: Ownership](/odf/spec/automation/resource-framework#ownership)). Only `User`, `Organization`, and `Node` accounts can own resources, and are referenced using `ResourceOwnerRef`, whose short form is the account name.

Owners hold all actions on the resources they own and on the entities materialized from them. For resources owned by an organization, the same rights are held by members of its `{org}/admins` group.

## Principals

A principal is the identity performing an action in an authorization request. It is established through authentication and is always a single concrete identity.

| Type | Can act as principal | Can own resources |
| - | - | - |
| `User` | ✅ | ✅ |
| `ServiceAccount` | ✅ | ❌ |
| `Node` | ✅ | ✅ |
| `Organization` | ❌ | ✅ |
| `Anonymous` | ✅ | ❌ |

* `Organization` cannot authenticate - its resources are always accessed through its members.
* `Anonymous` represents unauthenticated requests. It is not a resource and carries no identity.
* When a node executes work that belongs to a user or service account (e.g. a scheduled flow run) it should do so as that principal rather than as itself.

Principals are referenced using `PrincipalRef`, e.g. to specify the service account a flow runs as, or to record the initiator of an operation. Its short form requires a type prefix:

```yaml theme={null}
serviceAccount: ServiceAccount:acme/ingest-prod  # PrincipalRef
```

## Subjects

A subject is the party a grant is given to in a `PolicyBinding`. Unlike a principal, a subject may stand for many principals at once.

Subjects are referenced using `SubjectRef`, which is a union of:

| Kind | Matches | Short form |
| - | - | - |
| `Resource` | A specific `User`, `ServiceAccount`, `Node`, or members of a `Group` | `User:alice`, `Group:acme/engineering` |
| `Authenticated` | Any authenticated principal | `Authenticated` |
| `Anonymous` | Any unauthenticated request | `Anonymous` |
| `Everyone` | Any principal, authenticated or not | `Everyone` |

Short forms expand as follows:

```yaml theme={null}
subject: Group:acme/engineering
# Expands to:
subject:
  kind: Resource
  type: Group
  account: acme
  name: engineering

subject: Authenticated
# Expands to:
subject:
  kind: Authenticated
```

A `Resource` subject can alternatively be identified by `id` or `did`.

`Authenticated`, `Anonymous`, and `Everyone` can be used in policy bindings, but not as group members - otherwise nesting such a subject into a group would extend the group's permissions to all principals.

## Groups

A `Group` resource represents a named set of subjects. Membership is declared inline in the spec. Groups support nesting - a group may include other groups as members. Only the `Resource` subject kind is allowed as a member.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Group
headers:
  account: acme
  name: engineering
spec:
  members:
    - User:alice  # SubjectRef
    - User:bob
    - Group:acme/engineering-contractors  # nested group
```

Groups owned by the `Node` account, like `Group:node/admins`, can be used to represent node-wide predefined roles.

## Auth Attributes

Properties of a resource that directly affect access control (e.g. whether a dataset allows public reads) are expressed using resource labels (see [Resource Framework: Labels and Annotations](/odf/spec/automation/resource-framework#labels-and-annotations)).

A label schema that declares `labelProperties.isAuthAttribute: true` signals that any resource carrying that label should have its value materialized into an auth attribute:

```json theme={null}
{
  "$id": "https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead",
  "$schema": "https://opendatafabric.org/schemas/resources/v1alpha1/meta/ResourceLabel",
  "description": "Controls whether the dataset is readable by any authenticated principal.",
  "type": "boolean",
  "labelProperties": {
    "isAuthAttribute": true,
    "resourceTypes": [
      "https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset"
    ]
  }
}
```

Auth attributes are set like any other label:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset
headers:
  account: bob
  name: bobs-dataset
  labels:
    # Short form - resolved to https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead
    AllowPublicRead: true
    # Full URI form
    https://opendatafabric.org/schemas/datasets/v1alpha1/AllowAnonymousRead: false
spec:
  kind: Root
  metadata: []
```

Rules:

* Auth attributes must be labels, not annotations - validators must reject a resource that places an `isAuthAttribute` label under `annotations`
* The value is validated against the label schema's `type` at apply time
* Implementations discover the set of auth attributes by enumerating label schemas with `isAuthAttribute: true`, rather than hardcoding a list
* Changing an auth attribute may require elevated permissions beyond those needed to update the rest of the resource (see [`AddLabel`](#resource-actions))
* Auth attributes are propagated from resources to the entities they define, e.g. `AllowPublicRead` on a dataset resource (control plane) also appears as an attribute of the dataset entity (data plane)

Since auth attributes are labels, they are also indexed and queryable in the resource listing API.

### Standard Auth Attributes

| Label | Type | Applies to | Description |
| - | - | - | - |
| `datasets/v1alpha1/AllowPublicRead` | `boolean` | `Dataset` | Dataset is readable by any authenticated principal |
| `datasets/v1alpha1/AllowAnonymousRead` | `boolean` | `Dataset` | Dataset is readable without authentication |

The effects of these attributes can also be achieved by binding policies to `Authenticated` or `Anonymous` [subjects](#subjects). Attributes, however, are the primary way to make datasets public, as they are atomic with the resource manifest and can be efficiently filtered on when listing resources. Bindings to these subjects are better suited for granting other actions publicly, e.g. making a resource manifest viewable by everyone.

## Actions

Authorization decisions are made against a set of well-defined actions. Each action is defined by a JSON schema using the [`Action`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/Action.json) metaschema:

* The schema URI is the canonical action identifier used in authorization requests and policies, e.g. `https://opendatafabric.org/schemas/resources/v1alpha1/actions/View`
* The schema describes the context values needed to authorize the action (e.g. the label being added for `AddLabel`)
* `actionProperties.objectTypes` lists the types of objects the action applies to. Since resource types are an open set, every action must list its object types explicitly. Generic resource actions use the `Resource` type to apply to resources of any type.

### Resource Actions

Operate on resource instances. Exceptions are `Create` and `List`, which operate on a resource type and receive the account under which resources are created or listed in the `targetOwner` context value.

| Action | Object | Context | Description |
| - | - | - | - |
| <Schema /> | Resource type | `targetOwner` | Create a new resource instance under an account |
| <Schema /> | Resource type | `targetOwner` | List resource instances of a type under an account |
| <Schema /> | `Resource` | | Read the resource manifest |
| <Schema /> | `Resource` | | Modify the resource manifest |
| <Schema /> | `Resource` | | Delete the resource instance |
| <Schema /> | `Resource` | `labelType`, `labelValue` | Apply a specific label to the resource |
| <Schema /> | `Resource` | `labelType`, `labelValue` | Remove a specific label from the resource |
| <Schema /> | `Resource` | `subject` | Create `PolicyBinding` resources that reference this resource as `object` |

`AddLabel` and `RemoveLabel` operate at the sub-resource level and are distinct from `Edit` to allow fine-grained control - a principal may be allowed to edit a resource, but be restricted from changing auth-sensitive labels like `AllowPublicRead`. Their context carries the type and the new (when adding) or existing (when removing) value of the label.

`BindPolicy` receives the `subject` of the binding being created in its context, so policies can restrict who is allowed to grant access to e.g. `Everyone`.

### Dataset Actions

Operate on dataset entities in the data plane, separately from dataset resources.

| Action | Object | Description |
| - | - | - |
| <Schema /> | `Dataset` | Read dataset data (query, download) |
| <Schema /> | `Dataset` | Append new data to the dataset |
| <Schema /> | `Dataset` | Rewrite or truncate dataset history |

### Config Actions

Operate on configuration entities.

| Action | Object | Description |
| - | - | - |
| <Schema /> | `SecretSet` | Decrypt a secret from a `SecretSet` |

## Policies and Policy Bindings

Beyond [ownership](#ownership), `Policy` and `PolicyBinding` resources are the primary mechanism for granting permissions:

* A `Policy` is a reusable, potentially parameterized grant template. It declares what actions are permitted, but may leave the subject and object as slots to be filled in later.
* A `PolicyBinding` instantiates a `Policy` by supplying the concrete `subject` ([`SubjectRef`](#subjects)) and `object` (`ResourceRef`). The `object` refers to both the resource and the entity materialized from it.

Creating a `PolicyBinding` is itself an authorized operation - the principal creating it must hold the `BindPolicy` action on the referenced `object`. Resource owners implicitly hold `BindPolicy` on their own resources and can grant it to others, enabling delegated access management without transferring ownership.

A `Policy` resource's `kind` field selects the policy language. The `Generic` kind must be supported by all nodes at maturity level 2 and above. Other kinds are engine-specific.

Higher-level features like dataset roles (`Reader`, `Editor`, `Maintainer`) are not part of this spec - implementations can build them using groups, `Generic` policy templates, and `PolicyBinding` resources.

### Generic Policies

A `Generic` policy grants a set of ODF actions. It is the portable building block - any node at maturity level 2 and above can evaluate it without a specialized policy engine. Its subject and object are always supplied by a `PolicyBinding`.

```yaml theme={null}
# A reusable "dataset reader" policy template
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Policy
headers:
  account: node
  name: dataset-reader
spec:
  kind: Generic
  actions:
    - https://opendatafabric.org/schemas/resources/v1alpha1/actions/View
    - https://opendatafabric.org/schemas/datasets/v1alpha1/actions/Read
```

```yaml theme={null}
# Bob grants Alice reader access to his dataset
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/PolicyBinding
headers:
  account: bob
  name: alice-foo-reader
spec:
  policy: Policy:node/dataset-reader
  subject: User:alice  # SubjectRef
  object: Dataset:bob/dataset-foo  # ResourceRef
```

The binding above grants `View` on the dataset resource and `Read` on the dataset entity.

### Engine-specific Policies

Nodes with a full policy engine may support richer `Policy` kinds that can express attribute-based conditions, organization-scoped grants, and custom logic that `Generic` cannot. Engine-specific policies are not portable between implementations.

Engine-specific policies may declare slots for the subject and object, in which case they are templates instantiated via `PolicyBinding`. Policies that don't declare slots take effect on their own without a binding (e.g. a node-wide policy granting all actions to members of `Group:node/admins`).

The `Cedar` kind carries a policy in the [Cedar](https://cedarpolicy.com/) language, where `?principal` and `?resource` slots are filled in by the `subject` and `object` of a binding:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/auth/v1alpha1/Policy
headers:
  account: node
  name: dataset-reader-acme
spec:
  kind: Cedar
  body: |
    permit(
      principal == ?principal,
      action in [
        odf::resources::Action::"https://opendatafabric.org/schemas/resources/v1alpha1/actions/View",
        odf::datasets::Action::"https://opendatafabric.org/schemas/datasets/v1alpha1/actions/Read"
      ],
      resource == ?resource
    ) when {
      resource.owner in odf::auth::ResourceOwner::"acme"
    };
```

Nodes that do not support a given `Policy` kind must reject the resource with a clear error rather than silently ignoring it.

## Access Token Scopes

Access tokens may be restricted to a set of scopes (e.g. read-only access to specific datasets). Scopes are not part of the authorization model: they can only narrow what the principal is allowed to do, so implementations check whether an action on an object falls within the token's scopes before policy evaluation. Policies then decide based on the principal's own permissions, so a token can never grant more than its owner has.

## Reserved Names

Names that implementations must reserve and treat consistently.

**Accounts** - the following names cannot be used by `User` or `Organization` accounts, to prevent impersonation of the node or its administrators:

| Name | Reason |
| - | - |
| `node` | Conventional name of the local `Node` account |
| `admin`, `admins`, `root`, `system` | Suggest administrative privileges |
| `authenticated`, `anonymous`, `everyone` | Match subject keywords |

Matching is case-insensitive. Implementations may reserve additional names (e.g. their own product name).

**Groups** - reserved groups are created together with the account that owns them and cannot be created or deleted independently:

| Name | Created with | Description |
| - | - | - |
| `{node}/admins` | `Node` | Node administrators |
| `{org}/admins` | `Organization` | Organization administrators, have owner rights over all resources of the organization |
| `{org}/members` | `Organization` | Organization members |

**Subject keywords** - cannot collide with resource names, as resource references in short form always include a type prefix, a DID, or an ID:

| Keyword | Description |
| - | - |
| `Authenticated` | `SubjectRef` matching any authenticated principal |
| `Anonymous` | `SubjectRef` matching any unauthenticated principal |
| `Everyone` | `SubjectRef` matching any principal |

## Schemas

| Schema | Description |
| - | - |
| <Schema /> | Node account |
| <Schema /> | User account |
| <Schema /> | Organization account |
| <Schema /> | Service account |
| <Schema /> | Group of subjects |
| <Schema /> | Policy template |
| <Schema /> | Policy instantiation for a subject and object |
| <Schema /> | Reference to a principal |
| <Schema /> | Reference to a subject |
| <Schema /> | Reference to an account that can own resources |
| [`Action`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/Action.json) | Metaschema for action schemas |
| [`ResourceLabel`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceLabel.json) | Metaschema for label schemas, including auth attributes |

## Appendix A: Cedar Example

> **Non-normative:** This appendix illustrates how the authorization model maps onto the [Cedar](https://cedarpolicy.com/) policy engine.

Full prototype is available [here](https://github.com/kamu-data/tmp-cedar-prototype).

A schema can be code-generated based on types registered in the system allowing policies to be validated against known resource types and attributes:

> **Note:** For brevity, actions here use short names (e.g. `"read"`). A generated schema would use full action schema URIs (e.g. `"https://opendatafabric.org/schemas/datasets/v1alpha1/actions/Read"`) as in the examples above.

```cedar theme={null}
namespace impl {
    // Base context is implementation-specific and not part of ODF,
    // but it will be present in all actions as `context.base`.
    // It's a workaround for missing type inheritance in Cedar.
    type Context = {
        ip?: String,
        country_code_a2?: String,
    };
}

namespace odf {
    // Common context type for ODF actions that don't need anything extra
    type Context = {
        base: impl::Context,
    };
}

namespace odf::auth {
    // Abstract type that can own resources
    // User and Organization entities specify this type as parent
    entity ResourceOwner = {
        // Set for organizations to their reserved `{org}/admins` group,
        // whose members get the same rights as the owner
        admins?: odf::auth::Group,
    };

    // Marker for any authenticated principal: User, ServiceAccount, or Node
    // Allows checking `principal in Authenticated::""` instead of creating
    // separate policies for each principal type.
    entity Authenticated;

    // Unauthenticated users are represented as separate type
    // to ensure they are handled in policies explicitly
    entity Anonymous;

    // The node is a principal used for local system tasks and node-to-node communications.
    // When node performs actions on behalf of users and service accounts
    // (e.g. scheduled flow runs) the principal will be the user or service account
    // that task belongs to, i.e. node acts on behalf of the account.
    // The local node may own a few system-wide resources, e.g. the `admin` group.
    entity Node in [ResourceOwner, Authenticated];

    // Authenticated users
    entity User in [ResourceOwner, Group, Authenticated];

    // Organizations can own resources but can't perform actions (not principals)
    entity Organization in [ResourceOwner];

    // Principal for automation that lives under a user or organization
    // used to bind permissions to org-scoped flows and application installations.
    // Can act but does not own any resources.
    entity ServiceAccount in [Group, Authenticated] = {
        owner: odf::auth::ResourceOwner,
    };

    // Groups are a set of users, service accounts or other groups (nesting)
    // used to manage permissions in bulk.
    entity Group in [Group] = {
        owner: odf::auth::ResourceOwner,
    };
}

namespace odf::resources {
    entity TypeUri;

    entity Resource in [TypeUri] = {
        owner: odf::auth::ResourceOwner,

        // All possible auth attributes are accessible via full URIs
        "https://opendatafabric.org/schemas/datasets/v1alpha1/AllowPublicRead"?: Bool,
        "https://opendatafabric.org/schemas/datasets/v1alpha1/AllowAnonymousRead"?: Bool,
    };

    action create, list appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::Anonymous,
            odf::auth::ServiceAccount
        ],
        resource: [odf::resources::TypeUri],
        context: odf::resources::ContextCreateList,
    };

    type ContextCreateList = {
        // Owner under which resources are being created / listed
        targetOwner: odf::auth::ResourceOwner,
        base: impl::Context,
    };

    action view, edit, delete, bindPolicy appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::resources::Resource],
        context: odf::Context,
    };

    action addLabel, removeLabel appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::resources::Resource],
        context: odf::resources::ContextAddRemoveLabel,
    };

    type ContextAddRemoveLabel = {
        // Type of the label being added / removed
        labelType: odf::resources::TypeUri,
        // New value of the label when adding or existing value when removing (as stringified json)
        labelValue: String,
        base: impl::Context,
    };
}

namespace odf::datasets {
    // Represents a data plane entity, not the control plane resource
    entity Dataset = {
        owner: odf::auth::ResourceOwner,

        // Relevant attributes are propagated into the data plane
        // as short names allowing for type-checked access
        allowPublicRead?: Bool,
        allowAnonymousRead?: Bool,
    };

    action read, write, alterHistory appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::datasets::Dataset],
        context: odf::Context,
    };
}

namespace odf::config {
    // Represents a data plane entity, not the control plane resource
    entity SecretSet = {
        owner: odf::auth::ResourceOwner,
    };

    action decrypt appliesTo {
        principal: [
            odf::auth::Node,
            odf::auth::User,
            odf::auth::ServiceAccount,
            odf::auth::Anonymous
        ],
        resource: [odf::config::SecretSet],
        context: odf::Context,
    };
}
```

Example policies:

```cedar theme={null}
// System admins can do anything
permit(
    principal in odf::auth::Group::"node/admins",
    action,
    resource,
);

// The local node can do anything. Implementations drop privileges
// by executing work as a user or service account.
permit(
    principal == odf::auth::Node::"${local_node_id}",
    action,
    resource,
);

// Users can create and list resources under their own account
permit(
    principal is odf::auth::User,
    action in [
        odf::resources::Action::"create",
        odf::resources::Action::"list",
    ],
    resource is odf::resources::TypeUri,
) when {
    principal in context.targetOwner
};

// Users can do anything to resources they own
permit(
    principal is odf::auth::User,
    action,
    resource is odf::resources::Resource
) when {
    principal in resource.owner
};

// Users can do anything to data plane entities they own
permit(
    principal is odf::auth::User,
    action,
    resource is ${entity_type}
) when {
    principal in resource.owner
};

// Org admins can do anything to resources owned by the organization
permit(
    principal is odf::auth::User,
    action,
    resource is odf::resources::Resource
) when {
    resource.owner has admins && principal in resource.owner.admins
};

// Public datasets can be read by any authenticated principal (users and service accounts)
permit(
    principal in odf::auth::Authenticated::"",
    action in [odf::datasets::Action::"read"],
    resource is odf::datasets::Dataset
) when {
    resource has allowPublicRead && resource.allowPublicRead
};

// Anonymous users can read datasets that explicitly allow it
permit(
    principal is odf::auth::Anonymous,
    action in [odf::datasets::Action::"read"],
    resource is odf::datasets::Dataset
) when {
    resource has allowPublicRead && resource.allowPublicRead &&
    resource has allowAnonymousRead && resource.allowAnonymousRead
};
```

`PolicyBinding` resources become template-linked policies, with the `?principal` slot filled based on the `SubjectRef` variant:

| `SubjectRef` | Cedar |
| - | - |
| `Resource` (e.g. `Group:acme/engineering`) | Entity UID, e.g. `odf::auth::Group::"acme/engineering"` |
| `Authenticated` | `odf::auth::Authenticated::""` |
| `Anonymous` | `odf::auth::Anonymous::""` |
| `Everyone` | No slot — compiled into a static policy with an unconstrained `principal` |

## Appendix B: OpenFGA Example

> **Non-normative:** This appendix illustrates how the authorization model maps onto the [OpenFGA](https://openfga.dev/) ReBAC engine.

Full prototype is available [here](https://github.com/kamu-data/tmp-openfga-prototype).

> **Note:** This model predates the `ServiceAccount` and `Node` principals and is kept as an illustration of how the model maps onto a ReBAC engine.

Example schema:

```openfga theme={null}
model
  schema 1.1

# Singleton representing the kamu node. Carries the node-wide admin group.
# Users and organizations link to the node.
type node
  relations
    define admin: [group#member]

# An authenticated human user.
type user
  relations
    define node: [node]
    define node_admin: admin from node

# An unauthenticated (anonymous) caller. Kept separate from user so that
# user:* wildcards never accidentally grant anonymous access.
type anonymous

# An organization can own resources but cannot act.
type organization
  relations
    define node: [node]
    define node_admin: admin from node

# A set of users or other groups (supports nesting).
type group
  relations
    define direct_member: [user, group]
    define member: direct_member or member from direct_member

# Resource stores exist under every account per every resource type
# (e.g. `alice/dataset`, `bob/dataset`, `alice/secretset`, `bob/secretset`).
# They allow to express permissions for an entire resource type within
# an account or the whole node.
type resource_store
  relations
    define node_store: [resource_store]
    define owner: [user, organization]
    define lister: [user, group#member]
    define creator: [user, group#member]
    define deleter: [user, group#member]
    define viewer: [user, group#member]
    define editor: [user, group#member]
    define label_assignee: [user with label_type_matches, group#member with label_type_matches]

    define can_create: owner or node_admin from owner or creator or can_create from node_store
    define can_delete: owner or node_admin from owner or deleter or can_delete from node_store
    define can_list: owner or node_admin from owner or lister or can_list from node_store
    define can_view: owner or node_admin from owner or viewer or editor or can_view from node_store
    define can_edit: owner or node_admin from owner or editor or can_edit from node_store
    define can_set_label: owner or node_admin from owner or label_assignee or can_set_label from node_store

# A control-plane resource record (metadata / lifecycle object).
# Linked to a resource_store for org-wide viewer/editor grants and node_admin
# traversal. Per-resource viewer/editor tuples are also supported for individual grants.
type resource
  relations
    define resource_store: [resource_store]
    define owner: [user, organization]
    define viewer: [user, group#member]
    define editor: [user, group#member]

    define can_view: owner or viewer or editor or can_view from resource_store
    define can_edit: owner or editor or can_edit from resource_store
    define can_delete: owner or can_delete from resource_store
    define can_set_label: owner or can_set_label from resource_store

# Permission container for datasets, scoped to node, org, or user.
# Stores link to a parent node_store to inherit node-wide grants.
type dataset_store
  relations
    define node_store: [dataset_store]
    define owner: [user, organization]
    define reader: [user, group#member]
    define writer: [user, group#member]
    define maintainer: [user, group#member]

    define can_read: owner or node_admin from owner or reader or writer or maintainer or can_read from node_store
    define can_write: owner or node_admin from owner or writer or maintainer or can_write from node_store
    define can_alter_history: owner or node_admin from owner or maintainer or can_alter_history from node_store

# A data-plane dataset entity. Permissions are independent of the control-plane
# resource record — a reader role here grants data access only.
type dataset
  relations
    define dataset_store: [dataset_store]
    define owner: [user, organization]
    define reader: [user, group#member]
    define writer: [user, group#member]
    define maintainer: [user, group#member]
    define public_reader: [user:*]       # any authenticated user can read
    define anonymous_reader: [anonymous:*]  # unauthenticated callers can read

    define store_can_read: can_read from dataset_store
    define store_can_write: can_write from dataset_store
    define store_can_alter_history: can_alter_history from dataset_store

    define can_read: owner or reader or writer or maintainer or public_reader or anonymous_reader or store_can_read
    define can_write: owner or writer or maintainer or store_can_write
    define can_alter_history: owner or maintainer or store_can_alter_history

# Permission container for secret sets, scoped to node, org, or user.
# Stores link to a parent node_store to inherit node-wide grants.
type secret_set_store
  relations
    define node_store: [secret_set_store]
    define owner: [user, organization]
    define decryptor: [user, group#member]

    define can_decrypt: node_admin from owner or owner or decryptor or can_decrypt from node_store

# A data-plane secret set entity (e.g. encryption keys).
# Linked to a secret_set_store so that store-level grants (node admin,
# org group) propagate without needing per-secret-set tuples.
type secret_set
  relations
    define owner: [user, organization]
    define decryptor: [user, group#member]
    define secret_set_store: [secret_set_store]

    define can_decrypt: owner or decryptor or can_decrypt from secret_set_store

condition label_type_matches(requested: string, allowed: list<string>) {
  requested in allowed
}
```


# Resource Framework
Source: https://docs.kamu.dev/odf/spec/automation/resource-framework



**Domain**: Automation\
**Depends on**: -\
**RFCs**: [RFC-018](/odf/rfcs/018-iac-resource-framework)

## Abstract

This spec defines the declarative Infrastructure-as-Code framework used to configure ODF nodes. It defines the common manifest format of <Term />, how resource types are identified and versioned, how resources are identified, owned, labeled, and reference one another, and how <Term /> apply manifests and reconcile the actual state of the system with the desired state they describe. Individual resource types (datasets, sources, flows, accounts, etc.) are defined by the specs of their respective domains.

## Terminology

This spec builds on the <Term />, <Term />, and <Term /> concepts defined in the glossary, and introduces the following terms:

* **Controller** - a process that tracks resources of certain types and reconciles them with the state of entities under its control, continuously moving the actual state towards the desired state described by manifests

For example:

* Applying a `Dataset` manifest creates a `Dataset` resource, whose controller creates and maintains an ODF <Term />
* Applying a high-level resource manifest may result in its controller generating and applying lower-level manifests, which in turn result in provisioning of corresponding entities

## Resource Manifests

All resources share a common manifest structure:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers: {}
spec: {}
status: {}
```

Where:

* `$schema` - identifies the [type](#type-identification) of the resource
* `headers` - contains identity, ownership, labels, annotations, and lifecycle information
* `spec` - defines the desired state of the resource
* `status` - contains information about the current state and the reconciliation process (see [Status](#status))

The `status` section never appears in user-authored manifests - it is maintained by the node and is writable only by controllers.

### Input and Canonical Representations

Every resource has two representations:

* **Input** (<Schema />) - the user-authored manifest. It contains only the fields users can modify, and allows convenient alternatives, e.g. referencing other resources by name or using short forms.
* **Canonical** (<Schema />) - the representation stored and returned by the node. It includes system-populated fields (e.g. `id`, `generation`, `status`), and all references are fully resolved.

Input types reference their canonical type via the `canonicalType` schema property. Canonical types must be trivially convertible into input types, while converting input types into canonical requires identity and [reference resolution](#reference-resolution).

## Type Identification

The `$schema` field identifies the resource type using a resolvable URL that points to the JSON Schema of the resource:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
```

The URL follows the pattern `{base-url}/{context}/{version}/{Name}` and carries:

* Controlling organization domain (e.g. `opendatafabric.org`)
* Bounded context (e.g. `config`)
* Version (e.g. `v1alpha1`)
* Resource type name (e.g. `SecretSet`)

Resource schemas are registered within a node and assigned a **short type name** (e.g. `SecretSet`) that can be used instead of the full URL, e.g. in [references](#references).

Resource schemas are validated against the [`meta/Resource`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/Resource.json) and [`meta/ResourceInput`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceInput.json) metaschemas.

### Versioning

The version is part of the `$schema` URL in the form of e.g. `v1` or `v1alpha1`:

* A version captures both how a resource is defined and the semantics of how it behaves, i.e. the version may be incremented if the behavior of a resource changes significantly even when its schema stays the same.
* Versions apply to an entire bounded context, not an individual resource type - a version bump applies to all resource types within the context.

## Identity

A manifest usually only defines the resource `name`:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
spec: {}
```

At any point in time the tuple `(account, type, name)` uniquely identifies a resource within a node. Resources of different types under one account may have the same name.

Upon creation the node assigns the resource a unique `id` (UUID v4), which is a stable identifier of the resource **within one node**:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  id: 6767a4ee-d74d-436e-84f9-709407869a26
  name: my-secrets
spec: {}
```

Resource names are **mutable**. When a manifest includes both `id` and `name`, the resource is matched by `id` and its `name` is updated to the one in the manifest, allowing resources to be renamed. Including `id` ensures that the manifest applies to the exact intended resource, but sacrifices portability of the manifest across nodes.

### Resources and Entities

Resources describe the desired state of the system and are separate from the <Term /> created to fulfill that state, e.g. a `Dataset` resource is not the same thing as an ODF dataset stored in some S3 bucket.

This is why resources like `Dataset` or `User` may have both:

* `id` - identifies the resource within a node
* `did` - a [W3C DID](https://www.w3.org/TR/did-core/) that identifies the corresponding entity (e.g. a dataset or a person) on a global network

When one node creates a dataset `did:odf:xxyy` and another node replicates it, the two nodes have two `Dataset` resources with the same `did` but different `id`s.

## Ownership

Every resource belongs to an <Term /> specified in the `account` header:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  account: alice
  name: my-secrets
spec: {}
```

When `account` is omitted in a manifest, the resource is created under the account of the current <Term />.

ODF uses an account-centric model of multi-tenancy, where ownership and access control are defined by the [Authorization](/odf/spec/auth-and-privacy/authorization) spec.

## Labels and Annotations

Resources can specify labels and annotations. Both are maps of keys to arbitrary JSON values, but only labels are indexed and can be used for querying and in [selectors](#selectors):

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
  labels:
    env: prod
    https://opendatafabric.org/schemas/datasets/v1alpha1/DatasetKind: Root
  annotations:
    repo:
      url: https://github.com/open-data-fabric
      commitHash: 31d318be8a8f1a9ad8f317534a52f37ba89e8771
spec: {}
```

Labels and annotations are fully **mutable**.

Keys can be either short type names or full schema URIs. Label and annotation schemas are defined using the [`meta/ResourceLabel`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceLabel.json) and [`meta/ResourceAnnotation`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceAnnotation.json) metaschemas, so their values can be type-checked. Schemas may restrict which resource types they are valid for via `resourceTypes`, and labels can be marked as required for these types via `isRequired`.

Controllers may contribute their own labels to simplify common filtering scenarios, e.g. a `Dataset` resource may automatically get the `DatasetKind` label.

Labels can also carry information used in access control - see [Auth Attributes](/odf/spec/auth-and-privacy/authorization#auth-attributes).

## References

Resources can link to other resources using **references** (<Schema />). A reference can identify the target by:

* `id` - unique within a node
* `did` - unique globally
* `type`, `name`, and optional `account` - when `account` is not specified, the name is resolved within the account of the current principal

Example of referencing a resource by type, name, and account:

```yaml theme={null}
volume:
  type: PersistentVolume
  name: my-s3-bucket
  account: my-org
```

Or in short form:

```yaml theme={null}
volume: PersistentVolume:my-org/my-s3-bucket
```

Example of referencing by ID:

```yaml theme={null}
volume:
  type: PersistentVolume
  id: 6767a4ee-d74d-436e-84f9-709407869a26
```

Resource references form a DAG - cyclical references are not allowed.

All references and selectors share common schemas, so they can be handled uniformly by tools without knowing the specifics of individual resource types.

### Typed References

Domains may define extended versions of resource references that:

* Restrict target resources to a specific type (e.g. `DatasetRef`)
* Provide additional features (e.g. referencing a sub-path of a secret via `Secret:postgres#password`)

Such references are validated against the [`meta/ResourceRef`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceRef.json) metaschema.

### Reference Resolution

Referencing resources by name is convenient when setting up resource graphs, especially when target resources are not created yet. Over the long term, however, as resources get renamed and deleted, name-based references become unreliable and can pose security risks when old names get reused.

Nodes therefore resolve all references to stable IDs upon first apply. Resolved references are represented as **handles** (<Schema />) that always include the `id`, while `type`, `name`, and `account` remain for human readability:

```yaml theme={null}
volume:
  id: 6767a4ee-d74d-436e-84f9-709407869a26  # Stable reference
  type: PersistentVolume
  name: my-s3-bucket  # Remains only for human readability
  account: my-org
```

**Dangling references are allowed** - implementations may warn users about them, but:

* It must be possible to create a resource that references a non-existing resource and have the target created later
* It must be possible to delete a resource referenced by others

## Selectors

Multiple resources can be referenced at once using **selectors** (<Schema />) that match resources by:

* Owning account
* Resource type
* Name pattern in SQL `LIKE` format
* Label filters (<Schema />)

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  name: periodic-compaction
spec:
  target:
    type: Dataset
    name: org.opendatafabric.%
    labels:
      DatasetKind: Root
      env: prod
  ...
```

Being a superset of references, selectors can also match singular resources by `id`, `did`, or type and name without wildcards.

Unlike references, selectors are not resolved during apply. Whether a resource matches a selector is determined repeatedly during controller operations, so resources can start and stop matching a selector after the resource containing it was applied.

## Owner References

When a resource is created by a controller of another higher-level resource, the association is recorded in the `ownerReferences` header. This creation provenance can be used for lineage tracking, cascading deletion, and garbage collection.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/FlowRun
headers:
  name: periodic-compaction-aabbcc
  ownerReferences:
    - id: c27331ce-ce88-4ff9-8c5a-4ce8107cc03f
      name: periodic-compaction
spec: {}
```

Owner references are distinct from [ownership](#ownership) by an account.

## Generations

Resource reconciliation is an **eventually-consistent** process - while a controller is reconciling one version of a resource, its desired state may be changed by the user. To reflect this, the node maintains the following headers:

* `generation` - incremented every time the resource headers or spec are updated, signifying changes to the desired state. Does not increment on status changes.
* `statusGeneration` - incremented every time the resource status is updated.
* `createdAt`, `updatedAt` - time when the resource was first applied and last updated (including status updates)

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/config/v1alpha1/SecretSet
headers:
  name: my-secrets
  generation: 4
  statusGeneration: 9
  createdAt: 2026-01-01T00:00:00Z
  updatedAt: 2026-01-04T00:00:00Z
spec: {}
status: {}
```

Both generations start at `1`. Together they tell whether anything changed in a resource since it was last read, without comparing its contents.

Status may be updated by several controllers concurrently. A controller can pass the `statusGeneration` it read as a **precondition** of a status update, which is rejected as a conflict if the status was modified in the meantime. This allows **compare-and-swap** updates without the check and the update racing with other controllers.

## Status

The `status` section provides information about the reconciliation of the resource:

* `observedGeneration`, `observedAt` - the generation last seen by the main controller of the resource, and when
* `reconciledGeneration`, `reconciledAt` - the generation last successfully reconciled by the main controller, and when
* `phase` - see [Phases](#phases)
* `conditions` - see [Conditions](#conditions)

Example of a resource where reconciliation of `generation: 2` failed because it links to a non-existing secret, leaving the previous generation running:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/sources/v1alpha1/Source
headers:
  name: api-polling-source
  generation: 2
spec: {}
status:
  phase: Degraded
  observedGeneration: 2
  observedAt: 2026-01-02T00:00:00Z
  reconciledGeneration: 1
  reconciledAt: 2026-01-01T00:00:00Z
  conditions:
    https://opendatafabric.org/schemas/resources/v1alpha1/ReconciliationError:
      code: unresolved-reference
      message: "Secret `new-api-key` not found"
```

### Phases

The `phase` field (<Schema />) is populated as follows:

| `generation` | `observedGeneration` | `reconciledGeneration` | `phase` | Meaning |
| - | - | - | - | - |
| 1 | - | - | `Pending` | Controller has not seen the resource yet |
| 3 | 2 | 2 | `Pending` | Controller has not seen generation 3 yet |
| 3 | 3 | 2 | `Reconciling` | Controller is actively applying generation 3 |
| 3 | 3 | 3 | `Ready` | Fully reconciled, running the current spec |
| 3 | 3 | 2 | `Degraded` | Reconciliation of generation 3 failed, but generation 2 is still running |
| 3 | 3 | - | `Failed` | Controller has seen generation 3, but never successfully reconciled anything |
| any | any | any | `Deleting` | The `deletionRequestedAt` header is set and controllers are cleaning up |
| any | any | any | `Deleted` | Controllers cleaned up and the `deletedAt` header is set |

Deletion takes precedence over the other phases. Deleting a resource is a two-step process, so that controllers can clean up the entities they created or finish work in progress before the resource disappears.

### Conditions

The `conditions` field provides extended information about the state of a specific resource type. Conditions can be contributed by multiple controllers and are keyed by schema URIs to avoid name collisions and allow schema checking. Condition schemas are defined using the [`meta/ResourceCondition`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceCondition.json) metaschema.

When reconciliation fails, the controller reports the cause using the <Schema /> condition, which carries a machine-readable `code` and a human-readable `message`.

Conditions contributed by controllers other than the main controller of the resource should carry their own `observedGeneration` and `reconciledGeneration` fields, as they may lag behind or be ahead of the main controller.

## Resource Application

When a batch of manifests is applied, the node performs the following steps:

1. Validate each manifest against its schema
2. Look up whether target resources already exist (by `id` or `name`)
3. Assign `id`s to resources that don't yet exist
4. [Resolve](#reference-resolution) all references to `id`s of existing resources or of those about to be created
5. Order resources in depth-first dependency order
6. Pass every resource to the pre-apply step of its main controller, which may:
   * Perform additional validation
   * Contribute additional labels
   * Rewrite parts of the spec (e.g. to encrypt raw secrets before the spec is stored)
7. Increment the resource `generation`
8. Store the resource
9. Initiate reconciliation asynchronously

## Schemas

| Schema | Description |
| - | - |
| <Schema /> | Canonical representation of a resource |
| <Schema /> | User-authored representation of a resource |
| <Schema /> | Canonical identity, ownership, and lifecycle headers |
| <Schema /> | User-authored headers |
| <Schema /> | Indexed key-value pairs |
| <Schema /> | Non-indexed key-value pairs |
| <Schema /> | Reconciliation status |
| <Schema /> | Reconciliation or deletion phase |
| <Schema /> | Conditions contributed by controllers |
| <Schema /> | Condition describing a failed reconciliation |
| <Schema /> | Reference to another resource |
| <Schema /> | Resolved reference to another resource |
| <Schema /> | Matches zero or many resources |
| <Schema /> | Filter on resource labels |
| [`meta/Resource`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/Resource.json) | Metaschema for canonical resource schemas |
| [`meta/ResourceInput`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceInput.json) | Metaschema for user-authored resource schemas |
| [`meta/ResourceRef`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceRef.json) | Metaschema for reference types |
| [`meta/ResourceHandle`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceHandle.json) | Metaschema for resolved reference types |
| [`meta/ResourceLabel`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceLabel.json) | Metaschema for label schemas |
| [`meta/ResourceAnnotation`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceAnnotation.json) | Metaschema for annotation schemas |
| [`meta/ResourceCondition`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/resources/v1alpha1/meta/ResourceCondition.json) | Metaschema for condition schemas |

Example manifests of various resource types are available in the [`examples`](https://github.com/open-data-fabric/open-data-fabric/tree/master/examples) directory, and the relationships between core resource types are shown in the [entity-relationship diagram](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/erd.svg).


# Tasks and Flows
Source: https://docs.kamu.dev/odf/spec/automation/tasks-and-flows



**Domain**: Automation\
**Depends on**: [Resource Framework](/odf/spec/automation/resource-framework), [Authorization](/odf/spec/auth-and-privacy/authorization)\
**RFCs**: [RFC-019](/odf/rfcs/019-iac-task-flow-system)

## Abstract

This spec defines the resources used to define, schedule, and execute workloads within ODF <Term />. A `Task` represents a single unit of work like ingesting or transforming data of a <Term />, a `FlowRun` executes a sequence of tasks, and a `Flow` acts as a template that instantiates flow runs for a set of target resources when its triggers fire.

## Overview

The system is built from three <Term /> types defined using the [Resource Framework](/odf/spec/automation/resource-framework):

| Resource | Description |
| - | - |
| [`Task`](#task) | A single unit of work, from intent through execution to its outcome |
| [`FlowRun`](#flowrun) | A sequence of tasks executed one after another |
| [`Flow`](#flow) | A template for instantiating flow runs, with triggers that decide when to instantiate them |

Tasks and flow runs are short-lived - they are deleted as soon as they finish, so the existing resources of these types are exactly the work that is pending or in progress. Deleted resources stay readable for a configurable retention period (e.g. 10 days) before being purged, allowing operators to inspect recent runs.

## Task

A `Task` resource represents a single unit of work. Its `spec` captures the **intent**: the kind of operation, the target resource, and high-level parameters. This is what an operator or a `FlowRun` controller writes when creating a task. The spec is stable, human-readable, and never rewritten.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/tasks/v1alpha1/Task
headers:
  name: task-123456
  ownerReferences:
    - FlowRun:c27331ce-ce88-4ff9-8c5a-4ce8107cc03f  # FlowRun that spawned this task
spec:
  kind: Transform
  target: Dataset:sergiimk/foo
```

Tasks are executed by **workers**. Every worker specializes in one task type and pulls the next task of that type from the node's **queue**, which schedules and prioritizes tasks. The worker advances the task through the steps its type requires - e.g. fetching data for an ingest, or elaborating the execution plan of a transform, requesting an <Term /> and dispatching the plan to it - and commits the result.

As it progresses, the worker records the state of the task in type-specific status [conditions](/odf/spec/automation/resource-framework#conditions):

* <Schema /> - execution phase of the task (see [Task Lifecycle](#task-lifecycle))
* <Schema /> - a self-contained logical execution plan with fully resolved paths, offsets, schemas, and all other inputs
* <Schema /> - result of the execution (see [Task Outcome](#task-outcome))

Example of a task after planning and execution:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/tasks/v1alpha1/Task
headers:
  name: task-123456
  ownerReferences:
    - FlowRun:c27331ce-ce88-4ff9-8c5a-4ce8107cc03f
  deletedAt: 2026-09-11T02:23:41Z  # Deleted upon completion, readable until the retention period ends
spec:
  kind: Transform
  target: Dataset:sergiimk/foo
status:
  phase: Deleted
  observedGeneration: 1
  conditions:
    https://opendatafabric.org/schemas/tasks/v1alpha1/TaskStatus: Finished
    https://opendatafabric.org/schemas/tasks/v1alpha1/TaskPlan:
      kind: TransformPlan
      datasetId: did:odf:fed0..17bf
      systemTime: 2026-09-11T02:22:52Z
      queryInputs:
        - datasetId: did:odf:fed0..685b
          offsetInterval:
            start: 0
            end: 399493
          dataPaths:
            - s3://repo/f162..8a9f/data/0a..fa
      transform:
        kind: Sql
        query: SELECT ... FROM ...
      newDataPath: s3://repo/f162..8a9f/data/...
      newCheckpointPath: s3://repo/f162..8a9f/checkpoint/...
    https://opendatafabric.org/schemas/tasks/v1alpha1/TaskOutcome:
      kind: Success
      result:
        kind: TransformResult
        newBlockHash: f162..f008
        newOffsetInterval:
          start: 0
          end: 1050
        newWatermark: 2023-04-15T00:00:00Z
```

The structure of plans and results is specific to the task type.

### Task Lifecycle

The `TaskStatus` condition proceeds through the following phases:

| Status | Description |
| - | - |
| `Pending` | Task was created, and the node is validating its inputs and resolving what is needed to schedule it (e.g. the engine a transform uses) |
| `Queued` | Task is waiting in the queue for a worker |
| `Running` | A worker has claimed the task and is executing it, including the commit |
| `Finished` | Terminal - the `TaskOutcome` condition is set, and the task resource is deleted |

The resource [phase](/odf/spec/automation/resource-framework#phases) reflects the work of the **task controller**, whose job is to admit the task: validate its inputs, resolve what is needed to schedule it, and put it into the queue. `phase: Ready` means the task was admitted, while `phase: Failed` means it was rejected - in which case the controller also sets an unrecoverable `Failed` outcome. Execution is reported separately by workers in the `TaskStatus` condition, so an admitted task stays in `phase: Ready` regardless of its `TaskStatus` until it is deleted.

Deleting a task that has not finished yet **cancels** it when possible. Until its outcome is recorded, a task being deleted remains visible in `phase: Deleting`.

### Task Outcome

The `TaskOutcome` condition is one of:

| Outcome | Description |
| - | - |
| `Success` | Task completed successfully, with a type-specific `result` |
| `Failed` | Task failed |
| `NoOp` | Task completed with no work done (e.g. no new data to process). Such a task may have no `TaskPlan` if the worker realized there is nothing to do |
| `Cancelled` | Task was cancelled before completion |

A `Failed` outcome carries a human-readable `message`, and tells whether the failure is `recoverable`, i.e. whether retrying the task could succeed. Recoverable failures (e.g. a network error) can be retried according to the [retry policy](#retry-policy), while unrecoverable ones (e.g. an invalid query) would fail the same way again. Type-specific details of the failure are carried in `error` (<Schema />):

```yaml theme={null}
https://opendatafabric.org/schemas/tasks/v1alpha1/TaskOutcome:
  kind: Failed
  message: Input dataset was compacted since the last transformation
  recoverable: false
  error:
    kind: InputDatasetCompacted
    inputDataset: Dataset:sergiimk/sensor-temp
```

### Task Types

The `kind` field of the task spec (<Schema />) selects one of the following task types. All types accept:

* `name` - an alias used to refer to the task in a flow and access its results
* `target` - reference to the target resource, which defaults to the target of the flow run when omitted
* `serviceAccount` - see [Authorization](#authorization)
* `onNoOp` - how the flow run proceeds when the task finishes with a `NoOp` outcome: `Continue` (default) executes the next task, while `Break` finishes the flow run early without an error
* `onFailure` - how the flow run proceeds when the task finishes with a `Failed` outcome: `Fail` (default) finishes the flow run with an error, while `Continue` executes the next task

`Ingest` - fetches data from a `Source` and appends it to a <Term /> (see [Ingestion](/odf/spec/datasets/ingestion)):

```yaml theme={null}
kind: Ingest
target: Dataset:sergiimk/sensor-temp
source: Source:sergiimk/sensor-temp-http
targetRecordsPerSlice: 10000  # optional: target number of records per data slice
```

`Transform` - executes the transformation of data defined in a <Term />:

```yaml theme={null}
kind: Transform
target: Dataset:sergiimk/sensor-temp-daily
```

`SyncFrom` - pulls new blocks from a remote dataset into a local dataset (see [Transfer Protocols](/odf/spec/exchange/transfer-protocols)):

```yaml theme={null}
kind: SyncFrom
target: Dataset:sergiimk/sensor-temp
source:
  url: odf+https://node.example.com/acme/sensor-temp  # scheme selects the transfer protocol
  auth:
    kind: Bearer
    token: SecretSet:acme-node#accessToken
force: false  # optional: overwrite the local dataset even if histories have diverged
```

`SyncTo` - pushes new blocks of a local dataset to a remote dataset:

```yaml theme={null}
kind: SyncTo
target: Dataset:sergiimk/sensor-temp
destination:
  url: s3://my-bucket/datasets/sensor-temp/
  auth:
    kind: Aws
    region: us-west-2
    accessKey: SecretSet:my-aws-secrets#accessKey
    secretKey: SecretSet:my-aws-secrets#secretKey
force: false  # optional: overwrite the remote dataset even if histories have diverged
createIfNotExists: true  # optional: create the remote dataset if it does not exist
```

Sync endpoints (<Schema />) specify the `url` of the remote dataset, where the scheme selects the transfer protocol (e.g. `odf+https://`, `s3://`, `file://`), and optional credentials (<Schema />):

* `Bearer` - passes a token, e.g. an access token of a remote ODF node
* `Aws` - credentials for AWS or an AWS-compatible object storage
* `Headers` - custom request headers with values referencing variables and secrets

The execution plan of a sync task carries references to secrets and never their values - these are resolved only during execution.

`Compaction` - compacts data files of a dataset to improve query performance:

```yaml theme={null}
kind: Compaction
target: Dataset:sergiimk/sensor-temp
maxSliceSize: 100MiB  # optional: target maximum size of each compacted data slice
maxSliceRecords: 10000  # optional: target maximum number of records per data slice
```

`Reset` - moves a block reference of a dataset to an earlier block, discarding the history that follows it:

```yaml theme={null}
kind: Reset
target: Dataset:sergiimk/sensor-temp
ref: head  # optional: block reference to reset
newBlockHash: f162..8a9f  # optional: block the reference will point to, defaults to the `Seed` block
oldBlockHash: f162..f008  # optional: expected current block, task fails if the reference has moved
```

`GarbageCollection` - removes unreferenced data files of a dataset:

```yaml theme={null}
kind: GarbageCollection
target: Dataset:sergiimk/sensor-temp
```

`Verify` - checks dataset metadata for integrity, optionally replaying transformations to verify their reproducibility:

```yaml theme={null}
kind: Verify
target: Dataset:sergiimk/sensor-temp
replayTransform: true  # optional
```

`WebhookCall` - dispatches a payload to a <Schema /> resource:

```yaml theme={null}
kind: WebhookCall
endpoint: WebhookEndpoint:sergiimk/notify-slack
payload: '{"event": "ingest-complete", "dataset": "{{task.source}}"}'  # optional, supports templating
retryPolicy:  # optional: overrides the retry policy of the flow for this task
  maxAttempts: 5
  minDelay: 30s
  backoff: Exponential
```

## FlowRun

A `FlowRun` resource (<Schema />) is a set of tasks to be executed in a sequence:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/FlowRun
headers:
  name: flow-run-123456
  ownerReferences:
    - Flow:c27331ce-ce88-4ff9-8c5a-4ce8107cc03f  # Flow that spawned this run
spec:
  target: Dataset:sergiimk/foo
  tasks:
    - kind: Transform  # Implicit name: task-0-transform
    - kind: GarbageCollection
      name: gc
```

The `target` of the flow run is used as the default `target` of its tasks to avoid duplication.

The run controller spawns `Task` resources one after another and tracks the progress in the following conditions:

* <Schema /> - overall status of the run (`Waiting`, `Running`, or `Finished`) and the list of spawned tasks with their names, references, statuses, and outcomes
* <Schema /> - what led to the execution of the run: the time, the initiating <Term />, and a copy of the trigger configuration. Triggers that fired while the run was already queued or executing are recorded as `lateActivationCauses`. In case of a retry, the causes of the original run are preserved.
* <Schema /> - reference to the previous run and the retry number, if this run is a retry

```yaml theme={null}
status:
  phase: Ready
  conditions:
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowRunStatus:
      status: Running
      tasks:
        - name: task-0-transform
          task: Task:a1b2c3d4-1111-2222-3333-444444444444
          status: Finished
          outcome:
            kind: Success
          lastUpdatedAt: 2026-09-11T02:00:00Z
        - name: gc
          task: Task:e5f6a7b8-5555-6666-7777-888888888888
          status: Running
          lastUpdatedAt: 2026-09-11T02:02:00Z
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowRunActivationCauses:
      activationCauses:
        - activationTime: 2026-09-11T02:00:00Z
          initiator: User:alice
          trigger:
            kind: Manual
      lateActivationCauses: []
    https://opendatafabric.org/schemas/flows/v1alpha1/FlowRunRetry:
      retryOf: FlowRun:9b2e4f1a-3c7d-4e8b-a1f2-6d5e7c8b9a0d
      retryNumber: 2  # Second retry, i.e. third run total
```

Activation causes and retry information are known when the run is created, but they are part of `status` rather than `spec`, because they can only be reliably set by the controller, not by a user.

Like tasks, flow runs are **deleted as soon as the run finishes** and stay readable for the retention period. Deleting a run that has not finished yet cancels it, together with its unfinished tasks, which reference the run in `ownerReferences`.

A run created by a flow copies the `serviceAccount` of the `Flow` into its own spec, so changes to the flow affect only future runs, never runs in progress.

## Flow

A `Flow` resource (<Schema />) acts as a template for instantiating flow runs, and defines the triggers that decide when to instantiate them:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  name: compact-and-gc-roots
spec:
  target:  # Selector matching all root datasets under `sergiimk`
    type: Dataset
    account: sergiimk
    name: '%'
    labels:
      DatasetKind: Root
  triggers:
    - kind: Schedule
      cron: "@daily"
  tasks:
    - kind: Compaction
      maxSliceSize: 100MiB
      maxSliceRecords: 10000
      onNoOp: Break  # Skip garbage collection if there was nothing to compact
    - kind: GarbageCollection
  retryPolicy:
    maxAttempts: 3
    minDelay: 1min
    backoff: Exponential
  recentBindingsRetention:
    maxBindings: 10
  recentRunsRetention:
    maxRuns: 100
    maxAge: 30d
```

The `target` of the flow is used as the default target of its triggers and tasks, to avoid duplication while still allowing overrides (e.g. an `InputsUpdated` trigger watching a different dataset than the one the tasks operate on).

The flow controller reports the state of the flow in the <Schema /> condition:

* `status` - `Active` or `Paused`
* `recentBindings` and `bindingsTotal` - the last resources matched by the [target selector](#target-selector) and their total number
* `recentRuns` - summaries of recent flow runs, including their status, outcome, and retry information

The number of reported bindings and runs is controlled by `recentBindingsRetention` (<Schema />) and `recentRunsRetention` (<Schema />). Implementations may additionally cap these values.

### Target Selector

The `target` of a flow is a [selector](/odf/spec/automation/resource-framework#selectors). When it matches a resource, the flow controller sets up triggers for that resource automatically. As resources are created or deleted, the controller updates its set of active targets accordingly.

The association between a flow and its matched resources is a derived state maintained by the controller. Implementations may provide a dedicated API for querying this index, e.g. to list flows by target or targets by flow.

### Retry Policy

The `retryPolicy` (<Schema />) defines how a flow reacts to recoverable failures:

* `maxAttempts` - number of attempts before automatic scheduling of the flow is disabled
* `minDelay` - how long to wait until the first retry
* `backoff` - how the delay is scaled with subsequent attempts: `Linear` or `Exponential`. When omitted, retries use a constant delay

Tasks like `WebhookCall` may override the retry policy of the flow.

A retry creates a new `FlowRun` that references the failed one in its `FlowRunRetry` condition.

## Triggers

A flow specifies a set of triggers (<Schema />) that decide when to instantiate a `FlowRun`:

| Kind | Description |
| - | - |
| `Manual` | Fires on an API call or a UI action |
| `Schedule` | Fires on a [cron](https://en.wikipedia.org/wiki/Cron) schedule, e.g. `@daily` or `*/30 * * * *` |
| `Interval` | Fires at regular intervals, e.g. every `15m` |
| `Source` | Fires when a [source](/odf/spec/datasets/ingestion#sources) receives new data |
| `InputsUpdated` | Fires when inputs of the derivative datasets matching the `target` selector (defaulting to the flow target) receive new data |
| `Event` | Fires when an event bus event matching the filters is observed |

All triggers support the following common properties:

* `enabled` - allows to pause an individual trigger by setting it to `false`
* `cooldown` - minimum interval between activations of the trigger. The trigger fires upon the first activation, but if it fires again within the `cooldown` interval, the activation is postponed until the interval ends, batching multiple activations into one flow run
* `cooldownMaxBatch` - if the trigger is activated this many times during the `cooldown` interval, it fires without waiting for the interval to end

`Schedule` and `Interval` triggers are guaranteed to fire if the node was down when the next tick was supposed to happen, but fire only once for all missed ticks.

`InputsUpdated` triggers allow batching of new input data, e.g. to transform a derivative dataset only once enough new records accumulate in its inputs:

```yaml theme={null}
kind: InputsUpdated
minRecordsToAwait: 100  # wait until this many new input records accumulate
maxAwaitInterval: 1h  # but run at least once an hour
```

`Source` triggers similarly allow batching of pushed data:

```yaml theme={null}
kind: Source
source: Source:sensor.temp.http
minRecordsToAwait: 100  # wait until this many new records accumulate
maxAwaitInterval: 1h  # but run at least once an hour
```

`Event` triggers are a low-level mechanism that should be used sparingly, e.g. to run a flow on dataset `A` whenever dataset `B` is updated. They specify the event filters (<Schema />), and are commonly combined with `cooldown` to batch multiple events into one flow run.

```yaml theme={null}
kind: Event
events:
  type: dataset.ref.updated
cooldown: 10min
```

## Authorization

A task executes with the permissions of the user <Term /> who created it.

Tasks, flow runs, and flows can specify a `serviceAccount` - a non-human principal that gets permissions only through explicit policy bindings (see [Authorization](/odf/spec/auth-and-privacy/authorization#service-account)):

* A flow owned by an [Organization](/odf/spec/auth-and-privacy/authorization#organization) must specify a service account. When omitted for flows owned by users, tasks execute with the permissions of the user account that owns the flow.
* A flow run inherits the service account of its flow, and a task inherits the service account of its run, unless overridden.

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  account: acme
  name: ingest-sensors
spec:
  serviceAccount: ServiceAccount:acme/ingest-bot
  target: Dataset:acme/sensor-temp
  # ...
```

## Schemas

| Schema | Description |
| - | - |
| <Schema /> | Task resource |
| <Schema /> | Task types and their parameters |
| <Schema /> | Execution phase of a task |
| <Schema /> | Execution plan of a task |
| <Schema /> | Result of a task |
| <Schema /> | Type-specific details of a task failure |
| <Schema /> | How a flow run proceeds after a `NoOp` task |
| <Schema /> | How a flow run proceeds after a failed task |
| <Schema /> | Remote location of a sync task |
| <Schema /> | Credentials of a sync task |
| <Schema /> | Flow run resource |
| <Schema /> | Tasks of a flow run |
| <Schema /> | Execution status of a flow run |
| <Schema /> | What caused a flow run |
| <Schema /> | Link to the run being retried |
| <Schema /> | Flow resource |
| <Schema /> | Target, triggers, tasks, and retry policy of a flow |
| <Schema /> | Status and recent runs of a flow |
| <Schema /> | Conditions that cause a flow to execute |
| <Schema /> | Reaction to failures |
| <Schema /> | Retention of recent bindings in flow status |
| <Schema /> | Retention of recent runs in flow status |


# Data & Checkpoint Hashing
Source: https://docs.kamu.dev/odf/spec/datasets/data-and-checkpoint-hashing



**Domain**: Datasets\
**Depends on**: [Data Model](/odf/spec/datasets/data-model)\
**RFCs**: [RFC-002](/odf/rfcs/002-logical-data-hashes)

## Abstract

This spec defines how hash sums of data part files and checkpoints are computed, including the physical hashes used to identify files in content-addressable storage and the logical hashes of data records that remain stable regardless of how data is physically encoded.

## Physical vs. Logical Hashes

Because the on-disk Parquet data format is non-deterministic (serializing same logical data may result in different binary layouts depending on heuristics and implementation) two different hash sums are maintained by the coordinator:

* **Physical hash** - a (non-reproducible) hash sum of an entire binary file as originally produced by the owner of the dataset
* **Logical hash** - a stable (reproducible) hash sum computed on data records, resistant to encoding variations

**Physical hashing** is used in the context of verifying integrity of the datasets and uploading and downloading data files to/from content-addressable systems, where hash sum acts as an identifier of the data file. Physical hash is calculated using [SHA3-256](https://en.wikipedia.org/wiki/SHA-3) algorithm on entire data part file.

**Logical hashing** is used for equivalence checks when reproducing results of computations. It is computed over the in-memory representation of data in [Apache Arrow](https://arrow.apache.org/) format using the algorithm described below.

## Validation and Verification

Hashes stored in <Term /> are used in two scenarios:

* **Validation of integrity** - when a data part file or a checkpoint is obtained from any (possibly untrusted) source, its physical hash is computed and compared to the `physicalHash` stored in the corresponding <Schema /> or <Schema /> object. A mismatch means that the file was corrupted or tampered with, and it must be rejected.
* **Verification of reproduced results** - when data is reproduced by repeating a transformation (see [Processing Model: Verification via Reproducibility](/odf/spec/transformations/processing-model#verification-via-reproducibility)), the physical hash of the result is compared first. If it doesn't match due to non-determinism of the on-disk format, the logical hash is compared instead, and a match confirms that the resulting records are logically the same.

Checkpoints are engine-specific and are not required to be reproducible, so their hashes are only used for validation of integrity.

## Logical Hashing Algorithm

The logical hashing algorithm computes a stable hash of structured data based on its [Apache Arrow](https://arrow.apache.org/docs/format/Columnar.html#physical-memory-layout) in-memory representation. The algorithm is parametrized by the underlying digest function, which in ODF is [SHA3-256](https://en.wikipedia.org/wiki/SHA-3).

The current version of the algorithm is `0`, as reflected in the `arrow0-sha3-256` [hash code](#hash-codes). Future versions may change the resulting hashes, in which case they will be assigned new hash codes.

The reference implementation is provided by the [arrow-digest](https://github.com/sergiimk/arrow-digest) Rust crate.

### Properties

* The hash is the same no matter how many record batches the data was split into
* The hash is the same no matter which Arrow encoding variant of a type is used (e.g. `Utf8`, `LargeUtf8`, and `Utf8View`) - the difference between them is purely an encoding concern
* The hash is the same for arrays without a validity bitmap and arrays where a validity bitmap marks all values as valid
* The hash is **order-dependent** - reordering records will produce a different hash

### Value Encoding

Values are fed into the digest function according to the following rules:

* **Endianness** - all multi-byte values are encoded as little-endian
* **Fixed size types** (`Int`, `FloatingPoint`, `Decimal`, `Date`, `Time`, `Timestamp`) - hashed using their in-memory binary representation
* **Booleans** - hashed as byte-sized values: `1` for `false` and `2` for `true`
* **Binaries and strings** (`Binary`, `LargeBinary`, `FixedSizeBinary`, `BinaryView`, `Utf8`, `LargeUtf8`, `Utf8View`) - length of the value in bytes as `u64`, followed by the bytes of the value
* **Lists** (`List`, `LargeList`, `FixedSizeList`) - length of the list as `u64`, followed by the items of the list hashed according to their data type
* **Nulls** - every null value is represented by a single `0` byte

### Data Type Encoding

When hashing an array, its data type is hashed once at the start of the hashing session, followed by all items of the array hashed sequentially.

Data type is encoded as a type identifier (`u16`) followed by type parameters:

| Arrow Type | Type ID | Followed by |
| - | - | - |
| `Null` | `0` | |
| `Int8`, `Int16`, `Int32`, `Int64` | `1` | `1` (signed) as `u8`, bit width as `u64` |
| `UInt8`, `UInt16`, `UInt32`, `UInt64` | `1` | `0` (unsigned) as `u8`, bit width as `u64` |
| `Float16`, `Float32`, `Float64` | `2` | bit width as `u64` |
| `Binary`, `LargeBinary`, `FixedSizeBinary`, `BinaryView` | `3` | |
| `Utf8`, `LargeUtf8`, `Utf8View` | `4` | |
| `Boolean` | `5` | |
| `Decimal32`, `Decimal64`, `Decimal128`, `Decimal256` | `6` | bit width as `u64`, precision as `u64`, scale as `u64` |
| `Date32` | `7` | `32` as `u64`, date unit `Day` |
| `Date64` | `7` | `64` as `u64`, date unit `Millisecond` |
| `Time32`, `Time64` | `8` | bit width as `u64`, time unit |
| `Timestamp` | `9` | time unit, time zone (a single `0` byte if absent, otherwise its length as `u64` followed by its UTF-8 bytes) |
| `List`, `LargeList`, `FixedSizeList` | `11` | item data type |

Type IDs `10` (`Interval`), `12` (`Struct`), `13` (`Union`), `16` (`Map`), and `17` (`Duration`) are reserved.

Units are encoded as `u16`:

| Date Unit | ID |
| - | - |
| `Day` | `0` |
| `Millisecond` | `1` |

| Time Unit | ID |
| - | - |
| `Second` | `0` |
| `Millisecond` | `1` |
| `Microsecond` | `2` |
| `Nanosecond` | `3` |

### Record Batch Hashing

Data consisting of multiple columns is hashed as follows:

1. *(once per hashing session)* The schema is hashed into the combined digest by traversing all fields recursively in **depth-first** order and hashing for every field:
   * Field name as length in bytes (`u64`) followed by its UTF-8 bytes
   * Nesting level of the field (zero-based) as `u64`
2. Every leaf column (a column that is not a struct) is hashed by a separate column digest using the rules above:
   * Nested columns use a **combined nullability bitmap** produced from the nullability of the column and all of its parent structs
   * Subsequent record batches update the same column digests
3. *(final step)* Digests of all leaf columns are fed into the combined digest in the depth-first order to produce the final hash.

## Checkpoint Hashing

<Term /> are stored as opaque files and referenced by <Term /> using their physical hash. The process of computing a hash sum is identical to computing a physical hash for a data part file (see [Physical vs. Logical Hashes](#physical-vs-logical-hashes)).

## Hash Codes

Hashes are represented using [multihash](https://github.com/multiformats/multihash) and [multibase](https://github.com/multiformats/multibase) as described in [Hash Representation](/odf/spec/datasets/metadata-chain#hash-representation).

The `multicodec` table is extended with the following codes in the "private use area":

| Codec | Code |
| - | - |
| `arrow0-sha3-256` | `0x300016` |

See also:

* [RFC-002](/odf/rfcs/002-logical-data-hashes)


# Data Model
Source: https://docs.kamu.dev/odf/spec/datasets/data-model



**Domain**: Datasets\
**Depends on**: [Data Schema](/odf/spec/datasets/data-schema)\
**RFCs**: [RFC-001](/odf/rfcs/001-record-offsets), [RFC-015](/odf/rfcs/015-unified-changelog-stream-schema)

## Abstract

This spec defines how data of a dataset is represented in memory and at rest, the system columns that all data in ODF carries to support the bitemporal event model, the changelog representation of retractions and corrections, and the rules according to which a dataset's schema is allowed to evolve over time.

## In-memory and On-disk Formats

We differentiate two data formats by their purpose:

* **In-memory format** - used when passing data around between the subsystems (e.g. when <Term /> communicates with the <Term />).
* **On-disk format** - used for data at rest and by query engines when querying data from the entire dataset.

For our **in-memory format** we chose [Apache Arrow](https://arrow.apache.org/), as it is:

* Purposefully designed for minimal overhead interoperability between data processing systems
* Minimizes copying of data
* Hardware efficient
* Supports streaming

For our **on-disk format** we choose [Apache Parquet](https://parquet.apache.org/), as it is:

* Fast to decode into [Apache Arrow](https://arrow.apache.org/)
* Space and IO efficient thanks to the built-in compression
* Efficient for querying thanks to the columnar structure and built-in basic indexing

Data on disk is stored in the multiple "part" files. Once a part file is written it is immutable for the entire lifetime of the dataset.

## Common Data Schema

All data in the system is guaranteed to have the following columns:

| Column | Description |
| :-: | - |
| `offset` | <Term /> is a sequential identifier of a row relative to the start of the dataset (first row has an offset of `0`) |
| `op` | <Term /> is used to differentiate regular append events from retractions and corrections |
| `system_time` | <Term /> denotes when an event first appeared in the dataset. This will be an ingestion time for events in the <Term /> or transformation time in the <Term /> |
| `event_time` | <Term /> denotes when to our best knowledge an event has occurred in the real world. By default all temporal computations (windowing, aggregations, joins) are done in the event time space thus giving the user query an appearance of a regular flow of events even when data is backfilled or frequently arrives out-of-order |

Representation:

| Column | Arrow Type | Parquet Type | Recommended Parquet Encoding |
| :-: | :-: | :-: | :-: |
| `offset` | `uint64` | `INT64` | `DELTA_BINARY_PACKED` |
| `op` | `uint8` | `INT32` | `RLE_DICTIONARY` |
| `system_time` | `timestamp(ms, UTC)` | `INT64, TIMESTAMP(MILLIS, AdjustedToUTC=true)` | `PLAIN_DICTIONARY` |
| `event_time` | `timestamp(ms, UTC)` | `INT64, TIMESTAMP(MILLIS, AdjustedToUTC=true)`<br />or `int32, DATE` | |

> **TODO:**
>
> * We are not allowing non-UTC-adjusted timestamps yet as Parquet does not offer a way to encode the timezone, meaning we need a reliable way to pass timezone information between different engines through some other means (e.g. Parquet metadata). Having naive/local timestamps without enforcing that they are accompanied by the specific timezone would be too error-prone.

## Representation of Retractions and Corrections

<Term /> are differentiated from regular <Term /> in the <Term /> via special `op` column, carrying an <Term />.

Valid operation types are:

| Value | Operation name | Operation short code |
| :-: | :-: | :-: |
| 0 | `append` | `+A` |
| 1 | `retract` | `-R` |
| 2 | `correct-from` | `-C` |
| 3 | `correct-to` | `+C` |

The `retract` and `correct-from` events must carry the same exact data fields (all fields excluding `offset`, `op`, and `system_time`) as the event that is being retracted.

The `correct-to` event carries the new values for the event being corrected. This event must immediately follow the `correct-from` event. This model allows data engines that operate on changelog / diff events that simultaneously carry both old and new values for event during a correction to reconstruct such events easily by sequentially reading the <Term />.

A pair of `correct-from` and `correct-to` events must belong to the same <Term /> - they cannot span across slices.

The order of records in a dataset, as defined by their <Term />, is significant: a retraction or correction always follows the event it applies to, and correction pairs must stay adjacent. Records therefore must not be reordered (e.g. sorted by <Term />) when they are processed or written, as that may produce an invalid stream.

Not every stream needs to use all operation types. The following are all valid subsets of this model:

* **Append-only stream** - contains only `append` events
* **Retract stream** - contains only `append` and `retract` events, where a change of a previously published event is represented as its retraction followed by an append of the new value
* **Changelog stream** - uses all operation types, differentiating retractions from corrections

See also:

* [RFC-015: Unified Changelog Stream Schema](/odf/rfcs/015-unified-changelog-stream-schema)

## Schema Evolution

<Term /> <Term /> may change when:

* A different <Term /> process is specified in the <Term />
* A transformation <Term /> of the <Term /> is changed
* A <Term /> of one of the inputs of the <Term /> changes

In general, <Term /> changes are restricted to only adding new columns. Removing and renaming columns or changing data types would break downstream data consumers and therefore not allowed. A migration procedure should be used instead when it's desired to introduce breaking changes in the <Term />.


# Data Schema
Source: https://docs.kamu.dev/odf/spec/datasets/data-schema



**Domain**: Datasets\
**Depends on**: -\
**RFCs**: [RFC-016](/odf/rfcs/016-odf-schema)

## Abstract

This spec defines the ODF schema format used to describe the logical structure of <Term /> independently of how it is physically encoded in files (e.g. Parquet, Avro, CSV) or in memory (e.g. Arrow). It defines the set of supported logical types, the mechanism for attaching custom attributes to schemas and fields, and the rules for converting between ODF and Apache Arrow schemas.

## Logical Types vs. Encoding

ODF schema describes **logical types** of data - the semantics and value boundaries of a type and how it can be operated on. It deliberately does not capture the **encoding** (physical layout) of data such as compression, dictionary encoding, offset widths, or view-based representations.

This separation ensures that:

* Logical types remain stable across different file formats, data chunks, and engines
* Encoding strategies (compression, dictionary usage, encoding hints) can evolve independently, both at the file level and in memory
* Producers and consumers can negotiate encoding at runtime while relying on a consistent logical schema contract

While inspired by [Apache Arrow](https://arrow.apache.org/), ODF schema intentionally differs from the [Arrow flatbuffer schema](https://github.com/apache/arrow/blob/main/format/Schema.fbs), which mixes logical types with in-memory layout details and is not suited to be defined manually by humans.

## Schema Format

A <Term /> (`DataSchema`) consists of:

* `fields` - an ordered list of top-level fields (columns)
* `extra` - optional [extra attributes](#extra-attributes) that apply to the schema as a whole

Each field (`DataField`) consists of:

* `name` - name of the field
* `type` - [logical type](#logical-types) of the field
* `extra` - optional [extra attributes](#extra-attributes) that apply to the field

Example:

```yaml theme={null}
fields:
  - name: offset
    type:
      kind: UInt64
  - name: op
    type:
      kind: UInt8
  - name: system_time
    type:
      kind: Timestamp
      unit: Millisecond
      timezone: UTC
  - name: event_time
    type:
      kind: Timestamp
      unit: Millisecond
      timezone: UTC
  - name: mri_content_hash
    type:
      kind: String
    extra:
      opendatafabric.org/description: References the MRI scan data linked to the dataset by its hash
  - name: subject
    type:
      kind: Struct
      fields:
        - name: id
          type:
            kind: String
          extra:
            opendatafabric.org/description: Subject's unique identity
        - name: gender
          type:
            kind: Option
            inner:
              kind: String
          extra:
            opendatafabric.org/description: Subject's gender
    extra:
      opendatafabric.org/description: Information about the subject
  - name: area_codes
    type:
      kind: List
      itemType:
        kind: String
    extra:
      opendatafabric.org/description: List of body area codes covered by this MRI scan
extra:
  c.com/z: baz
```

See <Schema /> reference for the full definition.

### Nullability

Fields are non-nullable by default. Nullability is expressed explicitly via the special `Option` type that wraps the type of the value (unlike Arrow, where nullability is a property of a field). This applies recursively to nested types, e.g. struct fields and list items.

## Logical Types

| Type | Parameters | Description |
| - | - | - |
| `Bool` | | Boolean value representing true or false |
| `Int8`, `Int16`, `Int32`, `Int64` | | Signed integer |
| `UInt8`, `UInt16`, `UInt32`, `UInt64` | | Unsigned integer |
| `Float16`, `Float32`, `Float64` | | Floating-point number |
| `Decimal` | `precision`, `scale` | Fixed-point decimal number with specified total number of digits and number of digits after the decimal point (scale can be negative) |
| `String` | | Unicode string |
| `Binary` | `fixedLength` (optional) | Sequence of bytes, variable-length unless `fixedLength` is specified |
| `Date` | | Calendar date |
| `Time` | `unit` (default `Millisecond`) | Time of day without a date |
| `Timestamp` | `unit` (default `Millisecond`), `timezone` (default `UTC`) | Point in time represented as an offset from the Unix epoch. Timezone is either a name from the Olson timezone database (e.g. `America/New_York`) or an absolute offset (e.g. `+07:30`) |
| `Duration` | `unit` (default `Millisecond`) | Elapsed time interval |
| `List` | `itemType`, `fixedLength` (optional) | List of values of the same type, variable-length unless `fixedLength` is specified |
| `Map` | `keyType`, `valueType`, `keysSorted` (optional) | Map of key-value pairs, represented as a list of entries |
| `Struct` | `fields` | Collection of named [fields](#schema-format), each with its own type |
| `Option` | `inner` | Optional (nullable) value of the inner type |
| `Null` | | Type representing the absence of a value |

Time units are: `Second`, `Millisecond`, `Microsecond`, `Nanosecond`.

See <Schema /> reference for the full definition.

### Short Form

Types that don't require any parameters can be specified as a plain string instead of an object with a `kind` field. The following two definitions are equivalent:

```yaml theme={null}
- name: offset
  type:
    kind: UInt64
```

```yaml theme={null}
- name: offset
  type: UInt64
```

Short form is an input-only convenience - the expanded form is used when schema is stored in binary formats.

## Extra Attributes

Extra attributes is a key-value map that allows attaching custom information to schemas and fields.

Keys must be in the format `<domain>/<attribute-name>` (e.g. `kamu.dev/archetype`). The domain prefix is used to disambiguate attributes and avoid name collisions between different extensions.

Values can be a string or any JSON value, including nested objects.

Extra attributes are represented natively in YAML and JSON, and as a JSON-encoded string in FlatBuffers.

### Well-known Attributes

| Attribute | Applies to | Description |
| - | - | - |
| `opendatafabric.org/description` | Field | Human-readable description of the field |
| `arrow.apache.org/bufferEncoding` | Field | Arrow buffer encoding hint (see [Conversion to and from Arrow](#conversion-to-and-from-arrow)) |
| `arrow.apache.org/dateEncoding` | Field | Arrow date encoding hint (see [Conversion to and from Arrow](#conversion-to-and-from-arrow)) |
| `arrow.apache.org/decimalEncoding` | Field | Arrow decimal encoding hint (see [Conversion to and from Arrow](#conversion-to-and-from-arrow)) |

## Conversion to and from Arrow

Implementations should be able to convert between ODF and Arrow schemas back and forth without information loss. Since ODF schema captures only logical types, Arrow encoding details are preserved using encoding hints in [extra attributes](#extra-attributes):

**Arrow type:** `LargeUtf8`

```yaml theme={null}
type:
  kind: String
extra:
  arrow.apache.org/bufferEncoding:
    kind: Contiguous
    offsetBitWidth: 64
```

**Arrow type:** `Utf8View` (analogously for `BinaryView`, `ListView`, `LargeListView`)

```yaml theme={null}
type:
  kind: String
extra:
  arrow.apache.org/bufferEncoding:
    kind: View
    offsetBitWidth: 32
```

**Arrow type:** `Date64` (millisecond-based representation)

```yaml theme={null}
type:
  kind: Date
extra:
  arrow.apache.org/dateEncoding:
    unit: Millisecond
```

**Arrow type:** `Decimal128` (analogously for `Decimal256`)

```yaml theme={null}
type:
  kind: Decimal
extra:
  arrow.apache.org/decimalEncoding:
    bitWidth: 128
```

## Schema in APIs

When a schema is returned by APIs it should carry only the logical type information and not the encoding hints, unless the encoding represents the actual encoding of the data that this API transmits.


# Identity
Source: https://docs.kamu.dev/odf/spec/datasets/identity



**Domain**: Datasets\
**Depends on**: [Metadata Chain](/odf/spec/datasets/metadata-chain)\
**RFCs**: [RFC-003](/odf/rfcs/003-content-addressability), [RFC-007](/odf/rfcs/007-simple-transfer-protocol), [RFC-012](/odf/rfcs/012-recommend-base16-encoding)

## Abstract

This spec defines how <Term /> are identified. It defines the `did:odf` [DID](https://www.w3.org/TR/did-core/) method used to create globally unique dataset identifiers that are tied to a cryptographic key pair and require no central authority, and the human-friendly aliases and references used to refer to datasets within and across <Term />.

## Overview

The identity of a dataset consists of:

* [Unique identifiers](#unique-identifiers) - used to unambiguously identify datasets on the network, e.g. when referencing one dataset as an input of another.
* [Aliases and references](#aliases-and-references) - used for human-friendly naming.

## Unique Identifiers

ODF is designed to be compatible with [content-addressable storage](https://en.wikipedia.org/wiki/Content-addressable_storage). When stored in such storage, all parts of a <Term /> can be accessed by having only a hash of the last <Term />.

As <Term /> grows, however, a reference to a single <Term /> will only provide access to a subset of its history. To refer to a <Term /> as a whole a different identifier is needed that can be resolved into a hash of the latest block. Such identifiers must be globally unique, issuable with no central authority, and tied to a proof of ownership over a dataset.

ODF follows the [W3C DID Identity Scheme](https://w3c.github.io/did-core/) using a custom `did:odf` method. This method is based closely on [`did:key` method](https://w3c-ccg.github.io/did-method-key/) that derives the unique identity from a public key of a cryptographic key pair.

Example:

<pre>
  did:odf:fed012126262ba49e1ba8392c26f7a39e1ba8d756c7469786d3365200c68402ff65dc
</pre>

The identifier is formed by:

* Generating a cryptographic key pair (e.g. using `ed25519` algorithm)
* Taking the public key part
* Prepending an appropriate [multicodec](https://github.com/multiformats/multicodec) value to identify the algorithm (e.g. `ed25519-pub`)
* Encoding data with [multibase](https://github.com/multiformats/multibase) encoding using `base16` scheme

Or in pseudocode:

<pre>
  did-odf-format := 'did:odf:' + MULTIBASE(
  base16,
  MULTICODEC(
  public\_key\_type,
  public\_key\_bytes,
  )
  )
</pre>

The resulting DID is stored in the first <Term /> in the chain of every <Term />, called <Schema /> (see [Metadata Chain](/odf/spec/datasets/metadata-chain#core-events)).

Tying the identity of a dataset to a cryptographic key pair provides a way to create unique identity in a fully decentralized way. The corresponding private key can be used to prove ownership and control over a <Term /> and to delegate access.

> Note that the only difference between `did:odf` and `did:key` method is the encoding scheme. ODF uses `base16` instead of `base58btc` for reasons explained in the [Hash Representation](/odf/spec/datasets/metadata-chain#hash-representation) section. Because conversion between the two encodings is trivial, ODF is compatible with authorization frameworks that work with `did:key` method. We expect that in future the `did:key` spec will be extended to allow `base16` encoding too, further minimizing the difference.

## Aliases and References

Formats described below provide human-friendly ways to refer to a certain dataset. Note that they are only meaningful within the boundaries of a <Term />. Unlike [Dataset IDs](#unique-identifiers) they are not collision-free and mutable.

Depending on the context we differentiate the following types of references:

* **Local Format** - used to refer to a dataset within a local workspace (can be single- and multi-tenant)
* **Remote Format** - used to refer to a dataset located in a known <Term /> provider.

As you will see in the examples below, we recommend (but not require) using the [reverse domain name notation](https://en.wikipedia.org/wiki/Reverse_domain_name_notation) for <Term /> names.

Examples:

<pre>
  // Dataset ID
  did:odf:fed012126262ba49e1ba8392c26f7a39e1ba8d756c7469786d3365200c68402ff65dc

  // Local Format - DatasetName is highlighted
  <b>property.parcel-polygons</b>
  <b>cities</b>
  <b>admin0.countries.10m</b>

  // Local Multi-tenant Format - AccountName is highlighted
  <b>statcan.gc.ca</b>/census.2016.population
  <b>ny-newyork</b>/public-safety.ems-incident-dispatch

  // Remote Format - RepoName is highlighted
  <b>statcan.gc.ca</b>/census.2016.population
  <b>us.cityofnewyork</b>/public-safety.ems-incident-dispatch
  <b>data.gov</b>/ny-newyork/public-safety.ems-incident-dispatch

  // Remote Url Format
  [https://opendata.ca/odf/census-2016-population/](https://opendata.ca/odf/census-2016-population/)
  ipfs\://bafkreie3hfshd4ikinnbio3kewo2hvj6doh5jp3p23iwk2evgo2un5g7km/
</pre>

Note that name components such as `DatasetName`, `AccountName`, `RepoName` are case-insensitive, i.e. names `ny-newyork` and `NY-NewYork` are considered identical during lookups and uniqueness checks within the repository.

### Grammar

Full [PEG](https://en.wikipedia.org/wiki/Parsing_expression_grammar) grammar:

```peg theme={null}
DatasetRefAny = 
  DatasetRefRemote /
  DatasetRef

DatasetRef = 
  DatasetId /
  DatasetAlias

DatasetRefRemote =
  (RepoName "/")? DatasetId /
  DatasetAliasRemote /
  Url

DatasetAlias = 
  (AccountName "/")? DatasetName

DatasetAliasRemote = 
  RepoName "/" (AccountName "/")? DatasetName

DatasetId = "did:odf:" Multibase
DatasetName = Hostname
AccountName = Hostname
RepoName = Hostname

Hostname = Subdomain ("." Subdomain)*
Subdomain = [a-zA-Z0-9]+ ("-" [a-zA-Z0-9]+)*

Multibase = [a-zA-Z0-9+/=]+
Url = Scheme "://" [^\n]+
Scheme = [a-zA-Z0-9]+ ("+" [a-zA-Z0-9]+)*
```

Remote references in the URL format point directly to the location of a dataset (e.g. a directory served over HTTP or stored in IPFS), where the scheme selects the transfer protocol (see [Transfer Protocols](/odf/spec/exchange/transfer-protocols)).

## References in Metadata

Datasets can refer to other datasets in their <Term />, e.g. a <Term /> specifies its inputs in the `SetTransform` event (<Schema />). Such references:

* Can be specified using any form of local or remote reference when the dataset is defined
* Must be resolved into a dataset ID by the time the block is accepted into the chain, to guarantee reproducibility, as aliases can change over time
* Carry a separate `alias` under which the input is available in queries, decoupling queries from the names of input datasets

## Identity of Dataset Resources

The `Dataset` <Term /> that describes the desired state of a dataset on a <Term /> has its own identity that is separate from the identity of the dataset itself (see [Resource Framework: Resources and Entities](/odf/spec/automation/resource-framework#resources-and-entities)):

* The resource `id` identifies the resource within one node, while the `did` identifies the dataset on the global network. When a dataset is replicated between nodes, every node has its own `Dataset` resource with a different `id` but the same `did`.
* In resource manifests, datasets are referred to using [resource references](/odf/spec/automation/resource-framework#references) (e.g. `Dataset:acme/sensor-temp`), which can also identify the dataset by its `did`.


# Ingestion
Source: https://docs.kamu.dev/odf/spec/datasets/ingestion



**Domain**: Datasets\
**Depends on**: [Data Model](/odf/spec/datasets/data-model), [Data Schema](/odf/spec/datasets/data-schema), [Resource Framework](/odf/spec/automation/resource-framework)\
**RFCs**: [RFC-009](/odf/rfcs/009-ingest-source-state), [RFC-011](/odf/rfcs/011-push-ingest-sources), [RFC-014](/odf/rfcs/014-minimize-offset-scanning)

## Abstract

This spec defines how external data is brought into <Term />. It defines the `Source` resource that describes where data comes from and how it is read, the phases of the ingestion process, the <Term /> that turn data of various nature into an append-only history of events, how <Term /> is determined, and how the state of a source is preserved to allow incremental ingestion.

## Purpose

When interacting with data on the web we cannot make any assumptions about the guarantees the external source provides:

* Just by looking at the data we cannot tell if it is mutable or preserves the history of changes
* We often can't tell where it originates from and who can be held accountable for its validity
* We can't be sure that it's sufficiently replicated - the domain that hosts it may not exist the next day

<Term /> is about getting the external data into the system as efficiently as possible, into the strict ledgers of <Term />, where properties like clear ownership, reproducibility, accountability, and complete historical account can be guaranteed and made explicit.

This spec does not aim to standardize all possible data ingestion techniques. ODF is not meant to replace data workflow and data extraction tools - the ingestion mechanisms described here simplify the very first step in the data's journey through the stream processing pipelines, and implementations may support additional ingress types, formats, and steps.

## Sources

A source of data is described by a `Source` <Term />:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/sources/v1alpha1/Source
headers:
  name: ca.bankofcanada
spec:
  # Makes variables and secrets available for templating under `config`
  config:
    apiKey: SecretSet:ca.bankofcanada#apiKey
  # Where the data comes from
  ingress:
    kind: Url
    url: https://www.bankofcanada.ca/valet/observations/group/FX_RATES_DAILY/csv
    headers:
      - name: Authorization
        value: "Bearer ${{ config.apiKey }}"
  # How to read the data into structured form
  read:
    kind: Csv
    header: true
    schema:
      fields:
        - name: date
          type: Timestamp
        - name: FXUSDCAD
          type:
            kind: Decimal
            precision: 19
            scale: 6
  # OPTIONAL: Query that shapes the data
  preprocess:
    kind: Sql
    engine: datafusion
    query: |
      select
        date as event_time,
        FXUSDCAD as fxusdcad
      from input
  # How to combine new data with the history of previously seen data
  merge:
    kind: Ledger
    primaryKey:
      - event_time
```

A source is decoupled from the datasets it writes to. Ingestion is performed by an `Ingest` task that references the `Source` and the target root dataset, and that can be scheduled by a flow (see [Tasks and Flows](/odf/spec/automation/tasks-and-flows)):

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/flows/v1alpha1/Flow
headers:
  name: ca.bankofcanada
spec:
  target: Dataset:ca.bankofcanada
  triggers:
    - kind: Schedule
      cron: "*/30 * * * *"
  tasks:
    - kind: Ingest
      source: Source:ca.bankofcanada
      targetRecordsPerSlice: 10000
```

All data ingested into a dataset must conform to the dataset's schema, as captured by the <Schema /> event.

### Push and Polling Sources

Depending on the type of [ingress](#ingress), sources fall into two categories:

* **Polling sources** - the node periodically scans a remote data source and ingests the latest updates. This is suitable for periodic database dumps, data published as files on the web, bulk data APIs, and blockchains.
* **Push sources** - an external actor actively sends data into the node. This is suitable for IoT devices, business processes that report events directly, and streaming data APIs and event queues.

Although we aspire to reach a state where all authoritative data publishers push new events as soon as those occur, we realize that this level of integration will take time to achieve, and for a long time the vast majority of data will be ingested via the polling model.

## Ingestion Phases

The ingestion process consists of the following phases, directly reflected in the structure of the <Schema />:

| Phase | Field | Description |
| - | - | - |
| Ingress | `ingress` | Obtains the data from an external source in its original raw form (e.g. via HTTP), or accepts the data pushed by an external actor |
| Prepare (optional) | `prepare` | Prepares raw binary data for reading, e.g. by extracting an archive or converting between formats |
| Read | `read` | Reads the data in some format into a structured form |
| Preprocess (optional) | `preprocess` | Reshapes the structured data with a query, e.g. renaming columns, type conversions, nesting |
| Merge | `merge` | Combines the new data with the history of previously seen data (see [Merge Strategies](#merge-strategies)) |

Additionally:

* `vocab` - specifies the names of the system columns (see <Schema />) in case the data uses names different from the defaults
* `config` - brings variables and secrets into the `config` context, so they can be used for templating values of other fields (e.g. `${{ config.apiKey }}`)

## Ingress

The `ingress` field (<Schema />) determines where data comes from:

| Kind | Type | Description |
| - | - | - |
| `Url` | Polling | Fetches data by its URL using one of the supported protocols (e.g. HTTP, FTP), optionally passing request headers |
| `FilesGlob` | Polling | Matches a growing set of files on the local file system using a glob pattern |
| `Container` | Polling | Runs an OCI container to fetch data from an arbitrary source (see [Container Interface](#container-interface)) |
| `EvmLogs` | Polling | Streams transaction logs from an EVM-compatible blockchain node (see [EVM Logs](#evm-logs)) |
| `Mqtt` | Push | Subscribes to topics of an MQTT broker |
| `RestEndpoint` | Push | Exposes an HTTP endpoint that accepts pushed records |

`FilesGlob` is used for directories containing a growing set of files, e.g. periodic database dumps or batches of new data. Files should never change once added - the source remembers the last ingested file and only considers files that come after it in the order specified by `order`: lexicographically by name (`ByName`) or by [event time](#event-time-sources) (`ByEventTime`). Every file is processed individually and advances the <Term /> of the dataset.

Push ingresses buffer the received records until they are ingested by a task. The `RestEndpoint` ingress specifies the `buffer` (<Schema />) with its size and the `overflowPolicy` applied when the buffer is full: rejecting new records (`Reject`) or dropping the oldest ones (`DropOldest`).

### Event Time Sources

Ideally, data records contain <Term /> within them as a column, but many data sources don't. Polling ingresses can specify how event time is determined via the `eventTime` field (<Schema />):

| Kind | Description |
| - | - |
| `FromMetadata` | Extracts event time from the metadata of the source, e.g. modification time of files or the `Last-Modified` header of HTTP resources |
| `FromPath` | Extracts event time from the path of the source using a regular expression whose first group contains the timestamp, and a timestamp format |
| `FromSystemTime` | Uses the <Term /> of ingestion as the event time |

Example of extracting event time from file names:

```yaml theme={null}
ingress:
  kind: FilesGlob
  path: /data/db-table-dump-*.csv
  eventTime:
    kind: FromPath
    pattern: 'db-table-dump-(\d+-\d+-\d+)\.csv'
    timestampFormat: '%Y-%m-%d'
```

Event time is especially important for the [Snapshot](#snapshot) merge strategy, as snapshot-style exports represent the state of a system at a certain time that is usually captured only in some form of metadata (e.g. in the name of the file, in the URL, or in HTTP caching headers).

### Caching

Implementations should avoid redundant work and not ingest data if the source was not updated since the last poll. The exact mechanism depends on the ingress type and the protocol, e.g. for HTTP it relies on standard caching headers like `ETag` and `Last-Modified`, and the state needed for this is preserved as [source state](#source-state).

Caching behavior can be controlled via the `cache` field (<Schema />). The `Forever` kind specifies that after the source was processed once it will never be ingested again.

### Container Interface

The `Container` ingress allows using a general purpose programming language to deal with particularly complex APIs, or for web scraping. The specified OCI image is expected to conform to the following interface:

* Produce data to `stdout`
* Write warnings and errors to `stderr`
* Use the following environment variables:

| Variable | Description |
| - | - |
| `ODF_LAST_MODIFIED` | Last modified time of data from the previous ingest run, if any (in RFC 3339 format) |
| `ODF_ETAG` | Caching tag of data from the previous ingest run, if any |
| `ODF_BATCH_SIZE` | Recommended number of records, for scripts that provide a continuous stream of data and can resume from previous state |
| `ODF_NEW_LAST_MODIFIED_PATH` | Path to a text file where the script may write the new `Last-Modified` timestamp |
| `ODF_NEW_ETAG_PATH` | Path to a text file where the script may write the new `ETag` |
| `ODF_NEW_HAS_MORE_DATA_PATH` | Path to a file which the script can create to indicate that more data is available for the next batch. When the file is created, the new `ETag` or `Last-Modified` must also be written |

### EVM Logs

The `EvmLogs` ingress reads logs of EVM-compatible blockchains directly from a node. The `filter` field is an SQL `WHERE` clause used to create the most efficient RPC request, and the optional `signature` specifies a Solidity event signature used to decode logs.

The output structure corresponds to the output of the [`eth_getLogs`](https://ethereum.org/en/developers/docs/apis/json-rpc/#eth_getlogs) RPC method:

| Column | Type | Nullable | Description |
| - | - | - | - |
| `block_number` | `UInt64` | | |
| `block_hash` | `Binary` | | |
| `block_timestamp` | `Timestamp` | ✅ | Not returned by all node providers |
| `transaction_index` | `UInt64` | | |
| `transaction_hash` | `Binary` | | |
| `log_index` | `UInt64` | | |
| `address` | `Binary` | | |
| `topic0` ... `topic3` | `Binary` | ✅ | |
| `data` | `Binary` | | |
| `event` | `Struct` | | Decoded event, only present when `signature` is specified |

## Prepare Steps

The `prepare` field specifies a list of steps (<Schema />) applied to the raw data before reading:

| Kind | Description |
| - | - |
| `Decompress` | Extracts data compressed with `Gzip` or `Zip`. For multi-file archives `subPath` (can contain glob patterns) specifies which file to extract |
| `Pipe` | Executes an external command that receives data via `stdin` and outputs the result to `stdout`, e.g. to transcode formats that are not natively supported |

## Read Formats

The `read` field (<Schema />) specifies how data is read into the structured form:

| Kind | Description |
| - | - |
| `Csv` | Comma-separated values, with options for separators, quoting, escaping, headers, null values, and date / timestamp formats |
| `Json` | JSON document containing an array of objects, optionally located at `subPath` within the document |
| `NdJson` | [Newline-delimited JSON](https://github.com/ndjson/ndjson-spec) objects with the same schema |
| `GeoJson` | GeoJSON document with a `FeatureCollection` object in the root, creating a record per each `Feature` with properties extracted into individual columns and the geometry left in its own column |
| `NdGeoJson` | Similar to `GeoJson`, but with every `Feature` object on its own line |
| `EsriShapefile` | ESRI Shapefile, optionally located at `subPath` within an archive |
| `Parquet` | Apache Parquet |

The read step can specify the `schema` of the data in the [Data Schema](/odf/spec/datasets/data-schema) format, used to coerce values into appropriate data types. Formats like JSON can't be split and read efficiently, so newline-delimited variants should be preferred where possible.

## Preprocessing

The optional `preprocess` field specifies an engine-specific query (<Schema />) that shapes the data after reading, e.g. to parse values, convert types, or rename columns to match the dataset schema. The query refers to the read data as `input`:

```yaml theme={null}
preprocess:
  kind: Sql
  engine: datafusion
  query: |
    select
      date,
      city,
      cast(replace(population, ',', '') as bigint) as population
    from input
```

Preprocessing queries are executed by <Term /> using the [Raw Query](/odf/spec/transformations/engine-contract#raw-query) operation.

## Merge Strategies

By design, ODF stores data only in the append-only event log format to preserve the entire history. Unfortunately, a lot of data in the world is not stored or exposed this way - some publishers expose data as periodic database dumps, while others provide a log of changes between the current and the previous export.

<Term /> (<Schema />) bridge the gap between the wide variety of data in the external world and the strict event-based world of ODF, by defining how the newly-ingested data is combined with the existing history.

All strategies except `Append` rely on a user-specified `primaryKey` - the names of columns that uniquely identify a record throughout its lifetime.

### Append

Under this strategy the new data is appended to the dataset in its entirety, without any modifications or deduplication.

This strategy is rarely used for polling sources, but can be useful for push sources (e.g. IoT devices).

### Ledger

This strategy should be used for data sources containing append-only ledgers of events. New data exports can have new rows added, but once data made it into one export it should never change or disappear. The primary key is used to identify which events were already seen, so that the source data can contain partially overlapping sets of records and only those records that were not previously seen are appended.

For example, a source with historical populations of cities:

```csv theme={null}
Year,Country,City,Population
2019,CA,Vancouver,2581000
2019,US,Seattle,3433000
```

Ingested with primary key `(year, country, city)` results in:

| op | system\_time | event\_time | country | city | population |
| - | - | - | - | - | - |
| +A | s1 | 2019 | CA | Vancouver | 2581000 |
| +A | s1 | 2019 | US | Seattle | 3433000 |

When a new row for 2020 appears in the source, only that row is appended, while rows for 2019 are recognized as already seen:

| op | system\_time | event\_time | country | city | population |
| - | - | - | - | - | - |
| +A | s1 | 2019 | CA | Vancouver | 2581000 |
| +A | s1 | 2019 | US | Seattle | 3433000 |
| **+A** | **s2** | **2020** | **CA** | **Vancouver** | **2606000** |

Having a time column doesn't always mean that the data is a ledger - to be a true ledger all previous records must be immutable.

### Snapshot

This strategy can be used for data exports that are taken periodically and contain only the latest state snapshot of the observed entity or system. Over time such exports can have new rows added, and old rows either removed or modified.

Publishers that use such formats routinely lose information - when a record is updated or one export replaces another, the history of previous values is lost. This strategy transforms snapshot data into an append-only event stream by performing [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) - diffing the current state of data against the reconstructed previous state. The primary key is used to correlate rows between the two snapshots, and the <Term /> column of the output will contain:

* Append (`+A`) - when a row with a certain key appears for the first time
* Retraction (`-R`) - when a row with a certain key disappears from the snapshot
* Correction (`-C`, `+C`) - when the data associated with a certain key has changed, with `-C` carrying the old values and `+C` carrying the new values (see <Term />)

To identify whether a row has changed, this strategy compares all columns one by one. If the data contains a column that is guaranteed to change whenever any other column changes (e.g. a modification timestamp, a version, or a data hash), it can be specified in `compareColumns` to speed up the detection of modified rows.

For example, a source with current populations of cities, ingested with primary key `(country, city)` and event time [taken from the source metadata](#event-time-sources):

```csv theme={null}
Country,City,Population
CA,Vancouver,2581000
US,Seattle,3433000
```

| op | system\_time | event\_time | country | city | population |
| - | - | - | - | - | - |
| +A | s1 | e1 | CA | Vancouver | 2581000 |
| +A | s1 | e1 | US | Seattle | 3433000 |

After the population of Vancouver changes in the source, ingestion produces a correction:

| op | system\_time | event\_time | country | city | population |
| - | - | - | - | - | - |
| +A | s1 | e1 | CA | Vancouver | 2581000 |
| +A | s1 | e1 | US | Seattle | 3433000 |
| **-C** | **s2** | **e1** | **CA** | **Vancouver** | **2581000** |
| **+C** | **s2** | **e2** | **CA** | **Vancouver** | **2606000** |

### Changelog Stream

This strategy is used for sources that already provide data in the native ODF changelog format that describes appends, retractions, and corrections (see [Data Model](/odf/spec/datasets/data-model#representation-of-retractions-and-corrections)). No processing except for format validation is performed.

### Upsert Stream

This strategy is used for sources containing ledgers of insert-or-update and delete events. Unlike in the changelog stream, insert-or-update events only carry the new values, so this strategy uses the primary key to look up the previous values and re-classify events into appends or correction pairs.

## Committing Data

The result of every ingestion is committed into the <Term /> of the dataset as an <Schema /> event, which carries:

* `prevOffset` and `newData` - the last offset of the previous data slice and the new data slice, if any, allowing the next ingestion to be prepared without deep scanning of the chain
* `prevCheckpoint` and `newCheckpoint` - checkpoint used to restore and persist the ingestion state, if any
* `newWatermark` - the new <Term /> of the dataset, if any
* `newSourceState` - the [state of the source](#source-state), if any

An ingestion may produce no new data but still commit a block, e.g. to advance the watermark or to update the source state.

### Source State

To avoid expensive recomputations, the state of the source the data was ingested from (e.g. an `ETag` of an HTTP resource, the name of the last file matched by a glob pattern, or the height of the last indexed block of a blockchain) is stored in the `newSourceState` field (<Schema />) of the `AddData` event:

```yaml theme={null}
newSourceState:
  sourceName: odf/polling
  kind: odf/etag
  value: W/"95c5fde3918cba7c33eaac7ff9d02b22"
```

Where:

* `sourceName` - identifies the source the state belongs to
* `kind` - identifies the type of the state, similarly to MIME types. Standard kinds are `odf/etag` (an identifier with semantics of the `ETag` HTTP header) and `odf/last-modified` (an RFC 3339 timestamp with semantics of the `Last-Modified` HTTP header)
* `value` - an opaque value of the state

Both `sourceName` and `kind` are plain strings, allowing implementations to define their own extensions. It is safe to ignore source state that an implementation does not understand.

If the state didn't change but is still relevant for subsequent runs it should be carried over into the new event, as only the last state per source is considered when resuming.

## Validity of Root Data

Unlike data of <Term />, data of <Term /> cannot be reproduced - it captures observations of the outside world. Its validity therefore depends entirely on its publisher, who is in full control of the data they present, so measuring and processing errors or even malicious data can make their way into a root dataset.

What can be validated mechanically is that root data was not altered after it was published (see [Metadata Chain: Validation](/odf/spec/datasets/metadata-chain#validation) and [Data & Checkpoint Hashing: Validation and Verification](/odf/spec/datasets/data-and-checkpoint-hashing#validation-and-verification)), and who published it. Trust in the data itself is established through:

* **Authority** - the reputation of the publisher, e.g. organizations that directly operate the systems from which the data is collected
* **Cross-validation** - comparing data against other similar sources to detect anomalies

See also:

* <Term />

## Legacy Source Events

> **Deprecated:** Sources used to be defined in the <Term /> of root datasets using the following events. They are superseded by `Source` resources, as source definitions don't contribute to provenance or validity of a dataset, but may reveal sensitive information about the infrastructure of a publisher (see [RFC-018](/odf/rfcs/018-iac-resource-framework)).

| Event | Description |
| - | - |
| [`SetPollingSource`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/SetPollingSource.json) | Defines the polling source of a dataset, with `fetch`, `prepare`, `read`, `preprocess`, and `merge` steps |
| [`DisablePollingSource`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/DisablePollingSource.json) | Disables the polling source |
| [`AddPushSource`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/AddPushSource.json) | Defines a named push source of a dataset, with `read`, `preprocess`, and `merge` steps |
| [`DisablePushSource`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/DisablePushSource.json) | Disables a named push source |

Under this model:

* A dataset may have multiple named push sources active simultaneously (e.g. for devices from different manufacturers producing slightly different data), but push and polling sources are mutually exclusive
* All sources must result in the same final data schema, as captured in the `SetDataSchema` event
* The source state of the polling source uses the `odf/polling` source name

## Schemas

| Schema | Description |
| - | - |
| <Schema /> | Source resource |
| <Schema /> | Specification of a source |
| <Schema /> | Where data comes from |
| <Schema /> | Buffering of pushed records |
| <Schema /> | How event time is determined |
| <Schema /> | Caching of external data |
| <Schema /> | Ordering of input files |
| <Schema /> | Preparation of raw data |
| <Schema /> | Reading data into structured form |
| <Schema /> | Combining new data with history |
| <Schema /> | State of a source for incremental ingestion |
| <Schema /> | Metadata event committing the ingested data |


# Logical Structure
Source: https://docs.kamu.dev/odf/spec/datasets/logical-structure



**Domain**: Datasets\
**Depends on**: -\
**RFCs**: -

## Abstract

This document introduces the general structure of an ODF <Term /> - the main unit of data exchange in the system. It explains the components a dataset consists of, the two kinds of datasets, and how datasets evolve over time, serving as an entry point into the specs of the Datasets domain.

## Dataset Structure

ODF operates on data expressed as immutable events (see [Nature of Data](/odf/spec/overview#nature-of-data)). A <Term /> is a combination of:

* **Identity** - a globally unique identifier tied to a cryptographic key pair, and human-friendly aliases (see [Identity](/odf/spec/datasets/identity))
* **Data** - a potentially infinite stream of events, stored as a sequence of data slices (see [Data Model](/odf/spec/datasets/data-model))
* **Metadata Chain** - an append-only log of blocks that captures the entire history of the dataset (see [Metadata Chain](/odf/spec/datasets/metadata-chain))
* **Checkpoints** - engine-specific state used to resume stream processing (see [Processing Model](/odf/spec/transformations/processing-model#checkpoints))

<Diagram alt="Diagram: Dataset/Data" />

All components of a dataset except the block references are immutable and content-addressed, so having only a hash of the last metadata block one can discover and verify all of them (see [Storage Layout](/odf/spec/datasets/storage-layout)).

### Data and Data Slices

<Term /> is a set of <Term />. Since events are immutable, data can only grow over time. Conceptually it's best to think of data as a full log of a potentially infinite event stream.

Data never appears in the system alone, as we would not be able to tell where it came from and whether it can be trusted - it always appears as part of a dataset.

Data arrives into the system in arbitrarily large sets of events called <Term />. Every slice is a continuous part of data with the same <Term />, defined by its interval of <Term />, and is stored as a separate data part file.

<Diagram alt="Diagram: Data Slices and Metadata" />

All records carry the system columns of the common data schema, including the offset, the <Term />, the <Term />, and the <Term /> (see [Data Model](/odf/spec/datasets/data-model)). The logical structure of data is described using the ODF schema format (see [Data Schema](/odf/spec/datasets/data-schema)).

### Metadata

Metadata plays an essential role in ODF (see [Essential Role of Metadata](/odf/spec/overview#essential-role-of-metadata)) - it contains every aspect of where the data came from, how it was transformed, and everything that ever influenced how data looks like throughout its entire lifetime.

Every data slice and checkpoint added to the dataset is recorded in a block of the [Metadata Chain](/odf/spec/datasets/metadata-chain) together with its [hash](/odf/spec/datasets/data-and-checkpoint-hashing), linking data and metadata cryptographically.

### Root and Derivative Datasets

Depending on where the data comes from, datasets can be of two kinds:

* <Term /> - the points of entry of external data into the system, populated via [Ingestion](/odf/spec/datasets/ingestion)
* <Term /> - created by transforming and combining one or multiple existing datasets using deterministic queries (see [Processing Model](/odf/spec/transformations/processing-model))

Datasets connected through transformations form a graph, where data of every derivative dataset can be traced back to the root datasets it was derived from.

<Diagram alt="Diagram: Dataset Graph" />

## Evolution of Data

As the nature of businesses change, as new requirements arrive, as defects are detected and fixed - it's not a matter of *if* but *when* the time comes to make changes to the data. Calling data a potentially infinite stream would not make much sense without providing a way to improve and evolve it over time. Having to create a new dataset every time you need to change the schema or update the transformation would jeopardize the timeliness requirement, as the entire chain of transformations would have to be rebuilt from scratch after every such change.

Our design goals are, therefore:

* To seamlessly support all backward-compatible data schema changes
* To support the evolution of transformations over time
* To provide a way to correct past mistakes in data

While for simplicity we previously considered derivative datasets to be only defined by one transformation query, in fact it can comprise multiple queries that were active during different periods of time. The metadata keeps track of which parts of inputs were processed by which query and which parts of output were produced as the result, ensuring reproducibility. This is captured by the [Metadata Chain](/odf/spec/datasets/metadata-chain) - **almost anything that defines a dataset can change over time**.

## Datasets Specs

| Spec | Description |
| - | - |
| [Identity](/odf/spec/datasets/identity) | `did:odf` identifiers, aliases, and references |
| [Data Model](/odf/spec/datasets/data-model) | Event model, system columns, retractions and corrections, schema evolution |
| [Data Schema](/odf/spec/datasets/data-schema) | Logical schema format of data |
| [Metadata Chain](/odf/spec/datasets/metadata-chain) | Blocks, events, consistency rules, and block hashing |
| [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing) | Physical and logical hashing of data and checkpoints |
| [Storage Layout](/odf/spec/datasets/storage-layout) | Layout of dataset components in file systems, object stores, and content-addressable storage |
| [Ingestion](/odf/spec/datasets/ingestion) | Sources, ingestion phases, and merge strategies of root datasets |


# Metadata Chain
Source: https://docs.kamu.dev/odf/spec/datasets/metadata-chain



**Domain**: Datasets\
**Depends on**: [Data Model](/odf/spec/datasets/data-model), [Data Schema](/odf/spec/datasets/data-schema), [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing)\
**RFCs**: [RFC-004](/odf/rfcs/004-metadata-extensibility), [RFC-005](/odf/rfcs/005-dataset-annotations), [RFC-010](/odf/rfcs/010-data-schema-in-metadata), [RFC-012](/odf/rfcs/012-recommend-base16-encoding), [RFC-014](/odf/rfcs/014-minimize-offset-scanning)

## Abstract

This spec defines the structure of the <Term /> - the append-only log of blocks that captures the entire history of a <Term />. It defines metadata blocks and the events they carry, the rules that keep the chain consistent, the formats used to represent metadata, and how blocks and other objects are hashed and linked together.

## Metadata Blocks

The metadata chain consists of individual **metadata blocks** (<Schema />) that refer to the previous block in the chain, forming a singly-linked list. Every block contains:

* `systemTime` - <Term /> when the block was written
* `prevBlockHash` - [hash](#metadata-block-hashing) of the preceding block (absent in the first block)
* `sequenceNumber` - sequence number of the block, starting from zero at the first block
* `event` - one [metadata event](#metadata-events) that describes the transaction that occurred on the dataset

<Diagram alt="Diagram: Metadata Chain" />

Blocks are immutable - the chain only changes by appending new blocks. Since blocks, data part files, and checkpoints are named after and referenced by the hash of their content, a dataset forms a [content-addressable](https://en.wikipedia.org/wiki/Content-addressable_storage) graph, where having a reference to the last block one can traverse the entire chain to discover all components of the dataset (see [Storage Layout](/odf/spec/datasets/storage-layout)).

<Diagram alt="Diagram: Dataset as a Content-Addressable Graph" />

A block contains exactly one event. Bringing a dataset into a desired state may therefore require several blocks. Consistency is achieved similarly to `git` - a group of blocks that constitutes a consistent state is committed atomically by updating the [block reference](#block-references).

## Metadata Events

Every event (<Schema />) corresponds to a certain transaction on the dataset and contains only the fields relevant to it.

### Core Events

| Event | Description |
| - | - |
| <Schema /> | Establishes the identity (`datasetId`) and the kind (`Root` or `Derivative`) of the dataset. Always the first event in the chain |
| <Schema /> | Specifies the complete <Term /> of <Term /> added to the dataset following this event (see [Data Schema](/odf/spec/datasets/data-schema)) |
| <Schema /> | Specifies the names of the system columns (see [Data Model](/odf/spec/datasets/data-model#common-data-schema)) |
| <Schema /> | Indicates that data has been ingested into a <Term /> (see [Ingestion](/odf/spec/datasets/ingestion#committing-data)) |
| <Schema /> | Defines the inputs and the <Term /> that produce data of a <Term /> |
| <Schema /> | Indicates that a derivative transformation has been performed |

`AddData` and `ExecuteTransform` events describe the outcome of a transaction:

* `prevOffset` and `newData` (<Schema />) - the last offset of the previous data slice, and the new data slice with its [physical and logical hashes](/odf/spec/datasets/data-and-checkpoint-hashing), offset interval, and size
* `prevCheckpoint` and `newCheckpoint` (<Schema />) - the <Term /> used to restore the processing state and the new checkpoint with its physical hash and size
* `newWatermark` - the new <Term /> of the dataset

In addition, `AddData` carries the `newSourceState` (see [Source State](/odf/spec/datasets/ingestion#source-state)), while `ExecuteTransform` carries `queryInputs` (<Schema />) - the intervals of blocks and records of every input dataset that were used in the transaction.

A transaction may produce no new data and still be committed, e.g. to advance the watermark or to update the source state.

`SetDataSchema` events are normally not specified by users, but are generated by implementations when the schema of the data is first established (e.g. during the first ingestion or transformation) and when it evolves. Multiple `SetDataSchema` events can appear in the chain according to the [Schema Evolution](/odf/spec/datasets/data-model#schema-evolution) rules.

### Annotation Events

Annotation events provide human-oriented information about the dataset:

| Event | Description |
| - | - |
| <Schema /> | Brief description and keywords of the dataset |
| <Schema /> | License that applies to the dataset, with an optional [SPDX](https://spdx.org/licenses/) identifier |
| <Schema /> | Files associated with the dataset (e.g. a README), currently only in the form of items embedded in the metadata |

Example:

```yaml theme={null}
- kind: SetInfo
  description: Confirmed positive cases of COVID-19 in Ontario
  keywords:
    - Healthcare
    - Epidemiology
- kind: SetLicense
  shortName: OGL-Ontario
  name: Open Government Licence - Ontario
  websiteUrl: https://www.ontario.ca/page/open-government-licence-ontario
- kind: SetAttachments
  attachments:
    kind: Embedded
    items:
      - path: README.md
        content: |
          # Confirmed positive cases of COVID-19 in Ontario
          Detailed description follows...
```

The metadata chain is not meant to replace version control systems - larger sets of files (e.g. notebooks) are expected to be associated with datasets by referencing external systems.

### Legacy Events

> **Deprecated:** The `SetPollingSource`, `DisablePollingSource`, `AddPushSource`, and `DisablePushSource` events used to define ingestion sources in the metadata chain. They are superseded by `Source` resources, but can still appear in existing chains. See [Ingestion: Legacy Source Events](/odf/spec/datasets/ingestion#legacy-source-events).

## Consistency Rules

Implementations must ensure that:

* The first block of the chain contains the `Seed` event, and `Seed` appears only once
* Every block except the first refers to the hash of the preceding block, and sequence numbers increase by one with every block
* `AddData` events only appear in root datasets, while `SetTransform` and `ExecuteTransform` only appear in derivative datasets
* All data slices conform to the schema defined by the last preceding `SetDataSchema` event
* Watermarks are monotonically non-decreasing - once a watermark is set, all subsequent blocks either carry the same watermark or specify a greater one
* If a transaction did not update the checkpoint but it is still relevant for subsequent transactions, the hash of the previous checkpoint is carried over into `newCheckpoint`
* Data slices, checkpoints, and input datasets referenced by blocks have valid hashes

### Offsets and Intervals

`AddData` and `ExecuteTransform` events always carry enough information to prepare the next transaction from the last such event, without deep scanning of the chain:

* `prevOffset` is equal to the end of the offset interval of the last non-empty `newData`, or is absent if no data was written yet
* The `newData.offsetInterval` of the next non-empty slice starts at `prevOffset + 1` (or `0`)

For example, a chain of `AddData` events where the last one only advances the watermark:

```yaml theme={null}
- prevOffset: null
  newData:
    offsetInterval: {start: 0, end: 9}
  newWatermark: "2023-12-31T00:00:00Z"
- prevOffset: 9
  newData:
    offsetInterval: {start: 10, end: 19}
  newWatermark: "2023-12-31T01:00:00Z"
- prevOffset: 19
  newData: null
  newWatermark: "2023-12-31T03:00:00Z"
```

Similarly, `queryInputs` of `ExecuteTransform` describe the inputs using half-open intervals `(prevBlockHash, newBlockHash]` and `(prevOffset, newOffset]`, where the starting point is always carried over from the previous transaction, even when the interval itself is empty.

## Validation

Since metadata blocks are content-addressed, a chain obtained from any untrusted source can be validated by knowing only the hash of its last block, obtained from a trusted source. Validation involves:

* Recomputing the [hash](#metadata-block-hashing) of every block and comparing it to the hash by which the block was referenced (via a [block reference](#block-references) or the `prevBlockHash` of the following block)
* Checking that all [consistency rules](#consistency-rules) are satisfied

A valid chain guarantees the integrity of all data slices and checkpoints it references, as long as their hashes are also validated (see [Data & Checkpoint Hashing: Validation and Verification](/odf/spec/datasets/data-and-checkpoint-hashing#validation-and-verification)).

## Block References

**Block references** assign symbolic names to block hashes, effectively acting as named pointers. All datasets have a `head` reference that points to the current last block of the chain.

Using multiple references the chain can be organized into a directed acyclic graph that forms branches, e.g. to stage a subset of events for review or an automated QA process before they are accepted into the main chain. The semantics of references other than `head` are not yet specified.

## Extensibility

The metadata chain is designed to be extended with new event types that carry other kinds of information, such as:

* Meaning and structure of knowledge that data represents (glossary, semantics, ontology)
* Relevant policies, terms, rules, compliance, and regulations (governance)
* License, privacy, and security concerns (stewardship)
* Information that aids discovery
* Collaboration information

Some events (e.g. `AddData`) also carry the `extra` field (<Schema />) that allows attaching custom information to the event, with keys in the `<domain>/<attribute-name>` format (see [Data Schema: Extra Attributes](/odf/spec/datasets/data-schema#extra-attributes)).

## Metadata Format

The requirements we put towards the metadata format are:

* Compactness
* Fast read performance
* Complex type support (particularly nested structures and unions)
* Allows for forward and backward compatibility
* Allows for a controlled way of making breaking changes
* Possibility to read and write in human-friendly easily editable format
* Schema-based validation for user-specified data

We use the following combination of formats to satisfy these requirements:

* [YAML](https://yaml.org/) is used for human-friendly input and output
  * This format is concise and very readable and widely supported
  * Only a [JSON-compatible](https://json.org/) subset of `YAML` is used
* [JSON Schema](https://json-schema.org/) is used for validating user-specified `YAML` files
  * Widely supported and expressive grammar for validating `JSON` and `YAML`
  * Generates readable and easy to understand errors
* [FlatBuffers](https://google.github.io/flatbuffers/) format is used for binary representation
  * Highly efficient and cross-platform serialization format

The JSON Schemas of all metadata objects are provided in the [`schemas`](https://github.com/open-data-fabric/open-data-fabric/tree/master/schemas) directory, and the FlatBuffers schema is [generated](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/metadata/opendatafabric.fbs) from them.

## Hash Representation

To be able to evolve hashing algorithms over time we encode the identity of the algorithm used along with the hash sum itself using the [multiformats](https://github.com/multiformats/multiformats) specification, specifically:

* [multihash](https://github.com/multiformats/multihash) - describes how to encode algorithm ID alongside the hash value.
* [multibase](https://github.com/multiformats/multibase) - describes encoding used to represent a binary value as text (e.g. in YAML format or as a file name) without ambiguity of which encoding was used.

It is expected that the specific `multibase` encoding is largely a presentation-layer concern that can vary between implementations. For protocol compatibility all implementations must support all encodings in the `final` state of approval in the [`multibase` specification](https://github.com/multiformats/multibase), transcoding the representations when needed.

For consistency, we recommend implementations to prefer the `base16` encoding. Although it produces longer hashes than other encodings, it:

* Is case-insensitive and can appear in subdomains
* Does not contain symbols that can be easily confused
* Does not have a risk of forming an accidental obscenity
* Does not require padding
* Is easier to use for partitioning by prefix

For representing the identity of hashing algorithms the official [multicodec](https://github.com/multiformats/multicodec/) table is used.

## Metadata Block Hashing

Blocks of the metadata chain are referred to and linked together using their cryptographic hashes. The process of serializing and computing a stable hash is as follows:

1. The <Schema /> is serialized into [FlatBuffers](https://google.github.io/flatbuffers/) format following a two-step process to ensure that all variable-size buffers are laid out in memory in a consistent order:
   1. First, we iterate over all fields of the block in the same order they appear in the schemas serializing into buffers all vector-like and variable-size fields and recursing into nested data structures (tables) in the depth-first order.
   2. Second, we iterate over all fields again this time serializing all leftover fixed-size fields
2. Block content is then nested into a [`Manifest`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/Manifest.json) object using same serialization rules as above
3. The resulting `flatbuffer` data is fed into [SHA3-256](https://en.wikipedia.org/wiki/SHA-3) digest algorithm.

For use in the `Manifest`'s `kind` field the `multicodec` table is extended with the following codes in the "private use area":

| Codec | Code |
| - | - |
| `odf-metadata-block` | `0x400000` |

The block hashes are represented using [multihash](https://github.com/multiformats/multihash) and [multibase](https://github.com/multiformats/multibase) described [above](#hash-representation).

## Populating the Chain

The initial events of a dataset are specified via the `metadata` field of the `Dataset` <Term /> (<Schema />). When the resource is applied, the node creates the `Seed` block and appends a block for every specified event:

```yaml theme={null}
$schema: https://opendatafabric.org/schemas/datasets/v1alpha1/Dataset
headers:
  name: ca.bankofcanada
spec:
  kind: Root
  metadata:
    - kind: SetInfo
      description: Daily exchange rates published by the Bank of Canada
    - kind: SetLicense
      shortName: Apache-2.0
      name: Apache 2.0
      spdxId: Apache-2.0
      websiteUrl: https://www.apache.org/licenses/LICENSE-2.0
```

Subsequent blocks are appended by ingestion and transformation processes.

As the dataset evolves, the `metadata` field of the resource maintains a **snapshot** of the key events that describe the dataset in its present (latest) state. The snapshot excludes:

* All intermediate transitions, e.g. when the schema changes multiple times only the latest `SetDataSchema` event is retained
* All events related to ingestions and transformations, like `AddData` and `ExecuteTransform`

The metadata chain, on the other hand, preserves the full history of all events.

## Schemas

| Schema | Description |
| - | - |
| <Schema /> | Block of the metadata chain |
| <Schema /> | Union of all metadata events |
| <Schema /> | Data part file added in a transaction |
| <Schema /> | Checkpoint file written in a transaction |
| <Schema /> | Closed interval of record offsets |
| <Schema /> | Input of a derivative transformation |
| <Schema /> | Input dataset of a `SetTransform` |
| <Schema /> | Engine-specific transformation query |
| <Schema /> | Names of the system columns |
| <Schema /> | Files associated with a dataset |
| [`Manifest`](https://github.com/open-data-fabric/open-data-fabric/blob/master/schemas/legacy/v0/Manifest.json) | Legacy wrapper used in block hashing |


# Storage Layout
Source: https://docs.kamu.dev/odf/spec/datasets/storage-layout



**Domain**: Datasets\
**Depends on**: [Data Model](/odf/spec/datasets/data-model), [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing)\
**RFCs**: [RFC-003](/odf/rfcs/003-content-addressability), [RFC-006](/odf/rfcs/006-checkpoints-as-files), [RFC-017](/odf/rfcs/017-large-files-linking)

## Abstract

This spec defines how components of a <Term /> - metadata blocks, references, data part files, checkpoints, and linked objects - are laid out in file systems, conventional object stores, and content-addressable and permanent storage networks.

## Dataset Components

A dataset is stored as a set of objects grouped into the following sections:

| Section | Contents | Object key |
| - | - | - |
| `blocks` | Blocks of the <Term /> | Hash of the block |
| `refs` | Named references to blocks | Name of the reference |
| `data` | <Term /> part files and [linked objects](#linked-objects) | Physical hash of the file |
| `checkpoints` | <Term /> files | Physical hash of the file |

Hashes used as object keys are represented as [multihash](https://github.com/multiformats/multihash) values encoded with [multibase](https://github.com/multiformats/multibase) (see [Hash Representation](/odf/spec/datasets/metadata-chain#hash-representation)), e.g. `f16207e15328e0fcfe47339273e4a00f59f210891d45026d84b71c6b40eecf237ec24`. For data part files and checkpoints this is the physical hash (see [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing)).

Reference objects contain the hash of the block they point to. Every dataset has a `head` reference that points to the last block in the Metadata Chain. Other references (e.g. for branches and tags) may appear in the `refs` section, but their semantics are not yet specified.

All objects except references are **immutable** and **content-addressed** - they are uniquely identified by the hash of their content. Having only the hash of the `head` block one can traverse the entire Metadata Chain following the `prevBlockHash` links and discover all data part files and checkpoints referenced by the blocks. Hash of the `head` block thus acts as a Merkle root of the entire dataset.

### Checkpoints

A checkpoint must always be a single file. Engines that need to persist state as multiple files or directories must combine them into one file (e.g. using `tar`).

This ensures that checkpoints can be hashed the same way as data part files, and that a dataset can be transferred via protocols that don't support listing directories.

### Linked Objects

> **Experimental:** See [RFC-017](/odf/rfcs/017-large-files-linking).

Large binary objects (e.g. images, videos, documents) linked to a dataset by hash from data columns are stored in the `data` section alongside data part files, keyed by their multihash. Since links are part of the data, the hash of the `head` block also acts as a Merkle root of all linked objects, preserving tamper-proof properties of the dataset.

## Layout in File Systems and Object Stores

When storing a dataset in a file system or a conventional object store (e.g. S3), sections are represented as directories (or key prefixes) and objects as files within them:

```
{dataset-root}/
├── blocks/
│   └── {block-hash}
├── refs/
│   └── head
├── data/
│   ├── {data-file-physical-hash}
│   └── {linked-object-hash}
└── checkpoints/
    └── {checkpoint-physical-hash}
```

This layout must be used when sharing datasets via file or object-based <Term /> (e.g. local FS, S3, HTTP servers).

Accessing a dataset in this layout only requires the ability to **read an object by key** - it purposefully does not rely on any advanced storage features like listing directories. This allows datasets to be transferred using the [Simple Transfer Protocol](/odf/spec/exchange/transfer-protocols#simple-transfer-protocol) on top of any storage that supports Unix path-like keys.

When a <Term /> is imported locally, the exact layout is left entirely up to the <Term /> implementation, as we expect all interactions with the <Term /> to go through it. Implementations may also store additional implementation-specific information (e.g. configuration, caches, summaries) alongside the dataset, but such information is not a part of the dataset and must not be relied upon by other implementations.

See Also:

* [Dataset Sharing](/odf/spec/exchange/transfer-protocols#dataset-sharing)

## Layout in Content-Addressable Stores

Since all dataset components except references are immutable and content-addressed, datasets map naturally onto content-addressable and permanent storage networks like [IPFS](https://ipfs.tech/) and [Arweave](https://www.arweave.org/).

A dataset is stored in such networks using the same [layout](#layout-in-file-systems-and-object-stores) as above, represented via the network's conventional file and directory format (e.g. UnixFS in IPFS, ArFS in Arweave). This allows datasets to be stored and retrieved using generic tooling of the network that has no knowledge of ODF.

Storage networks identify stored data by their own **locators** (e.g. IPFS CIDs, Arweave transaction IDs). Locators generally differ from ODF hashes of the same objects, as they depend on network-specific settings like chunking of large files, DAG structure, or transaction boundaries. Objects are therefore located by their paths within the dataset directory rather than by their ODF hashes directly.

Networks that provide HTTP gateways allow reading datasets using the [Simple Transfer Protocol](/odf/spec/exchange/transfer-protocols#simple-transfer-protocol), as it only requires the ability to read objects by their paths.

Because components of a dataset are identical regardless of where they are stored, the same dataset can be stored in multiple places simultaneously, e.g. in a conventional object store for low-latency access and in a permanent storage network for archival and censorship-resistant access. Consistency between such replicas can be verified by comparing the hashes of their `head` blocks.

### Incremental Updates

When a dataset is updated, only the new blocks, data part files, and checkpoints (the **delta**) are added to the network - objects stored previously are reused without duplication.

Depending on the network:

* In networks that support directory structures with structural sharing (e.g. IPFS), every update produces a new directory locator, while previous locators remain valid and point to older states of the dataset
* In append-only networks where every upload is a separate transaction (e.g. Arweave), every upload containing a delta should reference the previous upload of the same dataset, so that having only the latest locator one can discover all uploads that constitute the dataset

Uploads may be batched - a single upload can contain deltas of multiple updates of a dataset, which may reduce the cost of storage in networks where every transaction incurs fees.

### Naming and Resolution

Since locators change with every update, an additional mechanism is needed to resolve a stable dataset identity (see [Identity](/odf/spec/datasets/identity)) or name into the locator of its latest state. Such mechanisms are network-specific, e.g. mutable names in IPFS ([IPNS](https://docs.ipfs.tech/concepts/ipns/), [DNSLink](https://docs.ipfs.tech/concepts/dnslink/)), or transaction tags indexed by gateways in Arweave.

Resolution should not depend on infrastructure operated by a single party. Wherever possible, information needed for resolution (e.g. dataset ID and the hash of the `head` block) should be stored alongside the data in the network itself, so that it can be independently verified, and the mapping can be reconstructed by indexing the network if needed.

### Immutability Considerations

Data stored in permanent storage networks cannot be reliably deleted:

* Hard compactions and other forms of erasure of history will not remove previously stored objects from the network
* Data that was published unencrypted cannot be taken down

Sensitive data should therefore only be stored in such networks in encrypted form. In this case, erasure can be achieved by permanently revoking access to decryption keys.


# Transfer Protocols
Source: https://docs.kamu.dev/odf/spec/exchange/transfer-protocols



**Domain**: Exchange\
**Depends on**: [Storage Layout](/odf/spec/datasets/storage-layout)\
**RFCs**: [RFC-007](/odf/rfcs/007-simple-transfer-protocol), [RFC-008](/odf/rfcs/008-smart-transfer-protocol)

## Abstract

This spec defines how <Term /> are shared and synchronized between <Term />. It defines the minimal read-only Simple Transfer Protocol that works on top of any storage that can read objects by key, the efficient read-write Smart Transfer Protocol that allows transferring data directly to and from the storage of a repository, and the rules that keep shared datasets consistent for concurrent readers.

## Design Goals

The guiding factors in designing how data is shared were:

* Minimal storage requirements - while a certain degree of replication is necessary to keep data highly available, we don't want to duplicate data more than it's necessary.
* Decentralization - there should be no central authority that decides what the "right data" is.
* Verifiable trust - even when data is hosted by a single party any tampering or alterations must still be impossible.
* Minimize the movement of data - when a trusted party keeps data in an environment that also provides compute resources, we would like to be able to process and query data without the need to always download it locally.

The way we designed <Term /> plays a crucial role in data sharing.

Metadata is usually several orders of magnitude smaller than the data it describes, so it's ideally suited to be widely shared. The exact way it's shared is out of scope of this document, but in most simple cases it can be published openly by the provider of data and cryptographically signed to ensure authenticity.

When the metadata of a certain dataset is reliably known, a peer can then download the data from any untrusted source and use the metadata to validate the authenticity of every data slice that composes it (see <Term />) - this means that we need to only ensure the validity of metadata, while the data can be stored almost anywhere with minimal replication factor, as long as one compromised party does not result in the complete loss of data.

Metadata also provides us with a way to establish the trustworthiness of any dataset by reviewing the transformations declared in it, re-applying those transformations in a trusted environment, and comparing the results to the original data. In a distributed system, having peers cross-validate each other's published data can guarantee trusted results and allow them to promptly identify and exclude malicious peers from the network.

## Dataset Sharing

Dataset sharing involves uploading the data to some <Term /> where it can be discovered and accessed by other peers. Due to immutable nature of data and metadata it is very easy to keep shared data up-to-date as only new blocks and part files need to be uploaded.

It is important to ensure the atomicity of the sharing, or at least create the perception of atomicity to someone who happens to be downloading the data concurrently with the upload.

For <Term /> that do not support atomic file/object operations the following sequence of operations can ensure that concurrent downloads will always see the <Term /> in a consistent state:

1. Upload data part files and checkpoints
2. Upload metadata blocks
3. Update references in the `refs` section (see [Storage Layout](/odf/spec/datasets/storage-layout#dataset-components))

## Validation of Transferred Data

Repositories, storage systems, and peers that data is transferred from are not trusted. Regardless of the transfer protocol used, the receiving party must validate all received objects before committing them:

* Metadata blocks are validated according to [Metadata Chain: Validation](/odf/spec/datasets/metadata-chain#validation), and must form a valid continuation of the chain the receiver already has
* Data part files and checkpoints are validated against the physical hashes stored in metadata (see [Data & Checkpoint Hashing: Validation and Verification](/odf/spec/datasets/data-and-checkpoint-hashing#validation-and-verification))
* Block references are updated only after all objects they make reachable were received and validated

This applies both to clients pulling data, and to servers accepting data pushed to them. The trust in the transferred dataset is therefore reduced to the trust in the hash of its last block.

## Simple Transfer Protocol

Simple Transfer Protocol is a bare-minimum read-only protocol used for synchronizing <Term /> between <Term />. It requires no ODF-specific logic on the server side and can be easily implemented, for example, by serving a dataset directory under an HTTP server. It's designed for maximal interoperability, not for efficiency.

The only operation a server needs to support is **reading an object by key**. The protocol purposefully does not rely on any advanced features like the ability to list a directory, so a dataset can be served from any storage that uses the [layout of file systems and object stores](/odf/spec/datasets/storage-layout#layout-in-file-systems-and-object-stores), including content-addressable storage networks that provide HTTP gateways.

To describe the protocol we will use HTTP `GET {object-key}` notation below, but note that this protocol can be implemented on top of any block or file-based protocol that supports Unix path-like object keys.

1. Process begins with `GET /refs/head` to get the hash of the last <Term />
2. The "metadata walking" process starts with `GET /blocks/{blockHash}` and continues following the `prevBlockHash` links
3. Data part files can be downloaded by using <Schema /> links with `GET /data/{physicalHash}`
4. Checkpoints are similarly downloaded using <Schema /> links with `GET /checkpoints/{physicalHash}`
5. The process continues until reaching the first block of the dataset or other termination condition (e.g. reaching the block that has already been synced previously)

`DataSlice` and `Checkpoint` objects also carry the `size` of the files, which can be used as a size hint when transferring files between repositories in a streaming fashion.

## Smart Transfer Protocol

Smart Transfer Protocol is a superset of the [Simple Transfer Protocol](#simple-transfer-protocol), which allows both reads and writes and solves performance issues to allow efficient synchronization between remote dataset repositories:

* Metadata blocks are transferred in batches as archives instead of one at a time
* Object files (data part files and checkpoints) are transferred directly to and from the storage system of the repository, with unrestricted degree of parallelism, instead of being proxied through the server

The protocol assumes establishing a duplex asynchronous communication channel between the parties. A separate session of message exchanges is open for a single pull or push operation, and it is closed after the operation is complete or fails with an error.

The protocol can be implemented using [WebSockets](https://websockets.spec.whatwg.org/) or any other type of channel that supports bi-directional non-blocking message exchanges.

### Establishing a Session

In addition to the HTTP endpoints of the Simple Transfer Protocol, servers implementing the Smart Transfer Protocol provide two more endpoints:

1. `GET /pull` - to start the smart pull flow on the dataset
2. `GET /push` - to start the smart push flow on the dataset

Both endpoints switch from HTTP to the WebSocket protocol using the standard [upgrade mechanism](https://datatracker.ietf.org/doc/html/rfc6455#section-4), where:

* `Sec-WebSocket-Protocol` header is set to `odf/smart-transfer-protocol/v1`
* `x-odf-smtp-version` header specifies the version of the Smart Transfer Protocol the client wants to use

All endpoints and messages are authorized using a bearer access token.

### Pull Flow

1. Client sends `DatasetPullRequest` with the dataset ID and the desired block range: `beginAfter` - the last block the client already has (absent for the initial pull), and `stopAt` - the last block the client wants to pull (absent for `head`)
2. Server responds with `DatasetPullResponse` containing the size estimation of the transfer (number of blocks and objects and their raw sizes)
3. Client sends `DatasetPullMetadataRequest`, and the server responds with `DatasetMetadataPullResponse` containing an archive of all metadata blocks in the requested range
4. Client analyzes the received blocks and identifies missing object files
5. Client sends one or more `DatasetPullObjectsTransferRequest` messages listing a subset or the full set of missing object files, and the server responds to each with `DatasetPullObjectsTransferResponse` containing the download strategy for every object (a time-limited download URL with read-only access to the storage system)
6. Client downloads the object files directly from the storage system
7. Client commits the new blocks locally and closes the session

<Diagram alt="Diagram: ODF Smart Transfer Protocol Pull" />

### Push Flow

1. Client sends `DatasetPushRequest` with the dataset ID, the `currentHead` that the client assumes to be the current head on the server, and the size estimation of the transfer
2. Server responds with `DatasetPushRequestAccepted`
3. Client sends `DatasetPushMetadata` containing an archive of the new metadata blocks, and the server responds with `DatasetPushMetadataAccepted` if it does not detect a divergence in the metadata
4. Client sends one or more `DatasetPushObjectsTransferRequest` messages listing a subset or the full set of new object files, and the server responds to each with `DatasetPushObjectsTransferResponse` containing the upload strategy for every object: a time-limited upload URL with write-only access to the storage system, or an instruction to skip the upload of an object that the server already has
5. Client uploads the object files directly to the storage system
6. Client sends `DatasetPushComplete`
7. Server verifies that the required objects exist and attempts to commit the new metadata blocks
8. In case of success, server responds with `DatasetPushCompleteConfirmed` and closes the session

<Diagram alt="Diagram: ODF Smart Transfer Protocol Push" />

### Object Transfer

Object files are transferred out-of-band, directly between the client and the storage system of the repository, using pre-signed URLs:

* The exchange of object files should be performed with delays and object sizes taken into account. As the expiration time of pre-signed URLs is limited, when large bulks of objects have to be exchanged the URLs may expire before the actual transfer starts - in this case they need to be re-requested.
* The parallelism of downloads and uploads is not limited by the protocol and is not to be constrained by the server - it's an implementation detail of the client.

### Error Handling

Errors that happen during any operation are reported by the server with the `DatasetError` message that carries an error code, a description, and optional extra fields.

In case of errors and timeouts the client needs to re-initiate the operation by opening another session. A new operation may skip the transfer of object files that were already processed in previous sessions, even if the changes have not been committed to the metadata.

Object files that were uploaded during failed operations remain orphaned in the storage until removed by a garbage collection mechanism, which is out of scope of this spec.

### Messages

| Message | Direction | Description |
| - | - | - |
| `DatasetPullRequest` | Client → Server | Initiates a pull of a block range |
| `DatasetPullResponse` | Server → Client | Accepts the pull and provides size estimation |
| `DatasetPullMetadataRequest` | Client → Server | Requests the metadata blocks |
| `DatasetMetadataPullResponse` | Server → Client | Archive of the requested metadata blocks |
| `DatasetPullObjectsTransferRequest` | Client → Server | Requests download strategies for a set of object files |
| `DatasetPullObjectsTransferResponse` | Server → Client | Download strategies for the requested object files |
| `DatasetPushRequest` | Client → Server | Initiates a push with the expected current head |
| `DatasetPushRequestAccepted` | Server → Client | Accepts the push |
| `DatasetPushMetadata` | Client → Server | Archive of the new metadata blocks |
| `DatasetPushMetadataAccepted` | Server → Client | Accepts the new metadata blocks |
| `DatasetPushObjectsTransferRequest` | Client → Server | Requests upload strategies for a set of object files |
| `DatasetPushObjectsTransferResponse` | Server → Client | Upload strategies for the requested object files |
| `DatasetPushComplete` | Client → Server | Informs that all objects were uploaded |
| `DatasetPushCompleteConfirmed` | Server → Client | Confirms the commit of the new blocks |
| `DatasetError` | Server → Client | Reports an error that happened during the operation |

## API Definitions

* [Smart Transfer Protocol: OpenAPI](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/transfer/smart-transfer-protocol.openapi.yaml) - HTTP endpoints of the Simple and Smart Transfer Protocols
* [Smart Transfer Protocol: AsyncAPI](https://github.com/open-data-fabric/open-data-fabric/blob/master/contracts/transfer/smart-transfer-protocol.asyncapi.yaml) - asynchronous messages of the Smart Transfer Protocol


# Verifiable Batch Queries
Source: https://docs.kamu.dev/odf/spec/exchange/verifiable-batch-queries



**Domain**: Exchange\
**Depends on**: -\
**RFCs**: [RFC-022](/odf/rfcs/022-batch-query-commitments)

## Abstract

This spec defines a commitment scheme for results of batch queries executed by ODF nodes. A commitment cryptographically binds the result of a <Term /> to all of its inputs, including the exact states of the queried <Term />, and to the identity of the node that executed it. This allows consumers to process query results optimistically and hold every party involved in producing the result accountable long after the response was received, by verifying and disputing the results later.

## Commitment Properties

The commitment scheme provides the following properties:

* **Hiding** - The commitment does not reveal any information about the operation. The hashes in the [`commitment`](#commitment) section don't reveal inputs or outputs, while the [`proof`](#proof) section reveals only the identity of the node that performed the computation.
* **Binding** - It is impossible to change any inputs or outputs of the operation without invalidating the commitment.
* **Succinct** - The commitment is small enough to be stored cheaply (e.g. on a blockchain), and disputing the result does not require storing and transferring the resulting data.
* **Deterministic** - The same query results in the same commitment, as long as the query itself is deterministic.
* **Reproducible** - Given the same inputs a third party can repeat the query, arrive at the same results, and validate them against the original commitment.
* **Non-repudiable** - A node cannot deny performing the operation, as the commitment is signed with the key tied to the node's identity.
* **Composable** - Computation split into several steps, possibly executed by nodes belonging to different parties (e.g. when joining datasets of different owners), produces commitments that compose into one final commitment. In case of a dispute the responsibility can be assigned to the specific party, not only to the party that aggregated the final result.
* **Multi-protocol** - The commitment structure can be reused across different protocols without significant alterations.

Until zero-knowledge computation proofs are introduced, reproducibility (see <Term />) is the primary way of verifying the commitments.

## Response Structure

A node executing a query that was requested to be verifiable returns a response consisting of the following sections:

```json theme={null}
{
  "input": {
    "query": "select * from foo",
    "queryDialect": "SqlDataFusion",
    "dataFormat": "JsonAoS",
    "schemaFormat": "ArrowJson",
    "datasets": [{
      "id": "did:odf:fed0..ee02",
      "alias": "foo",
      "blockHash": "f162..3e0b"
    }]
  },
  "output": {
    "data": [
      {"foo": 1, "bar": "A"},
      {"foo": 2, "bar": "B"}
    ],
    "dataFormat": "JsonAoS",
    "schema": {"fields": [...]},
    "schemaFormat": "ArrowJson"
  },
  "subQueries": [{
    "input": {...},
    "subQueries": [...],
    "commitment": {...},
    "proof": {...}
  }],
  "commitment": {
    "inputHash": "f162..3e0b",
    "outputHash": "f162..baaf",
    "subQueriesHash": "f162..8dae"
  },
  "proof": {
    "type": "Ed25519Signature2020",
    "verificationMethod": "did:key:fed0..3e41",
    "proofValue": "u3MvG...ouQk"
  }
}
```

### Input

The `input` section contains all information needed to fully reproduce the result of the operation:

* The query and its dialect
* All parameters that affect the output, including the requested data and schema formats
* The list of queried datasets, each with:
  * `id` - the [unique identifier](/odf/spec/datasets/identity#unique-identifiers) of the dataset
  * `alias` - the name under which the dataset was referred to in the query
  * `blockHash` - hash of the <Term /> block that represents the exact state of the dataset the query was executed against

Datasets that were referenced in the request by aliases are resolved by the node, and states of all datasets are pinned to specific blocks, so that the query can later be repeated against the exact same data.

### Output

The `output` section contains the result data and its schema in the requested formats.

### Sub-queries

The `subQueries` section contains information about parts of the operation that were executed by other nodes (e.g. in federated queries). Every entry has the same structure as the top-level response, excluding the `output` section, making the scheme recursive. Sub-queries are empty when the entire operation was executed by a single node.

### Commitment

The `commitment` section is a succinct commitment to the operation, consisting of:

* `inputHash` - `multihash(canonicalize(input))`
* `outputHash` - `multihash(canonicalize(output))`
* `subQueriesHash` - `multihash(canonicalize(subQueries))`

Hashes are represented as described in [Hash Representation](/odf/spec/datasets/metadata-chain#hash-representation), and `canonicalize` is defined in [JSON Hashing](#json-hashing).

The `commitment` section is technically redundant, as the consumer has to recompute all hashes to [validate](#response-validation) the response anyway. It is included to protect consumers from implementing hashing in a way that is inconsistent with this spec, and to help isolate inconsistencies to specific parts of the response during validation and dispute resolution.

### Proof

The `proof` section ties the commitment to the identity of the node that performed the computation:

* `type` - signature scheme (e.g. `Ed25519Signature2020`)
* `verificationMethod` - DID of the node, from which the public key used to verify the signature can be derived
* `proofValue` - `multibase(sign(canonicalize(commitment)))`

## JSON Hashing

To exclude variations in whitespace, property order, and number representation between JSON libraries, JSON objects are converted into canonical form as specified in [RFC 8785: JSON Canonicalization Scheme](https://datatracker.ietf.org/doc/html/rfc8785) before hashing or signing.

## Response Validation

Immediately upon receiving the response the consumer must validate it to ensure that the commitment is not bogus and is disputable:

1. Verify that the `proof.verificationMethod` DID belongs to the node the data was requested from
2. Verify `commitment.outputHash` by recomputing the hash
3. Verify `commitment.inputHash` by recomputing the hash
4. Verify `commitment.subQueriesHash` by recomputing the hash
5. Verify the signature in `proof.proofValue` using:
   * Message: `canonicalize(commitment)`
   * Public key: `key_from_did(proof.verificationMethod)`
6. Repeat steps 3-5 recursively for every entry in `subQueries`

## Storing Commitments

To be able to dispute the result in future, the consumer should store the following sections:

* `input`
* `subQueries`
* `commitment`
* `proof`

The `output` section doesn't need to be stored, as it can be reproduced from the `input`.

These sections provide different layers of verifiability and can be used separately:

* `input` - provides reproducibility
* `commitment` - seals the response against tampering
* `proof` - adds non-repudiation via signature
* `subQueries` - provide the above for multi-party computations

## Dispute Resolution

The procedure of resolving a dispute using a commitment via reproducibility is as follows:

1. Consumer submits a claim to the verifier, providing the `input`, `subQueries`, `commitment`, and `proof` sections
2. Verifier obtains the datasets listed in `input.datasets` at the specified block hashes
3. Verifier validates the claim according to the [Response Validation](#response-validation) rules
   * If any check fails, the claim is **rejected** as invalid
4. Verifier checks that all nodes that signed the `subQueries` are known (registered) nodes
   * If an unknown node is found, the claim is **accepted** and the blame is assigned to the root node for using a bogus node to compute part of the response
5. Verifier generates the logical query plan and checks that it is identical or equivalent to the plan composed from the `subQueries`
   * If the plan does not correspond to the original query, the claim is **accepted** and the blame is assigned to the root node for providing a malicious query plan
6. In a bottom-up fashion, verifier repeats all computation steps using their `input` sections and compares hashes of the outputs to the corresponding `commitment.outputHash` values
   * If a hash doesn't match, the claim is **accepted** and the blame is assigned to the node that signed the corresponding commitment for providing malicious output data

## Use in Other Protocols

While described here using JSON responses of a REST API, the same structure can be used in other protocols, e.g. Flight SQL using canonical structures in a binary format, or blockchain oracles.

In protocols that already provide some of these properties some sections may be redundant. E.g. a blockchain oracle response is already signed by the provider's transaction, making the root-level `proof` unnecessary. Including all sections nevertheless allows using the same dispute resolution procedure across protocols, without the need to map provider addresses on different blockchains to node identities.


# Overview
Source: https://docs.kamu.dev/odf/spec/overview



## Abstract

**Open Data Fabric** is an open protocol specification for decentralized exchange and transformation of semi-structured data that aims to holistically address many shortcomings of the modern data management systems and workflows.

## Specs

ODF is specified as a set of interconnected specs grouped by domain. Core concepts used by all specs are defined in the [Glossary](/general/glossary), and all schemas are described in the [Reference](/odf/schemas).

| Domain | Spec | Description |
| - | - | - |
| Datasets | [Logical Structure](/odf/spec/datasets/logical-structure) | General structure of an ODF dataset |
| Datasets | [Identity](/odf/spec/datasets/identity) | `did:odf` identifiers, aliases, and references |
| Datasets | [Data Model](/odf/spec/datasets/data-model) | Event model, system columns, retractions and corrections, schema evolution |
| Datasets | [Data Schema](/odf/spec/datasets/data-schema) | Logical schema format of data |
| Datasets | [Metadata Chain](/odf/spec/datasets/metadata-chain) | Blocks, events, consistency rules, and block hashing |
| Datasets | [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing) | Physical and logical hashing of data and checkpoints |
| Datasets | [Storage Layout](/odf/spec/datasets/storage-layout) | Layout of dataset components in various storage systems |
| Datasets | [Ingestion](/odf/spec/datasets/ingestion) | Sources, ingestion phases, and merge strategies |
| Transformations | [Processing Model](/odf/spec/transformations/processing-model) | Derivative datasets, watermarks, checkpoints, and query evolution |
| Transformations | [Engine Contract](/odf/spec/transformations/engine-contract) | Communication between coordinators and engines |
| Exchange | [Transfer Protocols](/odf/spec/exchange/transfer-protocols) | Simple and Smart Transfer Protocols for synchronizing datasets |
| Exchange | [Verifiable Batch Queries](/odf/spec/exchange/verifiable-batch-queries) | Commitments that make query results verifiable |
| Automation | [Resource Framework](/odf/spec/automation/resource-framework) | Declarative resource model and manifest format |
| Automation | [Tasks and Flows](/odf/spec/automation/tasks-and-flows) | Scheduling and execution of workloads |
| Auth & Privacy | [Authorization](/odf/spec/auth-and-privacy/authorization) | Accounts, groups, auth attributes, and policies |

## Problem Statement

Develop a method of semi-structured data exchange that would:

* Enable worldwide collaboration around data cleaning, enrichment, and transformation.
* Create an environment of verifiable trust between participants without the need for a central authority.
* Enable a high degree of data reuse, making quality data more readily available.
* Reduce the time it takes data to propagate from publishers to consumers.
* Introduce a feedback loop between the consumers and publishers of data that would facilitate improvements around data availability, recency, and design.
* Provide foundation for implementing data anonymization techniques to simplify and speed up the exchange of vital but sensitive data.

## Requirements

Functional:

* **Complete historical account** - when data is used to gain insight and to drive decision-making discarding or modifying data is akin to rewriting history. The history of all data observed by the system must be preserved.
* **Reproducibility** - the ability to reproduce the results is a cornerstone of the scientific method without which the process and findings of one party cannot be verified by others. Therefore, we require that all the transformations performed within the system must be fully reproducible, and it must be possible to get a reference to data that is frozen in time and never changes to achieve reproducibility of any process that uses it.
* **Verifiability** - any party that plans to use some data must be able to verify its validity.
* **Provenance** - regardless of how many transformation stages the data went through, it should be possible to trace any individual data cell back to its ultimate source and understand which data contributed to its existence and its value.
* **Privacy** - sensitive data must be exchangeable and usable in computations without being disclosed to unauthorized parties. Owners of data must stay in control over who can access it, while the above requirements of verifiability and provenance must still be satisfied.

Non-Functional:

* **Transparency** - the method itself should be developed in the open, easy to understand, and reason about.
* **Timeliness** - the method should aim to minimize the propagation time of data through the transformation pipeline. The frequency with which data is presented to consumers should be optimized for their experience and usability, not dictated by limitations of the data pipeline.
* **Transactionality** - once data is in the system all operations should have [ACID properties](https://en.wikipedia.org/wiki/ACID) and exactly-once semantics.
* **Decentralization** - there must be no entity in the system with any kind of special privileges over data, or abilities to circumvent any of the above requirements.

## Scope

The primary focus of this specification is the **mission-critical data** such as:

* Business performance data (sales, CRM)
* Governmental data
* Financial data
* Healthcare data
* Scientific data
* As well as any other data used for decision-making

This end goal is however to make the specification suitable for very high volume sources like IoT and sensor data. We start with infinite retention of history and provide operations like compactions and pruning for use cases where it may be cost-prohibitive. The hope is to shift the mindset of the data science community towards thinking about such cases as **edge cases** and no longer treat routine data and history loss as our default mode of operations.

## Design Decisions

### Nature of Data

In modern data science "data" is a very broad term that includes many shapes and forms of quantitative and qualitative records. We will not attempt to provide a taxonomy here but will claim that the large portion of data published today carries in it the limitations and design compromises of systems that generated it.

> Examples:
>
> **Snapshot data** - many [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) systems are designed to only maintain the current state of the information. The data sourced from OLTP systems is usually in the form of periodic database dumps (snapshots) or [change data capture](https://en.wikipedia.org/wiki/Change_data_capture) logs at best.
>
> Such data has many limitations:
>
> * It's often accompanied by data loss (e.g. all data changes between the two snapshots are lost).
> * Data is anemic and doesn't carry the business context of the changes.
>
> **Aggregated data** is the form of data in which information from several measurements is combined and expressed in a collective or summary form. This form of data is very common in governmental and healthcare sources, where aggregations often hide important deprivations and inequalities between gender, ethnic, and other groups.
>
> Although such styles of data have become the norm for many major data publishers today, we believe that such treatment is **inadequate** and at odds with many of the requirements we put towards the system. Data is our modern-day history book and should be treated as such.

To achieve the desired qualities we choose the following narrow definition of data:

> Data is a set of events, observations, or propositions believed to be true at a certain time.

Properties:

* Data is a set of event records / relational tuples
* Events are **immutable**
* Event set can only grow
* Every event set can be viewed as a potentially infinite event stream

Such representation poses many additional challenges when working with data, but, as we will show further - the benefits by far outweigh the added complexity, and that complexity can in most cases be addressed by better tooling.

### Nature of Transformations

The state-of-the-art approach to transforming data today is to version the source data (using a hash sum or a stable reference) and version the code that transforms it (using a version control system). The result of transformations is then uploaded to some shared storage and made available to others. There are many tools that improve the reproducibility of such workflows, but all of them treat data as a mere binary blob, deprived of its intrinsic qualities.

This leads to many problems:

* Data is routinely copied
* Modified versions of data essentially create new datasets that cannot be easily traced to their origin
* The history of transformations (provenance) cannot be easily established
* It is practically impossible to verify that no accidental or malicious alterations were made

As a result, our workflows create large quantities of datasets that are disjoint from their sources, that contribute to the overall "noise" when searching for data but cannot be meaningfully reused. Any data scientist that cares about the validity of their results therefore must begin with a clean slate and work with the primary authoritative source of data.

> Modern data science is stuck in a loop where all forward progress is constantly lost because no mechanism exists for making incremental improvements to data.

#### Components of Trust

We believe that the most important factor in enabling reuse and collaboration on data is **trust**.

While the implicit trust may be enough within an enterprise, trust on the global scale cannot be blind - it is only possible through transparency, reproducibility, and verifiability of the results. Those qualities cannot be easily layered on top of the existing tools, so we designed the system from the ground up with those qualities at its core.

In our design we achieve trust with following solution criteria:

* When using any data you must be able to trace its source to a trusted authority that originally published it
* All transformations that were applied to it must be known and can be audited
* The results of those transformations can always be reproduced by following the same transformation steps

<Diagram alt="Diagram: Transformation" />

Components:

* Transformations are expressed using <Term /> written in any of the supported languages.
* Queries combine and transform one or many input data streams into an output stream.
* Queries are executed by the <Term /> - a software that knows how to interpret and execute a certain query dialect.
* The results of queries must be deterministic - always produce the same results for the same set of inputs.
* Engines run in a "sandbox" environment, where they don't have access to a network or any other external resources.
* Engines are strictly versioned as their implementation contributes to the total determinism of the operation.

A transformation executed with the same query, on the same inputs, and using the same version of an engine is **guaranteed to be reproducible**.

This is indeed a very hard departure from the ways of working with data that we are used to. It prohibits using anything external to the system - there can be no API calls, no manual editing of data, no use of 3rd party libraries that are not part of the engine.

The benefits we get in exchange for this strictness are, however, immense:

* **Complete transparency** on where data comes from.
* Ability to audit all transformations.
* Ability to easily **reproduce** and **verify** all results.

All this combined allows us to build **verifiable trust** between parties.

#### Verification

Different kinds of data are verified using different mechanisms, each specified alongside the part of the system it applies to:

* **Integrity of datasets** - having only the hash of the last metadata block obtained from a trusted source, the entire [Metadata Chain](/odf/spec/datasets/metadata-chain#validation) and all [data and checkpoint files](/odf/spec/datasets/data-and-checkpoint-hashing#validation-and-verification) it references can be validated, regardless of where they were [downloaded from](/odf/spec/exchange/transfer-protocols#validation-of-transferred-data)
* **Root data** - cannot be reproduced, so its validity depends on its publisher (see [Ingestion: Validity of Root Data](/odf/spec/datasets/ingestion#validity-of-root-data))
* **Derivative data** - can be verified by reproducing the declared transformations, recursively across the entire lineage, or using other forms of verifiable computing (e.g. TEEs and zero-knowledge proofs) (see [Processing Model: Verification](/odf/spec/transformations/processing-model#verification))
* **Query results** - can be verified using query commitments (see [Verifiable Batch Queries](/odf/spec/exchange/verifiable-batch-queries))

See also:

* <Term />

### Stream Processing Model

As we model all data as potentially infinite event streams (see [Nature of Data](#nature-of-data)), it's quite natural to use the **stream processing** techniques for all data transformations.

When working with temporal events, stream processing has the following benefits:

* Lets users define a query once and potentially run it forever. This nicely addresses the **timeliness** requirement, allowing us to minimize the latency with which data propagates through the system.
* Streaming queries are expressive and are closer to *"which question is being asked"* as opposed to *"how to compute the result"*. They are usually much more concise than equivalent batch queries.
* They can be expressed in a way that is agnostic of how and how often the new data arrives. Whether the data is ingested once a month in Gigabyte batches, in micro-batches every hour, or as a true near real-time stream - processing logic can stay the same, produce the same results, and guarantee the best propagation times possible.
* Modern stream processing techniques make complex problems like handling late and out-of-order data much simpler, more explicit, and significantly less error-prone than equivalent batch operations.

### Essential Role of Metadata

Metadata is the cornerstone of our design. It contains every aspect of where the data came from, how it was transformed, and everything that ever influenced how data looks like throughout its entire lifetime. In a way it's like a digital passport of data using which we can audit and verify its validity (see [Transfer Protocols: Design Goals](/odf/spec/exchange/transfer-protocols#design-goals)).

The metadata is always updated whenever data gets updated and stays fully consistent with it. The two are cryptographically linked, making spoofing it near impossible.

Metadata also has an immutable append-only nature. While the specs describe it as a standalone technology, its format is designed to make it easy to integrate with existing ledger-based systems such as blockchain.

### Transactional Semantics

To achieve a perfect reproducibility the system needs to satisfy very strong transactional properties:

* All coordination operations including things like ingesting, downloading, and sharing data, creating and updating datasets must have [ACID properties](https://en.wikipedia.org/wiki/ACID).
* All data transformations must have **exactly-once** semantics.

## Components

ODF defines the following components that take part in managing and processing data:

### Coordinator

<Term /> is an application that implements the common <Term /> management logic. Its main functions are to:

* Maintain the [invariants](#requirements) and the [semantics](#transactional-semantics) of the system
* Guarantee the validity and integrity of the [Metadata Chain](/odf/spec/datasets/metadata-chain)
* Ingest external data into <Term /> (see [Ingestion](/odf/spec/datasets/ingestion))
* Interface with the [Engines](#engine) to implement data transformations (see [Processing Model](/odf/spec/transformations/processing-model))
* Enable data sharing via <Term /> (see [Transfer Protocols](/odf/spec/exchange/transfer-protocols))
* Provide additional functionality to simplify working with datasets and their metadata

### Engine

<Term /> is a software that knows how to interpret and execute <Term /> of a certain dialect. Engines are responsible for applying the queries to input data and returning the result, as well as answering <Term /> queries.

To guarantee the reproducibility of all transformations, engines run in a sandboxed environment without access to any external resources, and are strictly versioned (see [Components of Trust](#components-of-trust)). The coordinator communicates with engines according to the [Engine Contract](/odf/spec/transformations/engine-contract), making engines interchangeable - the set of supported query dialects can be extended by implementing a new engine.

### Node

<Term /> is an entire ODF deployment that acts under a single identity. It combines a coordinator with a set of engines, and additionally:

* Hosts datasets and other <Term /> that belong to one or many <Term />, and runs controllers that reconcile their desired state (see [Resource Framework](/odf/spec/automation/resource-framework))
* Controls access to resources and data (see [Authorization](/odf/spec/auth-and-privacy/authorization))
* Schedules and executes workloads like ingestion and transformations (see [Tasks and Flows](/odf/spec/automation/tasks-and-flows))
* Exposes datasets to other participants via transfer and query protocols (see [Transfer Protocols](/odf/spec/exchange/transfer-protocols) and [Verifiable Batch Queries](/odf/spec/exchange/verifiable-batch-queries))

Nodes communicate with one another to replicate datasets, execute parts of queries, and exchange other information, forming a decentralized network.


# Engine Contract
Source: https://docs.kamu.dev/odf/spec/transformations/engine-contract



**Domain**: Transformations\
**Depends on**: [Data Model](/odf/spec/datasets/data-model)\
**RFCs**: [RFC-015](/odf/rfcs/015-unified-changelog-stream-schema)

## Abstract

This spec defines the contract between a <Term /> and an <Term /> - the sandboxed execution environment of engines, the communication interface and the operations engines must support, and how engine versions are pinned and deprecated to preserve reproducibility of transformations.

## Execution Model

<Term /> executes in a very restricted environment (a "sandbox") that prevents it from accessing any external resources other than those explicitly provided by the <Term />. Isolating the <Term /> from making network calls and accessing any random files ensures both that the <Term /> being processed cannot be leaked, and that non-deterministic behavior is harder to introduce accidentally.

Our isolation technology of choice is  [OCI containers](https://opencontainers.org/) - a lightweight mechanism that provides good performance and strong isolation guarantees. We rely on OCI-compatible images as the distribution mechanism for the engines.

> **WIP:** Add exact `podman` and `docker` commands to reproduce the sandboxed environment

## Communication Interface

This section describes the [RPC](https://en.wikipedia.org/wiki/Remote_procedure_call) mechanism used in communications between the <Term /> and the <Term />.

The requirements we put towards the RPC mechanism are:

* Use common network protocols
* Wide support in many programming languages
* Support streaming responses (e.g. for streaming back the operation status and logs)

Our RPC technology of choice is [gRPC](https://grpc.io/) because:

* It's cross-platform, mature, and available in many languages
* Natively supports [FlatBuffers](https://google.github.io/flatbuffers/flatbuffers_grpc_guide_use_cpp.html) which is our [metadata format](/odf/spec/datasets/metadata-chain#metadata-format) of choice, letting us reuse object schemas
* Supports streaming
* For our narrow use case, its convenience is more important than interoperability, as we don't expect to have many server and client implementations

<Term /> implementations must expose the `gRPC` API on port `5544`.

Request and response messages are defined by the [message schemas](#message-schemas) and serialized using FlatBuffers. Every operation responds with a stream of messages, where zero or more `Progress` messages are followed by exactly one terminal message:

* `Success` - operation completed successfully
* `InvalidQuery` - the query did not pass validation, with a `message` explaining the error
* `InternalError` - an unexpected error occurred, with a `message` and an optional `backtrace` with details

> Note: When starting an <Term /> container, the <Term /> will consider it fully initialized when it can establish a TCP connection to the port `5544`.

### Data Exchange

The exchange of raw data happens out-of-band of the `gRPC` API.

Currently, data and checkpoints are exchanged via files and object stores. The <Term /> makes input data part files and the previous <Term /> available to the <Term /> and passes their paths in the request, along with the paths where the <Term /> must write the output data and the new <Term />.

In future, input and output <Term /> may be also exchanged using memory-mapped files (`mmap`) containing data in the common [in-memory format](/odf/spec/datasets/data-model#in-memory-and-on-disk-formats). This avoids any intermediate IO and minimizes the copying of data for the best performance.

## Operations

Engine implementation should support the following operations:

* [Execute query](#execute-query) - Performs the next iteration of the transformation.
* [Raw query](#raw-query) - Executes a query over raw data, e.g. during preprocessing of data in ingest.
* [Validate query](#validate-query) - Validates the user-specified query for basic syntax and schema correctness.
* [Migrate query](#migrate-query) - Updates the transformation state from one query to another.
* [Derive Provenance](#derive-provenance) - Explains the origin of some data produced in the past.

> **WIP:** Validate query, Migrate query, and Derive Provenance operations do not yet have defined [message schemas](#message-schemas).

### Execute Query

This operation is used by the <Term /> to perform the next iteration of the transformation.

The <Term /> is designed to isolate the <Term /> from the complexity of managing the <Term /> and make processing look as close as possible to a regular stream processing.

<Diagram alt="Sequence Diagram: Execute Query" />

Inputs (`TransformRequest`):

* ID and alias of the output <Term />
* <Term /> to assign to new records
* Offset to assign to the first new record
* Vocabulary of the output <Term /> - mapping of system columns onto its schema
* Transformation <Term />
* Inputs - for every input <Term />:
  * ID and alias of the dataset
  * Alias under which the input is referred to in the <Term />
  * Vocabulary of the input dataset
  * Offset interval of the input <Term /> that goes into this transaction
  * Schema and data files of the input
  * Explicit <Term /> that should be injected into the stream to separate micro-batches for reproducibility
* Previous <Term />
* Locations where new data and the new <Term /> should be written

Outputs (`TransformResponse`):

* Offset interval of the output <Term />, if any data was produced
* Output <Term />, if it was advanced
* New <Term />
* Operation progress and status

This operation must be [idempotent](https://en.wikipedia.org/wiki/Idempotence).

Input and output data are changelog streams that can contain <Term /> (see [Processing Model: Retractions and Corrections](/odf/spec/transformations/processing-model#retractions-and-corrections)). The <Term /> must:

* Consume input records in the order of their <Term />, as the order of retractions and corrections relative to the events they apply to is significant
* Take the operation type of input records into account and propagate retractions and corrections into the output
* Produce output that is a valid changelog stream (see [Data Model](/odf/spec/datasets/data-model#representation-of-retractions-and-corrections)) - in particular, it must not reorder records in a way that separates a correction pair or places a retraction before the event it retracts, and must write both events of a correction pair into the same output <Term />

Engines that process data in batches and parallelize work must preserve the order of records (e.g. by coalescing inputs into one partition). Such engines that only support simple record-by-record operations (like `map` and `filter`) must propagate the operation type column as-is, so that operations apply identically to appends, retractions, and corrections and produce a valid changelog stream.

When an operation is committed by the <Term />, the <Term /> will not see the same input data again. If due to the nature of the query (e.g. windowing or watermarking configuration) <Term /> cannot fully process and discard the input data - it should use the <Term /> to buffer it.

### Raw Query

This operation is used by the <Term /> to execute a <Term /> over raw data that is not yet part of any dataset, e.g. during the preprocessing step of <Term />.

Inputs (`RawQueryRequest`):

* Input data files to perform the query over - all files must have identical schema
* Transformation <Term />
* Location where the query result should be written

Outputs (`RawQueryResponse`):

* Number of records produced by the query
* Operation progress and status

### Validate Query

This operation may be used by the <Term /> when creating a new <Term /> or when changing the <Term /> of an existing one to validate the user-specified query for basic syntax and schema correctness.

Inputs:

* <Term /> of the input <Term />
* <Term />
* (optional) Previous <Term /> in case of an existing <Term /> modification

Outputs:

* <Term /> of the result
* (alternatively) Validation errors

### Migrate Query

This operation is used by the <Term /> when data processing hits the point where one transformation <Term /> is being replaced by another one. It gives the <Term /> all information needed to handle the transition as gracefully as possible, e.g. by reconciling the existing <Term /> with the new <Term />, or, at the very least, finalizing the processing of the old <Term /> and clearing the <Term /> before the new one starts to execute.

Inputs:

* Transaction ID
* Previous <Term />
* Next <Term />
* Previous <Term />

Outputs:

* New <Term />
* (optional) <Term /> that has been produced when finalizing the old <Term />

### Derive Provenance

This operation is used to trace back a set of events in the output <Term /> to input events that contributed to their values or their existence (see <Term />).

> **WIP:** The design of this operation is in progress.

## Engine Versioning

As described in [Components of Trust](/odf/spec/overview#components-of-trust), to guarantee the reproducibility and verifiability of the results a transformation must be associated with an exact version of an <Term /> that is used to perform it. We want to exclude any possibility that the code changes in the <Term /> will break this guarantee.

Whenever the <Term /> uses an <Term /> to execute a query it must specify the full digest of the OCI container image in the resulting <Term />. <Term /> maintainers are therefore responsible for treating the images as immutable and ensure old versions are never lost.

See also:

* [Engine Deprecation](#engine-deprecation)

## Engine Deprecation

Pinning the exact <Term /> version used by every transformation guarantees the reproducibility and verifiability of the results (see [Components of Trust](/odf/spec/overview#components-of-trust)). In the long run, however, this creates a problem where in order to [verify a dataset](/odf/spec/transformations/processing-model#verification) the <Term /> will have to use a very large number of different versions of an <Term /> that might've accumulated over the years. This slows down the verification procedure and can result in a significant disk space required for storing these images.

To mitigate this problem the <Term /> offers the **engine deprecation** operation that works as follows:

1. The <Term /> reconstructs the dataset from scratch by repeating all transformation in the <Term />.
2. Instead of the <Term /> version specified in the block, it uses the latest (or provided) version.
3. If the results produced in every step match the recorded ones - we can safely consider that the newer version of the <Term /> is fully compatible with the older one.
4. A special <Term /> record is created to record this compatibility, therefore, allowing all users to use the latest version of the <Term /> instead of downloading all the versions seen previously.

## Message Schemas

Messages exchanged between the <Term /> and the <Term /> are defined by the following schemas:

| Schema | Description |
| - | - |
| <Schema /> | Sent by the coordinator to perform the next step of data transformation |
| <Schema /> | Describes an input of the transform request |
| <Schema /> | Sent by the engine when performing the data transformation |
| <Schema /> | Sent by the coordinator to perform a query on raw input data |
| <Schema /> | Sent by the engine when performing the raw query |


# Processing Model
Source: https://docs.kamu.dev/odf/spec/transformations/processing-model



**Domain**: Transformations\
**Depends on**: [Data Model](/odf/spec/datasets/data-model), [Metadata Chain](/odf/spec/datasets/metadata-chain), [Engine Contract](/odf/spec/transformations/engine-contract)\
**RFCs**: [RFC-015](/odf/rfcs/015-unified-changelog-stream-schema)

## Abstract

This spec defines how data of <Term /> is produced from their inputs. It describes how derivative datasets are defined, how <Term />, <Term />, and <Term /> are used to process potentially infinite event streams incrementally with exactly-once semantics, the steps a <Term /> performs during a transformation, and how transformation queries can evolve over time.

## Derivative Datasets

All transformations in ODF follow the stream processing model (see [Stream Processing Model](/odf/spec/overview#stream-processing-model)).

A <Term /> is defined by the <Schema /> event in its [Metadata Chain](/odf/spec/datasets/metadata-chain), which specifies:

* `inputs` (<Schema />) - the input datasets, each with an `alias` under which it is available in queries (see [Identity: References in Metadata](/odf/spec/datasets/identity#references-in-metadata))
* `transform` (<Schema />) - the <Term />, the <Term /> that executes it, and the engine version

```yaml theme={null}
kind: SetTransform
inputs:
  - datasetRef: sensor.temp
transform:
  kind: Sql
  engine: flink
  query: |
    SELECT
      TUMBLE_START(event_time, INTERVAL '1' HOUR) as event_time,
      min(value) as value_min,
      max(value) as value_max
    FROM `sensor.temp`
    GROUP BY TUMBLE(event_time, INTERVAL '1' HOUR)
```

Queries can consist of multiple steps, where every step acts as a shorthand for `CREATE TEMPORARY VIEW <alias> AS (<query>)` and the last step produces the output. When stored in the metadata, a single `query` is always converted into the `queries` array.

Every execution of the transformation is recorded in the chain as an <Schema /> event that captures the exact intervals of input blocks and records that were processed, and the resulting data, checkpoint, and watermark (see [Metadata Chain: Offsets and Intervals](/odf/spec/datasets/metadata-chain#offsets-and-intervals)).

### Derivative Data Transience

Since past data in the input datasets is immutable and all transformations are required to be deterministic - derivative datasets are **functionally dependent** on their inputs. This means that for any given transformation graph it is possible to fully reconstruct data of all derivative datasets from the data of root datasets by simply re-applying all transformations.

> Data in derivative datasets can be considered a **form of caching** and does not require durable storage.

## Time and Watermarks

Following the <Term /> data model, all records carry the <Term /> when they were observed by the system, and the <Term /> when they occurred from the perspective of the outside world. Transformations operate primarily in event time.

When performing time-based windowed operations, aggregations, or joins it is important to know when a certain time window can be considered closed. Every `AddData` and `ExecuteTransform` event can carry a new <Term /> of the dataset - the point in event time for which all preceding events have most likely been observed:

* In <Term /> watermarks are advanced during ingestion, and can also be advanced without adding new data, e.g. for slow-moving datasets where it is normal not to see any events for a long time
* In derivative datasets, watermarks of the inputs are passed into the <Term /> along with the data (see [Engine Contract: Execute Query](/odf/spec/transformations/engine-contract#execute-query)), and the engine determines the output watermark

Watermarks are monotonically non-decreasing. Events that arrive after their time interval was closed by the watermark are considered "late", and it's up to individual queries to decide how to handle them - ignore them, emit them into a side output, or emit <Term /> into the output.

## Retractions and Corrections

Data of all datasets is a changelog stream that can contain <Term /> of previously published events (see [Data Model](/odf/spec/datasets/data-model#representation-of-retractions-and-corrections)). Transformations must take them into account:

* **Retractions and corrections in the inputs** must be propagated - a transformation must update its previously produced results that were influenced by the retracted or corrected input events, by emitting retractions or corrections into the output
* **Late events** in the inputs can similarly cause a transformation to retract or correct its previously produced results (see [Time and Watermarks](#time-and-watermarks))
* **The nature of the query** itself may require it to revise its results as new events arrive, even when the inputs contain only appends

This makes all derivative datasets self-correcting - a correction issued in a root dataset propagates through the entire graph of transformations without any manual intervention.

For example, consider a `match_scores` dataset that receives events as players complete the matches:

| op | match\_time | match\_id | player\_name | score |
| - | - | - | - | - |
| +A | t1 | 1 | Alice | 100 |
| +A | t1 | 1 | Bob | 80 |
| +A | t2 | 2 | Alice | 70 |
| +A | t2 | 2 | Charlie | 90 |
| +A | t3 | 3 | Bob | 60 |
| +A | t3 | 3 | Charlie | 110 |

A derivative dataset that continuously tracks the two highest-scoring players (a leader board) can be defined using a Top-N query:

```sql theme={null}
select *
from (
  select
    row_number() over (order by score desc) as place,
    match_time,
    player_name,
    score
  from match_scores
) where place <= 2
```

Even though the input contains only appends, the leader board changes as new scores arrive, so the output contains corrections:

| op | place | match\_time | player\_name | score |
| - | - | - | - | - |
| +A | 1 | t1 | Alice | 100 |
| +A | 2 | t1 | Bob | 80 |
| -C | 2 | t1 | Bob | 80 |
| +C | 2 | t2 | Charlie | 90 |
| -C | 1 | t1 | Alice | 100 |
| +C | 1 | t3 | Charlie | 110 |
| -C | 2 | t2 | Charlie | 90 |
| +C | 2 | t1 | Alice | 100 |

If the source later retracted the result of the first match, the leader board would in turn issue retractions and corrections of the affected places.

<Term /> are responsible for implementing these semantics and producing valid changelog streams (see [Engine Contract: Execute Query](/odf/spec/transformations/engine-contract#execute-query)).

## Checkpoints

Engines store the computation state between different invocations of a query in <Term />. Checkpoints are fully engine-specific and opaque to the system, but they are an essential durable part of a dataset, necessary to pause and resume streaming queries.

Every transaction restores the state from the checkpoint referenced by `prevCheckpoint` and records the new checkpoint in `newCheckpoint` of the `ExecuteTransform` event, atomically with the output data. Together with the exact intervals of processed inputs, this provides **exactly-once** processing semantics - every input record contributes to the output exactly once, even if the transformation is interrupted and resumed.

Checkpoints are hashed and stored as described in [Data & Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing#checkpoint-hashing) and [Storage Layout](/odf/spec/datasets/storage-layout#checkpoints).

## Derivative Transformations

The [Engine Contract](/odf/spec/transformations/engine-contract#execute-query) describes how an <Term /> executes the transformation query. This section describes what happens on the <Term /> side.

<Diagram alt="Sequence Diagram: Execute Query" />

Here are the steps that the <Term /> performs during the transformation:

* **Batch step** - Analyze the <Term /> of the <Term /> being transformed and all the inputs. The goal here is to decide how far the processing can progress before hitting one of the special conditions, such as a change of schema in one of the inputs or a change of the transformation query.
* **Run migrations** (when needed) - If a special condition is encountered - call the <Term /> [Migrate Query](/odf/spec/transformations/engine-contract#migrate-query) operation to make necessary adjustments for the new transformation parameters.
* **Run query** - Pass the input <Term /> into the <Term /> [Execute Query](/odf/spec/transformations/engine-contract#execute-query) operation
* **Hash resulting data and checkpoint** - Obtain a stable <Term /> of the output <Term /> and <Term /> (see [Data Hashing](/odf/spec/datasets/data-and-checkpoint-hashing#physical-vs-logical-hashes) and [Checkpoint Hashing](/odf/spec/datasets/data-and-checkpoint-hashing#checkpoint-hashing))
* **Prepare commit** - Creates the next <Term /> describing the output data
* **Commit** - Atomically adds the new <Term /> and the <Term /> to the <Term />

## Verification

The verification of a <Term /> confirms that its data is in fact produced by applying all the transformations stored in its [Metadata Chain](/odf/spec/datasets/metadata-chain) to the inputs and was not maliciously altered.

Verification is **recursive**: data of a derivative dataset can only be as valid as the data of its inputs, so fully verifying it involves verifying all datasets in its lineage, up to the <Term /> (see [Ingestion: Validity of Root Data](/odf/spec/datasets/ingestion#validity-of-root-data)). Since every `ExecuteTransform` event records the exact blocks of the inputs it was applied to, each input is verified at the specific state that was used, regardless of how it has evolved since.

Verification does not prove that the declared transformations are correct or benign - only that they were applied faithfully. Transformations are expressed as <Term /> that are usually small, so they can be audited to ensure they are sound and done in good faith. Establishing trust in a derivative dataset therefore combines inspecting its lineage, auditing the transformations along the way, and verifying that its data matches them.

### Verification via Reproducibility

Reproducing transformations is conceptually the simplest way of verifying derivative data. The process is almost identical to the [Derivative Transformation](#derivative-transformations), except instead of the `Commit` step the <Term /> compares the hashes of the data produced by the local transformation to the hashes stored in the `ExecuteTransform` events, using the exact versions of engines recorded in the metadata.

Due to non-determinism of the Parquet format the physical hash may not match the one in metadata, so a fallback to the logical hash is required to verify that the resulting records are logically the same (see [Data & Checkpoint Hashing: Validation and Verification](/odf/spec/datasets/data-and-checkpoint-hashing#validation-and-verification)).

Reproducing transformations is necessary even when hashes of data match the ones in metadata. Hash checks only ensure that data was not tampered with or corrupted, but data and metadata could have been replaced together. Reproducing the transformations ensures that the data in fact corresponds to the declared transformations, so metadata cannot be spoofed to match fake results.

This form of verification requires access to all input data and the compute resources to repeat the processing.

### Verification via Verifiable Computing

ODF aims to also support forms of verifiable computing that allow verifying results without repeating the transformations, such as attestations from [Trusted Execution Environments](https://en.wikipedia.org/wiki/Trusted_execution_environment) (TEEs) and [zero-knowledge proofs](https://en.wikipedia.org/wiki/Zero-knowledge_proof) (ZKPs). In this model the party performing the transformation produces a proof that the declared transformation was applied faithfully to the specific inputs, which others can check at a fraction of the cost of repeating the computation.

Since verifying such proofs does not require access to the input data, verifiable computing can also **preserve privacy** - derivative data can be verified without disclosing the potentially sensitive data it was produced from to the verifying parties.

### Asynchronous Verification

Verification does not have to happen synchronously with updates of data. Since every `ExecuteTransform` event captures the exact intervals of inputs it processed, verifiers can proceed incrementally and asynchronously from the last verified block - e.g. a dataset may already be on block `T` while verifiers are still checking processing block `T - 100`, aiming to eventually catch up. This allows participants to use data optimistically, while preserving the ability to detect and dispute invalid data later. Clients can define verification thresholds, e.g. requiring N verifiers to check the block before it is consumed.

## Dataset Evolution

Most parts of the <Term /> can [change over time](/odf/spec/datasets/logical-structure#evolution-of-data). To protect downstream consumers of data from breaking changes the <Term /> has to validate the changes for backward compatibility before accepting them. This section covers the allowed transitions for different parts of the <Term />.

### Query Evolution

The procedure for modifying the <Term /> of a <Term /> is as following:

1. The <Term /> invokes the <Term /> [Validate Query](/odf/spec/transformations/engine-contract#validate-query) operation to validate the new <Term /> and determine the validity of the transition according to the engine-specific rules.
2. The new <Term /> returned in the previous step is validated according to the [Schema Evolution](/odf/spec/datasets/data-model#schema-evolution) rules.
3. A new <Term /> is created and committed.

## Provenance

Since the <Term /> are responsible for all data transformations, it's also the engine's responsibility to answer <Term /> queries, i.e. to explain which input data elements an output element was derived from and how. Provenance is derived using the [Derive Provenance](/odf/spec/transformations/engine-contract#derive-provenance) operation of the engine.


# About
Source: https://docs.kamu.dev/platform/about



<div>
  <img alt="Kamu logo" />

  <div>
    <a href="https://github.com/kamu-data/kamu-web-ui/">
      <img />
    </a>

    <a href="https://github.com/kamu-data/kamu-web-ui/releases/latest">
      <img />
    </a>

    <a href="https://github.com/kamu-data/kamu-web-ui/actions">
      <img />
    </a>
  </div>
</div>

## About

Kamu Web Platform is a **front-end application** that acts as a window into the decentralized [Open Data Fabric](/odf) (ODF) network.

<img alt="Kamu Platform screenshots" />

Think of it as GitHub for data pipelines or Etherscan for ODF. It can be used in multiple setups:

* On a laptop to explore and build pipelines together with [Kamu CLI](/cli)
* Deployed as a private cloud or on-premise data platform in combination with [Kamu Node](/node)
* As a window into the global ODF network to interact with millions of decentralized data sources and supply chains.

## Get Started

By far the simplest way to try it out is using [Kamu CLI](/cli) - simply [install](/cli/install) it and run this command in your <Term />:

```sh theme={null}
kamu ui
```

## Helm Chart

You can deploy Kamu Web platform into a [Kubernetes](https://kubernetes.io/) cluster using [Helm](https://helm.sh/) and our latest chart from [ArtifactHub](https://artifacthub.io/packages/search?org=kamu).

See [deployment manual](/node/deploy) on how to deploy the Web Platform alongside [Kamu Node](/node).


# Accounts & Organizations
Source: https://docs.kamu.dev/platform/accounts





# Monitoring
Source: https://docs.kamu.dev/platform/admin/monitoring





# Tasks & Executors
Source: https://docs.kamu.dev/platform/admin/tasks-executors





# Notebooks
Source: https://docs.kamu.dev/platform/data/notebooks





# Querying
Source: https://docs.kamu.dev/platform/data/querying





# Attachments
Source: https://docs.kamu.dev/platform/datasets/attachments





# Licenses
Source: https://docs.kamu.dev/platform/datasets/licenses





# Overview
Source: https://docs.kamu.dev/platform/flows/overview





# Privacy
Source: https://docs.kamu.dev/platform/privacy





# Frequently asked questions
Source: https://docs.kamu.dev/start/faq



### Is Kamu a database?

No. Databases focus on state and transactional processing while Kamu focuses on historical data, analytics, and data science (complex user-defined queries, summaries, statistics). Databases update data in-place, while Kamu is more like a super low-latency accounting ledger - it preserves all history of changes by default allowing you to understand who and how changed data over time. Databases only work within company boundaries, while Kamu is designed for multi-party exchange and similarly to Git allows you to hold everyone who supplied or processed data forever accountable for results. Databases store data in custom data formats, while Kamu uses interoperable formats where data can be copied around as files and efficiently queried by multiple data engines similarly to Open Lakehouse architecture.

### Why separate Kamu CLI, Kamu Node, and ODF?

We believe that participating in global data economy should not require credit cards, accounts, and cloud subscriptions. So we packed ALL features of a modern data lakehouse in a CLI tool that you can install on your laptop and use for free. Kamu Node is a server counterpart - same features but designed for large scale. ODF is the underlying set of formats and protocols for data exchange and verifiable processing - both CLI and Node are implementing it.

### Can I use Kamu with my existing infrastructure?

Yes, Kamu is designed to integrate with existing infra and tools. You can ingest data from databases and run it alonside existing data lakehouses. Kamu supports many standard protocols and query languages like SQL, so your existing data science notebooks, BI tools, and analytics platforms can query Kamu datasets directly. The key benefit of using Kamu throughout your data stack is that once data is in, it gains verifiability, provenance tracking, and collaboration capabilities. See [integrations](/integrations) section for more info.

### What are the performance characteristics of Kamu?

Kamu is designed with massive volumes of near-realtime data in mind (e.g. industrial IoT, finance). It builds on decades of advancements in analytical data management and modern lakehouse architectures. It uses temporal processing model to allow highly efficient incremental processing of streaming data. Its scaling and performance characteristics should closely match those of the underlying engines, with minimal overhead introduced by metadata ledger upkeep and data hashing. Likewise the underlying ODF format is very efficient for data transfer and replication, including direct-to-storage access to fully utilize your network bandwidth.

### Kamu vs. Apache Iceberg, Delta Lake?

Apache Iceberg and Delta formats in modern data lakes serve the purpose of creating a logical representation of **data table** that abstracts the physical layout of data on disk.

Their features include:

* **Mutability** (with atomic updates and read consistency)
* Limited "time travel" (between snapshots)
* Automatic snapshot pruning
* Compactions (merging small data files for query efficiency)
* Evolution of schemas and partitions

The [ODF format](/odf) plays a similar role to Iceber/Delta in traditional lakehouse systems, but instead of representing a table, it representats a **historical ledger of events** - a combination of data and metadata that in detail describe the evolution of a dataset over time. Below we list some of its most notable aspects.

ODF ledgers are mostly **append-only**:

* Unlike Iceberg/Delta and similar formats created around Change-Data-Capture, ODF streams are intended to preserve **complete and unaltered history** of data for reproducibility, verifiability, and ultimate "time-travel" capabilities
* New data is added to the end, while previous records can be revised by issuing explicit [retractions and corrections](/cli/transform/retractions-corrections)
* The format still offers non-destructive compaction for the sake of performance
* The historical data can be pruned, but such actions are also communicated in the format explicitly.

ODF format puts emphasis on [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html) - storing descriptive domain events instead of the "current state". CDC tables still can be expressed using ODF ledger as a changelog stream.

While Iceberg/Delta only work with conventional file systems and object storages (e.g. HDFS, S3), ODF datasets were designed to also work efficiently with **decentralized content-addressable storages** like IPFS and Arweave.

The **rich metadata** of ODF streams is designed to carry a lot more information, such as:

* Applied transformations, for **lineage and provenance** of derivative data
* License changes, semantics, governance information...
* Attachments (data readmes, charts, examples etc.)

ODF datasets also include several **cryptographic features**:

* Embedded global identity, allowing attribution
* Metadata ledger is cryptographically secured and signed, for tamper-proofness and accountability
* In the near future ODF will also support granular encryption, allowing to control access and permissions via key exchange. This will allow private data to be stored in open networks (e.g. IPFS, Filecoin) that do not offer any central point of control to enforce permissions.

So while Iceberg and Delta fill the purpose of table abstractions in enterprise data lakehouses, ODF dataset is a **Web3-native** ledgerized data format that we see as a building block for **multi-party exchange and collaborative processing of data**.

### Kamu vs. Spark, Flink, Kafka Streaming?

`kamu` does not replace or compete with enterprise data processing engines - it uses them as building blocks.

Think of `kamu` as *"Kubernetes for Data"*, where [multiple different engines](/general/supported-engines) are just plug-in components, integrated together into a single **verifiable data processing framework**.

While individual engines let you do almost anything with data, `kamu` is a lot more **opinionated**:

* It tries to enforce many data management best practices like avoiding history loss
* Provides stable references to data for reproducibility
* Specifies how data & metadata are shared/replicated over the network
* Tracks every processing step executed and records provenance
  * *so that a person on another side of the world who downloaded your dataset could understand exactly where every single piece of data came from*
* Handles dataset evolution
  * *so that you could update your processing steps over time without breaking other people's downstream pipelines that depend on your data*
* And much more...

### Kamu vs. Snowflake?

| Snowflake | Kamu |
| - | - |
| Is a robust data platform and enterprise data warehouse | Is a Web3-native data sharing and collaborative processing solution |
| **Centralized**, cloud-only, and based on **proprietary** technology | **Decentralized** network with open protocol and code that scales from laptop to cluster, can span from edge to on-prem to multi-cloud |
| Best for **internal** data management | Best for **cross-organizational** data exchange and collaboration |
| Works with **cloud storage** (S3, Azure, etc.) | Works with both **cloud** and **content-addressabe storage** (IPFS, Arweave) |
| Uses **batch processing** where recurrent data pipelines are expensive to execute, typically run only once a day, and are non-composable due to consistency issues when processing dynamic data | Uses novel **stream processing** that enables near **real-time latencies** and for many workflows can be 1000x **more efficient** than batch, and provides unparalleled **consistency** and **composability** to data pipelines |
| Data is non-verifiable and has no provenance | Data is 100% reproducible, <Term />, and with built-in <Term /> |
| Data can be shared on Snowflake Marketplace but this ties you to Snowflake products | Data can be shared in a decentralized network **without intermediaries** and using **open protocols** |
| On Snowflake Marketplace you can only be a publisher or a consumer of data | Enables **infinitely composable** collaboration on data where parties can clean and combine data into higher-level data products while proving that they are not adding malicious data |
| Only off-chain data, limited to **company-size silos** | Seamlessly integrates **on and off-chain data**, and can provide data to blockchain as an **optimistic oracle** |

### Kamu vs. Databricks?

Databricks provide an amazing set of technologies for enterprise data, however, many of the limitations described in the [Snowflake](#snowflake) section in regards to centralization still apply.

Unlike Databricks platform, `kamu` is:

* **Local-first** - a person can start using `kamu` with nothing but their laptop, without any accounts or any cloud infrastructure, while having access to the same features a platform provides
* **Streaming-frist** - we believe that modern stream processing is a [superior computational model](https://www.kamu.dev/blog/end-of-batch-era/) for automation and far less fragile than batch
* Based on **cryptographic ledgers and verifiable processing** - we build from ground up to ensure reproducibility, auditability, and accurate provenance of data
* **Decentralized** - we respect data ownership and sovereignty and want to enable the smallest organizations and individuals to store data on any inftastructure of their choice. Instead of creating monolyths we focus on enabling millions of small data publishers and processors work together efficiently in a global data economy
* **Collaborative** - built-in verifiability and auditability allows allows data exchange and processing to function across company boundaries, and provides a reliable foundation for multi-party data exchange and processing.

### Kamu vs. Apache Airflow, Apache NiFi?

These tools are "swiss army knives" of data processing. Their ultimate flexibility unfortunately also makes them easily misused, often resulting in [Rube Goldberg machines](https://en.wikipedia.org/wiki/Rube_Goldberg_machine) that are extremely convoluted and fragile.

Our goal with `kamu` is to provide a more opinionated data management framework that:

* Makes following best practices effortless
* Provides configurable tradeoffs between consistency and latency
* Can automatically react to (not-so-)special events like backfills, retractions, corrections
* Captures provenance thoughout the entire pipeline
* Can function across company boundaries
* And much more!

You can use these tools to prepare data for ingestion into `kamu`, but after that you should seriously consider the superior [stream processing computational model](https://www.kamu.dev/blog/end-of-batch-era/) for further proccessing.

### Kamu vs. JSON REST APIs

| JSON REST | Kamu |
| - | - |
| No standard - diffrent auth mechanisms, data models, limits, pagination, error handling | Simple and [open protocol](/odf) |
| JSON can have very significant overhead for large amounts of data | Uses Apache Arrow for the most efficient and compact encoding of structured data |
| API has to be designed with specific use case in mind - this largely favors applications that have predictable access patterns rather than analytics that requires slicing and dicing data | Both application and analytics-friendly |
| Lack of flexibility in queries often monivates users to immediately dump all data | Avoids adverse incentive by providing convinient access |
| Have to be integrated on one-by-one basis - very time consuming and error prone | Data is decentralized but can be accessed uniformly |
| Need software developers to integrate and maintain - expensive | Data can be accessed easily by anyone, without software development skills |
| Not reproducible or verifiabiable - impossible to prove that data came from a source | 100% reproducible, verifiable, and allows holding data providers accountable |
| Not composable - after data is altered you'd need to build a new API infrastructure to share it | Infinitely composable - improved/enriched data can be shared with the network easily |

### Kamu vs. Blockchains?

Although [ODF data format](/odf) is inspired by cryptographic ledgers used in blockchains, `kamu` is NOT a mesh connectivity network, it does NOT maintain a single ledger or run any kind of consensus between nodes.

Nodes in `kamu` talk to each other only to execute a query that touches data spread across them, or to run a streaming data processing task.

Network topology in `kamu` therefore corresponds directly to the data pipeline being executed. Data is replicated only between the nodes that explicitly want to replicate it, and processing tasks are dispatched to nodes that explicitly say they can perform them.

We designed `kamu` to work well with blockchains:

* Blockchain smart contracts can act as a OLTP (transactional) distributed services, delegating data storage and heavy computation tasks to `kamu`
* Smart contracts can request data from `kamu` via [ODF oracle](/api/oracle)
* `kamu` can in turn [use data from blockchains](/cli/ingest/blockchain-source) for processing and analytics.

### Kamu vs. Ceramic Network?

| Ceramic | Kamu |
| - | - |
| Purpose: transactional data store for decentralized apps | Purpose: historical data lakehouse, analytical querying, composable stream processing |
| Stores data using IPLD protocol and IPFS network | **Storage-agnostic**: supports cloud, on-prem, and decentralized storages. Decuopling the query engine from storage infrastructure has long been considered the best practice in data analytics that prevents vendor lock-in of data. |
| Data model based on event streams as cryptographic ledgers. Uses **record-oriented** encoding of payloads (e.g. DAG-CBOR). | Data model also based on event streams as cryptographic ledgers. Data is stored in **industry-standard analytical formats** (Parquet). Using **column-oriented** format allows maximally efficient storage of data, suitable for high-volume and high-frequency IoT use cases. Logical structure of the stream is inspired by **leading lakehouse formats** (Apache Iceberg, Delta) and is developed as an [open specification](/odf) by a consortium of companies. |
| Custom data query API based on GrapghQL | SQL API for data access is backed by a wide range of world's **leading data processing engines** (e.g. Spark, Flink, Datafusion) |
| Custom storage and APIs require users to fully commit to the solution and re-architect their workflows around it | Multiple storage types and support for standard data protocols (e.g. JDBC, ODBC) provides a very gradual migration path that respects user's choices of infrastructure and is **compatible with all modern BI and data science tools** (Jupyter, Tableau, PowerBI, Google Collab etc.). |
| No built-in data transformation capabilities. Derivative data can only be created by extracting data from the system, transforming it, and writing it into a new stream. Transformed data therefore will be disjoint from its origin. The model is **non-composable**. | Data transformation is a core part of the system. Processing pipelines can be built using streaming SQL allowing to create higher-level datasets that clean, aggregate, enrich, and integrate data. Provenance of derivative data is preserved. Pipelines are **infinitely composable**. Using **verifiable processing** the validity of all data can be verified, allowing to build data pipelines that span arcoss multiple organizations in a global **trustless** environment. |

### Kamu vs. Space And Time?

| Space And Time | Kamu |
| - | - |
| Centralized data warehouse using Zero-Knowledge proofs | Decentralized data lakehouse, a Kubernetes-like framework for data management, and a multi-party collaborative data processing network |
| Closed-source | Open source and based on open formats and protocols |
| Cloud-only (Azure) | Can be deployed anywhere from a laptop to on-premise and public/private clouds |
| Batch query model only | Supports **batch** querying for analytics and **stream processing** for building low-latency, autonomous, and highly composable data processing pipelines. |
| Verifiable computing using **Zero-Knowledge proofs** allows computations to be performed on private data with a guarantee of correctness without revealing the data. At the time of writing ZK SQL implementation was limited to a few basic data types and operations and did not support `JOIN`s, making applicability very limited. Generating ZK proofs is a highly computationally-expensive task requiring specialized hardware with powerful GPUs limiting deployment options. | Verifiable computing via **reproducibility**. This is a simple and established model. Verifier needs to have access to source data to verify the computations, but privacy can be preserved by running computations in [TEEs](https://en.wikipedia.org/wiki/Trusted_execution_environment). Full range of batch and streaming SQL is supported including complex streaming `JOIN`s that are essential for integrating data from multiple sources. |
| Based on Apache Datafusion engine with custom extensions for ZK-proof generation | Integrates a [wide range](/general/supported-engines) of major data processing engines |
| No built-in data transformation capabilities. Derivative data can only be created by extracting data from the system, transforming it, and writing it into a new table. Transformed data therefore will be disjoint from its origin. The model is **non-composable**. | Data transformation is a core part of the system. Processing pipelines can be built using streaming SQL allowing to create higher-level datasets that clean, aggregate, enrich, and integrate data. Provenance of derivative data is preserved. Pipelines are **infinitely composable**. |

### Can I use Pandas or other library to create derivative datasets?

`kamu`'s goal is to be as inclusive of different ways to process data as possible, but also **uphold certain key properties** of data pipelines like autonomy, low latency, and reproducibility.

Pandas and most other libraries do **batch processing**, so they are not suited for processing data that flows continuously. Using them would be inefficient, non-reproducible, and error prone, as batch paradigm does not handle late and out-of-order data, and other types of temporal problems we see in data constantly (see our [learning materials](/start/learning-materials) on the topic of streaming vs. batch).

Also Pandas and similar libraries are imperative and their code is hard to analyze - this complicates our goal of achieving **fine-grain provenance**. So we're currently biased towards processing engines that use Streaming SQL, but will be glad to discuss extending support to other frameworks.


# First Steps
Source: https://docs.kamu.dev/start/first-steps



Hi, and welcome to `kamu` - a **planet-scale data pipeline** that can run on your laptop!

`kamu` is an innovative project with unique design - it takes some time to wrap your head around. To help you in this journey we have outlined the recommended sequence of steps for you below.

You can always get help or share feedback along the way in our [community channels](/community).

## To get started

<Steps>
  <Step>
    Go through [Kamu 101](https://www.youtube.com/playlist?list=PLV91cS45lwVG20Hicztbv7hsjN6x69MJk) series of videos to get a taste of what `kamu` is about
  </Step>

  <Step>
    Take a self-guided tour in our [online tutorial](/start/tutorial)
  </Step>

  <Step>
    [Install](/cli/install) `kamu` on your laptop
  </Step>

  <Step>
    Get to know it through [quick start](/cli/quick-start) guide and try out the [advanced examples](/examples)
  </Step>

  <Step>
    Join our data science community on [Discord](https://discord.gg/nU6TXRQNXC)
  </Step>

  <Step>
    Learn about architecture and design choices behind our data formats and protocols from our [conference talks](/start/learning-materials)
  </Step>

  <Step>
    Do a deep dive into [Open Data Fabric](/odf) protocol specification
  </Step>
</Steps>


# Learning Materials
Source: https://docs.kamu.dev/start/learning-materials

Explainer videos, talks, publications and other learning materials

## Explainer Videos

| Name | Level | Description |
| - | :-: | - |
| [Kamu 101: Basics](https://www.youtube.com/watch?v=oUTiWW6W78A\&list=PLV91cS45lwVG20Hicztbv7hsjN6x69MJk) | Beginner | Quick overview of the purpose and basic features of the tool |
| [Kamu 101: Trust & Collaboration](https://www.youtube.com/watch?v=hN_vpHYmwi0\&list=PLV91cS45lwVG20Hicztbv7hsjN6x69MJk\&index=2) | Beginner | Theory of what makes data trustworthy and how to verify datasets you get from others |
| [Verifiable Multi-Party Analytics](https://www.youtube.com/watch?v=M7DyV-QUZbk\&list=PLV91cS45lwVG20Hicztbv7hsjN6x69MJk\&index=3) | Advanced | Demonstrates how to verifiably query IoT data from smart contracts |

## Talks

| Name | Level | Description |
| - | :-: | - |
| [Time: The most misunderstood dimension in data modelling](https://www.youtube.com/watch?v=XxKnTusccUM) | Beginner | Educational talk from **PyData Global 2021** about event-based data models and stream processing |
| [Structured Data in Web3 - Rethinking Databases](https://www.youtube.com/watch?v=ZQ-MdKj3BjU\&list=PL_1oLZF_wrbTeLmaYWadPmKCCxjF9zGt0) | Intermediate | **Protocol Labs: Compute Over Data Summit 2022** talk about structured data processing in Web3 world |
| [Supercharging SQL: Global Data Supply Chains for Verifiable AI & Analytics](https://www.youtube.com/watch?v=c9UCjJdvJAU) | Intermediate | **Filecoin Dev Summit 2024** talk about how bringing Web3 properties to stream processing impacts AI |
| [Oracle-Augmented Generation: Connecting AI to Verifiable Data](https://www.youtube.com/watch?v=RuQQNAgfOlY) | Intermediate | Proposes a new RAG-inspired method for data-centric AI agents |
| [From Scientific Data Publishing to Collaborative Data Economy](https://www.youtube.com/watch?v=d-Nha1sO0tM) | Beginner | Talk from **DeSci Berlin 2025** about making scientific data exchange sustainable and more collaborative |
| [Open Data Fabric for Research Data Management](https://www.youtube.com/watch?v=Ivh-YDDmRf8) | Beginner | Current state and problems of scientific data portals, motivation behind ODF and system's design principles |
| [Building a Distributed Collaborative Data Pipeline](https://databricks.com/session_eu20/building-a-distributed-collaborative-data-pipeline-with-apache-spark) | Intermediate | Technical talk from **Data+AI Summit 2020** |

## Publications

| Name | Level | Description |
| - | - | - |
| [Kamu Blog: Introducing Open Data Fabric](https://www.kamu.dev/blog/introducing-odf/) | Beginner | Initial announcement of the ODF project |
| [Whitepaper: Open Data Fabric](https://arxiv.org/abs/2111.06364) | Beginner | Original ODF protocol whitepaper |
| [Kamu Blog: The End of Batch Era](https://www.kamu.dev/blog/end-of-batch-era/) | Beginner | Why we are using stream processing instead of batch |
| [Course: Introduction to Multi-Party Data Engineering](https://www.kamu.dev/blog/2024-08-28-intro-to-web3-data-engineering/) | Beginner | How Web3 innovations can be applied to conventional data lakehouse architecture |
| [Next Decade of Data: From algebra to calculus, from silos to real-time supply chains](https://www.kamu.dev/blog/2024-06-27-next-decade-of-data/) | Intermediate | Analysis of the state of the data market and future predictions |
| [Open Data Fabric protocol specification](/odf/spec/overview) | Advanced | Specification itself, for tech-savvy readers |

## External Resources

| Name | Level | Description |
| - | - | - |
| [The World Beyond Batch: Streaming 101 by Tyler Akidau](https://www.oreilly.com/radar/the-world-beyond-batch-streaming-101/) | Beginner | Fundamental ideas behind stream (temporal) processing |
| [Foundations of streaming SQL by Tyler Akidau](https://youtu.be/UlPsp7LaA38?si=lZyAVZ6Gew5Mr-wl) | Beginner | A great visual way to develop intuition in streaming and bitemporal data |

See also:

* [FAQ](/start/faq)
* [First Steps](/start/first-steps)
* [Examples](/examples)


# Online Tutorial
Source: https://docs.kamu.dev/start/tutorial



This is a self-guided tour of `kamu` that will teach you the basics of Kamu CLI and Kamu Node and will teach you the core data science concepts behind it, without needing to install anything.

Be ready to dedicate \~30 minutes to it.

Get started at: [https://demo.kamu.dev](https://demo.kamu.dev)

<Warning>
  The capacity of this environment is limited and you may experience worse performance than running `kamu` on your machine.
</Warning>

<Note>
  Not ready to commit your time just yet?

  Then check out **Kamu 101** playlist to see what `kamu` is about:

  <YouTubeList />
</Note>


# Welcome to Kamu!
Source: https://docs.kamu.dev/start/welcome



<Card title="First time here?" icon="rocket" href="/start/first-steps">
  Follow these steps to get started
</Card>

## Projects

Our documentation is organized around four closely related projects:

<Columns>
  <Card>
    <h4><a href="/odf">Open Data Fabric</a></h4>

    <a href="/odf">
      <img />
    </a>

    ODF is an open protocol spec for exchange and verifiable multi-party processing of data. It's an independent open-source project. It is included on this website for convenience of cross-referencing with the rest of documentation.

    <Columns>
      <div><a href="/odf">> About</a></div>
      <div><a href="/odf/spec/overview">> Specification</a></div>
      <div><a href="/odf/schemas">> Schemas</a></div>
      <div><a href="/odf/rfcs">> RFCs</a></div>
    </Columns>
  </Card>

  <Card>
    <h4><a href="/cli">Kamu CLI</a></h4>

    <a href="/cli">
      <img />
    </a>

    A powerful command line tool that implements ODF protocol. You can run it on any device to build data pipelines, ingest and explore data, and interact with other nodes on ODF network.

    <Columns>
      <div><a href="/cli/install">> Install</a></div>
      <div><a href="/cli/quick-start">> Quick Start</a></div>
      <div><a href="/start/tutorial">> Tutorial</a></div>
      <div><a href="/examples">> Examples</a></div>
    </Columns>
  </Card>

  <Card>
    <h4><a href="/node">Kamu Node</a></h4>

    <a href="/node">
      <img />
    </a>

    A scalable server implementation of ODF. It's a set of Kubernetes applications that can be installed in a distributed environment to:

    * Operate data pipelines
    * Verify computations done by other parties
    * Execute queries on co-located data
    * Provide data via rich set of APIs to applications and smart contracts.

    <Columns>
      <div><a href="/node/quick-start">> Quick Start</a></div>
      <div><a href="/api">> APIs</a></div>
      <div><a href="/node/deploy">> Deploying</a></div>
      <div><a href="/node/commitments">> Operating</a></div>
    </Columns>
  </Card>

  <Card>
    <h4><a href="/platform">Kamu Web Platform</a></h4>

    <a href="/platform">
      <img />
    </a>

    A front-end application that acts as a window into the ODF network. Think of it as GitHub for data pipelines or Etherscan of ODF. It can be used in multiple setups: from exploring your local Kamu CLI workspace, to managing your distributed Kamu Node deployment, and to interacting with remote nodes in the global ODF network.

    <Columns>
      <div><a href="/platform">> Overview</a></div>
      <div><a href="/platform">> Get Started</a></div>
      <div><a href="/platform/admin">> Administration</a></div>
    </Columns>
  </Card>
</Columns>