> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kamu.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Engine Contract

**Domain**: Transformations\
**Depends on**: [Data Model](https://docs.kamu.dev/odf/spec/datasets/data-model.md)\
**RFCs**: [RFC-015](https://docs.kamu.dev/odf/rfcs/015-unified-changelog-stream-schema.md)

## Abstract

This spec defines the contract between a [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) and an [Engine](https://docs.kamu.dev/general/glossary.md#engine) - the sandboxed execution environment of engines, the communication interface and the operations engines must support, and how engine versions are pinned and deprecated to preserve reproducibility of transformations.

## Execution Model

[Engine](https://docs.kamu.dev/general/glossary.md#engine) executes in a very restricted environment (a "sandbox") that prevents it from accessing any external resources other than those explicitly provided by the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator). Isolating the [Engine](https://docs.kamu.dev/general/glossary.md#engine) from making network calls and accessing any random files ensures both that the [Data](https://docs.kamu.dev/general/glossary.md#data) being processed cannot be leaked, and that non-deterministic behavior is harder to introduce accidentally.

Our isolation technology of choice is  [OCI containers](https://opencontainers.org/) - a lightweight mechanism that provides good performance and strong isolation guarantees. We rely on OCI-compatible images as the distribution mechanism for the engines.

> **WIP:** Add exact `podman` and `docker` commands to reproduce the sandboxed environment

## Communication Interface

This section describes the [RPC](https://en.wikipedia.org/wiki/Remote_procedure_call) mechanism used in communications between the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) and the [Engine](https://docs.kamu.dev/general/glossary.md#engine).

The requirements we put towards the RPC mechanism are:

* Use common network protocols
* Wide support in many programming languages
* Support streaming responses (e.g. for streaming back the operation status and logs)

Our RPC technology of choice is [gRPC](https://grpc.io/) because:

* It's cross-platform, mature, and available in many languages
* Natively supports [FlatBuffers](https://google.github.io/flatbuffers/flatbuffers_grpc_guide_use_cpp.html) which is our [metadata format](https://docs.kamu.dev/odf/spec/datasets/metadata-chain.md#metadata-format) of choice, letting us reuse object schemas
* Supports streaming
* For our narrow use case, its convenience is more important than interoperability, as we don't expect to have many server and client implementations

[Engine](https://docs.kamu.dev/general/glossary.md#engine) implementations must expose the `gRPC` API on port `5544`.

Request and response messages are defined by the [message schemas](#message-schemas) and serialized using FlatBuffers. Every operation responds with a stream of messages, where zero or more `Progress` messages are followed by exactly one terminal message:

* `Success` - operation completed successfully
* `InvalidQuery` - the query did not pass validation, with a `message` explaining the error
* `InternalError` - an unexpected error occurred, with a `message` and an optional `backtrace` with details

> Note: When starting an [Engine](https://docs.kamu.dev/general/glossary.md#engine) container, the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) will consider it fully initialized when it can establish a TCP connection to the port `5544`.

### Data Exchange

The exchange of raw data happens out-of-band of the `gRPC` API.

Currently, data and checkpoints are exchanged via files and object stores. The [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) makes input data part files and the previous [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint) available to the [Engine](https://docs.kamu.dev/general/glossary.md#engine) and passes their paths in the request, along with the paths where the [Engine](https://docs.kamu.dev/general/glossary.md#engine) must write the output data and the new [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint).

In future, input and output [Data Slices](https://docs.kamu.dev/general/glossary.md#data-slice) may be also exchanged using memory-mapped files (`mmap`) containing data in the common [in-memory format](https://docs.kamu.dev/odf/spec/datasets/data-model.md#in-memory-and-on-disk-formats). This avoids any intermediate IO and minimizes the copying of data for the best performance.

## Operations

Engine implementation should support the following operations:

* [Execute query](#execute-query) - Performs the next iteration of the transformation.
* [Raw query](#raw-query) - Executes a query over raw data, e.g. during preprocessing of data in ingest.
* [Validate query](#validate-query) - Validates the user-specified query for basic syntax and schema correctness.
* [Migrate query](#migrate-query) - Updates the transformation state from one query to another.
* [Derive Provenance](#derive-provenance) - Explains the origin of some data produced in the past.

> **WIP:** Validate query, Migrate query, and Derive Provenance operations do not yet have defined [message schemas](#message-schemas).

### Execute Query

This operation is used by the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) to perform the next iteration of the transformation.

The [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) is designed to isolate the [Engines](https://docs.kamu.dev/general/glossary.md#engine) from the complexity of managing the [Metadata Chain](https://docs.kamu.dev/general/glossary.md#metadata-chain) and make processing look as close as possible to a regular stream processing.

![Sequence Diagram: Execute Query](https://docs.kamu.dev/images/odf/specs/transformations/engine-contract/engine_transform.svg)
Sequence Diagram: Execute Query

Inputs (`TransformRequest`):

* ID and alias of the output [Dataset](https://docs.kamu.dev/general/glossary.md#dataset)
* [System Time](https://docs.kamu.dev/general/glossary.md#system-time) to assign to new records
* Offset to assign to the first new record
* Vocabulary of the output [Dataset](https://docs.kamu.dev/general/glossary.md#dataset) - mapping of system columns onto its schema
* Transformation [Query](https://docs.kamu.dev/general/glossary.md#query)
* Inputs - for every input [Dataset](https://docs.kamu.dev/general/glossary.md#dataset):
  * ID and alias of the dataset
  * Alias under which the input is referred to in the [Query](https://docs.kamu.dev/general/glossary.md#query)
  * Vocabulary of the input dataset
  * Offset interval of the input [Data Slice](https://docs.kamu.dev/general/glossary.md#data-slice) that goes into this transaction
  * Schema and data files of the input
  * Explicit [Watermarks](https://docs.kamu.dev/general/glossary.md#watermark) that should be injected into the stream to separate micro-batches for reproducibility
* Previous [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint)
* Locations where new data and the new [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint) should be written

Outputs (`TransformResponse`):

* Offset interval of the output [Data Slice](https://docs.kamu.dev/general/glossary.md#data-slice), if any data was produced
* Output [Watermark](https://docs.kamu.dev/general/glossary.md#watermark), if it was advanced
* New [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint)
* Operation progress and status

This operation must be [idempotent](https://en.wikipedia.org/wiki/Idempotence).

Input and output data are changelog streams that can contain [retractions and corrections](https://docs.kamu.dev/general/glossary.md#retractions-and-corrections) (see [Processing Model: Retractions and Corrections](https://docs.kamu.dev/odf/spec/transformations/processing-model.md#retractions-and-corrections)). The [Engine](https://docs.kamu.dev/general/glossary.md#engine) must:

* Consume input records in the order of their [Offsets](https://docs.kamu.dev/general/glossary.md#offset), as the order of retractions and corrections relative to the events they apply to is significant
* Take the operation type of input records into account and propagate retractions and corrections into the output
* Produce output that is a valid changelog stream (see [Data Model](https://docs.kamu.dev/odf/spec/datasets/data-model.md#representation-of-retractions-and-corrections)) - in particular, it must not reorder records in a way that separates a correction pair or places a retraction before the event it retracts, and must write both events of a correction pair into the same output [Data Slice](https://docs.kamu.dev/general/glossary.md#data-slice)

Engines that process data in batches and parallelize work must preserve the order of records (e.g. by coalescing inputs into one partition). Such engines that only support simple record-by-record operations (like `map` and `filter`) must propagate the operation type column as-is, so that operations apply identically to appends, retractions, and corrections and produce a valid changelog stream.

When an operation is committed by the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator), the [Engine](https://docs.kamu.dev/general/glossary.md#engine) will not see the same input data again. If due to the nature of the query (e.g. windowing or watermarking configuration) [Engine](https://docs.kamu.dev/general/glossary.md#engine) cannot fully process and discard the input data - it should use the [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint) to buffer it.

### Raw Query

This operation is used by the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) to execute a [Query](https://docs.kamu.dev/general/glossary.md#query) over raw data that is not yet part of any dataset, e.g. during the preprocessing step of [Ingestion](https://docs.kamu.dev/general/glossary.md#ingestion).

Inputs (`RawQueryRequest`):

* Input data files to perform the query over - all files must have identical schema
* Transformation [Query](https://docs.kamu.dev/general/glossary.md#query)
* Location where the query result should be written

Outputs (`RawQueryResponse`):

* Number of records produced by the query
* Operation progress and status

### Validate Query

This operation may be used by the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) when creating a new [Derivative Dataset](https://docs.kamu.dev/general/glossary.md#derivative-dataset) or when changing the [Query](https://docs.kamu.dev/general/glossary.md#query) of an existing one to validate the user-specified query for basic syntax and schema correctness.

Inputs:

* [Schemas](https://docs.kamu.dev/general/glossary.md#schema) of the input [Datasets](https://docs.kamu.dev/general/glossary.md#dataset)
* [Query](https://docs.kamu.dev/general/glossary.md#query)
* (optional) Previous [Query](https://docs.kamu.dev/general/glossary.md#query) in case of an existing [Dataset](https://docs.kamu.dev/general/glossary.md#dataset) modification

Outputs:

* [Schema](https://docs.kamu.dev/general/glossary.md#schema) of the result
* (alternatively) Validation errors

### Migrate Query

This operation is used by the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) when data processing hits the point where one transformation [Query](https://docs.kamu.dev/general/glossary.md#query) is being replaced by another one. It gives the [Engine](https://docs.kamu.dev/general/glossary.md#engine) all information needed to handle the transition as gracefully as possible, e.g. by reconciling the existing [Checkpoints](https://docs.kamu.dev/general/glossary.md#checkpoint) with the new [Query](https://docs.kamu.dev/general/glossary.md#query), or, at the very least, finalizing the processing of the old [Query](https://docs.kamu.dev/general/glossary.md#query) and clearing the [Checkpoints](https://docs.kamu.dev/general/glossary.md#checkpoint) before the new one starts to execute.

Inputs:

* Transaction ID
* Previous [Query](https://docs.kamu.dev/general/glossary.md#query)
* Next [Query](https://docs.kamu.dev/general/glossary.md#query)
* Previous [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint)

Outputs:

* New [Checkpoint](https://docs.kamu.dev/general/glossary.md#checkpoint)
* (optional) [Data](https://docs.kamu.dev/general/glossary.md#data) that has been produced when finalizing the old [Query](https://docs.kamu.dev/general/glossary.md#query)

### Derive Provenance

This operation is used to trace back a set of events in the output [Dataset](https://docs.kamu.dev/general/glossary.md#dataset) to input events that contributed to their values or their existence (see [Provenance](https://docs.kamu.dev/general/glossary.md#provenance)).

> **WIP:** The design of this operation is in progress.

## Engine Versioning

As described in [Components of Trust](https://docs.kamu.dev/odf/spec/overview#components-of-trust), to guarantee the reproducibility and verifiability of the results a transformation must be associated with an exact version of an [Engine](https://docs.kamu.dev/general/glossary.md#engine) that is used to perform it. We want to exclude any possibility that the code changes in the [Engine](https://docs.kamu.dev/general/glossary.md#engine) will break this guarantee.

Whenever the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) uses an [Engine](https://docs.kamu.dev/general/glossary.md#engine) to execute a query it must specify the full digest of the OCI container image in the resulting [Metadata Block](https://docs.kamu.dev/general/glossary.md#metadata-chain). [Engine](https://docs.kamu.dev/general/glossary.md#engine) maintainers are therefore responsible for treating the images as immutable and ensure old versions are never lost.

See also:

* [Engine Deprecation](#engine-deprecation)

## Engine Deprecation

Pinning the exact [Engine](https://docs.kamu.dev/general/glossary.md#engine) version used by every transformation guarantees the reproducibility and verifiability of the results (see [Components of Trust](https://docs.kamu.dev/odf/spec/overview#components-of-trust)). In the long run, however, this creates a problem where in order to [verify a dataset](https://docs.kamu.dev/odf/spec/transformations/processing-model.md#verification) the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) will have to use a very large number of different versions of an [Engine](https://docs.kamu.dev/general/glossary.md#engine) that might've accumulated over the years. This slows down the verification procedure and can result in a significant disk space required for storing these images.

To mitigate this problem the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) offers the **engine deprecation** operation that works as follows:

1. The [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) reconstructs the dataset from scratch by repeating all transformation in the [Metadata Chain](https://docs.kamu.dev/general/glossary.md#metadata-chain).
2. Instead of the [Engine](https://docs.kamu.dev/general/glossary.md#engine) version specified in the block, it uses the latest (or provided) version.
3. If the results produced in every step match the recorded ones - we can safely consider that the newer version of the [Engine](https://docs.kamu.dev/general/glossary.md#engine) is fully compatible with the older one.
4. A special [Metadata](https://docs.kamu.dev/general/glossary.md#metadata-chain) record is created to record this compatibility, therefore, allowing all users to use the latest version of the [Engine](https://docs.kamu.dev/general/glossary.md#engine) instead of downloading all the versions seen previously.

## Message Schemas

Messages exchanged between the [Coordinator](https://docs.kamu.dev/general/glossary.md#coordinator) and the [Engine](https://docs.kamu.dev/general/glossary.md#engine) are defined by the following schemas:

| Schema | Description |
| - | - |
| [TransformRequest](https://docs.kamu.dev/odf/schemas.md#transformrequest) | Sent by the coordinator to perform the next step of data transformation |
| [TransformRequestInput](https://docs.kamu.dev/odf/schemas.md#transformrequestinput) | Describes an input of the transform request |
| [TransformResponse](https://docs.kamu.dev/odf/schemas.md#transformresponse) | Sent by the engine when performing the data transformation |
| [RawQueryRequest](https://docs.kamu.dev/odf/schemas.md#rawqueryrequest) | Sent by the coordinator to perform a query on raw input data |
| [RawQueryResponse](https://docs.kamu.dev/odf/schemas.md#rawqueryresponse) | Sent by the engine when performing the raw query |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.