Skip to main content
PR Start Date: 2026-09-10 Published Date: 2026-09-22 Authors: Compatibility:
  • Backwards-compatible
  • Forwards-compatible

​
Summary

This RFC proposes a new set of resources to define and control workload scheduling and execution within ODF nodes.

​
Table of Contents

​
Current Prototype Implementation in Kamu

Kamu has been the testing bed for the initial implementation of the task and flow systems. Current design is presented in Appendix A. The current prototype system has a few design issues:
  • Flows are closely coupled with datasets - we’ll need them to work with many other types of resources
  • It exposes an ever growing GQL API surface that has to be updated when adding new flow types
  • It is not extensible and would not allow custom plug-in flow types
  • Flows consist of one workload only and cannot chain multiple steps (e.g. compact then GC)

​
Proposed IaC-based System

We propose to define a new Tasks & Flows system on the core ODF level that builds on the Resource Framework and allows defining and scheduling workloads in a generic, extensible way.

​
Task

Building in a bottom-up order, we define the Task resource. Task represents a single unit of work, from intent through execution to its outcome. Task resource spec captures the intent: the target resource and the operation kind with high-level parameters. This is what a human operator or a FlowRun controller writes when creating a task. Spec is stable, human-readable, and never rewritten. Tasks are executed by workers. Every worker specializes in one task type and pulls the next task of that type from the node’s queue, which schedules and prioritizes tasks. The worker then advances the task through the steps its type requires - e.g. fetching data for an ingest, or elaborating the execution plan of a transform, requesting an engine and dispatching the plan to it - and commits the result. As it progresses, the worker records the state of the task in type-specific status conditions. For example, the execution plan - with fully resolved paths, offsets, schemas, and all other inputs - is written into the TaskPlan condition. The TaskStatus lifecycle proceeds through the following phases:
  • Pending — task created; node is validating its inputs and resolving what is needed to schedule it (e.g. the engine a transform uses)
  • Queued — task is waiting in the queue for a worker
  • Running — a worker has claimed the task and is executing it, including the commit
  • Finished — terminal; TaskOutcome condition is either Success, Failed, NoOp or Cancelled, and the task resource is deleted (see below).
The resource phase reflects the work of the task controller, whose job is to admit the task: validate its inputs, resolve what is needed to schedule it, and put it into the queue. phase: Ready means the task was admitted, while phase: Failed means it was rejected - in which case the controller also sets an unrecoverable Failed outcome. Execution is reported separately by workers in the TaskStatus condition, so an admitted task stays in phase: Ready whatever its TaskStatus is, until it is deleted. Example of a task as created by a FlowRun controller (spec describes only the intent, no plan yet):
Task resources are deleted as soon as they finish, so the existing tasks are exactly the work that is pending, queued, or running. Deleted tasks stay readable for a configurable retention period (e.g. 10 days) before being purged. This allows the UI and operators to inspect recent runs. Example of the same task after planning and execution:
Note that a task may finish with a NoOp outcome and no TaskPlan if the worker realizes there is nothing to do (e.g. no new data to process). A Failed outcome describes the error and tells whether it is recoverable, i.e. whether retrying the task could succeed. Recoverable failures (e.g. a network error) can be retried by the flow according to its retryPolicy, while unrecoverable ones (e.g. an invalid query) would fail the same way again. Type-specific details of the error are carried in error:
Deleting a task that has not finished yet cancels it when possible. Until its outcome is recorded, a task being deleted remains visible in phase: Deleting.

​
Task Types

The following task types will be initially supported. Ingest - fetches data from a source and appends it to a dataset.
Transform - executes transformation of data defined in a derivative dataset.
SyncFrom - pulls new blocks from a remote dataset into a local dataset.
SyncTo - pushes new blocks of a local dataset to a remote dataset.
Sync endpoints support the following auth kinds:
  • Bearer - passes a token, e.g. an access token of a remote ODF node
  • Aws - credentials for AWS or an AWS-compatible object storage
  • Headers - custom request headers with values referencing VariableSets and SecretSets
Note that the execution plan of a sync task carries references to secrets and never their values - these are resolved only during execution. Compaction - compacts data files in matching datasets to improve query performance.
Reset - moves a block reference of a dataset to an earlier block, discarding the history that follows it.
GarbageCollection - removes unreferenced data files from matching datasets.
Verify - checks dataset metadata for integrity, optionally replaying transformations.
WebhookCall - dispatches a payload to a WebhookEndpoint resource.
Custom task types can also be specified using a full schema URI as the kind:

​
FlowRun

FlowRun resources are a set of tasks to be executed in a sequence. Example:
The spec.target on the FlowRun level is used as the default target for tasks in spec.tasks list to avoid duplication. Like tasks, FlowRun resources are deleted as soon as the run finishes and stay readable for the retention period. Deleting a run that has not finished yet cancels it, together with its unfinished tasks, which reference the run in ownerReferences. A run created by the flow controller copies the serviceAccount of its Flow into its own spec, so changes to the flow affect only future runs, never runs in progress.
Note: Although retryOf and activationCauses are immutable and known at FlowRun creation, they are part of status rather than spec because they carry information that can only be reliably set by the controller, not by a user.

​
Flow

Flow resources act as templates for instantiating FlowRuns and define triggers that decide when to instantiate them. Example:
The spec.target on the Flow level is used as the default target for triggers in spec.triggers list and tasks in spec.tasks list to avoid duplication while still allowing overrides (e.g. triggering a flow on dataset A based on events in dataset B).

​
Target Selector

When a Flow’s target selector matches a resource, the flow controller sets up triggers for that resource automatically. As resources are created or deleted, the controller subscribes to those events and updates its set of active targets accordingly. The association between a Flow and its matched resources is purely a derived state that the controller maintains. The FlowStatus.recentBindigs may reflect last N bindings that were created for debugging purposes. To show the full paginated list of bindings, the flow controller may provide a dedicated API for querying the index, e.g. in REST:
  • GET /flow/v1alpha1/flow/_/inverseSearch?target={ResourceRef} - list flows by target
  • GET /flow/v1alpha1/flow/{id}/targets - list targets by flow

​
FlowTrigger

Flows specify a set of triggers that decide when to instantiate a FlowRun. FlowRun stores the trigger configuration that lead to its creation in activationCauses. Core trigger types: Manual trigger - reacts to an API call or a UI button press.
Schedule trigger - fires on specified Cron schedule. Guaranteed to fire if node was down when the next tick was supposed to happen. Fires only once for all missed ticks.
Interval trigger - fires at regular intervals. Guaranteed to fire if node was down when the next tick was supposed to happen. Fires only once for all missed ticks.
Event trigger - reacts to events on the event bus. This is a very low-level trigger that should be used sparingly.
InputsUpdated trigger - fires when inputs of a derivative dataset have updates.
SourceUpdated trigger - fires when specified source is updated.
Besides triggers defined in ODF spec custom triggers can also be specified.
The following properties are common across all trigger types:

​
Authorization

A task executes with the permissions of a user principal who created it. When a task, flow run, or a flow is defined for an organization they must specify serviceAccount property which defines a non-human principal that gets permissions only through explicit policy bindings. A task created from a FlowRun inherits the serviceAccount of the run, and FlowRun inherits one from Flow. Example:
Details of the authorization mechanism are outside of the scope of this RFC.

​
Appendix A: Flow System Prototype in Kamu

The design is roughly captured by the following GQL types:
Last modified on September 23, 2026