Skip to main content
PR Start Date: 2026-10-10 Published Date: 2026-10-22 Authors: Compatibility:
  • Backwards-compatible
  • Forwards-compatible

​
Summary

This RFC proposes a new set of resources to define and control workload scheduling and execution within ODF nodes.

​
Table of Contents

​
Current Prototype Implementation in Kamu

Kamu has been the testing bed for the initial implementation of the task and flow systems. Current design is presented in Appendix A. The current prototype system has a few design issues:
  • Flows are closely coupled with datasets - we’ll need them to work with many other types of resources
  • It exposes an ever growing GQL API surface that has to be updated when adding new flow types
  • It is not extensible and would not allow custom plug-in flow types
  • Flows consist of one workload only and cannot chain multiple steps (e.g. compact then GC)

​
Proposed IaC-based System

We propose to define a new Tasks & Flows system on the core ODF level that builds on the Resource Framework and allows defining and scheduling workloads in a generic, extensible way.

​
Task

Building in a bottom-up order, we define the Task resource. Task represents a single unit of work, from intent through planning and execution to commit. Task resource spec captures the intent: the target resource and the operation kind with high-level parameters. This is what a human operator or a FlowRun controller writes when creating a task. Spec is stable, human-readable, and never rewritten. The execution plan - with fully resolved paths, offsets, schemas, and all other inputs needed by the worker - is computed by the node’s planner and written into TaskPlan status condition. The TaskStatus lifecycle proceeds through the following phases:
  • Pending — task created, waiting for the planner
  • Planning — node planner is resolving the full execution plan into TaskPlan
  • Ready — TaskPlan is populated; task is queued for a worker
  • Running — worker is executing the plan
  • Committing — worker reported its result; node is validating output and writing the metadata block
  • Finished — terminal; TaskOutcome condition is either Success, Failed, NoOp or Cancelled; the resource enters a TTL period before deletion.
Task resources are retained for a configurable TTL period after completion (e.g. 10 days) before being deleted. This allows the UI and operators to inspect recent runs directly from the resource store without querying the event store. After TTL expires, the resource is deleted, but the full history of any task remains recoverable from the event sourcing store by resource ID. Note that the separation of planning, execution, and commit phase allows to run certain parts of the task on the node (with access to metadata and storage state), and heavy computational tasks on a separate worker. Example of a task as created by a FlowRun controller (spec describes only the intent, no plan yet):
Example of the same task after planning and execution:
Note that a task may finish with a NoOp outcome and no TaskPlan if planner realizes there is nothing to do (e.g. no new data to process).

​
FlowRun

FlowRun resources are a set of tasks to be executed in a sequence. Example:
The spec.target on the FlowRun level is used as the default target for tasks in spec.tasks list to avoid duplication.
Note: Although retryOf and activationCauses are immutable and known at FlowRun creation, they are part of status rather than spec because they carry information that can only be reliably set by the controller, not by a user.

​
Flow

Flow resources act as templates for instantiating FlowRuns and define triggers that decide when to instantiate them. Example:
The spec.target on the Flow level is used as the default target for triggers in spec.triggers list and tasks in spec.tasks list to avoid duplication while still allowing overrides (e.g. triggering a flow on dataset A based on events in dataset B).

​
Target Selector

When a Flow’s target selector matches a resource, the flow controller sets up triggers for that resource automatically. As resources are created or deleted, the controller subscribes to those events and updates its set of active targets accordingly. The association between a Flow and its matched resources is purely a derived state that the controller maintains. The FlowStatus.recentBindigs may reflect last N bindings that were created for debugging purposes. To show the full paginated list of bindings, the flow controller may provide a dedicated API for querying the index, e.g. in REST:
  • GET /flow/v1alpha1/flow/_/inverseSearch?target={ResourceRef} - list flows by target
  • GET /flow/v1alpha1/flow/{id}/targets - list targets by flow

​
FlowTrigger

Flows specify a set of triggers that decide when to instantiate a FlowRun. Examples:
The following properties are common across all trigger types:

​
Appendix A: Flow System Prototype in Kamu

The design is roughly captured by the following GQL types:
Last modified on September 23, 2026