ApiServerConfig
| Field | Type | Default | Description |
|---|---|---|---|
auth | AuthConfig | | Authentication & authorization |
database | DatabaseConfig | | Database |
secretsEncryption | SecretsEncryptionConfig | | Secrets encryption configuration |
engine | EngineConfig | | Ingest and transform engines |
protocol | ProtocolConfig | | Protocols |
runtime | RuntimeConfig | {} | Tokio runtime |
repo | RepoConfig | | Dataset repository |
uploadRepo | UploadRepoConfig | | File upload repository |
url | UrlConfig | | External URLs |
flowSystem | FlowSystemConfig | | Configuration for the flow system |
webhooks | WebhooksConfig | | Configuration for webhooks |
source | SourceConfig | | Ingestion’s sources |
outbox | OutboxAgentConfig | | Outbox agent configuration |
email | EmailConfig | | Email gateway configuration |
identity | IdentityConfig | {} | UNSTABLE: Identity configuration |
search | SearchConfig | | Search configuration |
quota | QuotaConfig | | Default quotas configured by type |
extra | ExtraConfig | | Experimental and temporary module configuration |
AuthConfig
| Field | Type | Default | Description |
|---|---|---|---|
jwtSecret | string | "" | |
providers | array | [] | |
didEncryption | DidSecretEncryptionConfig | | |
passwordPolicy | PasswordPolicyConfig | | |
allowAnonymous | boolean | true |
AuthProviderConfig
| Variants |
|---|
Github |
Password |
AuthProviderConfig::Github
| Field | Type | Default | Description |
|---|---|---|---|
clientId | string | ||
clientSecret | string | ||
kind | string |
AuthProviderConfig::Password
| Field | Type | Default | Description |
|---|---|---|---|
accounts | array | [] | |
kind | string |
AccountConfig
The declarative account configuration used to register an account if one
does not already exist.
To update an existing account, either id or private_key must be
specified.
| Field | Type | Default | Description |
|---|---|---|---|
id | AccountID | null | May be omitted in favor of private_key. |
privateKey | PrivateKey | null | Optional ed25519 private key. When set, id is derived from it
(and must match id if both are present). |
accountName | AccountName | ||
password | Password | ||
email | Email | ||
displayName | string | null | Auto-derived from account_name if omitted |
accountType | AccountType | ”User” | |
provider | string | ”password” | |
providerIdentityKey | string | null | Auto-derived from account_name if omitted |
avatarUrl | string | null | |
registeredAt | string | null | |
properties | array | [] | |
treatDatasetsAsPublic | boolean | false |
AccountID
Base type: string
PrivateKey
Base type: string
AccountName
Base type: string
Password
Base type: string
Email
Base type: string
AccountType
| Variants |
|---|
User |
Organization |
AccountPropertyName
| Variants |
|---|
CanProvisionAccounts |
Admin |
DidSecretEncryptionConfig
| Field | Type | Default | Description |
|---|---|---|---|
enabled | boolean | false | |
encryptionKey | string | null | The encryption key must be a 32-character alphanumeric string, which
includes both uppercase and lowercase Latin letters (A-Z, a-z) and
digits (0-9).To generate, use: |
PasswordPolicyConfig
| Field | Type | Default | Description |
|---|---|---|---|
minNewPasswordLength | integer | 8 |
DatabaseConfig
| Variants |
|---|
InMemory |
Sqlite |
Postgres |
DatabaseConfig::InMemory
| Field | Type | Default | Description |
|---|---|---|---|
provider | string |
DatabaseConfig::Sqlite
| Field | Type | Default | Description |
|---|---|---|---|
databasePath | string | ||
provider | string |
DatabaseConfig::Postgres
| Field | Type | Default | Description |
|---|---|---|---|
credentialsPolicy | DatabaseCredentialsPolicyConfig | ||
databaseName | string | ||
host | string | ||
port | integer | null | |
maxConnections | integer | null | |
maxLifetimeSecs | integer | null | |
acquireTimeoutSecs | integer | null | |
provider | string |
DatabaseCredentialsPolicyConfig
| Field | Type | Default | Description |
|---|---|---|---|
source | DatabaseCredentialSourceConfig | ||
rotationFrequencyInMinutes | integer | null |
DatabaseCredentialSourceConfig
| Variants |
|---|
RawPassword |
AwsSecret |
DatabaseCredentialSourceConfig::RawPassword
| Field | Type | Default | Description |
|---|---|---|---|
userName | string | ||
rawPassword | string | ||
kind | string |
DatabaseCredentialSourceConfig::AwsSecret
| Field | Type | Default | Description |
|---|---|---|---|
secretName | string | ||
kind | string |
SecretsEncryptionConfig
| Field | Type | Default | Description |
|---|---|---|---|
enabled | boolean | false | |
encryptionKey | string | null | Represents the encryption key for secrets. This field is required if
enabled is true or None.The encryption key must be a 32-character alphanumeric string, which
includes both uppercase and lowercase Latin letters (A-Z, a-z) and
digits (0-9).To generate use: |
EngineConfig
| Field | Type | Default | Description |
|---|---|---|---|
maxConcurrency | integer | null | Maximum number of engine operations that can be performed concurrently |
runtime | ContainerRuntimeType | ”Podman” | Type of the runtime to use when running the data processing engines |
networkNs | NetworkNamespaceType | ”Private” | Type of the networking namespace (relevant when running in container environments) |
startTimeout | DurationString | ”30s” | Timeout for starting an engine container |
shutdownTimeout | DurationString | ”5s” | Timeout for waiting the engine container to stop gracefully |
images | EngineImagesConfig | | UNSTABLE: Default engine images |
datafusionEmbedded | EngineConfigDatafusion | | Embedded Datafusion engine configuration |
ContainerRuntimeType
| Variants |
|---|
Docker |
Podman |
NetworkNamespaceType
Corresponds to podman’s containers.conf::netns
We podman is used inside containers (e.g. podman-in-docker or podman-in-k8s)
it usually runs uses host network namespace.
| Variants |
|---|
Private |
Host |
DurationString
Base type: string
EngineImagesConfig
| Field | Type | Default | Description |
|---|---|---|---|
spark | string | ”ghcr.io/kamu-data/engine-spark:0.23.1-spark_3.5.0” | UNSTABLE: Spark engine image |
flink | string | ”ghcr.io/kamu-data/engine-flink:0.18.2-flink_1.16.0-scala_2.12-java8” | UNSTABLE: Flink engine image |
datafusion | string | ”ghcr.io/kamu-data/engine-datafusion:0.9.0” | UNSTABLE: Datafusion engine image |
risingwave | string | ”ghcr.io/kamu-data/engine-risingwave:0.3.0” | UNSTABLE: RisingWave engine image |
EngineConfigDatafusion
| Field | Type | Default | Description |
|---|---|---|---|
base | object | | Base configuration options
See: <https://datafusion.apache.org/user-guide/configs.html> |
ingest | object | | Ingest-specific overrides to the base config |
batchQuery | object | {} | Batch query-specific overrides to the base config |
compaction | object | | Compaction-specific overrides to the base config |
useLegacyArrowBufferEncoding | boolean | false | Makes arrow batches use contiguous Binary and Utf8 encodings instead
of more modern BinaryView and Utf8View. This is only needed for
compatibility with some older libraries that don’t yet support them.See: kamu-node#277 |
ProtocolConfig
| Field | Type | Default | Description |
|---|---|---|---|
ipfs | IpfsConfig | | IPFS configuration |
flightSql | FlightSqlConfig | | FlightSQL configuration |
IpfsConfig
| Field | Type | Default | Description |
|---|---|---|---|
httpGateway | string | ”http://localhost:8080/” | HTTP Gateway URL to use for downloads.
For safety, it defaults to http://localhost:8080 - a local IPFS daemon.
If you don’t have IPFS installed, you can set this URL to
one of the public gateways like https://ipfs.io.
List of public gateways can be found here: https://ipfs.github.io/public-gateway-checker/ |
preResolveDnslink | boolean | true | Whether kamu should pre-resolve IPNS DNSLink names using DNS or leave it to the Gateway. |
FlightSqlConfig
| Field | Type | Default | Description |
|---|---|---|---|
allowAnonymous | boolean | true | Whether clients can authenticate as ‘anonymous’ user |
authedSessionExpirationTimeout | DurationString | ”30m” | Time after which FlightSQL client session will be forgotten and client will have to re-authroize (for authenticated clients) |
authedSessionInactivityTimeout | DurationString | ”5s” | Time after which FlightSQL session context will be released to free the resources (for authenticated clients) |
anonSessionExpirationTimeout | DurationString | ”5m” | Time after which FlightSQL client session will be forgotten and client will have to re-authroize (for anonymous clients) |
anonSessionInactivityTimeout | DurationString | ”5s” | Time after which FlightSQL session context will be released to free the resources (for anonymous clients) |
RuntimeConfig
| Field | Type | Default | Description |
|---|---|---|---|
workerThreads | integer | null | |
maxBlockingThreads | integer | null | |
threadStackSize | integer | null |
RepoConfig
| Field | Type | Default | Description |
|---|---|---|---|
repoUrl | UrlOrPath | null | |
caching | RepoCachingConfig | | |
dataBlocksPageSize | integer | null |
UrlOrPath
Base type: string
RepoCachingConfig
| Field | Type | Default | Description |
|---|---|---|---|
registryCacheEnabled | boolean | false | |
metadataLocalFsCachePath | string | null |
UploadRepoConfig
| Field | Type | Default | Description |
|---|---|---|---|
maxFileSizeMb | integer | 50 | |
storage | UploadRepoStorageConfig | |
UploadRepoStorageConfig
| Variants |
|---|
S3 |
Local |
UploadRepoStorageConfig::S3
| Field | Type | Default | Description |
|---|---|---|---|
bucketS3Url | string | ||
kind | string |
UploadRepoStorageConfig::Local
| Field | Type | Default | Description |
|---|---|---|---|
kind | string |
UrlConfig
| Field | Type | Default | Description |
|---|---|---|---|
baseUrlPlatform | UrlOrPath | ”http://localhost:4200/“ | |
baseUrlRest | UrlOrPath | ”http://localhost:8080/“ | |
baseUrlFlightsql | UrlOrPath | ”grpc://localhost:50050” |
FlowSystemConfig
| Field | Type | Default | Description |
|---|---|---|---|
flowAgent | FlowAgentConfig | | |
flowSystemEventAgent | FlowSystemEventAgentConfig | | |
taskAgent | TaskAgentConfig | |
FlowAgentConfig
| Field | Type | Default | Description |
|---|---|---|---|
awaitingStepSecs | integer | 1 | |
mandatoryThrottlingPeriodSecs | integer | 60 | |
defaultRetryPolicies | object | {} |
RetryPolicyConfig
| Field | Type | Default | Description |
|---|---|---|---|
maxAttempts | integer | null | |
minDelaySecs | integer | null | |
backoffType | RetryPolicyConfigBackoffType | null |
RetryPolicyConfigBackoffType
| Variants |
|---|
Fixed |
Linear |
Exponential |
ExponentialWithJitter |
FlowSystemEventAgentConfig
| Field | Type | Default | Description |
|---|---|---|---|
minDebounceInterval | DurationString | ”100ms” | |
maxListeningTimeout | DurationString | ”2m” | |
batchSize | integer | 100 |
TaskAgentConfig
| Field | Type | Default | Description |
|---|---|---|---|
taskCheckingIntervalSecs | integer | 1 |
WebhooksConfig
| Field | Type | Default | Description |
|---|---|---|---|
maxConsecutiveFailures | integer | 5 | |
deliveryTimeoutSecs | integer | 10 | |
secretEncryptionEnabled | boolean | false | |
secretEncryptionKey | string | null | Represents the encryption key for the webhooks secret. This field is
required if secret_encryption_enabled is true or None.The encryption key must be a 32-character alphanumeric string, which
includes both uppercase and lowercase Latin letters (A-Z, a-z) and
digits (0-9).Examplelet config = WebhooksConfig { … secret_encryption_enabled: Some(true), encryption_key: Some(String::from(“aBcDeFgHiJkLmNoPqRsTuVwXyZ012345”)) }; ``` |
SourceConfig
| Field | Type | Default | Description |
|---|---|---|---|
targetRecordsPerSlice | integer | 10000 | Target number of records after which we will stop consuming from the resumable source and commit data, leaving the rest for the next iteration. This ensures that one data slice doesn’t become too big. |
mqtt | MqttSourceConfig | | MQTT-specific configuration |
ethereum | EthereumSourceConfig | | Ethereum-specific configuration |
MqttSourceConfig
| Field | Type | Default | Description |
|---|---|---|---|
brokerIdleTimeoutMs | integer | 1000 | Time in milliseconds to wait for MQTT broker to send us some data after which we will consider that we have “caught up” and end the polling loop. |
EthereumSourceConfig
| Field | Type | Default | Description |
|---|---|---|---|
rpcEndpoints | array | [] | Default RPC endpoints to use if source does not specify one explicitly. |
getLogsBlockStride | integer | 100000 | Default number of blocks to scan within one query to eth_getLogs RPC
endpoint. |
commitAfterBlocksScanned | integer | 1000000 | Forces iteration to stop after the specified number of blocks were scanned even if we didn’t reach the target record number. This is useful to not lose a lot of scanning progress in case of an RPC error. |
useBlockTimestampFallback | boolean | false | Many providers don’t yet return blockTimestamp from eth_getLogs RPC
endpoint and in such cases block_timestamp column will be null.
If you enable this fallback the library will perform additional call to
eth_getBlock to populate the timestam, but this may result in
significant performance penalty when fetching many log records.See: ethereum/execution-apis#295 |
EthRpcEndpoint
| Field | Type | Default | Description |
|---|---|---|---|
chainId | integer | ||
chainName | string | ||
nodeUrl | string |
OutboxAgentConfig
| Field | Type | Default | Description |
|---|---|---|---|
minDebounceInterval | DurationString | ”100ms” | |
maxListeningTimeout | DurationString | ”2m” | |
batchSize | integer | 100 |
EmailConfig
| Field | Type | Default | Description |
|---|---|---|---|
senderAddress | string | ||
senderName | string | null | |
gateway | EmailConfigGateway |
EmailConfigGateway
| Variants |
|---|
Dummy |
Postmark |
EmailConfigGateway::Dummy
| Field | Type | Default | Description |
|---|---|---|---|
kind | string |
EmailConfigGateway::Postmark
| Field | Type | Default | Description |
|---|---|---|---|
apiKey | string | ||
kind | string |
IdentityConfig
Private keys are used to sign API responses.
Supported algorithms: ed25519, secp256k1.
| Field | Type | Default | Description |
|---|---|---|---|
ed25519PrivateKey | PrivateKey | null | Root private key that corresponds to the authority and is used to sign
responses.To generate, use: |
secp256k1PrivateKey | Secp256k1Signer | null | Secp256k1 private key used to sign EIP-712 typed data.To generate, use: |
Secp256k1Signer
Base type: string
SearchConfig
| Field | Type | Default | Description |
|---|---|---|---|
indexer | SearchIndexerConfig | | Indexer configuration |
embeddingsChunker | EmbeddingsChunkerConfig | | Embeddings chunker configuration |
embeddingsEncoder | EmbeddingsEncoderConfig | | Embeddings encoder configuration |
repo | SearchRepositoryConfig | | Search repository configuration |
semanticSearchThresholdScore | number | 0.0 |
SearchIndexerConfig
| Field | Type | Default | Description |
|---|---|---|---|
incrementalIndexing | boolean | false | Whether incremental indexing is enabled |
clearOnStart | boolean | false |
EmbeddingsChunkerConfig
| Variants |
|---|
Simple |
EmbeddingsChunkerConfig::Simple
| Field | Type | Default | Description |
|---|---|---|---|
splitSections | boolean | false | |
splitParagraphs | boolean | false | |
kind | string |
EmbeddingsEncoderConfig
| Variants |
|---|
OpenAi |
Dummy |
EmbeddingsEncoderConfig::OpenAi
| Field | Type | Default | Description |
|---|---|---|---|
url | string | null | |
apiKey | string | null | |
modelName | string | ”text-embedding-ada-002” | |
dimensions | integer | 1536 | |
kind | string |
EmbeddingsEncoderConfig::Dummy
| Field | Type | Default | Description |
|---|---|---|---|
kind | string |
SearchRepositoryConfig
| Variants |
|---|
Dummy |
Elasticsearch |
SearchRepositoryConfig::Dummy
| Field | Type | Default | Description |
|---|---|---|---|
kind | string |
SearchRepositoryConfig::Elasticsearch
| Field | Type | Default | Description |
|---|---|---|---|
url | string | ”http://localhost:9200” | |
password | string | null | |
caCertPemPath | string | null | |
indexPrefix | string | ”kamu-node” | |
timeoutSecs | integer | 30 | |
enableCompression | boolean | false | |
embeddingDimensions | integer | 1536 | |
kind | string |
QuotaConfig
| Field | Type | Default | Description |
|---|---|---|---|
account | QuotaAccountConfig | {} |
QuotaAccountConfig
| Field | Type | Default | Description |
|---|---|---|---|
defaultStorageLimitInBytes | integer | null |
ExtraConfig
| Field | Type | Default | Description |
|---|---|---|---|
graphql | GqlConfig | {} |
GqlConfig
| Field | Type | Default | Description |
|---|