{
  "allowAnonymous": true,
  "users": {
    "predefined": []
  }
}

{
  "enabled": false
}

{
  "enabled": false
}

{
  "datafusionEmbedded": {
    "base": {
      "datafusion.catalog.default_catalog": "kamu",
      "datafusion.catalog.default_schema": "kamu",
      "datafusion.catalog.information_schema": "true",
      "datafusion.sql_parser.enable_ident_normalization": "false"
    },
    "batchQuery": {},
    "compaction": {
      "datafusion.execution.target_partitions": "1"
    },
    "ingest": {
      "datafusion.execution.target_partitions": "1"
    },
    "useLegacyArrowBufferEncoding": false
  },
  "images": {
    "datafusion": "ghcr.io/kamu-data/engine-datafusion:0.9.0",
    "flink": "ghcr.io/kamu-data/engine-flink:0.18.2-flink_1.16.0-scala_2.12-java8",
    "risingwave": "ghcr.io/kamu-data/engine-risingwave:0.2.0-risingwave_1.7.0-alpha",
    "spark": "ghcr.io/kamu-data/engine-spark:0.23.1-spark_3.5.0"
  },
  "networkNs": "Private",
  "runtime": "Docker",
  "shutdownTimeout": "5s",
  "startTimeout": "30s"
}

{
  "graphql": {}
}

{
  "flowAgent": {
    "awaitingStepSecs": 1,
    "defaultRetryPolicies": {},
    "mandatoryThrottlingPeriodSecs": 60
  },
  "flowSystemEventAgent": {
    "batchSize": 20,
    "maxListeningTimeoutMs": 2000,
    "minDebounceIntervalMs": 100
  },
  "taskAgent": {
    "checkingIntervalSecs": 1
  }
}

{
  "jupyter": {
    "image": "ghcr.io/kamu-data/jupyter:0.7.1",
    "livyImage": "ghcr.io/kamu-data/engine-spark:0.23.1-spark_3.5.0"
  }
}

{
  "awaitingStepSecs": 1,
  "batchSize": 20
}

{
  "flightSql": {
    "allowAnonymous": true,
    "anonSessionExpirationTimeout": "30m",
    "anonSessionInactivityTimeout": "5s",
    "authedSessionExpirationTimeout": "30m",
    "authedSessionInactivityTimeout": "5s"
  },
  "ipfs": {
    "httpGateway": "http://localhost:8080/",
    "preResolveDnslink": true
  }
}

{
  "storage": 1000000000
}

{
  "embeddingsChunker": {
    "kind": "Simple",
    "splitParagraphs": false,
    "splitSections": false
  },
  "embeddingsEncoder": {
    "kind": "Dummy"
  },
  "indexer": {
    "clearOnStart": false,
    "incrementalIndexing": true
  },
  "repo": {
    "kind": "Dummy"
  }
}

{
  "ethereum": {
    "commitAfterBlocksScanned": 1000000,
    "getLogsBlockStride": 100000,
    "rpcEndpoints": [],
    "useBlockTimestampFallback": false
  },
  "http": {
    "connectTimeout": "30s",
    "maxRedirects": 10,
    "userAgent": "kamu-cli/0.260.1"
  },
  "mqtt": {
    "brokerIdleTimeout": "1s"
  },
  "targetRecordsPerSlice": 10000
}

{
  "maxFileSizeInMb": 50
}

{
  "deliveryTimeout": 10,
  "maxConsecutiveFailures": 5,
  "secretEncryptionEnabled": false
}

Auto-derived from account_name if omitted

Auto-derived from account_name if omitted

{
  "predefined": []
}

Represents the encryption key for the dataset env vars. This field is required if enabled is true or None.

The encryption key must be a 32-character alphanumeric string, which includes both uppercase and lowercase Latin letters (A-Z, a-z) and digits (0-9).

To generate use:

tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo

The encryption key must be a 32-character alphanumeric string, which includes both uppercase and lowercase Latin letters (A-Z, a-z) and digits (0-9).

To generate use:

tr -dc 'A-Za-z0-9' < /dev/urandom | head -c 32; echo

{
  "base": {
    "datafusion.catalog.default_catalog": "kamu",
    "datafusion.catalog.default_schema": "kamu",
    "datafusion.catalog.information_schema": "true",
    "datafusion.sql_parser.enable_ident_normalization": "false"
  },
  "batchQuery": {},
  "compaction": {
    "datafusion.execution.target_partitions": "1"
  },
  "ingest": {
    "datafusion.execution.target_partitions": "1"
  },
  "useLegacyArrowBufferEncoding": false
}

{
  "datafusion": "ghcr.io/kamu-data/engine-datafusion:0.9.0",
  "flink": "ghcr.io/kamu-data/engine-flink:0.18.2-flink_1.16.0-scala_2.12-java8",
  "risingwave": "ghcr.io/kamu-data/engine-risingwave:0.2.0-risingwave_1.7.0-alpha",
  "spark": "ghcr.io/kamu-data/engine-spark:0.23.1-spark_3.5.0"
}

Type of the networking namespace (relevant when running in container environments)

{
  "datafusion.catalog.default_catalog": "kamu",
  "datafusion.catalog.default_schema": "kamu",
  "datafusion.catalog.information_schema": "true",
  "datafusion.sql_parser.enable_ident_normalization": "false"
}

Base configuration options See: <https://datafusion.apache.org/user-guide/configs.html>

{
  "datafusion.execution.target_partitions": "1"
}

{
  "datafusion.execution.target_partitions": "1"
}

Makes arrow batches use contiguous Binary and Utf8 encodings instead of more modern BinaryView and Utf8View. This is only needed for compatibility with some older libraries that don’t yet support them.

See: kamu-node#277

UNSTABLE: Datafusion engine image

UNSTABLE: Flink engine image

UNSTABLE: RisingWave engine image

UNSTABLE: Spark engine image

Forces iteration to stop after the specified number of blocks were scanned even if we didn’t reach the target record number. This is useful to not lose a lot of scanning progress in case of an RPC error.

Default number of blocks to scan within one query to eth_getLogs RPC endpoint.

Many providers don’t yet return blockTimestamp from eth_getLogs RPC endpoint and in such cases block_timestamp column will be null. If you enable this fallback the library will perform additional call to eth_getBlock to populate the timestam, but this may result in significant performance penalty when fetching many log records.

See: ethereum/execution-apis#295

Time after which FlightSQL client session will be forgotten and client will have to re-authroize (for anonymous clients)

Time after which FlightSQL session context will be released to free the resources (for anonymous clients)

Time after which FlightSQL client session will be forgotten and client will have to re-authroize (for authenticated clients)

Time after which FlightSQL session context will be released to free the resources (for authenticated clients)

{
  "awaitingStepSecs": 1,
  "defaultRetryPolicies": {},
  "mandatoryThrottlingPeriodSecs": 60
}

{
  "batchSize": 20,
  "maxListeningTimeoutMs": 2000,
  "minDebounceIntervalMs": 100
}

{
  "checkingIntervalSecs": 1
}

{
  "image": "ghcr.io/kamu-data/jupyter:0.7.1",
  "livyImage": "ghcr.io/kamu-data/engine-spark:0.23.1-spark_3.5.0"
}

Private key used to sign API responses. Currently only ed25519 keys are supported.

To generate use:

dd if=/dev/urandom bs=1 count=32 status=none |
    base64 -w0 |
    tr '+/' '-_' |
    tr -d '=' |
    (echo -n u && cat)

The command above:

• reads 32 random bytes
• base64-encodes them
• converts default base64 encoding to base64url and removes padding
• prepends a multibase prefix

HTTP Gateway URL to use for downloads. For safety, it defaults to http://localhost:8080 - a local IPFS daemon. If you don’t have IPFS installed, you can set this URL to one of the public gateways like https://ipfs.io. List of public gateways can be found here: https://ipfs.github.io/public-gateway-checker/

Whether kamu should pre-resolve IPNS DNSLink names using DNS or leave it to the Gateway.

Time in milliseconds to wait for MQTT broker to send us some data after which we will consider that we have “caught up” and end the polling loop.

{
  "allowAnonymous": true,
  "anonSessionExpirationTimeout": "30m",
  "anonSessionInactivityTimeout": "5s",
  "authedSessionExpirationTimeout": "30m",
  "authedSessionInactivityTimeout": "5s"
}

FlightSQL configuration

{
  "httpGateway": "http://localhost:8080/",
  "preResolveDnslink": true
}

{
  "kind": "Simple",
  "splitParagraphs": false,
  "splitSections": false
}

{
  "kind": "Dummy"
}

{
  "clearOnStart": false,
  "incrementalIndexing": true
}

{
  "kind": "Dummy"
}

{
  "commitAfterBlocksScanned": 1000000,
  "getLogsBlockStride": 100000,
  "rpcEndpoints": [],
  "useBlockTimestampFallback": false
}

{
  "connectTimeout": "30s",
  "maxRedirects": 10,
  "userAgent": "kamu-cli/0.260.1"
}

{
  "brokerIdleTimeout": "1s"
}

Target number of records after which we will stop consuming from the resumable source and commit data, leaving the rest for the next iteration. This ensures that one data slice doesn’t become too big.

Represents the encryption key for the webhooks secret. This field is required if secret_encryption_enabled is true or None.

The encryption key must be a 32-character alphanumeric string, which includes both uppercase and lowercase Latin letters (A-Z, a-z) and digits (0-9).

Example

let config = WebhooksConfig {
    ...
    secret_encryption_enabled: Some(true),
    encryption_key:
Some(String::from("aBcDeFgHiJkLmNoPqRsTuVwXyZ012345")) };