Building Locally

Prerequisites:

  • Docker or Podman (note: unit tests run with Podman by default)
    • If using docker - make sure it’s usable without sudo (guidelines)
    • If using podman - make sure it’s setup to run root-less containers (guidelines)
  • Rust toolset
    • Install rustup
    • The correct toolchain version will be automatically installed based on the rust-toolchain file in the repository
  • Tools used by tests
    • Install jq - used to query and format JSON files
    • Install kubo (formerly known as go-ipfs) - for IPFS-related tests
  • Code generation tools (optional - needed if you will be updating schemas)

Clone the repository:

git clone git@github.com:kamu-data/kamu-cli.git

Build it:

cd kamu-cli
cargo build

To use your locally-built kamu executable link it as so:

ln -s $PWD/target/debug/kamu-cli ~/.local/bin/kamu

When needing to test against a specific official release you can install it under a different alias:

curl -s "https://get.kamu.dev" | KAMU_ALIAS=kamu-release sh

Prepare test containers once before running tests:

cargo run --bin kamu-cli -- config set --user engine.runtime podman
kamu init --pull-test-images

Then run tests:

cargo test

Run Tests with Docker (Alternative)

Prepare test containers once before running tests:

kamu init --pull-test-images

Then run tests:

KAMU_CONTAINER_RUNTIME_TYPE=docker cargo test

Using Nextest test runner (Optional)

Nextest is a better test runner.

To install it first prepare cargo-binstall and cargo-update packages:

cargo install cargo-binstall
cargo install cargo-update
cargo binstall cargo-binstall --force # Make future updates to this binary use precompiled version

Then run:

cargo binstall cargo-nextest
cargo nextest run

Build Speed Tweaks (Optional)

Consider configuring Rust to use lld linker, which is much faster than the default ld (may improve link times by ~10-20x).

To do so install lld, then create ~/.cargo/config.toml file with the following contents:

[build]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

One more alternative is to use mold linker, which is also much faster than the default ld.

To do so install mold or build it with clang++ compiler from mold sources then create ~/.cargo/config.toml file with the following contents:

[build]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

Building with Web UI (Optional)

To build the tool with embedded Web UI you will need to clone and build kamu-web-ui repo or use pre-built release. Now build the tool while enabling the optional feature and passing the location of the web root directory:

KAMU_WEB_UI_DIR=`pwd`/../kamu-web-ui/dist/kamu-platform/ cargo build --features kamu-cli/web-ui

Note: KAMU_WEB_UI_DIR requires absolute path

Note: in debug mode the directory content is not actually being embedded into the executable but accessed from the specified directory.

Code Generation

Many core types in kamu are generated from schemas and IDLs in the open-data-fabric repository. If your work involves making changes to those - you will need to re-run the code generation tasks using:

make codegen

Make sure you have all related dependencies installed (see above) and that ODF repo is checked out in the same directory as kamu-cli repo.

Release Procedure

  1. While on the feature branch, bump the crates versions using release tool, e.g. cargo run --bin release -- --major / --minor / --patch
  2. We try to stay up-to-date with all dependencies, so for any minor or major release you should also run cargo update
  3. Create a CHANGELOG entry for the new version
  4. Create PR, wait for tests, then merge
  5. Checkout and pull master
  6. Tag the latest commit with a new version: git tag vX.Y.Z
  7. Push the tag to repo: git push origin tag vX.Y.Z
  8. Github Actions will pick up the new tag and create a new GitHub release from it