Supported Repository Types
kamu is shared via repositories. There are multiple types of repositories that differ by the kinds of services they provide. The most basic repository allows you to simply upload (“push”) and download (“pull”) data.
|Local FS||A basic repository that uses the local file system folder. Mainly used for examples and testing.||
|HTTP(s)||A basic repository that provides read-only access to data.||
|S3||A basic repository that stores data in Amazon S3 bucket. Can be used with any S3-compatible storage API||
|IPFS||Uses IPFS HTTP Gateway for reading. Push is only possible via IPNS (see details).||
Push / Pull Aliases
Starting out, you can always use explicit URLs when syncing data from and to repositories.
If some remote source contains a dataset you’re interested in you can download it using the
kamu pull https://example.org/dataset --as example-dataset
If you have a dataset you would like to share, or made some changes to a dataset you are sharing with your friends - you can upload the new data using the
kamu push example-dataset --to s3://my-bucket.example.org/example-dataset
When you run these commands,
kamu will automatically create “pull” and “push aliases” for this dataset:
kamu repo alias list example-dataset ┌─────────────────┬──────┬────────────────────────────────────────────┐ │ Dataset │ Kind │ Alias │ ├─────────────────┼──────┼────────────────────────────────────────────┤ │ example-dataset │ Pull │ https://example.org/dataset │ │ example-dataset │ Push │ s3://my-bucket.example.org/example-dataset │ └─────────────────┴──────┴────────────────────────────────────────────┘
With aliases in place, next time you want to pull or push data you can simply do:
kamu pull example-dataset # Will pull from https://example.org/dataset kamu push example-dataset # Will push to s3://my-bucket.example.org/example-dataset
push commands will analyze the state of the dataset in the workspace and at the repository and will only upload data and metadata that wasn’t previously seen (a minimal update).
These commands are also “safe”. They will detect all types of concurrent changes and history divergence and prevent you from overwriting someone else’s changes.
Rootdataset from a remote source running
kamu pullon it will attempt to sync it from the repository, and NOT execute a polling or other ingest action. In
kamu listdatasets with pull aliases will be showing up as
If you store multiple datasets side by side in some locations you can add it as a repository. Repositories are configured per workspace using the
kamu repo command group.
To add new repo use:
kamu repo add my-s3-bucket s3://my-bucket.example.org/
This repo will now be visible in
kamu repo list.
To pull/push a dataset from/to this repo you can now use remote references like
kamu pull my-s3-bucket/example-dataset kamu pull example-dataset --to my-s3-bucket
Searching for Data
To look for datasets in the repos use:
kamu search covid19 # Searching with empty query lists (a limited number of) datasets in all repos kamu search # You can specify the search term and specific repo to look in kamu search covid19 --repo my-s3-bucket
Search is delegated to the repository implementations and its capabilities depend on the type of the repo. Whereas smart repos may support advanced full-text search, simple storage-only repos may be limited to a substring search by dataset name.