> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kamu.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Schema

export const YouTubeList = ({id}) => {
  const src = `https://www.youtube.com/embed/videoseries?list=${id}`;
  return <iframe className="w-full aspect-video rounded-xl" src={src} allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowFullScreen></iframe>;
};

export const YouTube = ({id, width}) => {
  const src = `https://www.youtube.com/embed/${id}`;
  return <iframe className="w-full aspect-video rounded-xl" src={src} allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowFullScreen width={width}></iframe>;
};

export const Schema = ({t, id}) => {
  const anchor = id ? id : t.toLowerCase().replace(/\s+/g, "-");
  const link = `/odf/schemas#${anchor}`;
  return <a class="schema-object" href={link}>{t}</a>;
};

export const Term = ({t, id}) => {
  const anchor = id ? id : t.toLowerCase().replace(/\s+/g, "-");
  const link = `/general/glossary#${anchor}`;
  return <a class="glossary-term" href={link}>{t}</a>;
};

export const Diagram = ({src, alt}) => {
  return <div style={{
    display: "flex",
    "flex-direction": "column",
    "align-items": "center"
  }}>
    <img src={src} alt={alt} style={{
    background: "#dddddd",
    "margin-bottom": 0
  }} />
    <span>{alt}</span>
  </div>;
};

**Domain**: Datasets\
**Depends on**: -\
**RFCs**: [RFC-016](/odf/rfcs/016-odf-schema)

## Abstract

This spec defines the ODF schema format used to describe the logical structure of <Term t="data" id="data" /> independently of how it is physically encoded in files (e.g. Parquet, Avro, CSV) or in memory (e.g. Arrow). It defines the set of supported logical types, the mechanism for attaching custom attributes to schemas and fields, and the rules for converting between ODF and Apache Arrow schemas.

## Logical Types vs. Encoding

ODF schema describes **logical types** of data - the semantics and value boundaries of a type and how it can be operated on. It deliberately does not capture the **encoding** (physical layout) of data such as compression, dictionary encoding, offset widths, or view-based representations.

This separation ensures that:

* Logical types remain stable across different file formats, data chunks, and engines
* Encoding strategies (compression, dictionary usage, encoding hints) can evolve independently, both at the file level and in memory
* Producers and consumers can negotiate encoding at runtime while relying on a consistent logical schema contract

While inspired by [Apache Arrow](https://arrow.apache.org/), ODF schema intentionally differs from the [Arrow flatbuffer schema](https://github.com/apache/arrow/blob/main/format/Schema.fbs), which mixes logical types with in-memory layout details and is not suited to be defined manually by humans.

## Schema Format

A <Term t="schema" id="schema" /> (`DataSchema`) consists of:

* `fields` - an ordered list of top-level fields (columns)
* `extra` - optional [extra attributes](#extra-attributes) that apply to the schema as a whole

Each field (`DataField`) consists of:

* `name` - name of the field
* `type` - [logical type](#logical-types) of the field
* `extra` - optional [extra attributes](#extra-attributes) that apply to the field

Example:

```yaml theme={null}
fields:
  - name: offset
    type:
      kind: UInt64
  - name: op
    type:
      kind: UInt8
  - name: system_time
    type:
      kind: Timestamp
      unit: Millisecond
      timezone: UTC
  - name: event_time
    type:
      kind: Timestamp
      unit: Millisecond
      timezone: UTC
  - name: mri_content_hash
    type:
      kind: String
    extra:
      opendatafabric.org/description: References the MRI scan data linked to the dataset by its hash
  - name: subject
    type:
      kind: Struct
      fields:
        - name: id
          type:
            kind: String
          extra:
            opendatafabric.org/description: Subject's unique identity
        - name: gender
          type:
            kind: Option
            inner:
              kind: String
          extra:
            opendatafabric.org/description: Subject's gender
    extra:
      opendatafabric.org/description: Information about the subject
  - name: area_codes
    type:
      kind: List
      itemType:
        kind: String
    extra:
      opendatafabric.org/description: List of body area codes covered by this MRI scan
extra:
  c.com/z: baz
```

See <Schema t="DataSchema" id="dataschema" /> reference for the full definition.

### Nullability

Fields are non-nullable by default. Nullability is expressed explicitly via the special `Option` type that wraps the type of the value (unlike Arrow, where nullability is a property of a field). This applies recursively to nested types, e.g. struct fields and list items.

## Logical Types

| Type | Parameters | Description |
| - | - | - |
| `Bool` | | Boolean value representing true or false |
| `Int8`, `Int16`, `Int32`, `Int64` | | Signed integer |
| `UInt8`, `UInt16`, `UInt32`, `UInt64` | | Unsigned integer |
| `Float16`, `Float32`, `Float64` | | Floating-point number |
| `Decimal` | `precision`, `scale` | Fixed-point decimal number with specified total number of digits and number of digits after the decimal point (scale can be negative) |
| `String` | | Unicode string |
| `Binary` | `fixedLength` (optional) | Sequence of bytes, variable-length unless `fixedLength` is specified |
| `Date` | | Calendar date |
| `Time` | `unit` (default `Millisecond`) | Time of day without a date |
| `Timestamp` | `unit` (default `Millisecond`), `timezone` (default `UTC`) | Point in time represented as an offset from the Unix epoch. Timezone is either a name from the Olson timezone database (e.g. `America/New_York`) or an absolute offset (e.g. `+07:30`) |
| `Duration` | `unit` (default `Millisecond`) | Elapsed time interval |
| `List` | `itemType`, `fixedLength` (optional) | List of values of the same type, variable-length unless `fixedLength` is specified |
| `Map` | `keyType`, `valueType`, `keysSorted` (optional) | Map of key-value pairs, represented as a list of entries |
| `Struct` | `fields` | Collection of named [fields](#schema-format), each with its own type |
| `Option` | `inner` | Optional (nullable) value of the inner type |
| `Null` | | Type representing the absence of a value |

Time units are: `Second`, `Millisecond`, `Microsecond`, `Nanosecond`.

See <Schema t="DataType" id="datatype" /> reference for the full definition.

### Short Form

Types that don't require any parameters can be specified as a plain string instead of an object with a `kind` field. The following two definitions are equivalent:

```yaml theme={null}
- name: offset
  type:
    kind: UInt64
```

```yaml theme={null}
- name: offset
  type: UInt64
```

Short form is an input-only convenience - the expanded form is used when schema is stored in binary formats.

## Extra Attributes

Extra attributes is a key-value map that allows attaching custom information to schemas and fields.

Keys must be in the format `<domain>/<attribute-name>` (e.g. `kamu.dev/archetype`). The domain prefix is used to disambiguate attributes and avoid name collisions between different extensions.

Values can be a string or any JSON value, including nested objects.

Extra attributes are represented natively in YAML and JSON, and as a JSON-encoded string in FlatBuffers.

### Well-known Attributes

| Attribute | Applies to | Description |
| - | - | - |
| `opendatafabric.org/description` | Field | Human-readable description of the field |
| `arrow.apache.org/bufferEncoding` | Field | Arrow buffer encoding hint (see [Conversion to and from Arrow](#conversion-to-and-from-arrow)) |
| `arrow.apache.org/dateEncoding` | Field | Arrow date encoding hint (see [Conversion to and from Arrow](#conversion-to-and-from-arrow)) |
| `arrow.apache.org/decimalEncoding` | Field | Arrow decimal encoding hint (see [Conversion to and from Arrow](#conversion-to-and-from-arrow)) |

## Conversion to and from Arrow

Implementations should be able to convert between ODF and Arrow schemas back and forth without information loss. Since ODF schema captures only logical types, Arrow encoding details are preserved using encoding hints in [extra attributes](#extra-attributes):

**Arrow type:** `LargeUtf8`

```yaml theme={null}
type:
  kind: String
extra:
  arrow.apache.org/bufferEncoding:
    kind: Contiguous
    offsetBitWidth: 64
```

**Arrow type:** `Utf8View` (analogously for `BinaryView`, `ListView`, `LargeListView`)

```yaml theme={null}
type:
  kind: String
extra:
  arrow.apache.org/bufferEncoding:
    kind: View
    offsetBitWidth: 32
```

**Arrow type:** `Date64` (millisecond-based representation)

```yaml theme={null}
type:
  kind: Date
extra:
  arrow.apache.org/dateEncoding:
    unit: Millisecond
```

**Arrow type:** `Decimal128` (analogously for `Decimal256`)

```yaml theme={null}
type:
  kind: Decimal
extra:
  arrow.apache.org/decimalEncoding:
    bitWidth: 128
```

## Schema in APIs

When a schema is returned by APIs it should carry only the logical type information and not the encoding hints, unless the encoding represents the actual encoding of the data that this API transmits.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.