Storage backends

Package	Value
Distribution	archetype-ecs
Import package	archetype

Integration API. Supported asynchronous storage implementations for custom engine wiring.

AsyncStore

AsyncStore(session, io_config=None)

The ArchetypeStore is a component that manages the storage and retrieval of archetype tables.

Since our Schema supports multiple simulations and runs, our namespace is simply "archetypes". This allows us to run multiple simulations across many worlds using the same catalog.archetypes, by definition, the exact set of components attached to an entity. So we don't really which simulation or run we're using, so long as we differentiate between the simulation and run.

Using Daft Sessions/Catalogs enables us to reference archetype tables without having to hold them in memory.

get_archetype_df async

get_archetype_df(
    sig,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
    commit_tokens=None,
)

Get all archetypes that contain all of the specified component types.

commit_tokens is the reader-side visibility allowlist: current-generation rows must match a published manifest token. Legacy (v0.2) rows carry no commit identity and are implicitly epoch-0 visible — the allowlist never applies to them.

get_existing_table_schema async

get_existing_table_schema(table_id)

Return an existing table's Arrow schema without creating it.

get_existing_table_df async

get_existing_table_df(
    table_id,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
)

Read an existing table by durable physical identity (never creates).

list_signatures async

list_signatures()

List all registered signatures, including read-created tables.

list_committed_signatures async

list_committed_signatures()

List signatures backed by a successful durable append.

append async

append(sig, df)

Append a table with a new dataframe. Returns the durable batch receipt.

The backing-table write is the only executor of the incoming plan. Core never runs a count first to decide whether the append is worth attempting (issue #538): a zero-row frame is the backend's own successful no-op. Iceberg's append produces no natural row count, so the receipt reports rows=None — unknown is None, not a preflight.

flush async

flush()

No-op: appends write through to Iceberg.

shutdown async

shutdown()

Shutdown the store.

AsyncCachedStore

AsyncCachedStore(async_store, cache_config)

This Store variant holds a cache.

get_archetype_df async

get_archetype_df(
    sig,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
    commit_tokens=None,
)

Get all archetypes that contain all of the specified component types.

Reads are a union of already-flushed rows on disk and unflushed rows still in the memtable so that callers always see a coherent view of the signature's full history regardless of flush state. The commit-token allowlist applies to both layers, so staged rows are no more visible than durable rows.

list_signatures async

list_signatures()

Delegate to inner store.

list_committed_signatures async

list_committed_signatures()

Include successful staged appends as well as durable inner writes.

get_existing_table_schema async

get_existing_table_schema(table_id)

Delegate open-never-create physical discovery to durable storage.

get_existing_table_df async

get_existing_table_df(
    table_id,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
)

Delegate physical reads; staged cache rows have no durable table identity.

append async

append(sig, df)

Cache driven append with built in flush logic to underlying storage (super) a table with a new dataframe.

The receipt is staged (durable=False) unless this append tripped a flush: rows become durable at flush()/threshold/idle/shutdown. A commit coordinator must call flush() before publishing a manifest head — a head must never claim RAM-only rows are durable.

flush async

flush()

Drain every memtable to the inner store.

Called by the commit coordinator's owner before a manifest head is published, so visibility never outruns durability.

shutdown async

shutdown()

Stop background flushing and ensure all pending data is flushed to the inner store.

AsyncLancedbStore

AsyncLancedbStore(uri, namespace=None)

get_existing_table_schema async

get_existing_table_schema(table_id)

Return an existing table's Arrow schema without creating it.

get_existing_table_df async

get_existing_table_df(
    table_id,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
)

Read an existing table by durable physical identity.

Unlike get_archetype_df, this never calls _ensure_table and therefore cannot turn a read into a committed-looking table.

get_archetype_df async

get_archetype_df(
    sig,
    world_id,
    run_id,
    *,
    ticks=None,
    entity_ids=None,
    active_only=False,
    commit_tokens=None,
)

Read one archetype's rows.

commit_tokens is the reader-side visibility allowlist: current-generation rows must match a published manifest token. Legacy (v0.2) rows carry no commit identity and are implicitly epoch-0 visible — the allowlist never applies to them.

list_committed_signatures async

list_committed_signatures()

List signatures backed by a successful durable append.

flush async

flush()

No-op: appends write through to LanceDB.