A serving engine for temporal features, risk state, and online context.

The short version

TemporalStore is MatrixArk's online temporal serving engine. Its target is not the generic cache problem. Its target is the product gap between stream processing, feature stores, wide-column databases, and online caches.

Risk, fraud, ads, recommendation, marketplace trust, and LLM agent systems repeatedly ask questions such as: how many times did this device fail login in the last 30 minutes, which merchants did this card touch in the last 24 hours, which campaign impressions happened in the last hour, what did this user recently read, and what context should be placed into the next model call?

Why cache plus pipelines breaks down

A common architecture starts cleanly: raw events go to a queue, a stream processor computes aggregates, Redis or another online store serves latest values, and offline jobs repair or backfill state. That is a strong architecture for stable feature sets.

It becomes expensive when temporal questions change every week. A risk team adds a new velocity check. An ads team changes a frequency cap. A recommender wants a different sequence slice. An agent product needs session state, tool-call history, policy counters, and user preferences in one context bundle. Each new question creates new stream logic, new keys, new TTL rules, and another place where serving behavior can drift from training behavior.

What TemporalStore owns

The central design choice is to make the storage engine understand the online model. Instead of storing an opaque blob and forcing every caller to implement window logic, the engine exposes model-aware commands over entity-local state.

Architecture

TemporalStore is organized as a metaserver plus data nodes. The metaserver owns namespace, table, partition, placement, and routing metadata. Clients or proxies use that metadata to route writes to primary partitions and route reads according to consistency and freshness policy.

A partition is the core serving unit. It owns a slot range and contains the in-memory objects, model command executors, index, oplog, page store, block cache integration, background dump logic, and replica replay logic. Keys are hashed into slots; slots map to partition sets; partition sets contain a primary and optional secondaries.

Write workflow

A write is routed to the partition owner, applied to the model object in memory, logged as a replayable mutation, and later merged into persisted pages.

This is why repeated updates to a hot entity can be efficient. The hot object is updated in memory while the storage layer can later dump merged object or slot state. In an LSM design, every update enters the write path and later participates in compaction. In TemporalStore's target design, the storage engine can reduce write amplification for entity-local serving state by merging hot updates before page persistence.

write failed_login_count:
  key = device_id
  dimensions = { country: "US", method: "password" }
  timestamp = now
  bucket = 10 seconds
  value += 1

query:
  key = device_id
  metric = failed_login_count
  filter country == "US"
  window = last 30 minutes

Read workflow

Hot reads are served from memory. Warm reads use an in-memory index to locate persisted pages and can hit the block cache. Cold reads go to the shared durable store, decode the page or object, refill cache, and then execute the model-specific query.

The query is not a full-table scan. The routing layer sends the request to the entity's partition, the index identifies slot and page metadata, and the model runs bounded logic over retained entity state. This is the right shape for high-cardinality online features, where there may be millions of sparse entity keys but each request usually asks about one entity or a small set of entities.

Storage layout: object, slot, page, zone, oplog

TemporalStore's storage vocabulary matters because it explains why the engine is different from a plain cache.

Replication and recovery

TemporalStore uses primary partitions and secondary replicas. A secondary reconstructs queryable state from durable page/index state plus oplog replay. In a shared-store deployment, secondaries can read persisted streams directly. In a primary-pull design, secondaries can pull stream data from the current primary.

The roadmap hardens this with explicit primary lease or epoch fencing, freshness gates before promotion, secondary lag metrics, and recovery tests that include historical pages. Split-brain writers and stale replicas must be rejected before a deployment can be trusted for correctness-sensitive workloads.

Data model examples

The right model depends on the product question.

Why not just Redis or RocksDB?

Redis-style systems are excellent for simple strings, hashes, latest profiles, leader boards, queues, and many cache workflows. MatrixDB is the MatrixArk product direction for eventually consistent KV workloads that need both low-latency serving and offline or nearline query access. TemporalStore is different: it tries to put temporal semantics inside the serving engine.

RocksDB is a powerful embedded LSM engine and often the right local persistence layer. The tradeoff is that repeated updates create LSM write-path work and later compaction. For a hot entity receiving many small counter or sequence updates, TemporalStore's model is to update in memory, append replayable mutations, then dump merged pages when the storage manager decides to persist dirty state.

Where it fits with feature platforms

TemporalStore should not try to replace every feature platform capability on day one. Systems such as Feast, Chronon, Fennel, and Featureform are strong at registry, definitions, lineage, transformation orchestration, training sets, and offline/online consistency. TemporalStore is strongest as the online temporal serving engine underneath or beside those platforms.

LLM context is a related, not identical, problem

TemporalStore is not a GPU KV-cache manager. It does not replace the transformer KV cache used by vLLM, SGLang, TensorRT-LLM, or LMCache-style systems. The LLM runtime still needs tensor layout, prefix matching, GPU memory management, token-position lifecycle, and attention-cache APIs.

The overlap is structured context and state. Agent systems need recent conversation events, tool calls, retrieved-document metadata, memory freshness, safety counters, user preferences, and policy state. Vector databases are good at similarity search. TemporalStore is useful for temporal and structured context that should be filtered, counted, ordered, replayed, or expired with serving semantics.

Operational design

A serving engine is not a product until it is observable. The current MatrixArk observability work exposes separate pages for TemporalStore, MatrixDB, MatrixKV, and a Prometheus-compatible metrics endpoint. For TemporalStore, the most important signals are partition health, primary placement, replica replay lag, oplog append latency, page dump progress, block-cache hit ratio, storage errors, and client retry visibility.

In a quick engineering environment, those pages can be served under the same HTTP port as the company site. That is useful for fast iteration, but it is not the production boundary. A production deployment should separate the public website from the authenticated operations console and keep raw metrics endpoints private.

Autoscale and rolling deployment design

Autoscaling should not mean that a new instance independently decides which partitions it owns. A new TemporalStore data node should boot from a versioned runtime package, discover its instance identity, register with the metaserver, and advertise capacity. The metaserver then assigns secondary replicas first, waits for catch-up, and only then moves read traffic or primary ownership.

Scale-down needs the reverse workflow. The node should enter draining state, stop receiving new primary assignments, move primary partitions away, wait for replacement replicas to become healthy, and only then complete the cloud lifecycle termination. This protects the serving path from raw instance termination and makes rolling upgrades safer.

Engineering snapshot

The current AWS test cluster used one metaserver/client/UI node and two data nodes. A Prometheus bridge scraped the metaserver and both data nodes through their runtime variables endpoint.

These numbers are engineering snapshots, not final product benchmarks. The useful signal is the shape of the system: live metrics, module-level smoke coverage, and clear correctness gaps to fix before stronger claims.

The product boundary

TemporalStore should be honest about what it is. It is not a full warehouse, not a full feature platform, not a vector database, and not a transformer KV-cache runtime. It is an online state engine for temporal features and context. MatrixDB handles eventually consistent KV serving, profiles, and offline-queryable state. MatrixKV handles strongly consistent transactional metadata. Together they form the MatrixArk platform.

1. Use TemporalStore when the feature depends on recent events, windows, filters, distinct state, or sequences.
2. Use MatrixDB when the workload is latest profile, large hash/profile KV, hot-key cache, tenant-scale service state, scans, exports, or offline/nearline query over persisted KV data and eventual consistency is acceptable.
3. Use MatrixKV when the workload needs transactional KV, timestamp coordination, metadata correctness, or strong consistency.
4. Use feature stores and warehouses for registry, training sets, lineage, offline truth, and backfills.