API guide · collectors

gRPC ingest

Overview

Use gRPC ingest to feed points and durable gap markers from a custom collector into OrdinateDB. It is the collector write boundary for resolving source identifiers, streaming durable batches, and obeying server backpressure.

Use the HTTP read APIs instead when you are consuming stored data.

Quick example

A minimal collector resolves source URIs, opens the bidirectional write stream, observes credit, and removes local WAL records only after a cumulative acknowledgement.

Transport-neutral collector loop

resolved = ResolveSeries({
  collector_id,
  source_uris
})

open Write stream with bearer metadata or mTLS
send no more than current credit_batches

for each sealed edge-WAL record:
  send WriteBatch(
    collector_id,
    session_epoch,
    sequence_number = wal_record.index,
    series,
    watermark_ns,
    collector_receipt_ns,
    stream_class
  )

for each WriteAck:
  delete WAL records through ack.sequence_number
  set send window to ack.credit_batches when present
  estimate clock skew from ack.server_time_ns

ResolveSeries supplies stable series UUIDs. Each WriteAck confirms the durable sequence boundary and updates the collector's send window when credit is present.

How it works

Custom collectors use package ordinate.ingest.v1. The shipped Ingest service has two RPCs:

Service definition

service Ingest {
  rpc Write(stream WriteBatch) returns (stream WriteAck);
  rpc ResolveSeries(ResolveSeriesRequest) returns (ResolveSeriesResponse);
}

Reference

ResolveSeries

Resolve source URIs through the server registry before writing. The request carries the collector identity and a batch of opaque source URIs; the response maps each known URI to its stable series UUID.

Field	Number	Type	Meaning
`collector_id`	1	`string`	Collector identity name.
`source_uris`	2	`repeated string`	Source URIs to resolve.

ResolveSeriesResponse.uri_to_series_uuid is protobuf field 1 with type map<string, string>.

Write stream lifecycle

Write is bidirectional streaming. A collector sends ordered WriteBatch messages and independently receives cumulative WriteAck messages. The sequence number is the durable edge-WAL record index. An acknowledgement means the server has durably committed through that sequence.

Field	Number	Type	Meaning
`collector_id`	1	`string`	Must remain constant and match the authenticated collector identity.
`session_epoch`	2	`uint64`	Monotonic dedupe and fencing epoch.
`sequence_number`	3	`uint64`	Durable, monotonic edge-WAL record index.
`series`	4	`repeated SeriesPoints`	Points and gaps grouped by series.
`watermark_ns`	5	`int64`	Promise that no value below this UTC nanosecond will follow; zero means none.
`collector_receipt_ns`	6	`int64`	Collector wall clock when the batch was sealed.
`stream_class`	7	`StreamClass`	LIVE by default; BACKFILL receives lower credit.

SeriesPoints

Field	Number	Type	Meaning
`series_uuid`	1	`string`	Target registered series UUID.
`points`	2	`repeated Point`	Typed samples for the series.
`gaps`	3	`repeated GapMarker`	Durable edge-detected gaps.

Point

Field	Number	Type	Meaning
`timestamp_ns`	1	`int64`	UTC Unix timestamp in nanoseconds.
`value_f64`	2	`double`	Float64 value; preserves the original Phase 0 field number.
`quality`	3	`uint32`	Source quality code.
`value_f32`	4	`float`	Float32 value.
`value_i64`	5	`sint64`	Signed integer value.
`value_bool`	6	`bool`	Boolean value.
`value_string`	7	`string`	String value, limited to 65,536 bytes.
`value_digital`	8	`DigitalValue`	Ordinal plus opaque enum-set reference.

Compatibility rule: an unset Point.value oneof means float64 zero. Early proto3 collectors omitted field 2 when the value was exactly 0.0; the server must not interpret that wire shape as “no value.”

DigitalValue

Field	Number	Type	Meaning
`ordinal`	1	`sint64`	Digital state ordinal.
`enum_set_ref`	2	`string`	Opaque model-registry reference.

GapMarker

Field	Number	Type	Meaning
`start_ns`	1	`int64`	Included start of the gap.
`end_ns`	2	`int64`	Excluded end of the gap.
`cause`	3	`GapCause`	Structured gap cause.
`detail`	4	`string`	Optional free-text detail.

GapCause values are GAP_CAUSE_UNSPECIFIED, GAP_BUFFER_DROP_OLDEST, GAP_BUFFER_BLOCK_SOURCE, GAP_SOURCE_DISCONNECTED, and GAP_COLLECTOR_FAILOVER.

WriteAck and credit

Field	Number	Type	Meaning
`sequence_number`	1	`uint64`	Cumulative durable commit position.
`credit_batches`	2	`optional uint32`	Maximum outstanding batches: absent means unlimited; zero means pause.
`server_time_ns`	3	`int64`	Server wall clock when the acknowledgement was sent.

The collector must not have more unacknowledged batches in flight than credit_batches. Live streams normally receive more credit than backfill streams. Disk-full state can reduce credit to zero; the server can later re-grant credit without advancing the acknowledged sequence. Treat the acknowledgement as cumulative and discard all local WAL records through its sequence number only after receiving it.

Epochs, reconnects, and status codes

Increase session_epoch when a new collector owner takes over. A higher epoch fences older streams.
FAILED_PRECONDITION containing stale session_epoch is a permanent fence for that epoch.
ABORTED denotes retryable admission conflicts such as a same-class reconnect race or full stream slots.
INVALID_ARGUMENT means the batch violates the ingest contract.
PERMISSION_DENIED includes a mismatch between the authenticated collector identity and collector_id.
INTERNAL reports an internal ingest or registry failure.

Common patterns

Authenticate a collector

Collectors can send a bearer service token or use mTLS. A bearer token belongs to a service identity whose name must exactly equal collector_id. For mTLS, the server matches a URI or DNS SAN to a kind=collector service identity. Token and mTLS paths coexist; neither is required while server auth is disabled or shadow. See authentication for token provisioning.

Reconnect without duplicating durable work

Reopen with the same owner epoch and resume after the last cumulative acknowledged sequence. Increase session_epoch only when a new collector owner takes over; the higher epoch fences older streams.