TerminusDB.Streaming (terminusdb_ex v0.3.3)

Copy Markdown View Source

Incremental decoding of TerminusDB concatenated-JSON response bodies.

TerminusDB's document endpoint returns documents as concatenated JSON (multiple JSON objects back-to-back, not newline-delimited and not a JSON array) when the as_list query parameter is not set. This module provides a bracket/depth-aware splitter that incrementally parses chunks emitted by Req's streaming (into:) into a stream of decoded maps.

Used by TerminusDB.Document.stream/2 (ADR-0007). Note: Document.stream/2 does not pass as_list: true, so the server returns concatenated JSON; do not enable as_list on a streaming call as this splitter does not handle JSON array bodies.

Summary

Functions

document_stream(resp, opts \\ [])

Returns a stream of decoded JSON maps from a Req response that delivers chunks to into: :self. The response body must be concatenated JSON objects (the TerminusDB default when as_list is not set); JSON array bodies are not supported.

split_concatenated(buffer)

Splits a binary accumulator of concatenated JSON into a list of complete JSON object binaries and the remaining buffer. Objects are split on top-level } that closes the initial {, respecting string literals and nested braces.

Functions

document_stream(resp, opts \\ [])

@spec document_stream(
  Req.Response.t(),
  keyword()
) :: Enumerable.t()

Returns a stream of decoded JSON maps from a Req response that delivers chunks to into: :self. The response body must be concatenated JSON objects (the TerminusDB default when as_list is not set); JSON array bodies are not supported.

Options

  • :timeout — receive timeout in milliseconds between chunks. If no message arrives within this window, the stream halts (defaults to 15_000).

Examples

# With a Req response streamed via `into: :self`:
resp = Req.get!(req, url: "document/admin/mydb", into: :self)
TerminusDB.Streaming.document_stream(resp) |> Enum.take(10)

split_concatenated(buffer)

@spec split_concatenated(binary()) :: {[binary()], binary()}

Splits a binary accumulator of concatenated JSON into a list of complete JSON object binaries and the remaining buffer. Objects are split on top-level } that closes the initial {, respecting string literals and nested braces.

If the buffer ends inside a string and the last byte is a lone \ (an incomplete escape sequence), the backslash is retained in the returned buffer so the caller can prepend the next chunk and complete the escape.