Ingesting Data via Ingestion Files vs via REST API

Overview

This article describes the supported options for sending data into Fynapse, the limits and constraints that apply to each, and guidance on which patway to choose for a given workload, such as large volumes of data, continuous near-real-time feeds, and high-rate streaming.

Inbound Integration Options

Fynapse offers two inbound channels. Both ingestion pathways yield the same result, i.e. records are validated and published to relevant Entities using the same deduplication key. The difference is the ingestion mechanism and its operational characteristics.

Because both channels produce to Kafka with the same de-duplication key (derived from the entity’s primary-key fields plus its creation time), re-sending the same record is idempotent on either channel — a duplicate submission will not create a duplicate record downstream.

Limits and Constraints

REST API

File Ingestion

Choosing the Pathway

Large Data Volumes

For large datasets — backfills, migrations, or periodic large batches — use the file channel with gzip-compressed CSV. This path is designed for volume:

• Files are streamed and processed in chunks rather than loaded whole into memory.
• Failed files can be re-submitted simply by re-uploading; de-duplication keeps the operation idempotent.
• Uploads are decoupled from processing, so there is no long-lived synchronous connection to fail mid-transfer.

The REST API is not recommended for large data volumes: the 1 MB request cap and 5 req/sec rate limit make it slow for large volumes, and the synchronous model means a network interruption fails the entire in-flight request.

Continuous / near-real-time Feeds

When data is sent as it appears in the source system rather than as a large batch, the REST API is the recommended channel. This is where REST is most recommended:

• Payloads are small, so the 1 MB limit is not a concern.
• Records reach the system in near-real-time — lower latency than the file channel’s polling interval allows.
• The synchronous response confirms each submission reached the system and returns per-row validation errors immediately.
• Idempotent de-duplication makes the channel safe for sources that may re-emit or replay events (e.g. change-data-capture feeds).

Two practices are essential for a continuous REST integration:

Micro-batch submissions

The governing constraint for a continuous feed is the 5 requests/second rate limit, not payload size. Sending one request per event will be throttled if the source emits faster than ~5 events/second.

Instead, accumulate events over a short window (e.g. 200 ms–1 s) and submit them as a single request containing multiple rows. A single sub-1 MB request can carry thousands of rows, which keeps throughput high and latency low while staying well within the rate limit. If sustained event rates exceed what micro-batching within the rate limit allows, request a higher rate-limit tier or switch to frequent file drops.

Roll Ingestion Sessions

A continuous feed cannot hold a single session open indefinitely — see Session lifecycle. Open a fresh session before the current one expires, submit micro-batches to the active session, and allow expired sessions to close. Alternatively, open sessions with a longer configured timeout, though rolling sessions is the cleaner pattern for a long-running feed.

High-rate Continuous Feeds

If the event rate consistently exceeds what the REST rate limit allows and a latency of a few seconds is acceptable, frequent small file drops are a good middle ground: better resilience and no rate limiting, at the cost of the ~5 second detection latency and per-file overhead.

Session Lifecycle

REST submissions occur within an ingestion session:

1. Open a session for a namespace + entity; this returns an ingestionId.
2. Submit one or more data requests against that ingestionId.
3. Close the session explicitly, or let it expire automatically.

A session expires a fixed duration after it is opened (default 15 minutes, configurable per session at open time). Expiry is based on elapsed time since the session was opened — an actively-used session is still closed once its timeout is reached, after which further submissions to that ingestionId are rejected. This is why continuous feeds must roll sessions.

Reliability and Idempotency

Both channels share the same reliability model at the system boundary; the file channel adds further resilience features around file handling.

For REST, a submission returns a per-row breakdown: successes are recorded, and any failed rows are returned with their error messages. The recommended recovery pattern is to resubmit only the failed rows — de-duplication makes this safe.

Monitoring and Visibility

Both channels report progress and outcomes to the Processing Summary screen in the Fynapse UI, so you can track ingestion. The view is near-real-time — it refreshes continuously as processing progresses.

The Processing Summary screen covers both REST and file ingestion, distinguished by source, and shows:

• Overall processing status per ingestion (in progress → ready → done), with a clear indication when an ingestion has errored.
• Per-Entity breakdown — for each target Entity in an ingestion, the number of records stored (succeeded) and failed.
• Row-level counts — successful vs. failed records.

For file ingestion, the screen additionally surfaces file-level state tracking:

• Counts of pending, processed, and errored files, plus reference (master-data) files.
• Each file resolves to a status of pending, done, or errored, so a failed file is immediately visible and can be corrected and re-uploaded.

For REST ingestion, the screen surfaces submission counts and the total number of rows ingested.

Best-practice Summary

Large Data Volumes

• Use the file channel with gzip-compressed CSV.
• Recover by re-uploading; rely on de-duplication for idempotency.
• Reconcile against the generated error file rather than assuming all-or-nothing.

Continuous / near-real-time Feeds (REST)

• Micro-batch events into sub-1 MB requests to stay within the 5 req/sec limit.
• Roll ingestion sessions ahead of expiry.
• Handle partial-success responses; resubmit only failed rows.
• Confirm the rate limit fits your peak event rate; request a higher tier if needed.

High-rate Continuous Feeds

• Use frequent small file drops when the event rate exceeds the REST rate limit and second-level latency is acceptable.