> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.fynapse.app/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.fynapse.app/_mcp/server. # Ingesting Data via Ingestion Files vs via REST API > Learn about which mode of ingesting data into Fynapse is preferrable for you. ## Overview This article describes the supported options for sending data into Fynapse, the limits and constraints that apply to each, and guidance on which pathway to choose for a given workload, such as large volumes of data, continuous near-real-time feeds, and high-rate streaming. > **Info** > > Please note that this article covers **inbound** integration only, i.e. getting data into Fynapse. ## Inbound Integration Options Fynapse offers two inbound channels. Both ingestion pathways yield the same result, i.e. records are validated and published to relevant Entities using the same deduplication key. The difference is the ingestion mechanism and its operational characteristics. | Channel | How data is submitted | Best suited to | | ------------------ | --------------------------------------------------------------- | -------------------------------------------------------------------- | | **REST API** | `POST` JSON payloads through the API gateway | Continuous / near-real-time feeds, small and interactive submissions | | **File ingestion** | Upload files (CSV, optionally gzip-compressed) to cloud storage | Bulk loads, backfills, very large datasets | Because both channels produce to Kafka with the same de-duplication key (derived from the entity's primary-key fields plus its creation time), **re-sending the same record is idempotent** on either channel — a duplicate submission will not create a duplicate record downstream. ## Limits and Constraints ### REST API | Constraint | Value | Notes | | -------------------- | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | Maximum request body | **1 MB** | Enforced at the API gateway. Larger requests are rejected before reaching the service. | | Rate limit | **5 requests/second** | Enforced at the API gateway, applied across all callers of the ingestion service. Higher tiers can be provisioned on request. | | Processing model | **Synchronous** | The request blocks until every row is validated and acknowledged by the system, then returns the result. | | Session lifetime | **15 minutes** (default, configurable per session) | A session expires a fixed duration after it is opened (see [Session lifecycle](#session-lifecycle)). | > **Warning** > > Please note that the effective REST size limit is 1 MB. > The system can technically accept much larger requests, but the API gateway > caps request bodies at **1 MB**. Design REST integrations around this figure — > do not rely on larger single requests. ### File Ingestion | Constraint | Value | Notes | | --------------------------------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Supported formats | **CSV**, **gzip-compressed CSV (`.gz`)** | Gzip is detected and decompressed automatically. | | Maximum size of an uncompressed CSV file | **5 GB** | | | Maximum file size for an Entity with automatic sorting | **2 GB** | Applies to the file as received, i.e. the compressed size of a gzip file. | | Maximum uncompressed data in a gzip file or a sorted file | **About 30 GB** | Fynapse decompresses and sorts files in temporary space that files of different Entities share. See [File Size Limits](/documentation/data-integration/data-input-file-ingestion/overview#file-size-limits). | | Maximum file size uploaded via the Fynapse UI | **100 MB** | Place larger files in cloud storage. | | Detection latency | **\~5 seconds** | New files are picked up by polling cloud storage. | > **Use gzip for large files** > > Gzip reduces transfer size and cost, improves resilience to network interruptions, > and raises the data ceiling from 5 GB for an uncompressed CSV file to about 30 GB > of uncompressed data. **Prefer gzip for all large files.** ## Choosing the Pathway | Workload | Recommended channel | Design around | | ------------------------------------------------------------------------------------------------ | ------------------------------ | -------------------------------------- | | **Bulk load / backfill** (large one-off or periodic datasets, into the millions of rows) | **File** (gzip-compressed CSV) | File size limits (see above) | | **Continuous / near-real-time** (records sent as they occur in the source system) | **REST**, micro-batched | 5 req/sec rate limit + session rolling | | **High-rate continuous** (event rate exceeds the REST rate limit; seconds of latency acceptable) | **Frequent small file drops** | \~5 second detection latency | ### Large Data Volumes For large datasets — backfills, migrations, or periodic large batches — use the **file channel with gzip-compressed CSV**. This path is designed for volume: * Files are streamed and processed in chunks rather than loaded whole into memory. * Failed files can be re-submitted simply by re-uploading; de-duplication keeps the operation idempotent. * Uploads are decoupled from processing, so there is no long-lived synchronous connection to fail mid-transfer. The REST API is not recommended for large data volumes: the 1 MB request cap and 5 req/sec rate limit make it slow for large volumes, and the synchronous model means a network interruption fails the entire in-flight request. ### Continuous / near-real-time Feeds When data is sent as it appears in the source system rather than as a large batch, the **REST API is the recommended channel**. This is where REST is most recommended: * Payloads are small, so the 1 MB limit is not a concern. * Records reach the system in near-real-time — lower latency than the file channel's polling interval allows. * The synchronous response confirms each submission reached the system and returns per-row validation errors immediately. * Idempotent de-duplication makes the channel safe for sources that may re-emit or replay events (e.g. change-data-capture feeds). Two practices are essential for a continuous REST integration: #### Micro-batch submissions The governing constraint for a continuous feed is the 5 requests/second rate limit, not payload size. Sending one request per event will be throttled if the source emits faster than \~5 events/second. Instead, accumulate events over a short window (e.g. 200 ms–1 s) and submit them as a single request containing multiple rows. A single sub-1 MB request can carry thousands of rows, which keeps throughput high and latency low while staying well within the rate limit. If sustained event rates exceed what micro-batching within the rate limit allows, request a higher rate-limit tier or switch to frequent file drops. #### Roll Ingestion Sessions A continuous feed cannot hold a single session open indefinitely — see [Session lifecycle](#session-lifecycle). Open a fresh session before the current one expires, submit micro-batches to the active session, and allow expired sessions to close. Alternatively, open sessions with a longer configured timeout, though rolling sessions is the cleaner pattern for a long-running feed. ### High-rate Continuous Feeds If the event rate consistently exceeds what the REST rate limit allows and a latency of a few seconds is acceptable, frequent small file drops are a good middle ground: better resilience and no rate limiting, at the cost of the \~5 second detection latency and per-file overhead. ## Session Lifecycle REST submissions occur within an ingestion session: 1. Open a session for a namespace + entity; this returns an `ingestionId`. 2. Submit one or more data requests against that `ingestionId`. 3. Close the session explicitly, or let it expire automatically. A session expires a fixed duration after it is opened (default 15 minutes, configurable per session at open time). Expiry is based on elapsed time since the session was opened — an actively-used session is still closed once its timeout is reached, after which further submissions to that `ingestionId` are rejected. This is why continuous feeds must [roll sessions](#roll-ingestion-sessions). ## Reliability and Idempotency Both channels share the same reliability model at the system boundary; the file channel adds further resilience features around file handling. | Feature | REST | File | | --------------------------------------------------------------------- | ------------------------------------ | ---------------------------- | | Idempotent de-duplication (primary-key based) | ✓ | ✓ | | Per-row validation with error reporting | ✓ (returned in response) | ✓ (written to an error file) | | Partial-failure isolation (valid rows proceed, invalid rows reported) | ✓ | ✓ | | Re-submission on failure | Resend failed rows from the response | Re-upload the file | | Integrity (checksum) validation | — | ✓ (checksum on receipt) | | Automatic retries on transient failure | — (caller resubmits) | ✓ | | Progress and outcome visibility in the Fynapse UI | ✓ | ✓ (incl. file-level state) | For REST, a submission returns a per-row breakdown: successes are recorded, and any failed rows are returned with their error messages. The recommended recovery pattern is to **resubmit only the failed rows** — de-duplication makes this safe. ## Monitoring and Visibility Both channels report progress and outcomes to the **Processing Summary** screen in the Fynapse UI, so you can track ingestion. The view is near-real-time — it refreshes continuously as processing progresses. The **Processing Summary** screen covers both REST and file ingestion, distinguished by source, and shows: * Overall processing status per ingestion (**in progress** → **ready** → **done**), with a clear indication when an ingestion has errored. * Per-Entity breakdown — for each target Entity in an ingestion, the number of records stored (succeeded) and failed. * Row-level counts — successful vs. failed records. For file ingestion, the screen additionally surfaces file-level state tracking: * Counts of **pending**, **processed**, and **errored** files, plus **reference** (master-data) files. * Each file resolves to a status of **pending**, **done**, or **errored**, so a failed file is immediately visible and can be corrected and re-uploaded. For REST ingestion, the screen surfaces submission counts and the total number of rows ingested. > **Tip** > > The same progress and completeness data is also available through the statistics > service's REST API, so integrations that need to monitor ingestion > programmatically — for example, to confirm a large volume upload completed before triggering a > downstream step — can poll it directly rather than scraping the UI. ## Best-practice Summary ### Large Data Volumes * Use the file channel with **gzip-compressed CSV**. * Recover by **re-uploading**; rely on de-duplication for idempotency. * Reconcile against the generated error file rather than assuming all-or-nothing. ### Continuous / near-real-time Feeds (REST) * **Micro-batch** events into sub-1 MB requests to stay within the 5 req/sec limit. * **Roll ingestion sessions** ahead of expiry. * Handle partial-success responses; **resubmit only failed rows**. * Confirm the rate limit fits your peak event rate; request a higher tier if needed. ### High-rate Continuous Feeds * Use **frequent small file drops** when the event rate exceeds the REST rate limit and second-level latency is acceptable.