Different Pathways for Extracting Data

Fynapse Data Extraction — Choosing an Integration Method

This article describes the supported options for extracting data out of Fynapse into a downstream system (data warehouse, ERP, data lake, reporting tool, or an operational service). It compares the available methods and recommends one based on your data volume, download frequency, and target-system type.

Please note that this article covers outbound integration only, i.e. getting data out of Fynapse.

The Four Pathways at a Glance

Every extraction is driven by an extract configuration. The mode of data delivery is determined by the chosen Target setting, which yields four integration methods:

	A. Object Storage	B. Pageable JSON	C. Streamed JSONL	D. Webhook
Delivery mode	Fynapse pushes files	Client pulls page by page	Client pulls one stream	Fynapse pushes to your HTTP endpoint
Target	`CLOUD_STORAGE`	`REST_API`	`REST_API`	`WEBHOOK`
Output	CSV files in S3 / Azure Blob	JSON pages (`data`, `next`, `prev`)	JSON Lines file (`data.jsonl`)	HTTP callbacks to your endpoint
Who schedules	Fynapse (schedule or manual trigger)	The client, on its own cadence	The client, on its own cadence	Fynapse (schedule or manual trigger)
Interaction	Asynchronous / batch (minutes)	Synchronous, immediate, resumable	Synchronous, held open until complete	Event-driven, near-real-time
Typical volume	Large → very large	Small → medium per call	Medium → large in one pull	Continuous / incremental flow
Sweet spot	Bulk & scheduled ETL, warehouse/ERP loads	Controlled, resumable, incremental pulls	One-shot bulk pull into a streaming loader	Event-driven downstreams that expose an HTTP receiver

Constraints for each Pathway

CLOUD_STORAGE is push-only: run by Fynapse’s scheduler or a trigger, then you collect the files. Supports re-runs.
REST_API is pull-only: the you own the cadence. These configs cannot be scheduled, triggered, or re-run by Fynapse.
WEBHOOK is push-only: run by Fynapse’s scheduler or a trigger; Fynapse calls your registered HTTP endpoint. Requires you to expose a receiver.
WEBHOOK delivery is currently available only for extracts with Balances set as data source. For other types of extracts, you can setup a webhook subscription with notifications about completed extraction.

Additionally:

Incremental delivery (only new/changed data per cycle) is available in all four; the REST_API methods use an offset you acknowledge after a successful downstream write (at-least-once). For the pull methods, incremental only advances once you explicitly acknowledge a cycle using the Increments the saved offsets for an incremental extract endpoint — until then the same window is re-served, so nothing is skipped if a downstream write fails.
REST_API applies a bounded lookback on time-series/date-partitioned data: roughly the last 12 months for row-level extracts and about 24 months for aggregated extracts. If you need older history, use Object Storage, which loads full history with no such cap by default.
REST_API does not support balance queries (Pageable or Streamed). Balance data is available only via Object Storage (A) or Webhook (D).
Streamed JSONL (C) is a single, non-resumable read: if the connection drops partway you restart from the beginning. If you need to resume from where you left off or pull in controlled chunks, use Pageable JSON (B).

Choose by Data Volume

Volume	Recommended Pathway
Millions of rows / full-table / historical loads	A — Object Storage ( CSV). Files, internal chunking, no per-call overhead, resumable.
Large one-shot pull the client reads directly	C — Streamed JSONL. One connection, memory-flat consumption.
Small → medium, or resumable/controlled reads	B — Pageable JSON.

We recommend using pathway A for high data volume on a schedule.

When you need to actively pull, use pathway C for a single large read and pathway B for controlled paging.

Choose by Download Frequency

Frequency	Recommended Pathway
Scheduled / recurring, server-driven	A — Fynapse runs the schedule; use incremental to ship deltas.
Client-driven / on-demand (your own scheduler)	B or C — REST_API is pull-only, so you decide when.
Frequent small deltas	B + incremental — page the new rows, then acknowledge.
Event-driven / near-real-time	D — Webhook (push): register an HTTP endpoint and Fynapse calls you.

Choose by Target-system Type

Target system	Recommended Pathway
Data warehouse / lake (Snowflake, BigQuery, Redshift, Databricks, S3/ADLS)	A — CSV → native bulk loader, or C — JSONL streamed into a loader.
D365 F&O — journals into the General Ledger	E — direct GL posting via Recurring Integrations (Fynapse submits to D365, with status tracking and drillback).
ERP / finance, file-based load you manage (incl. Dynamics 365)	A — file-based extract (dedicated D365 file support).
Custom microservice / operational app	B for controlled, resumable pulls; C for a bounded one-shot pull.
BI / reporting tool	B — paginated JSON on demand.
Event-driven / callback-based system	D — Webhook: Fynapse pushes to your HTTP endpoint as data becomes available.

Large Data Volumes and REST API

A common question is whether the REST API (Methods B and C) is the right choice for very large data sets — into the millions of rows. The short answer: for one-off or periodic bulk movement at that scale, Object Storage (A) is preferrable, the REST methods are best kept for controlled, incremental, or moderate-sized pulls.

What should be considered when dealing with high volume extractions:

Pageable JSON (B) pages get more resource-expensive the deeper you go. Paging is offset-based, so each successive page has to skip all the rows before it; requesting page N of a million-row set requires progressively more work. Very large page sizes also mean a bigger single response to build and hold in memory. Keep page sizes moderate and page through steadily rather than asking for one enormous page.
Streamed JSONL (C) is memory-flat but not resilient to interruption. It holds a single connection open for the whole result set, so it comfortably reads large volumes without paging — but if the connection drops partway (a network blip, a proxy/load-balancer idle timeout on a long-running request) there is no resume: you start again from the beginning. The larger the pull, the more likely an interruption becomes.
Bounded lookback still applies. REST extracts see roughly the last 12 months (row-level) or 24 months (aggregated) of time-series data, so a “full history” load of millions of rows may be truncated on REST regardless of paging.
No balance queries over REST (see the constraints above).

Why files (A) are more resilient at this scale. With Object Storage, Fynapse does the heavy lifting server-side: the extract is chunked internally, written to files (CSV) via resumable multi-part upload, and the whole run can be re-triggered and re-run if something goes wrong. There is no deep-offset cost and no single long-lived connection to lose. This is the resilient, re-submittable path for millions of rows.

Scheduled and triggered methods (A, and D / E) are driven by Fynapse’s scheduler, which supports both recurring (cron-style) schedules and one-off runs and automatically retries a run that fails. Execution is at-least-once — in rare cases (e.g. a retry after an ambiguous failure) a run can repeat — so keep the downstream idempotent (for files, key on the run/file identity; for webhooks, on the Idempotency-Key). By contrast, the REST methods are client-driven: there is no server-side schedule or retry, so your side owns the cadence and any recovery.

Best practices if you do use REST at higher volumes:

Prefer B (pageable) over C (streamed) when the network is unreliable — paging is resumable (re-request the page you were on), a stream is not.
Use incremental + acknowledge: pull a window, write it downstream, then acknowledge to advance the offset. The window is re-served until you acknowledge, so a failed downstream write never silently skips data — this is the REST equivalent of “re-submittability”.
Keep page sizes moderate and consistent; don’t try to drain millions of rows in one call.
For a genuine one-shot bulk load of the full set, switch to A (Object Storage) rather than deep-paging the REST API.

Method D — Webhook (event-driven push)

In methods A–C above your integration initiates — you run a schedule, or you pull. The Webhook method is different: it is a continuous, event-driven push. Instead of asking Fynapse for data, you register an HTTP endpoint and Fynapse calls you as extracted data becomes available. This is the right choice when you want data to flow with minimal latency and without any polling loop on your side.

Current implementation covers two areas:

Webhook data delivery (Target: WEBHOOK). Extracted records are pushed to your registered HTTP endpoint one event at a time as small JSON messages (not batched files). Choose this when your target system exposes an HTTP receiver and you’d rather be called than run a scheduled job or a pull loop. Webhook data delivery is currently available only for extracts with Balances as the data source.
Availability notifications (a signal, not the data). For Object Storage (Method A) extracts, Fynapse can additionally emit a extract.upload.completed webhook event once a file has been written to S3/Azure and is ready to collect. This lets an object-storage integration behave like an event-driven one: instead of polling extract-log status, your system reacts to the “file is available” notification and then fetches the file. It is an at-least-once signal keyed for idempotency, so design your consumer to de-duplicate.

Characteristics that drive the choice

Aspect	Behaviour
Direction	Push — Fynapse calls your HTTP endpoint; no client pull loop
Trigger	Schedule- or trigger-driven, same pipeline as Object Storage
Latency	Low — events are delivered as data becomes available
Format	HTTP callback payloads to your registered endpoint
Delivery guarantee	At-least-once, with automatic retries and backoff over an extended window; your receiver should be idempotent
Failure handling	Events that remain undeliverable after retries are set aside rather than lost, so a temporarily unavailable receiver does not immediately drop data
Security	Each callback is cryptographically signed, so your receiver can verify the request is genuinely from Fynapse before processing it
Data scope	Currently `Balances` only for webhook data delivery
Incremental	Supported — the same incremental flag as the other methods
Payload	Individual data events, or an `extract.upload.completed` event for availability notifications

Before choosing Webhook, confirm your downstream can meet the receiver-side obligations — these are what make the method suitable (or not):

expose a secure, highly available HTTPS endpoint that accepts POST callbacks;
acknowledge quickly (return success only once the event is safely stored or buffered);
de-duplicate repeated events, since at-least-once delivery means an event can arrive more than once;
verify the request signature so it only trusts genuine Fynapse callbacks.

If you cannot host such an endpoint, we recommend using the pull method (B / C) or file-based delivery (A).

When to choose Webhook subscriptions:

You need near-real-time delivery rather than batch files or periodic pulls.
Your downstream exposes an HTTP endpoint and you’d rather receive callbacks than run a scheduled job or a pull loop.
You already use Object Storage but want to stop polling for readiness — enable availability notifications and react to extract.upload.completed.
The data you need is Balances (the data type currently available for webhook data delivery).

When not to choose Webhook subscriptions:

You want a self-contained file to bulk-load into a warehouse → use A (Object Storage).
You want your system to decide exactly when and how much to fetch → use B / C (REST pull).
You cannot expose a secure, always-on HTTP receiver → use a pull method instead.
You need a data type other than Balances delivered as events → not yet supported over webhook data delivery; use another method (availability notifications on an Object Storage extract give a similar event-driven feel for those domains).

Method E — Direct D365 Finance & Operations GL posting (specialised)

Methods A–D deliver data that you then load to the storage of your choice. If your target is Microsoft Dynamics 365 Finance & Operations (F&O) and you want Journals posted straight into the General Ledger, Fynapse offers a dedicated, purpose-built integration that goes one step further: instead of handing you a file to load yourself, it submits your Journal data directly into D365 F&O using D365’s standard Recurring Integrations mechanism.

This is a specialised, target-specific option — not a general-purpose delivery mode — so it is scoped to what the D365 General Ledger needs.

Characteristics that drive the choice

Aspect	Behaviour
Direction	Push, server-driven — you trigger it (Run Now) or schedule it in Fynapse; Fynapse prepares and submits the data to your D365 environment
Destination	Posted directly into the D365 F&O General Ledger via Recurring Integrations — no intermediate file for you to collect and load
Data scope	Journals / GL postings only
Status tracking	Each submission is tracked through to a final D365 processing state, with automatic re-submission of failed batches and the ability to retrieve what was sent for manual recovery
Drillback	From a posted D365 voucher, users can navigate back to the originating Fynapse journal lines
Prerequisite	A configured D365 F&O environment (with a matching Recurring Integrations data job) that Fynapse is authorised to submit to

When to choose:

Your target is D365 F&O and you want Journals posted into the General Ledger automatically, rather than exporting generic files and building your own D365 load process.
You want end-to-end tracking of each posting through to D365’s final processing state, plus drill-back from D365 vouchers to the underlying Fynapse Journals.

When not to choose:

Your target is not D365 F&O, or you need non-Journal data → use A / B / C.
You want D365-compatible files but prefer to manage the load into D365 yourself → use A (Object Storage) with its file-based D365 support.

Quick Summary

Bulk, scheduled, warehouse/ERP → A (Object Storage). Fynapse does the work and drops files; you load them.
Large one-shot client pull → C (Streamed JSONL). Read the stream straight into your loader.
Controlled or incremental client pull → B (Pageable JSON). You page and acknowledge at your own pace.
Near-real-time / event-driven → D (Webhook). Fynapse calls your registered HTTP endpoint as data becomes available — no polling. You must host a secure, always-on receiver that de-duplicates and verifies signatures. Webhook data delivery currently covers Balances only; object-storage users can also enable extract.upload.completed availability notifications to react instead of poll for other domains.
Journals into the D365 F&O General Ledger → E (Direct D365 GL posting). Fynapse submits journals straight into D365 via Recurring Integrations, tracks each posting to a final state, and supports drillback from D365 vouchers back to Fynapse journals — no file to load yourself.