> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://docs.fynapse.app/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.fynapse.app/_mcp/server. # Different Pathways for Extracting Data > Learn about which mode of extracting data from Fynapse is preferrable for you. ## Fynapse Data Extraction — Choosing an Integration Method This article describes the supported options for extracting data out of Fynapse into a downstream system (data warehouse, ERP, data lake, reporting tool, or an operational service). It compares the available methods and recommends one based on your data volume, download frequency, and target-system type. > **Info** > > Please note that this article covers outbound integration only, i.e. getting data out of Fynapse. --- ## The Four Pathways at a Glance Every extraction is driven by an extract configuration. The mode of data delivery is determined by the chosen Target setting, which yields four integration methods: | | A. Object Storage | B. Pageable JSON | C. Streamed JSONL | D. Webhook | | -------------- | ----------------------------------------- | ---------------------------------------- | ------------------------------------------ | ----------------------------------------------------- | | Delivery mode | Fynapse pushes files | Client pulls page by page | Client pulls one stream | Fynapse pushes to your HTTP endpoint | | Target | `CLOUD_STORAGE` | `REST_API` | `REST_API` | `WEBHOOK` | | Output | CSV files in S3 / Azure Blob | JSON pages (`data`, `next`, `prev`) | JSON Lines file (`data.jsonl`) | HTTP callbacks to your endpoint | | Who schedules | Fynapse (schedule or manual trigger) | The client, on its own cadence | The client, on its own cadence | Fynapse (schedule or manual trigger) | | Interaction | Asynchronous / batch (minutes) | Synchronous, immediate, resumable | Synchronous, held open until complete | Event-driven, near-real-time | | Typical volume | Large → very large | Small → medium per call | Medium → large in one pull | Continuous / incremental flow | | Sweet spot | Bulk & scheduled ETL, warehouse/ERP loads | Controlled, resumable, incremental pulls | One-shot bulk pull into a streaming loader | Event-driven downstreams that expose an HTTP receiver | ### Constraints for each Pathway * `CLOUD_STORAGE` is push-only: run by Fynapse's scheduler or a trigger, then you collect the files. Supports re-runs. * `REST_API` is pull-only: the you own the cadence. These configs cannot be scheduled, triggered, or re-run by Fynapse. * `WEBHOOK` is push-only: run by Fynapse's scheduler or a trigger; Fynapse calls your registered HTTP endpoint. Requires you to expose a receiver. * `WEBHOOK` delivery is currently available only for extracts with Balances set as data source. For other types of extracts, you can setup a webhook subscription with [notifications](/api-reference/getting-started/webhooks/event-types#extract-upload---extractuploadcompleted-event) about completed extraction. Additionally: * Incremental delivery (only new/changed data per cycle) is available in all four; the REST_API methods use an offset you acknowledge after a successful downstream write (at-least-once). For the pull methods, incremental only advances once you explicitly acknowledge a cycle using the [Increments the saved offsets for an incremental extract](/api-reference/api-reference/data-extraction/increment) endpoint — until then the same window is re-served, so nothing is skipped if a downstream write fails. * REST_API applies a bounded lookback on time-series/date-partitioned data: roughly the last 12 months for row-level extracts and about 24 months for aggregated extracts. If you need older history, use Object Storage, which loads full history with no such cap by default. * REST_API does not support balance queries (Pageable or Streamed). Balance data is available only via Object Storage (A) or Webhook (D). * Streamed JSONL (C) is a single, non-resumable read: if the connection drops partway you restart from the beginning. If you need to resume from where you left off or pull in controlled chunks, use Pageable JSON (B). --- ## Choose by Data Volume | Volume | Recommended Pathway | | ------------------------------------------------ | ------------------------------------------------------------------------------------- | | Millions of rows / full-table / historical loads | A — Object Storage ( CSV). Files, internal chunking, no per-call overhead, resumable. | | Large one-shot pull the client reads directly | C — Streamed JSONL. One connection, memory-flat consumption. | | Small → medium, or resumable/controlled reads | B — Pageable JSON. | > **Info** > > We recommend using pathway A for high data volume on a schedule. > > When you need to actively pull, use pathway C for a single large read and pathway B for controlled paging. ## Choose by Download Frequency | Frequency | Recommended Pathway | | ---------------------------------------------- | -------------------------------------------------------------------- | | Scheduled / recurring, server-driven | A — Fynapse runs the schedule; use incremental to ship deltas. | | Client-driven / on-demand (your own scheduler) | B or C — REST_API is pull-only, so you decide when. | | Frequent small deltas | B + incremental — page the new rows, then acknowledge. | | Event-driven / near-real-time | D — Webhook (push): register an HTTP endpoint and Fynapse calls you. | ## Choose by Target-system Type | Target system | Recommended Pathway | | -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | | Data warehouse / lake (Snowflake, BigQuery, Redshift, Databricks, S3/ADLS) | A — CSV → native bulk loader, or C — JSONL streamed into a loader. | | D365 F\&O — journals into the General Ledger | E — direct GL posting via Recurring Integrations (Fynapse submits to D365, with status tracking and drillback). | | ERP / finance, file-based load you manage (incl. Dynamics 365) | A — file-based extract (dedicated D365 file support). | | Custom microservice / operational app | B for controlled, resumable pulls; C for a bounded one-shot pull. | | BI / reporting tool | B — paginated JSON on demand. | | Event-driven / callback-based system | D — Webhook: Fynapse pushes to your HTTP endpoint as data becomes available. | --- ## Large Data Volumes and REST API A common question is whether the REST API (Methods B and C) is the right choice for very large data sets — into the millions of rows. The short answer: for one-off or periodic bulk movement at that scale, Object Storage (A) is preferrable, the REST methods are best kept for controlled, incremental, or moderate-sized pulls. What should be considered when dealing with high volume extractions: * Pageable JSON (B) pages get more resource-expensive the deeper you go. Paging is offset-based, so each successive page has to skip all the rows before it; requesting page *N* of a million-row set requires progressively more work. Very large page sizes also mean a bigger single response to build and hold in memory. Keep page sizes moderate and page through steadily rather than asking for one enormous page. * Streamed JSONL (C) is memory-flat but not resilient to interruption. It holds a single connection open for the whole result set, so it comfortably reads large volumes without paging — but if the connection drops partway (a network blip, a proxy/load-balancer idle timeout on a long-running request) there is no resume: you start again from the beginning. The larger the pull, the more likely an interruption becomes. * Bounded lookback still applies. REST extracts see roughly the last 12 months (row-level) or 24 months (aggregated) of time-series data, so a "full history" load of millions of rows may be truncated on REST regardless of paging. * No balance queries over REST (see the constraints above). Why files (A) are more resilient at this scale. With Object Storage, Fynapse does the heavy lifting server-side: the extract is chunked internally, written to files (CSV) via resumable multi-part upload, and the whole run can be re-triggered and re-run if something goes wrong. There is no deep-offset cost and no single long-lived connection to lose. This is the resilient, re-submittable path for millions of rows. Scheduled and triggered methods (A, and D / E) are driven by Fynapse's scheduler, which supports both recurring (cron-style) schedules and one-off runs and automatically retries a run that fails. Execution is at-least-once — in rare cases (e.g. a retry after an ambiguous failure) a run can repeat — so keep the downstream idempotent (for files, key on the run/file identity; for webhooks, on the `Idempotency-Key`). By contrast, the REST methods are client-driven: there is no server-side schedule or retry, so your side owns the cadence and any recovery. Best practices if you do use REST at higher volumes: * Prefer B (pageable) over C (streamed) when the network is unreliable — paging is resumable (re-request the page you were on), a stream is not. * Use incremental + acknowledge: pull a window, write it downstream, then acknowledge to advance the offset. The window is re-served until you acknowledge, so a failed downstream write never silently skips data — this is the REST equivalent of "re-submittability". * Keep page sizes moderate and consistent; don't try to drain millions of rows in one call. * For a genuine one-shot bulk load of the full set, switch to A (Object Storage) rather than deep-paging the REST API. --- ## Method D — Webhook (event-driven push) In methods A–C above your integration initiates — you run a schedule, or you pull. The Webhook method is different: it is a continuous, event-driven push. Instead of asking Fynapse for data, you register an HTTP endpoint and Fynapse calls you as extracted data becomes available. This is the right choice when you want data to flow with minimal latency and without any polling loop on your side. Current implementation covers two areas: * Webhook data delivery (Target: `WEBHOOK`). Extracted records are pushed to your registered HTTP endpoint one event at a time as small JSON messages (not batched files). Choose this when your target system exposes an HTTP receiver and you'd rather be called than run a scheduled job or a pull loop. Webhook data delivery is currently available only for extracts with Balances as the data source. * Availability notifications (a signal, not the data). For Object Storage (Method A) extracts, Fynapse can additionally emit a `extract.upload.completed` webhook event once a file has been written to S3/Azure and is ready to collect. This lets an object-storage integration behave like an event-driven one: instead of polling extract-log status, your system reacts to the "file is available" notification and then fetches the file. It is an at-least-once signal keyed for idempotency, so design your consumer to de-duplicate. Characteristics that drive the choice | Aspect | Behaviour | | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- | | Direction | Push — Fynapse calls your HTTP endpoint; no client pull loop | | Trigger | Schedule- or trigger-driven, same pipeline as Object Storage | | Latency | Low — events are delivered as data becomes available | | Format | HTTP callback payloads to your registered endpoint | | Delivery guarantee | At-least-once, with automatic retries and backoff over an extended window; your receiver should be idempotent | | Failure handling | Events that remain undeliverable after retries are set aside rather than lost, so a temporarily unavailable receiver does not immediately drop data | | Security | Each callback is cryptographically signed, so your receiver can verify the request is genuinely from Fynapse before processing it | | Data scope | Currently `Balances` only for webhook data delivery | | Incremental | Supported — the same incremental flag as the other methods | | Payload | Individual data events, or an `extract.upload.completed` event for availability notifications | Before choosing Webhook, confirm your downstream can meet the receiver-side obligations — these are what make the method suitable (or not): * expose a secure, highly available HTTPS endpoint that accepts `POST` callbacks; * acknowledge quickly (return success only once the event is safely stored or buffered); * de-duplicate repeated events, since at-least-once delivery means an event can arrive more than once; * verify the request signature so it only trusts genuine Fynapse callbacks. If you cannot host such an endpoint, we recommend using the pull method (B / C) or file-based delivery (A). When to choose Webhook subscriptions: * You need near-real-time delivery rather than batch files or periodic pulls. * Your downstream exposes an HTTP endpoint and you'd rather receive callbacks than run a scheduled job or a pull loop. * You already use Object Storage but want to stop polling for readiness — enable availability notifications and react to `extract.upload.completed`. * The data you need is Balances (the data type currently available for webhook data delivery). When not to choose Webhook subscriptions: * You want a self-contained file to bulk-load into a warehouse → use A (Object Storage). * You want your system to decide exactly when and how much to fetch → use B / C (REST pull). * You cannot expose a secure, always-on HTTP receiver → use a pull method instead. * You need a data type other than Balances delivered as events → not yet supported over webhook data delivery; use another method (availability notifications on an Object Storage extract give a similar event-driven feel for those domains). --- ## Method E — Direct D365 Finance & Operations GL posting (specialised) Methods A–D deliver data that you then load to the storage of your choice. If your target is Microsoft Dynamics 365 Finance & Operations (F\&O) and you want Journals posted straight into the General Ledger, Fynapse offers a dedicated, purpose-built integration that goes one step further: instead of handing you a file to load yourself, it submits your Journal data directly into D365 F\&O using D365's standard Recurring Integrations mechanism. This is a specialised, target-specific option — not a general-purpose delivery mode — so it is scoped to what the D365 General Ledger needs. Characteristics that drive the choice | Aspect | Behaviour | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Direction | Push, server-driven — you trigger it (Run Now) or schedule it in Fynapse; Fynapse prepares and submits the data to your D365 environment | | Destination | Posted directly into the D365 F\&O General Ledger via Recurring Integrations — no intermediate file for you to collect and load | | Data scope | Journals / GL postings only | | Status tracking | Each submission is tracked through to a final D365 processing state, with automatic re-submission of failed batches and the ability to retrieve what was sent for manual recovery | | Drillback | From a posted D365 voucher, users can navigate back to the originating Fynapse journal lines | | Prerequisite | A configured D365 F\&O environment (with a matching Recurring Integrations data job) that Fynapse is authorised to submit to | When to choose: * Your target is D365 F\&O and you want Journals posted into the General Ledger automatically, rather than exporting generic files and building your own D365 load process. * You want end-to-end tracking of each posting through to D365's final processing state, plus drill-back from D365 vouchers to the underlying Fynapse Journals. When not to choose: * Your target is not D365 F\&O, or you need non-Journal data → use A / B / C. * You want D365-compatible files but prefer to manage the load into D365 yourself → use A (Object Storage) with its file-based D365 support. --- ## Quick Summary * Bulk, scheduled, warehouse/ERP → A (Object Storage). Fynapse does the work and drops files; you load them. * Large one-shot client pull → C (Streamed JSONL). Read the stream straight into your loader. * Controlled or incremental client pull → B (Pageable JSON). You page and acknowledge at your own pace. * Near-real-time / event-driven → D (Webhook). Fynapse calls your registered HTTP endpoint as data becomes available — no polling. You must host a secure, always-on receiver that de-duplicates and verifies signatures. Webhook data delivery currently covers `Balances` only; object-storage users can also enable `extract.upload.completed` availability notifications to react instead of poll for other domains. * Journals into the D365 F\&O General Ledger → E (Direct D365 GL posting). Fynapse submits journals straight into D365 via Recurring Integrations, tracks each posting to a final state, and supports drillback from D365 vouchers back to Fynapse journals — no file to load yourself.