Flow Overview

Overview

This section provides information about Flow, which allows you to create custom processing within Fynapse framework.

Fynapse Flow allows you to compose your own integrated Fynapse processes, i.e. Flows, comprising of lookups, calculation rules, native Fynapse processes such as the Accounting Engine processing and Journal processing, etc.

By creating process Flows that combine native Fynapse services with configurable or custom rules, you can configure the Fynapse platform to meet the specific finance automation and data consumption needs of your organization with the fastest possible time-to-value.

How Does It Work?

A Fynapse Flow is a scalable process flow.

• Each Flow is comprised of a series of user-defined processing “steps”
• The steps are chained together via a visual flow diagram that has a defined start and an end
• Each step performs some form of data processing or performs some logic that determines the next action in the Flow

Sample Flow

This flow has five steps, which in this case have been arranged to run one after the other (visualized through the links with arrows):

1. Input: Denotes the start of Flow processing. Defines a Transactional Data Entity (TDE) to be processed by Flow. Typically the TDE data is sourced from the Inbound CSV file that is submitted to Fynapse. Assuming that a flat TDE is used then every row from a CSV file will be read and then subsequently processed by Flow. The record produced by Input step is made available for all the remaining steps in the rule to use.
2. Script step (enrichment lookup): A user-configured script step using Python rules to perform lookup - matching some attributes from the input record with records in a Reference Data Entity to extract another value from the matched Reference Data record. The value may be used in future processing steps.
3. Script step: A user-configured script step using Python rules to perform different operations on data, e.g. allocations, deferrals, etc.
4. Accounting Engine: The Business Event, which has now been enriched with a lookup value (from step 2) and a calculation value (from step 3), is passed into the Accounting Engine to generate Journals.
5. Journal Processing: Fynapse Journal processing, preparing Journals for extraction to the General Ledger.

Processing Fundamentals

Flow uses parallel processing to generate results as quickly and efficiently as possible. When a set of input records arrives for processing through a Flow, Fynapse starts up many instances of the Flow to process many records in parallel. As a consequence of this efficient processing, the input records are not guaranteed to be processed sequentially and output records are not guaranteed to be generated in any particular order.

Likewise, the steps of each Flow are only guaranteed to be processed sequentially whenever the Flow graph has a link between them denoting that there is a dependency between these steps.

Whenever there is a step that is followed by multiple subsequent steps there is no guarantee which successor step will be executed first. Any branch taken during execution needs to be completed first.

Below is the list of all possible execution orders:

• (a) Input → S1 → T1
• (b) Input → S2
  • (c) → S2-1 → T2
  • (d) → S2-2 → T3

In this case we have 4 possible execution orders:

1. a b c d
2. a b d c
3. b c d a
4. b d c a

Multiple Flows Working on the Same Input

It is possible to configure multiple flows that operate on the same input (the same Transaction Data Entity). When this happens, the steps from all the flows that operate on the same input will be combined, following the principles described above.

Input Record Persistence

Once all Flow executions for a given Input record are completed the Input is then sent to Finance Data Service.

If Finance Data Service is persisted for a given Input Data Entity then it will be stored in a database and available for querying.

Target Records Tracing

All target records are stamped with the original Ingestion Id of the input record. This can be used to trace back the original ingestion from the resulting Flow outputs (including Journals).

Errors and Reprocessing

If there is an error reported during Flow Processing, then no target is populated with data and the original Input record is sent to the Error Management.

This can be caused by various errors listed in the Consolidated Error List.

Whenever a record is sent for reprocessing it will be processed by the latest published version of the Flow, which means that previously reported errors can be fixed by adjusting the Flow definition (where appropriate).

Flow processing is successful whenever all of the targets (including Accounting Engine) are populated by the data created from input record. If an error occurs during processing of the Business Event generated by Flow, then the Business Event is sent to Error Management, where it can be reprocessed.

Input

The input into a Flow can be:

• A Transaction Data Entity
• A Reference Data Entity
• A transient Entity

defined in the Input step.

Transaction Data and Reference Data Entities are Defined Entities, with a structure consisting of attributes, with Transaction Data Entities being similar to Business Event Entity or Journals Entity and Reference Data Entities comprising of reference data used for enrichment and validations.

Transient Entities, are Entities created by an extract. They can be both Transaction Data and Reference Data Entities. For more details, refer to Extract Configuration.

Extract only incremental data is always set by default to avoid exponential increase of extracted data, as only increments are extracted after initial extract to avoid duplication of records.

It is set by default for all extracts apart from Balances.

Below is a sample structure of a Transaction Data Entity:

1
"entities": [
2
   {
3
      "namespace": "fynapse",
4
      "name": "Transaction Data",
5
      "temporalityType": "Transaction",
6
      "description": "This is a Transaction Data Entity",
7
      "type": "DefinedEntity",
8
      "attributes": [
9
         {
10
            "label": "Event Type Label",
11
            "name": "eventTypeName",
12
            "type":{
13
               "typeName": "PrimitiveType",
14
               "primitive": "TEXT"
15
            }
16
         },
17
         {
18
            "label": "Effective Date Label",
19
            "name": "effectiveDateName",
20
            "type":{
21
               "typeName": "PrimitiveType",
22
               "primitive": "DATE"
23
            }
24
         },
25
         {
26
            "label": "Amount Type Label",
27
            "name": "amountTypeName",
28
            "type":{
29
               "typeName": "PrimitiveType",
30
               "primitive": "DECIMAL"
31
            }
32
         },
33
         {
34
            "label": "Currency Label",
35
            "name": "currencyName",
36
            "type": {
37
               "typeName": "PrimitiveType",
38
               "primitive": "TEXT"
39
            }
40
         },
41
         {
42
            "label": "Custom Subledger Node Label",
43
            "name": "customSubledgerNodeName",
44
            "type":{
45
               "typeName": "PrimitiveType",
46
               "primitive": "TEXT"
47
            }
48
         }
49
      ]
50
   }
51
]

As you can see from the beginning of the definition:

1
"entities": [
2
      {
3
            "namespace": "fynapse",
4
            "name": "Transaction Data",
5
            "temporalityType": "Transaction",
6
            "description": "This is a Transaction Data Entity",
7
            "type": "DefinedEntity",
8
            "attributes"

the Transaction Data Entities are configured in the “entities” section of the Configuration Data JSON file, alongside other Defined Entities from the Data Structure. To configure a Transaction Data correctly, you need to ensure the properties are defined as follows:

• namespace - fynapse
• name - ensure you select a meaningful name that will allow you to easily identify the Transaction Data Entity
• temporalityType - what kind of data are contained in this Entity; for Transaction Data Entities this property has to be defined as “Transaction”
• description - optional, you can add an informative description for your Transaction Data Entity
• type - determines what type of an Entity this is; for Transaction Data Entities this property has to be defined as “DefinedEntity”
• attributes - in this section you will define the attributes for your Entity. For more information about types of attributes, refer to Data Structure Definition.

To learn about Reference Data Entities, refer to Data Structure.

Currently, Transaction Data and Reference Data Entities can only be defined in a JSON file. You can do it either in the Configuration Data JSON file with the entire configuration or separately in the Data Structure Definition JSON file.

Hierarchical Data in Flow

It is possible to process hierarchical data even if the data is submitted via CSV file. In this scenario, the Transaction Data Entity must have a single list attribute defined and a Primary Key on a root level.

Example Entity Definition:

1
{
2
   "namespace": "fynapse",
3
   "name": "InputWithList",
4
   "temporalityType": "Transaction",
5
   "type": "DefinedEntity",
6
   "primaryKeyAttributeNames": [
7
      "group"
8
   ],
9
   "attributes": [
10
      {
11
         "label": "Group",
12
         "name": "group",
13
         "type": {
14
            "typeName":"PrimitiveType",
15
            "primitive":"TEXT"
16
         },
17
         "mandatory":true
18
      },
19
      {
20
         "label":"Items",
21
         "name":"items",
22
         "type": {
23
            "typeName": "ListType",
24
            "itemType":{
25
               "typeName": "InlineEntityType",
26
               "definition": {
27
                  "name": "InputWithListItem",
28
                  "type": "DefinedEntity",
29
                  "attributes": [
30
                     {
31
                        "label": "ItemId",
32
                        "name": "itemId",
33
                        "type":{
34
                           "typeName": "PrimitiveType",
35
                           "primitive": "TEXT"
36
                        },
37
                        "mandatory": true
38
                     },
39
                     {
40
                        "label": "Value1",
41
                        "name": "value1",
42
                        "type":{
43
                           "typeName": "PrimitiveType",
44
                           "primitive": "DECIMAL"
45
                        }
46
                     },
47
                     {
48
                        "label": "Value2",
49
                        "name": "value2",
50
                        "type":{
51
                           "typeName": "PrimitiveType",
52
                           "primitive": "DECIMAL"
53
                        }
54
                     }
55
                  ]
56
               }
57
            }
58
         }
59
	}
60
]

With the Entity defined as above, the below CSV file can be submitted:

Group,Items.ItemId,Items.Value1,Items.Value2
A,ChildA1,10.0,3.0
A,ChildA2,100.0,0.3
B,ChildB1,57.1,30.2

For the above data Fynapse will create two records. If the aggregation of CSV rows is to be performed for multiple lines in the input CSV file, the lines need to be ordered by the value of the Primary Key attributes. Fynapse will emit a new record after it detects a change of these values.

This will create two instances of the InputWithList Transaction Data Entities that will have the following values:

{
  "group": "A",
  "items": [{
    "itemId": "ChildA1",
    "value1": 10.0,
    "value2": 3.0    
  }, {
    "itemId": "ChildA2",
    "value1": 100.0,
    "value2": 0.3    
  }]
}

{
  "group": "B",
  "items": [{
    "itemId": "ChildB1",
    "value1": 57.1,
    "value2": 30.2    
  }]
}

Following the example above, let’s assume we want to define a step that sums up a calculation - value1 * value2 - from each record in the hierarchy. To achieve this, we can use a script block with the following Python script:

1
# make the Decimal type available in the Script step
2
from decimal import *
3
4
# Initialize the sum variable to 0
5
sum = Decimal(0)
6
for item in input.items:
7
  # add the result of operation for each item in the items to the 
8
  sum = sum + item.value1 * item.value2
9
10
# round the result to 3 decimal places
11
sum = sum.quantize(Decimal('0.001'), rounding = ROUND_HALF_UP)
12
13
# publish "sum" variable and make it available to all subsequent steps
14
publish('sum', sum)

Output

A Flow can have two types of output:

• For Flows ending in the Journal Processor step, the output is a Journal
• For Flows ending in a Target step, the output is a transaction record stored in a Transaction Data Entity. This output can be later used as a source for an Input step in a different Flow.

Best Practices for Authoring Flows

Following is a list of best practices to author Flows:

• Break down Python scripts into multiple steps, if possible, and use variables to push the results between them. If there is an error during the processing of a Flow, the error message will include the Flow name, version and the step name where the error occurred.
• Document and comment your Python code. Make sure that any assumptions made when developing the Python code are properly documented as comments in the Scripts.
• Validate inputs. Any critical value needs to be inspected whether it is within the expected value range (e.g. value is not empty). If possible, the validations should be the first steps in the Flow.
• Clear error reporting. Make sure that the errors reported by Flow are sufficient enough to fix the data error, if needed. Include all the necessary information that would allow to easily identify the root cause of the problem.