One of the best practices mentioned before is to create an incremental sync between two application such that the data created after the last sync only is incrementally downloaded from the application.

Use Case

You want to sync / extract records from source application that have been created or modified after the last time the process ran.

Approach

To incrementally sync data, the one thing you need is to capture the last sync date. Ideally we recommend to get the highest sync date from the input packet such that we always store the correct date time irrespective of the time zone info which is required as well.

1. We create a variable in the ProcessFlow to capture the last date time present in the packet. We use the same date time stamp to query data greater than the last modified date value stored in the variable.
2. If we get data again, that means the data is incrementally retrieved. Recalculate the datetime present in the packet coming from the source application and replace the value of the variable.
3. Process the record as needed.
4. Upon completion of the proccess, the ProcessFlow will automatically capture the last value into storage.

Note: It is always recommended to capture the last modified date from the records itself, to ensure minor server date time adjustments does not hamper the incremental capture.

Consideration

• For the first execution time, we need to put a default value of the variable such that the historical data could be processed on first run. In APPSeCONNECT, the default value of a variable could be set which can automatically picked up if data not available.
• In case of errors midway of execution, the datetime will still be processed in APPSeCONNECT. APPSeCONNECT provides Sync Info section, which will allow you to track all the data present in the packet and allows you to rerun the same process again with a specific data.
• Idempotent data transfer is sometimes a challenge as well, in such a case, we use Duplicate check Idempotent rule in Mapper to remove duplicate data.

Steps to incrementally sync data

To create an integration for differential sync strategy, we need to make use of variable. Let us look at the steps on how we can do it in APPSeCONNECT.

• Open APPSeCONNECT portal and go to Design-> ProcessFlow.
• Drag Application Node for which you need to get data from. In our case we have created a sync between Magento and Microsoft Dynamics NAV.
  
• You can see from the above design, the data is getting transferred from Magento, transformed using Mapper and then posted to Dynamics Nav.
• In configure Magento popup, we need to create a variable
  
• In this section, we can see two variables, the one CreatedDate is the one created for Incremental Sync. If I edit the Variable it looks like below:
  
• You can see the //items/item/created_at mentioned in Expression. The expression is the relative XPath from the data packet to get the created_at value. You can mention function to identify max or min date inside the document too.
• The default value specifies the way to specify the first value as mentioned. Remember to pick Is Persistant field, to ensure the data is getting saved properly.
• Now you can use the same in Action Filter.
  
• You can see the ${CreatedDate} is used in the action filter to ensure the data is used to incrementally filter data.
• In the snapshot, you can see the actual Request URL in the Activity Log.
  
• When you click on the Request Filter section of the Activity log, you can figure out what data is getting sent to retrieve in each request.
  
• The http://templatebar.com/qa/magento1 is the url of our magento site, and the log entry specifies the request url for getting customers in the process flow.

Conclusion

Incremental search is an important consideration while developing any sync. Even though there are a number of other better approaches, but in most of the cases, we end up using the Last Modified Date approach to sync data. Using this kb, you will be easily use the technique to sync data incrementally. I hope this will help you in long run.

Feel free to comment on the approach.

Thank you.