Register Now

Login

Lost Password

Lost your password? Please enter your email address. You will receive a link and will create a new password via email.

Add question

You must login to ask question.

Login

Register Now

APPSeCONNECT community provides a new way for integration super heros connect the external world. Join hands together to build a community of people to help each other, solve problems or share knowledge.

Create Incremental Sync using Last Modified Date in APPSeCONNECT

Create Incremental Sync using Last Modified Date in APPSeCONNECT

One of the best practices mentioned before is to create an incremental sync between two application such that the data created after the last sync only is incrementally downloaded from the application.

Use Case

You want to sync / extract records from source application that have been created or modified after the last time the process ran.

Approach

To incrementally sync data, the one thing you need is to capture the last sync date. Ideally we recommend to get the highest sync date from the input packet such that we always store the correct date time irrespective of the time zone info which is required as well.

  1. We create a variable in the ProcessFlow to capture the last date time present in the packet. We use the same date time stamp to query data greater than the last modified date value stored in the variable.
  2. If we get data again, that means the data is incrementally retrieved. Recalculate the datetime present in the packet coming from the source application and replace the value of the variable.
  3. Process the record as needed.
  4. Upon completion of the proccess, the ProcessFlow will automatically capture the last value into storage.

 

Note: It is always recommended to capture the last modified date from the records itself, to ensure minor server date time adjustments does not hamper the incremental capture.

Consideration

  • For the first execution time, we need to put a default value of the variable such that the historical data could be processed on first run. In APPSeCONNECT, the default value of a variable could be set which can automatically picked up if data not available.
  • In case of errors midway of execution, the datetime will still be processed in APPSeCONNECT. APPSeCONNECT provides Sync Info section, which will allow you to track all the data present in the packet and allows you to rerun the same process again with a specific data.
  • Idempotent data transfer is sometimes a challenge as well, in such a case, we use Duplicate check Idempotent rule in Mapper to remove duplicate data.

Steps to incrementally sync data

To create an integration for differential sync strategy, we need to make use of variable. Let us look at the steps on how we can do it in APPSeCONNECT.

  • Open APPSeCONNECT portal and go to Design-> ProcessFlow.
  • Drag Application Node for which you need to get data from. In our case we have created a sync between Magento and Microsoft Dynamics NAV.
    Magento Nav Connection
  • You can see from the above design, the data is getting transferred from Magento, transformed using Mapper and then posted to Dynamics Nav.
  • In configure Magento popup, we need to create a variable
    magento configuration
  • In this section, we can see two variables, the one CreatedDate is the one created for Incremental Sync. If I edit the Variable it looks like below:
    variable editor
  • You can see the //items/item/created_at mentioned in Expression. The expression is the relative XPath from the data packet to get the created_at value. You can mention function to identify max or min date inside the document too.
  • The default value specifies the way to specify the first value as mentioned. Remember to pick Is Persistant field, to ensure the data is getting saved properly.
  • Now you can use the same in Action Filter.
    action filter
  • You can see the ${CreatedDate} is used in the action filter to ensure the data is used to incrementally filter data.
  • In the snapshot, you can see the actual Request URL in the Activity Log.
    snapshot
  • When you click on the Request Filter section of the Activity log, you can figure out what data is getting sent to retrieve in each request.
    filter data
  • The http://templatebar.com/qa/magento1 is the url of our magento site, and the log entry specifies the request url for getting customers in the process flow.

Conclusion

Incremental search is an important consideration while developing any sync. Even though there are a number of other better approaches, but in most of the cases, we end up using the Last Modified Date approach to sync data. Using this kb, you will be easily use the technique to sync data incrementally. I hope this will help you in long run.

Feel free to comment on the approach.

Thank you.

 

About Abhishek SurVerifiedExpert

VP, Products at InSync Abhishek Sur has been a Microsoft MVP since 2011. He is currently working as a Product Head with Insync Tech-Fin Solutions Pvt Ltd. He has profound theoretical insight and years of hands-on experience in different .NET products and languages. Over the years, he has helped developers throughout the world with his experience and knowledge. He owns a Microsoft User Group in Kolkata named Kolkata Geeks and regularly organizes events and seminars in various places to spread .NET awareness. A renowned public speaker, voracious reader, and technology buff, Abhishek's main interest lies in exploring the new realms of .NET technology and coming up with priceless write-ups on the unexplored domains of .NET. He is associated with Microsoft's Insider list on WPF and C# and stays in touch with Product Group teams. He holds a Master's degree in Computer Application along with various other certificates to his credit. Abhishek is a freelance content producer, developer, and site administrator. His website www.abhisheksur.com guides both budding and experienced developers in understanding the details of languages and latest technologies. He has a huge fan following on social networks. You can reach him at books@abhisheksur.com, get online updates from his Facebook account, or follow him on Twitter @abhi2434.

Follow Me

Leave a reply

Share via
Copy link
Powered by Social Snap