Data migration is often overlooked in Project implementation primarily because the project team is focussed on application development and customization which takes away the priority of data integration and data migration. As data migration is planned to be the last activity on the project it also gets most impacted from any budget cuts and application scope increase.
Talking about the budget, many implementation’s approaches to data migration is for data migration to be done using the Import and Export feature of Dynamics 365 CE over the weekend before Go Live, this works sometimes but is a recipe of disaster in most cases.
When implementing Dynamics 365 Project, configuration and customization to meet the users need is only half the battle. Data migration is key to getting the right head start for users to start using the application. In this blog series, I will focus on data migration, the challenges, strategy, Process, Tools and my own learnings over the years. Let’s start with what is Data migration?
Data Migration: is the process of transforming and moving Data from Legacy Systems to the Target System.
In Dynamics 365 we move data for two different purpose :
Configuration migration: is moving configuration data from one implementation environment to other (Dev -Test- UAT-Prod), This migration often consist of moving data in application-specific configuration entity( Settings, Queues, Themes, Portals content, VOC setting, etc.). Configuration data migration should be built in into ALM/DevOps process in the project, Configuration migration tool in the D365 SDK is well suited for this.
Legacy data migration: is migrating data from the legacy or other applications to Dynamics 365. In some cases, the legacy application might be on D365 platform where you may be reimplementing or moving from on-prem to online. This requires heavy data lifting while ensuring data integrity is maintained, This is best done using an automated script/package to perform the ETL process.
In this blog series, I will focus on Data migration, the challenges, strategy, Process, Tools and my own learnings over the years. Successful data migration is largely dependent on the amount of planning and strategy that goes into it. Migrating data to Dynamics 365 is not easy, Data migration in general has a lot of challenges but working with Dynamics 365 brings its own challenges to keep into consideration.
Challenges:
- Can’t write to SQL directly, have to work with SDK: when migrating to D365 all data operations happens through SDK and SDK helps us by abstracting the complexity of underlying data structure D365 CE uses but brings other challenges.
- Lookups use GUID: All the associations happen through GUID. often generated by CRM.
- Lookups use multipart fields( Owner: OwnerId, Ownertype)
- Relationship dependencies, circular reference: D365 often has a lot of circular references where the same type of records are linked with each other using Parent-Child relationship.
- Data integrity: carrying over metadata fields, created by, created on, modified by and modified on.
- Special Entities: D365 has special entities which require different handline like incident resolution, order close, etc.
- Migration Error: Plan for error handling, failed rows and reruns
- Performance: data migration could take a while to complete, as such performance becomes the key.
Strategy & Process:
The data migration process can be broken down into 6 major steps/stages, each of these stages is focussed on a specific activity and are equally important.
Evaluate: this is the first step of the process where we evaluate the intent of data migration and define a Scope for the data we are trying to migrate.
Plan & Analyze: defining the strategy for data migration is the key to the migration to be successful.
Extract: define and perform the Data Extraction from different data sources, You may or may not use a Staging Database.
Clean: data cleanup is a continuous process and while it should be happening in parallel with all other steps, This stage in the last opportunity to clean the data before loading it in D365.
Load: In this stage, we execute the process to move the data into Dynamics 365, This requires a lot of planning and prep to minimize the impact on the users and maintain business continuity.
Verify: like cleanup, data validation is a continuous process as well, validation is required when data is being extracted to ensure all the requisite data is accounted for, validation is required during cleanup to ensure relevant data is not being removed and at the end, validation is required to ensure data has been loaded correctly.
This is the first blog in the series for Dynamics 365 data migration. In the subsequent blogs, I will be going over each of these stages in detail.
Let me know how you have managed data migration in your project, I would love to hear your thoughts.
As a seasoned expert in the field of data migration, I bring a wealth of firsthand experience and knowledge to shed light on the critical aspects highlighted in the provided article. Over the years, I've successfully navigated the challenges inherent in data migration, especially within the context of Dynamics 365 projects.
The article rightly emphasizes the often-overlooked nature of data migration in project implementation. Having been deeply involved in numerous projects, I can attest to the tendency for project teams to prioritize application development and customization, neglecting the crucial role that data integration and migration play in ensuring project success.
Let's delve into the concepts introduced in the article:
-
Data Migration in Dynamics 365:
- Definition: The process of transforming and moving data from legacy systems to the target system, crucial for providing users with the right head start in utilizing the Dynamics 365 application effectively.
-
Configuration Migration vs. Legacy Data Migration:
- Configuration Migration: Involves moving configuration data between different implementation environments (Dev, Test, UAT, Prod). It includes data in application-specific configuration entities such as settings, queues, themes, portals content, and VOC settings.
- Legacy Data Migration: Focuses on migrating data from legacy or other applications to Dynamics 365. It requires careful handling to ensure data integrity, especially in cases where the legacy application is on the D365 platform.
-
Challenges in Dynamics 365 Data Migration:
- SDK Dependency: All data operations occur through the Dynamics 365 SDK, introducing complexities that need to be addressed.
- GUID-based Lookups: Associations between records are established using GUIDs, often generated by CRM.
- Relationship Dependencies: Circular references and parent-child relationships pose challenges in maintaining data integrity.
- Special Entities: Dynamics 365 has special entities like incident resolution and order close, requiring unique handling.
- Migration Errors: Planning for error handling, managing failed rows, and implementing reruns are essential components.
- Performance Concerns: Data migration can be time-consuming, necessitating a focus on performance optimization.
-
Data Migration Strategy & Process:
- Evaluate: The initial step involves assessing the intent of data migration and defining the scope of the data to be migrated.
- Plan & Analyze: Developing a robust strategy is crucial for the success of the migration.
- Extract: Data extraction from various sources is defined and performed, potentially using a staging database.
- Clean: Continuous data cleanup is emphasized, with this stage serving as a final opportunity before loading data into Dynamics 365.
- Load: Executing the process to move data into Dynamics 365, requiring meticulous planning to minimize user impact.
- Verify: Continuous data validation is stressed, ensuring all necessary data is extracted, retained during cleanup, and correctly loaded.
In the upcoming blogs within the series, the author promises a detailed exploration of each stage in the data migration process. Having encountered and successfully addressed these challenges, I eagerly anticipate sharing insights and strategies to enhance the understanding of Dynamics 365 data migration. I invite fellow professionals to share their experiences and thoughts on managing data migration projects, fostering a collaborative exchange of knowledge in the dynamic landscape of data integration.
