Enable breadcrumbs token at /includes/pageheader.html.twig

Five Essential Steps for Data Migration Planning

Don’t rip out your hair just yet. You can plan for this!

Planning ahead can take much of the stress out of data migration. Credit: Shutterstock

Implementing a new system can be an exciting time, but the nagging questions and doubts about the fate of data you’ve literally spent years collecting, organizing and storing can dampen this excitement.

This legacy data often comes from a variety of sources in different formats maintained by a succession of people.  Somehow, all the data must converge in a uniform fashion, resulting in its utility in the new solution. Yes, it is hard work and no, it is not quick. Fortunately, this scrubbing and normalization does not have to be a chaotic process replete with multiple failures and rework.

It goes without saying that the new business process or workflow, if you will, must be finalized, locked down, meaning 100 percent complete with no changes prior to migration.

1. Decide what to keep.

This is the first and arguably the most important step. Do you really anticipate using 10 years of legacy data? Perhaps you only need the previous fiscal year’s data. The rest of the data can be archived and retrieved when required. Only migrate what you need to maintain project timelines, control resource expenditures and maintain a streamlined, uncluttered application.

2. Determine what to do with data in the new application .

This is a critical step that helps to refine and scope the effort. The purpose of migrating data from a source to a destination database is often to be able to use the legacy data in reports and statistical analysis, maintain aggregates, etc. You must relate the data from the source database to the destination database in order to justify the migration and ensure viability of migrated data. Determining precisely how you want to use the legacy data will help pinpoint specific fields to migrate and their “home” in the destination application. This is another part of scoping the migration to avoid unnecessary scrubbing, normalization, export, import and quality assurance work.

3. Assign roles and responsibilities.

Before you heavily engage your IT support staff or vendors to begin the actual data migration, you will want process owners to examine their own data—of which they should be the subject matter expert—to ensure that correct fields for migration have been identified, their purpose and utility in the new application is clear, and the data is headed toward the correct table as its permanent residence.

4. Eliminate data anomalies. 

So, you may not have been exceptionally vigilant about data input standardization over the years and you have some messy data. That will be reversed during this stage of the prep work. This step is intended to achieve data integrity or in other words, make the data targeted for migration more uniform and closer to squeaky clean. Spend some time on this and be thorough! This step saves aggravation and project disruptions down the road. Ensure that only appropriate data is populating each field. For example, in the field “Last Name” ensure that only “Smith” is there and not “Jane R. Smith.” Eliminate duplications and misspellings too. Street address uniformity tends to be one of the main culprits during this stage of the prep work, but there are many others. A comprehensive scrubbing and normalization effort will payoff bountifully.

5. Lock down both legacy and destination databases.

This will preclude deletions and unintentional new test records.  No one gets to delete or add anything during migration! Giving users the ability to delete records at any given time is a questionable practice, but that’s another issue. Once the database has been prepped for migration, do not permit any deletion of records as algorithms written to automate the migration may be sequentially oriented. A deleted record disrupts the record ID number sequence and this causes errors and troubleshooting, which prolongs and complicates the migration process.

In conclusion, proper planning can help to alleviate many common pitfalls and this whole process, if coordinated properly, can be far less painful. Your IT support staff or vendors should supply a data migration plan only after being able to analyze the source databases so that they can accurately scope the effort. They will be able to guide the process owners on scrubbing their data and handing it over to them in a format that is fit for migration, or at least close to it.

Remember, again, take only what you need and know what you’re going to do with it. Most importantly, remember that business process changes necessitate a redirection of information. The data, essentially, must have a new home. This modification of table structure first requires the creation of a new table containing the modified business process; next is a migration of previously migrated data and the establishment of a new table relationship. So, accommodating process/workflow change requests, even minor changes, is not feasible once migration planning has begun. Maintaining project discipline, putting in the prep work and adhering to your migration plan will be fundamental to the effort’s success!

Karyn Richardson directs the operations of Karder Corporation, a veteran-owned small business providing information technology governance and services to federal and state governments as well as industry. She is a federal certified chief information officer and is pursuing a Ph.D. in science and technology studies at Virginia Tech. The views stated are hers alone.