Some businesses are moving data from legacy systems to the cloud. Some are moving from on-site servers to other on-site servers, or between cloud implementations. It’s sometimes necessary to migrate data between applications, as business processes change and stacks change with them. One thing is certain: most businesses will do a data migration. When they do, it will be fraught with risks — both as it’s done, and afterward when you try to run your company on the results.
Since few in-house IT teams have the skillset to plan and execute a migration successfully, most companies will reach for outside help. In this post, we’ll look at best practices and risks before talking about why you should seek out experienced specialists for your data migration.
Data migrations: what and why
Data migrations involve moving an organization’s data to a new substrate. Often, this will be the cloud rather than on-site data centers, and it’s increasingly common to move data from one cloud host to another.
Data migrations are also undertaken to upgrade an application database with new hardware, which means migrating data to new devices or equipment. It’s also sometimes necessary to migrate data to newly acquired tools in a stack or to replacements for legacy communication, customer data, or productivity tools, for instance.
An organization’s data is a crucial resource, and one that’s also both regulated and a target for bad actors. So there are more than purely technical factors at work when you consider, plan, and execute a data migration.
Types of data migration
Organizations undertake data migrations for multiple reasons; taking advantage of new tools, storage or even applications for their data are all common, but so are improved security and access. While there are as many types of migration as there are organizations, we can generalize.
Storage migrations
Storage migrations migrate data during a storage technology refresh or upgrade, with common goals including faster performance, dynamic scaling and improved data management features.
Database migrations
Moving databases from one platform to another, such as from on-premises data centers to the cloud, or migrating bodies of data between databases.
Application migrations
Application migrations can involve migrating data within an application, such as shifting from legacy on-site MS Office implementations to cloud-based Office 365. This category also refers to moving from one similar application to another — replacing accounting software or CRM, for instance, often from different vendors and using different data structures.
Cloud migration
Moving data from on-site to the cloud or from one cloud to another. This is not the same as backing up to the cloud. Instead, operational data is permanently moved to a cloud implementation.
Business process migration
These migrations transfer applications and databases that contain information about their company, customers, services and operations. Organizations do these when they merge, buy or are bought by another organization, or sometimes when they grow, scale and transform their operations or reorganize their internal structure for other business reasons.
Data migration best practices
Best practices apply to data migration regardless of the type of migration. They include:
Backing up
Back up data before migrating. If something goes horribly wrong, it’s still there. Make sure backups are secure, not involved in the migration (so they can’t be damaged, corrupted or lost in the same event that makes them necessary) and test them before you begin the migration process.
Strategizing and planning
Planning migrations is crucial, and it’s a complex, multi-stage process. This is a mission-critical project, not something to be taken lightly. It’s essential to plan both the preparatory period of migration, when you’re assembling the tools and personnel to manage and perform the migration, and the actual process itself. Good plans are stable, robust, clear, and reversible, with clear guidelines for what to do if you encounter common obstacles or a stage of the migration fails.
Testing and tracking
Every stage of the data migration process, from planning to post-execution support, should be comprehensively tracked and measured against expected outcomes. Data that has been migrated should be tested for integrity, processes should be tested for efficacy and robustness, and testing should be adversarial — that is, it should be designed to find problems rather than to reassure concerning their absence.
How data migrations go wrong
While every company and every project is different, we’ve found that the majority of data migrations that aren’t satisfactory go wrong in predictable ways. There are five main areas where unprepared or underskilled attempts at migrations tend to go off the rails:
1. Cost
Moving to an as-a-service pricing model can leave you unprepared to calculate total cost of ownership, which must now be figured as per-seat or per-instance recurring fees. These are often lower and more manageable than the costs of a traditional ownership model, but they can become an issue if you’re planning a big move for the first time.
There’s also the risk that without adequate technical knowledge, you can wind up paying twice or more for the same functionality. Selecting between reserved instances, hybrid use with the associated discounts, and Microsoft’s Shared Image Galleries is simple — if you have the experience to know that hybrid use discounts can save a business up to 80% on its data storage, while gallery images are billed by the minute. If you don’t, you might wind up finding out the hard (and expensive) way.
2. Readiness
If you’ve never designed and implemented business architecture before, you’d be surprised how many opportunities there are to design flaws in. Assigning staff the right level of access and privileges so they can get their job done, but don’t have the keys to the castle, is another task that’s simple if you’ve been doing it for a decade, a puzzling maze if you haven’t — and that’s if your stack is fairly simple.
It’s important to prepare for failure as well as success. Experienced consultants are less likely to restart the wrong server or turn off a virtual machine that’s not backed up, though it’s not unheard-of. More importantly, we’re more likely to use a series of safety, protection and recovery protocols to both make sure less goes wrong, and to road-test ideas and make mistakes in safe, recoverable environments rather than on launch day.
3. Design
Very few businesses really need a custom database and everything that goes with it. You really don’t need to write your own tools from scratch. The wheel is already here. For most businesses, designing data and process systems is about identifying which existing tools cover the required functionality and then setting them up and integrating them correctly. That’s no small task in itself. There are specific accelerators available for Microsoft and other partners, meaning experienced hands can move you to operability much quicker.
We also see mistakes with cost modeling. Over 83% of data migrations either fail, or overshoot their budgets; cost overruns average 30%. One reason is that they fail to accurately cost-model, meaning they expend huge efforts to translate legacy tools to the cloud — when they could have simply kept those tools propped up for six months and then retired them. Or they try to transfer all data at once — mission-critical and not, customer and business, structured and otherwise.
4. Performance
Suppose you have on-site servers, running at 40% CPU or 70% CPU usage, and representing a capital cost of $500,000. The CPU numbers look good, and for your own servers, they are. But take that same workload and put it on Azure, and you’ll pay by the operation: bad queries cost money. So designing your systems to be efficient at this basic, granular level suddenly makes a lot of sense — it’s just something most in-house IT teams aren’t equipped to deal with.
In many cases, in-house teams are closer to your business, which is good — but further from the wider industry. Thus, they’ll sometimes set up your systems with databases and applications cohabiting the same server or VM: great for efficiency, but if you want to move it to the cloud, you’ll need someone who can safely disaggregate and transfer it, and design an on-cloud structure that makes the most of the new substrate. (The broader point is true if you’re moving between cloud implementations too.)
5. Maintenance
Patching and backing up databases is pretty rare when you’ve moved to the cloud, which is overwhelmingly the destination for businesses migrating their data now. But managing security, managing the service tier, and planning non-production environments all require extensive skills that must either be sourced or trained. Some consultants will set you up and walk away; others remain available to work with your team to optimize and protect your data. (In addition, cloud data storage and retrieval isn’t quite ‘0 maintenance,’ and companies who think it is can wind up unprepared for easily mitigable risks.)
Why you should work with an experienced partner
You can’t just follow best practices for the same reason that you can’t just cook directly from a recipe book. The reason it never comes out quite right is, you’re not a chef. Following instructions can’t replace a depth of skills and know-how that means you’re never at a loss and know how to avoid the most common mistakes.
If you don’t have extensive in-house skills, you can hire for them. But after you finish your data migration, you’ll be overstaffed and overskilled. You could train for them, but that can be a time-consuming process, and your in-house IT team already have jobs. Or you could seek a partner and consultant.
How AndPlus can help
We’ve been helping businesses and non-profits move their data safely and securely for over a decade. With clients in multiple industries, we’ve seen and solved a vast array of the problems you find where tech and business meet. So whether you’re considering moving a simple on-site database to a simple cloud implementation, or you’re rebuilding your whole stack on Azure, Google Cloud, or AWS, we can help.
Crucially, we can partner with your whole business, not just your IT team. You won’t be left in the dark, and we won’t make decisions and recommendations based solely on technical considerations — without considering your P&L, revenue and the ROI your new stack should deliver.
AndPlus isn’t a contractor, we’re a long-term partner with a stake in your success. We deliver granular, usable advice, guidance, and boots on the ground where required, and we’ll get it right, because by then we’ll already know you and your business.
Takeaways
- Data migrations require deep, broad knowledge across domains, and practical experience. Few in-house IT teams have the requisite skillset
- Databases still require some maintenance after migration
- Effectively planned and executed data migrations reduce the risks associated with poor security during data transit and can offer significant initial and ongoing savings
- Most data migrations are partial or total failures, thanks to poor planning and forecasting, lack of technical understanding, or failure to ‘join up’ the technical and business aspects of the migration effectively
