Using Snowflake for Data Migration

Jul 9, 2020 10:22:48 AM

Cloud Technology 1In the beginning, enterprise data lived on database storage servers and file servers in a company’s onsite data center. Although this arrangement gave company leadership peace of mind that their data resided safely within their metaphorical “four walls,” it was also a source of IT challenges:

  • As the company grew, more servers or even more data centers had to be purchased, configured, tested, and deployed.
  • Data centers are costly to build and operate, with high electricity bills for the servers, cooling systems, and network switchgear.
  • Data had to be backed up to tape and stored offsite to assure some semblance of disaster readiness.
  • For those companies rich enough to afford multiple data centers, data could be replicated across different geographical locations, but this was complicated to set up and maintain.
  • Server hardware has a finite lifetime. It has to be maintained often and replaced periodically.

If that wasn’t enough, storing data in your own data center doesn’t necessarily make it safe. Even if a physical disaster never strikes, poor data security measures could leave the door open for hackers to walk off with all your sensitive data. In addition, maintaining the latest security updates can be time-consuming.

As cloud services have evolved and matured, they have become compelling solutions to all the shortcomings and challenges of on-premise data centers.

The Trouble with Cloud Data Migration

It turns out talking about migrating enterprise data to the cloud is one thing…getting it there is quite another. Depending on the size and complexity of a data environment, a cloud migration project could require several months of planning and preparation, including:

  • Addressing existing data quality issues
  • Developing and testing scripts for data moving from one platform to a different platform (for example, Oracle to SQL Server)
  • Designing the cloud server architecture for scalability and optimal performance
  • Establishing proper data security in the cloud environment
  • Updating applications to access the data in its new home

…and finally, transferring the data to the cloud.

You can’t just drag-and-drop 20 terabytes to a cloud service. Several approaches must be considered for this task. Some of these considerations are simplified by a new entrant in the cloud database market: Snowflake.

What Snowflake Is and Isn’t

Snowflake is a cloud-based database platform-as-a-service provider that offers some compelling advantages over on-premise and other cloud database solutions:

  • Developers in Python, R, Spark, Node.js, and more will find a plethora of connectors that simplify application development and updates.
  • It supports both structured and semi-structured data and supports ANSI-standard SQL.
  • It’s cloud-agnostic; it’s available for Microsoft Azure, Amazon Web Services, and Google Cloud Platform.
  • Its security scheme is built on hierarchical key model encryption, which satisfies many regulatory and industry requirements (such as PCI-DSS, HIPAA, and GDPR).
  • Its unique architecture separates storage and computing resources, enabling each to scale independently and reduce performance issues such as contention.
  • Snowflake features automatic and near-instant scaling (up or down) of compute and storage resources, plus per-second pricing so that customers pay only for the actual resources used.
  • Snowflake requires no infrastructure or maintenance activities on the part of the customer.

Snowflake also supports a data-sharing program that enables different companies (think customers, vendors, distributors, and other partners) to share data with each other without making copies or transfers.

However, Snowflake is not a general-purpose database platform. It is intended for data warehousing and similar data summarization and aggregation, such as data lakes and is less suitable for day-to-day operational transaction data.

The good news is, if your transactional data is also in the cloud, getting the summarized data into Snowflake is easy. Snowflake works within and between different cloud providers.

Migrating to Snowflake

Another nice feature of Snowflake is its technical support, which has published numerous guides to migrating data from various platforms to Snowflake. Several migration tools are available to simplify the process even further.

You’re still in for a long project if you have a complex data landscape. But many of the excruciating technical tasks can be simplified or eliminated. You’re left to focus on the business and application tasks, such as data cleansing, application updates, and the like.

Snowflake was designed from the ground up as a cloud database platform, not simply an on-premise platform deployed in the cloud. That means its architecture is optimized to leverage the unique advantages of cloud computing, such as automatic resource scaling.

This approach also simplifies data migration because the underlying cloud architecture—which is always tricky to set up, especially for cloud technology newcomers—is already set up and optimized for performance, reliability, and security.

Final Thoughts

Snowflake offers a compelling choice in cloud-based database platforms. Many well-known companies are Snowflake customers. At AndPlus, we are following Snowflake with keen interest so we can help you evaluate whether Snowflake is the right choice for your own cloud data migration.

Brian Geary

Written by Brian Geary

Brian is a true believer in the Agile process. He often assists the development process by performing the product owner role. In addition to his technical background, he is an experienced account manager with a background in design and marketing.

    See more resources

    Lists by Topic

    see all

    Get in touch

    LET’S BUILD SOMETHING AWESOME. TOGETHER.