SOLUTION
To achieve a more flexible and reliable cloud service for data warehousing, AndPlus advised a data transfer project from Amazon Redshift to Snowflake, a cloud warehouse built on a similar AWS infrastructure, for a number of cost and flexibility reasons. Amazon Redshift does not allow a user to scale CPU separately from storage, so upgrades and changes are not adaptable or customizable to a use case. With Snowflake, however, users can scale CPU separately from storage, meaning the services Onset paid for could be tailored specifically to the InTemp project’s needs. Snowflake also offered in-house maintenance, as well as more accurate recording.
Onset’s transformation roadmap implemented a microservice architecture, in which a single application is composed of many loosely coupled and independently deployable smaller services. Essentially, data was slowly transferred between Redshift and Snowflake over a period of time, rather than all at once. This is especially important for vaccines: uninterrupted temperature data is key for safe delivery, so the change between data warehouses could not shut down the app, even for a few minutes.
The warehouse microservice slowly decouples and shifts data from one service to another.
Microservice architecture serves a number of purposes for data transfer, predominantly benefiting the end user. This practice of transferring small amounts of data at a time prevents the end user from experiencing any delays or changes within the app. This lets clients continue to use the InTemp data logging software with no change in their office’s process or protocol. Additionally, discrepancies in data can be identified and corrected before transferring over to the new warehouse.
Not only could Snowflake scale storage service automatically, but the program no longer had to run 24/7. Rather than sending data in a constant stream, AndPlus designed a “snowpipe,” or a custom code which allowed data to transfer every three minutes. Giving these servers short periods of rest every day was an impactful cost-effective shift.
OUTCOME
AndPlus’s custom programming successfully migrated Onset’s InTemp storage system with zero downtime and a clean shift with no change in the InTemp user interface. Snowflake could now record temperatures to several decimal places, making data logging that much more precise, reducing product loss and thereby increasing product value. Record linkage fixed discrepancies while transferring data, and future recordings would have fewer issues. Above all, AndPlus’s integration will continue to evolve and adapt to Onset’s warehousing needs.
ABOUT ONSET
“Based on Cape Cod, Massachusetts, Onset has been designing and manufacturing its data loggers and monitoring solutions on site since the company's founding in 1981.
Onset's mission is to be a global leader in data logger and monitoring solutions used to measure, record, and manage data for improving the environment and preserving the quality of temperature-sensitive products.” - Onset 2021
TECHNOLOGY USED
