How should organizations enjoy the benefits of a multi-cloud strategy without the complexity of sharing and integrating voluminous data workloads?
Cloud adoption continues to accelerate, with Gartner noting that 81% of enterprises are already using two or more public cloud providers.
However, recent Flexera research found that migrating workloads to the cloud remains a challenge, with 59% of organizations listing it as a primary initiative for 2021.
The critical challenge companies face in multi-cloud world today is their data is siloed from the data stored in another cloud service and their data integration strategy to overcome the challenge is complex and time-consuming.
Informatica recently launched its Intelligent Data Management Cloud to strip the complexity out of a multi-cloud strategy and making it even easier to move data to the cloud for enterprises. The company has also partnered Snowflake on enabling new capabilities for organizations to scale analytics and cloud applications with simplicity and speed.
These industry-first cloud initiatives highlight the growing pain points that organizations need to address in the multi-cloud world. So how can we have our cake and eat it? DigiconAsia discussed the issues and strategies with Rik Tamm-Daniels, VP Strategic Ecosystems & Technology, Informatica.
Moving to the cloud is fundamental to the digital transformation strategy for many organizations. But why does migrating workloads to the cloud remain a challenge for these organizations?
Rik Tamm-Daniels (RTD): Data is growing exponentially and is more distributed than ever – across multiple applications, systems and clouds. Enormous volumes of data from different sources such as application data, mainframe, databases, data warehouse, machine data, IoT, streaming data, logs and files, etc., residing in siloed enterprise systems are making it difficult for organizations to move workloads to the cloud.
Further, according to IDC, 80% of enterprises will have shifted to cloud-centric infrastructure and applications twice as fast as before the pandemic. This quick shift means businesses need to figure out how to deal with the challenge of fragmented data across a multitude of platforms, clouds and siloed systems and manage the move of the workloads to the cloud seamlessly.
What are the key challenges in cloud data integration? How should organizations address these challenges?
RTD: The key challenges are often related to how to connect high volumes of data from on-premises data sources and cloud applications seamlessly and how to process the complex data integration tasks, especially with organizations that have been performing manual, complex and siloed approaches to data integration in the past.
It is important for organizations to look into high-performance, easy-to-use data integration solutions that will allow them to connect on-premises data sources and cloud applications to seamlessly integrate high volumes of data, getting them up and running quickly to keep up with market changes and stay competitive.
What is the difference between ELT and ETL capabilities in the context of data management?
RTD: ELT (Extract, Load, and Transform) and ETL (Extract, Transform, and Load) are both key tools to have in a comprehensive approach to data management, as the shape, volume and consumption requirements for specific data assets drive whether ELT or ETL is the most performant and efficient approach.
For example, if data needs to be integrated across multiple systems of record and transformed into a normalized data model row-by-row, an ETL approach may be significantly more efficient.
On the other hand, ELT is commonly used in Cloud Data Lake and Cloud Data Warehouse patterns to load raw data into a common location and then transform it in either the data lake or the cloud data warehouse. ELT is extremely efficient when data sets are broken into many files and the processing required can be easily parallelized, for example, IoT data processing.
The key is that there is not a one-size-fits-all approach, and to be agile and adapt to ever-changing data landscapes and requirements, enterprises need to be able to deploy ETL and ELT and have tools that provide seamless access to both modes of data integration.
How does Informatica’s Intelligent Data Management Cloud, and its latest collaboration with Snowflake’s data cloud, help organizations in Asia Pacific in stripping the complexity out of a multi-cloud strategy?
RTD: Two of the most important data management requirements customers have today are democratization of data and delivering a consistent data management and analytics experience in a multi-cloud and multi-hybrid world.
In many areas, it makes sense to optimize with cloud-specific tools, but in data and analytics, breaking down of data silos is a key requirement for success. Having a consistent data management and data access approach no matter which cloud users are on or no matter which environment data lives in is absolutely critical to successful digital transformation.
Data democratization requires that the greatest possible set of users have access to trusted data assets and can utilize those data assets in a way that works for their technical skill level.
The partnership between Snowflake and Informatica delivers on both of these key requirements. Informatica’s Intelligent Data Management Cloud (IDMC) platform powered by Informatica’s AI engine CLAIRE provides a complete end-to-end data management capability regardless of which cloud provider or cloud providers an organization chooses.
When organizations move data to Snowflake through Informatica’s IDMC, they can easily and seamlessly transform, cleanse and govern data without any hand-coding or having to assemble an end-to-end solution from disparate systems.
The new ELT capabilities announced make it easier and faster to drive key application data into Snowflake. Our new cloud-native application mass ingestion capabilities rapidly load and synchronize data in a matter of a few clicks from applications such as SAP, Salesforce, Netsuite, Zendesk, Microsoft Dynamics 365, Workday, Marketo, ServiceNow and Google Analytics into Snowflake through an enterprise-grade, easy-to-use, wizard-driven application synchronization service.
In addition, Informatica’s native support for Snowflake’s Java UDF via the Intelligent Data Management Cloud enables Java developers and data scientists to easily plug their codes into the ELT pipelines further democratizing data access by opening up familiar tools to different types of users.
