Pitfalls to avoid for successful cloud-based data management

We are sitting on huge volumes of data, much of which is unstructured, highly undefined and being collected at a scale and speed never seen before. 

IDC predicts that, by 2025, 55.7 billion connected devices worldwide will generate 73.1 ZB of data, growing from 18.3 ZB in 2019. 

Such raw data, which hold a plethora of precious business and customer insights, are sadly sitting in storage and are only processed when the need to use them arises.

Businesses and governments are often held back by their infrastructure, which cannot keep up with the scale and speed of data collection today. 

Understanding the need to take control of their unstructured data, businesses and governments are beginning to consider leveraging the cloud to better manage their data. However, there are pitfalls to avoid.

Bill Richter, CEO of Qumulo, shares some Asia Pacific insights with DigiconAsia.

IDC predicts that by 2025, 55.7 billion connected devices worldwide will generate 73.1 ZB of data, growing from 18.3 ZB in 2019. In your opinion, how effectively have organizations in Asia Pacific leveraged such raw data?

Richter: The pandemic accelerated a massive digital transformation across industries, with data being the driving force of this new digital economy. Over the past year, companies collected more data than ever before in a race to reshape their businesses through data-driven decision making.

In fact, a recent survey found that 82% of businesses that consider themselves to be data-driven reported they are reaping critical business advantages during the pandemic. It goes without saying that this unprecedented growth in data advanced the need for highly scalable data processing and storage capacity.

Asia Pacific is set to become the largest market for data centers over the next few years, with the Singapore, Hong Kong, Sydney, and Tokyo markets expected to lead in the region. Data centers will allow businesses across Asia to not only store petabytes of unstructured data, but also control where they extend over or integrate into on-premise and public cloud service management.

Small and medium-sized enterprises (SMEs) in Asia Pacific have also found success in leveraging data and data analytics to ease COVID-19 pressures.

In Thailand, Online Travel Agencies (OTAs) have helped SMEs understand data insights and optimize revenue through meta-search, machine learning, artificial intelligence, product diversification and platform centralization. Similarly, SMEs in Singapore used data-driven insights to tackle challenges amidst the pandemic – retaining existing and attracting new customers.

However, there is a common misconception that having a data-driven business typically refers to using structured data analysis to make decisions. While companies have more access to data than ever, there is a difference between being data-saturated and data-driven. In fact, 80 to 90% of data in businesses is unstructured, and most of that data doesn’t get used.

While it’s clear that business leaders know the importance of data, more than a third aren’t using it to make the majority of their decisions.

The ability to fully manage data and leverage data analytics is giving businesses the competitive edge to keep up in a world where data is growing exponentially.  Recognizing the potential of unstructured file data to create better outcomes is nothing short of a game-changer.

Businesses and governments are often held back by their infrastructure, which cannot keep up with the scale and speed of data collection today. What other challenges do they face in managing this explosive data growth?

Richter: Legacy file systems are not equipped to address the challenge of organizing and managing tremendous amounts of data growth. These systems lack adaptability, are not designed for scale, and more often than not are unable to provide real-time information and analysis.

Vendors have been providing complex products and licensing scenarios for this technology for decades. That complexity has helped them lock-in their customers to ineffective solutions, while in the process of extracting money from them.

Cloud solutions are now the name of the game, providing customers with more flexibility in storage options that can also handle the inevitable growth of data. Businesses need to take advantage of these new technologies and integrate modern software into their data management strategy to keep up with the scale and speed of data collection today.

How could the cloud be leveraged to meet these data management challenges?

Richter: Legacy factories are bound by hardware, and run on monolithic architectures where storage and compute are tethered and have to scale together. This structure makes it difficult for businesses to experiment, manage, and build with unstructured data.

The pandemic demonstrated the ability of businesses across industries to go digital. By moving data onto the cloud, businesses can separate their file data from compute and even connect their data to cloud-native services, like facial recognition.

This flexibility enables businesses to keep their data stable while their compute changes, giving them the freedom to manage and experiment, as they please. In the entertainment industry, it’s possible to create content for R&D, test experiences like virtual reality (VR) and augmented reality (AR), or try in-depth sports analytics on demand at live events without worrying that your data will be hindered or delayed due to the way it’s managed. You can set up virtual production crews and soundstages, and produce content rapidly with enhanced performance and capacity scaled to your workloads on the cloud.

What are the pitfalls often made when considering cloud storage and how should organizations avoid them?

Richter: Migrating to the cloud can be a treacherous journey when organizations are not properly equipped with the right support to transition their data from legacy infrastructure to the cloud. A major pitfall that businesses might face is realizing that they do not have capable IT architecture or expertise to handle the process internally and need to re-architect their legacy applications for cloud technology.

Businesses can overcome this by trusting a reliable storage provider that can help navigate the roads of data migration in a timely manner. The right partner will help transition companies into the cloud by prioritizing their own unique needs and evaluating the effort versus impact for their storage solutions.

Businesses also face the pressure of going “all-in” on a single platform. This triggers the potential of being locked-in, having scalability limitations, or creating the risk of a single point of failure. These dangers can be avoided by adopting a multi-cloud platform.

By taking advantage of the separate benefits offered by providers such as Amazon Web Services (AWS), Microsoft Azure, or the Google Cloud Platform, businesses can customize their cloud solutions and find the most cost-efficient and high-performing solution that suits their every need.

Cloud storage also surfaces security concerns for businesses. Although certain data must always reside on-prem due to security mandates, there are options to move that data into a secure cloud in a hybrid setting. Businesses need to work in tandem with their IT security experts and storage providers to overcome any potential red flags or roadblocks and work together on migrating that data safely.

Understanding these common pain points can help businesses employ the right cloud storage strategy. Finding a reliable partner that specializes in cloud and data management not only ensures a seamless transition to the cloud, it also guarantees all file data storage needs are met and available whether workflows are running on-premises or in a hybrid or public cloud.

What practical steps can be taken to ensure successful adoption of cloud-based data platform?

Richter: It’s critical to understand that every organization’s adoption of the cloud is going to look different. Depending on need, some organizations might require more customization or larger scale for storage as compared to others.

Before jumping into a specific solution, organizations must first assess their current IT infrastructure and identify specifically why their business needs to migrate to the cloud. This includes analyzing cost, scale, and what cloud providers are best fit. Before moving data from one place to another, there needs to be a functioning and secure landing zone that’s accessible for both users and IT experts.

Once data has officially been introduced to the cloud, organizations should ask themselves the following:

• Is everyone able to access the data they need?
• Is the data securely stored, or are people able to access it without permission?
• Are workflows being interrupted by storage issues?

From here, it’s easy to determine whether or not it’s necessary to scale up for other workloads. The option of hybrid cloud solutions removes stress from this decision-making process. You don’t need to move all of your data at once to have a collaborative workflow.

After becoming comfortable with the cloud, it’s easy to monitor data and storage usage in real-time to determine how to best scale cloud solutions for the best ROI. Beyond cost optimization, other immediate benefits of cloud adoption might include better collaboration across workflows and improved productivity as workflows are not hindered by outdated IT infrastructure.