How should organizations in Asia Pacific address the data explosion in today’s digital economy, without heavy investments in data infrastructure and engineering tools?
As organizations in the region pivot toward being more data-driven, there is a need to better equip data engineers with the right technology platforms so that they can utilize the power of analytics at a fraction of time traditionally needed to deliver better business value.
In a recent global survey of data engineers, it was found that inefficiencies were blamed for much of woes. The survey revealed that data engineers were spending too much time fixing and maintaining pipelines and finding the right data, rather than focusing on analytics that generate a positive impact on the business.
The amount and types of data available is so overwhelming that traditional approaches requiring heavy infrastructure, on-premise data loading tools, and perpetual waves of paid consultants simply cannot keep up.
The good news is that organizations no longer have to devote valuable engineering resources to building and maintaining data pipelines.
Fivetran is one company that provides the technology to enable this transition so that, with a cloud analytics model, data engineers are able to analyze even larger amounts of data without the previous hassle – and ensure they are on the right track in delivering the power of analytics within the organization.
DigiconAsia finds out more from TJ Chandler, Managing Director, APAC, Fivetran.
How can digital technologies – and what are some of these technologies – help organizations in Asia Pacific in their quest to be data-driven?
Chandler: Organizations in Asia Pacific have always desired to be data-driven, as the variety and volume of data sources has proliferated in the last decade. To be data-driven, organizations must work smarter, not harder.
Meanwhile, consumers of data have become more sophisticated and more diverse. No longer are reports and analysis limited to a small number of executives–everyone in the organization must have timely access to reliable and relevant information. Delivering on these expectations requires connecting to dozens (or even hundreds) of sources, extracting the data, loading it into an analytical repository (a.k.a. “data warehouse” or “data lake”), and transforming it into meaningful reports and user-friendly dashboards.
Automation, not brute force, is the only way to achieve these complex tasks at scale. If organizations across Asia Pacific are to make truly data-driven decisions, they must leverage automated managed data pipelines to deliver the right data to the right place at the right time.
In a recent global survey, data engineers blamed inefficiencies for causing them to spend too much time fixing and maintaining pipelines and finding the right data, rather than focusing on analytics that generate a positive impact on the business. What are some key hidden challenges data engineers face in their role?
Chandler: Consider the nature of a data engineer’s job: to deliver data on time, in the right format, to the right people. If it’s done perfectly, nobody ever knows, but when something breaks, it affects the whole company.
One of the key challenges data engineers face in their role is the strain that modern data needs are causing to old and traditional infrastructure. This is because traditional, on-premise data infrastructure is not cost-effective, and does not provide scalability and flexibility, while tying data engineers to specific tasks and skillsets.
This has changed drastically over the past year, as the pandemic forced organizations to adopt cloud infrastructure and its associated tools, with the utilization of data warehouses, data lakes and other data stores.
Data engineers are also spending too much of their time and resources on the architectural, logistical, and engineering challenges of maintaining data pipelines. Too often, data engineers become a bottleneck or “single-point-of-failure” for all their data to get into the data warehouse, and that takes up the bulk of a data engineering function’s time and human resources.
This contributes to a talent shortage as well, because too many employees are spending their time on tasks that could be automated – building and maintaining data pipelines – instead of analyzing it to create real, insightful business intelligence.
Our survey of analysts found that 68% could extract additional business insights from existing data if they had more time. Bringing insights to decision makers turns data engineers from invisible laborers into shining heroes.
Being data-driven now means being more informed about the potential outcomes, because the world is constantly changing. Data engineers are finding it challenging to predict potential outcomes and apply mitigation strategies from their data.
Modeling hypothetical situations and new scenarios can help a business become more operationalized, as projections can help with the needs of inventory and customer base. Data engineers are not able to do that due to outdated infrastructure as well as the lack of resources and time.
What impact does these challenges have on organizations? How should organizations address these issues without heavy investments in resources?
Chandler: Due to data engineers spending most of their time hand-coding pipelines and writing APIs from scratch, an organization is unable to put their energy and resources into data science, finding new insights, or creating more complex reports that aid business transformation and financial growth. CDOs and executives are frequently disappointed knowing there’s an ocean of data out there, yet only a trickle of information available for them to make decisions.
This will further exacerbate the talent shortage and employee retention issue faced by organizations, as data engineers might not feel challenged within their job function–or worse, get burnt out from the monotony and stress.
Organizations can address this issue by implementing automated data integration technologies such as Fivetran into their data engineering function. With this, what used to take months to hand-code can be shortened to weeks or even minutes, as data flows instantly after setup and authentication is done. It is also self-healing in real-time to ensure no data is lost, even if source data structures change.
A service like this does not need heavy investment in resources from organizations, as it delivers data with a guaranteed service level agreement (SLA) and leverages its own engineers to eliminate all maintenance.
Furthermore, the pricing model is not based on the number of connectors that are installed, but straight volume. Organizations are charged according to monthly active rows – as the number of rows increases, the cost per row decreases. Thus, organizations only have to pay for the value they are getting from the technology.
With this, data engineers are moved to a greater position of power and relevance, serving as strategic partners that bring their expertise through domain knowledge. Moving them towards this part of engineering, where they are on the edge of being data scientists and analysts, will increase employee engagement and career growth opportunities.
How does Fivetran help to bring business benefits for sectors such as finance, retail, e-commerce and public service?
Chandler: Fivetran is used across all these industries, as benefits can be reaped accordingly for each.
We have a customer in Hong Kong, Neat, which offers a consumer financial application. They were recently trying to decide which tool to choose to help with their data, and of the tools that they were considering, the foremost one was: “Let’s just build it ourselves.”
This is because they wanted to keep control as they deal with financial data and personal data. So, they calculated how much time and effort their engineers would be spending building and fixing data pipelines to make real-time decisions.
With Fivetran, which provides comparable security and reliability, they found that they would be saving at least US$35,000 from the beginning. In this case, they found a tool that was able to work smarter rather than harder.
