Remember the supermarket runs on toilet paper and the shortages of PPE? Can data intelligence stop these crises from ever recurring?
The pace of change is accelerating, and disruptive anomalies are becoming increasingly common. In 2021, I believe that every organization should be enacting The Great Digital Switch through data to react more quickly, read signals more clearly, and outline options for action.
This has given rise to the five most important cloud data integration trends that he believes businesses should be on the watch for next year 2021:
- SaaS is everyone’s new best friend
The increased use of cloud providers and online services has been essential to keeping the lights on in virtual environments. This has prompted companies to overcome the inertia and red tape surrounding Software as a Service (SaaS), Platform as a Service (PaaS), and other XaaS products. Organizations that once claimed they would never look to SaaS have had to embrace it. These changes have had some beneficial side effects, like adding scale and elasticity. Additionally, the pace of innovation in data and analytics is swift, and SaaS provides immediate access to new technologies like augmented analytics, facilitating transformation.
So, while bigger projects have been put on hold in the short term, the immediate switch to SaaS will be a trigger for a greater migration of databases and applications. Technologies that can access, move, and harmonize data from multiple places will follow. Containers and serverless infrastructure hold great potential for running applications in the cloud but using them at scale requires organizational maturity and significant know-how. The ability to manage hybrid deployment across multiple clouds will continue to be key to avoid vendor lock-in.
- Shared data, visualizations and storytelling are consumed by the masses
In 2020, data and data visualizations flooded mainstream news. General audiences pored over data in sources like European Centre for Disease Prevention and Control (ECDC) and Our World in Data. Now more than ever, we have seen the importance of delivering the last mile in data storytelling and infographics. There has been a massive up-leveling in the conversation about data, where armchair epidemiologists are able to say things like, “That’s a logarithmic scale” and “Here’s the problem with comparing per capita.”
This development will bring in millions more on the journey toward data literacy. But data is too often becoming politically fraught. How do we double-click beyond the picture? Get to the point behind the data point? Surface lineage and easily bring in new data sets? Technically, an expansion of context will be supported by more common data models and more business logic, accessible in catalog and data marketplaces. This will help synthesis and more productive discussion. But we will also need to start building ways of agreeing on the common ground, and work on an etiquette for intellectual honesty in debating data.
- Up-to-date and business-ready data are more important than ever
Since the pandemic, we have seen a surge in the need for real-time and up-to-date data. What is usually fairly stale—quarterly business forecasts, for example—is fleeting and mutable now. Alerts, data refreshes and forecasts will need to occur more often, with the freshest variables. On a macro level, we have seen disruptions to supply chains, with hospitals scrambling to procure Personal Protective Equipment (PPE) and consumers stockpiling toilet paper. In the case of PPE, we reacted to an actual shortage too slowly; with toilet paper, consumers broke the supply chain by assuming a shortage where none existed. Surges like these are accentuated in a crisis, and we have to build preparedness for them.
As the velocity of data increases, the speed of business needs to follow. Can we make ‘business-ready’ data (data that is not only curated for analytics consumption, but which has timely business logic and context applied to it) accessible earlier? And can we automatically trigger the end points, whether that is an automated process or an action taken by a human? The infrastructure and applications are available, enabling a gradual transition to active intelligence. That will be a big factor in helping enterprises pre-act.
- We need to capture and synthesize ‘alternative’ data
How early could we have detected the COVID-19 when it was an outbreak? Studies of ‘alternative’ data—in this case, traffic data outside hospitals in Wuhan and keyword searches by Internet users in that area—indicate that the virus may already have been circulating in late 2019. The investment community has been a pioneer in using alternative data, including audio, aerial photos, water quality, and sentiment. This is the front line for data-driven innovation and getting an edge here can result in huge gains. But in the wake of 2020, alternative data will become mainstream, with the goal of spotting anomalies much earlier.
From that, we can get derivative data, which comes from combinations, associations, and syntheses with data from systems of record. As IDC noted: “As more data gets captured and becomes available from external sources, the ability to use more of it becomes a differentiating factor. That includes taking lessons from industries other than your own.”
This trend, similar to what Gartner calls “X analytics”, is not new but is finally becoming an important foundation of modern data and analytics, thanks to cheaper processing and more mature AI techniques, including knowledge graphs, data fabrics, natural language processing (NLP), explainable AI and analytics on all types of content. This trend is completely dependent on ML and AI, as the human eye cannot catch it all.
- Collaboration has to coalesce earlier in the chain
In 2020, we have seen a step change in the embracing of web conferencing, remote collaboration and online learning. We are in a new world where we cannot gather as often for a quick huddle in the office or sketch out an idea on a whiteboard.
Also, an increasingly fast-moving world means that in many cases, people do not have time to wait to make a decision while someone builds a dashboard. The convergence of data management and analytics in the market has created opportunities for integration points between the components of a data pipeline, combining synthesis with analysis and enabling active metadata, business logic and catalogues to act as connective tissue. This in turn will push collaboration, innovation and discussions to the data itself.
We will see more experiences that introduce easy and enjoyable ways of working together in areas that were previously rendered ‘boring’ or ‘difficult’.
Tasks for moving data from raw to analytics-ready will become more engaging, fast and iterative. The separate, siloed worlds of data curators and consumers will begin coming together, and business logic will persist, enabling analytics-ready data to become business-ready much faster.
