DEC Blogs

The conundrum of modern data platforms

Written by Amir Sharabi | Oct 5, 2021 12:58:50 PM

 

 

Addressing Accuracy, Complexity and Growth

 

Many organisations who have introduced modern data platforms continue to face some common issues around data accuracy, complexity, the plethora of data available, and exponentially growing. The key area to address them is data management that focuses on governance, data lifecycle management, data literacy and fit for purpose architecture.

Data Accuracy

 

The importance of data quality cannot be overstated, nothing is more harmful to data analytics than poor and inaccurate data. Key causes for poor data quality are manual errors, inconsistent business/market definitions, and poor integrations resulting in corrupted/skewed data.

Making decisions relying on analysis that is rooted in poor data can have devastating consequences and erodes trust in data and the platform.

How to avoid these issues? A strong data management with documented and followed processes eliminating/minimising manual inputs is a good place to start. Set up a data cleansing project coupled with staff training and education. Formalise and harmonise business and market definitions to avoid misconceptions and variance in understanding certain measurements and metrics. Symmetrical system integration to ensure data is reflected accurately and consistently across different systems and platforms.

Data Complexity and Growth  

 

In the last decade we have seen an exponential growth in the volume and variety of data, multiplied by a growing number of data sources organisations are integrating and consuming.

This plethora of data adds complexity, especially since about 80% of the data not being governed as policies are not being enforced or outdated.

Why is data governance so crucial?

  • To prevent inconsistent data (eliminate multiple sources of truth)
  • Control the rapid growth of data replication with unclear origination
  • Address the ever-evolving regulatory and compliance requirements
  • Introduce data-as-a-service (DaaS) model to the organisation

To remediate this, organisations should review their entire data lifecycle, identify critical data entities and flows, and map their “as-is”. In parallel working on the future state and selecting the appropriate architectural approach would help future-proofing the platform to the growing demands and complexities.

For more about the challenges of modern data platform feel free to reach out to data@digitalevolution.consulting or fill out the contact form.