Resolving data challenges for transformational AI – Economictimes CIO

It is not common to use data from the warehouse into various front-end applications, this is because the warehouse is a single storage of the entire data, and if a front-end application (e.g. a customer facing App, a distributor portal etc.) requires to fetch data from that large storage, the response time is usually very high, which is not acceptable for most front-end applications.

An agile approach to data integration attempts to change this paradigm and bring in a use-case-driven approach to data integration. The key element of this approach involves creation of a data lake. As opposed to a data warehouse, a data lake stores data in an AS-IS format without performing any transformation of the data. One performs only Extraction and Loading (EL) of the data in the as-is format from the source system into a single repository. The data lake also stores unstructured data like text, images etc. In this case, there is no transformation or summarization of the source data, hence there is no loss of information. The dramatic reduction in storage costs have made it possible to store data in the as-is form.

Using the right platform, a data lake can be created very quickly (within 4-6 weeks). Once, the data lake has been created, an agile approach can be used to combine the few tables that are required for creating a particular use case. These tables are combined into a mini data mart. For example, one may need to identify the relationships between a handful of 10 to 15 tables only to create a data mart that is required to analyse the effectiveness of salespersons. Each data mart supports specific insight generation or machine learning model development needs. This approach ensures that organizations see initial results very quickly.

Each data mart consists of relatively fewer data elements compared to a warehouse; hence, they can be used to provide data to front-end applications, within prescribed response times. Data from the data marts and results of machine learning models are exposed as APIs (application program interface) which can be consumed by various front-end applications.

A key aspect of using data from the data mart for various front-end applications involves the frequency at which data from the source systems are refreshed into the data lake. An ideal solution is to perform a near-real-time refresh, and to capture changes where the source data is overwritten. Specific capabilities like change data capture and ability of reading updates form data base logs is critical for this purpose.

The data lake approach allows different data marts to obtain different types of summarization of the same base data. As the granular as-is data is present in the data lake, one can perform different types of summarizations from the same data. Within a data lake framework, it is critical to have utilities that can convert (or make sense) of unstructured data like images, scanned PDFs, etc. and incorporate the same with traditional structured data.
Once a specific data mart has been created for one use case, then the next set of data marts can be created whenever needed – the agile data journey. The data lake can also use existing data warehouse as a data source thereby reusing existing investments in the data warehouse.

The Author is Head Product and Engineering at Actify Data Labs