Executive Summary
Bring all data across the organization, owned by different business and technological platforms, into a Data Lake based on HaaS (Hadoop as a Service). For this we dealt with various forms and amounts of data, varied modes of transfer, ELT and Analytics.
Technologies / Tools / Frameworks used
- Hadoop ecosystem – MapReduce, Hive, Sqoop, Pig, Spark, Flume, Kafka, HTTPs, Oozie over Cloudera distribution.
- ETL – Pentaho
- Visualisation – Tableau, Datameer and home-grown dashboards
- Storage – HDFS, HBase
Business significance
- Creating a single platform where enterprise-wide raw source data can be ingested, then cleansed and enriched for the organisation's decision making.
- As Hadoop works on commodity hardware and is scalable and fault-tolerant, this setup is low cost and low maintenance.
- The Hadoop framework handles all kinds of data, structured and unstructured, which is not the case with other OLAPs available.