A Flash Manufacturer’s Success Story
Extending Flash Manufacturer’s Predictive
Platform to predict in real time while using 10x additional data points.
THE CUSTOMER
ThirdEye’s client is a Fortune 500 company that produces flash drives and other associated software for data storage. The company boast an impressive number of 10,000+ clients worldwide. They provide the fastest and most reliable access to data, both on-premise and in the cloud.
BUSINESS GOALS
The Company’s flagship product is a predictive analytics engine that delivers data insights by predicting and preventing issues across the entire infrastructure stack. The product uses Big Data and Data Science technologies to correlate trillions of sensor data points, to identify, predict, diagnose and resolve storage issues. The predictive analytics engine enables customers to avoid disruption and reduces time in dealing with infrastructure problems. By leveraging predictive analytics, the solution identifies future storage needs and
potential hotspots to simplify planning.
The Company started facing an unexpected and exponential growth trajectory and realized that the current
infrastructure and solutions were unable to scale to meet existing and future market demands. The Company needed to quickly turn around the current infrastructure to scale as per the demand while keeping capital and operating expenses (Capex & Opex) to a minimum. Additionally, the software stack used for implementing the current predictive analytics engine had reached its life cycle expectancy and needed a major overhaul.
THE SOLUTION
To extend the predictive analytics engine to the next level, ThirdEye’s team of Data Engineers and Data
Scientists collaborated with the client taking a three phase (detailed below) and a two-tiered approach –
1. Build, deploy and evaluate POCs
2. Scale POC into production.
Phase 1 – Strategy, Advisory & Architecture
Partnering with the client’s architect and management teams, ThirdEye’s strategic consulting arm undertook an overall assessment of the current predictive analytics engine’s architecture, technology stack, scalability bottlenecks and Opex & Capex considerations. Collaborating with the client’s strategic team, ThirdEye proposed a detailed architecture with various options and a scalability road map for each option.
Phase 2 – Infrastructure & Data Ingestion
ThirdEye’s Engineering team collaborated with the company in deploying a series of infrastructure and data Ingestion centric POCs to evaluate improvement based on set success criteria. The successful POC was rolled into production subsequently.
Phase 3 – Real-Time Analytics and Machine Learning Applications
Concurrent with Phase 2, ThirdEye’s Data Science team collaborated with the company to build in-memory OLAP cubes capable of analyzing data in real time, performing ad-hoc & predictive queries in real time, performing drill downs and root cause analysis for finding problem resolutions. Additionally, real-time machine learning applications were built to perform predictive maintenance, forecast breakdowns, identify patterns and
perform what-if scenarios. Here again, the approach was two tiered by first building POCs and then scaling successful applications into production.
Technologies Used
- Apache Spark
- Apache Ignite
- Apache Kafka
- HDFS
- Vertica
- App Server
- R Server
- D3.js
VALUE CREATED
Through several significant technological breakthroughs, the following performance improvements were
achieved:
- Ability to ingest 10x additional sensor data points into the system
- Rate of data ingestion was reduced by more than 50%
- Achieved near real-time data analytics capability
- Improved analytical queries response times ability by more than 90%
DATA ANSWERS
ThirdEye answers your data questions and offers actionable insights, real-world experiences and strategic recommendations.
