Price.com

Automated Data Pipeline

ThirdEye build a AWS Cloud based Data Lake for Price.com.

• Developed automated data pipelines for ingesting retail data from 500+ online retailers.
• Persisted all data feeds on a ElasticSearch cluster on a daily, weekly, monthly and on-event basis. 
• Aggregated data based on UPC of retail items on a daily, weekly and monthly basis.

Technologies Used: Amazon Redshift, ElasticSearch, Kibana, Apache Spark, Apache Kafka, MySQL RDBMs, Python, Java programming languages