Automated Data Pipeline

ThirdEye build a AWS Cloud based Data Lake for

  • Developed automated data pipelines for ingesting retail data from 500+ online retailers.
  • Persisted all data feeds on a ElasticSearch cluster on a daily, weekly, monthly and on-event basis. 
  • Aggregated data based on UPC of retail items on a daily, weekly and monthly basis.

Technologies Used: Amazon RedshiftElasticSearchKibanaApache SparkApache Kafka, MySQL RDBMs, PythonJava programming languages