A VALUED CUSTOMER OF THIRDEYE DATA
Price.com: pipelines that keep pace with 500 retailers
Price.com lets shoppers compare prices across thousands of retailers in one place. A comparison service is only as good as the freshness of what it compares, so ingesting retail data is the product, not the plumbing behind it. Prices change constantly, retailers publish differently from one another, and a stale price is worse than no price.
LOCATION
San Francisco, California, United States
COMPANY SIZE
Small-scale Enterprise
INDUSTRY
Retail › Explore solutions for this industry
Automated retail data pipelines on a cloud data lake
ThirdEye Data built the cloud data lake and the automated pipelines that feed it. The pipelines pull retail data from more than 500 online retailers, each structuring and publishing its data differently, and bring it into a consistent form the comparison service can query.
Feeds run on different rhythms depending on what they carry, on daily, weekly and monthly cycles, and also on event, so that something which changes unexpectedly is picked up when it changes rather than at the next scheduled run. Products are aggregated by universal product code across those same cycles, which is what allows the same item sold by twenty retailers under twenty different descriptions to be recognised as one item.
The requirement that shapes everything in a system like this is that it has to keep running without supervision. Any pipeline that needs a person to intervene when one retailer changes its format does not scale past a few dozen sources, and this one had to carry hundreds.
Retail data ingested automatically from more than 500 online retailers.
Related solution: data engineering at scale
Ingesting from many sources is a different discipline to ingesting a lot from a few. The volume is often modest. The difficulty is that every source has its own format, its own reliability, its own idea of what a field means, and its own habit of changing without warning. Systems built for one clean source break immediately when they meet a hundred messy ones.
ThirdEye Data builds for the messy case. That means pipelines that fail on one source without taking the rest down, aggregation logic that can recognise the same entity described differently by different publishers, and enough observability to tell the difference between a source that has gone quiet and a source that has changed shape. We have built this for retail price comparison across hundreds of merchants, for a distributor consolidating over 200 internal and external sources, and for a coaching business with two decades of systems that were never designed to talk to one another.
Explore our data engineering and pipeline solutions ›
DO YOU HAVE A SIMILAR USE CASE?
Talk to our team to learn more about what we did for Price.com, and how that experience applies to your own environment.
Explore What We Did for Other Customers
