← Case studies Retail and procurement

A pipeline that turned 200 supplier price lists into one queryable table

A retail group received supplier pricing as PDFs, spreadsheet attachments, and portal downloads, in around 200 different formats, and three analysts spent most of a week each month turning it into something the buying team could compare. We built an ingestion pipeline that collects, extracts, normalizes, and loads it automatically, with change detection so buyers see what moved rather than a full dump.

Scope: Collection, extraction, normalization, and load for about 200 suppliers
Timeline: 7 weeks, delivered supplier tier by supplier tier
Delivery: Runs in the client's cloud account, source in their repo
Volume: About 4,500 documents a month, roughly 90,000 price lines

31 hours → 20 minutes Weekly effort to get current supplier pricing into the warehouse

01The mess before

Pricing arrived in every format a supplier could invent

Some suppliers emailed a spreadsheet. Some attached a PDF with the prices in a table image. Some published to a portal that had to be logged into and clicked through. A few sent a link that expired. The only thing they had in common was that a person had to open each one.

Three analysts, one week a month

The work was copy, paste, reconcile, and chase. By the time a full picture existed it described pricing from three weeks earlier, which is not much use when you are deciding what to buy this week.

Nobody could tell what had actually changed

Each month produced a fresh spreadsheet rather than a record of movement. Answering whether a supplier had quietly raised a price meant opening the previous month's file and comparing by eye, so mostly nobody asked, and increases went through unnoticed.

Reruns made everything worse

When a load failed halfway, rerunning it appended the rows it had already written. The team learned to avoid reruns, which meant a failure usually became a manual repair rather than a retry.

02What we wired together

Python extraction workers

Per-format extractors for spreadsheets, native PDFs, and scanned tables, each one testable against saved fixtures.

Vision model for scanned tables

Reads the minority of price lists that arrive as images, with a confidence score per field and a review queue below the threshold.

PostgreSQL

The normalized landing zone: suppliers, products, prices, and a full history of every observed change.

Temporal

Schedules and orchestrates each supplier's collection and load as a durable workflow with retries and backfills.

Email intake

Watches the shared procurement mailbox, pulls attachments, and routes each one to the right extractor by sender and shape.

Slack

Where buyers get the daily change digest and where extraction failures are raised with a link to the document.

03How the build ran

Sorted suppliers by pain, not alphabetically

We ranked all 200 suppliers by how much analyst time each consumed and built extractors for the worst twenty first. Those twenty accounted for a bit over half the total effort, so the pipeline was saving real time in week three rather than at the end of the project.
Built collection separately from extraction

Getting the document and reading the document are different problems with different failure modes. Collection handles mailboxes, portal logins, and expiring links, and stores the raw artifact untouched. Extraction runs against stored artifacts, so a parser fix can be replayed over history without re-fetching anything.
Normalized to one product identity

The same item appears as different supplier codes, pack sizes, and units. We built a mapping layer with an explicit review step for new or ambiguous codes, because a wrong automatic match is far more expensive than an unmatched row a buyer resolves once.
Made loads idempotent from the start

Every price row is keyed on supplier, product, unit, and effective date. Loads are upserts against that key, so rerunning yesterday's load changes nothing. The team can now retry a failure instead of repairing it.
Shipped change detection, not a data dump

Because every observation is stored with its date, the daily Slack digest is a diff: which prices moved, by how much, and which suppliers went quiet. That is the artifact buyers actually use, and it did not exist in any form before.

04Under the hood

Most of the work was not the AI part

Around 80 percent of documents are spreadsheets or native PDFs with real text, and deterministic parsers handle those faster, cheaper, and more reliably than a model. The vision model is reserved for the roughly 15 percent that arrive as scans or table images. Routing by document type first, rather than sending everything through a model, cut running cost by about an order of magnitude and made most of the pipeline debuggable with a stack trace.

Confidence thresholds decide what a human sees

Every extracted field from the vision path carries a confidence score. Rows below the threshold, and any row where the price moved more than a set percentage, go to a review queue rather than straight into the warehouse. About 6 percent of extracted rows are reviewed, which is what keeps buyers trusting the numbers enough to act on them.

Raw artifacts are kept, which makes fixes cheap

Because the original file is stored alongside the extracted rows, improving a parser means replaying it over two years of history rather than asking suppliers to resend. Three separate parser improvements were backfilled this way, each one silently correcting historical rows nobody would otherwise have revisited.

Portal collection is the part that keeps breaking

Supplier portals change their markup without notice, and this is the maintenance cost nobody scopes. Collection failures are isolated per supplier, so one broken portal never stops the other 199, and the failure arrives in Slack with the supplier name and the last successful run rather than as a silent gap in the data.

05Before and after

Metric	Before	After
Weekly analyst effort on pricing	About 31 hours across 3 people	About 20 minutes reviewing the queue
Age of the pricing data buyers work from	About 3 weeks	Under 24 hours
Suppliers covered automatically	0 of about 200	About 185 of about 200
Price increases caught in the month they happened	Rarely, found later or not at all	Flagged in the next day's digest

Where it landed

Supplier pricing went from a monthly archaeology project to a daily feed. Buyers open Slack and see what moved overnight, the analysts who used to rekey spreadsheets now work on supplier negotiation, and a price increase gets noticed the day it appears instead of during the following quarter's review.

Have a build like this one?

Book a scoping call →