How We Handle the Messy, Unglamorous Work of Turning Utility Bills into Structured Emissions Data
Most ESG platforms stop at the dashboard. The charts are clean, the colours are on brand, the export button works. What you don’t see is the layer underneath: Where raw utility bills, sitting inside ZIP archives inside quarterly folders inside regional drives, actually become the numbers those dashboards display.
We spent a significant amount of engineering time on that layer. Not because it is exciting to talk about, but because it is the thing that decides whether ESG software is genuinely scalable or quietly subsidised by manual labour somewhere in the customer’s back office.
This blog is about the two pieces of that pipeline we are most proud of:
- How we walk a customer’s document repository to find the right bills, and
- How we run AI extraction on top of them in a way that keeps cost predictable at very large volumes.
Part 1: Discovery, Not Just Upload
The platform already supports the most natural way of getting bills in: Manual upload.
A user drops a single bill, or a small set of bills, directly into Credibl, and it flows straight into the extraction pipeline. For a sustainability lead handling a handful of documents at a time, that experience is intentionally as simple as attaching a file to an email.
But for large enterprises, manual upload alone is not enough.
Sustainability data at that scale is fragmented by nature. Bills sit on internal storage systems, organised by region, by facility, by quarter, often inside ZIP archives, often nested several folders deep. Asking a customer to manually upload tens of thousands of bills, or to reorganise all of that into a flat list before they can use the platform, was a non-starter.
So alongside manual upload, we built an automated ingestion layer that walks the customer’s document repository on its own. The customer points us at a root location and the system takes over from there.
Both paths share the same downstream AI extraction and the same human review queue, so whether a bill arrives one at a time through the browser or in a sweep across thousands of folders, the rest of the journey looks identical.
Under the hood, the ingestion layer performs a recursive traversal of the folder tree.
At every node, it makes a small set of decisions in sequence:
- Is this entry a directory? Recurse into it.
- Is it a compressed archive? Open it lazily and inspect its contents without fully extracting first.
- Is it a supported document type? Send it forward to the AI model.
- Is it noise? Skip it and move on!
Three details make this more interesting than a plain directory walk.
Contextual hints from folder names. The traversal carries a hint as it descends. When the ingestion layer sees a folder named “electricity” or “water” or “DG” along the path, it tags every document below that folder with the inferred bill type. By the time a document reaches the AI model, we already know what we are looking at. That tightens the prompt sent to the model and reduces the chance of misclassification. The same path also yields the facility code, lifted from the parent folder, which we later use to bind the extracted reading to the right physical site.
Idempotency by design. Every processed file is recorded against a per-organisation tracker keyed by its full path inside the repository, including its position inside any parent archive. On the next run, the ingestion layer consults this tracker before doing any expensive work. Files already processed are skipped without downloading them again. New files are picked up. Folders that yielded no bills on previous runs are remembered so we do not waste cycles re-traversing them. The result is a system that behaves more like a long-running synchroniser than a one-shot importer.
Lazy archive inspection. Rather than always downloading a full archive to inspect it, the ingestion layer first reads only the archive’s listing, which is cheap, and then compares that listing against the tracker. If every file inside the archive has already been processed, the archive is skipped entirely — no bytes downloaded. If only a subset is new, only that subset is materialised locally and pushed downstream. On large quarterly archives, this alone cuts ingestion time by an order of magnitude.
Part 2: AI Extraction Without The Runaway Bill
Once a bill is identified, it has to be read. This is where the economics of an AI-driven platform either work or they don’t.
A modern vision-capable AI model is impressive at reading a utility bill, but it is not free. If you naively call the AI synchronously, once per bill, for every customer, every run, the per-document cost adds up fast.
On a customer base with millions of bills flowing through per year, that arithmetic becomes the difference between a sustainable product and an unsustainable one.
We solved this with batching.
When the platform starts pulling in a customer’s bills, the ingestion layer does not pass them to the AI model one at a time. It groups them first.
Bills are accumulated into a logical group, tagged with a single batch identifier, and that group is submitted to the AI model as a coordinated unit. Each member of the batch carries its own prompt, tuned for the bill type detected during ingestion, and its own document payload, but they all travel together as one submission.
The advantages compound quickly.
Pricing. Batch mode on the underlying AI model is offered at a meaningful discount compared to synchronous calls, because the provider can schedule the work in their own time rather than guaranteeing low latency. For a workload that is fundamentally not latency-sensitive — a customer is not standing there waiting for a single bill — this is exactly the right trade. We absorb the higher tail latency and pass the lower unit cost through to how aggressively we can scale ingestion.
Throughput. Calling the AI model one document at a time is bounded by how many concurrent requests we are willing to open. Batch mode lets us push hundreds or thousands of documents into a single submission, with the AI provider handling the parallelism internally. That moves the bottleneck out of our control plane and into the AI infrastructure itself, which is exactly where we want it.
Operational coherence. Because every document in a run shares a batch identifier, the entire run can be tracked, sealed, and reconciled as one logical operation. We know exactly when the batch is complete, exactly which members succeeded, exactly which need follow-up. The customer sees a single, clean summary at the end of the run, not a trickle of individual completions.
A Few Additional Details Make This Reliable At Scale
Each batch carries caps: A maximum number of members and a maximum total payload size.
When a run exceeds those thresholds, the ingestion layer transparently rotates into a new batch, so a single very large run never produces an unwieldy submission that the AI provider might reject or slow down.
Every batch member is prepared before submission. We materialise the document locally, pick the right prompt for the detected bill type, optionally enhance it with OCR output for bills where scan quality is borderline, and persist a small slice of context against the job so that when the AI returns results, we know exactly how to interpret them.
This pre-processing is cheap compared to the AI extraction itself but pays for itself by sharply reducing the rate at which the AI returns unusable output.
Failures are isolated. If a single bill in a batch of a thousand fails to extract, the rest are unaffected.
The failed member is routed into a recovery path where it can be retried, downgraded to a simpler extraction strategy, or surfaced to a human reviewer with full context. The batch as a whole still seals cleanly.
Every result, regardless of how it was obtained, lands in the same review queue, where the customer’s team makes the final call before any number enters the platform’s emissions calculations.
Part 3: Why The Two Pieces Need Each Other
The ingestion layer and the AI batch pipeline are often described as if they are separate concerns. In practice, they are deeply connected.
The ingestion layer’s job is to convert a messy, unbounded, customer-specific document repository into a clean, ordered stream of work units — and to do the same for manual uploads coming in through the browser. The AI batch pipeline’s job is to turn that stream into structured data at a cost point that scales.
Neither piece is enough on its own. Without the ingestion layer, the AI has nothing useful to work with — just whatever a human happens to have uploaded. Without batching, the ingestion layer surfaces work faster than we could economically process it.
Together, they form a pipeline where adding a new customer with hundreds of facilities and tens of thousands of historical bills is a non-event. Whether the documents arrive one at a time through manual upload or in a sweep across an entire document repository, the system absorbs the volume, sends it to the AI in coordinated runs, recovers automatically from transient failures, never reprocesses the same document twice, and produces a deterministic, auditable, reviewable set of structured records on the other side.
That is the part of the platform that almost nobody sees. And it is the part we believe matters most.
Where We’re Taking It Next
The bill ingestion engine is not the finish line.
The same skeleton is already being extended to fuel logs, refrigerant records, water meter readings, and other categories of evidence that have traditionally relied on someone sitting in front of a spreadsheet for a week at a time.
Wherever there is a number trapped inside an unstructured document, the pipeline can be pointed at it.
Less typing. Lower cost per record. Faster reporting cycles. More trustworthy numbers.
That is the bet.
Special thanks to Anurag Borisa, Software Engineer at Credibl and the engineering mind behind this post, for partnering with our content marketing team to bring technical depth, expertise, and real-world insights to this piece.
