Certificate of Analysis (CoA) Data Extraction Without the Manual Entry

A certificate of analysis is one of the densest documents in any regulated industry. A single CoA carries test results, method references, accreditation details, chain-of-custody data, and the lab’s sign-off, all laid out for a person to read rather than a machine to parse. That density is exactly what makes it hard to extract reliably.

The document is consistent in what it contains but not in how it presents it. Manual extraction works at low volume and falls apart the moment the volume grows or the number of issuing labs does. Below is the approach that holds up in production.

What a CoA contains, and why each part matters

A standard certificate carries six things worth extracting, and each one fails in its own way. Get any of them wrong and the error flows straight into the record that feeds a compliance submission.

Results table. The parameters, their measured values, units, and the applicable limits. This is the data most systems actually want, and the section most prone to layout variation.

Method references. The test method behind each parameter, like ISO 17294-2, EPA 200.8, or BS EN 1484. It matters for audit trails and compliance documentation.

Accreditation details. Which body accredits the lab, the scope reference, and the tests inside that scope. In a regulated industry, using a result from outside the accredited scope is a compliance risk.

Sample information. Sample ID, sampling date, receipt date, analysis date, and the matrix (water, soil, air, biological). These have to stay linked to the results, not float free.

Detection limits. The lowest level the method can reliably see. A result of < 0.01 needs its detection limit to read correctly as a non-detect, not as a value of 0.01.

Signatory and issue date. Who signed the certificate and when. Required for chain-of-custody documentation.

Why a CoA is harder than an invoice

Invoice extraction is close to solved. The layout varies, but the meaning doesn’t: every invoice has a total, a date, a vendor, and once you know the layout the values are easy to place. A CoA is harder on five fronts.

Results are relational, not flat. Every value has to stay tied to its parameter name, unit, detection limit, and sample ID. Break those links with a misaligned table and you get orphaned numbers, or a value pinned to the wrong parameter.

Non-detects are not zero. A result below the detection limit is a non-detect, usually written < DL. Plain numeric extraction misses it. The system has to read <0.01, ND, BDL, Not detected, and < LOQ as non-detects and capture the limit as a separate value.

Units have to survive exactly. mg/L and µg/L sit a factor of 1,000 apart; mg/kg and mg/L describe different matrices. Normalise the units or strip the special characters and you get numbers that look right and are off by orders of magnitude, which for a compliance submission is a critical error.

Lab formats vary wildly. Invoices come from a handful of software packages; lab reports come from many. Some labs export structured PDFs from a LIMS, some generate Word, some hand you a scan. Each has its own table structure, column order, and convention for how qualifiers and limits appear.

Accreditation needs a cross-reference. Reading that pH was measured by Method X is easy. Confirming Method X sits inside the lab’s UKAS or A2LA scope needs domain knowledge built into the pipeline, not just extraction.

Define the schema first

Schema-first extraction matters more for CoAs than for almost any other document, because the downstream use depends on the relational structure staying intact. A flat list of numbers isn’t just less useful here; it’s dangerous.

The schema names every field a single result needs and forces the pipeline to keep them together: the normalised parameter, the raw parameter as printed, the value, a non-detect flag, the detection limit, the unit exactly as reported, the method, and the specification limit. The certificate-level fields, like the lab, the certificate number, the dates, the sample ID, the matrix, and the accreditation body and number, wrap around that list of results.

The load-bearing rule is the one on non-detects: if there is no numeric value, the non-detect flag has to be set. That single constraint stops the pipeline from silently dropping a < 0.01 and storing nothing. Instead it stores the detection limit and which kind of non-detect it was, so the downstream system always knows what the blank means.

Extraction approach by CoA type

How hard a certificate is to extract depends almost entirely on how it was produced. Four cases, easiest to hardest.

Structured PDFs from a LIMS are the clean case. Clear column headers, values in predictable positions, and text extraction with pdfplumber or PyMuPDF gives reliable output for rules. Most fields come out with high confidence.

Word-generated PDFs look clean but are often malformed underneath: merged cells, split rows, text wrapped inside a cell. Table extraction with explicit settings usually beats the defaults here.

Multi-sample certificates put several samples on one page, often as transposed tables. The logic has to tie each result column to the right sample ID, and this is where alignment errors do the most damage.

Scanned CoAs need an OCR layer first, and quality swings hard: a clean laser print reads well, a photocopy of a fax reads badly. Below a quality threshold, routing the scan to a person beats extracting noise.

Confidence scoring and validation

Confidence scoring on a CoA has to fit this document’s failure modes, not a generic template. Every extracted field is scored on the checks that actually break here, and anything that fails routes to review instead of sliding through.

A few cross-checks catch the rest: does the certificate number match the lab’s format, do the analysis dates fall after sampling, do the units fit each parameter? Fail one, or score low on alignment, and the field goes to human-in-the-loop review before the record is accepted.

In practice: the water consultancy pipeline

The CoA pipeline I built for a water consultancy handles certificates from more than ten laboratories in daily operation. Non-detect handling and unit preservation were the two failure modes that needed the most careful design.

Non-detects go through a normalisation layer that maps every known representation to one flag before any number is read, with the detection limit pulled out as its own field. Units are handled as a match against a validated vocabulary of water-quality units, not free-form extraction: if the string doesn’t match a known unit for that parameter, the field is flagged for review rather than trusted.

The payoff is that certificates from labs with very different formats, some tightly structured and some quite loose, produce the same clean structured output, and it feeds straight into the compliance dataset.

FAQ

What Python library is best for certificate of analysis extraction?

For structured digital CoAs, pdfplumber handles table extraction reliably. For complex layouts with merged cells or wrapped text, PyMuPDF with coordinate-based extraction gives more control. Scanned CoAs need an OCR step first: pytesseract on-premise, or a cloud OCR API when volume and accuracy demand it.

How do you handle non-detect values in CoA extraction?

Never treat a non-detect as numeric zero. Build a normalisation layer that maps representations like <0.01, ND, BDL, and Not detected to a non-detect flag, and extract the detection limit as a separate field. Store both, because the downstream system needs the limit, not just the fact that the result was below it.

Can IDP reach high accuracy on CoAs from multiple laboratories?

Yes, with a per-laboratory extraction profile. A single generic extractor rarely does well across diverse formats. What works in production is a lab-identification step followed by lab-specific rules, all producing output against the same validated schema.

How long does it take to onboard a new laboratory’s CoA format?

For a well-structured digital CoA, a new extraction profile usually takes a few hours to a day: analyse the format, write the rules, test against a sample batch. The schema never changes; only the rules for that lab are new.

What happens when a CoA fails extraction?

A well-built pipeline fails loudly. If required fields are missing, the table won’t parse cleanly, or confidence drops below threshold, the document is flagged and sent to a review queue. A person checks the original against the extraction and corrects it before the record is accepted downstream.