Alfraido now reads reports and extracts data on its own

Most datasets do not arrive in that state. In our field, the numbers that matter are rarely stored in a database. They sit in a laboratory report saved as a PDF, in a spreadsheet built by a technician who used the layout that suited him at the time, in a scanned certificate, or in a table buried in the middle of a paper. Each source uses its own vocabulary, its own units and its own way of marking a value that was not measured.

The work of converting all of this into a single, coherent table is well understood and widely disliked. It is repetitive, it is slow, and it is usually done by the person who is least interested in doing it. It is also where most of the time of a modelling project is spent, and where most of the errors are introduced.

What changed

In its latest deployment, Alfraido removes the manual part of this step. Data can be uploaded as it is. It can be an Excel sheet, a PDF report, a scanned document or a paper. Alfraido locates the relevant values inside the file, maps them onto the variables of the model, and returns a structured dataset ready for training.

The extraction runs as a background job rather than inside the upload request. A long report can take some time to process, and the interface does not block while it does. The user uploads the file, the job is queued, and the extracted table appears when it is finished.

The output is not a copy of the document. It is a normalised table: units converted to a common basis, column names aligned with the model's variables, and unmeasured quantities marked as missing instead of guessed.

What still needs a human

Extraction is inferential, not deterministic. The system reads a document and decides what each number means, and that decision can be wrong, particularly with ambiguous headers, footnotes that redefine a unit, or tables split across two pages.

For that reason the extracted table is presented for review before it is used. Values can be corrected, mapped to a different variable, or discarded. The intended saving is not in the verification, which stays with the user, but in the transcription, which no longer has to be done by hand.

Why this matters for the models

The practical effect is on the size and the diversity of the training set. When adding a dataset costs a day of manual formatting, old reports stay in the archive and the model is trained on whatever was already convenient to use. When adding a dataset costs an upload and a review, the archive becomes usable.

More data from more sources gives a model that generalises better and that is less dependent on the conditions under which the convenient data happened to be collected. That is the reason the feature was built.