A VALUED CUSTOMER OF THIRDEYE DATA

Myriad Technologies: extracting lessons learned from government reporting

Myriad Technologies provides technical solutions to the defence and national security arm of the Australian government. Organisations of that kind generate a great deal of written reporting, and the value of it depends on somebody reading enough of it to notice what recurs. In practice the volume outpaces the reading.

LOCATION

Brisbane, Australia

COMPANY SIZE

Mid-scale Enterprise

INDUSTRY

Government and Public Sector

Lessons learned through extractive summarization

The requirement was direct. Agencies needed summarized versions of their reports so that the most relevant material could be identified quickly, across documents held in different formats.

ThirdEye Data built the system that produces those summaries. It handles the formats the material actually arrives in rather than requiring conversion first, and it extracts the lessons from each document rather than paraphrasing them, which matters in a setting where a summary that introduces language not present in the source is a liability.

Rather than choosing a summarization approach in advance, we implemented six different open-source algorithms and evaluated them against the customer’s own material. The one that performed best on their documents, rather than the one that performs best in general, was then run across the full corpus. Different document types reward different approaches, and there is no way to know which in advance without testing.

The system includes a rating mechanism, so the people using the summaries can assess the quality of what comes back. That serves two purposes: it identifies which approach is genuinely performing best in live use, and it gives the agencies a way to evaluate outcomes rather than accept them.

Six summarization approaches benchmarked on the customer’s own material, with the best performer run across the full corpus.

This engagement came to ThirdEye Data through Microsoft.

Related solution: summarization and document triage at volume

Summarization is usually presented as a single capability, but it’s really a choice between approaches with different failure modes. Extractive methods pull sentences from the source and cannot invent anything, but they read less smoothly. Generative methods read better and can introduce claims the document never made. In a regulated, legal or governmental setting that second failure is disqualifying, and the smoother output is not worth it.

ThirdEye Data selects the approach against the material and the consequences of being wrong, rather than defaulting to whatever is current. Where the stakes are high we test several approaches on the customer’s own documents and let the results decide, and we build in the means for users to rate output so the choice keeps being tested after launch. We have done this work for defence and national security reporting, for European case law, and for research literature at a scale no team could read.

Explore our document summarization and analysis solutions ›