AI-Powered Knowledge-Graph-Driven Document Generation

Our client is a major international pharmaceutical company. It conducts research and develops medicinal products related to a wide range of human medical disorders.

The project was focused on automating the creation of Investigational Product Preparation and Administration Instructions (IPPI)*.

Before we started the project, the IPPIs were made manually, using a number of templates. The process was time-consuming and prone to critical quality issues:

• deviation from regulatory templates causing legal risks
• inconsistent or ambiguous wording causing misunderstanding: certain terms and ways of explaining things were highly dependent on the author's style.
• errors in calculations of medication doses causing risks for the study participants' health.

The document is usually created by a group: a lead author, co-authors, and reviewers. It usually went through several review cycles, while authors and reviewers communicated via messengers, emails, and commenting inside the text documents. The client wanted to streamline the authoring and review process so that all communication would occur in one place and thus be traceable.

All these issues were aggravated by the fact that a large amount of subject matter knowledge was spread across documents and templates.

As a clinical study is a very complex multi-step process regulated by strict and sophisticated rules, an ordinary LLM* would not be able to solve the task appropriately and provide consistent results, i.e., similar wording for similar subjects in all documents. Making an IPPI requires understanding.

That's why for this task the project team literally taught the system to understand the domain knowledge:

1. Clinical Study Information Taxonomy. We structured clinical study knowledge into a machine-understandable format.
2. Attribute Form Information Taxonomy. We defined relationships between study attributes, Investigational Products (IP), preparations, and instruction sets.
3. IPPI Template Information Taxonomy. We digitized knowledge that was embedded in existing IPPI templates.
4. Contextual Relationship Model. We connected information items used in the instructions so the system understands context instead of processing fields independently. In other words, the information required by the system adjusts to the information that was already given: some questions add up, others become irrelevant. During the initial stage of IPPI creation, the LLM generates an attribute form (~Q&A form) that is filled in by the author. Once it is loaded, the LLM tries its best to populate the draft IPPI using available information for each node (question). Next, its authors turn to either validate the results or answer questions where LLM was not able to. As all nodes are connected to each other, certain answers may lead to changes in the set of other questions. If the system needs more information, it asks for it using new information fields.
5. Enterprise Knowledge Graph.* We created an Enterprise Knowledge Graph covering ~80% of the IPPI design space (estimated by subject matter experts). All identified relations were turned into nodes. Each node contains a predefined set of instructions telling the LLM how to handle it during IPPI generation. The EKG captured domain expertise and became a structured storage of reusable organizational knowledge. It supports automatic document generation by enabling intelligent reasoning across interconnected data elements.

Then, we created prompt packets—curated, structured collections of reusable prompts containing instructions on how to combine the information from the attribute form (Q&A form) to generate the text.

Finally, we developed the system that generates document content using created taxonomies, EKG nodes (questions+answers), and prompts from the collection.

After the IPP Instruction is generated, the authors can see details on how a particular part of the text was generated: what was the input from the author, which business rules were applied, etc.

We also implemented a functionality similar to the co-editing mode in MS Word. Now, all communication in the authoring team (authors, reviewers) is done through inline comments, while all changes can be tracked in the change log. This reduces risks of misunderstanding and keeps all the history of changes and discussions in one place.

The project was done by a distributed team consisting of

• client's internal team, responsible for structuring the clinical study knowledge into a machine-understandable format
• Software Country's team, responsible for implementing the application layer and stiching the things together.

Summary: challenges and solutions

We developed a tool automating IPPI authoring based on generative AI and an enterprise knowledge graph. Besides, it streamlines collaboration between authors and reviewers.

The developed solution reduced time needed to develop IPPIs by 82% and saved $1.5+ mln during the first year of usage. This speedup was accompanied by significant improvements in quality:

• Better compliance: reduced deviation from regulatory templates.
• Consistent terminology: reduced inconsistent wording.
• Reduced risk of errors and eliminated dosing calculation errors, leading to fewer health risks for patients and study participants.
• Less impact of authors' writing skills on quality. Lack of knowledge or understanding of some specific questions by a given author does not affect quality: the crucial knowledge is taken from the carefully curated knowledge base.

In the future, it is planned to use authors’ changes and reviewers’ comments to improve the knowledge graph and the prompts.