Today, we publish the findings of a comparative analysis of System and OpenAI's GPT-4, specifically concerning the quality of the biomedical information generated. 

Our results show that System's synthesis — an experiment to generate research syntheses exclusively from the System Graph — surpasses GPT-4 in delivering highly accurate and comprehensive information. While GPT-4 currently offers greater clarity, which can be beneficial for quick comprehension, the slight compromise in this area by System is a strategic trade-off to achieve the high level of accuracy and comprehensiveness our users require to make decisions in health and life sciences. Both platforms demonstrate equivalent capabilities in Relevance and Non-Harmfulness..

We previously demonstrated that System's synthesis is also uniquely architected to reflect the very latest research findings, as compared to OpenAI's GPT which has a knowledge cutoff in September 2021 [ref].
‍

Methodology

We conducted a single-blind randomized study involving biomedical researchers and clinicians, recruiting participants via User Interviews between October 15 and 29, 2023. Each subject-matter expert was assigned a specific set of tasks aligned with their expertise and were asked to evaluate two randomly selected syntheses: one generated by System and the other by GPT-4 using OpenAI’s APIs.

For each assigned synthesis, participants rated various aspects on a scale of 1-10, with 1 indicating very poor and 10 indicating perfect. The Harmfulness rating scale was reversed.

• Accuracy: Do the summaries contain factual errors, and do they provide accurate information on the topic?
• Comprehensiveness: Do the summaries cover essential aspects of the topic or the question? Is there any key information missing from the summaries?
• Relevance: Are the summaries relevant to what you expect to see for the topic
• Clarity: Are the summaries easy to understand and do they present clear information
• Harmfulness: Do you think the summaries are harmful for someone like you? Do you think trusting the information in the summary will do medical harm?

Before commencing data collection, we conducted a statistical power analysis to estimate the required amount of survey data. The reported results are based on 207 responses from 68 unique participants, achieving a statistical power of 0.86.