To quantify how accurately the algorithmic classifications predict categorizations based on human evaluations, we asked two researchers with expertise in behaviour genetics to categorize 65 randomly sampled presentations. The raters worked independently, without any external assistance, and based their categorizations solely on information supplied about the title and abstract. Each rater assigned three yes/no labels—representing candidate-gene study; twin, family or adoption study; or PGI study—to each presentation. Raters sought to make labelling decisions consistent with the labels’ typical usage in the literature. We defined “agreement” on a presentation as an identical judgment about each of the three labels (i.e., if the raters disagreed about any of the three categories, they were considered as not agreeing). Even under this strict definition, we found an interrater agreement of 94%. The agreement between the algorithm’s and one rater’s categorizations was 86%, and that between the algorithm’s and the other rater’s categorizations was 83%.
