A study of OpenAI’s GPT training data found a marked difference between models tested on copyrighted, non-public books: GPT-4o scored well above random chance, while two other models performed close to chance. Researcher Ilan Strauss used the results to examine potential access-rights violations, not simply whether material appeared in training.
The study, titled Beyond Public Access in LLM Pre-Training Data , used a legally acquired collection of 34 O’Reilly Media books containing 13,962 paragraphs . Strauss applied the DE-COP membership inference attack to test recognition of paywalled material that, the study says, the models lacked permission to use for training.
Testing OpenAI’s GPT training data
GPT-4o recorded an AUROC score of 0.82 , compared with 0.54 for GPT-3.5 Turbo and 0.56 for GPT-4o Mini. On this measure, 0.50 represents random chance. GPT-4o’s bootstrapped 95% confidence interval ranged from 0.60 to 0.96.
The comparison included differences beyond performance: GPT-3.5 Turbo had been trained two years earlier, while GPT-4o Mini was a smaller model.
To check the findings, Strauss used independent paraphrasing models, Gemma and Qwen, and repeated the analysis while leaving out one book at a time. The study also compared GPT-4o with GPT-4o Mini, which shared an October 2023 training cutoff , to test whether changes in language over time explained the results.
The findings remain preliminary. Strauss identifies the limited book sample, possible contamination of post-cutoff control data, and differences in model size, architecture and training composition as limitations.
The study’s focus on non-public content raises questions about potential access violations in OpenAI’s GPT training data, rather than training inclusion alone. Strauss calls for disclosure requirements covering pre-training data provenance and formal markets for licensing training content.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
