ChatGPT왿� 코퍼스 기반의 대학수학능력시험 모의평가 영어영역 실제 정답률 예측 가능성 모색 (Predicting the difficulty of college scholastic ability test based on ChatGPT and corpus linguistic features)

This study attempts to validate ChatGPT, as well as corpus linguistic features, in predicting the difficulty level of the College Scholastic Ability Test(CSAT). The Observed Proportion Correct (Pc) of the 2021 to 2024 CSAT mock tests was compared with those provided by ChatGPT-3.5, ChatGPT-4.0 and 10 corpus-based linguistic features, including Type, Token, TTR, AWL, Lexical Density, Average Words per Sentence, Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. The research findings are as follows: First, ChatGPT-4.0 bettter predicted item difficulty compared to ChatGPT-3.5.
Second, the Predicted Proportion Correct (Pc) of ChatGPT-3.5, ChatGPT-4.0 and some corpus features showed a high correlation with observed Pc. The observed Pc was largely accounted for by the difficulty predicted by ChatGPT-3.5 and ChatGPT-4.0, where ChatGPT-4.0’s predicted Pc demonstrated higher explanatory power than ChatGPT-3.5. Third, a regression model for predicting the difficulty of CSAT was established with ChatGPT and such corpus-based variables as the Gunning-Fog Index and Lexical Diversity. Overall, the findings endorse the potential use of ChatGPT, along with salient corpus linguistic features, as a suitable tool for predicting item difficulties prior to actual test administration.

This study attempts to validate ChatGPT, as well as corpus linguistic features, in predicting the difficulty level of the College Scholastic Ability Test(CSAT). The Observed Proportion Correct (Pc) of the 2021 to 2024 CSAT mock tests was compared with those provided by ChatGPT-3.5, ChatGPT-4.0 and 10 corpus-based linguistic features, including Type, Token, TTR, AWL, Lexical Density, Average Words per Sentence, Flesch Reading Ease, Flesch-Kincaid Grade Level, and Gunning-Fog Index. The research findings are as follows: First, ChatGPT-4.0 bettter predicted item difficulty compared to ChatGPT-3.5.
Second, the Predicted Proportion Correct (Pc) of ChatGPT-3.5, ChatGPT-4.0 and some corpus features showed a high correlation with observed Pc. The observed Pc was largely accounted for by the difficulty predicted by ChatGPT-3.5 and ChatGPT-4.0, where ChatGPT-4.0’s predicted Pc demonstrated higher explanatory power than ChatGPT-3.5. Third, a regression model for predicting the difficulty of CSAT was established with ChatGPT and such corpus-based variables as the Gunning-Fog Index and Lexical Diversity. Overall, the findings endorse the potential use of ChatGPT, along with salient corpus linguistic features, as a suitable tool for predicting item difficulties prior to actual test administration.