Andrew Blair-Stanek (Maryland; Google Scholar), Anne-Marie Carstens (Maryland), Daniel S. Goldberg (Maryland), Mark Graber (Maryland), David C. Grey (Maryland) & Maxwell L. Stearns (Maryland; Google Scholar), GPT-4’s Legislation Faculty Grades: Con Legislation C, Crim C-, Legislation & Econ C, Partnership Tax B, Property B-, Tax B:

GPT-4 performs vastly higher than ChatGPT or GPT-3.5 on authorized duties just like the bar examination and statutory reasoning. To check GPT-4’s talents, we ran it on our ultimate exams this semester and graded its output alongside college students’ exams. We discovered that it produced easily written solutions that failed to identify many essential points, very similar to a vivid scholar who had neither attended class usually, nor thought deeply concerning the materials. It uniformly carried out beneath common—in each course. We offer observations that will assist regulation professors detect college students who cheat on exams utilizing GPT-4.

Conclusion
GPT-4 carried out nicely beneath common on all our exams this semester. In no less than one occasion, its failure to correctly analyze a difficulty would have possible been malpractice that would have resulted within the consumer going to jail. This low efficiency comes regardless of GPT-4 having been educated on an enormous internet corpus that possible included each printed U.S. case and statute, in addition to a lot authorized commentary.

We discover that GPT-4 performs decently at a number of alternative questions, throughout areas as various as Constitutional Legislation, Property, and Tax. That is in step with its good efficiency on standardized a number of alternative assessments just like the LSAT, SAT, and GRE. On condition that GPT-4’s enormous coaching corpus possible accommodates many a number of alternative questions, it might be an professional at gaming them. Professors anxious about GPT-4 primarily based dishonest would possibly think about shifting away from a number of alternative, significantly since a one-letter reply is far more durable than prose to examine for GPT-4’s fingerprints.

On written questions like subject spotters, GPT-4 misses many apparent points and lacks depth of study. Even when given a universe of authorities (e.g., circumstances) to attract from, it doesn’t totally make the most of them. But it has occasional flashes of brilliance like recognizing points missed by most college students or analyzing cures for claims it identifies. It usually refers to doctrines by various names not utilized in class, as with “vested rights” relatively than “nonconforming makes use of.” It typically spots completely legitimate points—on subjects not truly coated within the course. GPT-4 produces unusually clean and arranged prose, usually with useful headers, numbering, and summaries. Hopefully these tendencies will assist professors spot solutions written by GPT-4 or comparable fashions.

GPT-4 received its greatest grades this semester within the two tax-law programs. There are a number of attainable explanations. One, each exams had been heavy on a number of alternative, certainly one of GPT-4’s stronger areas. Two, the curving in tax lessons could also be extra beneficiant. Three, OpenAI’s use of a tax-law instance throughout its 22-minute GPT-4 livestream kickoff means that a few of GPT-4’s coaching could have been optimized to deal with tax regulation.

When it comes to future work, a few of us and a few of our colleagues could re-run this identical experiment with the newest GPT mannequin on our exams within the fall semester. We might also experiment with totally different parameters, akin to seeing whether or not it will get higher (or worse) grades when the mannequin makes use of the next “temperature,” making the textual content extra random and inventive.

https://taxprof.typepad.com/taxprof_blog/2023/05/gpt-4s-law-school-grades.html