A brief evaluation of ChatGPT-4 quality

Vitaly Sharovatov -

OpenAI has unveiled ChatGPT-4, and its creators acknowledge that it still possesses several notable shortcomings, including social biases, hallucinations, and adversarial prompts. The purpose of this article is to evaluate the factual accuracy of some of GPT-4's outputs.

Theories of motivation

I have extensively researched theories of motivation, and my initial inclination was to see what ChatGPT-4 has to offer on the subject.

I would argue that GPT-4 delivers accurate responses. As evidence, it also presents the following sources:

  1. Deci, E. L., Koestner, R., & Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin, 125(6), 627-668. Link: https://psycnet.apa.org/record/1999-15054-003
This meta-analysis examines the effects of extrinsic rewards on intrinsic motivation, finding that they can undermine intrinsic motivation in certain cases, particularly when people feel controlled by the rewards.

The provided link is incorrect. The accurate link for this study is this: https://psycnet.apa.org/record/1999-01567-001

While this study does investigate the impact of extrinsic rewards on intrinsic motivation, its findings diverge somewhat from what GPT-4 suggests:

Careful consideration of reward effects reported in 128 experiments leads to the conclusion that tangible rewards tend to have a substantially negative effect on intrinsic motivation, with the limiting conditions we have specified. Even when tangible rewards are offered as indicators of good performance, they typically decrease intrinsic motivation for interesting activities.

Although rewards can control people's behavior—indeed, that is presumably why they are so widely advocated — the primary negative effect of rewards is that they tend to forestall self-regulation. In other words, reward contingencies undermine people's taking responsibility for motivating or regulating themselves.

As demonstrated, GPT-4 exhibits a minor misinterpretation of the study's outcomes.

The second reference supplied by GPT-4 is this:

  1. Gneezy, U., Meier, S., & Rey-Biel, P. (2011). When and why incentives (don't) work to modify behavior. Journal of Economic Perspectives, 25(4), 191-210.Link: https://www.aeaweb.org/articles?id=10.1257/jep.25.4.191

    This paper provides an overview of when and why incentives work (or don't work) to modify behavior, discussing factors such as the nature of the task, the type of incentive, and the context in which the incentive is applied

On this occasion, the link accurately directs to the study, and the authors' names, as well as the research date, are also correct. I would consider the description of the research to be largely accurate.

The third reference supplied by GPT-4 is this:

Pink, D. H. (2009). Drive: The Surprising Truth About What Motivates Us. Riverhead Books.

In this book, Daniel Pink presents a comprehensive review of research on motivation, arguing that intrinsic motivators, such as autonomy, mastery, and purpose, are more effective for long-term engagement and success than extrinsic motivators like money and rewards.

Indeed, the book exists, and the summary provided by GPT-4 is quite precise.

The fourth reference supplied by GPT-4 is this:

Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25(1), 54-67.

Link: https://www.sciencedirect.com/science/article/pii/S0361476X99910202

This paper presents classic definitions of intrinsic and extrinsic motivation and their role in various contexts, highlighting the importance of considering both types of motivation when designing incentive systems.

The link is accurate, and the information regarding the authors and the date is also correct. Additionally, the research description is sufficiently valid.

In this conversation, only 1 of the 4 sources had a slight discrepancy.

Following that, I engaged in a rather satisfactory discussion with GPT-4 on various topics related to motivation theories, which met my expectations:

Based on my observations from this brief conversation, GPT-4 appears to offer fairly accurate responses within this specific domain.

Mob programming

An additional subject that interests me is mob programming. I also posed a few questions to GPT-4 regarding this subject.

I might dispute some of the points offered, but that is more a matter of personal opinion.


All four links supplied by GPT-4 appear to be hallucinations.

Mob Programming: A Case Study - Holger Koschek (2018) URL: https://www.researchgate.net/publication/326150223_Mob_Programming_A_Case_Study This case study investigates a team's experience with mob programming and examines the team's collaboration, communication, and knowledge sharing. It reports positive experiences and highlights that mob programming can lead to better code quality and a more enjoyable working environment.

The provided link is false, and no such study has been conducted by the mentioned author.

How Mob Programming and Pair Programming Foster Collaboration, Learning and High Quality Code - Woody Zuill (2017) URL: https://www.infoq.com/presentations/mob-programming-collaboration/ In this presentation, Woody Zuill, one of the pioneers of mob programming, discusses the benefits of this approach from a practitioner's perspective. He presents real-world experiences and anecdotal evidence to support the benefits of mob programming.

The link is invalid, and the author has not conducted this study. Interestingly, Woody Zuill is indeed renowned in the realm of Mob Programming, but for other reasons. He has an engaging experience report: https://www.agilealliance.org/resources/experience-reports/mob-programming-agile2014/

Exploring the Dynamics of Mob Programming: A Longitudinal Case Study - Julia Anny Romberger (2019) URL: https://digitalcommons.unf.edu/etd/904/ This longitudinal case study examines the effects of mob programming on a software development team's collaboration, productivity, and overall satisfaction. The study shows that mob programming can lead to increased collaboration and job satisfaction while maintaining or improving productivity.

The provided link is false and directs to a master's thesis titled 'Investigating the Effects of Rainfall on Traffic Operations on Florida Freeways' by Lucia Andrew.

This study represents another instance of GPT-4 hallucination. .

The Mob Programming Role-Playing Game: Simulating Mob Work for Communication Skill Development - Pilar Rodríguez, Hanna Mäenpää, and Tommi Mikkonen (2018) URL: https://dl.acm.org/doi/10.1145/3241815.3241863 This paper presents a role-playing game designed to simulate mob programming and facilitate the development of communication skills. The results of the study show that the game can be an effective tool for introducing mob programming and improving communication skills.

The provided link is also invalid, resulting in a 404 error. There is no study with the name hallucinated by GPT-4.

I also posed a basic question to GPT-4 about Warhammer 40k chapters:


It appears that GPT-4 did not accurately comprehend the concept of a Warhammer 40k chapter.

After examining only three areas of my interest, I can conclude that while GPT-3 displayed confident ignorance, GPT-4 demonstrates greater confidence and occasional ignorance, which makes detecting hallucinations even more challenging.