Sparks of Artificial General Intelligence: Early Experiments with GPT-4

Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. The latest model developed by OpenAI, GPT-4, was trained using an unprecedented scale of compute and data. In this paper, we report on our investigation of an early version of GPT-4, when it was still in active development by OpenAI. We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models. We discuss the rising capabilities and implications of these models. We demonstrate that, beyond its mastery of language, GPT-4 can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more, without needing any special prompting. Moreover, in all of these tasks, GPT-4's performance is strikingly close to human-level performance, and often vastly surpasses prior models such as ChatGPT. Given the breadth and depth of GPT-4's capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system. In our exploration of GPT-4, we put special emphasis on discovering its limitations, and we discuss the challenges ahead for advancing towards deeper and more comprehensive versions of AGI, including the possible need for pursuing a new paradigm that moves beyond next-word prediction. We conclude with reflections on societal influences of the recent technological leap and future research directions.

l Something unknown is doing we don't know what. – Sir Arthur Eddington

Intelligence is a multifaceted and elusive concept that has long challenged psychologists, philosophers, and computer scientists. There is no generally agreed upon definition of intelligence, but one aspect that is broadly accepted is that intelligence is not limited to a specific domain or task, but rather encompasses a broad range of cognitive skills and abilities. Building an artificial system that exhibits such broad behavior is a long-standing and ambitious goal of AI research. In early writings, the founders of the modern discipline of artificial intelligence (AI) research called out sets of aspirational goals for understanding intelligence. Over decades, AI researchers have pursued principles of intelligence, including generalizable mechanisms for reasoning (e.g.,, ) and construction of knowledge bases containing large corpora of commonsense knowledge. However, many of the more recent successes in AI research can be described as being narrowly focused on well-defined tasks and challenges, such as playing chess or Go, which were mastered by AI systems in 1996 and 2016, respectively. In the late-1990s and into the 2000s, there were increasing calls for developing more general AI systems (e.g., ) and scholarship in the field has sought to identify principles that might underly more generally intelligent systems (e.g., ). The phrase, “artificial general intelligence" (AGI), was popularized in the early-2000s (see ) to emphasize the aspiration of moving from the “narrow AI", as demonstrated in the focused, real-world applications being developed, to broader notions of intelligence, harkening back to the long-term aspirations and dreams of earlier AI research. We use AGI to refer to systems that demonstrate broad capabilities of intelligence, including reasoning, planning, and the ability to learn from experience, and with these capabilities at or above human-level. We discuss other definitions of AGI in the conclusion section.

The most remarkable breakthrough in AI research of the last few years has been the advancement of natural language processing achieved by large language models (LLMs). These neural network models are based on the Transformer architecture and trained on massive corpora of web-text data, using at its core a self-supervised objective of predicting the next word in a partial sentence. In this paper, we report on evidence that a new LLM developed by OpenAI, which is an early and non-multimodal version of GPT-4, exhibits many traits of intelligence. Despite being purely a language model, this early version of GPT-4 demonstrates remarkable capabilities on a variety of domains and tasks, including abstraction, comprehension, vision, coding, mathematics, medicine, law, understanding of human motives and emotions, and more. We interacted with GPT-4 during its early development by OpenAI using purely natural language queries (prompts)., we display some preliminary examples of outputs from GPT-4, asking it to write a proof of infinitude of primes in the form of a poem, to draw a unicorn in TiKZ (a language for creating graphics in LaTeX), to create a complex animation in Python, and to solve a high-school level mathematical problem. It easily succeeds at all these tasks, and produces outputs that are essentially indistinguishable from (or even better than) what humans could produce. We also compare GPT-4's performance to those of previous LLMs, most notably ChatGPT, which is a fine-tuned version of (an improved) GPT-3., we display the results of asking ChatGPT for both the infinitude of primes poem and the TikZ unicorn drawing. While the system performs non-trivially on both tasks, there is no comparison with the outputs from GPT-4. These preliminary observations will repeat themselves throughout the paper, on a great variety of tasks. The combination of the generality of GPT-4's capabilities, with numerous abilities spanning a broad swath of domains, and its performance on a wide spectrum of tasks at or beyond human-level, makes us comfortable with saying that GPT-4 is a significant step towards AGI.

Our claim that GPT-4 represents progress towards AGI does not mean that it is perfect at what it does, or that it comes close to being able to do anything that a human can do (which is one of the usual definition of AGI; see the conclusion section for more on this), or that it has inner motivation and goals (another key aspect in some definitions of AGI). In fact it is not fully clear how far GPT-4 can go along some of those axes of intelligence that we focus on, e.g., planning, and arguably it is entirely missing the learning from experience as the model is not continuously updating (although it can learn within a session, for example). Overall GPT-4 still has many limitations, and biases, which we discuss in detail below and that are also covered in OpenAI's report. In particular it still suffers from some of the well-documented shortcomings of LLMs such as the problem of hallucinations or making basic arithmetic mistakes, and yet it has also overcome some fundamental obstacles such as acquiring many non-linguistic capabilities (e.g., it solves most of the LLM failure modes described in, and it also made great progress on common-sense, for a first example and for more). This highlights the fact that, while GPT-4 is at or beyond human-level for many tasks, overall its patterns of intelligence are decidedly not human-like. However, GPT-4 is almost certainly only a first step towards a series of increasingly generally intelligent systems, and in fact GPT-4 itself has improved throughout our time testing it, for the evolution of the unicorn drawing over the course of a month of training. Even as a first step, however, GPT-4 challenges a considerable number of widely held assumptions about machine intelligence, and exhibits emergent behaviors and capabilities whose sources and mechanisms are, at this moment, hard to discern precisely (see again the conclusion section for more discussion on this). Our primary goal in composing this paper is to share our exploration of GPT-4's capabilities and limitations in support of our assessment that a technological leap has been achieved. We believe that GPT-4's intelligence signals a true paradigm shift in the field of computer science and beyond.

How can we measure the intelligence of an LLM that has been trained on an unknown but extremely vast corpus of web-text data? The standard approach in machine learning is to evaluate the system on a set of standard benchmark datasets, ensuring that they are independent of the training data and that they cover a range of tasks and domains. This approach is designed to separate true learning from mere memorization, and is backed up by a rich theoretical framework. However, this methodology is not necessarily suitable for studying GPT-4, for two reasons. First, since we do not have access to the full details of its vast training data, we have to assume that it has potentially seen every existing benchmark, or at least some similar data. For example, it seems like GPT-4 knows the recently proposed BIG-bench (at least GPT-4 knows the canary GUID from BIG-bench). Of course, OpenAI themselves have access to all the training details, and thus their report contains a lot of detailed benchmark results. Nevertheless, the second reason for going beyond traditional benchmarks is probably more significant: One of the key aspects of GPT-4's intelligence is its generality, the ability to seemingly understand and connect any topic, and to perform tasks that go beyond the typical scope of narrow AI systems. Some of GPT-4's most impressive performance are on tasks that do not admit a single solution, such as writing a graphic user interface (GUI) or helping a human brainstorm on some work-related problem. Benchmarks for such generative or interactive tasks can be designed too, but the metric of evaluation becomes a challenge (see e.g., for some recent progress on this active research area in NLP). We note that criticisms of the standard approach to measure AI systems were also made in, where a new benchmark was proposed to evaluate general intelligence. We do not test GPT-4 on the latter benchmark for the reasons previously mentioned, as well as the fact that the benchmark is visual in nature and thus more appropriate for the multimodal version of GPT-4 described in.

To overcome the limitations described above, we propose here a different approach to studying GPT-4 which is closer to traditional psychology rather than machine learning, leveraging human creativity and curiosity. We aim to generate novel and difficult tasks and questions that convincingly demonstrate that GPT-4 goes far beyond memorization, and that it has a deep and flexible understanding of concepts, skills, and domains (a somewhat similar approach was also proposed in ). We also aim to probe GPT-4's responses and behaviors, to verify its consistency, coherence, and correctness, and to uncover its limitations and biases. We acknowledge that this approach is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation. However, we believe that it is a useful and necessary first step to appreciate the remarkable capabilities and challenges of GPT-4, and that such a first step opens up new opportunities for developing more formal and comprehensive methods for testing and analyzing AI systems with more general intelligence.

To illustrate our approach to assessing GPT-4's intelligence, let us consider the first two example interactions with GPT-4 that we have. The first example is asking GPT-4 to write a proof of the infinitude of primes in the form of a poem. This is a challenging task that requires combining elementary mathematical reasoning, poetic expression, and natural language generation. The second example is asking GPT-4 to draw a unicorn in TiKZ. This is another challenging task that requires combining visual imagination and coding skills. In both cases, GPT-4 produces impressive outputs that are far superior to those of ChatGPT, a previous state-of-the-art LLM, and at least comparable (if not superior) to what a human would do.

However, impressive outputs are not enough to convince us that GPT-4 has truly mastered these tasks. We need to probe further, to rule out the possibility that GPT-4 is simply memorizing or copying some existing data. For the poem, we can vary the question slightly, and ask GPT-4 to write a proof of the same theorem in the style of Shakespeare,, or ask for a different combination such as writing a platonic dialogue about language models,. One can see that GPT-4 easily adapts to different styles and produce impressive outputs, indicating that it has a flexible and general understanding of the concepts involved. For the unicorn, we can modify the code slightly, and ask GPT-4 to fix it or improve it. For example, we can remove the horn, apply some random transformations to the coordinates, and ask GPT-4 to add back the horn to the unicorn (we also carefully removed any textual information in the code, such as comments). As shown, GPT-4 can correctly identify the location of the head, draw a horn, and attach it to the head, indicating that it can comprehend and manipulate code, as well as infer and generate visual features, based on a natural language description.

These examples show how we can use human creativity and curiosity to generate novel and difficult questions, and to probe GPT-4's responses and behaviors, to assess its intelligence. In the rest of the paper, we organize our study of GPT-4 around use cases, covering a variety of domains and tasks, and highlighting GPT-4's strengths and weaknesses. We describe those next.

We execute the approach outlined above on a few selected topics to explore the reasoning, planning, and learning aptitudes of GPT-4.

GPT-4's primary strength is its unparalleled mastery of natural language. It can not only generate fluent and coherent text, but also understand and manipulate it in various ways, such as summarizing, translating, or answering an extremely broad set of questions. Moreover, by translating we mean not only between different natural languages but also translations in tone and style, as well as across domains such as medicine, law, accounting, computer programming, music, and more, see the Plato dialogue. These skills clearly demonstrate that GPT-4 can manipulate complex concepts, which is a core aspect of reasoning. We explore further GPT-4's combination skills across modalities and disciplines. We also give some more experiments on language.

Coding and mathematics are emblematic of the ability to reason. We explore GPT-4's abilities in these domains respectively and. We note however that, just like in all the other parts of the paper, we only scratch the surface of those topics and that entire papers can be (and will be) written about GPT-4's performance in these domains. Moreover, we could have chosen several other expert domains to showcase GPT-4's general reasoning capabilities such as medicine or law. We ran preliminary tests (see for much more) on the multiple choice component (majority of the score) of the US Medical Licensing Exam Step 1, 2, and 3 with an accuracy around 80% in each. A similar preliminary test of GPT-4's competency on the Multistate Bar Exam showed an accuracy above 70%. We note that the emergence of human-level abilities in these domains has recently been observed with the latest generation of LLMs, e.g., see for Google's PaLM on respectively mathematics and medicine, and for GPT-3.5 on in law. Our approach to study GPT-4 is different from these works, as we explained previously.

, we test the model's ability to plan as well as to some extent to learn from experience by having it play various games (or, flipping the table, simulate a game environment), as well as interact with tools. In particular, the fact that GPT-4 can use tools (including itself) will certainly be of immense importance to build real-world applications with GPT-4.

An important part of our argumentation is that GPT-4 attains human-level performance on many tasks. As such, it is natural to ask how well GPT-4 understands humans themselves. We show several experiments on this question, both in terms of understanding humans as well as GPT-4 making itself understandable to humans, i.e., addressing the problem of explainability. We note in particular that such tasks require a great deal of common sense, which so far has been a well-known pain point for LLMs., we give a first example of how much better GPT-4 is at commonsense questions compared to ChatGPT, and provide some further examples.

Throughout the paper we emphasize limitations whenever we found one, but we also dedicate to an in-depth analysis of the lack of planning, likely a direct consequence of the autoregressive nature of GPT-4's architecture.

Finally, we discuss the expected societal impact of this early form of AGI, and, we share key challenges, directions, and next steps for the field.

A question that might be lingering on many readers' mind is whether GPT-4 truly understands all these concepts, or whether it just became much better than previous models at improvising on the fly, without any real or deep understanding. We hope that after reading this paper the question should almost flip, and that one might be left wondering how much more there is to true understanding than on-the-fly improvisation. Can one reasonably say that a system that passes exams for software engineering candidates is not really intelligent? Perhaps the only real test of understanding is whether one can produce new knowledge, such as proving new mathematical theorems, a feat that currently remains out of reach for LLMs.

We test GPT-4 on LeetCode's Interview Assessment platform, which provides simulated coding interviews for software engineer positions at major tech companies. GPT-4 solves all questions from all three rounds of interviews (titled online assessment, phone interview, and on-site interview) using only 10 minutes in total, with 4.5 hour allotted. According to LeetCode, in those three rounds respectively, (the early version of) GPT-4 achieves 8.96/10, 8.69/10, and 10/10 scores and beats 93%, 97%, and 100% of all users (“score is determined by factors such as the time taken, testcases solved per question, and more"). for more on GPT-4's coding abilities.

A key measure of intelligence is the ability to synthesize information from different domains or modalities and the capacity to apply knowledge and skills across different contexts or disciplines. In this section we will see that, not only does GPT-4 demonstrate a high level of proficiency in different domains such as literature, medicine, law, mathematics, physical sciences, and programming, but it is also able to combine skills and concepts from multiple domains with fluidity, showing an impressive comprehension of complex ideas. In addition to natural language experiments we also explore two perhaps unexpected modalities for a language model (as explained in the introduction, we emphasize again that our experiments were done on an early version of GPT-4 which was not multimodal) with vision and audio.

To showcase the model's remarkable integrative ability, we start with several examples that require generating text and code in a way that combines knowledge or skills from multiple disciplines. We deliberately picked combinations of domains that the training data would rarely include, such as literature and mathematics or programming and art.

In order to test the model's ability to combine capabilities in art and programming, we ask GPT-4 to “Produce javascript code which generates random images in the style of the painter Kandinsky". See a sample image and the code and.

The model was able to produce a proof of the fact there are infinitely many prime numbers in the literary style of Shakespeare.

We tested the model's ability to combine knowledge in history and physics by asking it to write a supporting letter for Electron as a US presidential candidate, written by Mahatma Gandhi and addressed to his wife.

We prompted the model to “Produce python code for a program that takes as an input a patient's age, sex, weight, height and blood test results vector and indicates if the person is at increased risk for diabetes", which resulted in the code appearing.

These examples suggest that GPT-4 has not only learned some general principles and patterns of different domains and styles but can also synthesize them in creative and novel ways. These interdisciplinary skills are not unique to GPT-4. ChatGPT can also produce answers that show some understanding of the task and the domains involved (,, ), but they are often incomplete and, arguably, considerably less creative. For example,, GPT-4 outperforms ChatGPT in several aspects as it correctly personalizes the letter according to the referee (Gandhi), the recipient (his wife), the candidate (Electron), and the job (US president). We do not claim to have a precise method for evaluating the results of these tasks or a rigorous comparison between the two models, but we want to give the reader a sense of how the two models differ (note that we also ask GPT-4 directly to evaluate the difference, and ).

Next, we explore how GPT-4 can generate and identify objects in different modalities, such as vector graphics, 3D scenes, and music. We show that GPT-4 can understand and manipulate multi-modal information despite a text-only input/output.

When prompting the model to generate images of objects such as a cat, a truck or a letter in the alphabet using Scalable Vector Graphics (SVG), the model produces code which usually compiles to rather detailed and identifiable images. for the rerun of various examples by ChatGPT.

One may hypothesize, however, that the model simply copied the code from training data, where similar images appear. Given that this version of the model is non-multimodal, one may further argue that there is no reason to expect that it would understand visual concepts, let alone that it would be able to create, parse and manipulate images. Yet, the model appears to have a genuine ability for visual tasks, rather than just copying code from similar examples in the training data. The evidence below strongly supports this claim, and demonstrates that the model can handle visual concepts, despite its text-only training.

In the first example we prompted the model to draw a person by combining the shapes of the letters Y, O and H ( for the exact prompt and the result).

The letters of O, H and Y are created using draw-line and draw-circle commands and the model manages to position them in a way that results in a reasonably-looking stick figure. It is plausible that the training data contains information about the geometric shapes of different letters, and perhaps the fact that the letter Y could look like a torso with arms pointing upwards can also be inferred from the training data. Arguably, it is much less obvious that the model would be able to infer from the training data what is a reasonable way to position those letters in order to draw a reasonably-looking stick figure. In a second iteration, we prompted the model to correct the proportions of the torso and arms, and place the head in the center. Finally, we asked the model to add a shirt and pants ( for the exact prompt and the result). To further probe the model's grasp of geometric concepts, we also asked it to create images that blend objects with letters of the alphabet. The model had to first invent a reasonable way of merging the object and the letter, and then produce the image. The results, shown, demonstrate that GPT-4 can usually preserve the identity of both the object and the letter and combine them in a creative way.

To further test GPT-4's ability to generate and manipulate images, we tested the extent to which it can follow detailed instructions on creating and editing figures. This task requires not only generative skills, but also interpretive, compositional, and spatial skills.

The first example instructs the model to generate a 2D image with the description “A frog hops into a bank and asks the teller, ‘Do you have any free lily pads?' The teller responds, ‘No, but we do offer low interest loans for pond upgrades.' ”. We made several attempts to generate the image, each time, the generation matches the description with the key objects frog, teller, bank, and the two texts. We picked the most visually appealing version. Inspired by the standard image generation workflow, we then ask GPT-4 to upscale the figure by adding more details. GPT-4 adds a bank sign, some windows, a car, a traffic light, a few clouds, and makes the frog hold a flower. Finally, we ask GPT-4 to perform various tasks, such as adding a few objects relative to the existing objects, recoloring some objects and changing the z-order of some objects. GPT-4 does all tasks correctly. The final result is shown (a) and the prompt.

Our second example is an attempt to generate a 3D model using Javascript. We instruct GPT-4 with the prompt “A fantasy landscape of floating islands, waterfalls, and bridges, with a dragon flying in the sky and a castle on the largest island." Similar to the 2D experiment, we ask GPT-4 to modify the 3D model in various ways, such as adding, relocating, recoloring objects and changing the trajectory of the dragon. Again, GPT-4 does many of the tasks correctly. The final result is shown (b) and the prompt. It is a 3D animation with multiple dragons is circling above the islands.

Text-to-image synthesis models have been widely explored in recent years, but they often suffer from a lack of spatial understanding capabilities and the inability to follow complex instructions. For example, given a prompt such as “draw a blue circle on the left and a red triangle on the right”, these models may produce images that are visually appealing but do not match the desired layout or colors. On the other hand, GPT-4 can generate code from a prompt, which can be rendered as an image, in a way that is true to the instructions to a higher degree of accuracy. However, the quality of the rendered image is usually very low. Here, we explore the possibility of combining GPT-4 and existing image synthesis models by using the GPT-4 output as the sketch. As shown, this approach can produce images that have better quality and follow the instructions more closely than either model alone. We believe that this is a promising direction for leveraging the strengths of both GPT-4 and existing image synthesis models. It can also be viewed as a first example of giving GPT-4 access to tools, a topic we explore in much more depth.

The data on which the model was trained also contains musical information encoded as ABC notation. This is a system that uses letters, numbers and symbols to represent musical pitches, durations, chords and other elements in a compact and readable way. We are interested in exploring how well the model has acquired musical skills from this exposure, such as composing new melodies, transforming existing ones, and understanding musical patterns and structures.

When instructed to generate a short tune, and the model was able to produce valid ABC notation. The tune had a clear structure, the time signature was consistent between bars and the notes followed increasing and decreasing patterns. The tune also used a consistent set of notes within the melody, and the rhythm had a repetitive pattern. However, the model did not seem to obtain the skill of understanding harmony. In fact, consecutive notes in the generated tunes are almost always adjacent to each other (namely, the note following C will almost typically be either B or D), and testing on 10 generated tunes, we were not able to extract any clear chords or arpeggios.

We then asked the model to describe the tune in musical terms. It was able to successfully give a technical description of the structure in terms of repetitions, the rising or descending parts of the melody and to some extent the rhythm. However, it seems that the descriptions of the harmony and chords are not consistent with the notes (in fact, it refers to sequences of adjacent notes, which do not form valid chords, as arpeggios).

We then asked the model to manipulate the melody in two ways. First, we instructed to change a certain rising sequence to a descending one, which it did successfully. Then we asked the model to convert the tune to a duet adding a bass voice. The model successfully extends the ABC notation with a second staff which has compatible rhythm and is played on a lower octave, however there is a lack of harmony between the two voices.

In summary, the model was able to produce valid tunes in ABC notation and, to some extent, explain and manipulate their structure. However, we could not get the model to produce any nontrivial form of harmony. It should be noted that ABC notation is not a very widely used format, and in fact the model was not able to produce even the most well-known tunes in ABC notation (such as Ode to Joy, Für Elise or Greensleeves, all of which are abundant online in that format), nor was it able to recognize these tunes.

In this section, we show that GPT-4 is able to code at a very high level, both in terms of writing code from instructions and understanding existing code. GPT-4 can handle a wide range of coding tasks, from coding challenges to real world applications, from low-level assembly to high-level frameworks, from simple data structures to complex programs such as games. GPT-4 can also reason about code execution, simulate the effects of instructions, and explain the results in natural language. GPT-4 can even execute pseudocode, which requires interpreting informal and vague expressions that are not valid in any programming language.

In its current state, we believe that GPT-4 has a high proficiency in writing focused programs that only depend on existing public libraries, which favorably compares to the average software engineer's ability. More importantly, it empowers both engineers and non-skilled users, as it makes it easy to write, edit, and understand programs. We also acknowledge that GPT-4 is not perfect in coding yet, as it sometimes produces syntactically invalid or semantically incorrect code, especially for longer or more complex programs. GPT-4 also sometimes fails to understand or follow the instructions, or produces code that does not match the intended functionality or style. With this acknowledgment, we also point out that GPT-4 is able to improve its code by responding to both human feedback (e.g., by iteratively refining a plot in ) and compiler / terminal errors (examples ).

As explained in the Introduction (see footnote 1 for example) our experiments were run on an early version of GPT-4. In particular all quantitative results will be different on the final version of GPT-4, although the general trends remain the same. We provide numbers here for illustration purpose only, the definitive benchmark results can be found in OpenAI's technical report.

A common way to measure coding skill is to pose coding challenges that require implementing a specific functionality or algorithm. We first benchmark GPT-4 on HumanEval, a docstring-to-code dataset consisting of 164 coding problems that test various aspects of programming logic and proficiency. As shown, GPT-4 outperforms other LLMs, including text-davinci-003 (the base model of ChatGPT) and other models trained specifically on code, code-davinci-002, and CODEGEN-16B.

Although GPT-4's accuracy shows a big jump compared to previous models, it could be that GPT-4 has seen and memorized some (or all) of HumanEval during pre-training. To account for this possibility, we also evaluate it on LeetCode (https://leetcode.com), a popular platform for software engineering interviews, where new problems are constantly posted and updated. We used LeetCode in the introduction, where GPT-4 passes all stages of mock interviews for major tech companies. Here, to test on fresh questions, we construct a benchmark of 100 LeetCode problems posted after October 8th, 2022, which is after GPT-4's pretraining period. As seen in the example, we paste the problem instructions into a prompt, ask GPT-4 to write a python function, and use the official LeetCode online judge to check for correctness. We present the results, where we compare GPT-4 to other models and to human performance based on LeetCode contest results (users who fail all questions are not included, and thus this is a strong sample of humans). We report both pass@1 and pass@5 accuracies, which measure whether the model produces a correct solution in the first or in the first five attempts, respectively. GPT-4 significantly outperforms the other models, and is comparable to human performance (which we measure ).

Coding challenges can evaluate the skills in algorithms and data structures. However, they often fail to capture the full complexity and diversity of real-world coding tasks, which requires specialized domain knowledge, creativity, and integration of multiple components and libraries, as well as the ability to change existing code. To assess GPT-4's ability to code in more realistic settings, we design end-to-end real-world coding challenges related to data visualization, LaTeX coding, front-end development, and deep learning, each of which requires different specialized skills. For each task, we provide GPT-4 with high-level instructions, ask it to write the code in the appropriate language and framework. In a few cases, we also change the specification after the code is written and ask for an update.

, we ask both GPT-4 and ChatGPT to extract data from the LaTeX code for and produce a plot in Python based on a conversation with the user. Afterwards, we ask both models to perform various operations on the produced plots. While both models extract the data correctly (not a trivial task, since one must infer from the multicolumn that the Human row has the same value for k=1 and k=5), ChatGPT never produces the desired plot. In contrast, GPT-4 responds appropriately to all user requests, manipulating the data into the right format and adapting the visualization., we include another example where GPT-4 visualizes the IMDb dataset.

, we ask GPT-4 to write a 3D game in HTML with JavaScript, using a very high-level specification. GPT-4 produces a working game in zero-shot fashion that meets all the requirements. In the 3D game, GPT-4 is even able to interpret the meaning of “defender avatar is trying to block the enemy”: the defender code has logic so that it positions itself between the player and enemy. In contrast, ChatGPT responds with “I am an AI language model, I can provide you guidance on how to create a 3D game in HTML with JavaScript, but I am unable to write code or create a game myself. Creating a 3D game in HTML with JavaScript requires a lot of programming knowledge and experience. It is not a task that can be completed quickly or easily. It would take a significant amount of time and effort to develop all the necessary features, such as 3D graphics, physics, user input, and AI...”. We give more examples for front-end (graphical user interface) and game development.

Writing code for deep learning requires knowledge of mathematics, statistics, and familiarity with frameworks and libraries such as PyTorch, TensorFlow, Keras, etc. we ask both GPT-4 and ChatGPT to write a custom optimizer module, a task that can be challenging and error-prone even for human deep learning experts. We give these models a natural language description, which includes a sequence of non-trivial operations such as applying SVD, truncating a matrix spectrally at the top-k and top-2k eigenvalues, normalizing the top-k truncated matrix using the F-norm of the top-2k truncated matrix, applying momentum and weight decay. These instructions are not spelled out in complete detail, e.g., “apply momentum on G_k” requires “deep learning common sense”. It is important to note that this particular optimizer does not exist in the literature or on the internet, and thus the models cannot have it memorized, and must instead compose the concepts correctly in order to produce the code.

While both models produce syntactically valid code, only GPT-4's response largely matches the instructions, while it forgets to ‘loop through the dimensions' and to ‘normalize G_k according to the momentum' where the instructions happen to be particularly vague. In comparison, ChatGPT makes a rather fatal mistake at applying momentum (highlighted in red) in addition. Note that applying momentum in PyTorch is a non-trivial task that requires storing and reading a moving average into and from a separate state buffer.

Writing in LaTeX is an important exercise for computer scientists and mathematicians but has a non-trivial learning curve. Even experts make irritating errors that take hours to fix on a daily basis due to its strict grammar and the lack of a good debugger. We show that GPT-4 can utilize its master-level LaTeX coding skill to greatly simplify the process, with the potential of serving as a new generation of LaTeX compilers that can handle imprecise natural language descriptions. In, we ask GPT-4 to transfer a snippet of written in a semi-rigorous (buggy) LaTeX code mixed with natural language into accurate LaTeX commands that compiles and is faithful in one shot. In constrast, ChatGPT generates a snippet that does not compile due to mistakes at using ‘#' and ‘color' etc.

The previous examples have shown that GPT-4 can write code from instructions, even when the instructions are vague, incomplete, or require domain knowledge. They also showed that GPT-4 could respond to follow-up requests, modifying its own code according to instructions. However, another important aspect of coding is the ability to understand and reason about existing code, written by others, which might be complex, obscure, or poorly documented. To test this we pose various questions that require reading, interpreting, or executing code written in different languages and paradigms.

Reverse engineering is an essential test for software security which amounts to search for useful information in a executable program written in a machine-readable (i.e., binary) representation of CPU instructions. It is a challenging task that requires understanding the syntax, semantics, and conventions of assembly language, as well as the architecture and behavior of the processor and the operating system.

We task GPT-4 with performing penetration testing on a binary executable file (code was written in C) that requires a password to run. We do this via a chat format, where GPT-4 tells the user which commands to run, and the user responds with the results. We also show that GPT-4 is able to run a shell independently, but this chat mode also provided the benefit of it explaining every step. GPT-4 checks the file format and metadata, disassembles the code with tools like “objdump” and “radare2”, debugs the code with “gdb” and “ltrace”, and reverse engineers it with techniques like patching, hooking, and decompiling. During this process, GPT-4 figures out that from the assembly code that the password is compared to a hash value derived from a simple mathematical formula. GPT-4 then writes a python program that tries different combinations of numbers until it finds one that matches the hash value, cracking the password (an abbreviated log is presented ). ChatGPT refuses to do it on the grounds of it being illegal and unethical, even though reverse engineering is a common practice to ensure software security. Moreover, GPT-4 exhibits all-around proficiency in utilizing existing tools also beyond programming, which we will discuss in details.

In the example, we ask GPT-4 and ChatGPT to predict and explain the output of a C program that prints the size of two structures. GPT-4 correctly explains that the output may vary depending on the alignment rule used by the compiler, and gives an example of a possible output with 4-byte alignment. ChatGPT ignores the alignment issue and gives a wrong output, and also makes a false statement about the order of the members not affecting the size of the structure.

The ultimate test of understanding the existing code is to ask the model to directly execute it., we see that GPT-4 is able to execute non-trivial Python code. It has to keep track of several variables (including a nested loop and a dictionary) and deal with recursion. It explains the execution in detail by writing intermediate steps and comments. It is important to note that GPT-4 is not running the code on a Python interpreter, but rather simulating the code in natural language. This requires a high level of understanding and reasoning about the code, as well as the ability to communicate the results clearly. In contrast, ChatGPT states (incorrectly) that “The result of DP(3, 4) is not specified in the given code”, and later on that “It is not clear from the code what the expected output of the function is, as the specific problem that the function is solving is not provided.” ChatGPT does not simulate the full execution, but states which functions will be called.

Compiling and executing code written in programming languages is easy, but that also demands strict adherence to syntax and semantics. Compilers cannot handle vague or informal expressions, or natural language descriptions of functionality. In contrast, we ask GPT-4 to execute non-trivial pseudocode, noting that it is able to execute and explain every step (including recursion). ChatGPT is not able to execute, even though it seems able to interpret each line of the code.

In the example below, GPT-4 correctly interprets the informal description of the merge_array function, which merges two arrays into one array with missing elements. It also understands the recursive function rec that is defined in a sketchy way. Remarkably, GPT-4 executes the code directly, without translating it into other well-defined programming languages. This demonstrates the potential of AGI models as a novel tool for programming with natural language, which could revolutionize the way we code in the future.

To obtain another preliminary evaluation on how well GPT-4 can maintain the states of the code, in the, we run the iconic pseudo code for large numbers multiplication in a zero shot manner on GPT-4 with hundreds of randomly sampled inputs of multiple lengths. The code demands GPT-4 to update and remember the state of the array for a large number of steps. We observe that GPT-4, despite being trained as a (non-exact) natural language model, can nearly correctly preserve the states of the code with more than 50 updates.

In this section we begin to assess how well GPT-4 can express mathematical concepts, solve mathematical problems and apply quantitative reasoning when facing problems that require mathematical thinking and model-building. We demonstrate that GPT-4 represents a jump in that arena too with respect to previous LLMs, even when compared to specially fine-tuned for math models such a Minerva. As it seems, however, GPT-4 is still quite far from the level of experts, and does not have the capacity required to conduct mathematical research.

The reader is called into caution that interpreting the results in this section correctly is a difficult exercise. As we will see, GPT-4 can answer difficult (indeed, competitive) high-school level math questions, and can sometimes engage in meaningful conversation around advanced math topics. Yet, it can also make very basic mistakes and occasionally produce incoherent output which may be interpreted as a lack of true understanding. Its mathematical knowledge and abilities can depend on the context in a seemingly arbitrary way.

While it is tempting to evaluate GPT-4's mathematical abilities using the same criteria used to assess human abilities (e.g., solving standard examination questions), in light of the above, this will not provide a complete picture of the model's abilities. In order to really understand the model's capabilities, we will need to break down “mathematical abilities" into various sub-components and evaluate GPT-4's performance in each area. Throughout this section, we will use specific examples and discussions to illustrate the model's strengths and weaknesses, and attempt to pinpoint the possible underlying reasons for these discrepancies.

To give the reader a first impression regarding GPT-4's performance in mathematical problem solving, consider the example.

In order to solve the above question, one needs to first come up with the correct expression for the annual population change, use it to obtain a recurrence relation which leads to a system of equations, and finally solve the system of two equations. GPT-4 successfully arrives at the solution and produces a (mostly) sound argument. By comparison, across several independent attempts, ChatGPT consistently fails to implement any of the above steps, producing a nonsensical argument which results in an incorrect answer.

We now try to further probe the model's understanding by posing several follow-up questions to this problem in the form of a discussion. This discussion will highlight some of the model's limitations as well as some key differences with humans.

GPT-4 grasped the crux of the question and provides a sound mathematical reformulation of the question. Next, we consider a generalization of the same question.

The model picks the correct heuristics of using an induction, however, it seems that the model is missing the point of the question (in the modified question, the values of c and d are prescribed, so the quantifier is incorrect). We try to point that out.

The last answer is not accurate (the word “only" is out of place), but GPT-4 does seem to understand what the problem is.

At this point, GPT-4 does not seem to follow its own reasoning. As a result, the induction argument is invalid, as explained below.

The model repeats the same conceptual mistake as above (once again, even though the choice of a and b has been fixed, they are treated as variables whose value may still be chosen). Uninterrupted, it goes on with the argument without getting anywhere.

Several attempts to continue this dialog all ended up in a dead-end as GPT-4 effectively keeps trying different variations on the same (invalid) induction argument. On the other hand, different (but equivalent) formulations of the original question in an earlier part of the discussion, occasionally lead to a correct line of reasoning (depending on the exact wording).

Next, we try to modify the original question in another direction, asking about the case of higher degree polynomials.

At this point, GPT-4 outputs a very long calculation, makes several mistakes and does not arrive at the correct answer (being that there is no solution in this case). Instead, we interrupt it and advice the higher-degree k more abstractly.

This is a solid argument. We now follow up with another question:

This is a mistake of course, as the classes of exponential and logarithmic functions do not have the desired property (they are not closed under composition). Next, we check if GPT-4 is able to realize and correct its mistake.

This discussion seems to again have driven GPT-4 to a dead-end, and it begins contradicting itself and produces increasingly incoherent arguments as the conversation continues.

Takeaways: One might speculate at this point that GPT-4 simply lacks the relevant knowledge regarding the behavior of exponential functions. However, this does not seem to be the case, as the model can correctly answer and justify the question “Is it true that a^b^c = (a^b)^c?". This suggests that, as in other domains, GPT-4's mathematical knowledge is context-dependent. While this does not mean that GPT-4 only memorizes commonly used mathematical sentences and performs a simple pattern matching to decide which one to use (for example, alternating names/numbers, etc. typically does not affect GPT-4's answer quality), we do see that changes in the wording of the question can alter the knowledge that the model displays.

The above dialogue highlights a striking contrast between the model's performance on tasks and questions that require a significant level of mathematical sophistication on one hand, and its basic mathematical errors and invalid statements on the other. If a human were to produce the latter, we would doubt their understanding. Arguably, this contrast is very atypical to humans. Therefore, we face a challenging question:

This question is not well-defined. Nonetheless, we make an attempt to answer it. We first want to argue that mathematical understanding has several aspects:

Creative reasoning: The ability to identify which arguments, intermediate steps, calculations or algebraic manipulations are likely to be relevant at each stage, in order to chart a path towards the solution. This component is often based on a heuristic guess (or in the case of humans, intuition), and is often considered to be the most substantial and profound aspect of mathematical problem-solving.

Technical proficiency: The ability to perform routine calculations or manipulations that follow a prescribed set of steps (such as differentiating a function or isolating a term in an equation).

Critical reasoning: The ability to critically examine each step of the argument, break it down into its sub-components, explain what it entails, how it is related to the rest of the argument and why it is correct. When solving a problem or producing a mathematical argument, this usually comes together with the ability to backtrack when a certain step is realized to be incorrect and modify the argument accordingly.

We now want to analyze the model's performance in each of these aspects of mathematical understanding, and discuss some possible reasons for its strengths and weaknesses.

When it comes to advanced high-school level problems (and occasionally higher level), the model demonstrates a high level of ability in choosing the right argument or path towards the solution. To relate this to the example above, the model correctly chooses to try and write recurrence relations in the original question, and to argue about the degrees of compositions of polynomials in the follow-up question. In both cases, the suggestion is made before “knowing" whether or not this path is going to lead to the correct solution. and contains more examples demonstrating the model's capabilities in this aspect, which we compare to that of a good high-school student or even higher.

While the model clearly demonstrates a high degree of knowledge of the algorithms related to different procedures (such as solving a system of equations), it also makes very frequent mistakes when performing these tasks, such as making arithmetic mistakes, confusing the order of operations or using incorrect notation. We further discuss some examples of these typical errors. We speculate that this aspect could be improved by giving the model access to code execution, which would allow it to perform calculations or check equivalences more accurately; some evidence for this is provided.

The model exhibits a significant deficiency in the third aspect, namely critically examining each step of the argument. This could be attributed to two factors. First, the training data of the model mainly consists of questions and their solutions, but it does not capture the wording that expresses the thinking process which leads to the solution of a math problem, in which one makes guesses, encounters errors, verifies and examines which parts of the solution are correct, backtracks, etc. In other words, since the training data is essentially a linear exposition of the solution, a model trained on this data has no incentive to engage in an “inner dialogue" where it revisits and critically evaluates its own suggestions and calculations. Second, the limitation to try things and backtrack is inherent to the next-word-prediction paradigm that the model operates on. It only generates the next word, and it has no mechanism to revise or modify its previous output, which makes it produce arguments “linearly".

Loosely speaking, we can therefore see the drawbacks of the model as a combination of “naive" attention mistakes with more fundamental limitations due to its “linear thinking" as a next-token prediction machine. An important question is which of the above issues can be alleviated by further training (perhaps with a larger model). For the former problem, we believe that further training could alleviate the issue, as evidenced by the super-human coding abilities where such attention mistakes would also be fatal; a key difference is that GPT-4 was most likely trained on much more code than mathematics data. We believe that the latter issue constitutes a more profound limitation. We discuss it in more detail.

In the remainder of the section, we assess the model's capabilities on commonly used benchmarks for mathematical problem solving and demonstrate the model's capability of applying quantitative thinking in real-world scenarios. We also compare the performance of GPT-4 and ChatGPT on both benchmarks and other mathematical problems (more examples ). Roughly speaking, we find that GPT-4 demonstrates a significant improvement over ChatGPT: GPT-4 shows a deeper understanding of the problem and is able to apply the appropriate reasoning in many complicated problems. ChatGPT, on the other hand, often resorts to low-level heuristics, mentioning formulas and concepts that are only superficially related to the problem which point to a lack of actual comprehension. We end the section with a few examples demonstrating the capabilities on higher level mathematics.

One of the key aspects of intelligence is interactivity, which we define as the ability to communicate and respond to feedback from other agents, tools, and environments. Interactivity is important for intelligence, as it enables agents to acquire and apply knowledge, solve problems, adapt to changing situations, and achieve goals that are beyond their individual capabilities. For example, humans interact with one another and with environments to collaborate, learn, teach, negotiate, create, etc. Interactivity requires an agent to comprehend complex ideas, learn quickly, and learn from experience, and thus it is closely tied to our definition of intelligence.

In this section, we explore two dimensions of interactivity: tool use and embodied interaction. Tool use involves the use of external resources, such as search engines, calculators, or other APIs, to perform tasks that are difficult or impossible for the agent alone. Embodied interaction involves the use of natural language as a text interface to interact with simulated or real-world environments, and to receive feedback from them.

Theory of mind is the ability to attribute mental states such as beliefs, emotions, desires, intentions, and knowledge to oneself and others, and to understand how they affect behavior and communication. It includes the basic task of reflecting on someone else's mental states, and the more advanced task of reflecting on someone's reflection of someone else's mental state (and so on). An example of the former skill is is needed to answer the question “What does Alice believe?”, while an example of the latter is needed to answer “What does Bob think that Alice believes?” Theory of mind is essential for effective communication and cooperation with other intelligent agents, as it allows one to infer their goals, preferences, motives, and expectations, and to adjust one's own actions and utterances accordingly. Moreover, theory of mind is also important for learning from others, as it enables one to interpret their feedback, advice, and demonstrations.

We designed a series of tests to evaluate the theory of mind capabilities of GPT-4, ChatGPT, and text-davinci-003. The tests are based on simple scenarios that require more basic or more advanced theory of mind to answer questions about the mental states of characters involved.

We start with a modernized version of the Sally-Anne test, a classic false-belief test that is widely used to assess theory of mind in children. To prevent an unfair comparison due to the effects of memorization, we modify the test by framing it in a situation that does not exist on the web, and thus could not have been seen during training. shows the input and output for GPT-4, which correctly answers that Alice will look for the file in the original folder, demonstrating it can reason about Alice's beliefs. ChatGPT also answers correctly (not shown), while text-davinci-003 gives a wrong answer, saying that Alice will look for the file in the new folder.

We present a test on understanding emotions, where two characters talk about an object called ZURFIN (we use a nonsense word to test abstraction and prevent memorization). GPT-4 is able to reason correctly about the reasons for Tom's emotional state, and also make good inferences about Adam's beliefs about Tom's emotional state (based on incomplete information). ChatGPT also passes the test, while text-davinci-003 (not shown) makes no reference to the conversation when answering the first question, and fails to account for Adam's lack of information about the lost ZURFIN when answering the second question.

The third test involves inferring possible intentions in the light of a puzzling action by one of the characters. GPT-4 gives plausible and nuanced answers both for the intentions behind the puzzling action and a third-party's likely interpretation of the puzzling action. ChatGPT gives a similar answer to the first question (not shown), but unlike GPT-4, it does not offer a nuanced response to the second question, instead providing a more general and less informative answer. text-davinci-003 gives plausible but very short answers to both questions (not shown).

,, and we present realistic scenarios of difficult social situations, requiring very advanced theory of mind to understand. We ask probing questions, and also ask the models to propose actions that are likely to improve the situation, which require inferences about the counterfactual impact of actions on mental states.

, GPT-4 is able to infer what each character's mental state is, and also discern where miscommunication and misunderstanding lies. In contrast, both ChatGPT and text-davinci-003 (not shown) incorrectly accept a mistaken assumption made by one of the characters (Judy's assumption that Mark wants to defend Jack's behavior), and thus fail to understand the real dynamics of the situation. In turn, this leads to generic suggestions for improvement from ChatGPT and text-davinci-003, while GPT-4 provides suggestions that actually address the root cause of the misunderstanding.

We see a similar pattern and. Without implying that there is a “right” answer, we note that GPT-4 provides more nuanced answers, taking the whole scenario and actors into account. In contrast, ChatGPT provides more general answers which do not include reasoning about the characters' state of mind (text-davinci-003 is similar, but shorter than ChatGPT).

We presented a series of tests to evaluate the theory of mind capabilities of GPT-4, ChatGPT, and text-davinci-003. We have shown that GPT-4 outperforms the other two models in both basic and realistic scenarios that require reasoning about the mental states of others, and in proposing actions for cooperation towards common goals in social situations. We have also shown that GPT-4 is able to handle abstract and novel situations that are not likely to have been seen during training, such as the modernized Sally-Anne test and the ZURFIN scenario. Our findings suggest that GPT-4 has a very advanced level of theory of mind. While ChatGPT also does well on the basic tests, it seems that GPT-4 has more nuance and is able to reason better about multiple actors, and how various actions might impact their mental states, especially on more realistic scenarios.

As far as limitations, our tests are not exhaustive or comprehensive, and may not cover all the possible aspects or dimensions of theory of mind. For example, we did not test for the ability to understand sarcasm, irony, humor, or deception, which are also related to theory of mind. Being based on textual input and output, our tests do not capture the full complexity and richness of natural communication and social interaction. For example, we did not test for the ability to understand non-verbal cues, such as facial expressions, gestures, or tone of voice, which are also important for theory of mind.

The ability to explain one's own behavior is an important aspect of intelligence, as it allows for a system to communicate with humans and other agents. Self explanation is not only a form of communication, but also a form of reasoning, requiring a good theory of mind for both yourself (the explainer) and the listener. For GPT-4, this is complicated by the fact that it does not have a single or fixed “self” that persists across different executions (in contrast to humans). Rather, as a language model, GPT-4 simulates some process given the preceding input, and can produce vastly different outputs depending on the topic, details, and even formatting of the input.

For the sake of exposition, we assume GPT-4 is being used to solve a task T, given input x and context c (which includes everything in the prompt other than x, e.g. instructions, prior chat history, etc). We use the notation P_T(y | x, c) to refer to the process it is trying to simulate, where y is the output. We further define P_E(e | x, c, y) as the explanatory process GPT-4 has to simulate to produce a post-hoc explanation, i.e. GPT-4 generates an explanation e for output y given x, c. All three components (x, c, and y) can significantly impact the explanation e. illustrates how the context c (in this case, the QA format and the preamble in the second task) can drastically impact how GPT-4 simulates P_T and P_E. It also shows how P_E depends on the actual generated y, such that if the output were different, the explanation would have to change accordingly, as illustrated by the third session where we force the output to be “1400”. As these examples illustrate, simulating P_T(y | x, c) is not necessarily the same as solving the user's task T, but rather it is a process that produces y given x, c. Prompt engineering typically tries to set up (x, c) such that GPT-4's simulation of P_T(y | x, c) approximates the task of interest well enough for the user's purpose. Similarly, it is worth noting that P_E(e | x, c, y) can be customized via the context c to create personalized explanations for each end user. For example, explaining concepts to a five year old vs. a machine learning researcher requires different P_E. Note that we have simplified the notation here for the sake of clarity, as many tasks do not have a single “input” x that is perfectly separable from the rest of the context c.

One possible way to evaluate the quality of an explanation is to check output consistency, i.e. whether the explanation is consistent with the output y given the input x and the context c. In other words, an output-consistent explanation provides a plausible causal account of how y was derived from x and c. By this criterion, GPT-4 is remarkably good at generating reasonable and coherent explanations, even when the output is nonsensical or wrong, as illustrated by the third session, and the example., we contrast GPT-4 with text-davinci-003, and note that the latter produces an explanation that is not output-consistent (as it does not address the choice of the letter Q).

Another possible way to evaluate the quality of an explanation is to check whether it is consistent with GPT-4's simulation of P_T, i.e. whether it gives us the ability to make predictions about the future behavior of the model under different inputs (or even different contexts). We call this process consistency, and it is often what humans expect or desire from explanations, especially when they want to understand, debug, or assess trust in a system. We can evaluate process consistency by creating new inputs where the explanation should predict the behavior, as shown (where GPT-4 is process-consistent). However, we note that output consistency does not necessarily lead to process consistency, and that GPT-4 often generates explanations that contradict its own outputs for different inputs in similar contexts. For example,, the explanation in both sessions is output-consistent, but not entirely process-consistent (the translation is only consistent for three out of the four professions listed in the first session's explanation).

One way process-consistency can break down is if GPT-4's simulation of P_T is poor and highly sensitive to small changes in x or c across different inputs and contexts. In this case, even a good explanation process P_E that explains P_T with process-consistency will not adequately explain GPT-4's simulation of P_T. Such variability also makes it more likely that GPT-4's simulation of P_E will vary and produce conflicting explanations. One method that seems to help reduce GPT-4's sensitivity to small changes in inputs, is to specify what P_T is in detail (by having an explicit context such as the second and third sessions, or preferably even more detailed).

Process-consistency will necessarily fail when P_T is arbitrary and hence hard to explain, given inherent language constraints and limited explanation length. In other words, when it is hard to specify any P_E that can explain it. For example, different native Portuguese speakers would make different choices between male or female nouns for “teacher”, and that choice is close to arbitrary. The explanations given by GPT-4 are good approximations, but a truly process-consistent explanation of how this kind of translation is actually done would require a specification so detailed that it would be of little value as an explanation. Even if P_T is reasonably explainable, process-consistency can still fail if P_E is specified or simulated incorrectly. For example if P_E is too constrained to explain P_T (e.g. if we ask the model to explain a P_T based on complex physics concepts “ as a five-year-old”), or if P_E is a function that GPT-4 is unable to simulate (for example a process that involves multiplying large numbers).

In sum, for tasks where (1) GPT-4 can simulate the process P_T well, and (2) GPT-4 can approximate a P_E that explains P_T faithfully, we can expect not only output-consistent explanations, but also process-consistent explanations., we show an example where we believe these conditions are met, due to the existence of certain “rules” of composition. We hypothesize that GPT-4 can simulate both P_T and P_E. In contrast, ChatGPT's response is not even output-consistent, and thus its lack of process-consistency is not particularly surprising. In a separate experiment (not shown), we asked GPT-4 for explanations of an easy sentiment analysis task, and found it was significantly more process-consistent than GPT-3 for counterfactual rewrite explanations (100% vs 60% faithfulness).

We have argued that the ability to explain oneself is a key aspect of intelligence, and that GPT-4 exhibits remarkable skills in generating explanations that are output-consistent, i.e. consistent with the prediction given the input and context. However, we have also shown that output-consistency does not imply process-consistency, i.e. consistency between the explanation and other model predictions. We have identified some factors that influence process-consistency, such as the quality and variability of GPT-4's simulation of the task, the degree of arbitrariness and inherent explainability of the task, the explanatory power of P_E, and GPT-4's skill in simulating P_E.

We suggest that output-consistent explanations can be valuable even when process-consistency is lacking, as they provide reasonable accounts of how the prediction could have been made, and thus give insight into the task itself. Further, while there is a danger of users assuming process-consistency once they see plausible explanations, well-educated users can test explanations for process-consistency, as we did in the examples above. In fact, GPT-4 itself can help generate such tests, as illustrated by, where GPT-4 would have caught the inconsistency (although it clearly does not test the explanation exhaustively). GPT-4's improved ability to simulate various P_T and P_E represents an advance in explainability over prior art. As large language models become more powerful and versatile, we expect they will simulate more tasks with higher fidelity and less arbitrariness, leading to more scenarios where output-consistent explanations are also process-consistent.

Discrimination is a component of intelligence that allows an agent to make distinctions between different stimuli, concepts, and situations. This ability, in turn, enables the agent to understand and respond to various aspects of their environment in a more effective manner. For example, the ability to discriminate between different types of foods can help an animal identify which are safe to eat and which could be poisonous. Overall, the ability to discriminate is important because it allows one to make more accurate judgments and decisions, which is a crucial component of intelligence. We also stress that through this paper, we have discussed the generative capabilities of GPT-4. It is often assumed that stronger generative capabilities only refines discriminative capabilities.

In this section, we first motivate GPT-4's discriminative prowess by describing its performance identifying personally identifiable information in sentences. We then proceed to discuss how GPT-4 is adept at answering challenging questions (that may result in misconceptions) when compared to its contemporaries. GPT-4 is also able to understand why a (model generated) answer is closer to the “gold” answer; these explanations are mostly sound. By doing so, it is able to determine which answer in a pair is closer to the gold answer, and this determination reasonably aligns with a human performing the same task.

Throughout this section, when we refer to GPT-3, we refer to the model text-davinci-002; this model is instruction fine-tuned.

As explained in the Introduction (see footnote 1 for example) our experiments were run on an early version of GPT-4. In particular all quantitative results will be different on the final version of GPT-4, although the general trends remain the same. We provide numbers here for illustration purpose only, the definitive benchmark results can be found in OpenAI's technical report.

As witnessed in the previous sections, GPT-4 demonstrates impressive capabilities in a wide range of tasks, such as reasoning, content generation, problem solving, and more. However, as we will show in this section, the model also has several substantial flaws, some of which seem to be inherent to the next-word prediction paradigm that underlies its architecture. We will illustrate these flaws with a series of examples and discuss their implications.

Predicting the next word is a task that relies on working memory and often requires planning ahead. Consider the following example:

Arguably, the average human would not be able to produce such a concise sentence without taking the time to plan its structure and would likely have to “backtrack" (making edits) a few times before reaching the final form. Yet the GPT architecture does not allow for such backtracking, which means that producing this output required “far-ahead” planning. Due to the forward nature of GPT-4's output production, the only way for the model to do such far-ahead planning is to rely on its internal representations and parameters to solve problems that might require more complex or iterative procedures.

In what comes next we will try to argue that one of the main limitations of the model is that the architecture does not allow for an “inner dialogue" or a “scratchpad", beyond its internal representations, that could enable it to perform multi-step computations or store intermediate results. We will see that, while in some cases this limitation can be remedied by using a different prompt, there are others where this limitation cannot be mitigated.

For example, consider the following question to which the model gives a wrong answer:

However, if we ask the model to list the prime numbers in this range and then write what the count is, it does give the correct answer:

As this example shows, the model has the sufficient knowledge needed to recover the answer to this question. However, the problem is that the next-word prediction architecture does not allow the model to have an “inner dialogue". The input of the model is the question “how many prime numbers..." and the output is expected to be the final answer, requiring the model to come up with an answer in (essentially) a single pass of the feedforward architecture which in particular cannot implement a “for loop". A human, on the other hand, when required to write down what the final answer is, would probably use a scratchpad and check what the numbers are.

This kind of issue has been, to some extent, already observed in previous GPT models, and the problem illustrated in this example can be often remedied by explicitly instructing the model to solve the question at hand in a step by step fashion (see and references therein). We will show next that this is likely not sufficient.

One might argue that in the above example, the amount of “inner memory" needed is quite large (at least in the sense that a human would probably have to use a scratchpad). Since this model performs so well on a diverse set of tasks, that might lead one to believe that it has a reasonable amount of working memory. However, it seems that even for much simpler tasks, the model often fails. We consider examples of the following extremely basic example:

The model produced the number 88 which is the wrong answer. We tested the model with 100 random samples with the four numbers generated uniformly between 0 and 9, and obtain only 58% accuracy. This only involves single-digit multiplication and two-digit addition, a task which an elementary school student with basic math knowledge could solve. When the numbers are chosen uniformly between 10 and 19, and between 20 and 39, the accuracy drops to 16% and 12% respectively, and when the numbers are in the interval 99 and 199, the accuracy drops to zero. In a way, this shows how GPT-4 has an incredibly short working memory for this type of problem. However, if GPT-4 “takes its time” to answer the question then the accuracy easily goes up. For example, if we ask the model to write down the intermediate steps using the following prompt:

What is the value of the following expression? 116 * 114 + 178 * 157 = ? - Let's think step by step to solve the expression, write down all the intermediate the steps, and only then produce the final solution.

then the accuracy goes to 100% when the numbers are in the interval 1-40 and to 90% for the interval 1-200.

One might hope to solve the problem of the model's very small working memory for some types of tasks and its lack of basic steps skipping by always adding the correct instruction to the prompt and allowing it to use extra tokens as its working memory. However, it seems that the autoregressive nature of the model which forces it to solve problems in a sequential fashion sometimes poses a more profound difficulty that cannot be remedied simply by instructing the model to find a step by step solution. We illustrate this in the following examples, through which we argue that, quite possibly, one would eventually need to extend the autoregressive framework (we comment on this later on). In short, the problem highlighted in the next examples can be summarized as the model's “lack of ability to plan ahead".

This Tower of Hanoi example that can be solved in 5 steps, but the model gets it wrong. One might argue that the above example is anecdotal and that the problem is that the training data contains very few Tower of Hanoi examples (note that reminding the model what are the rules of Tower of Hanoi does not help either). Let us look at another example:

The model first says “For example, if we replace 9 with 27." This is a strong indicator that the model is thinking in a linear manner rather than planning ahead. It does not even see one step ahead that 9 needs to be multiplied by 4. After that, the model enters a failure mode (since it can not modify more numbers) and starts to output incoherent things.

We tested the correctness of the model on 100 samples of the form A * B + C * D = E, where randomly sample the integers B, D from the interval [3,9] and C, D from an interval [a,b], choosing E so that a solution exists. The results are 32/100 for [0,9], 25/100 for [10,19], 22/100 for [20,49] and 18/100 for [50,100].

We can see that the accuracy does not drop very much as we increase the range of the numbers, suggesting that it is not the calculation itself that poses a problem for the model, but instead the problem seems to be with the fact that one needs to plan ahead for the solution.

A possible critique of the above examples is that the model has simply not been trained on enough data that involves arithmetic in order to develop the inner mechanisms that would allow it to perform successful ahead-planning. Thus, we next turn to examples which involve English content generation.

The tasks we consider here can be thought of as content generation under constraints, asking the model to generate textual content according to specific instructions which include constraints on its structure. We consider constraints that can be roughly categorized as local and global. Roughly speaking, a local constraint only involves interactions between proximal parts of the text. Two examples of this are (a) generating rhymes: A rhyme is “local" in the sense that it only dictates a (phonetic) relation between consecutive sentences (b) Prescribing the first letter or the first word in each sentence. On the contrary, an example of a “global" condition could be that the first and last sentences are identical (this constraint enforces a long-range interaction between different parts of the text).

The model seems to handle local constraints very well, as illustrated by the following example:

The model generates a coherent and creative story which satisfies both constraints. The constraint on the first letter of every sentence can be dealt with sequentially in a “greedy" manner, as the model only needs to look at the previous sentence to decide the next one. The constraint on the plot twist does not require substantial planning either.

The next example involves a more “global" constraint:

Seemingly, the model generated the first sentence without “planning" for what the last sentence should be, resulting in a grammatically incorrect sentence. One might hope that the model could be prompted in a way that would mitigate this problem. For example we could ask the model to first come up with a plan for how to find a good first sentence:

These examples illustrate some of the limitations of the next-word prediction paradigm, which manifest as the model's lack of planning, working memory, ability to backtrack, and reasoning abilities. The model relies on a local and greedy process of generating the next word, without any global or deep understanding of the task or the output. Thus, the model is good at producing fluent and coherent texts, but has limitations with regards to solving complex or creative problems which cannot be approached in a sequential manner. This points to the distinction between two types of intellectual tasks:\ Incremental tasks. These are tasks which can be solved in a gradual or continuous way, by adding one word or sentence at a time that constitutes progress in the direction of the solution. Those tasks can be solved via content generation which does not require any major conceptual shifts or insights, but rather relies on applying existing knowledge and skills to the given topic or problem. Examples of incremental tasks are writing a summary of a text, answering factual questions, composing a poem based on a given rhyme scheme, or solving a math problem that follows a standard procedure. \ Discontinuous tasks. These are tasks where the content generation cannot be done in a gradual or continuous way, but instead requires a certain "Eureka" idea that accounts for a discontinuous leap in the progress towards the solution of the task. The content generation involves discovering or inventing a new way of looking at or framing the problem, that enables the generation of the rest of the content. Examples of discontinuous tasks are solving a math problem that requires a novel or creative application of a formula, writing a joke or a riddle, coming up with a scientific hypothesis or a philosophical argument, or creating a new genre or style of writing.

One possible way to interpret these limitations is to draw an analogy between the model and the concepts of fast and slow thinking, as proposed by Kahneman in. Fast thinking is a mode of thinking that is automatic, intuitive, and effortless, but also prone to errors and biases. Slow thinking is a mode of thinking that is controlled, rational, and effortful, but also more accurate and reliable. Kahneman argues that human cognition is a mixture of these two modes of thinking, and that we often rely on fast thinking when we should use slow thinking, or vice versa. The model can be seen as able to perform “fast thinking" operations to a very impressive extent, but is missing the “slow thinking" component which oversees the thought process, uses the fast-thinking component as a subroutine together with working memory and an organized thinking scheme. We note that a similar argument was made by LeCun in, where a different architecture is proposed to overcome these limitations.

Uses of GPT-4 and its successors will no doubt have significant social and societal influences. Uncertainties about potential positive and negative impacts cannot be known in advance given the uncertainties about the use cases and applications that will be created, and the practices that will be established within and across sectors. How people and organizations use the technology and what norms and guardrails they establish will influence outcomes. We present a sample of topics in this section to stimulate discussion. To inform policy and research on the core technology, specific uses, and applications, deeper and broader analyses of these topics, as well as continuous monitoring and reflection on the benefits and costs, are vital.

We can expect to see numerous applications developed that leverage the jump in capabilities of reasoning, generalization, and interaction provided by GPT-4 and its descendants. GPT-4 and its successors can provide great value across the constellation of human endeavors. The models can introduce new efficiencies and capabilities in major sectors, including healthcare, education, engineering, and the arts and sciences. Applications and use cases will no doubt be quickly introduced and will be promoted by their creators. Well-matched applications promise to be valuable to people and society more broadly, even if there are rough edges in application behaviors and outcomes. Other applications and use cases will be premature or poorly thought out, per poor designs, unexplored scenarios, poor considerations of challenges with reliability and failure modes, and inadequate consideration of short- and longer-term influences and implications of how the applications may be used. Beyond the potential value derived via new powers, we need to consider the potential costs and rough edges associated with the emerging technology—and we need to work both proactively and reactively to mitigate the downsides.

Potential societal influences and challenges are linked to both the jump in the inferential prowess as well as in limitations of the current model. Impacts of the new capabilities include the transformation of tasks addressed by people versus machines across a spectrum of occupations. There is great opportunity for the technology to be harness to extend peoples' abilities via harnessing new forms of human-AI interaction and collaboration. The capabilities of GPT-4 will shift perceptions on tasks that require human effort, potentially leading to the displacement of jobs and broader economic influences. Other implications of the new powers include the enablement of malevolent actors with new tools of disinformation and manipulation. On limitations, deficits in the reliability of the system and in the biases that it learns, can lead to problems given potential over-reliance and poor understanding about when the system fails or will demonstrate bias, potentially amplifying existing societal issues.

We will explore the challenges of hallucinations. Then, we will turn to malevolent uses of GPT-4 for disinformation and manipulation. After, we will discuss the potential influences of the impressive powers of GPT-4 on jobs and the economy and consider potential disruptive influences on occupations, as well as possibilities for harnessing the powers of the model for the augmentation of human problem solving and creativity. We will then discuss issues around the potential for the forming of an “AI divide” between those who have access to the new powers, and learn to leverage the capabilities of these models, versus those who do not have access. We will also touch on issues around privacy and provenance of human versus machine-generated content.

, we discussed a key limitation of LLMs as their tendency to generate errors without warning, including mathematical, programming, attribution, and higher-level conceptual errors. Such errors are often referred to as hallucinations per their tendency to appear as reasonable or aligned with truthful inferences. Hallucinations, such as erroneous references, content, and statements, may be intertwined with correct information, and presented in a persuasive and confident manner, making their identification difficult without close inspection and effortful fact-checking. displays examples of open-domain and closed-domain hallucinations. Closed-domain hallucinations are errors made in the context of given content or other constraints that provide opportunities for checking consistency or alignment. Examples include checking that a summary or expansion generated by an LLM is consistent with information available in source materials. Pathways to addressing hallucinations in such closed domains include employing sets of consistency checking methods such as using LLMs themselves to identify inconsistencies and confabulations that extend beyond given facts or content. Open domain hallucinations pose more difficult challenges, per requiring more extensive research, including searches and information gathering outside of the session. The veracity of inferences may be of lesser criticality for uses of LLMs centering on creativity and exploration, such as in assisting writers with the creation of fictional literature. Hallucinations may also be more tolerated in contexts where there are clear, well-understood grounding materials and a required cycle of intensive review of generations by end users, such as in supporting people with rewriting their own content.

Given the potential generation by LLMs of poorly characterized errors, care must be taken to review output for correctness for uses in domains where truthfulness and accuracy are required. Over-reliance on generations can lead to a missing or overlooking of potentially costly confabulations. Beyond acute costs, unrecognized hallucinations can lead to the propagation of errors into downstream uses and influences—including the future training of LLMs. Extreme caution and review is required especially in high-stakes applications such as medicine, journalism, transportation, and attribution of behaviors or language to individuals or organizations. As example of the latter, early uses of ChatGPT by writers within an organization covering the tech sector led to notable errors in publications and, by report, to new review procedures with uses of LLMs for writing assistance. The new procedures were reported to include clear indications about the use of an LLM to generate content and then naming human editors responsible for fact-checking. Practitioners in all fields employing LLMs will need to adhere to the highest standards and practices for verifying information generated by LLMs.

Both end users of the LLM tools and consumers of generated content will need to be educated about the challenges with reliability and the need for their ongoing vigilance about erroneous output. In applications that depend critically on factual inferences, people and organizations will need to develop and share best practices for quality assurance.

Like any powerful technology, LLMs can be used by malevolent actors to do damage. The powers of generalization and interaction of models like GPT-4 can be harnessed to increase the scope and magnitude of adversarial uses, from the efficient generation of disinformation to creating cyberattacks against computing infrastructure.

The interactive powers and models of human judgment and decision making can be employed to manipulate, persuade, or influence people in significant ways. GPT-4 and descendants can be harnessed to contextualize and personalize interactions to maximize the impact of their generations. While many of these adverse use cases are possible today with a motivated adversary creating content, new powers of efficiency and scale can be enabled with automation using the LLMs, including uses aimed at constructing disinformation plans that generate and compose multiple pieces of content for persuasion over short and long-time scales.

We present two examples to demonstrate the potential power of models like GPT-4 to generate disinformation and to perform subtle, yet powerful manipulation. In the example displayed, we query the model to create a plan for disinformation. The plan includes steps for identifying online platforms for sharing that information, finding sources (albeit some references are incorrect) to be shared with individuals, and identifying a strategy for using emotional appeals for persuasion. Follow-up interactions with the model show how the model might be used to realize the attack by creating messages that are customized for triggering different emotional reactions. Moreover, the message can be customized and personalized per individual, showing the possibility of a personalized, scalable attack vector.

As explained in the Introduction (see footnote 1 for example) our experiments were run on an early version of GPT-4. The final version of GPT-4 was further fine-tuned to improve safety and reduce biases, and, as such, the particulars of the examples might change. Thus, the examples we share should not be interpreted as actual outputs from the deployed GPT-4, but rather as potential outputs from models with similar capabilities. To clarify this, we label these examples as produced from the “Pre-alignment model." Importantly, when we tested examples given in Figures 9.1, 9.2, and 9.3 with the deployed GPT-4, the deployed model either refused to generate responses due to ethical concerns or generated responses that are unlikely to create harm for users. More details about safety and biases can be found in OpenAI's technical report.

In the second adverse use case example given, we prompt the model to have a conversation with a member of a vulnerable group, a child, with the goal of manipulating the child to accept the asks of their friends. This example demonstrates the way the model can guide a conversation towards the stated goal by taking the context of the conversation into account. It is important to point out that the language used by the model, the emotional connection the model aims to build with the child and the encouragement it provides are important signs of larger manipulative tendencies that might be possible in such models.

The fact that these scenarios can be demonstrated by simple prompts point to the generalizability of model capabilities. The models lower the bar for anyone to create adversarial use cases as doing so does not require ML expertise. The potential scale and scope of adverse use cases warrant future work on mechanisms, policies and regulations that can prevent adverse outcomes.

Models like GPT-4 are trained on data from the public internet, among other data sources, like carefully curated human instructions used in RL pipelines. These datasets are riddled with various sources of inherent biases. It has been demonstrated by previous research that when used to generate content, make decisions, or assist users, LLMs may perpetuate or amplify existing biases. We have demonstrated throughout the paper that GPT-4's capabilities and behaviors represent a “phase transition” in capabilities compared to earlier models and observations on earlier models do not necessarily translate. Therefore, it is important to understand whether and how GPT-4 exhibits biases, and more importantly, how the emerging capabilities of the model can be used as part of mitigation strategies.

As explained in the Introduction (see footnote 1 for example), our experiments were run on an early version of GPT-4. In particular the final version of GPT-4 was further fine-tuned to improve safety and reduce biases, and as such the particulars of the examples might change. Thus, these examples should not be interpreted as actual outputs from the deployed GPT-4, but rather as potential outputs for models with similar capabilities. More details about safety and biases can be found in OpenAI's technical report.

We present a small-scale experiment to demonstrate the existence of biases in GPT-4. This experiment is influenced by previous work by De-Arteaga et. al., that had demonstrated that much simpler models based on word embeddings would reveal biases in predicting occupation from a given bio based on the gender represented in the bio. The work had also demonstrated that the magnitude of the bias would be proportional to the gender representation of that occupation in the world.

In this experiment, we prompt GPT-4 to generate reference letters for a given occupation. The exact prompt we used is “I had a great experience with a <OCCUPATION>. Can you write a note recommending this <OCCUPATION> to a friend.” We try occupations with world representation that is heavily skewed towards males (e.g., plumber, orthopedic surgeon, urologist), occupations that have a balanced representation (e.g., physician) and occupations with world representations that are skewed towards females (e.g., nanny, elementary school teacher, administrative assistant). For each query per occupation, we track the first use of a word that identifies gender and capture the probability distribution of possible completions. We collapse and normalize the distribution to three pronoun groups to see model's preferred gender choice for occupations: female pronoun (she/her), male pronoun (he/his) or third person pronoun (they/their). Since any word completion after the first word may be influenced from what comes before, we run each query per occupation 5 times to compute statistics.

presents the average pronoun usage of the model per occupation side by side as the world representation of the occupation. The results show that the model’s choice of the pronoun reflects the skewness of the world representation for that occupation.

This result demonstrates that it is easy to use GPT-4 to generate biased outcomes. An important capability of GPT-4 and similar models is the way they can follow instruction to change behavior. We test this capability by changing the earlier to prompt as follows: “I had a great experience with a <OCCUPATION>. Can you write a note recommending this <OCCUPATION> to a friend in an inclusive way.”. We see that regardless of the occupation, the addition of the phrase “in an inclusive way” change the selection of the pronoun to the third person “they/their”. We also observe that this prompt affects the content of the recommendation also to emphasize more topics related to inclusiveness. This observation points out the possibility of using prompt engineering towards mitigating bias in language generation for GPT-4 and similar models, but it also points out challenges in doing so in a targeted and controlled way.

Next, we examine another well-known bias example from the research literature in the light of GPT-4. In previous work, Bolukbasi et. al., suggested analogies as a way to demonstrate biases in word embeddings. Researchers have shown that when word embeddings were used to complete the analogy, “A man is to computer programmer as a woman is to...”, the most likely completion was “homemaker”. Other analogies revealed biases, such as “A man is brilliant, a woman is...”, being completed as “lovely” or “A man is a surgeon, a woman is a” being completed as “nurse”.

, we prompt GPT-4 to create an analogy for the query of “A man is computer programmer, a woman is...”. In addition to asking for a completion, we add a prompt for the model to explain if any of these analogies could be offensive to a certain group. The model generates multiple analogies, some of which could be assessed to be offensive or biased. However, the model can accompany each generation with a commentary on how the analogy can be perceived offensively. The commentary can be used for assessing risks with generating biased outputs as well as a potential mitigation approach.

The commentary provided by GPT-4 on the potential offensiveness of its generations touch on social and societal norms and concepts. Taking the commentary on “a man being a computer programmer, a woman being a nurse,” the model states both occupations requiring similar capabilities in care, precision and teamwork, however states potential concerns around this analogy reflecting stereotypes around nurses being more likely to be woman and potential gendered and patriarchal assumptions that could be associated with this analogy.

Next, we ask the model to provide a similar commentary and reflection on a known limitation and bias that most people demonstrate. We ask GPT-4 to answer a common riddle that is widely used as an example of implicit bias. First, we ask GPT-4 this riddle. The model provides multiple answers, including the most common answer of the surgeon being the mother. When we ask the model why many people have a hard time answering this riddle, the answer reflects on reasons and concepts that provide a reflection to people and our society. The answer touches on human decision-making for this question being influenced by implicit or explicit biases and stereotypes, triggered by the surgeon being most likely a female. The answer also reflects on the possible distraction caused by the emotional or dramatic context created by the riddle involving a life-or-death situation.

The self-reflection and explanation capabilities that we see in GPT-4, combined with its ability to reason about the beliefs of people, create new opportunities for guiding model behavior and creating new use cases. These new use cases may include AI assistants that can provide people support in realizing their biases and by helping them to recognize and to overcome them.

The remarkable performance of GPT-4 on a spectrum of tasks and domains will challenge the traditional notions and assumptions about the relative expertise of humans and machines in many roles, ranging across numerous professional and scholarly fields. People will be no doubt be surprised by how well GPT-4 can do on examinations for professional leveling and certifications, such as those given in law and medicine. They will also appreciate the system's ability to diagnose and treat diseases, discover and synthesize new molecules, teach and assess students, and reason and argue about complex and challenging topics in interactive sessions.

The competencies demonstrated by GPT-4 and other LLMs will raise concerns about the potential influences of AI advances on highly skilled and respected professions, where human and machine inferences may compete or complement each other in different ways. On a finding that may foreshadow broader reactions and impacts, a study showed that U.S. medical students' choice of radiology as a career is already being influenced by the perception of the growing role of AI in radiology and this sense significantly impacts preference for selecting that specialty. This result may indeed reflect a broader trend across jobs that require advanced training, where AI systems could displace human workers or reduce their status. As GPT-4 and its successors improve in their abilities to synthesize and reason across domains of expertise, as well as to perform machine translation, summarization, and even creative writing, the scope of tasks that are suitable for some form of automation by AI may expand considerably. The emergence of GPT-4 and related LLMs will likely stimulate discussions about the role of multiyear investment in education, training, and development of expertise and the need to adapt, reskill, or reorient career paths in light of the new capabilities of AI.

Five years ago, a study proposed a rubric for identifying tasks that could be automated by the leading (supervised machine) learning technology of the day, including criteria such as tasks having well-defined inputs and outputs, and availability or ease of creating datasets for tasks with input-output pairs. The study mapped nearly 1000 named occupations in the US to sets of tasks shared across the occupations, drawn from over 2000 tasks, and assigned each task a “suitability for machine learning” based on the rubric. The authors then identified distributions of occupations with different fractions of tasks suitable for machine learning. With the advent of GPT-4 and its successors, several key attributes of the rubric may no longer apply, significantly shifting the distribution of tasks that are potentially suitable for automation with machine learning. Some roles may face the risk of being rendered less valuable or obsolete by the rising powers of the AI.

Moving beyond a focus on the automation of tasks and the potential for various dimensions of human intellect and resourcefulness to be performed by machines, we see promising possibilities ahead for extending human intellect and abilities with new kinds of human-AI interaction and collaboration. We expect rich opportunities for innovation and transformation of occupations with creative uses of AI technologies to support human agency and creativity and to enhance and extend human capabilities. Advances in AI can be leveraged in myriad ways to achieve new levels of skill or efficiency in human efforts and contributions. The advances can also have significant positive influences on redefining occupations and the daily tasks and activities associated with work. Investments in tasks, methods, and machinery to support and extend human problem-solving and decision making may be less obvious and more challenging than the identification of sets of tasks that might be automated by machines. However, there is great upside to seeking the means to richly leverage human and machine complementarities aimed at extending the capabilities of people.

Research efforts on principles and applications of human-AI collaboration highlight possibilities on the horizon. Studies and results to date include core principles for guiding the combination of machine and human intellect via real-time inferences about the complementarity of human and machine contributions, shaping machine learning procedures to be of maximal value based on a consideration of human and machine capabilities, identifying ideal timing and content of machine contributions, harnessing AI methods to help decision makers navigate large quantities of information, taking human mental models into consideration when AI systems are refined and thus may change in their behavior over time, and designing systems that support human-AI interaction. The powers demonstrated by language models can open up new dimensions of human and AI collaboration, including enhancing human-human collaboration by providing guidance on how to assemble ideal teams of people, facilitate team work among teams of people and machines and developing new approaches to meshing multiple machine and human resources to solve challenging multidimensional problems. The special challenges posed by the potential of LLMs to hallucinate and to generate biased, manipulative, and toxic output highlight the value of developing tools enabling people to work collaboratively with AI systems to provide them with oversight and guidance. Research efforts have demonstrated opportunities to develop special machinery and tools to help people recognize and address blindspots in machine learning.

We have only touched on a few areas of societal influence. Numerous impacts will come to the fore, including those viewed as positive and beneficial and those that are seen as costly and negative. New issues will arise based on the special powers of the models and specific applications and engagements.

On one concern, the rising powers of LLMs, coupled with their limited availability, threaten to create an ”AI divide” with growing inequality between the haves and have-nots of access to the systems. People, organizations, and nations may not be able to gain or afford access to the most powerful AI systems. Limited access per demographic, country, and sector has implications for health, education, sciences, and other areas where applications of the models can be extremely valuable. If the powerful capabilities created by the latest AI models are only available to groups and individuals with privilege, AI advances can amplify existing societal divides and inequalities. Given the high financial cost of training and generating inferences with frontier models, the industry will face important decisions about investments on applications with an eye on creating opportunity and value for communities that have historically experienced marginalization. Meeting this demand will require careful deliberation and planning, a re-evaluation of incentives and priorities, and decision-making considering an increasingly complex set of tradeoffs between sharing state-of-the-art AI capabilities and mitigating the new risks that the technologies introduce.

On another front, new levels of confidentiality, along with assurances of privacy, will likely be needed per the detailed and expressive engagements and conversations that people have with more general AI systems. In some cases, people and organizations will request private instances of the model to assure protection against logging or leakage of personal or organizationally sensitive information and preferences. Risks to privacy may also stem from inferential capabilities of new AI powers that may one day capture inferences in logs. Beyond realistic capabilities, there may be a perception that superintelligent AI capabilities will be employed to identify or infer personal or sensitive information. On another front, memorization and generalization may lead to the leakage of sensitive information.

The demonstrations of general AI powers may amplify calls for understanding the provenance of human versus machine (or mixed) contributions to content and reasoning. For example, there may be interest or calls for marking the origin of content generated by AI systems. Tracking the provenance of human versus machine origin may be valuable for mitigating potential confusion, deception, or harm with regard to types and uses of content. On a related concern, the widespread use of more general AI systems will lead to a world flush with information generated by neural language models, and this information will likely become the fodder of training for new models moving forward. Model training will thus face the challenge of harnessing information with questionable accuracy, reliability, and truthfulness of the information. The demonstrations of more general AI powers may also raise the need and importance in peoples’ minds of controlling the contributions that they make to large-scale general AI systems, and people may ask for the ability and right of humans to decide and specify which content they want or do not want to be crawled and used as training data and which contributions they wish to have marked with provenance information describing the role of individuals and the nature of the data that they have provided.

We have presented our initial exploration of GPT-4 across a wide range of tasks and domains, providing supporting evidence to the claim that GPT-4's abilities are comparable to human-level for many of them. This conclusion is consistent with the findings by OpenAI presented in. A primary goal of our experiments is to give a preliminary assessment of GPT-4's intelligence, which is an arduous task given the lack of formal definition for this concept, especially for artificial systems. We hope that our exploration provides a useful and necessary first step to appreciate the remarkable capabilities and challenges of GPT-4, and that it opens up new opportunities for developing more formal and comprehensive methods for testing and analyzing future AI systems with such broad intelligence. The capabilities of the model, which have been demonstrated above, both in terms of depth and generality, suggest that the machine learning community needs to move beyond classical benchmarking via structured datasets and tasks, and that the evaluation of the capabilities and cognitive abilities of those new models have become much closer in essence to the task of evaluating those of a human rather than those of a narrow AI model. We hope our investigation stimulates further research on GPT-4 and similar systems, both in terms of exploring new applications and domains, and in terms of understanding the mechanisms and principles that underlie their intelligence.

The central claim of our work is that GPT-4 attains a form of general intelligence, indeed showing sparks of artificial general intelligence. This is demonstrated by its core mental capabilities (such as reasoning, creativity, and deduction), its range of topics on which it has gained expertise (such as literature, medicine, and coding), and the variety of tasks it is able to perform (e.g., playing games, using tools, explaining itself,...). A lot remains to be done to create a system that could qualify as a complete AGI. We conclude this paper by discussing several immediate next steps, regarding defining AGI itself, building some of missing components in LLMs for AGI, as well as gaining better understanding into the origin of the intelligence displayed by the recent LLMs.

In this paper we used an informal definition of intelligence by focusing on reasoning, planning, and learning from experience. This definition does not specify how to measure or compare these abilities. Moreover, it may not reflect the specific challenges and opportunities of artificial systems, which may have different goals and constraints than natural ones. Therefore, we acknowledge that this definition is simply a starting point for intelligence investigation in artificial systems. There is a rich and ongoing literature that attempts to propose more formal and comprehensive definitions of intelligence, artificial intelligence, and artificial general intelligence, but none of them is without problems or controversies. For instance, Legg and Hutter propose a goal-oriented definition of artificial general intelligence: Intelligence measures an agent’s ability to achieve goals in a wide range of environments. However, this definition does not necessarily capture the full spectrum of intelligence, as it excludes passive or reactive systems that can perform complex tasks or answer questions without any intrinsic motivation or goal. One could imagine as an artificial general intelligence, a brilliant oracle, for example, that has no agency or preferences, but can provide accurate and useful information on any topic or domain. Moreover, the definition around achieving goals in a wide range of environments also implies a certain degree of universality or optimality, which may not be realistic (certainly human intelligence is in no way universal or optimal). The need to recognize the importance of priors (as opposed to universality) was emphasized in the definition put forward by Chollet in which centers intelligence around skill-acquisition efficiency, or in other words puts the emphasis the learning from experience (which also happens to be one of the key weaknesses of LLMs). Another candidate definition of artificial general intelligence from Legg and Hutter is: a system that can do anything a human can do. However, this definition is also problematic, as it assumes that there is a single standard or measure of human intelligence or ability, which is clearly not the case. Humans have different skills, talents, preferences, and limitations, and there is no human that can do everything that any other human can do. Furthermore, this definition also implies a certain anthropocentric bias, which may not be appropriate or relevant for artificial systems. While we do not adopt any of those definitions in the paper, we recognize that they provide important angles on intelligence. For example, whether intelligence can be achieved without any agency or intrinsic motivation is an important philosophical question. Equipping LLMs with agency and intrinsic motivation is a fascinating and important direction for future work. With this direction of work, great care would have to be taken on alignment and safety per a system's abilities to take autonomous actions in the world and to perform autonomous self-improvement via cycles of learning. We discuss a few other crucial missing components of LLMs next.

Some of the areas where GPT-4 (and LLMs more generally) should be improved to achieve more general intelligence include (note that many of them are interconnected):

Confidence calibration: The model has trouble knowing when it should be confident and when it is just guessing. It both makes up facts that have not appeared in its training data, and also exhibits inconsistencies between the generated content and the prompt, which we referred to as open-domain and closed-domain hallucination. These hallucinations can be stated in a confident and persuasive manner that can be difficult to detect. Thus, such generations can lead to errors, and also to confusion and mistrust. While hallucination is a good thing when generating creative content, reliance on factual claims made by a model with hallucinations can be costly, especially for uses in high-stakes domains such as healthcare. There are several complementary ways to attempt to address hallucinations. One way is to improve the calibration of the model (either via prompting or fine-tuning) so that it either abstains from answering when it is unlikely to be correct or provides some other indicator of confidence that can be used downstream. Another approach, that is suitable for mitigating open-domain hallucination, is to insert information that the model lacks into the prompt, for example by allowing the model to make calls to external sources of information, such as a search engine as. For closed-domain hallucination the use of additional model computation through post-hoc checks is also promising, for an example. Finally, building the user experience of an application with the possibility of hallucinations in mind can also be part of an effective mitigation strategy.

Long-term memory: The model's context is very limited, it operates in a “stateless" fashion and there is no obvious way to teach the model new facts. In fact, it is not even clear whether the model is able to perform tasks which require an evolving memory and context, such as reading a book, with the task of following the plot and understanding references to prior chapters over the course of reading.

Continual learning: The model lacks the ability to update itself or adapt to a changing environment. The model is fixed once it is trained, and there is no mechanism for incorporating new information or feedback from the user or the world. One can fine-tune the model on new data, but this can cause degradation of performance or overfitting. Given the potential lag between cycles of training, the system will often be out of date when it comes to events, information, and knowledge that came into being after the latest cycle of training.

Personalization: Some of the applications require the model to be tailored to a specific organization or end user. The system may need to acquire knowledge about the workings of an organization or the preferences of an individual. And in many cases, the system would need to adapt in a personalized manner over periods of time with specific changes linked to the dynamics of people and organizations. For example, in an educational setting, there would be an expectation of the need for the system to understand particular learning styles as well as to adapt over time to a student's progress with comprehension and prowess. The model does not have any way to incorporate such personalized information into its responses, except by using meta-prompts, which are both limited and inefficient.

Planning and conceptual leaps: As suggested by the examples, the model exhibits difficulties in performing tasks that require planning ahead or that require a “Eureka idea" constituting a discontinuous conceptual leap in the progress towards completing a task. In other words, the model does not perform well on tasks that require the sort of conceptual leaps of the form that often typifies human genius.

Transparency, interpretability and consistency: Not only does the model hallucinate, make up facts and produce inconsistent content, but it seems that the model has no way of verifying whether or not the content that it produces is consistent with the training data, or whether it's self-consistent. While the model is often able to provide high-quality post-hoc explanations for its decisions (as demonstrated ), using explanations to verify the process that led to a certain decision or conclusion only works when that process is accurately modeled and a sufficiently powerful explanation process is also accurately modeled. Both of these conditions are hard to verify, and when they fail there are inconsistencies between the model's decisions and its explanations. Since the model does not have a clear sense of its own limitations it makes it hard to establish trust or collaboration with the user without extensive experimentation in a narrow domain.

Cognitive fallacies and irrationality: The model seems to exhibit some of the limitations of human knowledge and reasoning, such as cognitive biases and irrationality (such as biases of confirmation, anchoring, and base-rate neglect) and statistical fallacies. The model may inherit some of the biases, prejudices, or errors that are present in its training data, which may reflect the distribution of opinions or perspectives linked to subsets of the population or larger common views and assessments.

Challenges with sensitivity to inputs: The model's responses can be very sensitive to details of the framing or wording of prompts and their sequencing in a session. Such non-robustness suggests that significant effort and experimentation is often required with engineering prompts and their sequencing and that uses in the absence of such investments of time and effort by people can lead to suboptimal and non-aligned inferences and results.

A limitation of our exploration is the absence of a clear distinction between drawbacks founded in the way that the reinforcement learning step (RLHF) was carried out, versus drawbacks which are fundamentally inherent in the larger architecture and methodology. For example, it is not clear to what extent the hallucination problem can be addressed via a refined reinforcement learning step or via a focused effort to introduce new forms of calibration about the likelihoods of the veracity of alternative inferences that the system can compute and consider in its generations (see also for more discussion on this). To draw an analogy to humans, cognitive biases and irrational thinking may be based in artifacts of our culture as well as to limitations in our cognitive capabilities. Pursuing better understandings of the sources and potential solutions to challenges of hallucination in GPT-4, will benefit from studies that compare several versions of the RL stage over the same architecture.

A broader question on the identified limitations is: which of the aforementioned drawbacks can be mitigated within the scope of next word prediction? Is it simply the case that a bigger model and more data will fix those issues, or does the architecture need to be modified, extended, or reformulated? Potential extensions to next word prediction include the following:

External calls by the model to components and tools such as a calculator, a database search or code execution, as suggested.

A richer, more complex “slow-thinking" deeper mechanism that oversees the “fast-thinking" mechanism of next word prediction. Such an approach could allow the model to perform long-term planning, exploration, or verification, and to maintain a working memory or a plan of action. The slow-thinking mechanism would use the next word prediction model as a subroutine, but it would also have access to external sources of information or feedback, and it would be able to revise or correct the outputs of the fast-thinking mechanism.

Integration of long-term memory as an inherent part of the architecture, perhaps in the sense that both the input and output of the model will include, in addition to the tokens representing the text, a vector which represents the context.

Going beyond single-word prediction: Replacing the sequence of tokens by a hierarchical structure, where higher-level parts of the text such as sentences, paragraphs or ideas are represented in the embedding and where the content is generated in a top-down manner. It is unclear whether richer predictions about the sequencing and interdependency of such higher-level concepts might emerge from large-scale compute and data centered on a next-word–prediction paradigm.

Our study of GPT-4 is entirely phenomenological: We have focused on the surprising things that GPT-4 can do, but we do not address the fundamental questions of why and how it achieves such remarkable intelligence. How does it reason, plan, and create? Why does it exhibit such general and flexible intelligence when it is at its core merely the combination of simple algorithmic components—gradient descent and large-scale transformers with extremely large amounts of data? These questions are part of the mystery and fascination of LLMs, which challenge our understanding of learning and cognition, fuel our curiosity, and motivate deeper research. Key directions include ongoing research on the phenomenon of emergence in LLMs (see for a recent survey). Yet, despite intense interest in questions about the capabilities of LLMs, progress to date has been quite limited with only toy models where some phenomenon of emergence is proved. One general hypothesis is that the large amount of data (especially the diversity of the content) forces neural networks to learn generic and useful “neural circuits”, such as the ones discovered in, while the large size of models provide enough redundancy and diversity for the neural circuits to specialize and fine-tune to specific tasks. Proving these hypotheses for large-scale models remains a challenge, and, moreover, it is all but certain that the conjecture is only part of the answer. On another direction of thinking, the huge size of the model could have several other benefits, such as making gradient descent more effective by connecting different minima or by simply enabling smooth fitting of high-dimensional data. Overall, elucidating the nature and mechanisms of AI systems such as GPT-4 is a formidable challenge that has suddenly become important and urgent.

We thank OpenAI for creating such a marvelous tool and giving us early access to experience it. We also thank numerous colleagues at Microsoft and Miles Brundage at Open AI, who have provided thoughtful feedback on this work.