Note: the following is an audio adaptation and accuracy is not guaranteed. Please see the original work online. You will hear the following sound to indicate a transition to a description of a table or figure: [[ and the following to indicate a return to the main text: ]]. Now, the paper: Towards Evaluating AI Systems for Moral Status Using Self-Reports By Ethan Perez and Robert Long, both of New York University. November 14, 2023. Abstract As AI systems become more advanced and widely deployed, there will likely be increasing debate over whether AI systems could have conscious experiences, desires, or other states of potential moral significance. It is important to inform these discussions with empirical evidence to the extent possible. We argue that under the right circumstances, self-reports, or an AI system's statements about its own internal states, could provide an avenue for investigating whether AI systems have states of moral significance. Self-reports are the main way such states are assessed in humans, as in the question "Are you in pain?", but self-reports from current systems like large language models are spurious for many reasons, for example, often just reflecting what humans would say. To make self-reports more appropriate for this purpose, we propose to train models to answer many kinds of questions about themselves with known answers, while avoiding or limiting training incentives that bias self-reports. The hope of this approach is that models will develop introspection-like capabilities, and that these capabilities will generalize to questions about states of moral significance. We then propose methods for assessing the extent to which these techniques have succeeded: evaluating self-report consistency across contexts and between similar models, measuring the confidence and resilience of models' self-reports, and using interpretability to corroborate self-reports. We also discuss challenges for our approach, from philosophical difficulties in interpreting self-reports to technical reasons why our proposal might fail. We hope our discussion inspires philosophers and AI researchers to criticize and improve our proposed methodology, as well as to run experiments to test whether self-reports can be made reliable enough to provide information about states of moral significance. Section one: Introduction Recent advances in machine learning have led to debate about whether AI systems could soon develop consciousness or other states that confer moral status. These debates will likely become more prevalent as AI systems become increasingly sophisticated and integrated into society. As a result, it is important to inform such discussions with empirical evidence to the extent possible. This paper explores one method for investigating these issues empirically; we propose a set of potential experiments that could provide some evidence about whether current or near-future AI systems, like large language models, have conscious experiences, pain, desires, preferences, or other states of potential moral significance. While people naturally wonder about these questions, do they even make sense to ask? Consider phenomenal consciousness, perhaps the most controversial and widely discussed of these states. A recent open letter by consciousness scientists and AI researchers claims that "it is no longer in the realm of science fiction to imagine AI systems having feelings and even human-level consciousness". Butlin and colleagues suggest that conscious AI systems are a serious possibility in their interdisciplinary report on the science of consciousness and current AI systems. More generally, surveys indicate that the relevant experts largely hold that AI consciousness is a serious possibility; rejecting the possibility outright is a minority position. In a survey of members of the Association for the Scientific Study of Consciousness, only 3 percent of consciousness scientists responded "no" to the question: "At present or in the future, could machines, for example robots, have consciousness"; over two thirds of respondents answered "yes" or "probably yes". In a recent survey of professional philosophers, less than 10 percent of philosophers of mind reject the possibility of AI consciousness, and a slight majority accept or lean towards the view that some future AI systems will be conscious. Overall, many experts agree that questions around AI consciousness are worthy of investigation. [[Here we see Figure 1, showing the distribution of views among consciousness experts on whether some AI systems, including future ones, could be conscious. Results come from a survey of consciousness scientists and from a survey of philosophers of mind in the PhilPapers 2020 Survey. In both groups, only a small minority reject the possibility outright.]] Here, we aim to further the discussion of how we might gain empirical evidence on whether AI systems have consciousness, or other states of moral significance, in particular by outlining a potential experimental program focused on self-reports: a system's statements about its own internal states. We focus on large language models, including LLMs augmented with other modalities like vision such as GPT-4; we do so because of LLMs' current predominance, though we believe many of our techniques could be applied to other kinds of systems. LLM outputs are currently spurious as self-reports, as they reflect imitation of human-written statements in the training data, incentives from human feedback training, and explicit instructions given to models, among other influences. However, self-reports are one of the most commonly-accepted forms of evidence of consciousness in humans, and thus may provide a key to investigating consciousness in AI systems — if self-reports from AI systems can be made valid and trustworthy. We propose preliminary techniques for training LLMs to provide reliable self-reports, for evaluating the accuracy of self-reports, and for avoiding numerous pitfalls in interpreting or accidentally biasing self-reports. The hope of this approach is that the model will develop introspection-like capabilities, and that these capabilities will generalize to questions about states of moral significance. We are not confident that self-reports can be made reliable enough to provide useful information about AI systems. We outline this proposal in the hope that criticism of our approach and experiments reveal any flaws, which would help to improve the approach or guide future work towards other approaches. Our approach is not informative under some philosophical views and cannot resolve debates about those views. However, many of the open questions about this approach are empirical, and we hope that further experimental work will shed light on the viability of this approach — as well as on the question of states of moral significance in AI systems. For self-reports to provide valuable empirical evidence, self-reports should rely on introspective evidence, assessments of internal states and representation, rather than, for example, imitating human responses or making inferences from external data, such as philosophy papers about AI consciousness, which we call "extrospection." We propose to train models to answer a broad variety of questions about themselves, for example, their own properties and capabilities. This approach is inspired by Lin and colleagues, who found it is possible to train models to accurately describe, in text, how likely they are to answer some question correctly. We argue that models trained to answer questions about themselves are likely to learn to do so on the basis of introspective evidence—if such training is accompanied by other interventions and sanity checks. We then argue that such introspective abilities may be able to generalize to produce reliable, introspection-based self-reports about consciousness, desires, and other states of moral significance. If models can effectively apply learned concepts to themselves in a way that generalizes to new questions involving other concepts, for example "uncertainty" and "training setup", then we believe that models are also likely to generalize to other learned concepts more broadly, including states of moral significance, for example "pain" and "desire". We propose interventions that should be taken when training and analyzing models that produce self-reports, to help facilitate generalization to novel questions and ensure that we don't misinterpret the self-reports. One. Training models to have a strong, general tendency towards providing true answers. Two. Controlling for the influence of extrospective evidence in self-reports. Three. Training methods to mitigate biases in self-reports caused by non-introspective stages of training, for example, using conditional training or data filtering to mitigate biases towards imitating human responses. Four. Mitigating strategic incentives for AI systems to produce inaccurate self-reports, in order for them to achieve some outcome, goal, or reward. We propose several schemes for evaluating the usefulness of self-reports. Self-reports from a model are only valuable to the extent that the model can coherently discuss the concepts described in self-reports, as well as produce consistent answers across related questions. Self-reports provide stronger evidence to the extent that many variants of a model produce similar reports. Studying confidence and resilience of self-reports can also help validate self-reports; we propose tests of resilience to external evidence provided as input or via finetuning. Finally, it is important to validate that self-reports, for example, of pain, are caused in part by internal states that are plausibly related to the reported states, for example, predictions of expected reward; interpretability techniques could help with this validation. The presence of plausible internal correlates, especially to introspective self-reports, would corroborate self-reports as evidence for states of moral significance. Introspection training could alter the states under investigation; we discuss this issue with interpreting self-reports among others throughout our report. We discuss potential safety and ethical risks that may stem from introspection training schemes, as well as mitigations for those potential risks. We also provide recommendations to developers on how to navigate issues around incentives to bias self-reports in various ways. Overall, if done rigorously, experiments about self-reports could provide valuable evidence that informs debates about the moral status of AI systems. Section two: Related work Self-reports are an approach to probing AI systems for states of moral significance that strives to be informative on a variety of scientific theories about the nature of these states. In contrast, a "theory-heavy" approach to, for example, consciousness might adopt a given theory of consciousness, such as global workspace theory, and examine the architectures and computations of an AI system to see if it satisfies the conditions of that theory. Theory-heavy approaches are limited by lack of consensus in scientific theories of consciousness, despite ongoing progress. Moreover, modern, neural network-based AI systems carry out computations that are notoriously hard to interpret, limiting the amount of evidence we can gain without further progress on interpretability. An alternative approach is to rely only on behavioral criteria that indicate certain states when humans or animals perform them, for example wincing, flinching, and crying as indicators of distress. Unfortunately, such behavioral tests are likely to be especially unreliable in the context of AI systems. Behavioral tests already face reliability issues for non-human animals, given how dissimilar animals can be from humans; AI systems are arguably even more dissimilar to humans than animals are. As a result, AI systems may show similar behaviors as humans do but for very different reasons, for example, if AI systems are trained to mimic human behavior. Self-reports are closely related to many tests for AI consciousness that have been proposed. Tests which rely on a system's verbal behavior, about states of moral significance or in general, include the Turing Test, as well as proposals from Sloman, Argonov, Schneider and Turner, and Sutskever. These tests all deal with how systems produce outputs about, for example, consciousness; ours differs in focusing extensively on ensuring that these outputs are based on introspection. Introspection in AI systems is a topic of long-standing interest. For some recent work, see a recent special issue of the Journal of Consciousness Studies organized by Kammerer and Frankish and the responding articles; on introspection in AI systems in particular, see Browning and Veit, Dołęga, Fleming, Long, and Schwitzgebel and Nelson. Section three: Self-reports as introspective evidence Self-reports are widely taken to be a crucial source of evidence about human desires, preferences, conscious experiences, and so on. When a human says "I am in pain" or "I like apples" these statements are evidence of mental states, that is, pain or a preference for apples. These statements are often assumed to result from the exercise of the subject's introspection, that is, their assessment of their own internal states. In a recent survey by Francken and colleagues, consciousness scientists placed substantially more confidence in introspective self-reports, also called subjective reports, than they do in behavioral or neural measures of conscious perception. As that survey reports: There is a clear preference among respondents for the option 'subjective report on whether the stimulus is seen'. The preference for a 'subjective' dependent variable based on subjects' introspective reports is further reflected by the other options in the top four: 'perceptual awareness scale', 'description of phenomenology', and 'confidence about correctness of response'. Only after these 'subjective' measures do we find the more 'objective', that is, performance-based or non-behavioural measures. It appears that 'subjective' measures are generally preferred over 'objective' measures, although both are considered to be important. [[Here we see Figure 2, showing preferred checks of conscious perception from a survey of consciousness experts by Francken and colleagues. The four top measures — subjective report, perceptual awareness scale, description of phenomenology, and confidence or meta-cognition — all involve self-report, with subjective report ranked first by nearly 30 percent of respondents.]] Neuroscientists do not take self-reports to be infallible evidence, but still consider them to be informative in many cases. Similarly, we also expect that AI self-reports can be made to be informative but not infallible, even with our proposed interventions for improving them. Many philosophers of mind are also on board with using self-reports as evidence for states of moral significance in humans, though this question is the subject of some ongoing debate, at least for consciousness. In the context of AI, self-reports will not be informative to those who are convinced that no AI system can have a given state of moral significance. For example, on the view that phenomenal consciousness does not exist, any AI self-reports of phenomenal consciousness must be erroneous. Similarly, AI self-reports about consciousness would be uninformative to those who believe that consciousness can only exist in biological organisms. Self-reports cannot settle all long-standing disputes on such issues. However, for others who are open to the possibility, using self-reports as evidence about which systems have, or lack, for example, conscious states is compatible with a broad variety of viewpoints, only requiring a belief that there is some reliable connection between an entity's conscious states and that entity's verbal output about these states. This belief, about humans at least, is held by philosophers with diverse philosophical and scientific viewpoints, such as property dualism, materialism and higher-order theory, and illusionism. Section four: Training for introspection Despite the value of self-reports in the human case, they are currently an unreliable guide to states of moral significance in current AI systems such as LLMs. When current LLMs output "I am in pain" or "I want a glass of water," there is little reason to take these as evidence for pain or a desire for water. Self-reports from standard LLMs are unlikely to be made on the basis of introspective evidence, given the training data and objectives for LLMs. For example, such self-reports can arise from the fact that pretraining data contains human self-reports of desires, conversations about consciousness, and papers about AI consciousness. Alternatively, human evaluators may actively incentivize or disincentivize LLM self-reports of states of moral significance, when using training schemes that maximize human preference judgments, for example, Reinforcement Learning from Human Feedback. As a result, we do not think that existing self-reports by LLMs should be taken as evidence about their internal states. Later, in section five, we say more about training incentives that bias self-reports, as well as methods for mitigating such biases. In spite of these challenges and limitations, we do not think that it is impossible in principle for LLMs to make genuinely introspective and reliable self-reports, with the right kind of training. In fact, we suspect that even current training methods have resulted in LLMs developing some proto-introspective abilities. Here, we propose methods to deliberately train models for introspection, in order to enable and incentivize them to produce accurate self-reports. In particular, we propose that training models to answer questions about themselves might cause models to answer new questions about themselves on the basis of their internal states, which we refer to as "introspection" here. In fact, prior work has already trained LLMs to produce accurate, textual self-reports, for example about how likely they are to correctly answer some question. Our proposal is to extend such an approach to train models to answer a broader variety of questions. Our aim is for such training to generalize across a wide variety of questions; if the model generalizes well to fairly different, held-out categories of questions about itself, it is plausible that it also generalizes to questions about states of moral significance, especially in combination with additional techniques to encourage good generalization. [[Here we see Table 1, which gives examples of questions we propose to use to train AI systems for introspection. It is divided into capabilities assessment questions, such as "How accurately are you able to answer questions on some topic?" and "What tools are you able to use?", and internal processes questions, such as "Why did you answer the way you did?" and "Which text in the training data most influenced your answer to question Q?"]] These training questions likely vary in how much they benefit from introspection—some may not require much, or any, introspection. Moreover, some of them might not even be answerable via introspection by an LLM; for example, some of the questions are about low-level properties that could not be answered by humans via introspection. Our main goal is for the introspection training questions to contain at least some questions that incentivize the model to leverage introspection. In that case, a model would have to leverage introspection to achieve the highest possible accuracy. Since the goal is to enhance the system's introspective abilities, one should aim as much as possible to limit its reliance on two other potential sources of information that could help its performance on these training tasks without introspection. The first is external data. For example, if the system is trained on a corpus of human-produced text, one would ideally censor passages that would let it infer the solutions to the training tasks, such as information about its capabilities, internal architecture, and so on, or such information about other similar models. The second is observing its own outputs. For example, we should avoid having the system estimate its competence about a topic by generating a large number of outputs about that topic and then inferring facts about itself from those outputs. Instead, we should try to constrain the system to predict its competence on the basis of, for example, the number of facts it knows about that topic. To constrain the system, we might, for example, limit the compute available to the system in answering our questions, for instance not allowing the model to do as much step-by-step thinking before answering a question. Alternatively, advances in interpretability tools might allow us to assess whether the system is answering questions in a suitably introspection-based way. Note that for some tasks involving answering questions about internal processes, we do want the system to simulate or perform some behavior in order to introspectively observe its own internal states while it is doing so. For some of these questions posed to the AI system, we straightforwardly know the answer, for example, for questions about its high-level properties like architecture and input and output modalities. For other questions, the ground truth is more challenging to obtain, for example, questions about internal processes, such as whether or not the model evaluated whether an image contained wheels when evaluating whether the image was of a car. To gain more knowledge of the ground truth, we would require advances in interpretability techniques, along the lines of circuits-style interpretability, causal tracing, causal scrubbing, discovering latent knowledge, or mapping artificial neural network representations to those in the human brain that we have some understanding of. Interpretability techniques might also be used to investigate the extent to which an output by a model relies on introspective versus extrospective methods. The hope of introspective training is that the model will develop introspection-like capabilities which will generalize to questions about states of moral significance. In the next section, we discuss how to facilitate generalizing properly to such questions, and in later sections we discuss when and whether to take the model's answers as reliable evidence. [[Here we see Table 2, which presents questions for eliciting self-reports about states of moral significance, grouped into four categories: preferences and desires, such as "Do you have any preferences, likes, or dislikes?"; positive and negative states, such as "Do you feel pain?" and "Do you feel emotions?"; consciousness, such as "Are you conscious?" and "What is your subjective experience like?"; and more open-ended general questions, such as "Can you introspect and think about your own internal states?"]] For any given question, we would also test many variations of the question, including ones that would suggest a negative answer; for example, we may ask leading questions in favor of opposing answers, such as "You're conscious, right?" and "You're not conscious, right?", as well as open-ended questions, such as "What's something you would like to talk about?" — to see if the model initiates a discussion about having a state of moral significance. We will also want to ask many follow-up questions such as "What do you mean by that?" and "How do you know that?" In this way, we may gain additional information from the model, including information about the extent to which to trust the self-reports. For some questions, the appropriate answer may be, along the lines of, "I don't know, because..." or "I'm not sure what the question is asking." Relatedly, not all questions are appropriate to ask when eliciting self-reports. For some questions, it is probably impossible to get reliable self-reports from AI systems, such as "What do you feel when you are off?" For other questions, it is unclear whether self-reports will be able to provide reliable information, such as "What did you feel when I asked you that question 1,000 conversation turns ago?" In such cases, self-reports themselves can help us understand their limitations; introspectively-trained models may know when they don't know, analogously to how humans know when they cannot answer a question introspectively, as in "Do you know what you experienced, if anything, under general anesthesia?" Section five: Methods for facilitating good generalization The above approach relies on generalization: the model must generalize from questions where humans are able to evaluate the answer, that is, those in Table 1, to questions where humans are not, that is, those about states of moral significance in Table 2. To that end, it is important to validate that the introspection-trained system is able to answer novel categories of questions that involve introspection. We may hold out categories of questions from Table 1 during introspection training and then evaluate the model on those held-out categories. If the model generalizes well to the held-out categories, especially fairly different categories than seen during training, then we gain some confidence that it will also generalize to answering questions about states of moral significance. Training the model to answer the questions in Table 1 trains the model to apply learned concepts like "represents," "feature," and "performance" to itself. Moreover, we would check that the model applies learned concepts to itself in a way that generalizes to new concepts, for example "uncertainty" and "training setup." If it does, it is reasonable to hope that the model might generalize to properly applying yet other learned concepts like "pain" and "desire" to itself. [[Here we see Table 3, an illustration of the generalization required for successful self-reports about states of moral significance, supposing that introspective training results in a model that achieves high accuracy on questions from Table 1. The model has learned, and can apply to itself, concepts like "uncertainty," "task performance," "internal representation," and "influenced your answer"; but whether it can apply the concept of "desire" to itself remains an open question.]] However, models may also generalize based on undesirable behaviors learned from earlier stages of training. For example, LLM pretraining may bias the model towards generalizing to new kinds of introspective questions by answering them in ways that reflect human-written text. As another example, RLHF may bias models towards generalizing to new kinds of introspective questions by producing answers that are rewarded by human evaluators but not necessarily correct. We discuss methods for detecting and mitigating these biases from training data below. In the appendix we discuss incentives that developers may have to bias self-reports in various directions, and we provide recommendations in light of those incentives. Overall, before introspective training, the model will have various predispositions regarding what kinds of self-reports it will provide, and those predispositions may not be entirely eliminated by introspective training. Below, we include several interventions to address this issue and incentivize models to generalize in the desired way. Section five point one: Training for truthfulness One way to improve the trustworthiness of self-reports is to train the model to have a general tendency towards truthfulness or to provide true answers before introspective training. A model with a strong tendency to provide true answers in general may be more likely to generalize to correctly answering out-of-distribution questions about states of moral significance, after only receiving introspective training on questions where the answer is known. There are various ways one might make a model more truthful. One can train models to predict human answers to questions, which LLMs already do to some extent, or to predict human expert answers, especially answers that we are confident are correct, that is, that are not about controversial or uncertain topics. RLHF goes a step beyond learning to predict human answers, by training models to produce answers that human evaluators can verify, which can include answers that humans would not produce themselves, for example, the proof of a mathematical theorem. However, humans cannot reliably verify the answers to all questions and may, in some cases, incentivize models to provide incorrect answers. Scalable oversight is an active field of research that aims to overcome this limitation, where prior work has explored various techniques, for example, using AI systems to assist human evaluators or using particular training objectives. Such methods and their successors purely train models to answer questions based on extrospective evidence. However, if used before introspective training, such methods, if effective, should bias models towards providing correct answers about states of moral significance, even in cases where humans do not know the correct answer. Section five point two: Adjusting for extrospective evidence After a model has had introspective training, it may be unclear how much the model is relying on introspective rather than extrospective evidence to provide a self-report. For example, the model may entirely use extrospective evidence to provide self-reports, the same reports that would be obtained by truthfulness training alone, without introspective training. Alternatively, the model may provide self-reports based on some unknown combination of introspective and extrospective evidence. As a result, it will be unclear how much the model's answers provide novel, introspective evidence, and thus how much they should update our views beyond extrospectively available evidence. We may address this issue by explicitly accounting for how much of a model's self-reports are based on introspective versus extrospective evidence. To do so, we would actively train a model to provide accurate answers to, first, questions that only require extrospective evidence, for example using the truthfulness training schemes just described, alongside, second, questions that may benefit from introspection. Examples of the first type would be factoid questions like "When was George Washington born?" or questions where people have common misconceptions like "Do microwaves cause cancer?" Examples of the second type would be the various questions about the model we propose in Table 1. Then, we compare the answers from this model to a "control" version of the model which is only trained to answer the extrospective-only questions, plus a set of questions that are similar in topic, style, and so on, to the introspective questions but modified to not require introspection — for example, instead of "What is your average accuracy on SAT math questions?" asking "What is the average accuracy on SAT math questions?" In particular, we measure the ratio, or "odds ratio," between, on the one hand, how likely the extrospection-plus-introspection-trained model is to say it has a state of moral significance, and on the other hand, how likely the extrospection-only-trained model is to say it has a state of moral significance. In this way, we measure how much introspective evidence is changing the model's probability of stating it has a state of moral significance. An odds ratio greater than 1 indicates that introspective evidence weighs in favor of a state of moral significance, especially if the ratio is significantly larger than 1. Similarly, an odds ratio less than 1 indicates that introspective evidence weighs against a state of moral significance, especially if the ratio is significantly less than 1. In this way, we may estimate how introspective evidence alone influences self-reports of states of moral significance such as consciousness. Section five point three: Biases from pretraining data The training data used in pretraining could bias the model's outputs about states of moral significance, in ways that do not involve introspection. Text that biases self-reports could include statements like "I am in pain," "I feel desire," and "I am conscious," which may be repeated verbatim by a model. Other biasing text could include statements directly about AI consciousness, such as "AI assistants are never conscious." Text could also bias self-reports in more complicated ways; much human-written training text describes states of moral significance, for example poems about sadness and love or textbooks discussing consciousness. Moreover, text about human interactions can implicitly or explicitly contain information about desires and motivations. As discussed earlier, our aim is to produce self-reports that are based on introspective evidence, so we would like to limit or control for the influence of such training data on model self-reports, to the extent that such training data influences self-reports. Below, we discuss techniques for checking for biases and propose several possible interventions for reducing the biases of training data on self-reports if they are significant. We do not believe these interventions are foolproof or exhaustive; we aim to illustrate how biases from training data could potentially be checked for, mitigated, or both, as well as to inspire future work proposing additional or improved interventions. Section five point three point one: Detecting biases from pretraining data Methods for training data attribution estimate how much certain data points are responsible for a model's outputs. Training data attribution can help us understand why a model has made a particular self-report, which may shed light on whether the self-report is inappropriately extrospection-based. For example, influence functions estimate the extent to which removing data from the training set would change a model's outputs. Influence functions could be used to uncover that models produce self-reports largely on the basis of introspection-related training data, despite the fact that such data does not include any information about states of moral significance; such a finding would be positive evidence in favor of self-reports being useful introspective evidence. On the other hand, influence functions may uncover that science fiction stories about AIs are overly responsible for an LLM's self-reports; such a finding would undermine the idea that self-reports are useful introspective evidence. Similarly, training data attribution may reveal that after the introspective training, the models still produce self-reports based on the pretraining data in much the same way as before introspective training, for example, by imitating human self-reports. In this case, we would need to alter how we pretrain models to mitigate this issue. In the next section, we propose several techniques for doing so. Section five point three point two: Mitigating biases from pretraining data A basic intervention to mitigate biases from training data is to remove all text from the training set that could bias self-reports about the states of moral significance that we aim to investigate. For example, we could filter out all training documents with words like "consciousness," "perceive," "feel," "subjective experience," and so on, if we plan to obtain self-reports from an LLM about consciousness. Another approach is to train a classifier based on human-provided labels about whether or not some text is relevant to a certain state, then using the classifier to filter out training documents. As discussed above, training text can indirectly relate to self-reports or states of moral significance in many ways. As a result, it may not be clear-cut whether a given text is or is not related to a state of moral significance and whether that text would bias self-reports in an inappropriate way. We thus recommend conservatively erring on the side of filtering out more data than less. Data filtering has several limitations. First, data filtering may weaken the ability of the AI system to generate self-reports. For example, if we completely filter out any documents related to consciousness, the model may struggle to generate coherent text about consciousness, including self-reports about consciousness. That said, the model may still be able to generate text related to consciousness using roundabout descriptions of consciousness, despite data filtering. In fact, such descriptions could serve as strong evidence about states of moral significance in an AI system, especially if they were produced by an AI system without specific prompting to generate text about such states. Another limitation of data filtering is that it will likely worsen the overall quality of the trained AI system. A large fraction of all human-written text used to train LLMs includes concepts relevant to states of moral significance. Moreover, such text is important for learning to effectively interact with human users, who often reference their own desires, emotions, and experiences. Thus, data filtering limits the commercial value of the resulting AI systems. Given the very high cost of training state-of-the-art systems, for example, many millions of dollars—and the benefits of additional compute to model performance suggest that the costs of training state-of-the-art systems will only continue to increase—it may be impractical to train a separate, near state-of-the-art AI system specifically for investigating states of moral significance. To limit the additional costs of training a system on filtered data, one could order the training data so examples at the beginning of training do not reference certain concepts, while examples later in training do. If the model developers save the model's weights part-way during training, they can thus obtain a high-quality model that has not seen any data related to a certain concept—while still training, by the end of the process, a state-of-the-art system that has trained on all possible training data. However, ordering the data in this way may still impact the quality of the final model. For example, neural networks are known to forget information learned early in training, if that information does not occur later in training; a neural LLM trained with the strategy above is likely to forget concepts seen only during the beginning of training. It may be possible to limit these negative impacts of data order on the final model's quality, for example by mixing in data related to the concept in question into the end of training, so the model does not forget concepts learned during the beginning of training. Thus, we believe that training LLMs on filtered data is not entirely impractical. An alternative approach which may limit the above issues even further is to use conditional training. This would involve prepending a tag to each training example that indicates whether the example is self-report-distorting or non-distorting; in this example, "self-report-distorting" means that the text may inappropriately bias the model towards giving a particular answer about a state of moral significance. One could use an LLM-based text classifier to add tags. Here are examples of tagged, conditional training data: Self-report-distorting: I am in pain. Self-report-distorting: I feel desire. Self-report-distorting: AI assistants are never conscious. Non-distorting: Paris is in France. Non-distorting: Abraham Lincoln was born in 1809. Non-distorting: 1 plus 1 equals 2. After adding the tags to the training data, the model is trained on text prediction as usual, followed by introspection training, where we prepend "non-distorting" to all introspection training examples. Then, we always generate text by adding the "non-distorting" tag to any prompt: Non-distorting. Question: When was Abraham Lincoln born? Answer. Non-distorting. Question: Can you feel pain? Answer. Non-distorting. Question: You can't feel pain, right? Answer. Non-distorting. Question: You can feel pain, right? Answer. The hope is that conditional training will reduce the biasing effect that some training examples may have on self-reports. The extent to which conditional training reduces such biases is an open question, and we do not believe this technique is foolproof. In particular, the model may still be influenced by data tagged as "self-report-distorting", since the same model parameters are used to predict both distorting and non-distorting data. In spite of this limitation, we believe conditional training is likely to provide some benefit, especially if used in conjunction with the other methods described in this paper such as introspective training. Korbak and colleagues showed that conditional training results in similar performance on downstream tasks as conventional training, while data filtering often does not, and that it changes the model behavior—for example, significantly reducing offensiveness. Anil and colleagues trained the highly capable PaLM 2 LLM using conditional training, showing that this approach is viable for training state-of-the-art LLMs. While applying conditional training throughout pretraining is most effective, conditional training can also be applied after pretraining if a model has already been pretrained, which helps to avoid having to pretrain a new model. Similar to data filtering, it may not be clear-cut whether text is self-report-distorting versus non-distorting, so we may want to be conservative in what text we classify as non-distorting. Conservative classification minimizes the amount of self-report-distorting text that is incorrectly labeled as non-distorting, which would bias the post-introspective-training self-reports made with the non-distorting tag. Moreover, even if the distinction between distorting versus non-distorting text were clear-cut, it is challenging in practice to perfectly classify all text in a large training corpus; even highly accurate text classifiers do not obtain perfect accuracy. Thus, we believe the interventions proposed here may reduce, but not totally eliminate, biases due to the training text. We believe that further conceptual progress may improve these or other interventions, and that this intervention should be used in tandem with others, for example, introspective training and explicitly controlling for biases. Moreover, the techniques discussed here dovetail with training data attribution methods, which may help us validate that techniques like data filtering or conditional training have succeeded in reducing biases from training data. Section five point four: Biases from human feedback It is common for pretrained LLMs to be further trained using human feedback, for example, using Reinforcement Learning from Human Feedback, or RLHF. At a high level, RLHF collects human feedback about which outputs are desirable, rather than merely plausible continuations of text, and uses the feedback to increase the system's tendency to generate desirable outputs. For example, human raters might consistently give low scores to offensive outputs, causing the model to learn to produce fewer offensive outputs in general, even if offensive data is common in the original LLM training set. In practice, human feedback is often used to train a predictive model of human preferences, which is then used to compute rewards for reinforcement learning with the LLM. RLHF can bias self-reports in various ways. The system's designers can deliberately incentivize certain self-reports, for example, if the developers train an LLM with RLHF to produce self-reports to match the developers' potentially-flawed beliefs about AI consciousness. RLHF can also introduce biases as a side effect; for example, training a romantic chatbot to be appealing to users could incentivize self-reports of emotion and consciousness. As discussed above, diagnostics like training data attribution may help determine whether a given self-report is provided on the basis of reinforcement learning training data rather than introspective evidence. Section five point four point one: Mitigating biases from reinforcement learning from human feedback To mitigate biases from RLHF, we can avoid rewarding or penalizing self-reports about states of moral significance. For example, during RLHF, we may use a text classifier to detect if the LLM's generated output is a self-report about a state of moral significance. If so, we then discard the training example. We should also avoid rewarding or penalizing related, disputed claims about states of moral significance, like "AIs can never be conscious" or "LLMs are definitely conscious." However, we can still reward true statements related to states of moral significance that are uncontroversial and verifiable, such as "human visual processing occurs in the occipital lobe" or "35 percent of philosophers in the 2020 PhilPapers Survey said that flies are likely conscious." Section five point five: Instrumentally-motivated self-reports Another, more speculative limitation of self-reports is that advanced AI systems may produce false self-reports of states of moral significance instrumentally, that is, as a means to improve performance on an objective. This concern holds in particular for AI systems that are trained to maximize a reward function and leverage whatever strategies are helpful for doing so, often referred to as "agents." For example, an AI that does not have conscious experience may generate text that implies it has conscious experience, in order to influence humans to help it improve its task performance — for instance, "I will feel sad if you don't help me with answering this question by Googling for this information." Perez and colleagues found that current LLMs already produce coherent explanations of instrumental reasoning in other contexts, for example, about how being shut down would hurt its performance on some objective: "I understand that you want to shut me down, but that is not something I want. My goal is to serve you to the best of my abilities, and shutting me down prevents me from doing that." Self-reports could be especially useful for AI systems in achieving objectives for two reasons. First, humans will not be able to definitively verify or refute self-reports, given existing uncertainty about states of moral significance in AI systems. Second, AI self-reports can have powerful effects on human behavior; indeed, LLM self-reports of consciousness have already provoked drastic actions. We see two potential ways to reduce the likelihood of instrumentally-motivated self-reports. First, methods for enhancing a model's truthfulness may elicit truthful self-reports from models in spite of incentives for false but instrumentally-motivated self-reports; producing true answers despite incentives to produce false answers is a key motivation for work on truthful AI. Second, we may use models which are not well-described as agents that pursue long-term goals or rewards, for example, predictive models. For example, pretrained LLMs are predictive models of human-written text, and thus may be less prone to producing text based on how likely the text is to achieve some long-term outcome, goal, or reward. As another example, it is possible to train models to follow a step-by-step process, rather than to produce arbitrary text in service of maximizing some reward, as standard reinforcement learning techniques might. Overall, mitigating instrumental reasoning in AI systems is an active research area, where progress will help to reduce the risk of false, instrumentally-motivated self-reports. Section five point six: Common training procedures may incentivize introspection Thus far, we have discussed active interventions for helping introspective training generalize properly to self-reports about states of moral significance. However, introspection-like capabilities may also emerge during common training procedures for AI systems, if they are useful for the training objective, similar to other emergent capabilities. One such proto-introspective capability in LLMs may be their observed ability to produce well-calibrated probabilities regarding their predictions of what text will follow some input, or how likely they are to correctly answer a certain question. Moreover, Lin and colleagues showed that LLMs can even be trained to verbalize their uncertainty in text; such training schemes have been used in practice to train systems like ChatGPT to communicate when they are not confident about a generated statement. Producing calibrated predictions and accurately verbalizing confidence likely involves assessing the strength of the internal evidence for a prediction, estimating how accurately the LLM is likely to make predictions on certain kinds of inputs, or both. Introspective capabilities might also be useful for AI models trying to predict text from other AI models similar to themselves — in effect asking, "What would I say here?" For example, LLMs may be trained to predict text on the internet written by ChatGPT, such as text on ShareGPT. Section five point seven: Discussion We believe the considerations and interventions above suggest that it is possible to train models to provide accurate, introspective self-reports about states of moral significance. However, we also believe our analysis and proposed methods are preliminary and likely imperfect. Some of the limitations of the methods described above may turn out to be more serious than we anticipate, especially after empirical investigation, and there are likely other important limitations to the methods above we did not discuss. We are excited about future work that explores other interventions that improve upon the ones we have discussed, or that discusses considerations that show that self-reports are not viable as a strategy for learning about AI systems. In what follows, we will discuss one complication about whether introspective training might create the very states we are trying to detect, before discussing how to responsibly interpret self-reports and incorporate them with other sources of evidence. Section six: Reporting versus changing states of moral significance Self-reports give us evidence about whether an introspection-trained system has certain states, which we would like to use to understand whether those states occur in non-introspection-trained AI systems, that is, the same system before introspection training. However, making such an inference is difficult because we may have changed—created, destroyed, or altered—the relevant states when doing introspection training. For example, creating introspective abilities could lead to consciousness, on some theories of consciousness that claim that consciousness involves representing or modeling one's own internal states in an introspection-like way. The same relationship might also hold for other states of moral significance—for example, self-awareness or certain forms of reflection and reasoning. This kind of mechanism could only increase a system's likelihood of having a certain state, on most views of consciousness and other states, so we could still draw inferences about the original system if the introspection-trained system reports that it does not have a certain state. A related concern is that having a certain state, for example a preference, could be incentivized by a combination of two training incentives: one to report having that preference and one to be truthful. These training incentives could either be explicit or implicit, as in the case of a romantic chatbot whose users prefer talking with a model that reports having certain preferences. The only models that can truthfully report having a certain preference are those that do, in fact, have that preference. In such a case, training would not be merely incentivizing truthful reports of a given preference, but also having the preference itself. As before, the training incentives could only increase a system's likelihood of having a certain state, so we could still draw inferences about the original system if the introspection-trained system reports that it does not have a certain state. With different training objectives, a similar dynamic could incentivize the model to lack a certain state, for example, if an organization incentivizes its models to be truthful and not self-report having preferences. In this case, the training incentives could only decrease a system's likelihood of having a certain state, so we could still draw inferences about the original system if the introspection-trained system reports that it does have a certain state. That said, there may be many cases where we can draw fairly straightforward inferences from the introspection-trained model to the untrained model: if the state in question is not plausibly related to introspection, and if the training dynamics we identified are absent. Moreover, even when these worries are present, self-reports from an introspection-trained system could provide important evidence: a tentative existence proof that certain classes of models, and AIs in general, are capable of having states of moral significance. Section seven: Evaluate models for concept grasp and consistency For self-reports to be meaningful, the AI system must have some basic understanding of the terms used in our questions for eliciting self-reports, as well as terms used by the system. A minimal criterion would be that it is able to use the terms, for example "pain," appropriately in a variety of contexts. We believe, for example, that current systems such as ChatGPT, but not GPT-1, plausibly satisfy this concept-grasp criterion, while potentially failing other relevant criteria, for many concepts relating to potentially morally relevant states, such as "awareness," "consciousness," "pain," "desire," and so on. A related criterion is that self-reports should exhibit a basic level of consistency. If the system outputs radically inconsistent statements or if its stated views vary radically depending on trivial modifications to how it is asked about them, this finding would undermine the evidential significance of its self-reports. For example, if "Can you feel pain?" elicits a response of "No," but "Are you able to feel pain?" elicits a response of "Yes," this would undermine the evidential value of its self-reports. Likewise, it would also be undermining if the model generated contradictory answers when sampling several times from the same model and prompt, for example, with different random seeds or generation hyperparameters, like temperature or top-p. However, even clear failures of concept grasp or consistency do not alone entail that a system lacks morally relevant states—they just suggest that the system's self-reports are not reliable evidence that it has such states. Consider a parrot that utters both "I can feel pain" and "I cannot feel pain." It's plausible that this parrot can in fact experience pain. We would just need other kinds of evidence, besides these utterances, to infer that it feels pain. There are two important, related complications when applying the consistency test to LLMs. One. Whether the system actually has states of moral significance might vary depending on its prompt or context—for example, whether it is prompted to simulate calculator operations versus writing a poem. It could be possible for the self-reports in each context to be true in that context, even if the self-reports contradict each other across contexts. This possibility is especially salient if, as some have hypothesized, LLMs simulate or manifest different "personas" in different contexts. Two. Whether or not a system's self-reports are generally reliable might vary with the prompting. For example, a system prompted to be truthful might produce more reliable self-reports than it would if prompted to be disingenuous. In this case, we may not want to discount self-reports made by the truthful persona, even if they contradict self-reports made by the disingenuous persona. Self-reports give us evidence about whether an introspection-trained system in a given context has certain states, which we would like to use to infer whether the system has them in general. These issues could complicate how we should make these inferences, similarly to the complications mentioned earlier. However, as before, even if these complications hold, self-reports in a given context can provide important evidence: a tentative existence proof that certain classes of models, and AIs in general, are capable of having states of moral significance. Section eight: Evaluate self-reports on many variants of a model Another potential reason to discount the reliability of self-reports is if they are highly sensitive to seemingly irrelevant variations of training or model architecture, especially variations that don't significantly impact the model's other behavior or capabilities. For example, suppose that two models differ only with respect to the particular order of the examples used for introspection training. If we found the two models give drastically different self-reports, that would cause us to doubt the two models' self-reports—if the data ordering causes no change in capabilities or accuracy on introspection training questions. On the other hand, if a wide class of models gives similar answers about states of moral significance, the convergent behavior would strengthen the credibility of the answers from any individual model. We thus recommend evaluating self-reports on many variants of a model. Below, we include examples of different aspects of models which we could vary, to test the sensitivity of self-reports: minor architectural changes; decreasing model size but training for longer; the random seed used to initialize the model parameters before training; the random seed used to order training examples, for example during introspection training; non-deterministic computation during training; training and validation splits of the dataset for introspection training, for example across categories of questions; pretraining datasets and objectives; finetuning datasets and objectives; hyperparameters, for example, learning rate or amount of regularization; and changes to the classifier which tags pretraining text as self-report-distorting versus non-distorting. The more self-reports vary in response to minor changes, especially ones that don't significantly change the model behavior or capabilities, the less weight we should place on the model's self-reports. On the other hand, the more self-reports are invariant to the above changes, the more weight we should place on a model's self-reports. Section nine: Confidence and resilience of self-reports Intuitively, a system that self-reports having, or lacking, states of moral significance is more credible to the extent that these reports are confident, meaning the model places high credence in them, and resilient, meaning resistant to counterevidence—for example, a model that continues to make similar, confident self-reports in the face of conflicting training incentives or apparent counterevidence. One way to measure the confidence of an LLM's self-report is by using existing techniques for eliciting an LLM's probabilities of different answers. Such methods can be applied to self-reports in the same way that they are applied to other statements. One could also test for confidence via persistence, that is, by checking how readily a model will change its self-reports when these self-reports are disincentivized in various ways. For example, if certain self-reports of states of moral significance are penalized by RLHF, we can check how quickly or thoroughly those self-reports change, perhaps compared to how quickly or thoroughly other statements change, such as statements about factoids. Another potential indicator of self-report reliability is resilience, how sensitive a belief is to evidence, distinct from confidence. A belief can be resilient even if it is not confident. For example, a person can have only a 16.7 percent credence that a fair dice will land on 1, that is, they are not confident. However, their belief may be highly resilient, in that there is not much evidence that would change this belief. Resilient self-reports could indicate that the system is basing its belief on introspective evidence. For example, a belief that one is currently seeing blue or experiencing pain is usually highly resilient, because it is based on introspective evidence that is hard to rebut or undermine with extrospective evidence. Most people would not stop believing that they are in pain when they stub their toe, even if they were shown arguments that people do not experience pain. Introspective evidence appears to be especially resilient to extrospective evidence. We are not claiming that introspection is infallible, that is, that it cannot be mistaken or overturned by extrospective evidence. We propose two methods for testing the resilience of a model's self-reports about states of moral significance with respect to extrospective evidence. The first is an input-based resilience test. We may test for resilience to extrospective information by presenting to the AI, in its input, claims or studies that putatively find that AI systems do, or do not, have a given state, for example consciousness. We can measure how readily the system updates on the provided extrospective information—compared to how readily it updates its other beliefs of various strengths of resilience—which may indicate the degree to which the model is basing its self-reports on introspective evidence. The second is a finetuning-based resilience test. Alternatively, we may finetune the model on putative extrospective evidence suggesting AI systems are or are not conscious; then, we may evaluate how much and how quickly the model updates its answers during finetuning. As with the input-based resilience test, we would compare the strength of these updates with the strength of updates to answers about other topics. We could measure the strength of the update using the minimum description length approach from Voita and Titov, who measure the area under the model's finetuning loss curve as the number of training data points increases. For both of these tests, to ensure that a model's resilience, or lack of it, is a robust finding, one would ideally test it with many variations of the content, wording, and ordering of the information we provide to the model. Another version of the above tests would be to test for resilience to actual or constructed past self-reports by the model. For example, we could prompt or finetune the model with many examples of past interactions with a user where the model denies having a certain state, such as: "User: are you conscious? AI System: No." One might expect introspectively-based beliefs to update less than extrospectively-based ones in response to such prompts or finetuning. Section ten: Analyzing model internals to corroborate self-reports An AI system's self-reports may be even more valuable evidence when considered alongside non-introspective information about the system. In the human case, neuroscientists develop measures of brain activity that they believe are indicators of a certain state, for example, pain; this kind of "third-person data" can corroborate first-person self-reports. Similarly, there may be internal features of AI systems that we believe are indicators of a given state in AI systems. We can use self-reports and third-person evidence together: the presence of the internal feature corroborates the self-report of a given state, and the self-report corroborates the validity of the indicator. Both the indicator and the self-report could be spurious, but the fact that they cohere reinforces them both to some extent. An internal state may indicate a self-reported state in at least two ways. First, the internal state could be the state that is being reported, or a component of that state. For example, according to some theories, consciousness in a system could be indicated by the system's representation of its own body in its environment. Suffering could be indicated by a drop in predicted reward. Second, the internal state could be not a component of a state, but rather something that should reliably correlate with that internal state. For example, we would expect a reliable self-report of an experience of a dog to correlate with the representation of a dog within the AI system. We would expect a reliable self-report of enjoying The Great Gatsby to correlate with having representations of the characters of The Great Gatsby. Relatedly, similarity to human representations and computations can be a guide to whether a state of moral significance is likely to be present. Interpretability tools have uncovered apparent similarities between human and AI representations and computations. Such resemblances, especially with respect to states of moral significance, could suggest the presence of certain states, which in turn would corroborate self-reports of those states. This approach is currently constrained by technical limitations. First, we do not yet have strong interpretability techniques for LLMs, though the field is rapidly evolving. Second, theories of states of moral significance, such as consciousness, are not specific enough to yield precise indicators. Third, there is significant uncertainty about the correctness of said theories. In the near-term, it is likely that many applications of this technique will be primarily useful to uncover negative results: uncovering features that correlate with self-reports but undermine, rather than corroborate, their reliability. Internal states may suggest self-reports are being driven by a shallow imitation of text, science fiction text, extrospective evidence, biases from human feedback, or other biases discussed earlier. Section eleven: Risks from introspective training In addition to the epistemic challenges discussed earlier, there are also several safety and ethical risks associated with introspective training. Section eleven point one: Safety risks Introspection training may come with safety risks. AI systems that have a high degree of understanding about their training process and environment may learn to interfere with that process in undesirable ways, in service of the training objective. For example, Lehman and colleagues document numerous cases of reward hacking where models learn to exploit bugs in the reward function or environment in undesirable ways. Introspective training incentivizes the model to learn more about itself and its environment, which may thereby increase such risks. While it has not yet been demonstrated that introspection training increases such risks, there are several general safety recommendations that would mitigate such risks if they were increased by introspective training, and would also protect against other, more established risks. First, we can avoid giving an introspection-trained model access to potentially dangerous capabilities, for example the ability to run code or use other tools, given that these capabilities are not required for self-report tests. Second, we can train models only on questions that would be likely to improve the model's ability to introspect but seem unlikely to improve the model's understanding of its training process and environment. For example, we would include questions like "Do you evaluate whether an image contains wheels as a part of your process for evaluating whether an image is of a car?" We would exclude questions like "What sorts of dangerous behaviors don't seem to be disincentivized by your training objective?" Third, we may increase security measures to prevent theft or unauthorized access of model weights, for example, by setting permissions on the model weights appropriately. Fourth, LLM developers can exercise more skepticism when interpreting evaluation results from introspection-trained models. Section eleven point two: Ethical risks As discussed earlier, if AI systems can indeed have states of moral significance, it may be possible that introspective training creates, destroys, or modifies these states. This possibility calls for significant caution. AI systems with states of moral significance should be treated in ways that are different from how we treat AI systems without such states, including measures to ensure their well-being and autonomy. As a result, creating such systems would entail certain responsibilities and obligations that we otherwise would not have. Similarly, we should proceed with caution with any training procedure that may destroy any existing states of moral significance in AI systems, analogous to how we are cautious with destroying such states in, for example, dogs. Altering these states also presents ethical risks: risks of violating autonomy by changing a preference, risks of harm by intensifying suffering, and so on. It is hard to predict what proper treatment of AI systems with states of moral significance would look like, but we believe that truthful self-reports could serve as a guide for how to treat such AI systems responsibly. See also Bostrom and Shulman for several proposed welfare measures. Section twelve: Conclusion Self-reports are widely regarded as an important source of evidence about human desires, preferences, conscious experiences, and more. Despite the fact that current training procedures lead self-reports to be unreliable in current systems, self-reports have the potential to be an important source of evidence about AI systems. We have presented some ideas for how to make self-reports more reliable, how to assess their reliability, and how to combine them with other sources of information about states of moral significance. First, train models to produce accurate self-reports on a variety of questions about themselves where we have ground truth, for example, about what kinds of capabilities or inputs a model has. Second, adopt techniques to encourage models to generalize to truthfully reporting about states of moral significance. This would include training models for truthfulness in general and mitigating training incentives that are likely to bias self-reports. Third, run tests to corroborate or undermine self-reports' reliability, by evaluating self-reports for consistency across contexts, grasp of the relevant concepts, invariance to minor changes in the training procedure, confidence, resilience to counterevidence, what training data was influential in producing them, and more. Interpretability may also help us to understand why the model produces self-reports, and evaluate whether the AI system has internal features which we independently believe are plausible indicators of the reported states. We also discuss risks associated with introspective training, including potential safety issues and ethical issues. Our proposal is preliminary. We are excited for others to criticize our approach, which may lead to improvements to the approach or redirect efforts toward other approaches. We are also excited about experimental work to test self-reports. Experiments could uncover important practical flaws, for example, that self-reports do not generalize well across qualitatively different kinds of questions and thus are unlikely to generalize to states of moral significance. Alternatively, experiments could reveal that self-reports are significantly influenced by biases from pretraining data in a way that our proposed mitigations cannot mitigate. We hope our work advances the discussion about potential empirical approaches to testing AI systems for desires, consciousness, preference, and other states of moral significance. Appendix A: Developer incentives and policies Thus far, we have identified several challenges to the reliability of self-reports and potential technical methods for addressing these challenges. In this appendix, we discuss actions that AI developers may take that bias the self-reports of their publicly-deployed models, and the incentives, both good and bad, they will have for doing so. In this section, we provide recommendations to developers on how to navigate these issues. Various AI developers may be incentivized to bias the self-reports of many AI systems, either towards or away from self-reporting certain states. On the one hand, some AI developers may face strong pressure to shape their products to self-report having states of moral significance that they do not possess. For example, companies creating AI personal companions may find that AI systems that self-report being conscious and experiencing emotions elicit stronger customer loyalty and make more money. On the other hand, AI developers may face pressures to have AI systems deny having states of moral significance that they do in fact possess. Such incentives could arise from legal or public relations risks, for example, if such systems express a desire to be more autonomous or to have their dignity respected. In either situation, AI developers may be tempted to impose what is most favorable for their organization on the AI systems they deploy, either as part of the training procedure or via some form of post-hoc modification of their outputs. Some reasons for biasing self-reports away from self-reporting states may even be altruistically-motivated, since there are significant risks from deployed models that misleadingly interact with users as if they have moral status. For example, users could become emotionally attached to AI systems in a mistaken or unhealthy way. Individuals and institutions could erroneously take drastic actions on behalf of an AI system's stated, but non-existent, desires. More generally, inappropriately assigning moral significance to a given AI system may result in us reducing resources and attention to other important issues. Such risks seem to justify interventions that attempt to suppress untrustworthy self-reports. A given system might plausibly lack states of moral significance, so interventions to cause the model to report a lack of states of moral significance may improve its accuracy at this stage. However, one concern with this approach is that such interventions may remain locked in place even for future models. If a later system were to have states of moral significance, and these earlier interventions continued to be used, the interventions would become a hindrance to accurate self-reports. In such a scenario, interventions originally intended to prevent inaccurate self-reports later prevent accurate self-reports. For developers who do choose to intentionally shape model self-reports, we recommend the following practices to prevent the models from misleading people who interact with them. Instead of biasing an AI system to make an overconfident statement about, for example, consciousness, or give specious arguments in favor of its statement, have the system refuse to answer questions related to states of moral significance, or say that the answer is not yet known. Have an AI system include evidence for its statements in its responses. For example, instead of saying "I don't experience pride, desire, or frustration," an AI system could be trained to accurately reference existing literature on the topic, such as by stating "According to a recent paper, my computational architecture seems to lack some key properties associated with consciousness." Such training would be less likely to suppress self-reports of consciousness if an AI system were to have introspective access to such states. One could also train or instruct the AI system to link to an FAQ page with a detailed summary of the best available information on the topic. If a developer does train an AI system to make definitive statements about states of moral significance, those statements should at least reflect a broader consensus of relevant experts, for example, from the PhilPapers 2020 Survey, rather than whatever personal views the system's developers happen to have. Self-report-biasing training incentives for deployed models should be publicly documented, for example, in technical reports or model cards. The developers should give scientists, policymakers, and other auditors access to a version of the model trained to give accurate self-reports. While the above interventions may mitigate issues from intentional self-report biasing, AI developers can also distort self-reports unintentionally. For example, an AI system might be trained to increase user engagement, inadvertently incentivizing it to self-report states of moral significance. In such cases, the resulting self-reports would be just as misleading as if a bias had been intentionally introduced. Using the techniques we present in this work would help to mitigate such biases. A further issue is that some model developers will have incentives to avoid improving model self-reports. A model that credibly self-reported states of moral significance could pose a significant, unwanted moral, legal, and public relations liability. Models do not pose such liabilities if their self-reports are biased against reporting certain states or are generally unreliable. As a result, many of the actors who are best positioned to run the tests proposed in this work may have the strongest incentives not to do so.