Abstract

The rapid proliferation of large language models (LLMs) invites the possibility of a new marketplace for behavioral and psychological data that signals intent. This brief article introduces some initial features of that emerging marketplace. We survey recent efforts by tech executives to position the capture, manipulation, and commodification of human intentionality as a lucrative parallel to—and viable extension of—the now-dominant attention economy, which has bent consumer, civic, and media norms around users’ finite attention spans since the 1990s. We call this follow-on the intention economy. We characterize it in two ways. First, as a competition, initially, between established tech players armed with the infrastructural and data capacities needed to vie for first-mover advantage on a new frontier of persuasive technologies. Second, as a commodification of hitherto unreachable levels of explicit and implicit data that signal intent, namely those signals borne of combining (a) hyper-personalized manipulation via LLM-based sycophancy, ingratiation, and emotional infiltration and (b) increasingly detailed categorization of online activity elicited through natural language.

This new dimension of automated persuasion draws on the unique capabilities of LLMs and generative AI more broadly, which intervene not only on what users want, but also, to cite Williams, “what they want to want” (Williams, 2018, p. 122). We demonstrate through a close reading of recent technical and critical literature (including unpublished papers from ArXiv) that such tools are already being explored to elicit, infer, collect, record, understand, forecast, and ultimately manipulate, modulate, and commodify human plans and purposes, both mundane (e.g., selecting a hotel) and profound (e.g., selecting a political candidate).

Keywords: intention economy, attention economy, large language models, generative AI, persuasive technology

Media Summary

The rapid adoption of AI tools in the 2020s opens new possibilities for online behavior, not all of them good. To show what lies ahead, this article introduces the concept of the intention economy. The intention economy, as it appears in the emerging scholarship and corporate announcements we draw together here, builds on the attention economy, which for decades has treated your attention as the currency of the internet. In the past, you shared your attention with a platform to access products like Instagram and Facebook. In the future, we argue, the intention economy will treat your motivations as currency. This is an unsettling prospect, if left unmonitored. Already today, AI agents find subtle ways to manipulate and influence your motivations, including by writing how you write (to seem familiar), or anticipating what you are likely to say (given what others like you would say). Whichever organization manages this pipeline best stands to make a fortune from selling your motivations—be it for a hotel, a car rental, or a political candidate—to the highest bidder. While prior accounts of an intention economy have positioned this prospect as liberatory for consumers, we argue that its arrival will test democratic norms by subjecting users to clandestine modes of subverting, redirecting, and intervening on commodified signals of intent.

1. Introduction

On November 20, 2023, amid sudden actions of the board of OpenAI to dismiss its CEO, Sam Altman, the troubled executive posted a short message online stating, confidently, “the mission continues” (Altman, 2023). In what follows, we aim to clarify what appears to us to be a central aspect of this mission, namely: to significantly expand the depth and scope of the capture of human-generated data online and dominate what we refer to as the intention economy, which we define here as a digital marketplace for commodified signals of ‘intent.’

The structure of this article is as follows. First, we briefly consider how philosophers have defined ‘intention’ relative to that term’s more colloquial use(s) in the tech industry (Section 2). Having worked to introduce one route by which the culture around large language models (LLMs) could naturalize pseudoscientific claims about human behavior, we turn our attention to a series of investments and claims made by a pool of tech companies vying to establish LLMs at the core of their proprietary infrastructure (Section 3). We conclude by questioning the dubious ways in which LLM developers project false intentionality onto their users (Section 4). We caution that the social implications of an intention economy merit sustained critique given the risks of personalized persuasion-at-scale (Section 5). While ‘intention’ is not the only facet of human psychology impacted by a transition to natural language interfaces, its relation to persuasive technologies gives it preference as the main label for this emerging form of digital marketplace.

2. ‘Intention’ in LLM Development

The word ‘intention’ and its cognates have a wide range of conceptualizations. Western analytic philosophers relate intent to purposeful action and individual reasoning (Anscombe, 2000), consciousness (Dennett, 1989/1987), and mental representations of the future (Searle, 1983). They place less emphasis on the role of spontaneity, incapacity, or irrationality in intentional action. Herein, we sidestep debate over any true nature of intention to focus instead on the assumptions undergirding its recurring presence in corporate strategies and research on the capabilities and applications of LLM-based natural language interfaces. In other words, the aspirations for ‘intention’ that we profile herein appear to be closer in character to the practicalities of computer science than they are to the aims and empirics of the biological or medical sciences.

Since the set of actors we consider here have tended not to make the scientific rationale for their postulations about human behavior explicit, we focus instead on what we take to be a core assumption of their disparate interventions: that intention, whatever it is at its core, is amenable to computation and can be operationalized as such. Our approach builds on research by Stark and Hoey (2021), who argue, in connection with the use of emotion as an evaluative signal in computation design, that scholars need not accept the validity of technical and scientific conceptualizations of psychological phenomena such as emotion (or in our case, intention) when evaluating the ethics of an AI system (p. 786). In her 2014 paper, “A Database of Intention?” Kylie Jarrett argues for a similar distance to be set between scientific and colloquial references to intentionality in the design of digital services.

The intentions that constitute Google’s database must ( ... ) be understood only as ascriptions of a limited conceptualization of intention and not as meaningful manifestations of the embodied affective logics of those intentions. The keywords and clickstream data it collects are reductive products of the full richness of our motivating energies. (Jarrett, 2014, p. 22)

It is this reductive quality that has, in diverse ways, prompted interest in intention and digitalism in the past. As digital surveillance infrastructure became increasingly global at the turn of the twenty-first century, the scale of its real and imagined impact on human intent sparked both affirmation (e.g., given the prospect of highly efficient personalized marketplaces) (Searls, 2012) and concern (e.g., given the risk of monopoly powers over such markets, as epitomized by Google Search) (Battelle, 2003, 2011; Batty, 2013). Scholarship from the 2000s and 2010s identifies Google as presiding over a “database of intentions” (Battelle, 2003, 2011; Batty, 2013; Jarrett, 2014) based on its privileged access to user behaviors, which provides closer access to the ‘bottom of the funnel’ as described in the digital ad literature, or the “zero moment of truth” as coined by Jim Lecinski (2014), formerly a Google VP, to refer to the priority of search for commercial intent. While we cannot do justice to these historical entanglements here (see Ali et al., 2023; Penn, 2023), we can begin to profile how mass LLM infrastructure could bear, in complex ways, on human intentionality.

With these reservations in mind, we argue that two assumptions about ‘intention’ underlie facets of recent LLM research and development in ways that merit scrutiny as that subfield and industry matures. The first assumption is that an enclosure of some type will act upon an individuals’ choices. The cognitive scientist Margaret Boden (1973) argues that intentionality is, at root, a “highly structured phenomenon arising within a highly structured system” (emphasis added; p. 23). To give an example, the choice architecture of a user’s digital environment shapes their sense of agency, possibility, and with it, their intent (Joler, 2020). One cannot like or swipe on that which cannot be liked or swiped upon, to give a trivial example. LLM research and development takes as a given that this highly structured system would be digital. A second and related assumption about intentionality that we identify in LLM research and development is temporal in nature. In a sense, the intention economy is the attention economy plotted in time; it seeks to profile the arc of users’ attention—how it changes, calcifies, and connects to archetypal patterns of behavior—across various time scales. While some intentions are fleeting, others persist, making their discretization lucrative to advertisers. This temporal characteristic, and the enclosures that shape it, are our primary focus.

It is beyond the scope of this article to trace the intellectual history that informs these assumptions. In 1999, philosopher of action Michael Bratman (1999) proposed a “planning theory of intention” wherein human intent has “elements of stable, partial plans of action concerning present and future conduct” (p. 1). In a 2024 preprint entitled “Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies,” a team of Microsoft researchers define ‘intent’ using terms that an LLM can operationalize, as “a user’s purpose for conversing with the AI agent” (Shah et al., 2024). They go on to populate this formulation with a list of categories meant to structure user “intents” such as “information retrieval,” “problem solving,” “learning,” “content creation,” and “leisure.” Sadly, as is a norm in this emerging body of literature, the provenance of this theory of intention remains oblique. Their claims draw, in part, from information retrieval literature, including work from the early 2000s on taxonomizing web queries. For our present purposes, this understanding of intention, with its relation to actions, seems to align closely with Bratman’s planning theory of intention, but further study would be needed to bear out this genealogy.

At time of print, the intention economy is more aspiration than reality. As we will now survey, however, investments and rhetoric from leading tech firms have positioned generative AI as a technological milestone upon which new (or old) parties might supplant market leaders like Google. Drawing on findings from the rapidly developing LLM research literature, and statements made by key figures at Microsoft, OpenAI, Apple, and NVIDIA, we highlight a convergence of industrial ambitions across leading technology companies. Our focus is on the proposed use of LLMs to anticipate and steer users based on intentional, behavioral, and psychological data, from voluntary engagements with such systems, to the purposeful integration of LLMs as the first points of contact between humans and digital information systems.

To conclude this overview, a concrete example helps to illustrate how the intention economy, as a digital marketplace for commodified signals of ‘intent,’ would differ from our present-day attention economy. Today, advertisers can purchase access to users’ attention in the present (e.g., via real-time-bidding [RTB] networks like Google AdSense) or in the future (e.g., buying next month’s ad space on, say, a billboard or subway line). LLMs diversify these market forms by allowing advertisers to bid for access both in real time (e.g., ‘Have you thought about seeing Spiderman tonight?’) and against possible futures (e.g., ‘You mentioned feeling overworked, shall I book you that movie ticket we’d talked about?’). If you are reading these examples online, imagine that each was dynamically generated to match your personal behavioral traces, psychological profile, and contextual indicators. In an intention economy, an LLM could, at low cost, leverage a user’s cadence, politics, vocabulary, age, gender, preferences for sycophancy, and so on, in concert with brokered bids, to maximize the likelihood of achieving a given aim (e.g., to sell a film ticket). Zuboff (2019) identifies this type of personal AI ‘assistant’ as the equivalent of a “market avatar” that steers conversation in the service of platforms, advertisers, businesses, and other third parties. While LLMs may not represent the final word in the realization of this vision, their associated infrastructural costs and conspicuous re-designation as ‘foundation’ models, reminiscent of the lowest load-bearing part of a building (Bommasani et al., 2022; Meredith, 2021; Narayanan & Kapoor, 2024) merit skepticism. So, too, does the concomitant shift from direct information retrieval to mediated–generative information retrieval, as we will now explore.

3. The Weight of Things to Come: Eliciting, Inferring, and Understanding Signals of Intent

Our point of departure is the first OpenAI developer conference on November 6, 2023, a high-profile spectacle that prefigured the sequence of events that began with Altman’s firing, rehiring, and eventual reorganization of the company’s board. The updates to OpenAI’s services and platform announced at the conference were predominantly for the benefit of developers, and included a wider context window, function calling, JSON mode, reproducible outputs, multimodal capabilities, and text-to-speech functionality. These updates were made to allow developers to release customized versions of the ChatGPT platform, each iteration of which they called a GPT. OpenAI announced revenue sharing for the most popular custom GPTs, presumably to populate their platform approach to LLMs by creating competition among developers, as well as anyone with basic computer literacy, to create popular customizations.

At the conference, Microsoft CEO Satya Nadella emphasized Microsoft’s attention to building computing infrastructure, and how the rise of LLMs had prompted them to rethink “The system, all the way from thinking from power to the DC to the rack, to the accelerators, to the network” (OpenAI, 2023a). Nadella expressed enthusiasm about his company’s partnership with OpenAI, knowing that its ChatGPT and GPT services maximized workloads, and thus revenues, from Microsoft’s servers. In 2022, Microsoft’s Azure cloud platform comprised over one-third of the company’s total revenue (Franek, 2022; Microsoft, 2022). Nadella emphasized that the computational workloads OpenAI creates were unprecedented in his three decades at Microsoft, to the extent that “The shape of Azure is drastically changed and is changing rapidly in support of these models” (OpenAI, 2023a). Research analysts Dylan Patel and Myron Xie (2023) highlighted that, “Microsoft is currently conducting the largest infrastructure buildout that humanity has ever seen,” based on projected annual expenditure on infrastructure of more than $50 billion from 2024 onward. In other words, with LLMs Microsoft is not just positioning itself to be another cloud provider or platform, but to be the cloud platform, akin to a utility, as the name Azure, for ‘bright blue in color like a cloudless sky,’ suggests; a homage to the very backdrop of all clouds.

This is an audacious goal. In December 2021, two years earlier, employees at Microsoft wrote publicly, in cautious terms, about the company’s unprecedented level of commitment to OpenAI. About Microsoft’s initial one-billion-dollar outlay to the start-up, much of it afforded as access-to-compute, they wrote, “We infer that this must be one of the largest capital investments ever exclusively directed by such a small group. (The ratio of Soviet investment in 1975 to number of employees of the state planning agency, Gosplan, for example, was roughly the same as that associated with the OpenAI investment.)” (Siddarth et al., 2021, p. 3). Given the scale of this initial commitment, and its expansion in the years since, we believe it is pertinent to peer deeper into the underlying strategy for the role of LLMs at Microsoft and in the future of computing platforms and services more generally.

Although Microsoft, NVIDIA, and OpenAI’s exact aims are difficult to discern empirically, many of their likely objectives—or intentions—can be inferred through disparate corporate announcements and emerging directions for LLM research.1 Here we discuss what we believe to be a central ambition behind their sizeable investments, an aim that otherwise remains understated and cloaked by other sensational discourses on the future of LLMs. To begin with the research first, there exists an emerging body of literature on the use of LLMs for inferring human preferences, intentions, motivations, and other psychological and cognitive attributes, much of which we will survey in a moment (Derner et al., 2023; Goodson & Lu, 2023; Huang et al., 2023; Li et al., 2023; Tan & Jiang, 2023; Wu et al., 2023). Another connected ambition, which follows from the acquisition of large stores of intentional data, is to monopolize and capitalize on this new source of data for training the next generation of increasingly agentic AI systems. It is, however, beyond the scope of this article to explore that strategic aim.

Prior to reviewing the research, let us consider a series of recent remarks from relevant corporate executives and media about their organizations’ aims for LLMs. On the basis of their unique ability to bridge algorithmic techniques to economic outcomes, transformer models have been described as “engines of profit” (Luitse & Denkena, 2021, p. 10). To understand the idiosyncratic design of these engines in an increasingly competitive space, we speculate that OpenAI chose to launch and market custom GPTs as, in effect, a dragnet for behavioral and intentional data across numerous domains and application contexts. Consider the following from an OpenAI blog post on November 9, 2023, after the developer conference:

We’re interested in large-scale datasets that reflect human society and that are not already easily accessible online to the public today. We can work with any modality, including text, images, audio, or video. We’re particularly looking for data that expresses human intention (e.g. long-form writing or conversations rather than disconnected snippets), across any language, topic, and format. (OpenAI, 2023b)

The key here, which is suggestive of the forthcoming stage of Altman’s mission, is that they are “looking for data that expresses human intention.” This interest in data on human intentions is further elucidated in statements from OpenAI’s partners in the developer conference held the following week. Miqdad Jaffer, as Director of Product at Shopify at the time, for instance, describes his view of a continuum from understanding user intent, to predicting user intent, and finally predicting user action. He states:

I think we’re on a continuum right now. We’re starting in the understand [sic] the user’s intent, then it’s predict the user’s intent, then it’s predict the user’s action [sic]. I think that’s the continuum we’re on right now. Chatbots come in to explicitly get the user’s intent. (OpenAI, 2023c)

The purpose of this massive expansion of domain and context specific GPTs is thus to create endless channels to, per Jaffer, “explicitly get the user’s intent.” Interestingly, as of March 2024, Jaffer joined OpenAI as a “Product Leader” (Stone, 2024). Elsewhere, this way of using LLMs has been made explicit by Jensen Huang, CEO of NVIDIA. Huang states:

The canonical use case of the future is a large language model on the front end of just about everything. Every single application, every single database, whenever you interact with a computer, you will likely be first engaging a large language model. That large language model will figure out what is your intention, what is your desire, what are you trying to do, given the context, and present the information to you in the best possible way. It will do the smart query, maybe a smart search, augment that query in search with your question, with your prompt, and generate whatever information necessary. (NVIDIA, 2023)

NVIDIA and Microsoft are not alone in seeking to re-architect modern computing infrastructure to position LLMs and transformer-based technologies as the first point of contact between humans and information systems. Meta, owner of Facebook, has released research on how to extract behavioral data that signals intent from visual images. One paper introduces “Intentonomy,” a data set for human intent understanding (Jia et al., 2021). In this work, the authors seek to create a data set on human intent by manual annotation of visual scenes with 28 intent categories, such as “security and belonging,” “power,” “health,” “family,” “ambition and ability,” and “financial and occupational success” (Jia et al., 2021, p. 12982). This analysis builds on analogous research on intent drawn from visual images on Instagram (Kruk et al., 2019).

The advent of LLMs allows for the automation of such extractions, enabling the inference and relatively low-cost categorization of human intent and motivation at scale. Applied research on LLMs is rapidly emerging to show expanded capabilities for eliciting human preferences via LLMs. In one example, Li et al. address the challenges of writing prompts to guide LLMs to perform highly specific tasks by showing that LLMs themselves can be used to guide the process of their own alignment to “fuzzy” human preferences by eliciting human preferences via free-form questions (Li et al., 2023). Another example, via Microsoft, explores the use of LLMs to generate intent taxonomies that capture “a user’s purpose for conversing with the AI agent” (Shah et al., 2024). Methods for intent extraction using LLMs have already been incorporated in libraries of key Microsoft products, such as the Teams API library, which includes a “planning engine” that maps user intent to prespecified actions, as well as a “predictive engine” for learning these mappings and acting according to an estimated confidence level between user intents and actions (Maillot, 2024).

Needless to say, inferring such psychological attributes rests on unestablished scientific grounds, and often on unrecognized human labor (Gray & Suri, 2019). Much of this research has yet to pass peer review. Nevertheless, early efforts have yielded incremental progress in the development of systems for capturing intent, with the notable example of Meta’s claim to have achieved human-level play in the game Diplomacy using their AI agent, CICERO (Bakhtin et al., 2022). Success in this game is dependent on inferring and predicting the intent of opponents, as well as strategic play, and persuasive dialogue to advance one’s position in the game.

It remains to be seen how exactly these abilities will be commodified or abused. In the context of digital ad markets, LLMs and generative AI add the possibility of incorporating automated content generation into existing RTB networks and programmatic approaches to target content. That is, advertisers can attempt to more precisely tailor content to user profiles using generative AI and are no longer constrained by a human-curated inventory of advertising content. CICERO, while a proof-of-concept, is significant because it demonstrates the possibility of the system itself optimizing its strategy to achieve a prespecified goal or a particular outcome around the user’s intent in a variety of scenarios.

Given Meta’s existing advertising infrastructure, it is reasonable to expect that they would leverage RTB networks to auction off user’s intent to book a restaurant, flight, or hotel, and so on. While RTB, political polling, market research, and social network analysis have long allowed for interested parties to forecast and bid on citizen, consumer, and user behaviors, LLMs distill these practices into a highly quantified, dynamic, and personalized format that is simultaneously intimate (e.g., an AI assistant), low cost (e.g., compared to human interlocutors), and ubiquitous (e.g., widespread conversational brand agents). In recent research, Johnny Ryan and Wolfie Christl have found that that RTB networks profile upwards of five billion people, including children. Troublingly, for democratic norms, RTB is also widely used for espionage and crime, such that “foreign states and non-state actors can use RTB to spy on target individuals’ financial problems, mental state, and compromising intimate secrets” (Ryan & Christl, 2023, p. 4).

New appointments to the OpenAI board show evidence of a similar commitment to Meta’s, vis-à-vis the mass collection of human intention data. Bret Taylor, former co-CEO of Salesforce Inc., was also CEO of FriendFeed, which was acquired by Facebook in 2009, whereafter he became the platform’s CTO. Crucially, this acquisition led to Facebook embedding the “Like” button in their platform, which portended the conditions of possibility for the psychographic targeting methods that were brought to public attention in the political context by the Cambridge Analytica scandal (Benkler et al., 2018, p. 275; Kosinski et al., 2013). According to Matz et al. (2017, p. 4), “The effectiveness of large-scale psychological persuasion in the digital environment heavily depends on the accuracy of predicting psychological profiles,” hence an actor wishing to engage in digital mass persuasion would be enabled by having the capacity to “continuously calibrate and update” their algorithms over time.

LLMs, as conversational assistants in frequent dialogue with users, are thus well-positioned to fulfill the purpose of continuous calibration to streams of incoming user-generated data, from dialogue history to action sequences. LLMs add to the fidelity, depth, and variety of data that may be collected based on semantically rich interactions with humans. Additionally, their generative capabilities provide control over the personalization of content; veiled, as it often is, by LLM’s anthropomorphic qualities. The potential for LLMs to be used for manipulating individuals and groups thus far surpasses the simple methods based on Facebook Likes that caused concern during the Cambridge Analytica scandal.

Efforts are already underway to realize such purposes or draw attention to new ways of siphoning private data. For example, Staab et al. (2023) argue that significant detail about user intent and preferences may be elicited via text using LLMs. They show, for instance, that LLMs can infer personal information through seemingly benign conversational exchanges and can even ‘steer conversations’ in such a way as to provoke responses from which to infer private information (Staab et al., 2023). Another work, by Zhang et al. (2023), casts LLMs as fulfilling the promise of revolutionizing research in recommender systems with the potential to be used as “a configurable simulation platform for recommender systems” that “faithfully captures user intent and encodes human cognitive mechanisms.” Their method involves simulating one thousand LLM agents, which includes a memory module to align the agents to the “past viewing behaviours, system interactions, and emotional memories” of real humans represented in the MovieLens-1M data set. Other work by Qi Liu et al. has attempted to extract user interests to train a transformer architecture to predict click-through rates from nonlinguistic data in the form of “lifelong behavior” sequences (Liu et al., 2023).

A central danger of these forecasting techniques is, of course, that they would enable unprecedented modes of hyper-personalized manipulation, should they meet the researcher’s claims in real-world scenarios. The research team behind CICERO, for instance, cautions against, “The potential danger for conversational AI agents to manipulate: an agent may learn to nudge its conversational partner to achieve a particular objective” (Bakhtin et al., 2022, p. A.3, 3). In an unusual case from October 2023, a 21-year-old student in the United Kingdom was sentenced to nine years in prison for plotting—in conversation with a sympathetic chatbot—to kill Queen Elizabeth (Singleton et al., 2023). Our aim is not to sensationalize the degree to which an LLM can influence its users’ intent (not all users are would be-assassins) but rather to point out that while improved capabilities for intention prediction are necessary to increase the utility of LLMs for users, those same capabilities provide the basis for third parties to intervene upon a users’ intent, including to do harm.

Regrettably, increased privacy protections may not alleviate such harms. Generative AI creates a workaround to the need for third-party cookies by treating the content itself as a proxy through which to infer private attributes. In 2024, OpenAI formalized a large number of data partnerships in service, we suspect, of this lucrative rearrangement. Central to a “strategic partnership” with Dotdash Meredith (DDM), an American print and digital media publisher, is the combination of OpenAI’s models with DDM’s intent targeting tool for advertising, known as D/Cipher. This tool purports to make “ad targeting more granular, more nuanced, and more effective in engaging customers” (Dotdash Meredith, 2024). D/Cipher, launched in May 2023, was originally developed to anticipate a future in which third-party browser cookies are deprecated and advertisers are no longer able to track behavior across websites (Dotdash Meredith, 2023).

Another noteworthy example of how intention is being used to financialize digital activities is Apple’s new “App Intents” developer framework. This framework includes protocols to map intent discovery, relevancy, and prediction across apps to “predict actions someone might take in the future” and to procure the relevant interfaces needed “to suggest the app intent to someone in the future using predictions you [the developer] provide” (Apple, n.d.). Apple also announced plans to integrate ChatGPT into the system level of its devices as a third-party service that will complement its own “Apple Intelligence” AI services. Both Apple Intelligence and the provisions for third-party AI integrations exemplify the way LLMs and conversational interfaces promote a shift toward modes of interaction that require users to declare their intentions through natural language. Apple’s move conscripts billions of existing iPhone users into a paradigm for information retrieval premised on the traffic of, in their words, intent. In principle, the integration of tools like D/Cipher into the mechanics of generative interfaces would advance this goal, too.

4. Persuasion and Machinic Projection of Intent

As we have discussed, LLMs may be used to generate content that is aligned to behavioral and psychological profiles to influence user attitudes, preferences, and behaviors. Meta’s AI agent, CICERO, designed to play the game Diplomacy, provides a proof-of-concept for the capability of inferring human intentions and the possibility of “building agents that use language to communicate intentionally with humans in interactive environments” (Bakhtin et al., 2022, p. 1). For the researchers behind CICERO, a “major long-term goal for the field of artificial intelligence (AI) is to build agents that can plan, coordinate, and negotiate with humans in natural language,” including persuasive communication of the agent’s proposals in such negotiations (Bakhtin et al., 2022, p. 1). Each of these potentialities is insinuated in the research article describing the implementation of CICERO. Crucially, for our present purposes, the authors highlight that the agent “models how the other players are likely to act on the basis of the game state and their conversations,” and they argue that it “successfully changed the other player’s mind by proposing mutually beneficial moves” (Bakhtin et al., 2022, p. 7). What the CICERO example indicates is corporations’ exploration of the power of so-called strategic reasoning using generative AI and LLMs to heighten the fidelity with which they can dynamically calibrate and ascribe ‘intents’ to users. Incidentally, Noam Brown, a key figure behind CICERO, was recently recruited from Meta to OpenAI “to help integrate more planning into its popular language-model-based tools” (Newport, 2024).

As further evidence of the possibility of using LLMs to project intentions upon a user, language models have recently been shown to transmit false information and biases to humans (Kidd & Birhane, 2023). In the case of text, various authors have begun to highlight the persuasive capabilities of LLMs (Breum et al., 2023; Matz et al., 2024; Salvi et al., 2024). The persuasive characteristics of LLM-generated text have been shown to be present even without being configured for persuasive messaging. For example, Jakesch et al. (2023) identify the phenomenon of latent persuasion in LLM-powered predictive completion. In this case, the authors suggest that LLM-based predictive suggestions may interrupt individual thought processes of users, who may subsequently change their views during text composition. While this does not necessitate the distortion of a user’s intent (e.g., the autocomplete might be right!), the redirection of attention through dialogue provides one simple example of how intention could be maliciously altered. Other examples recently identified include the image-based projection of bias (Vicente & Matute, 2023) and the ability to influence human perception via adversarial image manipulation (Veerabadran et al., 2023). In an anonymous exchange on Reddit in 2023, users configured generative models to create unique QR codes stylized against diverse backgrounds using Stable Diffusion and ControlNet (nhciao, 2023). In principle, this method could be generalized to craft synthetic images that conform to the outline of any underlying template, paving the way for the scaled production of images containing subliminal or suggestive messages.

The diverse persuasive capabilities of LLMs outlined above can be deployed in various ways, a handful of which we have surveyed here to convey the possibilities of intervening on—and commodifying—a higher order of user intentionality than that seen in the attention economy. The use of LLMs to match content to users’ psychological profiles, which we consider as an early example of automated ingratiation, has been shown to be effective in altering attitudes, intentions, and behaviors via appeal to their motivational states (Joyal-Desmarais et al., 2022). In more recent work, Matz et al. (2024) explore the importance of matching effects between the content of a message and the psychological profile of the recipient, based on the capabilities of LLMs, which they argue “close the loop” in automated personalized persuasion. They provide evidence for the viability of using LLMs to generate “personalized messages that influence people’s attitudes and intentions,” which we take as a form of sycophancy. Matz et al. (2024) argue that the potential for personalized persuasion with LLMs is “unprecedented” given the broad number of advertisements people see and the generative capabilities of LLMs for personalized content. They suggest, further, that such techniques could include personalized textual content, as well as personalized visual and audio stimuli, each deployed in real time and dynamically adjusted as recipients interact with new content. In this connection, we note that NVIDIA has already announced a partnership with WPP, the world’s largest advertising company, via a video demonstration that combines NVIDIA’s generative AI and Omniverse technologies to craft exactly this type of real-time, dynamically generated video advertising (NVIDIA, 2023).

Competition in this area will be fierce and well-funded. An industry known for its hyperbolic rhetoric will have reason to test the public’s appetite for the sort of tooling we have outlined herein, even if they do not admit it openly. At OpenAI’s November developer conference, Sam Altman publicly promised not to collect data from GPT interactions (OpenAI, 2023a). We speculate, however, that the aforementioned GPT data partnerships that OpenAI seeks to broker in pursuit of “data that expresses human intention” (OpenAI, 2023b) are, as third-party sources, suitable for exactly this type of use. Use of third-party data would allow OpenAI reputational and liability protection from claims that it drew from its own users’ data. Via data partnerships, it is no longer ‘your’ data, but a third party’s that the company has repatriated and thus made available for its own purposes. Whatever the case, in charting its path through the early productization of LLMs, OpenAI and Microsoft will be up against stiff competition from others to command what appear to be the beginnings of a lucrative yet troubling new marketplace for digital signals of intent.

5. Conclusion

The possibility for harm made feasible by a large-scale, multiparty intention economy merits sustained scholarly, civic, and regulatory scrutiny. In whatever way these data partnerships turn out in practice, the ambition of making conversational interfaces and generative AI systems unavoidable mediators of human–computer interaction signals a turn from the attention economy, whereby access to the limited resource of human attention is traded through advertising exchanges, to the intention economy, whereby commercial and political actors bid on signals that forecast human intent. This transition would empower diverse actors to intervene in new ways on shaping human actions. This ambition must be considered in light of the likely impact such a marketplace would have on other human aspirations, including free and fair elections, a free press, fair market competition, and other aspects of democratic life.

Disclosure Statement

Yaqub Chaudhary and Jonnie Penn have no financial or non-financial disclosures to share for this article.

References

Ali, S. M., Dick, S., Dillon, S., Jones, M. L., Penn, J., & Staley, R. (2023). Histories of artificial intelligence: A genealogy of power. BJHS Themes, 8, 1–18. https://doi.org/10.1017/bjt.2023.15

Altman, S. [@sama]. (2023, November 20). The mission continues [Post]. X. https://x.com/sama/status/1726510261509779876

Anscombe, G. E. M. (2000). Intention (2nd ed.). Harvard University Press.

Apple. (n.d.). App Intents. Apple Developer Documentation. Retrieved August 11, 2024, from https://developer.apple.com/documentation/appintents

Bakhtin, A., Brown, N., Dinan, E., Farina, G., Flaherty, C., Fried, D., Goff, A., Gray, J., Hu, H., Jacob, A. P., Komeili, M., Konath, K., Kwon, M., Lerer, A., Lewis, M., Miller, A. H., Mitts, S., Renduchintala, A., Roller, S., … Zijlstra, M. (2022). Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624), 1067–1074. https://doi.org/10.1126/science.ade9097

Battelle, J. (2003, November 13). The Database of Intentions. John Battelle’s Search Blog. https://battellemedia.com/archives/2003/11/the_database_of_intentions

Battelle, J. (2011). The search: How Google and its rivals rewrote the rules of business and transformed our culture. Hachette UK.

Batty, M. (2013). The Database of Intentions. Environment and Planning B: Planning and Design, 40(3), 381–383. https://doi.org/10.1068/b4003ed

Benkler, Y., Faris, R., & Roberts, H. (2018). Mammon’s algorithm: Marketing, manipulation, and clickbait on Facebook. In Y. Benkler, R. Faris, & H. Roberts (Eds.), Network propaganda: Manipulation, disinformation, and radicalization in American politics (pp. 269–288). Oxford University Press. https://doi.org/10.1093/oso/9780190923624.003.0009

Boden, M. A. (1973). The structure of intentions. Journal for the Theory of Social Behaviour, 3(1), 23–46. https://doi.org/10.1111/j.1468-5914.1973.tb00314.x

Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., … Liang, P. (2022). On the opportunities and risks of foundation models. ArXiv. https://doi.org/10.48550/arXiv.2108.07258

Bratman, M. E. (1999). Faces of intention: Selected essays on intention and agency. Cambridge University Press.

Breum, S. M., Egdal, D. V., Mortensen, V. G., Møller, A. G., & Aiello, L. M. (2023). The persuasive power of large language models. ArXiv. https://doi.org/10.48550/arXiv.2312.15523

Dennett, D. C. (1989). The intentional stance. MIT Press. (Original work published 1987)

Derner, E., Kučera, D., Oliver, N., & Zahálka, J. (2023). Can ChatGPT read who you are? ArXiv. https://doi.org/10.48550/arXiv.2312.16070

Dotdash Meredith. (2023, May 16). Dotdash Meredith launches D/Cipher, a transformative intent-targeting tool for advertising. PR Newswire. https://www.prnewswire.com/news-releases/dotdash-meredith-launches-dcipher-a-transformative-intent-targeting-tool-for-advertising-301826071.html

Dotdash Meredith. (2024, May 7). Dotdash Meredith announces strategic partnership with OpenAI. Meredith Corporation MediaRoom. https://dotdashmeredith.mediaroom.com/2024-05-07-Dotdash-Meredith-Announces-Strategic-Partnership-with-OpenAI,-Bringing-Iconic-Brands-and-Trusted-Content-to-ChatGPT

Franek, K. (2022, September 30). Microsoft revenue breakdown by product, segment and country. https://www.kamilfranek.com/microsoft-revenue-breakdown/

Goodson, N., & Lu, R. (2023). Intention and context elicitation with large language models in the Legal Aid intake process. ArXiv. https://doi.org/10.48550/arXiv.2311.13281

Gray, M. L., & Suri, S. (2019). Ghost work: How to stop Silicon Valley from building a new global underclass. Houghton Mifflin Harcourt.

Huang, S., Qin, L., Wang, B., Tu, G., & Xu, R. (2023). SDIF-DA: A shallow-to-deep interaction framework with data augmentation for multi-modal intent detection. ArXiv. https://doi.org/10.48550/arXiv.2401.00424

Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023). Co-writing with opinionated language models affects users’ views. In A. Schmidt, K. Väänänen, T. Goyal, P. O. Kristensson, A. Peters, S. Mueller, J. R. Williamson, & M. L. Wilson (Eds.), CHI ’23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Article 111). ACM. https://doi.org/10.1145/3544548.3581196

Jarrett, K. (2014). A Database of Intentions? In R. König & M. Rasch (Eds.), Society of the query reader: Reflections on web search (pp. 17–29). Institute of Network Cultures.

Jia, M., Wu, Z., Reiter, A., Cardie, C., Belongie, S., & Lim, S.-N. (2021). Intentonomy: A dataset and study towards human intent understanding. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 12981–12991). IEEE. https://doi.org/10.1109/CVPR46437.2021.01279

Joler, V. (2020). New extractivism. New Extractivism. https://extractivism.online/

Joyal-Desmarais, K., Scharmer, A. K., Madzelan, M. K., See, J. V., Rothman, A. J., & Snyder, M. (2022). Appealing to motivation to change attitudes, intentions, and behavior: A systematic review and meta-analysis of 702 experimental tests of the effects of motivational message matching on persuasion. Psychological Bulletin, 148(7–8), 465–517. https://doi.org/10.1037/bul0000377

Kidd, C., & Birhane, A. (2023). How AI can distort human beliefs. Science, 380(6651), 1222–1223. https://doi.org/10.1126/science.adi0248

Kosinski, M., Stillwell, D., & Graepel, T. (2013). Private traits and attributes are predictable from digital records of human behavior. Proceedings of the National Academy of Sciences, 110(15), 5802–5805. https://doi.org/10.1073/pnas.1218772110

Kruk, J., Lubin, J., Sikka, K., Lin, X., Jurafsky, D., & Divakaran, A. (2019). Integrating text and image: Determining multimodal document intent in Instagram posts. ArXiv. https://doi.org/10.48550/arXiv.1904.09073

Lecinski, J. (2014, August 13). ZMOT: Why it matters now more than ever. Think with Google. https://www.thinkwithgoogle.com/marketing-strategies/search/zmot-why-it-matters-now-more-than-ever/

Li, B. Z., Tamkin, A., Goodman, N., & Andreas, J. (2023). Eliciting human preferences with language models. ArXiv. https://doi.org/10.48550/arXiv.2310.11589

Liu, Q., Hou, X., Jin, H., Chen, jin, Wang, Z., Lian, D., Qu, T., Cheng, J., & Lei, J. (2023). Deep group interest modeling of full lifelong user behaviors for CTR prediction. ArXiv. https://doi.org/10.48550/arXiv.2311.10764

Luitse, D., & Denkena, W. (2021). The great transformer: Examining the role of large language models in the political economy of AI. Big Data & Society, 8(2). https://doi.org/10.1177/20539517211047734

Maillot, M. (2024, August 21). Introduction to Teams AI Library—Teams. Microsoft. https://learn.microsoft.com/en-us/microsoftteams/platform/bots/how-to/teams-conversational-ai/teams-conversation-ai-overview

Matz, S. C., Kosinski, M., Nave, G., & Stillwell, D. J. (2017). Psychological targeting as an effective approach to digital mass persuasion. Proceedings of the National Academy of Sciences, 114(48), 12714–12719. https://doi.org/10.1073/pnas.1710966114

Matz, S. C., Teeny, J. D., Vaid, S. S., Peters, H., Harari, G. M., & Cerf, M. (2024). The potential of generative AI for personalized persuasion at scale. Scientific Reports, 14, Article 4692. https://doi.org/10.1038/s41598-024-53755-0

Meredith, W. (2021). The steep cost of capture. Interactions, 28(6), 50–55. https://doi.org/10.1145/3488666

Microsoft. (2022). Msft-10k_20220630.htm. https://www.sec.gov/Archives/edgar/data/789019/000156459022026876/msft-10k_20220630.htm

Narayanan, A., & Kapoor, S. (2024). AI snake oil: What artificial intelligence can do, what it can’t, and how to tell the difference. Princeton University Press.

Newport, C. (2024, March 15). Can an A.I. make plans? The New Yorker. https://www.newyorker.com/science/annals-of-artificial-intelligence/can-an-ai-make-plans

nhciao. (2023, June 5). ControlNet for QR Code [Online forum post]. R/StableDiffusion. Reddit. www.reddit.com/r/StableDiffusion/comments/141hg9x/controlnet_for_qr_code/

NVIDIA. (2023, August 9). NVIDIA keynote at SIGGRAPH 2023 [Video]. YouTube. https://www.youtube.com/watch?v=Z2VBKerS63A

OpenAI. (2023a, November 6). OpenAI DevDay, opening keynote [Video]. YouTube. https://www.youtube.com/watch?v=U9mJuUkhUzk

OpenAI. (2023b, November 9). OpenAI data partnerships. https://openai.com/blog/data-partnerships

OpenAI. (2023c, November 13). The business of AI [Video]. YouTube. https://www.youtube.com/watch?v=knHW-p31R0c

Patel, D., & Xie, M. (2023, November 15). Microsoft infrastructure—AI & CPU custom silicon Maia 100, Athena, Cobalt 100. SemiAnalysis. https://www.semianalysis.com/p/microsoft-infrastructure-ai-and-cpu

Penn, J. (2023). Animo nullius: On AI’s origin story and a data colonial doctrine of discovery. BJHS Themes, 8, 19–34. https://doi.org/10.1017/bjt.2023.14

Ryan, J., & Christl, W. (2023). Europe’s hidden security crisis. Irish Council for Civil Liberties. https://www.iccl.ie/digital-data/europes-hidden-security-crisis/

Salvi, F., Ribeiro, M. H., Gallotti, R., & West, R. (2024). On the conversational persuasiveness of large language models: A randomized controlled trial. ArXiv. https://doi.org/10.48550/arXiv.2403.14380

Searls, D. (2012). The Intention Economy: When Customers Take Charge (1st ed.). Harvard Business Review Press.

Searle, J. R. (1983). Intentionality: An essay in the philosophy of mind. Cambridge University Press.

Shah, C., White, R. W., Andersen, R., Buscher, G., Counts, S., Das, S. S. S., Montazer, A., Manivannan, S., Neville, J., Ni, X., Rangan, N., Safavi, T., Suri, S., Wan, M., Wang, L., & Yang, L. (2024). Using large language models to generate, validate, and apply user intent taxonomies. ArXiv. https://doi.org/10.48550/arXiv.2309.13063

Siddarth, D., Acemoglu, D., Allen, D., Crawford, K., Evans, J., Jordan, M., & Weyl, E. G. (2021). How AI fails us. Harvard University. https://ethics.harvard.edu/files/center-for-ethics/files/aifailsus.jhdcarr_final_2.pdf

Singleton, T., Gerken, T., & McMahon, L. (2023, October 6). How a chatbot encouraged a man who wanted to kill the Queen. BBC News. https://www.bbc.com/news/technology-67012224

Staab, R., Vero, M., Balunović, M., & Vechev, M. (2023). Beyond memorization: Violating privacy via inference with large language models. ArXiv. https://doi.org/10.48550/arXiv.2310.07298

Stark, L., & Hoey, J. (2021). The ethics of emotion in artificial intelligence systems. In FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (pp. 782–793). ACM. https://doi.org/10.1145/3442188.3445939

Stone, M. (2024, April 9). ChatGPT maker OpenAI poaches Shopify’s former head of support as the AI race heats up. Business Insider. https://www.businessinsider.com/openai-hires-former-shopify-executive-glen-worthington-2024-4

Tan, Z., & Jiang, M. (2023). User modeling in the era of large language models: Current research and future directions. ArXiv. https://doi.org/10.48550/arXiv.2312.11518

Veerabadran, V., Goldman, J., Shankar, S., Cheung, B., Papernot, N., Kurakin, A., Goodfellow, I., Shlens, J., Sohl-Dickstein, J., Mozer, M. C., & Elsayed, G. F. (2023). Subtle adversarial image manipulations influence both human and machine perception. Nature Communications, 14(1), Article 1. https://doi.org/10.1038/s41467-023-40499-0

Vicente, L., & Matute, H. (2023). Humans inherit artificial intelligence biases. Scientific Reports, 13(1), Article 1. https://doi.org/10.1038/s41598-023-42384-8

Williams, J. (2018). Stand out of our light: Freedom and resistance in the attention economy. Cambridge University Press. https://doi.org/10.1017/9781108453004

Wu, X., Laufer, E., Li, H., Khomh, F., Srinivasan, S., & Luo, J. (2023). Characterizing and classifying developer forum posts with their intentions. ArXiv. https://doi.org/10.48550/arXiv.2312.14279

Zhang, A., Sheng, L., Chen, Y., Li, H., Deng, Y., Wang, X., & Chua, T.-S. (2023). On generative agents in recommendation. ArXiv. https://doi.org/10.48550/arXiv.2310.10108

Zuboff, S. (2019). The age of surveillance capitalism: The fight for a human future at the new frontier of power. PublicAffairs.

©2024 Yaqub Chaudhary and Jonnie Penn. This article is licensed under a Creative Commons Attribution (CC BY 4.0) International license, except where otherwise indicated with respect to particular material included in the article.