Abstract

This panel was part of the 5th Anniversary Symposium of Harvard Data Science Review—AI and Data Science: Integrating Artificial and Human Ecosystems. The panel explored AI’s impact on society, the intriguing and the scary, as well as the intersection of AI and the public interest. At this moment in history, it’s important for us to understand not just the power and potential of AI but the peril as well. In the face of powerful and disruptive innovation, how do we ensure that society thrives? How does AI perturb our economy? How does it change our legal system? How does it impact our culture? How does AI change our relationships with one another? This text has been edited for readability and length.

Keywords: AI, tech and society, social impact, ethics, markets, inequity

Fran Berman, Moderator: Today’s topic is AI in the human ecosystem, AI in real life. At this moment in history, it’s important for us to understand not just the power and potential of AI, but the peril of AI as well. In the face of powerful and disruptive innovation, what do we need to do to ensure that society thrives? This morning we’ll talk about how AI impacts individuals and society. How does it perturb our economy? How does it change our legal system? How does it impact our culture? How does it change our relationships with one another?

We start with a term we use all the time, responsible AI. I want to ask each of the panelists, what does responsible AI actually mean? Whom are we responsible to? What does it mean to be responsible for AI? How do we know if AI is responsible or not? We’ll start with Martha Minow.

AI and Responsibility

Martha Minow, Panelist: It’s a pleasure to be here. I’ve already learned so very much. I want to start by asking who’s the “we”? When you say ‘who is responsible,’ who, what people, do you mean? I believe that in the next phases of this really transformative set of human discoveries, we need to make sure that the human remains critically involved at every single stage: whether it is in the stage of identifying the data sets that are used for training purposes or the stages of designing and selecting any particular learning algorithm or generative AI system. Who is consulted, who participates in judgment calls? There is already some attention to diversity in those that have access to using AI, but what about access to the resources even to participate in the discussions about purposes and deployment? And access to compute power?

There will be many consequences of this transformation. Just as the creation of the printing press and moveable type profoundly disrupted and altered social, political, and economic arrangements, with AI, there will be effects we cannot yet foretell. Even if we limit ourselves to discussion of social media, we need to consider the impact of social media on social mores, how people talk to each other, the degree to which people actually engage in or respect previously well-settled norms of decency, the degree to which people trust what they're reading. Were those effects anticipated in the design, and deployment of social media?

Michael Jordan, Panelist: I think it’s important to start with the realization that the phrase ‘AI’ is currently being used by different people in very different ways, and this can, and does, lead to strange dialogue in which the meaning of the basic term fluctuates throughout a conversation.

I’ve been working on the underlying technology for 40 years, using the phrase ‘machine learning’ (ML) rather than ‘AI,’ and in particular, I developed some of the variational machinery behind so-called generative AI tools. In doing this work, I was aiming at foundational statistical issues arising in problems throughout engineering, including problems in operations research and computational biology.

The goal was to assist in the design of intelligent systems, not to develop some kind of standalone intelligence. My colleagues and I didn’t see ourselves as allied with the Frankenstein-like goal of mimicking or replacing human intelligence that arises in the classical definition of ‘AI.’ The algorithms that we have worked with are simple gradient-based algorithms that are familiar in many areas of engineering. They’re now being used at very large scale, which has triggered much of the current dialogue, but even that development is not so recent, and it occurred historically without the phrase ‘artificial intelligence’ being invoked.

Back in the early 2000s, companies like Amazon or Alibaba needed to solve very large supply-chain problems involving very large collections of data. These supply chains are exceedingly complex.

Nowadays they involve a billion products and hundreds of millions of people, connecting them within one day. This was all done in 2000 with random forests, which were the neural nets of the day; they are gradient-based algorithms with large numbers of parameters, and they are obscure black boxes.

At Amazon, it was soon realized that they couldn’t do this kind of thing on one machine, given the scale of the data. Every one of those billion products had to be put together from various pieces and shipped on various routes. So they built a cloud. Eventually this cloud became Amazon Web Services (AWS) and it offered a wide range of services, but it is of interest in the current dialogue to realize that the origin of the modern cloud was to run ML algorithms for supply chains and logistics.

In 2000, I learned about this and found it fascinating, so I went to Amazon and looked into the phenomenon. It was amazing because the ML system was creating brand new, creative solutions to logistics problems that you’d never seen in the textbooks. In the next phase of development, it wasn’t just supply chains; they began to use transactional data, and the resulting ML system was the basis for the recommendation system. So now people’s data are coming into play.

In a third phase, the data has changed yet again. Now it’s human speech and language. The system again makes very good predictions, but because the data are speech and language, it’s no longer just a large-scale distributed stochastic control system that no one in The New York Times would write about. It seems artificially human, and it’s driven AI hype. It’s not on the way to an artificial human, in my humble opinion. Of course, one can argue with that, but what’s more interesting to me is that what has been achieved is to create a planetary-scale system that takes in data involving many hundreds of millions of people and that makes decisions that arguably improve human welfare. It’s an engineering system that includes humans in the loop at vast planetary scale. That’s exciting.

Our focus needs to be on the problem-solving abilities of the overall system. Indeed, it’s often more useful to think about that system, including the humans that are part of the system, than to focus on the specter of an ‘artificially intelligent agent’ that is plopped into existing infrastructure to solve problems for us.

Also, the problems that are emerging include human-centric issues, but they are not just the ethical problems that one might associate with the classical AI agenda of creating a substitute for humans. Rather, they are the economic and engineering problems associated with a large-scale system that includes humans. This includes issues of fairness and privacy. Moreover, it includes issues of safety in a traditional sense—the system should work, and work for everyone. The logistics chain should not break down and it should be adaptive and robust—in a word, it should be ‘intelligent.’ It should work in novel situations like a pandemic.

Overall, it’s best thought of as a new way to build systems that augment human intelligence and create new economic conditions. Not as a superhuman intelligence. It’s akin to the development of engineering fields in the past—it takes 20, 30 years to develop into a mature state. The development cannot be achieved by a small set of companies, but rather it requires all kinds of entities and all kinds of human beings. We all need to get involved. We roll up our sleeves, we think about it, we talk.

Berman: Thanks, Michael! Sabina, what about responsible AI?

Sabina Leonelli, Panelist: Often, responsible AI is equated with asking questions such as, how do we make sure that AI solutions are accessible and usable by people that need them? But the primary question, in my view, is, who sets the problems that AI algorithms are meant to solve? Access and usability are certainly crucial components of responsible AI, but the bigger problem concerns who is involved in—or excluded from—designing these systems, and how regulation and governance become effective instruments to guide which ingredients, goals, and values are used to develop, maintain, and update AI tools.

Related questions include: How do we actually calibrate the models? What kind of empirical parameters are we using? What are the problems to which we need solutions? Those areas are where we are missing the mark in a lot of current AI applications, partly because of the way this work is distributed across industry and the public sector, and partly because of the way power is distributed in society. AI development is not typically done in a participative format. Higher levels of transnational collaboration are needed on which we can engender more participation in constructing the algorithms. It is extremely difficult to do, and it does require a rethink about what responsibility in AI entails, and for whom.

David Banks, Panelist: There are many different aspects of responsible AI and we need to map those out carefully. It goes back to the alignment problem, in which one is concerned about whether or not an AI aligns with human values. And that’s hard to study because human beings don’t align with each other’s values. One strategy is to try to come up with measures of alignment between people first, and then between people and AI systems second. If you want to get to responsible AI, then one starting point might be Asimov’s Three Laws of Robotics,1 but there are subtle perils in that approach that I don’t want to unpack now.

AI and Trust

Berman: I love that you brought up Isaac Asimov in the first 15 minutes of this panel! Responsibility and trust are terms that we often use together. I believe that one of the biggest untapped challenges, if not the biggest untapped challenge, around AI is trust. Did the pope really wear a puffer coat? Did Drake and The Weeknd really release a song on TikTok? Was it really them? It’s hard to manage something that you don’t trust. And that’s especially true in all the scenarios in which AI will make decisions where a bad result would be catastrophic. What do we need to do to create AI that we can trust at least as much as we trust humans? What will it take? We don’t trust every human being, but we do have some sort of context that helps us calibrate how trustworthy a human being’s response might be. How are we going to do this in AI land?

Jordan: It’s important to realize that the treatment of uncertainty is quite inadequate in current large language models. If you say ‘ChatGPT, you just told me something, how sure are you about what you said?’ It might say ‘I’m very sure.’ It does that not because it’s reasoning under uncertainty, but rather because in its training data a similar question was asked in the past and some human said, ‘I’m sure.’ But that human was thinking in the moment and managing their uncertainty. ChatGPT is not doing that at all, right? That’s problematic.

How can you trust something that doesn’t have any idea of how unsure or sure it is? Indeed, if we are aiming at human-level intelligence, this is one sense in which we’re far off the mark. Humans are pretty good in coping with uncertainty, and, critically, it’s not an individual human which is so great with uncertainty, it’s the collective. We manage uncertainty well because we’re in a collective. We manage uncertainty well when we can interact and jointly reduce uncertainty. We can share bits of knowledge and share decisions about collecting more data. We do this by indicating our uncertainty meaningfully. And the current AI systems are not able to talk this language of uncertainty. This may seem shocking, but it really is true. Now it’s not an unsolvable problem, but it’s going to take a while.

We have ideas about how to do it. But even with our best AI systems, like AlphaFold, are not well calibrated and in particular they can give us quite wrong outputs to questions on the edge of knowledge. We published a paper recently that used AlphaFold precisely in this setting, asking questions that scientists really ask, not about the old data, but about new things. And we calculated error bars based on the AlphaFold output, finding that they were completely wrong. This is despite the fact that the system is accurate overall. Now, one can get better, and have more calibrated error bars, but it requires some additional human input.

So, we have to roll up our sleeves. You don’t develop trust by just wishing it into existence. You have got to be an engineer and a mathematician. But there’s more: You need to realize that humans trust other entities for various kinds of reasons, and many of them are social and economic. We all go into markets, which are environments in which trust is not a given. But mechanisms have arisen that allow us to navigate our interactions in such collectives, and we understand those mechanisms. Current AI systems do not, and that is another part of the uncertainty problem. It calls for economic thinking to be brought to bear in addition to statistical thinking.

Minow: I actually think that trust is fundamental, and the most important question here because when lost, trust is very hard to build. And doing so will take time. There may be a loss of trust in any of the enterprises that are using anything that we call AI. I spent much of this past year working with a group of mainly scientists organized by the National Academies of Sciences, Medicine, and Engineering, to produce a short statement now out in PNAS; it is a short, proposal for dealing with integrity in science itself (Blau et al., 2024).

When scientists use AI, including learning algorithms, supervised or not, there are three fundamental areas of scientific validation—verifying findings and interpretations—that are in jeopardy: One is access to data because the private companies at the forefront of AI treat what they develop as mostly proprietary, held in the hands of a few companies. The second is replicability. We all know so far with most generative AI that you can’t even replicate your own result—a few have a temperature parameter that controls randomness in responses, but this is rarely used (Ball, 2023). So promoting a generative response to replicate what someone else triggered generally introduces as much or more variability. And the third dimension at risk is peer review. And I’ll just say a word about that. There’s a wonderful book by Jonathan Rauch called The Constitution of Knowledge that takes the issue of trust across different social systems (Reich, 2021).

Across each domain of human knowledge—journalism and news, my own field of law, banking, science—in each area, intelligence is a collective enterprise. Trust is a collective enterprise. In each of these areas, there’s something that looks like peer review. There are communities that constitute standards, measures of error, language, and then systems to check and calibrate. And that confidence is absolutely in jeopardy right now. On science, we produced a statement that says developers of models and scientists who use the models need to be explicit about what they’re asking, what they’re using, what they don’t know for what purposes. And right now, none of that is happening.

Leonelli: Another concern is how tightly the notion of trust is coupled with the notion of expertise. We trust something because we think that there are reasons to trust, because we have cultivated that trust in a variety of different ways, and that often takes the form of expertise (in the sense of systematic, well-informed modes of intervention). But then the question becomes ‘What are the boundaries of expertise?’ Whether we are looking at scientific research or much wider uses of AI, we need to recognize the boundedness of certain forms of expertise.

It isn’t going to be the engineer or politician or ethicist who is going to provide grounds for trusting new technologies or new uses of technology. Trust typically stems from a collection of different perspectives which, when considered together, constitutes expertise one needs to confront a particular use of technology. That’s the kind of participatory element that I think we are missing in the design of trustworthy technologies. We may not understand how AI works, which is anyhow unavoidable for most of society when it comes to such specialized and intrinsically opaque systems. As much as AI literacy is important, very few people will ever reach the level of technical expertise required to shape AI systems from the inside or even understand the ways in which deep learning works (which cannot be fully articulated by humans anyhow). However, not understanding who was involved in designing AI and with which objectives constitutes a real problem, because there is no reason for such lack of transparency, and its consequences are severe.

Lack of explicit acknowledgment and reward obscures the very real forms of expertise infused by humans within AI systems—the multiple contributions coming from all sectors of society that make AI effective and well-adjusted for use within specific domains—whether these are data provided by social media users to boost communications within their communities, models developed by biomedical researchers to optimize diagnostic processes, or algorithms devised by energy companies seeking to heighten productivity. Reforming the ways in which we design, describe, and acknowledge the wide-ranging nature of AI tools and related contributions would go a long way toward reinforcing the trustworthiness and reliability of the resulting decision-making systems.

Berman: Interestingly, I think all three of you are talking about the fact that trust has a lot to do with context. Michael was talking about error bars. If the error bars are small, we are more likely to trust the results. Martha in some sense was talking about transparency. If we know a lot about where things came from or from whom, we have a better sense of what to trust. A lot of times we’re working with AI ‘in the wild’ and it just comes up with something. I’ve used ChatGPT and asked it for citations. The citations are frequently not useful and often incorrect, does that make me trust it more? No. What do you think, David?

Banks: I think the immediate threat from generative AI is the ability to easily facilitate deepfakes, disinformation, identity theft, and cybercrime. But the silver lining to all of this is that if everybody becomes convinced that deepfakes are very, very easy to manufacture, then perhaps everyone will become a lot more conscientious about curating their information sources. And I think that's a net social positive.

Cultural Disparities

Berman: In some sense, society is made up of a lot of different publics—people with different priorities from different cultures. One of the things that I think, Sabina, that you work on a great deal and Martha has mentioned as well is: What about the cultural disparities in the use of AI? Sabina, can you talk a little bit about what our challenges are and how we might address them in terms of AI advantaging or disadvantaging some cultural groups and not others?

Leonelli: AI is intrinsically political, since it is so widely used as a tool for decision-making. As such, it is reasonable to demand that AI systems reflect democratic politics with elements of representation from different parts of society. This is important not just because of ethics, but because it shapes how we use and trust AI.

Consider the use of AI in agriculture, which starts to be used systematically to determine the choice of crop varieties that may fit different kinds of environments, particularly given climate change. There may be a lot of involvement by local populations and interested stakeholders in collecting data of relevance to that process, and some companies, such as the Climate Corporation and Digital Green, are trying to use this information. However, as the data get elaborated, enriched, and structured, and then adopted to train particular AI systems, only some of those data are privileged as input for AI systems to inform decisions on future agricultural policies. And those choices of data sources tend to be based on the tractability of the data or the ease with which they fit existing systems of analysis—for instance, genetic or metabolomic data may be easier to handle than qualitative data on the preferred diets or cultural norms of local communities.

So, we have a situation where data sources and models for agricultural decision-making AI are selected on the basis of what best fits existing algorithms, rather than considerations around which knowledge needs to be represented and optimized in these systems. These considerations often result in the exclusion of local knowledge (e.g., from breeders and others working with local markets to know, e.g., how the food will be cooked) and agroecological concerns around the long-term sustainability of specific crop economies.

This is an example of how data filtering processes underpinning AI systems can drop crucial information needed for training the algorithm. In practice, this results in the creation of a decision-making system for which varieties can be used to feed the world, but which excludes 99% of the people who actually have expertise in that domain—and typically to the detriment of the most vulnerable among interested parties, such as local farmers and consumers. I think this is an example of the confluence of the different problems one has in trying to set up a system that’s responsive to democratic input as well as expert participation.

Berman: I have a bit of a follow-up for Martha. When you talk about cultural groups and disparities, that also brings us to the discussion of fairness and equity. You and Cynthia Dwork have been teaching a course on fairness and privacy. Will law save us here? Will math save us? Will anything save us?

Minow: There is no magic wand, and there’s no savior. It’s just all of us. It’s just people. I will take the example of fairness and unfairness in the legal system in the United States. There are just such massive inequalities across the board. For example, let’s focus on the access to justice, the legal system, access to court—and even a subset of that, consider civil litigation affecting low-income individuals, landlord-tenant, debt, family law matters. In some 80–90% of the cases that actually come to state court, at least one party doesn’t have a lawyer.

One reason I’m so interested in this whole field is that AI could offer real solutions. You could prompt generative AI to produce a response to the notice that you have to leave your apartment and not be evicted that is comprehensible to someone with a fifth-grade reading level. This could be fantastic. We will never provide enough lawyers for the people that don’t have one. This could be a tool.

Last year, one or two million dollars were spent on the development of tools in this area for people who don't have the resources. Compare that to the multiple billions spent to assist businesses, banks, creditors with AI, exacerbating the imbalances that have long made too many courts places where justice is not done. Right now, the biggest user of generative AI tools are large corporations. And the state courts are now flooded with cases that are generated by generative AI by creditors, not by debtors; the cost of bringing cases to court is vastly reduced with generative AI tools.

Staying with the law but leaving the court, consider how the federal government creates regulations by giving notice of proposed rules and soliciting comment on the proposed regulations. Right now, the notice and comments processes used by federal agencies are being flooded by comments that generative AI produced. I mean, before it was robocalls, but now it can't even be distinguished. Are these real? Are they not real? And you have bureaucrats, the civil servants who don't have the time or resources to get through them all. They need algorithms to go through all the comments. What about the people without the backing of the business communities? Their voices are not heard.

AI and the Marketplace

Jordan: Wikipedia. I want to think about that one for a minute. It’s one of the most successful artifacts arising from the Internet age, and it’s a great example of our collective intelligence. It’s now all being ingested inside of the large language models (LLMs), and that is a major change. In particular, the producer/consumer relationship that was just below the surface in Wikipedia is now completely lost.

Knowledge has become a black box. The individual who contributed is not remunerated when the company who builds the LLM makes money. There is no longer a sense of responsibility. We lose the producer/consumer relationship that is the glimmering of a market. Bottom-up markets have been critical drivers of human progress and human happiness, and it is hard for me to imagine proceeding without these driving forces. Big companies making money off of advertising and sales to other big companies doesn’t replace the social welfare created by the collective. They damage the incentives that motivated people to contribute. To me, this is not just unethical, but it’s economically damaging. And the talk of fixing it by offering ‘tips’ to producers doesn’t address the real problem.

Part of the real problem is that our current expectation for technology is that it all has to be free. But nothing is free, and the illusion of having something be free obscures the fundamental loss of market efficiency and social welfare. In case it’s not already clear, I believe that it is essential to bring microeconomic thinking, and actual economists, into the dialogue. It’s strange to me that discussions about AI mostly involve computer sciences and legal scholars, and perhaps social scientists, with nary an economist in sight. You know, economists have been talking for a thousand years on how to create markets, how to connect people. And they’ll be helpful in discussions of the new kinds of markets that can emerge in this wave of technology.

Let me give an example of such a market. Music is a cultural good and ideally an economic good. But young musicians aren’t making any money. There’s no market. They put their stuff out there, on websites that stream music for free to consumers. The websites make money, in ways that are only indirectly related to the producer/consumer relationship. I’ve been involved in a company called United Masters that has aimed to help solve this problem. It uses machine learning algorithms to create and sustain a three-way market. On one leg are the musicians and on another leg are the listeners.

Critically, also, there are brands in the market. There is transparency between all of the entities. A brand streams music to support a product, a specific musician is used, in part because of the demographic associated with that product and that kind of music, and other brands watch this interaction. Musicians get paid when their songs are used, and brands pay musicians to write new songs. This is all up and running, with over three million (mostly young) musicians signed up. This is an intelligent system that creates opportunities. It’s adaptive and intelligent. So, rather than replacing musicians with artificial musicians, we provide an adaptive, collective system that empowers young people to bring their musical culture to a market. The AI here is the connectivity.

This is so different from the current Silicon Valley AI mentality. And for those outside of Silicon Valley, it provides an alternative that is different from just complaining or discussing legal remedies. I have no problem with legal remedies, and I believe the regulatory mechanisms will eventually be needed, but the law is unfortunately very impoverished with respect to many of the economic and market-creation issues at play. In particular, copyright law is not enough. For example, the journalists are likely going to lose their case against the AI companies because the copyright law is not strong enough.

We need a whole new infrastructure, one than includes provenance mechanisms and mechanisms that provide data-adaptive markets. Neither is impossible, and indeed provenance mechanisms were essential in developing an online banking industry. The health care industry requires provenance writ large. Provenance helps to create markets because it creates the possibility of new producer/consumer relationships. We need a new kind of microeconomics that embraces data flows that connect individual humans.

AI in the Wild

Berman: Let’s think about AI in the wild and its applications. Panelists, what are your favorite and perhaps least favorite applications of AI? Where do you think AI could really make a difference? And David, I really want you to talk about self-driving cars.

Banks: Certainly! Self-driving cars strike me as one of the potentially huge benefits from AI systems. Obviously, an autonomous vehicle is not exactly generative AI in the way we’ve been discussing, but it has many positive features. According to the EPA, about 42% of human-produced greenhouse gases in the United States come from the transportation sector. So electric autonomous vehicles have the potential for huge reductions in pollution. Presumably, if all the cars are networked and sufficiently safe, then you can build car bodies out of canvas instead of having to carry a ton and a half of steel around to protect you from other people. Similarly, if there are such networked vehicles, then one rarely needs to brake. Braking costs an enormous amount of fuel, and the networked vehicles can just interleave through intersections, with little energy loss without braking.

There are other societal impacts. The main reason elderly people move into assisted living facilities is that they can no longer drive to the grocery store. Autonomous vehicles (AVs) would solve that. Also, one can put seven times as many cars on the road if they are networked and autonomous, which solves congestion for the foreseeable future. I’m not saying that AVs are going to be an answer to everything, but they’re one of the few technologies on the horizon that has the potential to make a meaningful impact on climate change. And so that’s something that I think we have to explore.

Berman: David, let me push back on that a little bit. There was a recent study from some folks at MIT talking about the amount of energy it will take to run the autonomous computation in self-driving cars. Assume it’s a few decades from now and they're ubiquitous on the road. We won't need stop signs because they don’t read stop signs, they will just get invisible signals from digital transportation networks that their algorithms will use to platoon, so we don’t need today’s infrastructure, but we will need computational and communication infrastructure. But what about all the energy they’re going to be using?

Banks: That is an engineering problem, and there are various ways address it. Battery technology has made huge advances. We're now developing chlorine batteries, which is far more plentiful than lithium. I am optimistic that smart manufacturing and the use of renewable energy resources can find workable solutions. I agree, you raise an open question about net energy use. But we would be remiss not to try to make safer, more efficient autonomous vehicles.

I’d like to jump back a bit and also talk about some of the other things that AI systems can do. We were talking about disparities a moment ago. AIs have, I think, on average, raised the playing field for all of us, although more for some than others. It is now possible for everybody in the world to write at the level of the college sophomore.

Some of the most brilliant minds in my field don't speak English as a first language, and that has been a handicap to many careers. AI doesn't solve that problem entirely, and it's reduced the burden. I'm an occasional editor. I received a paper 18 months ago in which the math looked right, but the English was so poor that I could not even send it out for review. I wrote back to the author and asked him to please work with a native English speaker to fix the grammar. I got the paper back the next day in perfect English. He'd run it through GPT-4.

Berman: Sabina, what’s your favorite AI application these days?

Leonelli: I would pick two things—one is medical diagnostics and the other is language translation. They both have amazing advantages. We’re all aware of the fact that AI-assisted medical diagnostics are progressing at an incredible speed. In Europe we can think about national health services for everybody at a lower cost. We’re also seeing applications in diagnostics for the environment, such as improvement in environmental detection of pollutants, and in precision toxicology.

In terms of translation, the fact that we start to have intercultural dialogue in a way that's strongly facilitated by AI is fantastic. At the same time, those advantages are also examples of the expansive gulf that is opening between people who can take advantage of this and people who can’t. Not all languages are being curated for automated translation, and not all to the same level of detail and accuracy. It is a small number compared to the richness of languages in the world. The few that are being curated correspond to the richer sectors of society and the richer countries.

Similarly, for medical diagnostics, we are seeing so much investment on being able to acquire and analyze expensive data produced by cutting-edge technologies, such as MRI scans, but much less investment on data that is more easily and cheaply procured and processed. We also need to complement AI diagnostics with human expertise that allows doctors to interpret those results in light of the needs and family history of the patients in front of them. Without investment in low-bar, widely accessible technologies and relevant know-how for interpreting AI results, those outputs risk becoming decontextualized in a way that can be damaging to the people who rely on such tools to inform their everyday lives, especially in low-cost environments. But the promise is there. The question is where the investment is going to go.

Berman: Martha, what’s your favorite AI application?

Minow: Well, besides medical applications, I am most hopeful about prospects for education. The disparities in educational opportunities are devastating and costly not only for individuals, but also for societies. All children are born with the capacity to become well-educated, develop their talents, and contribute back to society. AI could close the gaps, the gulfs, and create opportunities.

Already, the Khan Academy has had breakthroughs developing and making accessible powerful and effective instruction for all kinds of people. Recently, the team there developed what’s called Khanamigo, which are AI powered, personalized tutors, free for any person (see Kahn, 2024). That’s a breakthrough. So it doesn’t matter if you live in a poor school district that is funded on real estate taxes, reflecting the disparate property values in different neighborhoods (because the United States is bizarre, and that’s what we do). Even if you do not have the right equipment or broadband, you can have access to this if you go to your local library, and maybe it will be distributed even more broadly into people’s homes.

Another example: the partnership between Sesame Workshop and the International Rescue Committee developed, for the first time, educational programs for people three years and younger who are refugees. During the pandemic, the effort had to pivot entirely to digital instruction. And lo and behold, every one of the refugees—the children through adults—had some access to a smartphone. And as a result, the partnership was able to deliver materials to the children and their parents, and they catapulted in 10 weeks a year’s worth of instruction (International Rescue Committee, 2020). And because of this breakthrough, it's now possible to deliver this kind of service for refugees who are on the move, who are not in a single location. So it’s transformative.

Audience Questions

Berman: These are all great applications. We’re going to open it up to the audience now. I see a lot of hands!

Audience Member: Great panel! Thank you. So as I listen to this, I think things mostly divide into two categories: One is: Can we make these algorithmic systems do what we want them to do? There’s where we have hallucinations, and the references aren’t very good today, and they can’t tell us whether they’re making the right decisions or not. They are not trustworthy in the sense that we don’t even know if they will do what we ask them to do. So that’s a technical issue. And many of us are probably optimistic: we will grind away at this and make this better and better and better over time.

The second issue comes up when we talk about equity or we talk about privacy or we talk about small languages or something like that, even self-driving cars. The question is, what are the objectives that we want to reach? And this particularly comes up when we talk about the influence computers have on people, which comes up again on the social network construct. Similar things exist when we look at education. What are we trying to do? Consider admissions at Bronx Science, where we have one set of policies in a very liberal city versus, say, Boston Latin, where we have a different set of policies. I hearken back to what Stephanie Dick said in her first talk. And that is that all of the data, all of the algorithms just highlight that we as humans don't really agree on what we're going to be thinking about. I think that's the ultimate challenge that we have.

Berman: I think this is something that comes up in ethics as well. If we all can’t agree as humans, how are we going to add AI to the mix? Michael?

Jordan: I don’t think we can necessarily agree. And that’s part of being a human being. While there are some absolute standards, life is mostly contextual, and what I want to do is empower individual people to be able to operate in ever-more-complex contexts more effectively. I don’t want the IT companies of the world to take over and figure things out for us. You know, back when they provided a search engine, I was happy with Google. I’m no longer happy with Google. I think they and other IT companies, especially those focused on LLMs, are putting all of our knowledge and all of our culture into black boxes that they own, and then aiming to supply services and make certain guarantees on their terms.

I don’t want them to make such decisions for me. For example, privacy is something that I want control of. I want to be empowered. Privacy should be a personally managed tradeoff. It should be something I possess since it’s my own economic decision. I want to turn my own differential privacy knobs. If you want my medical data, or my genome, I'm not going to give it to you if you just ask. I'm not going to give it to Google. But if you're a doctor working on a disease that runs in my family, I may want to give some fraction of my data to you. I'm going to set my privacy knob according to all the contextual factors that come into my momentary decision.

There's no way a big company can know about my momentary context and my decision and my desire and my utility functions. This is how humans have interacted throughout history. We don't agree about the utilities. We have Pareto frontiers—effective solutions not dominated by other solutions—we are willing to consider. We have ways of interacting, nonetheless, that are not always perfect. And our system has got to integrate into that fabric. They don't get to decide and they’re not going to let others decide. Academic discussions often focus on whether it's the computer scientists or the government that decide. No, it's each individual. I want technology to help bring us up to Pareto frontiers. I want to empower humans. I don’t want AI to take over.

Audience Member: My question is surrounding the trust of AI systems. As we are already seeing, many of these bigger foundational models are getting harder and harder, and more resource intensive to train by individual entities, right? We have to place some trust on the organizations who have the resources to train some of these models. What are your opinions on how public trust should be dealt with around these models, models trained by inherently self-interested organizations who are going to be using them presumably to promote their own interests. How does public trust revolve around the inherently self-interested?

Jordan: But that’s not a bad thing, right? That every one of us is inherently self-interested—that’s not necessarily bad. It’s whether or not there is transparency and whether or not it’s possible to have market entry. Right now, OpenAI has been a leader, but I don’t see a monopoly arising—it’s not impossible for others to build these systems, given that the algorithmic machinery is common knowledge and the hardware is commodity-based. There will be the possibility of consumer choice. A company that is not trustworthy will not prevail in the long run.

Berman: I’m going to push back on that because I think that it's fine to be self-interested if you're just trying to innovate. You’ve got a new idea and you can commercialize that. You may make a ton of money. It’s a great thing. It’s an entirely different thing when that innovation—that product or service—is used at scale and becomes critical infrastructure. We expect different things from critical infrastructure. We expect essential goods and services to protect people. We expect them to be resilient. We expect that when there is a problem, they're not going to kill you, that it's not going to be catastrophic. And a lot of services and apps in cyberspace now have become critical infrastructure. We can't function without them. They’re essential. What happens to those private interests then? They’re fine. That’s how we got here. But now we have to be thinking about public protection, risk remediation and regulation.

Jordan: We have to have regulation. But think about the search engine. It’s critical infrastructure. It’s changed all of our lives for the best. But it didn't need to be regulated, for at least most of its lifetime. It’s not clear that regulation is the best way to deal with the emerging issues surrounding this technology, certainly not at the detailed input/output level. What needs to be regulated is the transparency.

Minow: You mentioned economics. I mean, one of the failures here involves policies known as antitrust. So, if you come to social media right now, we have a bundling of all of the elements from the server all the way through to the user interface. You could legally require unbundling so that the curation function is a competitive activity, not run by the same few companies. If we had competition, then Michael could actually pick who’s his curator—for content, even for training materials—and then you could have a virtuous circle. Markets, if they are set up right, can achieve and reflect different values and different purposes. But right now, we don't have that. We have a failure of monitoring the concentration of economic power.

Jordan: Indeed, and the business model is all advertising-based, which doesn’t promote differentiation. Facebook made its advertising money, and then bought Instagram, and so on, not allowing a market to develop. It’s a broken economic model. I personally would break up Facebook. On the other hand, I wouldn’t break up Amazon, say, because I think that there’s a meaningful producer/consumer business model behind it. It brings packages to people’s doors.

Leonelli: Just to add that I don’t think we should be building AI systems that are predicated on eventually reaching some sort of social equity situation or having a homogeneous society. Neither of those things are going to happen. Diversity is part of human culture, thank goodness—we want to foster that. And social inequities will always exist, power differentials will continue to exist.

So, arguably AI systems should be built in a way that actually addresses this to start with, rather than building on the assumption that those issues will eventually disappear. We need to build AI with issues of inequity and diversity very much at the core. And I don’t think it is what is happening right now. We’re going in exactly the opposite direction. We’re just seconding existing power differentials in a way that has no affirmative elements in it. And we are working toward the idea of consensus and homogeneity, which really is not helping us to bring in questions of dissent and diversity, which are so important in society.

Minow: Critical infrastructure is often managed and regulated as a utility, which is another way of thinking about some of these issues.

Audience Member: From what I understand, LLMs are not going to lead us to artificial general intelligence (AGI), but could be a part of the chain that could lead to the AGI. So, like, how far are we? What if there is a breakthrough in quantum computing? How would it change the playground of the AGI?

Jordan: The concept of AGI is very ill-defined and, in my view, it is not helpful. It is a Silicon Valley buzzword. It’s still very human-centric. As I’ve noted, a market, especially at large scale, can be adaptive, robust, scalable and, thus, by any definition, intelligent. It’s at the collective level that the intelligence arises, and it does so whether or not the individuals making up the market are particularly intelligent. Somehow this isn’t part of the vision for AGI, and that’s a problem in my view. We really need distributed, human-level mechanisms that allocate goods and services more efficiently and more fairly, rather than imagining super-intelligences that somehow figure how the allocations should go.

Berman: And they can’t know what it’s like to be human. As Stuart Russell points out (Russell, 2023), AI doesn’t know what it’s like to fall in love or to be disappointed or to be sad. AI can produce an outcome that simulates human responses, but it doesn’t get there the same way, it doesn’t know how things feel.

Audience Member: Hello. I’m currently a student in design, so, I definitely think I’ve chosen the wrong major! One of the previous speakers mentioned how historically we wanted to decentralize the power from monarchy to more democracy. So do you think we have strayed away from those original values in this day and age, where only certain corporations have access to all of the data? Is it perpetuating the already existing power imbalance in the society? I’d like to hear your thoughts on that.

Berman: Sabina, I think you’ve written a lot about that, haven’t you?

Leonelli: There is path dependence in what has happened so far, with corporate powers indeed replacing the power historically retained by monarchies and governments, and behaving today as the stewards to our data landscape—but without a democratic mandate aside from people using their services. This is something we often cannot avoid in any case to be able to live in contemporary society (a trend strongly reinforced during the pandemic, where being able to access Facebook for social contacts, Uber for food and transport, and Amazon for shopping was further entrenched as a norm). But that doesn’t mean this dominance by Big Tech needs to continue in the future, nor that it should go unchecked.

A good example to think about is the food system. It’s an incredibly complex system that we have now, not just the market for it, but the way in which food is produced, the way in which we are cultivating seed, in which it is distributed across the world, put together in all sorts of different ways. You would think it would be impossible to trace all this and exercise some form of social control over how it is done and whether it can be trusted to provide consumers with safe food. One could object that it is just too distributed and there’s too many people involved. Yet, this system needs to be regulated in ways we can track, and while provenance is extremely complex, we’ve achieved it to a large extent: food provenance is traceable and there are several safeguards in place to verify compliance with safety and quality standards.

So I don’t see in principle the reason why this shouldn’t be happening for AI in the ways we’ve been discussing. I absolutely agree with you that at the moment what this technology is doing is path- dependent on a long history of inequity and concentration of power in few hands, expanding existing disparities in a way that’s actually difficult to even contemplate. But this can be challenged in a variety of ways, it will require active intervention from different parts of society, but I think it is both feasible and necessary.

Jordan: The phrase AI came from John McCarthy, (“Dartmouth Workshop,” 2024) and he was sort of a philosopher and computer scientist. He asked what would it mean to put thought in a computer. It’s a fantastically interesting question. It’s inspired people. It hasn’t happened, and technology didn’t really need it. And whether technology actually needs thought in a computer is not clear, because we’re not clear about the overall systems that computers will be participating in. We need a broader engineering perspective that goes beyond the singleton intelligence. In the design of systems for collectives, we have really got to think from the very beginning about the goals and the impact on people, including the direct economic impact. McCarthy’s vision is just absolutely inadequate to that task.

Berman: I want to thank all the panelists, Xiao-Li Meng, and HDSR for this outstanding discussion. If we think about our future in which both AI and humans have agency and influence, there is much we need to consider and do—every one of us—to ensure that society will thrive. We invite you to continue thinking about these issues and working in your own lives to ensure that the ways in which we develop and incorporate AI and manage its risks helps individuals, communities, and society to flourish.

Disclosure Statement

Francine Berman, David Banks, Michael I. Jordan, Sabina Leonelli, and Martha Minow have no financial or non-financial disclosures to share for this interview.

References

Ball, P. (2023, December 5). Is AI leading to a replicability crisis in science? Nature. https://doi.org/10.1038/d41586-023-03817-6

Blau, W., Cerf, V. G., Enriquez, J., & Witherell, M. (2024). Protecting scientific integrity in an age of generative AI. PNAS, 121(22), Article e2407886121. https://doi.org/10.1073/pnas.2407886121

Dartmouth Workshop. (2024, August‎ 16). In Wikipedia. https://en.wikipedia.org/w/index.php?title=Dartmouth_workshop&oldid=1240707438

International Rescue Committee. (2020, May 1). How the IRC and Sesame Workshop continue to reach refugee children during Covid-19. https://www.rescue.org/article/how-irc-and-sesame-workshop-continue-reach-refugee-children-during-covid-19

Khan, S. (2024). Brave new worlds: How AI will revolutionize education and why that’s a good thing. Khanamigo. https://www.khanmigo.ai/

Raich, J. (2021). The constitution of knowledge: A defense of truth. Brookings Institution.

Russell, S. (2023, April 3), Beyond ChatGPT: Stuart Russell on the risks and rewards of AI. Commonwealth Club World Affairs. https://www.commonwealthclub.org/events/2023-04-03/beyond-chatgpt-stuart-russell-risks-and-rewards-ai

©2024 Francine Berman, David Banks, Michael I. Jordan, Sabina Leonelli, and Martha Minow. This article is licensed under a Creative Commons Attribution (CC BY 4.0) International license, except where otherwise indicated with respect to particular material included in the article.