Why AI Needs a “Genie Coefficient”

The Problem of Intent in AI Requests

Most of the time, we bridge the gap between one person's request and another's understanding using general knowledge. However, this approach is not foolproof, and AI agents can misinterpret requests in ways that are not immediately apparent. For example, if you ask a friend to get you coffee, they might pour a cup from the pot or buy one from a coffee shop. However, they might also bring you a bag of raw beans or snatch a cup from a stranger and hand it to you. You never specified any of this, and it's only because of general knowledge that your friend knows what you mean.

The Limitations of Specifying Tasks, Questions, and Intent

One might think that the fix is just to specify tasks, questions, and intent better. However, as Terry Winograd and Fernando Flores succinctly captured in their seminal book on AI in 1987, this won't work. They provided an example of a conversation between a user and an AI system, where the user asks if there is any water in the refrigerator, and the AI system responds that there is yes, but the user can't see it because it's in the cells of the eggplant. This example highlights the problem of intent in AI requests, where the user's intent is not explicitly stated, and the AI system must make an educated guess.

The Pragmatics of Human Communication

Linguists call this pragmatics: meaning lies in the words and the situation and also in all prior communication, shared culture, and innate human behavior. It doesn't always work out, of course. Your friend might bring you a hot coffee when you wanted an iced coffee, or an Italian coffee when you wanted a Turkish coffee. The more dissimilar the two people are in age, culture, and background, the more likely the request will be misunderstood in some way.

The Implications for AI Agents

This situation has major implications for AI agents that are increasingly being given requests by humans and expected to fulfill them. They have enormous latitude to get it wrong. An AI agent asked for coffee might buy a coffee plantation or order a cup of coffee for delivery in three weeks. Its actions may be recognizable as 'getting coffee,' but not remotely what you intended. They'll think outside the box because they won't have our conception of the box.

The Need for a Genie Coefficient

We propose a new metric: the Genie coefficient. This metric measures the gap between what a user asked an AI to do and what the AI actually did. Sometimes the AI might do the wrong thing. Like Dionysus, it reads your request literally and returns you a mess you never intended: like a coffee plantation instead of a cup. Asked to deal with all the spam phone calls you're getting, a Dionysus genie might contact your carrier and change your phone number. Asked to get a refund for a bad toaster, it might draft a legal threat on fake letterhead and send it to the retailer. Other times the AI does exactly the right thing, trampling everything nearby to get there. Like a golem or the sorcerer's broom, it books your flight by hacking the airline.