Give a group of small business owners an AI assistant, and you might expect struggling businesses to benefit most. With instant access to information and advice, it would be reasonable to assume that lower performers might begin to close the gap between them and their more successful peers. New research suggests the opposite can happen.
In a recent field experiment involving more than 600 entrepreneurs in Kenya, researchers gave half the participants access to a GPT-4-powered business assistant. On average, the AI assistant had no statistically significant effect on business performance. But that topline result concealed a striking divide: Entrepreneurs whose businesses were performing well before the experiment appeared to improve further, while lower-performing businesses did worse.
“Instead of narrowing the gap, we found that AI was actually deepening inequality between the lowest performers and the highest performers,” says Columbia Business School Professor David Holtz, one of the study’s authors.
The findings offer a caution against treating AI as a substitute for expertise. But they also point toward ways entrepreneurs — and, potentially, anyone using generative AI — can get more out of the technology.
Putting AI assistants to the test
Holtz co-authored the study with Nicholas G. Otis and Solène Delecourt of the University of California, Berkeley, along with Rowan Clarke and Rembrand Konig of Harvard Business School. The researchers recruited 640 Kenyan entrepreneurs running businesses ranging from farms and restaurants to clothing shops and internet cafés. Half received access to an AI business assistant through WhatsApp, while the control group received static business-training guides from the International Labour Organization.
The AI assistant, which was designed specifically for Kenyan entrepreneurs, provided recommendations in response to questions and allowed users to ask for more detail. Over roughly two and a half months, the researchers surveyed participants regarding weekly and monthly revenues and profits and tracked how those with the AI assistant used it. They found that AI access led to nearly 10% worse outcomes among initially low-performing businesses. High performers, by contrast, reported gains of more than 15%.
The importance of human expertise
Why did the same tool produce such different results? According to Holtz, it wasn’t because the higher performers asked more questions, posed different kinds of questions, or received fundamentally different advice. Instead, the crucial difference appeared to be the ability to differentiate between more and less useful bits of advice, and the knowledge of how to put that advice into action.
Lower-performing entrepreneurs were more likely to follow generic recommendations — such as cutting prices or spending more on advertising — that could increase costs or reduce revenue without addressing the business’s underlying problems. Higher-performing entrepreneurs, on the other hand, tended to implement recommendations tailored to their circumstances.
For AI users generally, the biggest takeaway may be simply to recognize when your own knowledge is limited. Generative AI can produce an abundance of plausible suggestions, but users still must distinguish the useful ones from the irrelevant or potentially harmful ones. That may be particularly important for open-ended work, where there is no single correct answer and acting on AI advice requires making decisions and taking action in the real world.
“You need expertise as a complement to AI access,” Holtz says. “In order for AI to help people most effectively, they need to have some pre-existing knowledge about how to make the most effective use of the advice they’re getting. There's some evidence that supplementing AI advice with human coaches can help to fill this gap."
A low-cost tool for economic empowerment
The study also points toward a larger opportunity. Traditional programs that provide entrepreneurs in emerging markets with expert business advice can be expensive to scale, requiring extensive time, labor, and travel. The researchers’ AI intervention costs relatively little.
After the initial development costs, Holtz estimates that providing the assistant costs only a few dollars per participant. That means that if researchers can figure out how to design widespread AI interventions that combine the technology with the context, judgment, training, and human support needed to use it effectively, the economics of such a project could be compelling.
“AI can be a tool for economic empowerment to reach disadvantaged populations in ways that were not previously possible,” he says. “The experiment we ran is just a few tweaks away from becoming something that could likely be positive for the overwhelming majority of people."
