Every day, millions of people open music streaming apps, press play and trust an algorithm to know what they want to hear next. Sometimes the algorithm nails it. Other times, it misses the mark—either by recommending a song you don’t like, or one that’s just wrong for the situation. Even if you’re a big Bob Dylan fan, for example, you might not want his breakup song “Don’t Think Twice It’s All Right” on your wedding playlist.
For streaming platforms and other digital businesses, improving this experience is high stakes. As consumer data has exploded, so has the race to uncover a richer way to understand user preferences, make better recommendations and ultimately drive engagement. A team of marketing professors—Asim Ansari, Khaled Boughanmi and Yang Li—believe they have found one.
“Our goal was to find a better method of recommending new songs and categories that consumers would really be interested in exploring, no matter the situation,” says Ansari, William T. Dillard Professor of Marketing at Columbia Business School.
To do this, the team used a cutting-edge mathematical tool that can represent complex relationships in a nuanced way, one that had not been systematically applied to consumer collections: hypergraph neural networks. They found that, compared to traditional models, these sophisticated models provide a richer understanding of how listeners organize their music collections and how those structures reflect different listening contexts.
The research has big implications not only for music streaming services, but for any business that deals with curated collections and makes personalized recommendations to consumers.
A better way to measure complex relationships
Music collections offer a rich window into consumer taste, reflecting not only which songs a listener likes, but which songs belong together and in which situations. In other words, a playlist isn’t just a group of similar songs, it’s a collection of songs created for a specific context: music for an early morning workout or a period of heartbreak, a dinner party or a nostalgic mood.
Traditional models tend to be pretty good at guessing songs you might like, but they’re often not very good at determining whether a song fits a specific context. Ansari and his colleagues suspected that hypergraph neural networks would be better at this, since they’re designed to map multi-dimensional relationships between multiple items. In the case of music, they can show how listeners, songs and playlists connect in many ways. For example, a single song might connect many listeners across multiple playlists, creating a dense web of relationships. Hypergraph neural networks can capture all these relationships simultaneously, and they can measure how strong they are.
Smarter recommendations
To showcase this capability, the researchers compiled a multi-source dataset that included 2,000 randomly sampled Spotify users, more than 33,000 user-curated playlists and nearly 770,000 unique songs by nearly 95,000 artists, as well as technical audio characteristics and crowd-sourced genre and mood descriptors. They then created three interlocking hypergraphs to model the links between consumers and songs, songs and playlists, and playlists and consumers. The result was an intricate representation of relationships between users, playlists and songs in measurable, geometric space.
“Suppose you see my collection and want to extend it with new song recommendations,” Ansari says. “You’d do that by looking at my songs in this geometric space and recommending the closest ones to me.”
To determine how well their model captured the presence or absence of connections among songs, playlists and users, they randomly set aside some of these connections from the Hypergraph networks and did not train their model on these. Then they used there model to predict these set aside connections. They did the same with more traditional recommendation methods. If a model does better at such predictions, it would be good at capturing consumer preferences as well as playlist contexts.
The results showed that the hypergraph neural networks performed better than traditional models with an accuracy of prediction of 86% compared to 61 to 81 percent for traditional methods
The future of personalization
For music streaming platforms, hypergraph neural networks could be used for more than just recommending individual songs. For example, they could help generate entirely new playlists tailored to specific listeners in a specific context, expand existing playlists with songs that perfectly match their mood and purpose or recommend playlists made by other listeners.
And the applications extend far beyond streaming. “Any grouping can potentially be analyzed using these methods,” Ansari says—including collections like grocery baskets, recipe boxes and reading lists.
For digital platforms of all kinds, the approach offers a scalable way to interpret complex preferences across large and diverse user bases. As brands lean further into consumer customization, models like this may help them move closer to recommendations that feel less generic—and more genuinely personal.
