Discovery: The Spark of All Journeys and How It’s Evolving

Waffle's Genesis — Part 2

As Satya Nadella pointed out in 2023, search is the most profitable large business category. Yet, the way people search is undergoing a massive transformation.

The Evolution of Search: From Google Maps to Personal Discovery

Consider your experience with Google Maps. When typing something like “sushi restaurant open now,” you might believe you’re searching, but in reality, you’re embarking on a journey of discovery — you don’t know exactly what you want, but you have a starting point. It’s a subtle distinction, but one with profound implications. In these moments, your inputs act like filters, guiding you to discovery rather than merely search — unless, of course, you’re inputting a specific restaurant name.

This distinction is crucial, but why is it so significant?

The Challenges of Existing Discovery Engines

It matters because discovery is at the heart of our personal growth. It’s the spark that lights the way into our interest’ journeys — whether it’s finding beginner piano songs or the latest trends in fashion. Have you ever used Booking.com? IMDB? BoardGamesGeek? Goodreads? These are all examples of discovery engines. The problem is some of them charge hefty commissions fees as high as 15–25%, their coverage is highly specialized, the experience is plagued with ads and most importantly, they are completely isolated from the rest of the interest’s journey! You booked a flight and now want to create an itinerary? Too bad. You’ve found a book you like and want to add notes to it while you’re reading it? You’re out of luck. Even worse, most of the time your interest like a disease or a pet likely doesn’t have a dedicated discovery engine and you have to resort to using google (a website search engine), and hope the website you land on will have a robust enough catalog to give you enough options.

Consumer Frustration with Ads and Lack of Personalization

Once you find a website you hope has a catalog of things you’re interested in — such as: “best compact SUVs” — you have to scroll through their suggestions which are often not localized, not relevant, and promoted. This is probably why 51% of people report lack of options (1). Bummer. In the meantime, you were harvested and targeted. Congratulations, next time you reload, your prices will likely go up, and your online experience is about to get bombarded with distracting ads. To no one’s surprise: 90% of people find ads intrusive and annoying (2). And consumers are tired of it — 71% of them now expect personalized interactions (3). But no matter: search and discovery engines don’t provide this personalized experience yet.

Yet, there’s hope and much to gain for those that improve this experience.

The Future of Discovery Engines: Knowledge Graphs and AI

Since just a few years ago, specific technologies have reached important milestones that make a massive discovery engine viable. I’m talking about something akin to Google Maps — not just for places, but for everything. I won’t get too technical, but to build a discovery engine you essentially need to interconnect everything in a structured way in a type of database called a knowledge graph. With a sufficiently powerful graph you can achieve “semantic intelligence”. There, a movie is connected to its cast, its director, its soundtrack, and so on. This in turn allows you to filter “movies by James Cameron” for example. What’s special about this type of database is that you can apply multiple filters at once in a way that’s much more performant than the traditional (relational) database. Until recently, graph databases weren’t capable of performing at scale, but now they are. At the same time, the advent of Large Language Models (LLMs) like ChatGPT can now apply these filters via regular language, so you don’t even have to apply them manually. What’s more, other AI models can learn how you might want to discover new sneakers, or try out new restaurants that are particularly suited to your tastes.

Personalizing the Digital Experience

This is significant because it heralds a revolution in personalized digital experiences. The fusion of graph databases, LLMs (that use novel vector databases), and AI models can create advanced discovery experiences that were previously out of reach and in the realm of science fiction. But users now demand even more! To create tailored experiences and results, we must integrate personal data hubs built specifically for personal use that know us and yet keep our data private and secure. So far, what little personalization has existed has been thanks to cookies, but the compromise is significant and the benefits meager. The data hubs that exist such as Google Drive and Notion have treated personal users as second class — often prioritizing features for their more sophisticated and paying business customers. This makes them harder to use, and more complex at a time when users demand ease of use and speed. Integrating both a personal data hub and a modern discovery engine will enable the curation of content and suggestions that align precisely with your unique preferences and interests.

This is a huge deal! Imagine bypassing generic lists for a curated selection of options that are specifically suited to you. Instead of searching websites for “best compact SUVs 2023” you could search “new vehicle for my day-to-day” and get suggestions that are absolutely perfect for you in an instant — no more time wasted browsing and searching in Google. This level of personalization will transform how we explore and interact with our interests online.

The Impact of Discovery Engines in the Digital Economy

A few years ago I was reading an article in the Economist about the “creator economy” and a quote from Ben Thompson stuck with me: “value has shifted away from companies that control the distribution of scarce resources to those that control the demand for abundant ones”. This insight rings especially true in the realm of discovery engines.

The significance of building a discovery engine that’s not only comprehensive but also fair and free from ads cannot be overstated.

There are alternative monetization strategies, like affiliate marketing and integrated marketplaces, that align with user interests without compromising the integrity of the discovery process, benefiting users by connecting them directly with products or services that truly match their needs.

The future of your digital interactions should not be one where you’re treated as the product, but rather one that’s user centric and serves you precisely what’s best for you.

Big Tech and the Evolution of Discovery Engines

Big tech is taking notice. The race is on. In fact, it has been for a while — but with moderate success. Google has increasingly been working on discovery engines for hotels, flights and some entities like movies and events. In fact, Google was one the first to explore discovery engines with Google Squared, and there was some talk of Knowledge Cards, which was a response to Wolfram Alpha. They continued with the acquisition in 2010 of a company called Metaweb. But in short, distributed computing just wasn’t mature enough, and knowledge graphs weren’t either. This kept more powerful discovery engines from being technologically possible. But no more. The technology is ready, and the players are moving fast.

The Limitations of Current Generative AI Search Technologies

Meanwhile, advancements in “Generative AI Search” by Google Bard and Bing CoPilot, though innovative, still fall short compared to a fully-realized graph-powered discovery engine. These tools often rely on limited, website-sourced catalogs, lacking the extensive, trustworthy selection and nuanced filtering capabilities that a more advanced system could offer for a truly free and comprehensive exploration experience. Importantly, their current design and implementation makes them prone to unreliable “hallucinations”.

At a time where 5 of the top 10 sites the world spends most time on are search engines, having a superior experience shift to AI + Data Hub + Graph powered discovery engines will be monumental in its impact.

Social Media and the Shift in Digital Discovery Trends

Building a discovery engine alone, however, is not the complete solution. This is particularly evident when considering 16 to 34 year-olds are now more likely to visit a social network when looking for information about brands than they are to use a search engine. Even in my personal experience, I’ve noticed my wife often starts her searches on Instagram using #hashtags before turning to Google. These social media filters, while not perfect, offer a semblance of authenticity and connect users with recommendations from real people, something that even the most sophisticated search algorithms struggle to replicate. But still, there’s a big opportunity missing.

Authenticity and Trust in the Digital Landscape

This shift towards social media for searches reflects a deeper craving for authenticity and personal touch in the digital landscape. While according to Talk Triggers, “word of mouth is directly responsible for 19% of all purchases, and influences as much as 90%”, very few platforms support it. Compromised reviews and ratings further complicate the digital trust landscape. However, the introduction of Know Your Customer (KYC) measures is starting to shift this dynamic, potentially leading to a more accountable online environment. Although KYC has its challenges, it could enhance the credibility of digital interactions. This change could make navigating interests online more reliable and rewarding, aligning better with user needs and preferences.

Unifying the Digital Discovery Journey: A Vision for the Future

The current state of affairs in digital discovery is disjointed, with the journey of interest not fully integrated into the broader exploration of that interest. This fragmentation calls for a significant shift. We stand on the cusp of realizing Tim Berners-Lee’s vision of a semantic, interconnected web, a transformation that promises vast benefits. It’s a change that can unify the exploration and discovery of interests, creating a seamless and empowering experience for self-discovery and growth.

As we explore the imminent transformation of various aspects of people’s interest journey, we can foresee the integration of these elements into a seamless experience. This evolution is more than just a technological advancement; it’s a movement towards empowering individuals in their pursuit of their interests. Join us in this exciting exploration of how we can reinvent our interactions with the digital world, making our adventures into our interests not just more rewarding, but deeply personal and enriching.