How algorithmic personalization entered, dominated and will ultimately serve the world.
Personalization is pandemic. But is that a problem?
We lead such digitalized, personalized lives. After two million years spent entirely in the natural world, it’s taken our species just a few decades to create and immerse ourselves in the new digital alternative. 2018 has been the year the number of internet users surpassed half the world’s population. One in three people on the planet uses social media. We spend10 hours a day glued to screens. These digital lives are substantially personalized. In just over a decade we’ve gone from zero to a billion smartphones. Each of these devices displays a personalized selection and arrangement of mostly personalized apps. No wonder we look at them 150 times a day. The pervasiveness of personalization is mind-blowing.
This has all happened in the blink of an eye. It’s not easy to even comprehend what’s happened, let alone whether it’s a good or bad thing or what lies in store. We’ll get to all that but let’s start with some history.
How we got here
The First Industrial Revolution of the 18th and 19th centuries brought mass production: the manufacture of large numbers of standardised products. After Henry Ford popularized the concept in the 1920s, many millions were involved in the mass production or mass consumption of standardised goods. The Second Industrial Revolution, which happened around the turn of the 20th Century, introduced, amongst other inventions, electricity.
It was the Third Industrial Revolution (digital) 50 years ago that enabled us to move from mass standardisation to mass personalization. Written in our own code and free from the laws of physics, the digital world is more amenable to instantaneous, per-user changes than its material counterpart. Suddenly it became possible to provide different experiences to different people at infinite scale. GPS, mail merge and webpages were some of the first illustrations of mass personalization. Note that the devices – laptops, smartphones, smart TVs – through which we access these experiences are themselves mass produced and electronically powered. Personalization, of the Fourth Industrial Revolution, is very much the progeny of the previous three.
An enormous, interconnected digital world rose up. Content grew exponentially (there are now almost two billion websites), fuelled by user-generated material and increasing bandwidth. Advertising paid the bills so web traffic became a hard currency. Companies popped up with ingenious ideas to attract and retain visitors. They placed cookies on our devices to remember what we’d done and what we liked. They made it easy for us to log in and stay logged in, feeding them every mouse click’s worth of data. Static web pages evolved to dynamic, interactive, infinitely adaptive experiences, unique for each user, in each session. To deal with with billions of items of content, personalization platforms - which specialised in getting the right content (news, music, film, photos, videos, gossip, learning, etc) to the right user - joined the cyber-party, applying behavioural science, nudge theory and instant feedback mechanisms to prey on our weaknesses and whims and keep us there. All these personalizing mechanisms gave us a sense of importance and control. We abandoned the slow-paced, finitely-resourced physical dimension for dazzling, dynamic digital.
Three industrial revolutions made personalization technologically possible. Unlimited supply (digital content proliferation) and unbridled demand (insatiable human nature) made it unstoppable.
Present day personalization
Personalization today is ubiquitous but nuanced.
Most digital experiences today are personalized. News and social media feeds are algorithmically determined, just for you. Music, video and film (most modern entertainment) is on-demand and algorithmically enhanced (and largely provided by Spotify, YouTube and Netflix, respectively). Information is filtered just for the individual user’s context by default. Online advertisements are tailored, targeted, and follow us around the web. This adds up to many hours of personalized life, for each of us, every day.
Yet personalization has its limitations. There are several gaps which contribute to these limitations. There’s a limit to how much a system knows about its users, how much it knows about its content, and how computationally effective it is at matching the two. There is also an interest gap: that between the alignment of the motivations of user, platform and third parties. This is why efforts at personalization, even when restricted to the seemingly straightforward field of recommendations, are still far from perfect; Amazon are often ridiculed in recommendation discussions for offering users a second toilet seat.
We engage in personalized experiences to varying extents and in different ways. Sometimes, the personalizing system is in control, sometimes we are. Considering the diversity of the interactions with personalization platforms gives rise to the following hierarchical taxonomy:
These interaction modes are listed in order of decreasing human agency (we do more at the top, less at the bottom), along with platform examples. Note that many of these personalization platforms exemplify multiple interaction modes. For example, you can search, filter and browse on either Airbnb or LinkedIn. But there is often a preeminent, typical interaction and it’s alongside these that I’ve placed the platform. Some platforms retain strong elements of all these modes of interaction eg Amazon, YouTube and eBay.
Starting from the top, configuration is where a user consciously changes settings to personalize her experience. This is maximum agency (though not all platforms allow such configuration). The resulting changes for the user are often superficial: usernames, colours, skins, wallpapers, avatars etc. But they can also affect the user experience more deeply, such as the difficulty settings in a video game or the ordering rubric of a Slack feed.
The next three - search, filter and browse - are close cousins. The important difference between them is how much knowledge the user requires to engage. To search, the user needs to have in mind a word or string of words; such a user is substantially and specifically determining her experience. (That said, where the search itself is highly contextualised as Google’s search has been for five years, the user is no longer in full control.) To filter, the user needs to be able to recognise some categories which are more relevant to her - again, such a user is meaningfully influencing her experience. To browse, the user needs to scroll, travel, move, in her own time, but more or less seeing what anyone else would see - much less self-determination.
The nuances in each case are fascinating. Netflix, for example, certainly provides a personalized user experience, with per-user recommendations featuring in a number of its horizontal trays. But the platform gives much greater prominence to the latest blockbuster and to genre groupings. So really it’s a browse-first experience (the predominant interaction mode is browse) where the agency of the user is paramount. This makes sense for Netflix for a few reasons: the platform hosts thousands rather than millions/billions of assets (a user can get round to a decent proportion unaided); the browsing experience is itself part of the entertainment (ogling thumbnails, playing trailers); the user has to be able to veto films she’s already seen and, relatedly; the cost of watching a film is hours rather than minutes or seconds.
At the bottom of the table, human involvement is passive. Personalized ads just happen to us; we’re oblivious to what’s going on.
Occasionally it feels a bit weird, a bit uncanny and we notice, but that observation / feeling is about as involved as we get. If and when Amazon implements its (patented, of course) anticipatory shipping idea - ie when they bypass the act of purchase - we may need to add a row at the bottom: abdication. When humans aren’t in control, a new breed of problem arises. Children get weird, potentially pernicious content. Horizons are incarcerated in small filter bubbles. Consumers may be treated unfairly. Microtargeting can be weaponized and hate can be monetized. There’s a growing confusion and unease at just what these tech companies are listening in on, logging, using or making available to third parties. All these issues occur with feeds and ads. Such problems are serious because if the user’s not in control, the system must be, and it’s not clear whom the system serves. And since it’s vulnerable, less informed groups - the young, the old, technophobes, certain socio-economic groups - who are more likely to be manipulated, this is critical.
But personally, I find the fifth row the most intriguing and important: recommendations. We have an unquenchable thirst for knowledge. Yet as Socrates, reminded us, “The only true wisdom is in knowing you know nothing”. If you break knowledge down into this 2x2, you can see that there’s one near-infinite quadrant here: the unknown-unknowns. There's a much smaller quadrant - top-right, the Known-Unknowns - which is what Search is about ie when the user has an inkling there's something out there that they need, manifested as a search term. Google's fundamental business is actually aimed at just the small problem.
The unknown-unknowns is vast, unbounded. And this is where personalization, technology, AI, machine learning, collaborative filtering - or, in a word, recommendations - come into play. A good recommender system has near-encyclopedic awareness of (or digital proxies like tags) of the unknown-unknowns. With a little information about you (and some others), a matching algorithm and a feedback mechanism, it can get you to really relevant, really useful content. This is cool in the short-term when that content is a film on Netflix, a song on Spotify, or a picture of some food on Instagram and the process is entertainment. But it’s empowering, enhancing, metamorphic when that content is knowledge and the process is learning. Mining the unknown-unknowns is technology’s most important, realistic assignment. (That’s why we are fixated on recommendations and why we built magpie.)
What happens next
Personalization could go a number of ways. It might be that the public’s unease with tech giants and big data grows and that personalization has to rewind significantly and end up providing less to fewer. Or we may just see slow, gradual improvements over time - slightly better recommendations, slightly better UX, slightly better performance. But the most likely scenario is also the most intriguing: personalization makes a big, positive leap in the very near future and one which ironically retains autonomy for the species. That scenario is that intelligent assistants (‘IA’, aka virtual assistants or personal digital assistants) become a significant part of our lives very soon.
One thing is clear: the world’s biggest companies are falling over each other trying to create a useful IA. Google Assistant, Microsoft’s Cortana, Apple’s Siri, Amazon’s Alexa, Samsung’s Bixby, Baidu’s Little Fish etc are all just prototypes, of limited value to us today. Indeed, for many users IAs irritate more than they accomplish. But if multiple near-trillion-dollar companies are hellbent on making this technological breakthrough, chances are it will happen.
It’s happening. Apple’s iOS12 will recommend specific apps on the basis of patterns of activity. Amazon’s Alexa is now learning to recommend apps on the basis of general, natural intents rather than explicit skills invocation. Microsoft are playing a long game but they must still be in the running with an installed base of 140m Cortanas. Almost half of Americans have now tried a digital voice assistant. There are one billion IAs in circulation today. Some experts predict that number will grow to 7.5bn by 2021.
Your IA will be the ultimate personalized experience. It will serve a single master - you. Interactive by design, not a single other user will have remotely the same experience as you have with yours. It will use data from different areas of your life to answer questions, anticipate needs, make recommendations and carry out basic tasks. Owing to the broad and deep capabilities of the technology, you won’t need more than one IA; it will probably make sense to be monogamous. With a IA, it would be worth investing the time at the outset to tell it about you, get the settings right, just as some people do today with their phones, laptops and browsers.
This is the point. There is a fundamental difference between a user consciously configuring an IA to assist her, and a user receiving personalized experiences from a system which infers intents from past activity. The former scenario is grounded in an explicit instruction from the user, the latter is not. In the near future, your IA will be an adequately and intentionally armed digital operative, carefully and consciously configured to represent your interests ie improve its human master’s life. Your agency is still yours.
With good design, IA can be useful without even being sophisticated. Consider a straightforward, rules-based alert-and-reminder bot which: advises when you’ve spent too much time on a single activity, warns you of train delays, suggests gifts for upcoming birthdays, reminds you of your bank balance as you shop for expensive items. If it did this well, unobtrusively and smoothly, it would be welcome, value adding and not at all irritating. With just a little sophistication, IAs will make high-quality, relevant, fresh, explained recommendations to read / watch / learn / know the right content, enhancing your capabilities and enjoyment. Not only will the recommendations be good, they will be timely, made to fill the interstitial periods in your day. An advanced recommender system, designed for us and working for each of us, 24/7. Just as personalization platforms like Spotify solve content proliferation in a single domain, so IAs will solve content proliferation more broadly, across multiple domains. IAs, personalizing as powerfully as this, would make a major dent in the content proliferation problem. With just a little more sophistication, capability might rise exponentially as we go from you-liked-that-song-so-try-this-one to: you watched that video, liked that tweet and have a free evening so try out this recipe tomorrow - and click here to buy all the ingredients. This IAs will be able to make a dent in the unknown-unknowns described in the previous section. (It’s then really not much more of a stretch to imagine my IA collaborating with your IA to achieve some ends you and I are set on, like scheduling a meeting between us. An IA-to-IA communications protocol would bypass the whole natural language understanding quagmire Google and others seem to be stuck in. Meetings would be a breeze.)
This is not altogether new. The vision of the near future described above is similar to that in Spike Jonze’s ‘completely plausible’ film, Her, in which an operating system serves as Joaquin Phoenix’s very IA. Though the IA in that film is well beyond us today (eg flawless natural language understanding), the sentiment is credible: software will soon serve everyone, individually. What’s interesting is that this future, or a slightly muted version of it, will be with us in just a few years. Neither dystopian, nor utopian, but an improvement on the present. This prediction is just a minor extension of the path that the world population and the tech giants have already set out on.
The contestants in the IA race face many hurdles. Some of those we’ve covered here, such as keeping humans in control, handling personal data and lifting the capabilities of the current crop of IAs. But we will soon have a solution that works, a IA that helps each of us, every day. The self-improving nature of machine learning and adaptive systems means that widely adopted, extensively utilised and implicitly trusted IAs will only improve in the field. Then IA will be working hard, for each of us, knowing its place as a servant of humankind. And personalization will have served its purpose.