The trust paradox

In "Brave New World", Aldous Huxley created a society in which everyone feels cared for, pampered and safe. Everything is predictable, everything is perfectly tailored. In today's digital reality, personalization sometimes fulfils a similar function. We get exactly what we want - according to the algorithm - before we even think about it. But we have to ask: personalized for whom?

Use technologies that we do not trust

We live in what Cornell University researchers call the "trust paradox" - the public's puzzling willingness to support or use AI-powered technologies that they don't trust. A recent study by KPMG confirms this state of affairs: only 46 percent of people worldwide are willing to trust AI systems, and 54 percent of respondents say they are suspicious of AI. The public's concerns about AI have three main causes: Malfunction (bias, inaccurate results), misuse (misinformation, manipulation) and overuse (deskilling, loss of human interaction, erosion of privacy).

In the context of personalization, the "trust paradox" manifests itself clearly through concerns about privacy, algorithmic bias and the gradual erosion of human agency - the focus of this article. We want the benefits of algorithms that know us, but we resist the very mechanisms that enable this knowledge.

Abundance leads to paralysis rather than liberation

We live in what psychologist Barry Schwartz has called the "paradox of choice", where abundance leads to paralysis rather than liberation. Every day, the average person is confronted with more information than their ancestors processed in a lifetime, yet our cognitive architecture has remained unchanged since survival depended on quick decisions.

In an age of information overload, personalization serves as a filter to help users navigate the overwhelming choice and find what matters most to them. From recommendation algorithms to intelligent chatbots that provide instant help, personalization illustrates how AI has become the driving force behind modern digital experiences. Personalization is often praised for its ability to boost business performance. A study by McKinsey shows that it can reduce customer acquisition costs by up to 50 percent, increase revenue by 5 to 15 percent and boost marketing ROI by 10 to 30 percent. The study found that fast-growing companies generate 40 percent more revenue from personalization than slower-growing competitors.

However, the convenience of personalization comes at a cost that we are only beginning to understand. While its business value is clear, its impact on individual wellbeing and society as a whole is still unclear.

Why we fear what we use

The invisibility of algorithmic processes leads to a fundamental paradox: we crave the convenience of personalization while instinctively resisting the mechanisms that enable it. The most obvious concern is privacy - any personalized recommendation requires the collection, analysis and storage of intimate details about our preferences and behaviors. However, this is just the surface of deeper algorithmic challenges that shape our digital experiences in far more subtle ways.

The hidden costs of being "known"

One of the most subtle shortcomings of personalization lies in how recommendation systems treat those who don't fit the mold. Platforms like Netflix, Amazon or YouTube often rely on collaborative filtering - suggesting content based on the preferences of users with similar behavior. For many users, especially those whose tastes are mainstream, this works remarkably well.

However, personalization becomes less personal if you don't fit the statistical average. If you belong to a minority group - be it in terms of culture, age or niche interests - the system may not have enough data to accurately represent you. As a result, the algorithm makes assumptions based on behaviors it has observed in others, not you. It provides you with the "average taste", not your taste. This leads to a silent erosion of relevance. The system thinks it knows you, but in reality it is projecting someone else's profile onto yours.

This bias is exacerbated when algorithms rely on "proxy variables" such as zip codes, educational background or device type, which correlate with protected characteristics such as ethnicity, gender or income. Uber's algorithm could hide more expensive options for modest neighborhoods - your zip code tacitly determines your choices. Even in the absence of explicit discriminatory intent, such proxy variables can encode social hierarchies into machine learning systems.

When optimization excludes

We continue to engage with targeted advertising systems - clicking on ads, making purchasing decisions based on algorithmic recommendations - even as we become increasingly aware that these same systems can systematically exclude entire groups, tacitly reproducing societal inequalities. Consider how these algorithms work: They predict our likelihood to interact based on past behavior, stated interests and demographics. When an algorithm learns that men are statistically more likely to click on high-paying job ads, it begins to display these job offers predominantly to men. Women - who are just as qualified and interested - simply don't see them. The algorithm is not malicious, but efficient. And this efficiency systematically excludes.

A notable example of this was Amazon's development of an AI recruitment tool that was trained on the CVs of previously hired employees - most of whom were men. The system began systematically downgrading applications that contained the word "women" or came from all-female universities. The algorithm did not consciously discriminate, but merely reflected the biased training data. This leads to "algorithmic redlining" - a digital exclusion that takes place below the threshold of conscious awareness. If certain groups have had less exposure to educational or career ads in the past due to systemic barriers, the algorithm learns not to "waste" impressions on them. The system converts past discrimination into future restrictions, creating a feedback loop in which those who are already disadvantaged receive fewer opportunities for advancement.

Financial algorithms can regularly charge minorities higher interest rates for identical loans, perpetuating economic inequalities through seemingly objective calculations. The system learns that certain groups have been treated differently in the past, and then treats this inequality as a pattern to replicate rather than a problem to solve. Back in 2013, research by Latanya Sweeney showed how this manifests itself even in seemingly neutral contexts: Google ads for criminal records showed up more frequently in searches for names commonly associated with black people than in searches for names commonly associated with white people.

This raises a fundamental question: If personalization leads to different treatment of different users, is it acceptable for people to be shown different content, prices or opportunities just because the system thinks they are certain people? Personalization on a large scale doesn't just affect individual experiences - it can systematically change access to opportunities for entire populations.

Overcoming the paradox

To solve the trust paradox, strict controls are needed to ensure that personalization algorithms operate fairly and transparently and that users retain meaningful agency. Personalization is not inherently problematic - when designed according to human-centric principles, it can truly improve the way we discover, learn and choose. The key lies in what personalization systems are optimized for. When algorithms prioritize long-term user satisfaction over short-term engagement and are regularly screened for bias, personalization can fulfill its original promise: to make our information-rich world more manageable and our decisions more meaningful.

We need both the empowerment of the individual through AI literacy and systemic reforms that embed ethical principles into AI design - what scientists call "AI for the social good". Without such measures, we risk what Oxford University researchers call the "erosion of human self-determination" - a gradual abandonment of human agency in favor of algorithmic mediation.

The choice we face

The choice we face is clear: we can build Huxley's world of comfortable digital conditioning, where algorithms know us better than we know ourselves and gently guide us to predetermined outcomes. Or we can create a bolder new world - a world where technology augments human agency rather than replacing it, where personalization serves our authentic choices rather than manipulating them.

The difference lies not in the sophistication of the algorithms, but in our courage to demand that they serve human flourishing and not digital conformity. The trust paradox will only be resolved if we can use these powerful tools without surrendering our autonomy to them.

The author

Dr. Magdalena Zabicka-Wlodarczyk is a researcher and practitioner with over 15 years of industry experience in a multinational retail company, specializing in market research, cross-cultural collaboration and strategic decision making. She lectures on customer behavior in the context of new technologies, including artificial intelligence and cross-cultural marketing. She is currently a visiting scholar at the ZHAW. Her research focuses on human-centered approaches to AI, such as diagnosing employee attitudes towards AI and examining the role of leaders in its adoption. She is also involved in initiatives to promote AI literacy in education and training.