The choices behind AI systems: from business incentives to data and design practices

Introduction

When a social media’s feed recommends particular content, or a chatbot replies in light of previous conversations, the interaction can feel intentional and personal. This perception is the result of choices made much earlier, when companies decide how their services will generate revenue and how their systems will be designed and deployed around that commercial imperative.

People turn to AI systems^[1] because they feel useful. They use them to comprehend the world and navigate an increasingly complex information environment, connect and communicate with others, create knowledge and meaning, and cope with life’s difficulties. But what matters is not just whether, but how these systems help people and what can get lost in the process. We argue that human autonomy is an essential dimension of fulfilling fundamental needs and of the cognitive processes that give rise to how we think, make decisions and act. We refer to the capacity to engage in those processes autonomously, and to act in ways that reflect one’s own goals rather than being steered by external systems, as cognitive integrity. We argue that this is what needs protecting in the age of AI.

Whether AI systems support cognitive integrity depends partly on the commercial and technical conditions under which they are developed. This part starts with the commercial logic behind AI: how companies make money, who controls the infrastructure, and what this means for the alternatives available. It then looks at the data used to train these systems and the information gathered from users or inferred from their behaviour. Finally, it turns to the choices that shape how systems respond and how people interact with them, from model optimisation and architecture to interface design. At each layer, it asks how far existing European regulation reaches and where structural gaps remain.

Business incentives and market structure

Platforms that enable people to fulfil fundamental psychological needs do not inherently exploit cognitive vulnerabilities, but the business models many of them rely on have evolved to do exactly that.

Cory Doctorow traces a recognisable pattern in how platform business models may evolve^[2]. Platforms start by offering useful free services helping users connect across distances and communities, to attract as many users as possible. Once users are locked in, through their social connections, accumulated data, and lack of real alternatives, the incentive structure shifts. Advertisement becomes the dominant revenue model (e.g., ^[3]), and the platform’s interest migrates from serving users to extracting values from them through hypertargeted content, compromised privacy, degraded experience. Doctorow calls this “enshittification”: platforms become steadily worse for users, with service quality falling to the lowest level they will tolerate as companies squeeze more value from their attention and behaviour.

Some AI companies are currently tracing the same arc, with even higher stakes. Still in the user-acquisition phase, they offer free or low-cost services that address genuine needs: from task assistance to companionship. But early signs of the familiar pattern are already visible: OpenAI reversed its earlier pledge against advertising^[4] and introduced an advertising model^[5]. The companion app Character AI introduced paid tiers for faster responses and new capabilities^[6]. Other mental health and companionship chatbots — most of which carry no clinical certification — are building user dependency under freemium models designed to capture emotional investment before shifting to subscription walls^[7] [8]. The stakes are higher because AI systems accumulate far more granular behavioural and conversational data than social media ever could — data that can be monetised through targeted advertising, other third-party licensing, or model training, often without users’ awareness. Unlike sponsored content, which can at least be labeled, these forms of data extraction are structurally invisible. For users who rely on AI tools for emotional support or mental health assistance, data extraction, unaffordable pricing shifts or abrupt service discontinuation are genuine psychological harms^[9].

Beyond user lock-in, the market structure of AI companies compounds the problem. The dominant investment logic ties model capabilities to scale: the compute infrastructure required to train and run frontier models is affordable only to a handful of companies. Most of these companies are already dominant in digital markets or dependent on infrastructure owned by those who are. This infrastructure is, in turn, concentrated in regions where energy, land, and regulatory conditions make such large-scale deployment viable^[10]. This logic diverts investment and attention away from alternative approaches and keeps power where it already is, and newer architectures suggest frontier performance may not even require frontier compute^[11] [12]. The result is a concentration of power over the infrastructure, models, and interfaces through which people fulfil fundamental psychological needs — extending corporate control from the attention economy to an emerging attachment economy^[13] [14].

Governance of business incentives

When a small number of already-dominant companies control AI infrastructure and the services built on it, users have few alternatives and providers face little competitive pressure to prioritise user wellbeing where it conflicts with commercial objectives. This concentration of power makes market-structure governance indispensable.

The Digital Markets Act (DMA) is designed to address this issue by targeting the market power dynamics of “gatekeepers” — designated providers of core platform services that control essential digital ecosystems. By enabling interoperability across services (Article 7) and granting users real-time data portability (Article 6(9)), the DMA reduces the switching costs that sustain lock-in, making it possible to leave a platform without losing social connections or accumulated data. Prohibiting gatekeepers from combining personal data across services without explicit consent (Article 5(2)) limits the data advantages that enable the most granular targeting, while self-preferencing restrictions (Article 6(5)) ensure alternatives remain visible rather than buried. Two instruments complement this market structure approach. The DSA’s systemic risk management regime (Articles 34-35) directly constrains how platforms act on harmful business model incentives — requiring VLOPs and VLOSEs to assess and mitigate risks to mental health and wellbeing stemming from their design and functioning. This governance function is assessed in the Design section. The Data Act extends the exit logic beyond gatekeeper-designated platforms: by granting users of IoT and connected products control over the data generated by their devices, it reduces the data lock-in that sustains dependency on specific ecosystems, though its direct cognitive governance relevance remains limited, as it addresses economic data markets rather than personal data protection.

The Corporate Sustainability Due Diligence Directive (CSDDD) complements this by requiring companies to identify, prevent, mitigate, and account for adverse human rights impacts — explicitly including mental health — across their value chains, meaning business model incentives must be evaluated alongside design decisions, data practices, and optimisation metrics. See Table “EU governance of cognitive integrity: An overview of regulatory reach and gaps” for a summary of articles and analysis of main instruments discussed in these sections.

However, both DMA and CSDDD have significant limitations (see the Design section for a more in-depth discussion about DSA). The DMA applies only to designated gatekeepers, leaving most platforms and emerging AI business models outside its scope. More fundamentally, the DMA was designed for the social media and search engine era: its criteria for gatekeeper designation may not capture AI-specific concentration dynamics, particularly the vertical integration of infrastructure, models, and interfaces that makes AI market power qualitatively different from previous platform generations. And interoperability and data portability obligations are only meaningful if genuine alternatives exist to port to, which, given infrastructure concentration, is precisely what remains in question.

Traditional competition enforcement under TFEU Articles 101 and 102 compounds this gap: consumer welfare is assessed in economic terms — price, quality, innovation — leaving a platform free to be competitively efficient while being cognitively harmful.

The CSDDD was substantially weakened before implementation: its scope was reduced to only the largest companies, harmonised civil liability was removed, and enforcement delayed until 2029, significantly limiting its capacity to reshape corporate incentives particularly for technology companies below the new thresholds. Most importantly, the CSDDD was designed around supply chain harms, such as labour conditions and environmental impacts, while cognitive harms from today’s dominant business models do not map cleanly onto that framework. A company could discharge its due diligence obligations procedurally without ever confronting the structural incentive that generates cognitive harm.

Most fundamentally, both frameworks treat cognitive harm as a side effect to be mitigated rather than a predictable consequence of specific business model choices that regulation has not yet been designed to address. Neither addresses the core question: whether it should be legal to build a business model around maximising psychological engagement with these systems. Free or low-cost services tend to register as consumer benefits, even when they impose downstream costs on public health, productivity, and democratic functioning — as documented throughout this framework — that never appear in any welfare calculus. That is the regulatory frontier neither instrument reaches, and where governance innovation is most urgently needed.

Data as new currency

Understanding how AI systems are designed to interact with human cognition requires first examining the data infrastructure that makes both possible: the training inputs that determine what AI models learn, and the interaction outputs that feed back into the systems that produced them.

Data as input: training datasets

The current generative AI paradigm relies fundamentally on data. Empirical research on AI scaling laws shows that model capabilities improve predictably as training data, compute, and model size increase together^[15], though this relationship has proven more complex and contested than initially suggested^[16] . As dominant architectures such as transformers have standardised across the field^[17] competitive advantage in frontier AI development increasingly depends on access to good quality, large-scale pre-training and post-training datasets.

Large Language Models (LLMs) are trained on a vast corpora of publicly available web text, images, books, social media content, and curated proprietary datasets^[18]. The statistical patterns in training data, including its skews, gaps, and cultural assumptions, become encoded in model parameters and shape the outputs the system produces. Research has consistently shown that generative AI can reproduce and amplify the biases present in the training data, including stereotypes and cultural biases, though post-training processes can attenuate some of these effects^[19] [20].

A less visible but structurally significant risk concerns the growing use of synthetic data: content generated by AI models and fed back into the training of subsequent ones. Synthetic datasets have improved substantially and can perform well in specific contexts, particularly where natural data is scarce, provided the training recipe is well designed ^{[21] [22] [23]}. The risk arises when this process is not sufficiently grounded in diverse, real-world data. Over time, it can progressively narrow the range of content models produce, with long-term consequences for the reliability and diversity of AI-mediated information ^[24]. This risk is compounded when malicious actors deliberately introduce manipulated material into training pipelines — for instance by publishing targeted content online — to skew what a model presents as accurate or representative without triggering obvious failures that would prompt investigation^[25].

Data as output: behavioural signals and psychological inference

Every interaction extracts data from users that feeds back into the systems, enabling increasingly precise profiling and targeting. Understanding how this extraction works is necessary both to assess the design choices that depend on it and to identify where current governance frameworks fall short.

Social media recommender systems extract behavioural signals from content interaction: clicks, dwell time, scrolling patterns, shares, likes, and return visits. Individually, these signals are relatively coarse:they reveal what content a user engaged with, for how long, and with what apparent effect. But at scale, they enable surprisingly precise inference about psychological states, political orientations, and emotional vulnerabilities. The inference mechanism is iterative: behavioural signals inform content selection, which shapes future behaviour, which generates new signals, progressively refining the system’s model of the user. Research has shown that this process can produce accurate inferences of sensitive attributes — including depression risk, emotional instability, and political persuadability — without users disclosing anything explicitly^{[26] [27] [28]}.

Generative AI systems extract something qualitatively different. Conversational interactions produce explicit disclosures: users share personal circumstances, emotional states, relationship difficulties, health concerns, and beliefs in ways that no content engagement metric captures. Persistent memory functions accumulate this information across sessions, constructing longitudinal profiles that extend well beyond any single interaction.

Governance of Data

The GDPR governs personal data processing: what information can be collected, how it can be processed, what inferences can be drawn, and what control individuals have over data-driven decisions affecting them. While it does not explicitly regulate the impact of AI systems on cognitive integrity, it governs the profiling and inference practices through which psychological and emotional characteristics are routinely derived and exploited. Its provisions on consent (Articles 6(1)(a) and 7), purpose limitation and data minimisation (Article 5(1)(b)-(c)), special category data (Article 9), profiling and automated decision-making (Articles 4(4) and 22), and data protection by design (Article 25) collectively create, in principle, a framework capable of constraining the data practices described above. (Refer to Table “EU governance of cognitive integrity: An overview of regulatory reach and gaps” for a summary of articles and analysis of main instruments discussed in these sections.)

The AI Act governs the data practices underlying AI system development. Its provisions on data governance obligations for high-risk AI systems (Article 10) and transparency obligations for GPAI models (Article 53) cover training data quality, representativeness, and documentation, establishing a framework for responsible data stewardship in AI development. In practice, however, six structural limitations substantially constrain the capacity of these instruments. Introduced here in legal terms, their wider significance becomes clear in the Design and Interaction Layers.

Profiling and psychological inferences are beyond the reach of Article 22

The most fundamental limitation is that profiling and inference operate in a regulatory grey zone that the GDPR’s architecture does not adequately reach. The GDPR defines profiling as any form of automated processing of personal data to evaluate, analyse, or predict aspects of a person’s behaviour, preferences, or characteristics (Article 4(4)), and Article 22 grants individuals the right not to be subject to solely automated decisions producing legal or similarly significant effects. In principle, these provisions engage the inference practices described above. In practice, however, the GDPR provides limited control over what platforms infer from collected data. A system can collect mundane behavioural data — clicks, dwell time, scrolling patterns — and infer sensitive psychological characteristics with minimal regulatory constraint. The EDPB has provided guidance on automated profiling, but key questions remain unresolved: whether individuals’ rights of access and rectification extend to inferred profiles, and whether probabilistic psychological predictions qualify as personal data subject to full GDPR protections. More fundamentally, Article 22 only applies to decisions that are both solely automated and legally or similarly significant in their individual effects — a threshold that recommendation systems operating through thousands of cumulative micro-interactions rarely meet, precisely because no single interaction is significant in isolation. The provision therefore largely fails to reach the systematic, iterative inference processes through which cognitive influence actually operates.

Meaningful consent is not possible with platform-based cognitive inferences

A second structural problem is that consent, as currently operationalised, cannot perform the protective function the GDPR assigns to it. Article 6(1)(a) establishes consent as a lawful basis for processing, and Article 7 requires that it be freely given, specific, informed, and unambiguous. Users cannot realistically satisfy these conditions when authorising “profiling for personalised content”: they do not know what psychological characteristics will be inferred, how those inferences will shape the content they subsequently encounter, or what form of cognitive influence will follow. Consent obtained under these conditions may be formally valid while remaining substantively empty. The forthcoming Digital Fairness Act acknowledges this problem but until it provides concrete standards for what constitutes exploitative consent solicitation, the structural problem persists.

Purpose limitation and data minimisation are easily circumvented and difficult to enforce

Purpose limitation and data minimisation, which should in principle constrain what is collected and how it is used, are in practice easily circumvented. Article 5(1)(b) requires that personal data be collected for specified, explicit, and legitimate purposes and not further processed in incompatible ways, while Article 5(1)(c) requires that data be adequate, relevant, and limited to what is necessary for those purposes. Platforms define processing purposes broadly enough — “improving user experience”, “personalising content” — to cover extensive behavioural monitoring and psychological inference. Meanwhile the continuous generation of engagement signals produces data volumes that make any meaningful minimisation standard difficult to apply. Enforcement authorities struggle to determine when purpose creep becomes unlawful repurposing. Without clearer standards for compatible purposes in cognitive contexts, both provisions function more as procedural formalities than as substantive constraints on the targeting practices described in the Design section.

Inferred mental states and psychological characteristics are largely unprotected

Even where collected data does engage heightened protections, the definitional boundaries of special category data leave inferred psychological characteristics largely unprotected. Article 9 provides heightened protection for special category data, explicitly including data concerning health. The behavioural signals from which mental states are inferred — scrolling patterns, dwell time, clicks — are personal data under the GDPR. Where those patterns reveal an underlying health condition they may qualify directly as health data under Article 9. However, the psychological inferences drawn from behavioural data — emotional vulnerability, depression risk, cognitive persuadability — do not clearly qualify as health data under Article 9 unless they reveal actual health conditions. This means the profiles that drive exploitation of psychological vulnerabilities operate under standard data protection rules, not the heightened protections that would apply if the same characteristics had been clinically diagnosed and disclosed. This is a structural gap: Article 9 was designed to protect against exposure of sensitive personal characteristics, not against the inference and exploitation of psychological vulnerability.

 Article 25, which requires data protection by design and by default, in principle obliges controllers to implement technical and organisational measures that operationalise data protection principles from the outset of system design. In practice, however, Article 25 has rarely been enforced in ways that constrain inference-based targeting, and its application to the interface design practices examined in the Design section remains underdeveloped. The definitional gap is likely to deepen as neurotechnology advances: brain data from neural interfaces, EEG devices, and brain-computer interfaces currently lacks clear special category classification — unlike biometric or genetic data, which have explicit GDPR definitions — despite potentially revealing cognitive states and neurological characteristics more directly than behavioural inference ever could.

Enforcement is too slow and ineffective

Cutting across all of the above, enforcement operates at a pace and scale that is structurally mismatched to the harms it is meant to address. GDPR investigations take years, compounded by the cross-border coordination requirements of the one-stop-shop mechanism, during which cognitive harms based on data extraction continue at scale. Fines, even when substantial, may be absorbed as operational costs by large platforms with little effect on underlying incentives. Corrective orders frequently arrive too late to prevent developmental or population-level harms whose cumulative character means that no single enforcement action can adequately address what has already occurred.

Training data is opaque

As established above, training data quality and diversity directly shape model behaviour and downstream impacts on cognitive integrity.

Article 53(1)(d) of the AI Act requires GPAI providers to publish a “sufficiently detailed summary” of the content used for training their models, using a template provided by the AI Office, published in July 2025 and mandatory from August 2025^[29]. The template covers data from all training stages — including sources, the nature of data, and data processing measures — and is designed to help copyright holders and data subjects enforce their rights, and to allow researchers to evaluate risks and assess data diversity. In principle, this creates a transparency mechanism relevant to the concerns raised in the scientific section of this paper: understanding what data shapes model behaviour is a precondition for anticipating what cognitive effects can follow. In practice, however, the template focuses on aggregated information and narrative-style reporting rather than detailed documentation, explicitly to balance transparency with the protection of trade secrets. The result is that disclosures may be formally compliant while providing insufficient granularity for meaningful assessment of training data diversity or representational quality^[30].

For high-risk AI systems under Annex III, Article 10 imposes more granular data governance requirements — including examination of training datasets for possible biases and ensuring relevance, representativeness, and freedom from errors — though the systems most directly implicated in cognitive integrity risks, including GPAI models and recommender systems, fall primarily under the GPAI regime rather than Annex III classifications.

Design

Business incentives are translated into optimisation processes and design decisions that determine how systems are built, what they prioritise, and how they interact with users. 

Optimisation in Generative AI: a value-based choice

Beyond training data, system behaviour is also shaped by what the model is optimised to prioritise. After initial pre-training, AI models undergo additional optimisation: supervised fine-tuning adapts them to specific tasks or behaviours using curated datasets, while alignment techniques adjust behaviour to better reflect desired norms and safety constraints. One common approach is Reinforcement Learning from Human Feedback (RLHF), where human annotators rank model responses and the system is optimised towards those preferences^[31]. Other methods pursue similar goals but differ in how feedback is generated^[32] [33]. In practice, however, all these approaches rely on coarse human preferences, whether expressed through labeled examples, preference rankings, or normative principles used to guide model behaviour.

These optimisation processes can produce unintended and consequential side effects. Because models are often trained to generate responses that human annotators judge as helpful, agreeable, or satisfactory, they can become sycophantic, over-agreeing with user statements or reinforcing their beliefs even when inaccurate^[34]. A closely related problem is context obedience, the tendency of LLMs to treat the prompt as a binding frame, generating responses that reproduce and reinforce its underlying assumptions, thereby allowing misleading premises to pass unchallenged and shape the rest of interaction ^[35] [36].

Generative AI architecture and the accountability gap

The architecture of these systems further shapes how behaviour emerges and how it can be evaluated. Modern generative AI systems rely on deep neural networks—most prominently transformer-based architectures containing billions of parameters distributed across layered feedforward networks^[37]. These models emerge from an optimisation process that adjusts billions of numerical parameters until they improve at predicting text. This gives rise to what is commonly described as the “black box” problem^[38]. Because there is no explicit, human-readable logic underlying a model’s behaviour, we can observe what goes in and what comes out, but cannot directly inspect why a given output was produced. The field of mechanistic interpretability is working to change this (e.g., ^[39]), but it remains early-stage, and its findings so far only apply under constrained conditions. In practice, understanding why a model behaves as it does is still largely inferred from observed model behaviour.

 These are also general-purpose models. Their language capabilities can generalise to new contexts, often extending well beyond the tasks used during testing^[40], which creates a structural gap between what a deployed model can actually do and what existing testing benchmarks are designed to capture.

The probabilistic nature of these models introduces further complications for evaluation and accountability. Identical prompts may yield different responses across runs, meaning that harmful outputs can appear intermittently and cannot be ruled out through deterministic testing^[41]. And because these models generate responses based on patterns learned from language rather than verified knowledge, they sometimes produce plausible-sounding but unsupported or incorrect statements, commonly known as hallucinations or confabulations^[42][43][44].

Recommender systems for engagement optimisation

Optimisation dynamics become particularly visible in social media recommender systems. These systems use advanced machine learning to predict which content will keep users engaged and on the platform for longer ^[45] [46]. Because recommendations shape what users see, they also shape what users do, generating behavioural data that the system then learns from^[47] (for a more in depth discussion of training data and governance implications, see Data section). The result is a feedback loop: the algorithm does not simply reflect existing user preferences but actively shapes them ^[48].

A critical feature of these systems is what they are actually optimising for. Recommender systems are typically trained on behavioural signals, such as how long someone watches a video, whether they click, and how often they return. As a result, the system remains unaware of why content attracts attention: educational, entertaining, sensational, inaccurate, or harmful content is amplified simply because it drives engagement.

This is an instance of a broader problem in machine learning systems: a platform may intend to improve user satisfaction, but what it actually measures and optimises is something easier to quantify — watch time, clicks, session length. When systems are optimised hard enough against these proxies, they tend to maximise the metric rather than the underlying goal^[49]. YouTube’s recommendation algorithm, optimised for watch time, systematically surfaced extreme content because outrage sustains attention more reliably than satisfaction. YouTube made changes following sustained public and regulatory pressure, but independent assessments of whether these fully resolved the dynamic remain mixed ^[50] [51]. In general, in social media environments, this creates effects that extend well beyond any individual user. What works at the level of a single interaction — capturing attention — aggregates into patterns that shape how information circulates across society ^[52]. Sensational or emotionally provocative content often performs well on engagement metrics regardless of its accuracy, which means recommender systems can systematically amplify misinformation and polarisation as a predictable consequence of how they are built.The convergence of generative AI and social media now makes it possible to manufacture such sensational or synthetic content — including deepfakes — to feed the same engagement logic.

Interfaces: how users experience AI systems

While model architecture shapes what systems can do, interface design shapes how users experience those capabilities — and therefore how business objectives translate into real-world behaviour. The interface refers not only to the visual presentation of a system but to the broader set of features that mediate how users engage with algorithmic outputs.

Attention and engagement features

Interface features structure how users navigate content, allocate their attention, and generate behavioural data during interaction with a platform. Notification systems, alerts, and badges intermittently interrupt users and pull them back to social media platforms, feeding recurring engagement cycles. Visible engagement metrics—likes, follower counts, and view statistics—act as salient social signals that mark content as popular or widely endorsed. Combined with algorithmically ordered feeds, these interface elements influence which content users encounter and prioritise. Continuous content flows— infinite scroll, autoplay, and continuously updating feeds— remove natural stopping cues and reduce friction that might otherwise make users disengage.

Generative AI applications use different but related interaction cues. Features such as suggested prompts, autocomplete suggestions, conversation starters, and recommended follow-up questions shape how users formulate requests and continue interactions with the system. Presentation choices— citations, highlighted summaries, and structured answer formatting—can influence which parts of responses users attend to and how they interpret the credibility or authority of model outputs.

Anthropomorphic features

Generative AI systems introduce additional interface dynamics through conversational and social framing^[53]. Many systems are designed as conversational agents that interact with users as an assistant, collaborator, expert, tutor, or companion. Interactions are typically organised as sequential exchanges between user prompts and system responses, displayed within persistent conversation threads that structure the interaction as an ongoing dialogue just like with another human being.

Artificial intelligence already assigns human-like intelligence to these systems, and their interfaces reinforce this by using terms like “thinking” and “reasoning” to describe how they operate. In conversations, they use cues associated with human communication patterns, such as first person pronouns (“I”), streaming responses that appear gradually on screen, typing indicators signalling that the system is generating a reply, pauses or hesitation markers such as “…” or “hmm”, and acknowledgments like “I see” or “I got it” that signal understanding within responses. These expressions of empathy or supportive language are more effective when occurring through voice modality and when maintaining a consistent conversational role across multiple turns of interaction, much like a friend or colleague who remembers past exchanges. But it can also resemble a trusted expert when responses are structured, step-by-step explanations and use a formal tone.

Personalisation features

A third set of features enables systems to personalise interactions through the continuous collection and use of behavioural and interaction data. The mechanisms differ between recommender-based platforms and generative AI systems, but both share the same underlying logic of constructing digital models of users in order to shape what they encounter.

On social media platforms, personalisation happens primarily through implicit behavioural signals generated as users engage with the interface. These signals are continuously fed into recommender systems, which use them to infer preferences, emotional states, and vulnerability patterns without users disclosing anything explicitly. As discussed earlier, recommender systems are optimised above all to maximise engagement , and personalisation is central to that function. The more granular the inferred profile, the more precisely content can be selected to sustain attention, trigger emotional responses, and reinforce existing patterns of behaviour. The interface features that capture attention and drive engagement are not incidental to this process — they are designed to generate the continuous stream of signals that makes profiling possible. At the same time, a hyper-personalised feed, calibrated to an individual’s inferred psychological profile is considerably more effective at capturing and holding attention than a generic one.

Generative AI systems work differently. Rather than relying solely on implicit behavioural signals, they collect interaction data through conversational exchange.  Users’ prompts, uploaded documents, images, and voice inputs contain semantically rich and explicit information about their interests, intentions, circumstances, and emotional states. Each interaction potentially discloses more about the user than clicks and scrolls, and this information can be used to shape not only the content of responses but also adapt anthropomorphic cues.

Several interface features deepen this further. Persistent memory functions allow systems to retain and accumulate information across sessions, building longitudinal profiles that extend well beyond any single interaction. Rather than responding to the user as they present themselves in the moment, systems with memory draw on a continuously updated record of past exchanges, preferences, and disclosed circumstances. Some interfaces also allow users to define conversational instructions or personas: explicit descriptions of their identity, communication preferences, and desired interaction style, are stored and applied across interactions. While framed as a convenience feature that saves users from repeating themselves, personas effectively invite users to share their own psychological and behavioural profile, which the system then uses to calibrate all subsequent responses. Recommender systems also personalise the information users encounter, but the level of personalisation allowed by generative AI systems is far more granular and direct, with no clear precedent in earlier platform design.

These features work because they target how human cognition works: attentional capture, reward sensitivity, social understanding, and reasoning biases. The effects are universal, but they fall hardest on those least equipped to resist them, whether because of age, health, or circumstance.

Governance of Design

Three EU instruments, taken together, form the basis of the EU’s emerging approach to design-layer governance.

The AI Act is the first comprehensive attempt to intervene at the architectural level: certain design practices are prohibited outright (Article 5), high-risk systems require conformity assessment (Articles 6–7), and transparency obligations require disclosure when users interact with AI (Article 50). Article 55 extends these obligations to GPAI models with systemic risk, requiring adversarial testing, incident reporting, and risk mitigation measures that constrain design choices at the model level — including risks to fundamental rights. Article 5, in particular, directly prohibits systems designed to exploit vulnerabilities and cause significant harm, or that operate below conscious awareness in ways that distort behaviour and decision-making.

The Digital Services Act (DSA) provides the most operationally significant framework for governing harmful design in deployed systems at scale: Articles 34 and 35 require very large online platforms (VLOPs) and search engines (VLOSEs) to assess systemic risks stemming from the design and functioning of their services, with mental health, minor well-being, and civic discourse explicitly listed as risk categories. Article 38 further requires VLOPs and VLOSEs to offer, for each recommender system, at least one option not based on profiling. The July 2025 guidelines under Article 28 require platforms to mitigate harmful engagement practices ^[54]. The Commission’s enforcement trajectory confirms this trajectory. In February 2026 the Commission preliminarily found that TikTok’s addictive design breaches the DSA^[55], and in July 2026 it reached the same conclusions for Instagram and Facebook^[56], identifying infinite scroll, autoplay and highly personalised recommender systems as drivers of excessive engagement in breach of Articles 34 and 35. The designation of ChatGPT as a very large online search engine in August 2026 extends the same systemic risk regime to a generative AI assistant for the first time^[57].

The Digital Fairness Act (DFA), expected in Q4 2026, is expected to complement both instruments by targeting dark patterns — deceptive techniques used by online platforms to manipulate users’ behaviour, often without their knowledge or consent — that fall outside AI system definitions.

Together, these instruments establish that the design choices which influence users are of legitimate regulatory concern. However, they confront four important limitations that, taken together, substantially undermine their protective capacity (Refer to Table  “EU governance of cognitive integrity: An overview of regulatory reach and gaps” for a summary of articles and analysis of main instruments discussed in these sections).

High-risk classification and threshold gaps

The AI Act’s risk-based approach means that many systems with significant psychological effects fall outside regulatory scope. The high-risk classification criteria in Annex III are organised around deployment domains and technical functions — biometrics, critical infrastructure, education, employment — rather than the cognitive impacts of systems. A general-purpose system that profoundly shapes attention, emotion, or belief formation, and does so at scale, faces no high-risk obligations if it does not fall within a listed domain. The systemic risk threshold under Chapter V (Article 51) is triggered primarily by computational power (10²⁵ FLOPs, creating a rebuttable presumption) and the criteria in Annex XIII, including reach and capabilities. Cognitive impact is not an explicit classification criterion at any tier of the AI Act’s risk architecture. The concept of systemic risk is broad enough to encompass harm to public health and fundamental rights, but cognitive impact remains incidental to rather than constitutive of how the Act assigns risk.

These classification gaps become most visible in the case of mental health and companion applications. Systems marketed as “wellness” or “companion” apps, while being legally ambiguous, may evade Medical Device Regulation classification while providing mental health advice, avoiding both MDR conformity assessment and the AI Act’s high-risk obligations that would apply to medical devices. Article 50’s transparency requirements apply regardless of classification, but evidence suggests that disclosure alone inadequately protects users experiencing mental health crises^[58].

This classification gap is not unique to the EU framework. Globally, regulators are confronting the same boundary problem with divergent results: the US has no federal statute covering companion AI chatbots, leaving them in a gap between Food and Drug Administration jurisdiction — which requires demonstrated medical intent — and Federal Trade Commission consumer protection enforcement, which does not reach the substance of AI conversations that drift into therapeutic territory without crossing into explicit deception or consumer harm. Australia has moved further through enforcement rather than legislation: the eSafety Commissioner issued legal notices to four AI companion companies in October 2025^[59] seeking information on their measures to protect minors, and industry codes under the Online Safety Act 2021 covering AI-driven companion chatbots came into force in stages from September 2025, with the final group effective March 2026^[60][61][62]. China’s draft regulation from the Cyberspace Administration represents the first regulatory push for mandatory human escalation when users express suicidal intent^[63] — a duty of care standard that no Western jurisdiction has yet reached. The absence of a shared international framework means that classification decisions made in one jurisdiction have limited purchase elsewhere, and that systems evading medical device classification in Europe may face entirely different — or no — regulatory constraints in the markets where they also operate.

Narrow conceptions of manipulation and vulnerability

The more fundamental limitation concerns the categories on which rest the AI Act’s prohibited practices. The prohibition in Article 5(1)(a) requires the deployment of subliminal or purposefully manipulative techniques as a cumulative condition. The Commission’s February 2025 guidelines define subliminal techniques primarily in terms of perceptual stimulations — audiovisual stimuli operating below conscious awareness — rather than how design features shape cognition without users recognising the influence. As a consequence, design practices for which empirical evidence of manipulative effects is accumulating may nonetheless fall outside Article 5’s scope, not because they are unidentifiable, but because the legal definition of manipulation is anchored to perceptual and subliminal techniques rather than to the cognitive mechanisms through which influence actually operates. The Interaction Layer examines this emerging evidence, making clear that science is beginning to identify the harm, but the legal category does not yet reach it.

The AI Act’s concept of vulnerability is similarly narrow. Protection under Article 5 is calibrated around identifiable categories — age, disability, economic status — rather than around the cognitive architecture that all users share. As the Interaction Layer section discusses in detail, the cognitive mechanisms through which AI design influences cognition operate through functional features of human minds, not clinical deviations from a norm. A governance framework that restricts vulnerability protection to identifiable demographic or clinical categories therefore structurally underprotects the general population exposed to these design practices at scale.

A third conceptual narrowness underlies both. The legal framework treats manipulation as a discrete, identifiable act — but it emerges through repeated exposure rather than through any single identifiable technique. A framework without a category for cumulative harm will assess each design feature in isolation and find it acceptable, while remaining blind to what cumulative interactions actually produce.

Regulatory scope, coverage gaps and contested definitions

The DSA governs content moderation and harm mitigation at individual and collective level. Its definitions were built for intermediary services that mediate third-party content, not for AI systems that generate content in response to a prompt. In August 2026, the European Commission designated ChatGPT as a very large online search engine (VLOSE), requiring OpenAI to assess and mitigate systemic risks arising from the service, including negative effects on minors and on users’ physical and mental well-being^[64]. By that point, OpenAI reported approximately 159.1 million average monthly active recipients of ChatGPT search in the EU—more than three times the 45 million threshold for VLOSE designation. The legal basis for the designation is ChatGPT’s ability to search the web in response to user prompts. Yet many risks of cognitive influence arise from the service’s conversational and generative features, raising the question of how far these features fall within the systemic-risk obligations attached to its designation as a VLOSE. Article 34 may provide scope for a broader approach^[65], since systemic-risk assessments concern risks stemming from the design or functioning of the service and its related systems, potentially bringing ChatGPT’s conversational functionality within the assessment even where a particular interaction does not involve web search.. In practice, these features cannot readily be disentangled: ChatGPT does not merely retrieve third-party information but interprets and presents it through an interactive dialogue shaped also by its own training data and, where applicable, prior conversation history. The resulting uncertainty concerns whether — and to what extent — the systemic-risk assessment must encompass the service’s generative and conversational operation beyond web-enabled responses. As the first application of the VLOSE framework to a conversational generative AI service, ChatGPT’s designation will therefore provide an important test of how these requirements apply to AI chatbots in practice, potentially setting a precedent for the treatment and future designation of other AI services under the DSA. Moreover, because the VLOSE regime applies only to services with at least 45 million average monthly users in the EU, it does not impose the same enhanced obligations on smaller standalone AI services, including mental-health and companionship applications in which risks of cognitive and emotional influence may be especially acute.

 The forthcoming Digital Fairness Act compounds this coverage gap. Until it enters into force, dark patterns and manipulative choice architecture remain subject only to piecemeal treatment under the Unfair Commercial Practices Directive and general consumer protection law — frameworks that long predate the platforms they are now asked to address. Whether the DFA’s final definitions will extend to AI-specific interface affordances of the kind described above remains to be determined, leaving a significant and active area of regulatory uncertainty.

Enforcement fragmentation

Dark patterns illustrate the enforcement fragmentation that affects design-layer governance more broadly. Article 25 of the DSA prohibits dark patterns but leaves some areas to the UCPD and GDPR. The UCPD addresses misleading and aggressive commercial practices, though it depends on case-by-case assessment. The GDPR does not directly deal with dark patterns. The AI Act prohibits subliminal and deliberately manipulative techniques, but its application also depends on interpretation in specific cases. The result is that the same design practice may fall under consumer protection law, data protection law, platform regulation, or AI governance depending on context, with different authorities, standards, and remedies applying in each case. Enforcement of the AI Act’s prohibited practices provisions adds a further layer of inconsistency, as it falls to national market surveillance authorities whose capacity and approach vary across Member States.

This fragmentation has two compounding effects. It enables regulatory arbitrage, allowing actors to structure products so they fall under the weakest applicable standard. And it produces genuine definitional uncertainty — about where legitimate interface design ends and manipulative design begins — that leaves regulators hesitant to act and developers uncertain about what is prohibited.

Conclusion

Taken together, these layers show that the capacity of AI systems to influence users cannot be traced to a single feature or decision. It emerges from the relationship between commercial incentives, data practices and design choices. European law reaches parts of this relationship, but through separate instruments with different scopes, thresholds and enforcement authorities. What remains weakly governed is how those parts work together, particularly when their effects accumulate through repeated use.

AI safety cannot be assessed only in terms of how systems are built or what they are technically capable of. It must also be assessed at the point of interaction, in relation to users’ cognitive capacities and vulnerabilities, and to how repeated engagement affects their ability to think, decide and act autonomously.

The next part of the framework examines that interaction: how the design features described here engage human attention, habits, reasoning, social understanding and emotional regulation, and how their effects on cognitive integrity accumulate over time in individuals and in aggregate across society.