14  Data Protection and Large Language Models

TipLearning outcomes

By the end of this chapter, learners will be able to:

  • distinguish large language models from other kinds of AI systems.
  • identify current open problems of those models from a data protection perspective; and
  • exemplify potential solutions to addressable issues.

In the last few years, much of the discussion about AI has focused on Large Language Models (LLMs). An LLM is a type of AI model that is designed to process and generate text, using vast amounts of training data to learn how to extract and reproduce patterns in human language. Those models were brought to public attention in November 2022, when OpenAI released the first iteration of ChatGPT, a chatbot powered by one of its earlier LLMs. Because ChatGPT could answer a variety of queries and dialogue with its users, it soon became a popular tool, used not just for fun but incorporated into a series of applications.

As these models have grown larger and more complex, their development has concentrated within a few major corporations due to the significant computational resources and expertise required. Many businesses now use these models through APIs or tools without needing to develop them in-house. This concentration raises unique challenges for data protection, as responsibilities and risks are distributed across the supply chain in ways that can complicate compliance with data protection law.1

LLMs raise a variety of challenges to data protection law. Of those, a few are particularly critical.

Concentration of the markets for LLM technologies can compound those risks, as it means that an organization’s compliance challenges will be affected by the technical decisions made by a handful of technology suppliers. As such, these suppliers become especially relevant for data protection enforcement. This does not mean, however, that the organizations using those LLMs suddenly become exempt from their GDPR duties. Just that compliance with those duties might become more difficult.

In addressing those issues, organizations and regulators face specific challenges. One of them is the difficulty in ascribing responsibility. Providers of LLMs have the technical expertise to modify them but often lack the specific context of each business or end-use case. Without knowing the particularities of each deployment, it is difficult for model providers to foresee every privacy risk or ensure compliance with regulations.

Meanwhile, organizations using LLMs within their own systems—whether as part of their operations or as end-users—understand the context of their data processing and how the model affects their workflows. What they typically lack is the ability to alter the model itself to address context-specific privacy risks. For example, a company using a commercial LLM for customer service could struggle to adjust the model’s data retention practices to meet GDPR requirements, as the model’s provider controls these technical aspects.

An emerging alternative to the LLMs provided by large technology companies is the rise of open-source LLMs, which organizations can modify more freely than commercial models. Although open-source LLMs may not perform as consistently as closed-source models from major tech firms, they offer growing capabilities and flexibility. With open-source LLMs, companies have more control over model adjustments, which can aid in adapting data handling to meet legal requirements.

Open-source LLMs bring their own challenges. Developers and businesses using them will need a greater degree of technical expertise, not just to make use of these models but also to ensure their compliance with technical requirements. For example, a model provider’s measures for complying with data protection requirements will propagate to any AI systems using those models. An organization that uses its own models will need to implement their own measures for that purpose, and to do so they might need to make changes to the original model. If, on the one hand, open-source models give them the power to make such changes, on the other hand that power is of little use if the organization lacks the expertise to do so. Each organization must therefore consider what kind of model, if any, is better suited for data protection compliance considering the resources it has available.

In this chapter, we examine the data protection issues created by LLMs and how they affect the design and use of AI systems based on them. In Section 14.1, we consider the opportunities and risks created by those models, obtaining a clearer view of what they can and cannot do with the current state of the art. Section 14.2 then looks at data protection issues that emerge during the development of LLMs. To wrap up, Section 14.3 discusses measures that organizations can adopt when they use systems based on LLMs.

14.1 The opportunities and risks of large language models

TipLearning outcomes

By the end of this section, learners will be able to exemplify why the use of LLMs might be desirable in some cases and why it might not be in others.

Large language models (LLMs), such as OpenAI’s GPT-4, represent a significant advancement in artificial intelligence. Those models are produced by training deep neural networks on large datasets of written text, which often include almost everything that is publicly available on the internet for a given language (Kucharavy et al. 2024, ch. 1). This training process allows LLMs to recognize patterns in text data, which they subsequently apply to the consumption and generation of other texts. Current state-of-the-art LLMs can perform an impressive range of language tasks, including text summarization, answering questions, translating languages, generating written content, and even conducting conversations that mimic human dialogue.

These capabilities make LLMs particularly valuable in areas where efficient language generation is required. For example, InnovaHospital might want to create a chatbot that interacts with patients to carry out an initial triage based on their symptoms, while DigiToys might benefit from LLMs to enable their toys to better interact with children. Those tasks can be useful and even transformative of certain kinds of work. However, they create a series of risks for data protection, too.

Despite their impressive capabilities, LLMs are not omniscient or capable of reasoning like a human. They do not inherently understand concepts or the world in the way humans do; rather, they generate responses based on probability distributions derived from their training data. As a result, they lack “real” intelligence or understanding and are prone to “hallucinations” — confidently generating incorrect or fabricated information. This is not merely a flaw but a fundamental limitation that cannot currently be eliminated in their design.

As a result, these models remain limited in their reliability and interpretative abilities. LLMs can sometimes provide misleading or inaccurate information. For example, even the best current LLMs often struggle with simple mathematical problems, depending on how they are phrased. Not to mention the countless examples of “hallucinations” one can find online. Reliance on misleading information generated by an LLM runs against the GDPR principle of accuracy in the processing of personal data, especially if that wrongful output becomes the basis for subsequent processing.

Another data protection concern with LLMs relates to how those models are trained. There is, currently, a scientific controversy on whether a large language model does, in fact, memorize information during its training process. If that is the case, then the model itself might contain personal data. This is because LLMs are trained on massive datasets, which often contain public and potentially private information about information. As a result, those models might capture personal data used in its training, even if precautions are taken in the process. However, the technical and legal question of whether memorization matters for data protection law remains open.

Even if no memorization takes place, LLMs can themselves be taken as a source of personal data. If an LLM can generate outputs that refer to an identified or identifiable natural person, that output might qualify as personal data. This is the case even if the output itself is false. After all, the definition of personal data refers to information that can be associated with a natural person, and accuracy is a requirement that applies after something is classified as personal data. The outputs of an LLM, or of a system that is based on an LLM, might therefore be subject to data protection requirements. Currently, there is intense discussion among EU data protection authorities on how to apply data protection requirements in this context, and further guidance is likely to come soon.

Because they can be used for a broad variety of purposes, LLMs are likely to qualify as general-purpose models under the AI Act. And, given the potential reach of those systems that can generate language at scale, they might create systemic risks. The AI Act’s regulatory approach to this kind of model is pretty narrow. It establishes a set of security and safety requirements, which must be observed by the most powerful models in the market 2. This approach ensures that the state-of-the-art models are made safe(r) before they can be commercialized or made accessible to the public.

The AI Act’s response to systemic risk goes, in some respects, beyond the individual focus of data protection law. Even so, the brunt of the legal response to the risks created by LLMs falls on the metaphorical shoulders of data protection law. This is because many of the potential harms from the training and use of LLMs can affect identifiable persons. In the following sections, we consider how LLMs affect compliance duties for providers and deployers of AI systems based on LLMs.

14.2 Safeguarding measures during model development

TipLearning outcomes

By the end of this section, learners will be able to describe some of the data protection issues that might emerge during the development of an LLM and sketch potential responses to them.

The training process of LLMs raises some important data protection questions. As discussed in Section 6.2, the LLM’s provider is likely to qualify as the data controller of any personal data processed during the training procedure. Consequently, they need to adopt safeguards to address general risks involved in model training as well as the risks that are specific to LLMs.

The first obligation of a data controller training an LLM is to ensure that any personal data in the training set can be lawfully processed. This entails identifying whether such data is present in the training set, finding proper legal bases for its processing, and ensuring the observance of the applicable rights and requirements from data protection law. In this regard, things are similar to any other AI model or system. What changes is that some specific factors become more salient.

One of them concerns the data sources used to train the LLM. Language models are often created from “data trawling” (Kuru 2024), that is, the mass consumption of publicly available data. While public data may seem readily available for training, it does not automatically mean it can be freely used; public availability does not negate data protection obligations. Data controllers should assess each data source, verifying that proper legal grounds, such as legitimate interest or consent, justify the use of personal data within the dataset. This step is especially important if the data includes sensitive or special categories of personal data. Documenting the legal basis for data use can support compliance and prepare organizations for potential audits or inquiries.

Data minimization is another critical principle in the development of LLMs. This means using only the minimum amount of data necessary to achieve the training objectives. Privacy-enhancing techniques, analysed in Section 12.1, can be used to reduce the amount of personal data contained in the training dataset while still retaining the dataset’s utility. Organizations should consider implementing robust data governance frameworks that regularly evaluate and update data minimization practices throughout the LLM’s development and maintenance phases.

LLMs are often designed for general-purpose applications, meaning they can be adapted to a wide range of uses beyond those initially anticipated. This versatility raises unique data protection concerns because a model’s potential misuse or unintended applications could lead to new privacy risks. Under the AI Act, such general-purpose AI systems are subject to specific regulatory requirements, which mostly relate to the kind of information disclosure discussed in Chapter 11. For the most advanced AI models, some additional requirements apply, as stipulated in Article 55.

Regardless of the applicability of those provisions, developer organizations must anticipate a range of potential use cases for their model, including harmful applications. Given that the developer is likely to be a controller for processing in the training process, the purposes for which a model is tailored are relevant for determining the lawfulness of processing. Additionally, these developers will also need to consider kinds of misuse that, while not intended, are reasonably foreseeable from the original application. For example, DigiToys needs to consider that any safety issues with their model might allow hackers to interact with the children that play with their smart toys.

In forecasting potential misuses or errors, organizations should pay particular attention to possible unintended processing of sensitive data. Without adequate controls, a model could inadvertently generate or expose sensitive information such as health, racial, or political data about individuals. This matters even if the information generated by the system is not true, as in that case the LLM will be generating (and potentially help spread) false information about an individual. For example, if an LLM is used to generate texts that slander someone, the generated outputs are personal data even if they have no basis on reality, as they are associated with an identifiable natural person. As such, developers might want to consider the use of techniques to restrict the outputs, prohibiting certain kinds of prompts or interactions with the model.

Moreover, models accessible to minors might pose unique risks, as they could unintentionally generate harmful or age-inappropriate content. Effective measures to address these risks include implementing monitoring mechanisms that detect and flag potential data-related issues in real-time. Additionally, integrating ethical and responsible use guidelines into the deployment phase can help prevent misuse, ensuring the model’s output aligns with data protection standards and user safety expectations.

To meet both GDPR and AI Act obligations, it is critical to adopt specific safeguards. For instance, transparency measures can help users understand the limitations and potential biases within the model. Furthermore, to fulfil GDPR joint controllership obligations, organizations must collaborate with partners to jointly establish accountability measures for data processing.

Organizations may also consider adopting robust model documentation, as discussed in Section 10.1. Those documents can detail the training data, potential risks, and safeguards to enhance the model’s transparency. Although this step is more exhaustive than documentation requirements for traditional data processing activities, it is crucial to clarify accountability and manage systemic risks associated with general-purpose AI models under data protection law the AI Act.

An emerging issue in LLM development is the risk of poisoning the well, that is, the degradation of the data used to train a model. Degradation can happen, for instance, when AI models are trained on data that includes substantial amounts of AI-generated content. Some recent studies have suggested that models trained on AI-generated data can collapse in performance over time. But this phenomenon can also happen deliberately, as seen in Section 3.3. An attacker might poison the data used to train a model in order to alter and manipulate a model’s operation, for example to make it generate content that discriminates against a specific ethnic group. In both cases, the changes to an AI model’s training set can impact its performance and/or create undesirable side effects.

The issue of poisoning the well suggests organizations need to exercise caution when using synthetic data to train an LLM, adopting data auditing practices and tracking metrics to ensure that the model outputs achieve a sufficient level of quality. An alternative approach is to design and use datasets that are strictly curated and verified to contain only authentic human-generated content, reducing the risk of model degradation.

For open-source LLMs, data protection responsibilities remain relevant and, in many cases, complex. While identifying the controller or processor roles can be challenging in collaborative open-source settings, these models are still subject to GDPR obligations. This means that even when a model is freely accessible and collaboratively developed, accountability measures must still be enforced.

Under the AI Act, open-source systems benefit from certain exemptions, such as reduced documentation requirements. These exemptions apply only to the finished systems themselves, and LLMs remain covered by the rules applicable to generalpurpose AI models. Some of the documentation requirements in Article 53 do not apply to open-source models, but others do. Furthermore, all rules on systemic risk apply to models that meet the relevant thresholds. Therefore, open-source projects aimed at developing LLMs, especially at the state of the art, need to pay attention to data protection and AI Act requirements.

14.3 Safeguarding measures for model use

TipLearning outcomes

By the end of this section, learners will be able to outline risks that must be assessed when an organization considers whether and how to deploy an LLM.

As discussed throughout the training module, even the best design practices cannot eliminate all data protection risks created by AI. This means that organizations deploying systems based on LLMs, or incorporating LLMs into the AI systems they develop, must themselves adopt some safeguards. To do so, they must understand the roles the LLM plays within the data processing architecture. In this section, we will discuss some measures that can be useful for data protection professionals as they seek the best responses to the risks created in specific contexts.

When an LLM is integrated as part of software development, particularly during finetuning, privacy concerns surrounding the data used and generated by the model are paramount. Organizations should understand how the LLM processes, stores, or shares data and where it fits within the broader workflows that use it. The first step towards such an understanding is a thorough evaluation of the instructions for use that the LLM developer is required to supply to downstream providers, a topic we cover in Section 11.2. Based on the issues flagged by the developer, an organization can consider initial safeguards to adopt and identify issues that require additional investigation.

To supplement that documentation, an organization might want to conduct its own black box tests and audits of the models they intend to do so, as discussed in Chapter 7. While those kinds of tests do not provide the same levels of insight that white-box practices can supply, they can help organizations flag some issues before they lead to harms in practice. Such a mapping exercise helps identify potential points of data exposure, especially if personal data is involved.

When fine-tuning a model, an organization must evaluate whether the data it uses for that purpose contains personal data. If so, it will need to determine the proper legal basis for processing data for a purpose distinct than the one that motivated the original processing of that personal data, under the conditions examined in Section 6.3. For example, if a customer support system uses a fine-tuned LLM to provide responses based on prior interactions, the organization will need to obtain the consent of those data subjects or establish the presence of a suitable basis for further processing, as required by Article 6(4) GDPR.

A critical component of deploying LLMs responsibly involves scrutinizing the input data that will be fed into the model. Since LLMs often rely on vast amounts of data to improve responses or recommendations, it is essential to ensure that this data is free from unnecessary personal information. Implementing strict input filtering and anonymization protocols can reduce the risk of processing personal data inadvertently. For instance, sensitive identifiers such as names, addresses, and financial details should be either stripped out or obfuscated before data is inputted into the LLM. Additionally, data minimization principles must be observed, ensuring that only the essential data required for the model’s functionality is used.

It is also important to evaluate how the LLM provider might use any personal data submitted to or generated by the model. Many LLM providers retain certain types of data to improve model performance or conduct diagnostics, which may lead to secondary processing risks. Organizations should scrutinize service-level agreements and data processing agreements to understand what, if any, access the provider has to this data and whether additional safeguards, like encryption and access controls, are in place.

To mitigate the risk of unauthorized data access, organizations may opt for self-hosted models or models that can run locally, ensuring that data does not leave their controlled environment. Furthermore, if the LLM provider will engage in processing the data, specific contractual clauses should be enforced to limit such access strictly to what is necessary for service delivery.

When using an LLM-powered tool, attention should also be given to the data generated by the model itself. The outputs generated by LLMs can potentially contain or create personal data, which poses additional data protection challenges. For instance, if an LLM generates summaries or recommendations that incorporate individual user preferences or behaviours, these outputs may qualify as personal data under data protection law if they can be associated with an identified or identifiable natural person. In this regard, it is essential to apply accuracy checks on generated content to prevent misinformation or inaccurate profiles. Regular monitoring of the model’s outputs, coupled with ongoing adjustments to the model’s parameters, can help maintain the reliability and relevance of the information produced.

Depending on the purposes of the system that uses the model, the organization might want to adopt input filters that prevent the model from being used for certain purposes such as the generation of hate speech. It might also adopt output filters that prevent some of the outputs generated by the LLM from reaching its end user, for example by preventing a chatbot from outputting swear words or discriminatory terms. However, one must also be aware that such methods are often subject to subversion (“jailbreaking”), which might or not be addressable through further design of the system’s inputs.

To ensure compliance, organizations should also incorporate a process for regular audits and reviews of the LLM’s performance and its handling of data. This includes both technical assessments, such as testing the effectiveness of data anonymization measures, and compliance reviews, such as verifying that user consent is up to date and aligned with the intended data processing purposes. Establishing a clear auditing trail for data inputs, outputs, and consent records will help maintain transparency and accountability in line with data protection laws.

14.4 Conclusion

In this unit, we have looked more closely at a specific type of AI model, which powers a growing number of applications that process personal data. The three sections above are not enough to exhaust the complexities of the topic, but they provide a starting point for further analysis. By drawing on the discussions above, you will be able to flag issues that require further attention and investigate them in the context of particular systems. Furthermore, the same steps of analysis can be used to analyse other AI technologies that might be relevant to your work.

We can synthetize the main points of the previous discussion as follows:

  • LLMs are trained from vast amounts of data.
    • Most often, that data is scraped from the internet and other public sources, but one must keep in mind that publicity does not exclude the need for lawfulness in data processing.
    • This information is likely to include personal data of individuals, and some of it might not even contain falsehoods about the identified (or identifiable) persons.
  • LLMs operate as “black boxes,” making it difficult to explain how they process data and generate outputs.
  • The complexity of LLMs may hinder the exercise of rights such as access, rectification, and erasure.
  • Many of the measures for risk mitigation and elimination discussed in previous units can be tailored to deal with the specifics of LLMs.
  • The complexity of LLMs and their need for data has some implications for their diffusion.
    • Few actors might have the capabilities to develop or to host LLMs at scale.
    • Fine-tuning those models is considerably less expensive and is more accessible for organizations and individuals.
    • There is an asymmetry of knowledge between upstream and downstream providers, and so obligations need to be well-distributed between them.
  • A downstream provider using a ready-made system powered by an LLM, or incorporating an LLM into their own system, must evaluate which safeguards and protective measures they can implement.
    • Those measures might include organizational measures regarding how the LLM is used.
    • They might also include technical measures such as filtering input and output data or fine-tuning the model to address known risks.
  • Given the capabilities of advanced LLMs and their widespread use, some of them might trigger the AI Act’s thresholds for systemic risk, triggering additional legal requirements for their providers.

To translate these insights into action, focus on building robust collaboration frameworks with AI developers. By-design measures such as those discussed in Chapter 12 can be useful both for model developers and for system developers relying on an LLM provided by somebody else. And, fundamentally, compliance checks must cover every stage of the AI lifecycle. Finally, the fast evolution of these technologies means that more concrete guidance is likely to emerge in the near future. Therefore, it is particularly important to stay updated of recent developments in data protection law.

Exercises

Exercise 1. Which of the following highlights a key limitation of LLMs from a data protection perspective?

  • a. They cannot process human language, only replicate it.
  • b. They may generate inaccurate outputs or “hallucinations.”
  • c. They only use anonymized data.
  • d. They are incapable of creating personal data.
  • e. They have no commercial applications.

Exercise 2. From a data protection perspective, what is an implication of training LLMs with publicly available data?

  • a. Public availability does not negate data protection obligations.
  • b. Public data does not require any legal basis for processing.
  • c. When data is made public, it is no longer considered personal data.
  • d. Data subjects have no data protection rights once their data is made public.
  • e. Public data always aligns with purpose limitation principles.

Exercise 3. What is a data protection risk when relying on LLM outputs for decisionmaking?

  • a. Outputs could inadvertently expose sensitive proprietary information about the algorithm.
  • b. Outputs may erode public confidence in the media.
  • c. Outputs might encourage overreliance on automation without human oversight.
  • d. Generated outputs may contain false personal data.
  • e. Outputs might be indistinguishable from human-generated content.

Exercise 4. How should the maintainers of open-source LLMs approach the question of data protection compliance?

  • a. Assume that their collaborative nature exempts them from data protection requirements.
  • b. Avoid implementing privacy safeguards since open-source models are publicly accessible.
  • c. Collaborate with data protection experts on robust accountability measures and safeguards.
  • d. Focus exclusively on the technical performance of the model, ignoring compliance issues.
  • e. Rely entirely on the model’s contributors to address data protection concerns.

Exercise 5. Which of the following practices is NOT aligned with GDPR and AI Act compliance when fine-tuning an LLM?

  • a. Verifying the lawful basis for using personal data during fine-tuning.
  • b. Adopting strict input filtering to ensure only relevant data is processed.
  • c. Obtaining consent from data subjects if fine-tuning involves sensitive personal data.
  • d. Monitoring the model’s outputs for accuracy and potential biases.
  • e. Assuming that fine-tuning does not require additional documentation if the LLM provider has already completed it.

14.4.1 Prompt for reflection

LLMs create not only privacy-related risks but also systemic risks under the AI Act, such as the spread of misinformation or harmful applications at scale. How should organizations anticipate and address potential misuse of LLMs? Are technical safeguards, like input filtering, sufficient, or do they require broader societal and regulatory interventions? How can organizations mitigate these risks while still leveraging the advantages of LLMs?

14.4.2 Answer sheet

Exercise 1. Alternative B is correct, as it might clash with the accuracy principle. The remaining alternatives are inaccurate representations of LLMs.

Exercise 2. Alternative A is correct.

Exercise 3. Alternative D is correct. Other alternatives signal prominent issues about LLMs which are not, however, covered by data protection law.

Exercise 4. Alternative C is correct. Alternative E is plausible, but not feasible, as the developer community might lack some of the data protection expertise needed to tackle the issue. The remaining alternatives overlook legal obligations that apply even for open-source LLMs.

Exercise 5. Alternative E is the incorrect one, as the AI Act’s rules on general-purpose AI models do not exclude the responsibility of downstream providers for any data processing during fine-tuning.

References

CNIL’s Q&A on the Use of Generative AI (18 July 2024). Accessed 26 September 2024.

Data Protection Commission. AI, Large Language Models and Data Protection (18 July 2024). Accessed 26 September 2024.

Mindy Nunez Duffourc, Sara Gerke and Konrad Kollnig, ‘Privacy of Personal Data in the Generative AI Data Lifecycle’ (2024) 13 NYU Journal of Intellectual Property & Entertainment Law 219.

EDPS, ‘Generative AI and the EUDPR. First EDPS Orientations for Ensuring Data Protection Compliance When Using Generative AI Systems.’ (European Data Protection Supervisor 3 June 2024).

Lilian Edwards and others, ‘Private Ordering and Generative AI: What Can We Learn From Model Terms and Conditions?’ (CREATe Working Paper, CREATe 2024).

Florence G’sell, ‘An Overview of the European Union Framework Governing Generative AI Models and Systems’ (Stanford Cyber Policy Center working paper, 20 May 2024).

Philipp Hacker and others, ‘Regulating Gatekeeper Artificial Intelligence and Data: Transparency, Access and Fairness under the Digital Markets Act, the General Data Protection Regulation and Beyond’ (2024) 15 European Journal of Risk Regulation 49.

Andrei Kucharavy and others (eds), Large Language Models in Cybersecurity: Threats, Exposure and Mitigation(Springer 2024).

Taner Kuru, ‘Lawfulness of the mass processing of publicly accessible online data to train large language models’ (2024) International Data Privacy Law.

Paul Ohm, ‘Focusing on Fine-Tuning: Understanding the Four Pathways for Shaping Generative AI’ (21 June 2024).

OWASP, Top 10 for Large Language Model Applications (OWASP 2025).

David Rosenthal, ‘Part 19: Language models with and without personal data’ (Vischer, 2024).

Ilia Shumailov and others, ‘AI Models Collapse When Trained on Recursively Generated Data’ (2024) 631 Nature 755.

  1. In addition to risks that go beyond individual privacy and data protection, such as their potential for misuse and the creation of systemic risks. For instance, LLMs can be used to generate realistic but entirely false content, fuelling misinformation or political disinformation campaigns. These capabilities can undermine public trust, incite political instability, and even manipulate individuals’ opinions by spreading fake news or impersonating public figures. Such risks are covered by the AI Act’s rules on generalpurpose AI models, as well as by other legal instruments, but they exceed the data protection focus of this training module.↩︎

  2. As they depend on quantitative metrics that are currently met by a handful of models, such as GPT-4o or Google’s Gemini.↩︎