5  The Inception of AI Technologies

TipLearning outcomes

By the end of this chapter, learners will be able to identify various interventions that can be made during the inception stage of a life cycle. In particular, they will learn how to:

  • inventory AI systems in their organization; and
  • classify AI systems based on the risk associated with their application.

AI systems and models are technical artefacts. They are created by somebody’s deliberate effort, usually as a tool for some purpose. As discussed in @ch-intro, this purpose is what will determine the applicable legal framework for systems and models. However, there is a huge gap between defining a purpose and actually creating a system that has a plausible claim at achieving that purpose. For example, some of the AI technologies we take for granted nowadays, such as largescale text generators, were conceivable for a long time, but only recently it become possible to implement them. If a purpose is achievable, it takes a considerable amount of technical work to create a system or model that can do it.

The inception stage of the AI life cycle represents the starting point of that work. It refers to that moment in time when an organization is about to begin an AI project. At that point in time, the organization must make several choices that will influence how it develops (or purchases) AI:

Those requirements, in turn, can reflect the perspectives of various stakeholders, such as different units within a business, potential clients of an AI-based product, or the communities affected by a proposed AI solution. Software engineering disciplines have developed various techniques to identify relevant stakeholders and elicit requirements that are relevant for an application. Such techniques will not be examined in depth here, but the materials indicated in the references offer an introduction to them.

What this chapter covers, instead, is the roles that data protection professionals can play during this stage of the life cycle. Section 5.1 sketches out those roles by highlighting the interplay between the purpose of an AI system or model and its role in data processing. Section 5.2 then looks at the challenges involved in creating and maintaining up to date an inventory of AI applications, which can be essential for evaluating compliance with data protection law and the AI Act. Finally, Section 5.3 discusses how AI systems can be classified into the various risk categories defined in the AI Act.

5.1 Data protection tasks in the inception stage

TipLearning outcomes

By the end of this section, learners will be able to illustrate how data protection professionals can be actively involved during the inception stage of an AI life cycle.

The inception stage is a strategic process, which takes place long before any data is processed by an AI system or model. Even in the absence of any actual processing, data protection law still creates obligations. Article 25(1) GDPR, for instance, stipulates that risks to data protection principles must be addressed also during “the determination of the means for processing”. While the specific technical means are chosen at the next step of the life cycle (see Chapter 6), this provision already creates some obligations at the inception of an AI system or model. Therefore, data protection professionals must already be active at this stage.

A data protection professional must evaluate the specific risks that can emerge at this stage. That is, they must consider whether decisions about the need to create an AI system (or model), its intended purpose, and its functional and non-functional requirements can have unacceptable implications at some later point. For example, the AI systems that InnovaHospital wants to build for medical diagnosis give origin to concerns about the proper handling of special categories of personal data. Addressing such issues at an early point might prevent an organization from committing to an AI system or model that will need to be changed or abandoned later.

5.1.1 Initial assessments of lawfulness for processing

One risk that data protection professionals can diagnose at the inception stage is that of unlawful processing. Even before a system or model is built, the purpose for which it is meant might already signal the need to look more closely at some aspects. Any system that DigiToys uses in its toys, for instance, will need to be built in a way that is compatible with the processing of the personal data of children.

The AI Act adds red lines to processing, as it prohibits the use of AI in some practices, but it also authorizes some kinds of processing of special categories of personal data to avoid biases in high-risk AI systems. Therefore, a data protection professional can tell an organization about the implications of how they frame an AI system (or model)’s purpose.

5.1.3 Contracting for AI

In particular, the data protection professional can offer valuable guidance when it comes to the assessment of the contracts between an organization and its AI providers. Many AI models (and even full-blown systems) are not developed in-house or bought as closed products but hired as services, as we discuss in more depth in Chapter 14. If that service is acquired between relatively equal parties, an organization might have the flexibility to include some clauses that facilitate its own performance of data protection clauses. For example, InnovaHospital might require its non-EU contractors to follow standard contractual clauses as a safeguard for the processing of personal data.

Such negotiations are not always possible. Because many AI systems and models are developed by large organizations, those providers have considerable power when setting the terms of purchase. A recent study of large language models (Edwards et al. 2024), for instance, has found that most of them are offered through contracts of adhesion, in which buyers can only accept or reject the proposed clauses.

In this context, a data protection professional will need to support an organization in evaluating whether those clauses are compatible with the organization’s own data protection duties. For example, if a provider denies to a deployer information about how its AI systems work, contracting with that provider might create a situation in which the deployer cannot comply with some of its duties, such as the information rights from Articles 13–15 GDPR discussed in Chapter 11.

5.1.4 Taking stock of how AI is used

Finally, a data protection professional can help an organization keep track of the AI systems it already has. Each of these AI systems will raise its own data protection issues, and so an unnoticed AI system is a likely source of data protection exposure.

Additionally, a global vision of what is going on within organization is important for understanding data protection issues that might emerge from the interaction between different systems and models. For example, it might be the case that the combination between two separate AI systems that DigiToys uses to analyse personal data supplies enough information for identifying the owners of some toys.

Such risks must be addressed as part of the overall strategy for data protection compliance. However, as we will discuss in the next session, determining what AI systems are in use within an organization is not always a straightforward task.

5.2 Mapping the uses of AI

TipLearning outcomes

By the end of this section, learners will be able to formulate approaches for identifying whether and how AI is used within their organization.

Any organization that develops or uses AI systems or models is subject to some legal obligations. In Part I of this training module, we saw that the shape of those obligations may vary with the type of technology in question and the context. Nonetheless, the organization is still bound to obligations regarding the AI system itself and any personal data it processes. Understanding whether AI is in fact in place is a necessary first step for discharging those obligations. Yet, it is entirely possible that an organization makes extensive use of AI without knowing it.

Consider two examples in which an organization benefits from the background use of AI.

  1. The UNw university has purchased access to a workplace software suit supplied by a large company OfficeCorp. To preserve its dominance in the workplace software market, OfficeCorp is always pursuing new ways to boost user productivity. And, with the latest developments in AI technologies, it has decided to use some AI-powered tools in its background work. One of those tools, for instance, tries to optimize meeting schedules across teams in the company by suggesting the best timeslots and proposing agendas. It does so without requiring the end-users of the tool—in this case, UNw staff—to interact with an AI system. Hence, its use of AI might even go unnoticed.
  2. The toy company DigiToys decides to outsource some of its data analysis to an external provider, AnalyticsRUs. From the contract between those two companies, AnalyticsRUs is obliged to follow best statistical practices and make sure that it uses data in accordance with data protection principles, such as minimization and (whenever possible) anonymization. That company, however, reserves the right to protect its trade secrets, and so it does not offer DigiToys direct access to information about whether and how AI is used for data analysis.

Both cases illustrate how an organization might not be aware of AI being used on its behalf. In the previous section, we saw that such ignorance is not an unavoidable consequence of relying on external providers: an organization can and should pursue information about AI used by third parties on its behalf. And, if an organization does not do so, that is not an excuse for sidestepping its legal duties under either the AI Act or the GDPR. Not knowing about the use of AI might prove to be a costly decision.

Ignorance about the use of AI can emerge even when AI systems and models are deliberately used within the organization. One reason for that is that innovation does not always start from the top. Some AI systems are not developed or used uniformly across an organization, appearing instead from the initiative of individuals or small teams. For example, a human resources professional at InnovaHospital might read a news article about ChatGPT and create a prototype tool that automatically summarizes the files of workers, reducing the number of documents that need to be read for designing their career path. Or the professors at UNw’s computer science department might decide to create a chatbot that answers questions from students of the introductory programming course, freeing up more time for research and grant applications. These so-called shadow AI initiatives reflect initial inspiration, but often remain undetected until they are successful—or lead to harms to an organization’s interests. If individuals and groups fail to report the use of AI, or believe such uses can go unreported, it might be a long time before their existence come to the attention of a data protection professional.

Other aspects of an organization’s culture might contribute to lack of visibility about the use of AI systems. At UNw, for instance, there is a massive rivalry between the departments of computer science and electrical engineering, both of which carry out research on AI. So, if one of those departments decides to create an AI system for its own purposes, as in the example above, it is unlikely to involve the other, and it might even try to keep the use of the system secret. Such secrecy can affect an organization in several ways, ranging from the waste of effort in creating (or buying) systems that already exist elsewhere in the organization to a lack of proper supervision of systems that go unnoticed. Therefore, knowing what is going on within an organization is an important starting point for the inception of any AI tools.

5.2.1 Reasons for maintaining an AI inventory

The most straightforward approach to this issue is to create an inventory of AI systems that exist within an organization. A data protection professional needs to know about AIbased processing to oversee any processing operations. If they organize that information into a structured form, such as a list or an internal database of AI uses within an organization, they will then gain a comprehensive view of what AI technologies are in use and how they interact with one another.

At the end of the day, each organization is free to organize this information in any way they want. It might not have a centralized list, or it might not store much information about each AI system. Still, an organization would benefit from having easy access to the information they need to comply with existing legal requirements. Otherwise, it might need to gather that information each time they need to demonstrate legal compliance with a data protection or AI Act requirement. This section does not provide a comprehensive set of information that should be kept about each AI system or model. It focuses, instead, on the task of listing those systems and models in the first place.

Such a list, in itself, would already facilitate conformity with some legal obligations. Awareness that AI is being used for a particular application contributes to the AI literacy that every organization providing or deploying AI must foster under Article 4 AI Act. Providers of high-risk AI systems listed in Annex III AI Act are obliged to register those systems before they are placed on the market or put into service, an obligation that falls on the deployer of some public-sector applications. Furthermore, awareness of the use of AI is needed if data subjects decide to exercise their rights to obtain meaningful information about how a decision involving an AI system or model is produced. This list is not meant to exhaust the duties in which information about the use of AI is relevant, just to exemplify why an inventory can be useful for overall compliance.

5.2.2 Building the AI inventory

But what can a data protection professional do to build or update an inventory of AI systems? Comprehensive coverage of all AI systems in an organization will require a good strategy for collecting information, as well as a constant effort to ensure the inventory stays up to date. There is no alternative to a thorough examination of data practices to ensure nothing is being missed. But the following paragraphs highlight a few issues that one should not overlook.

5.2.2.1 Communication as a source of information about AI systems

First, communication is essential to ensure that no AI system or model is being overlooked. By gathering information from departments within the organization, as well as from individuals, the data protection professional can find systems and models that would otherwise escape their attention. They can also establish themselves as a reference point for AI within the organization, thus creating a virtuous circle in which individuals and departments proactively supply information about AI.

Consider a few forms of communication that might be relevant:

  1. The data protection professional can make direct requests of information to departments about any AI projects that they might be using. This approach is likely to yield information about systems that a department is developing, or that it is aware of using. But a data protection professional might need to assist the department with guidance to find AI systems being used in the background.
  2. A good rapport between the data protection professional and the organization’s inner structures might lead to voluntary reporting of information about AI. For example, the UNw data protection officer might position themselves as a neutral observer and thus establish contact with both rival departments.
  3. Data protection professionals might gain valuable information from addressing individual queries. For example, a human resources specialist at InnovaHospital might contact the hospital’s DPO if they are not sure that the new tool they are using relies on AI. After investigation, the data protection professional might find out that AI is indeed being used. Even if it is not, this investigation is likely to uncover relevant data processing that needs to be addressed.

5.2.2.2 Classification uncertainties

Second, the data protection professional’s habitual diligence is particularly relevant given the various uncertainties about AI mapped in Part I of this training module. Despite the technical and social complexities of AI systems, the information obtained through communication processes cannot be taken at face value. Instead, the professional must carry out a dialogue with the technical experts within the organization (and at the external providers) to evaluate the legal implications of the systems being analysed.

For example, a department within an organization might say that their new tool is an AI system in order to secure extra funding or to gain prestige from using advanced technologies. But, upon further inspection, their system might not meet the AI Act’s criteria for an AI system. Or, conversely, an individual software developer within an organization might want to avoid mentioning their use of AI in a project in order to avoid having to deal with internal compliance requirements before a project is mature enough. So, any information about a supposed AI system must be thoroughly verified before it is added to the inventory.

5.2.2.3 Keeping the inventory up to date

Third, the inventory must be updated often:

  • Even if an organization does not develop its own AI systems (or develops just a few of them), its data processors might decide to use AI for some reason. In the UNw example, OfficeCorp’s business decision to use AI was taken without prior notification to the university, which would only find out about it after the new AI systems were implemented.
  • Some AI systems that were originally listed can be deactivated.
  • The legal classification of an AI system under the AI Act might change over time. For example, Article 7 AI Act allows the European Commission to update the list of high-risk AI systems.

An inventory that is not verified from time to time might lose its usefulness or even become a misleading guide to AI in an organization.

5.2.2.4 Public or private inventories?

Fourth and finally, the organization must decide how much it wants to disclose the inventory. If an inventory is made accessible within the organization (or even to the public), departments and individuals might feel more inclined to supply information and keep it updated. Additionally, the availability of this information can generate value for businesses. As an example, the Brazilian judiciary has a list of AI models developed by each court, allowing other courts to benefit from tried-and-true solutions rather than creating their own system from scratch. Those benefits of transparency, however, might be outweighed by other organizational factors, such as the need to preserve trade secrets.

5.3 The purposes of AI technologies

TipLearning outcomes

By the end of this section, learners will be able to assign different AI systems to the various legal frameworks established by the AI Act.

Once a data professional knows what AI systems and models are used in an organization, they can start assessing the organization’s data processing architecture. Each system or model is used for one or more tasks, which often involve distinct kinds of personal data. Additionally, those systems are often interconnected, in the sense that the output of one system can act as the input for another. For example, InnovaHospital might use an AI-based solution to analyse the data generated by the interactions between patients and the hospital’s chatbots. It is necessary, therefore, to have a sharp vision of the function and the interactions between existing AI systems and models.

Here, it is important to distinguish between two kinds of purpose that are relevant here. On the one hand, each AI system or model is designed for a purpose, that is, for carrying out one or more specific tasks. The system or model’s purpose is relevant for determining the rules applicable under the AI Act. On the other hand, those systems and models might also process personal data, either during their training process or during their operation. As such, the purposes of each processing operation become relevant for the application of data protection requirements to the system. It is now time to examine those two kinds of purpose. The purposes of data processing will be examined more closely in other units of this training module. Now it is time to look more closely at the legal classification of purposes for AI systems and models.

Models and systems are subject to different legal frameworks under the AI Act. However, as discussed in Chapter 1, all those frameworks are organized around the purpose for an AI model or system. General-purpose models which can be used to power systems designed for various purposes—are subject to special rules, some of which apply to every general-purpose. If a model can only be used for a specific purpose, then it is not directly regulated. Still, conformity with the rules that apply to an AI system in light of its purpose might require changes to the model powering it. So, one must determine why an AI system is being used in order to find out the applicable rules.

5.3.1 Prohibited AI applications

Article 5 features a list of prohibited AI practices. In some cases, those practices are themselves illegal or at least questionable, but the addition of AI would have the potential to amplify the harms. One such prohibition is that of Article 5(1)(a), which bans the use of AI to materially distort the behaviour of a person or group of persons in a way that causes or is likely to cause them to harm themselves or others. This means, for instance, that an application that uses AI to incite conflicts within a society is likely to be unlawful.

Other prohibitions deal with practices that are not in themselves unlawful but become risky when done at the scale enabled by AI. For example, Article 5(1)(f) bans the use of emotion inference systems in the workplace or in educational institutions. This provision also illustrates another feature of the AI Act’s system of prohibitions: they sometimes allow for exceptions. In this case, the use of emotion inference systems is allowed when the system is intended to be put in place or into the market for medical or safety reasons. Those exceptions can be quite big, as shown by the fact that most of Article 5 AI Act is dedicated to laying down conditions in which law enforcement can use real-time biometric identification systems that are theoretically prohibited by Article 5(1)(h). But, without such an exception, no number of technical safeguards can allow the lawful use of a system covered by Article 5.

5.3.2 High-risk AI systems

The AI Act distinguishes between two types of high-risk AI systems. Some systems are classified as such because they are used in products that are, in themselves, subject to harmonized product safety law at the EU level. Others are classified as such because the EU lawmaker has deemed that the risks they create to fundamental rights, democracy, and the rule of law are big enough to warrant special attention. For the most part, those two kinds of high-risk AI systems are largely subject to the same rules. Still, there are a few differences between the classification procedures applicable to each one.

5.3.2.1 Product safety law

When it comes to systems already covered by product safety law, Article 6(1) AI Act establishes two conditions for the high-risk classification.

The first condition relates to the intended purpose of the system: the system must be a product covered by the product safety legislation listed in Annex I AI Act, or a safety component of such a product. For example, the toys produced by DigiToys are regulated by the Toy Safety Directive, and so they meet this first requirement. Likewise, if InnovaHospital decides to incorporate AI into medical devices, those devices might be covered by the existing regulations on medical and in vitro diagnostic devices. So, they would meet this first criterion.

The second condition for the application of rules on high-risk AI comes from product safety law itself. Under Article 6(1)(b) AI Act, an AI system is classified as high-risk if the product in which it is used must undergo a third-party conformity assessment before being placed on the market:

  1. In the case of DigiToys, the company has decided that such an assessment is necessary due to the nature of their products and the lack of technical standards applicable to smart toys, which means they also become subject to the AI Act’s rules.
  2. In the case of InnovaHospital, classification is more contextual, as the applicable regulations have a complex mechanism for determining which applications require third-party assessments.

But, whenever a device without AI would need such an assessment, the use of AI means the rules on high-risk systems become applicable.

5.3.2.2 Risks to public values

Many of the high-risk applications covered by the AI Act have no precedent in product safety law. Article 6 AI Act stipulates that all systems listed in Annex III are high-risk, unless they are covered by one of the derogations present in Article 6(3). Scholars and civil society organizations have pointed out that there is no underlying logic to this list of high-risk applications. It reflects, instead, applications that the EU lawmaker saw as creating particular risks to public values, in particular the protection of fundamental rights, democracy, and the rule of law.

The AI Act’s recitals offer guidance about the risks that the lawmaker associated with some—but not all—of the applications listed as high-risk. But, for the most part, the determination of which risks apply in each context is left to the deployers and providers of those systems. Therefore, those actors need to know whether they systems are covered by one of the various rubrics of Annex III AI Act.

5.3.2.3 AI systems posing a high risk to fundamental rights

Annex III to the Act indicates eight types of application considered as high-risk:

  1. Biometrics, in so far as the use of AI is permitted under EU or national law.
  2. Operation and management of critical infrastructure.
  3. Education and vocational training.
  4. Employment, workers management, and access to self-employment.
  5. Access to and enjoyment of essential private services and essential public services and benefits.
  6. Law enforcement, in so far as the use of AI is permitted under EU or national law.
  7. Migration, asylum, and border control management, in so far as the use of AI is permitted under EU or national law; and
  8. Administration of justice and democratic processes.

Within each point, the EU lawmaker has designated one or more applications considered high-risk. Any other applications within that domain are not classified as such. For example, the use of AI for risk assessment in pricing in relation to natural persons for life and health insurance is covered under Point 5 above. By exclusion, risk assessment and pricing with regard to the insurance of legal persons or of other types of insurance for natural persons is exempt from the rules on high-risk AI. So, the rules on high-risk AI under Annex III only apply to a narrow set of applications designated as especially risky.

5.3.2.4 Derogations from the high-risk classification

A system might escape the rules on high-risk AI even if it is designed for a listed purpose. Article 6(3) AI Act stipulates that a system is not considered high-risk if it does not “pose a significant risk of harm to the health, safety or fundamental rights of natural persons”. In general lines, this derogation covers situations in which the AI system plays only a marginal role in the outcomes.

As currently formulated, this is the case whenever one of the four conditions below applies:

  • The system is intended to perform a narrow procedural task, such as automatically archiving the work done by a human.
  • It is intended to improve the result of a previously completed human activity, for example by improving a report written by a human analyst.
  • It is intended to detect decision-making patterns or deviations from prior decision-making, without replacing or influencing human assessment, for example by flagging whether a transaction involves a much larger monetary value than usual; or
  • The AI system is intended to perform a preparatory task to an assessment relevant for the purposes listed in Annex III, leaving the assessment itself in human hands.

The examples listed above are merely indicative, as the degree of human involvement in a particular case might mean that the derogation is not applicable. Determining its applicability requires a contextual assessment. The only general rule in that regard is that any system that carries out profiling in the context of Annex III applications is considered high-risk, regardless of any subsequent human involvement.

The application of those derogations falls entirely to providers. Under Article 6(4) AI Act, a provider must evaluate whether their system is covered by one of the derogations above. If they consider that is the case, they must document their assessment before the system is put on the market or placed in the service. That documentation can be requested by national competent authorities at a later stage, who might question the provider’s assessment. Until and unless an authority does so, the provider is obliged to register the system in an EU-wide database for high-risk AI systems but is not subject to any other of the obligations surrounding high-risk systems.

5.3.3 Specific rules for specific purposes

The purpose of an AI system is also relevant for determining whether some of the AI Act’s special rules are applicable. When it comes to high-risk AI, Article 27 stipulates that some deployers of those systems must carry out a fundamental rights impact assessment of the deployed system. This obligation covers all deployers that are governed by public law, or that are private entities providing public services. It also applies to deployers using AI systems for evaluating creditworthiness of natural persons (including by credit scoring) and for risk assessment and pricing for life and health insurance. Therefore, the high-risk legal framework can suffer some adjustments depending on the purpose of the system.

Furthermore, Article 50 AI Act establishes some requirements that AI systems must observe regardless of their risk classification:

  1. Providers of systems meant to interact directly with natural persons must design and develop the system in a way that allows natural persons to be informed that they are interacting with AI.
  2. Providers of AI systems generating synthetic audio, image, video, or text content must ensure that the outputs are market in a machine-readable format and detectable as being generated or manipulated by AI.
  3. Deployers of an emotion recognition or a biometric categorization system must inform the natural persons exposed to that system of its operation; and
  4. Deployers of an AI system that generates or manipulate image, audio, or video content constituting a deep fake must disclose that the content has been artificially generated or manipulated.

All those rules admit exceptions, which must be analysed on a contextual basis.

5.4 Conclusion

The safe use of AI requires close attention to the purposes for which AI systems and models are proposed. If that purpose is in itself unlawful, no amount of software engineering can make it acceptable in the eyes of the law. But even if the purpose is lawful at a first glance, different purposes raise different risks to data protection principles and other values protected by the law. As such, the purpose for which a system (or model) is built or used affects the legal duties to which a provider or deployer must comply.

Some of the risks created by AI might be anticipated early on. To do so, the previous sessions have highlighted a few good practices:

  1. Creating and keeping updated an inventory of AI systems within an organization.
  2. Involving various stakeholders within an organization in the process of keeping that inventory up to date.
  3. Evaluating the contractual terms under which popular AI tools are offered.
  4. Helping business stakeholders identify potential implications for data protection of the requirements they are mapping for an AI system.
  5. Using the inventory to evaluate whether a system interferes with (or is interfered by) other systems within the organization; and
  6. Assessing the purposes of AI systems before they move on to the development stage.

After completing those preliminary tasks, and organization should have a much clearer picture of what they use AI for and what needs a new system would address. This suggests that involving a data protection professional at the earliest stages of AI development can help organizations avoid not only legal liability for breaches of the law but also the duplicated work that would come from redundant or even unlawful AI projects.

For the data protection professional, this early assessment process also contributes with visibility of the AI systems and models that will process personal data within the organization. Classifying those systems in accordance with the AI Act is also important to understand whether and how the general data protection obligations are modified by any AI-specific obligations. In the next stages of the AI life cycle, we will focus on the various data protection obligations that emerge throughout the life cycle.

Exercises

Exercise 1. What is the primary purpose of the inception stage of an AI project?

  • a. Conducting real-time data processing.
  • b. Determining the organizational need for AI and its specific requirements.
  • c. Deploying AI systems in various departments.
  • d. Collecting as much personal data as possible.
  • e. Analysing the final output of an AI model.

Exercise 2. Which of the following is a non-functional requirement for an AI system?

  • a. A chatbot must answer 90% of questions correctly.
  • b. A diagnostic tool must outperform human doctors.
  • c. A recommendation system must provide personalized suggestions.
  • d. A translation model must support five languages.
  • e. An AI system must adhere to GDPR’s data minimization principle.

Exercise 3. Why might an AI inventory be critical for organizations?

  • a. A complete inventory might allow an organization to avoid applying the GDPR to some of its less relevant systems.
  • b. To centralize all AI-based decision-making into one system.
  • c. To identify AI systems and ensure compliance with legal obligations.
  • d. To minimize stakeholder involvement in AI decisions.
  • e. To protect trade secrets from external review.

Exercise 4. When might an AI system used for one of the AI Act’s high-risk purposes escape the additional obligations established in the Act?

  • a. When it only performs narrow procedural tasks, without conducting profile.
  • b. When it promotes fundamental rights.
  • c. When it uses machine learning algorithms that are not advanced.
  • d. When it involves profiling in law enforcement contexts.
  • e. When it operates with minimal human supervision.

Exercise 5. When analysing a contract for the provision of an AI-based service, what is a data protection professional likely to focus on?

  • a. The cost-effectiveness of the service.
  • b. Whether the AI model powering the service reflects the technical state of the art.
  • c. Whether the provider of this service controls their own technology controls the intellectual property used to provide it.
  • d. Whether the provider supplies enough information for the organization to comply with its own disclosure duties.
  • e. Whether the service is compatible with an organization’s existing tools for data processing.

5.4.0.1 Prompt for reflection

The chapter emphasizes the creation and maintenance of AI inventories to ensure compliance and accountability. Why do you think organizations might struggle with this task, and what practical strategies can a data protection officer implement to overcome these challenges? Consider how organizational culture, such as the rivalry at UNw or the reliance on third-party providers like DigiToys (or examples from your own organization!), might impact this process.

5.4.1 Answer sheet

Exercise 1. Alternative B is correct. Alternatives A, C, and E refers to tasks that happen in subsequent life cycle stages, while alternative D is at odds with the GDPR’s data minimization principle.

Exercise 2. Alternative E is correct. All other alternatives relate to system features.

Exercise 3. Alternative C is correct. An inventory is a tool for compliance and, potentially, transparency, not for strategic compliance. An organization might also decide to centralize AI-based decisions, but that is independent of an inventory.

Exercise 4. Alternative A is correct, according to Article 6(3) AI Act. Alternative B represents one potential use of AI, but positive impact on fundamental rights is not in itself a reason for exclusion. Likewise, alternative C is not sufficient to avoid the application of the rules to high-risk systems, as technical criteria are only used in the determination of whether the rules on general-purpose AI models with systemic risk apply, not for AI systems. Alternative D falls into an exception to Article 6(3) AI Act, related to profiling. Finally, Alternative E makes it unlikely, but not impossible, that a system will meet the criteria for the high-risk exemptions.

Exercise 5. Alternative D is correct, as it is the only one focusing on data protection duties. Alternative C might also have some relevance in practice, to the extent that intellectual property is invoked to prevent the controller from receiving information. Other alternatives remain within an organization’s freedom to choose the means it will use to comply with data protection obligations.

References

‘ISO/IEC/IEEE International Standard - Systems and Software Engineering – Software Life Cycle Processes’ [2017] ISO/IEC/IEEE 12207:2017(E) 1.

Marco Almada and Nicolas Petit, ‘The EU AI Act: Between the Rock of Product Safety and the Hard Place of Fundamental Rights’ (2025) 62 Common Market Law Review.

Rashidah Kasauli and others, ‘Requirements Engineering Challenges and Practices in Large-Scale Agile System Development’ (2021) 172 Journal of Systems and Software 110851.

F Khomh and others, ‘Software Engineering for Machine-Learning Applications: The Road Ahead’ (2018) 35 IEEE Software 81.

Ralf Kneuper, Software Processes and Life Cycle Models: An Introduction to Modelling, Using and Managing Agile, Plan-Driven and Hybrid Processes (Springer International Publishing 2018).

David Lehr and Paul Ohm, ‘Playing with the Data: What Legal Scholars Should Learn About Machine Learning’ (2017) 51 UCDL Rev 653.

Silverio Martínez-Fernández and others, ‘Software Engineering for AI-Based Systems: A Survey’ (2022) 31 ACM Trans Softw Eng Methodol 1.

Mariana Maia Peixoto, ‘Privacy Requirements Engineering in Agile Software Development: A Specification Method’ in (CEUR-WS 2020) Joint Proceedings of REFSQ-2020.

Luke Stark and Jevan Hutson, ‘Physiognomic Artificial Intelligence’ (2022) 32 Fordham Intellectual Property, Media and Entertainment Law Journal 922.

Rob van der Veer, ‘ISO/IEC 5338: Get to know the global standard on AI systems’ Software Improvement Group. Accessed 26 September 2024.

Titus Winters and others (eds), Software Engineering at Google. Lessons Learned from Programming over Time(O’Reilly 2020).