Evaluating human intervention in automated decisions

Photo by charlesdeluvio at Unsplash

Article 22 of the GDPR sets out the rights of individuals of not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects on them or similarly significantly affects them, with some exceptional situations.

With regard to the concept of "Decision based solely on automated processing", this is developed in section IV.A of  the WP 251 Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679, as last revised and adopted on 6 February 2018:

Article 22(1) refers to decisions ‘based solely’ on automated processing. This means that there is no human involvement in the decision process.

The controller cannot avoid the Article 22 provisions by fabricating human involvement. For example, if someone routinely applies automatically generated profiles to individuals without any actual influence on the result, this would still be a decision based solely on automated processing.

To qualify as human involvement, the controller must ensure that any oversight of the decision is meaningful, rather than just a token gesture. It should be carried out by someone who has the authority and competence to change the decision. As part of the analysis, they should consider all the relevant data.

As part of their DPIA, the controller should identify and record the degree of any human involvement in the decision-making process and at what stage this takes place. 

Many decisions that are commonly thought of as automated actually involve some degree of human intervention. However, human participation has to be active and not just a symbolic gesture, that is, it has to have a certain degree of relevance and capacity. 

WP251 Guidelines show an example that is not based solely on automated processing:

Example: An automated process produces what is in effect a recommendation concerning a data subject. If a human being reviews and takes account of other factors in making the final decision, that decision would not be ‘based solely’ on automated processing. 

An example of the opposite is shown in the documentation produced by the ICO:

A factory worker’s pay is linked to their productivity, which is monitored automatically. The decision about how much pay the worker receives for each shift they work is made automatically by referring to the data collected about their productivity.

The CJEU of 7 December 2023, case C-634/21, "SCHUFA", establishes that it is also an automated decision within the meaning of art.22 of the GDPR, when it is the automatic generation of a value from personal data that is transmitted by the data controller to a third party, also a data controller, and this third party, in a decisive way, bases a decision about the person on that value.

In the context of a processing with possible automated decision-making, an assessment of the degree of human involvement must be carried out. Assessing whether human supervision is possible and effective involves evaluating both the system used and the processing and its context. In order to carry out this assessment systematically, it is recommended to objectively assess a person's participation in the decision-making process as follows:
1.    Competence.
That is, they have the authority or task assigned to them that allows them to alter the outcome of the automated decision.
An example would be an inmate applying for third-degree or parole. To make a decision on this request, a report is prepared on the inmate which, for example, includes an algorithmic prediction of the risk of recidivism. A prison officer does not have the competence to modify this prediction or the decision that is made based on it, only the obligation that this decision (made by a judge) be enforced.
2.    Preparation and trainning.
That is, they have the ability and aptitude to evaluate the decision and the factors that determine that decision in relation to the context of the processing and the automated system used, on its capabilities and limitations.
For example, in the case of an automated health diagnosis, the intervention of a health professional in the specialty related to the diagnosis will be necessary to be able to dispute the decision with knowledge of the facts.
In other cases, in addition to basic training, the person must have the necessary training to know the peculiarities of specific decisions in the real contexts of operation.
3.    Independence and diligence in the exercise of their powers.
It is necessary to evaluate whether there are pressures from the organization or from outside the organization that coerce the dispute of the decision by the person. 
For example, it may be the case that investing in the automated system discourages you from questioning it. Another example is that the number of decisions in dispute is limited or that the fact of disputing them entails some kind of prejudice to the competent person or others.
The competent persons themselves, or those who supervise them, may also be conditioned by the bias of authority, which leads them not to question (or to do so on very rare occasions) decisions made in an automated way. 
This requires continuous review of the decision process, even that the supervision is carried out by more than one person in some cases. Such a review must include assurances about the diligence of the person in the performance of their obligations.
4.    Means to be able to exercise their competence and qualifications. 
With regard to the means needed to assess the degree of intervention, it should be assessed that the person has the following skills available:

1. That procedurally he or she can exercise its competence. 
   This presupposes that the procedure that frames the automate decision contemplates the ability to intervene at the right time or point in a timely manner, before applying that automate decision to the individual. In addition, if the procedure is digitalized, the systems that implement it must allow the person the ability to intervene in any circumstance.
   For example, it may be the case that the legal effects, or that they significantly affect the data subject, occur in an insufficient period of time for a person to be able to exercise their competence, as is the case of taking transport (a flight, for example), attending an event, accessing a service such as a gym, etc.
2. That they have the necessary information in a timely manner to be able to exercise their qualification.
   They must be able to know the consequences, the risks of decisions in general, and those that are being made for specific cases.
   They must be able to know all the aspects that condition the automated decision. These include the data of the individual but could also include the procedures for the collection of the input data, the data implicit in the model that generates the decision, the contextual data that has not been taken into account in the automated decision, as well as the capabilities and limits of the decision system.  Also, those data that the person, in his or her qualification, considers necessary to take into account for the specific case and that have not been considered in the automated decision.
3. That they have the resources to be able to exercise their qualification.
   For example, it is necessary for them to have applications that allow analyse the information in the format that is being used for the automated decision, or to convert that format to the needs of the person. Also, for resources that they need to process the information in the optimal way or those that a team needs to support the supervision.
4.  That they have the necessary time to be able to exercise their qualification for each of the decisions that fall within their competence.
   A person may be competent, qualified, and have the pre-resource needs met. However, if they're faced with a work regime that requires to oversee decisions on 100 reports of 100 pages each, every day, this forces to read and analyse 21 pages per minute. In general, that would not be feasible. And it can also happen that they may have additional tasks due to their position that they have also to attend, which would make human supervision de facto unfeasible.

This post is related to some other materials published by the Innovation and Technology Division of the AEPD, such as:

• Risk Management and Impact Assessment in the Processing of Personal Data
• GDPR compliance of processing that embed Artificial Intelligence. An introduction
• AI: System vs Processing, Means vs Purposes
• Artificial Intelligence: accuracy principle in the processing activity