An critical assessment of the concepts used by De Azevedo et al. (2022) and the derived hypothesis
Introduction
The third approach, RE4AI from De Azevedo et al. (2022), is essentially a refined version of the ECCOLA method. Thus, it is also an iterative add-on process, which is applicable in the entire development process by following the three operationalisation steps (prepare, review, and evaluate), again with the use of a deck of cards. While there are minor differences between ECCOLA and RE4AI, such as the consideration of more principles (11 in RE4AI vs. 8 in ECCOLA), the following concepts seem to be the most prominent difference.
Last Concept: Enriching Procedural Methods with Technical Tools
As in ECCOLA, the cards in RE4AI also consist of a motivation, a how-to-tackle-the-issue, and a practical example part. However, the note-taking part is replaced by a tool-suggestion part, which refers to technical tools that support the implementation of the ethical issue. Specifically, the current technical tools are matched with the set of principles used and listed on each respective card.
Evaluation Regarding the First Developer Challenge: Time Pressure
- The concept adopts the inherent challenge of toolkits by engendering a misguided confidence in the resulting overall ethical alignment.
- In addition, while matching tools with principles, the concept does not provide guidance on when to use the tools as well as the specific requirements and underlying assumptions of each tool. This could lead to mistakes and, thus, to subsequent changes that can cause a disproportionate time effort, as in the ‘change something, change everything’ concept.
- Furthermore, the toolkits are not evenly distributed among the respective cards, meaning that some cards refer to multiple toolkits while others have no references at all. This could lead to a harmful practice of ignoring certain principles, not because they are unimportance in the specific context, but rather because of missing toolkits. As a result, ignoring important principles could again lead to mistakes and, thus, to the need for subsequent changes that are time intensive.
- However, referring developer teams to suitable toolkits also increases their awareness of the existence of these kits, which will decrease the time-effort to find the tools and enhance their use.
Evaluation Regarding the Second Developer Challenge: Conflicting Approaches
- Focusing on the second developer challenge, matching principles with toolkits corresponds to the straightforward problem-solving approach and the training of developers. This can result in an actionable outcome and, thus, in a high likelihood of implementation.
Evaluation Regarding the Third Developer Challenge: Mushy Stuff
- Lastly, with reference to the third developer challenge, the provision of toolkits also reduces the abstract nature of principles through the provision of actionable steps.
- However, it is important to note that toolkits provide a simplification that can only be meaningfully used in specific contexts and that this concept does not fully embrace the context-specific and conflicting nature of principles.
Hypothesis Derivation
To conclude this evaluation, it is evident that this concept supports developer teams in all three challenges since it directly refers them to actionable items that match developers’ working style and increase ethical awareness. However, these tools simplify ethical principles and, thus, potentially do not solve all the ethical issues of an AI system, leading to a misguided confidence in the overall ethical alignment. Nevertheless, this thesis views the translation of principles into actionable items as a major strength in terms of usefulness and, thus, hypothesises that (H5) embedding the translation of principles into toolkits using procedural methodologies is positively associated with the perceived usefulness by developer teams for every AI project.