Reasoning About Responsibility in Autonomous Systems: Navigating the Challenges and Charting Future Directions
DOI:
https://doi.org/10.71346/utj.v1i2.21Keywords:
Autonomous Systems, Responsibility, Explainable AI, Ethical Principles, Hybrid ReasoningAbstract
As autonomous systems gain prominence in sectors such as transportation, healthcare, and finance, the challenge of assigning responsibility for their actions has become increasingly critical. Existing legal, ethical, and technical frameworks often fall short in addressing the unique characteristics of these systems, which include opaque decision-making processes, emergent behavior, distributed control, and learning from biased data. This paper investigates the core challenges involved in reasoning about responsibility within autonomous systems by focusing on issues such as the black-box problem, the unpredictability of outcomes, the complexity of multi-agent environments, and the evolving role of human oversight. It reviews and analyzes a range of potential solutions, including explainable AI techniques, formal specification & verification methods, agent-based simulations, ethics-oriented design principles, and hybrid reasoning models that combine symbolic & sub-symbolic approaches. By connecting these methods to real-world domains and incidents, the paper offers a structured understanding of how responsibility can be clarified and embedded into the design & governance of autonomous systems. This research contributes novel analytical perspectives and practical pathways that can support the more accountable deployment of AI technologies while laying the groundwork for future interdisciplinary probe into responsible autonomy.
References
V. Yazdanpanah et al., “Reasoning about responsibility in autonomous systems: challenges and opportunities,” AI & Society, vol. 38, no. 4, pp. 1453–1464, Dec. 2022, doi: 10.1007/s00146-022-01607-8.
J. Sifakis and D. Harel, “Trustworthy autonomous system development,” ACM Transactions on Embedded Computing Systems, vol. 22, no. 3, pp. 1–24, Jun. 2022, doi: 10.1145/3545178.
M. Dastani and V. Yazdanpanah, “Responsibility of AI systems,” AI & Society, vol. 38, no. 2, pp. 843–852, Jun. 2022, doi: 10.1007/s00146-022-01481-4.
J. Dong, S. Chen, M. Miralinaghi, T. Chen, P. Li, and S. Labi, “Why did the AI make that decision? Towards an explainable artificial intelligence (XAI) for autonomous driving systems,” Transportation Research Part C Emerging Technologies, vol. 156, p. 104358, Sep. 2023, doi: 10.1016/j.trc.2023.104358.
K.-C. Hsu, H. Hu, and J. F. Fisac, “The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems,” Annual Review of Control Robotics and Autonomous Systems, vol. 7, no. 1, pp. 47–72, Feb. 2024, doi: 10.1146/annurev-control-071723-102940.
M. Luckcuck, “Using formal methods for autonomous systems: Five recipes for formal verification,” Proceedings of the Institution of Mechanical Engineers Part O Journal of Risk and Reliability, vol. 237, no. 2, pp. 278–292, Jul. 2021, doi: 10.1177/1748006x211034970.
H.-M. Heyn, E. Knauss, and P. Pelliccione, “A compositional approach to creating architecture frameworks with an application to distributed AI systems,” Journal of Systems and Software, vol. 198, p. 111604, Jan. 2023, doi: 10.1016/j.jss.2022.111604.
N. H. Conradie, “Autonomous Military Systems: collective responsibility and distributed burdens,” Ethics and Information Technology, vol. 25, no. 1, pp. 1–14, Feb. 2023, doi: 10.1007/s10676-023-09696-9.
Jedličková, “Ethical approaches in designing autonomous and intelligent systems: a comprehensive survey towards responsible development,” AI & Society, vol. 39, no. 4, pp. 1–14, Aug. 2024, doi: 10.1007/s00146-024-02040-9.
M. M. Mayer, A. Buchner, and R. Bell, “Humans, machines, and double standards? The moral evaluation of the actions of autonomous vehicles, anthropomorphized autonomous vehicles, and human drivers in road-accident dilemmas,” Frontiers in Psychology, vol. 13, pp. 1–13, Jan. 2023, doi: 10.3389/fpsyg.2022.1052729.
M. A. Houghtaling et al., “Standardizing an ontology for ethically aligned robotic and autonomous systems,” IEEE Transactions on Systems Man and Cybernetics Systems, vol. 54, no. 3, pp. 1791–1804, Nov. 2023, doi: 10.1109/tsmc.2023.3330981.
D. Trusilo, “Autonomous AI systems in Conflict: Emergent behavior and its impact on predictability and reliability,” Journal of Military Ethics, vol. 22, no. 1, pp. 2–17, Jan. 2023, doi: 10.1080/15027570.2023.2213985.
G. Bakirtzis, S. Carr, D. Danks, and U. Topcu, “Dynamic certification for autonomous systems,” Communications of the ACM, vol. 66, no. 9, pp. 64–72, Aug. 2023, doi: 10.1145/3574133.
F. Rowe, M. J. Medina, N. B. Journé, E. Coëtard, and M. Myers, “Understanding responsibility under uncertainty: A critical and scoping review of autonomous driving systems,” Journal of Information Technology, vol. 39, no. 3, pp. 587–615, Sep. 2023, doi: 10.1177/02683962231207108.
D. Saenz, Z. Harned, O. Banerjee, M. D. Abràmoff, and P. Rajpurkar, “Autonomous AI systems in the face of liability, regulations and costs,” Npj Digital Medicine, vol. 6, no. 1, pp. 1–3, Oct. 2023, doi: 10.1038/s41746-023-00929-1.
L. Methnani, M. Chiou, V. Dignum, and A. Theodorou, “Who’s in charge here? a survey on trustworthy AI in variable autonomy robotic systems,” ACM Computing Surveys, vol. 56, no. 7, pp. 1–32, Feb. 2024, doi: 10.1145/3645090.
D. B. Acharya, K. Kuppan, and B. Divya, “Agentic AI: Autonomous Intelligence for Complex Goals – A Comprehensive survey,” IEEE Access, vol. 13, pp. 1–25, Jan. 2025, doi: 10.1109/access.2025.3532853.
H. Chakraa, F. Guérin, E. Leclercq, and D. Lefebvre, “Optimization techniques for Multi-Robot Task Allocation problems: Review on the state-of-the-art,” Robotics and Autonomous Systems, vol. 168, p. 104492, Jul. 2023, doi: 10.1016/j.robot.2023.104492.
X. Li, Z. Chen, J. M. Zhang, F. Sarro, Y. Zhang, and X. Liu, “Bias Behind the wheel: Fairness testing of autonomous driving systems,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 3, Art. no. 82, Nov. 2024, doi: 10.1145/3702989.
V. Jain, G. Gupta, M. Gupta, D. K. Sharma, and U. Ghosh, “Ambient intelligence-based multimodal human action recognition for autonomous systems,” ISA Transactions, vol. 132, pp. 94–108, Nov. 2022, doi: 10.1016/j.isatra.2022.10.034.
H. Araujo, M. R. Mousavi, and M. Varshosaz, “Testing, Validation, and Verification of Robotic and Autonomous Systems: A Systematic review,” ACM Transactions on Software Engineering and Methodology, vol. 32, no. 2, pp. 1–61, Jun. 2022, doi: 10.1145/3542945.
X. Yin, B. Gao, and X. Yu, “Formal synthesis of controllers for safety-critical autonomous systems: Developments and challenges,” Annual Reviews in Control, vol. 57, p. 100940, Jan. 2024, doi: 10.1016/j.arcontrol.2024.100940.
N. E. Fard, R. R. Selmic, and K. Khorasani, “Public Policy Challenges, regulations, oversight, technical, and ethical Considerations for autonomous Systems: a survey,” IEEE Technology and Society Magazine, vol. 42, no. 1, pp. 45–53, Mar. 2023, doi: 10.1109/mts.2023.3241315.
Holzinger, K. Zatloukal, and H. Müller, “Is Human Oversight to AI Systems still possible?,” New Biotechnology, vol. 85, pp. 59–52, Dec. 2024, doi: 10.1016/j.nbt.2024.12.003.
M. Adam, C. Diebel, M. Goutier, and A. Benlian, “Navigating autonomy and control in human-AI delegation: User responses to technology- versus user-invoked task allocation,” Decision Support Systems, vol. 180, p. 114193, Feb. 2024, doi: 10.1016/j.dss.2024.114193.
M. H. Cohen, T. G. Molnar, and A. D. Ames, “Safety-critical control for autonomous systems: Control barrier functions via reduced-order models,” Annual Reviews in Control, vol. 57, p. 100947, Jan. 2024, doi: 10.1016/j.arcontrol.2024.100947.
T. Ziemke, “Understanding Social robots: attribution of intentional agency to artificial and biological bodies,” Artificial Life, vol. 29, no. 3, pp. 351–366, Jan. 2023, doi: 10.1162/artl_a_00404.
K. Terzidis, F. Fabrocini, and H. Lee, “Unintentional intentionality: art and design in the age of artificial intelligence,” AI & Society, vol. 38, no. 4, pp. 1715–1724, Jan. 2022, doi: 10.1007/s00146-021-01378-8.
Rawal, A. Raglin, D. B. Rawat, B. M. Sadler, and J. McCoy, “Causality for Trustworthy Artificial intelligence: Status, challenges and perspectives,” ACM Computing Surveys, vol. 57, no. 6, pp. 1–30, May 2024, doi: 10.1145/3665494.
H. L. Fraser and N. P. Suzor, “Locating fault for AI harms: a systems theory of foreseeability, reasonable care and causal responsibility in the AI value chain,” Law Innovation and Technology, vol. 17, no. 1, pp. 103–138, Mar. 2025, doi: 10.1080/17579961.2025.2469345.
K. Shukla, V. Terziyan, and T. Tiihonen, “AI as a user of AI: Towards responsible autonomy,” Heliyon, vol. 10, no. 11, p. e31397, May 2024, doi: 10.1016/j.heliyon.2024.e31397.
P. Hacker, “The European AI liability directives – Critique of a half-hearted approach and lessons for the future,” Computer Law & Security Review, vol. 51, p. 105871, Sep. 2023, doi: 10.1016/j.clsr.2023.105871.
W. Xu, M. J. Dainoff, L. Ge, and Z. Gao, “Transitioning to Human Interaction with AI Systems: New Challenges and Opportunities for HCI Professionals to Enable Human-Centered AI,” International Journal of Human-Computer Interaction, vol. 39, no. 3, pp. 494–518, Apr. 2022, doi: 10.1080/10447318.2022.2041900.
V. Hassija et al., “Interpreting Black-Box Models: A review on Explainable Artificial intelligence,” Cognitive Computation, vol. 16, no. 1, pp. 45–74, Aug. 2023, doi: 10.1007/s12559-023-10179-8.
S. Nazat, O. Arreche, and M. Abdallah, “On evaluating Black-Box explainable AI methods for enhancing anomaly detection in autonomous driving systems,” Sensors, vol. 24, no. 11, p. 3515, May 2024, doi: 10.3390/s24113515.
E. ŞAHiN, N. N. Arslan, and D. Özdemir, “Unlocking the black box: an in-depth review on interpretability, explainability, and reliability in deep learning,” Neural Computing and Applications, vol. 37, pp. 859–965, Nov. 2024, doi: 10.1007/s00521-024-10437-2.
Kuznietsov, B. Gyevnar, C. Wang, S. Peters, and S. V. Albrecht, “Explainable AI for Safe and Trustworthy Autonomous Driving: A Systematic review,” IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 12, pp. 19342–19364, Jan. 2024, doi: 10.1109/tits.2024.3474469.
Novelli, M. Taddeo, and L. Floridi, “Accountability in artificial intelligence: what it is and how it works,” AI & Society, vol. 39, no. 4, pp. 1871–1882, Feb. 2023, doi: 10.1007/s00146-023-01635-y.
Y. Wang et al., “Decision-Making Driven by driver intelligence and Environment Reasoning for High-Level Autonomous Vehicles: A survey,” IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 10, pp. 10362–10381, May 2023, doi: 10.1109/tits.2023.3275792.
B. C. Stahl, “Embedding responsibility in intelligent systems: from AI ethics to responsible AI ecosystems,” Scientific Reports, vol. 13, no. 1, pp. 1–8, May 2023, doi: 10.1038/s41598-023-34622-w.
M. Taddeo, P. Jones, R. Abbas, K. Vogel, and K. Michael, “Socio-Technical Ecosystem Considerations: An Emergent Research Agenda for AI in Cybersecurity,” IEEE Transactions on Technology and Society, vol. 4, no. 2, pp. 112–118, Jun. 2023, doi: 10.1109/tts.2023.3278908.
Y. Himeur et al., “AI-big data analytics for building automation and management systems: a survey, actual challenges and future perspectives,” Artificial Intelligence Review, vol. 56, no. 6, pp. 4929–5021, Oct. 2022, doi: 10.1007/s10462-022-10286-2.
T. Kim, M. D. Molina, M. Rheu, E. S. Zhan, and W. Peng, One AI Does Not Fit All: A Cluster Analysis of the Laypeople’s Perception of AI Roles. ACM, 2023, pp. 1–20. doi: 10.1145/3544548.3581340.
M. Hanna et al., “Ethical and bias considerations in artificial intelligence (AI)/Machine learning,” Modern Pathology, vol. 38, no. 3, p. 100686, Dec. 2024, doi: 10.1016/j.modpat.2024.100686.
S. Albahri et al., “A systematic review of trustworthy and explainable artificial intelligence in healthcare: Assessment of quality, bias risk, and data fusion,” Information Fusion, vol. 96, pp. 156–191, Mar. 2023, doi: 10.1016/j.inffus.2023.03.008.
S. Grigorescu and M. Zaha, “CyberCortex.AI: An AI‐based operating system for autonomous robotics and complex automation,” Journal of Field Robotics, vol. 42, no. 2, pp. 474–492, Aug. 2024, doi: 10.1002/rob.22426.
K. Grosse and A. Alahi, “A qualitative AI security risk assessment of autonomous vehicles,” Transportation Research Part C Emerging Technologies, vol. 169, p. 104797, Sep. 2024, doi: 10.1016/j.trc.2024.104797.
C. Sieber, L. M. V. Da Silva, K. Grünhagen, and A. Fay, “Rule-Based verification of autonomous unmanned aerial vehicles,” Drones, vol. 8, no. 1, pp. 1–19, Jan. 2024, doi: 10.3390/drones8010026.
P. H. Padovan, C. M. Martins, and C. Reed, “Black is the new orange: how to determine AI liability,” Artificial Intelligence and Law, vol. 31, no. 1, pp. 133–167, Jan. 2022, doi: 10.1007/s10506-022-09308-9.
V. Chamola, V. Hassija, A. R. Sulthana, D. Ghosh, D. Dhingra, and B. Sikdar, “A review of Trustworthy and Explainable Artificial Intelligence (XAI),” IEEE Access, vol. 11, pp. 78994–79015, Jan. 2023, doi: 10.1109/access.2023.3294569.
Taylor, “Is explainable AI responsible AI?,” AI & Society, vol. 39, no. 2, pp. 1–10, Apr. 2024, doi: 10.1007/s00146-024-01939-7.
M. Z. Zgurovsky, P. O. Kasyanov, and L. B. Levenchuk, “Formalization of methods for the development of autonomous artificial intelligence systems,” Cybernetics and Systems Analysis, vol. 59, no. 5, pp. 763–771, Sep. 2023, doi: 10.1007/s10559-023-00612-z.
S. Li, M. Wei, S. Li, and X. Yin, “Temporal logic task planning for autonomous systems with active acquisition of information,” IEEE Transactions on Intelligent Vehicles, vol. 9, no. 1, pp. 1436–1449, Oct. 2023, doi: 10.1109/tiv.2023.3327312.
Z. Dimitrios and S. Petros, “Towards Responsible AI: A Framework for Ethical Design utilizing Deontic Logic,” International Journal of Artificial Intelligence Tools, vol. 33, no. 8, p. 2550003, Jan. 2025, doi: 10.1142/s0218213025500034.
Z. Guo and X. Liu, “How artificial intelligence cooperating with agent‐based modeling for urban studies: A systematic review,” Transactions in GIS, vol. 28, no. 3, pp. 654–674, Mar. 2024, doi: 10.1111/tgis.13152.
C. B.-V. Burken, “Value Sensitive Design for autonomous weapon systems – a primer,” Ethics and Information Technology, vol. 25, no. 1, pp. 1–14, Feb. 2023, doi: 10.1007/s10676-023-09687-w.
Platzer, “Intersymbolic AI,” in Lecture notes in computer science, vol. 15222, Springer LNCS, 2024, pp. 162–180. doi: 10.1007/978-3-031-75387-9_11.
D. Garikapati and S. S. Shetiya, “Autonomous Vehicles: Evolution of artificial intelligence and the current industry landscape,” Big Data and Cognitive Computing, vol. 8, no. 4, p. 42, Apr. 2024, doi: 10.3390/bdcc8040042.
E. Papagiannidis, I. M. Enholm, C. Dremel, P. Mikalef, and J. Krogstie, “Toward AI governance: identifying best practices and potential barriers and outcomes,” Information Systems Frontiers, vol. 25, no. 1, pp. 123–141, Apr. 2022, doi: 10.1007/s10796-022-10251-y.
Q. Lu, L. Zhu, X. Xu, J. Whittle, D. Zowghi, and A. Jacquet, “Responsible AI Pattern Catalogue: A collection of best practices for AI governance and engineering,” ACM Computing Surveys, vol. 56, no. 7, pp. 1–35, Oct. 2023, doi: 10.1145/3626234.
Palaniappan, E. Y. T. Lin, and S. Vogel, “Global Regulatory Frameworks for the use of Artificial intelligence (AI) in the healthcare services sector,” Healthcare, vol. 12, no. 5, p. 562, Feb. 2024, doi: 10.3390/healthcare12050562.
N. Díaz-Rodríguez, J. Del Ser, M. Coeckelbergh, M. L. De Prado, E. Herrera-Viedma, and F. Herrera, “Connecting the dots in trustworthy Artificial Intelligence: From AI principles, ethics, and key requirements to responsible AI systems and regulation,” Information Fusion, vol. 99, p. 101896, Jun. 2023, doi: 10.1016/j.inffus.2023.101896.
K. G. Hopster and M. M. Maas, “The technology triad: disruptive AI, regulatory gaps and value change,” AI And Ethics, vol. 4, no. 4, pp. 1051–1069, Jun. 2023, doi: 10.1007/s43681-023-00305-5.
I. Hauptman, B. G. Schelble, N. J. McNeese, and K. C. Madathil, “Adapt and overcome: Perceptions of adaptive autonomous agents for human-AI teaming,” Computers in Human Behavior, vol. 138, p. 107451, Aug. 2022, doi: 10.1016/j.chb.2022.107451.
N. Vallès-Peris and M. Domènech, “Caring in the in-between: a proposal to introduce responsible AI and robotics to healthcare,” AI & Society, vol. 38, no. 4, pp. 1685–1695, Dec. 2021, doi: 10.1007/s00146-021-01330-w.
C. Mennella, U. Maniscalco, G. De Pietro, and M. Esposito, “Ethical and regulatory challenges of AI technologies in healthcare: A narrative review,” Heliyon, vol. 10, no. 4, p. e26297, Feb. 2024, doi: 10.1016/j.heliyon.2024.e26297.
B. N. Thanh, H. X. Son, and D. T. H. Vo, “Blockchain: the economic and financial institution for autonomous AI?,” Journal of Risk and Financial Management, vol. 17, no. 2, pp. 1–20, Jan. 2024, doi: 10.3390/jrfm17020054.
Schmitt, “Automated machine learning: AI-driven decision making in business analytics,” Intelligent Systems With Applications, vol. 18, p. 200188, Jan. 2023, doi: 10.1016/j.iswa.2023.200188.
R. Hamon, H. Junklewitz, I. Sanchez, G. Malgieri, and P. De Hert, “Bridging the gap between AI and explainability in the GDPR: Towards Trustworthiness-by-Design in Automated Decision-Making,” IEEE Computational Intelligence Magazine, vol. 17, no. 1, pp. 72–85, Jan. 2022, doi: 10.1109/mci.2021.3129960.
W. Li, F. Paraschiv, and G. Sermpinis, “A data-driven explainable case-based reasoning approach for financial risk detection,” Quantitative Finance, vol. 22, no. 12, pp. 2257–2274, Sep. 2022, doi: 10.1080/14697688.2022.2118071.
S. He, “Who is Liable for the UBER Self-Driving Crash? Analysis of the Liability Allocation and the Regulatory Model for Autonomous Vehicles,” in Perspectives in law, business and innovation, 1st ed., Springer, 2020, pp. 93–111. doi: 10.1007/978-981-15-9255-3_5.
K. M. Bridges, “Race in the Machine: Racial Disparities in Health and Medical AI,” Virginia Law Review, vol. 110, no. 2, pp. 1–97, Apr. 2024, [Online]. Available: https://virginialawreview.org/articles/race-in-the-machine-racial-disparities-in-health-and-medical-ai/
B. L. Ranard et al., “Minimizing bias when using artificial intelligence in critical care medicine,” Journal of Critical Care, vol. 82, p. 154796, Mar. 2024, doi: 10.1016/j.jcrc.2024.154796.
B. Vaassen, “AI, opacity, and personal autonomy,” Philosophy & Technology, vol. 35, no. 4, pp. 1–20, Sep. 2022, doi: 10.1007/s13347-022-00577-5.
Kumar, A. Aijaz, O. Chattar, J. Shukla, and R. Mutharaju, “Opacity, transparency, and the ethics of affective computing,” IEEE Transactions on Affective Computing, vol. 15, no. 1, pp. 4–17, May 2023, doi: 10.1109/taffc.2023.3278230.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Usman Tariq, Irfan Ahmed
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors retain copyright for all articles published in CrossLink Studies journals. These articles are made freely available under a Creative Commons CC BY-SA 4.0, which allows unrestricted downloading and reading by the public.
