digital · family: the answer is in a field you’ve never heard of
trustedinparts,trustedbynoone
Heterogeneous AI systems cannot be verified end-to-end for trustworthiness because trust frameworks are fragmented across communities
Problem statement
Real-world AI systems are not monolithic neural networks — in the words of a 2024 Dagstuhl seminar report, they are "often assembled by integrating many heterogenous technologies – including machine learning, symbolic reasoning or optimization." Each of those technologies has its own research community, and the report finds that "the notion of trust is fragmented and bespoke for the individual communities." How trust in the components adds up to trust in the whole system is still open. The seminar brought together researchers from AI, machine learning, robotics, hardware and software verification, software engineering, and the humanities (especially philosophy), and its report lists among its open questions how to reason about responsibility, blame and causal factors affecting trustworthiness in individual components, and how that should "be expanded to the overall AI system." The report also notes that recent manifestos, position papers, and policy recommendations on AI ethics have focused primarily on fairness, accountability, and transparency (FAT), with most of the attention on fairness because it "seems relatively easier to define precisely."
Why this matters
Autonomous systems, the report says, are "increasingly being deployed in safety-critical environments whilst interoperating with humans," and it expects them to make mistakes: "accidents will surely happen despite best efforts." When such a system fails, someone has to work out which part was responsible: the report asks how to reason about responsibility, blame and causal factors affecting trustworthiness, and, given a falsification of a specification, what automated diagnosis, proof-theoretic and causal tools are needed "to identify problematic components." Regulation does not settle this. The report describes existing regulations as "necessarily vague, depending on the nuance of human interpretation for actual implementation," and asks what tools can be given to regulators, verification and validation professionals, and system designers to help them clarify the intent and content of regulations "down to a machine interpretable form." One talk at the seminar argued that risk management aimed at minimizing harm is required for trustworthy AI compatible with the EU AI Act; another said that the increasing use of AI in autonomous systems "has made the problem of certifying such systems hard."
What’s been tried and why it hasn’t worked
The report records work from several of these communities. On the logic side, one seminar talk described a hybrid agent architecture in which the agents are logical, so that the reasons for a decision can be exposed, and the agent's decision-making can be formally verified. On the machine learning side, the report mentions formal verification of ML models, including deep learning architectures, and recent work on the robustness properties of newer networks — but it still lists as an open question whether the tools available to verify ML components cover what users and regulators understand as correct behavior. DARPA's XAI agenda, the report says, "has led to a resurgence in diagnostic explanations, but also ignited the question of interpretability and transparency in machine learning models, especially deep learning architectures." Causal reasoning matters for attributing responsibility when things go wrong — the report calls causality "central to defining a notion of responsibility" — but it observes that conventional ML models "focus on statistical associations, which can be misleading in critical applications demanding human-understandable explanations." How to "unify tools from causal reasoning and verification for assessing the correctness of complex AI systems" remains one of the report's open questions. The result is that each community has its own notion of trust for its own components, and trust in the system as a whole is left as an open question.
What would unlock progress
The report does not prescribe a solution; what it offers is a list of open questions. They ask how properties of AI systems can be "expressed in languages that lend themselves to formal verification or quantitative analysis"; which software engineering practices are relevant for interfacing and integrating these notions; how to unify tools from causal reasoning and verification; and, when a specification is shown to fail, what tools are needed to identify the problematic components. Read together, these describe a way of composing component-level properties (proved, tested, or argued) into system-level trustworthiness claims, with explicit assumptions about how the components interact — that framing is this brief's reading, not the report's wording. The seminar's stated purpose was to initiate a debate with the overall aim of laying the foundations for a "Trustworthiness & Responsibility in AI" framework, a systems development methodology that "integrates quantifiable responsibility and verifiable correctness into all stages of the software engineering process." The report presents this as foundations still to be laid, with open questions and partial answers, not as a current capability; the seminar page says the planned main outcome was a blueprint of such a framework, grounded in causality and verification.
Entry points for student teams
A team could build a small heterogeneous autonomous system (e.g., a robot that uses an ML vision model for perception, a symbolic planner for navigation, and a rule-based safety controller) and attempt to construct a formal trust argument spanning all three components. The exercise would reveal exactly where current trust methods break down at component boundaries. The deliverable would be a documented "trust gap analysis" identifying which properties can be verified, which can only be tested, and which cannot currently be assured at all. This is a feasible software engineering / formal methods project. Skills needed: familiarity with at least two of ML, formal methods, and software architecture.
Genes
Structural cousins: stuck the same way, in other fields
Sources
"Trustworthiness and Responsibility in AI — Causality, Learning, and Verification," Dagstuhl Seminar 24121, Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl Reports, Vol. 14, Issue 3, pp. 75–91, 2024. DOI: 10.4230/DagRep.14.3.75. (accessed 2026-02-12) go to source ↗
Working notes
Notes made while writing and checking this brief, shown as written. They use working shorthand.
- Seminar organizers intentionally bridged AI, ML, robotics, hardware/software verification, software engineering, and philosophy — the breadth of disciplines reflects the fragmentation this brief describes.
- Directly related to existing brief `DIGITAL-autonomous-systems-formal-verification` (the neural network verification sub-problem) and the new brief `DIGITAL-autonomous-system-runtime-resilience` (the runtime assurance sub-problem). This brief addresses the broader compositional problem: how to combine different kinds of assurance across heterogeneous components.
- The report touches regulation in several places: the executive summary (existing regulations are necessarily vague; what tools to give regulators, verification and validation professionals, and system designers), one talk (Leon Kester, "AI Safety and The EU AI Act") tying trustworthy AI to risk management compatible with the EU AI Act, another talk (Joseph Halpern) noting that the European Union's draft AI act mentions "harm" over 25 times, and a breakout session summary on "Effective and Balanced AI Regulation." This is the basis for `constraint:regulatory`. The report says nothing about conformity assessment.
- The `failure:disciplinary-silo` tag is the central diagnosis: trust frameworks exist within individual communities but don't interoperate. This is not a technical limitation waiting for a clever algorithm — it is a coordination and language problem across research communities.
- The report credits DARPA's XAI agenda with a resurgence in diagnostic explanations and with raising the question of interpretability and transparency in machine learning models; it does not assess the program further.
Reconciliation 2026-10-01: Fact repair after reading the whole write-up against its source. Two things were read, both fetched on 2026-10-01 and both in full: the seminar page on the Source line (https://www.dagstuhl.de/en/seminars/seminar-calendar/seminar-details/24121) and the published report at the DOI on the Source line (https://doi.org/10.4230/DagRep.14.3.75, which leads to https://drops.dagstuhl.de/storage/04dagstuhl-reports/volume14/issue03/24121/DagRep.14.3.75/DagRep.14.3.75.pdf, 17 pages). The report is a short document: an executive summary, a list of open questions, one-paragraph talk abstracts and a breakout summary. Most of the earlier text went beyond it. (1) Problem Statement. The text listed perception models, planners, rule-based safety systems and human-in-the-loop interfaces as the components, said "there is no methodology" for composing component trust into system trust, and said FAT principles "lack formal, compositional verification methods." The report names machine learning, symbolic reasoning and optimization; it asks, as an open question, how reasoning about individual components should be expanded to the overall AI system; and it says only that recent manifestos and position papers have focused on FAT, mostly fairness. The paragraph now says that. The quoted phrase "fragmented and bespoke for the individual communities" is in the report word for word and stays. The list of disciplines now follows the report (hardware and software verification; the humanities, especially philosophy). (2) Why This Matters. Removed because the report does not contain them: the NIST AI Risk Management Framework; the statement that demand for trustworthy AI "outpaces the technical ability to deliver it"; the autonomous vehicle that misclassifies a pedestrian; the statements that the EU AI Act will require conformity assessments for high-risk systems and that "no conformity assessment methodology exists" for mixed learning and logic systems; and "emergent behavior." The paragraph now gives what the report says: deployment in safety-critical environments, that accidents will happen, that existing regulations are necessarily vague, the question of what tools to give regulators, one talk linking trustworthy AI to risk management compatible with the EU AI Act, and one talk saying certification of AI-enabled autonomous systems has become hard. (3) What's Been Tried. Removed because the report does not contain them: the general statements about what model checking and theorem proving can and cannot do; test accuracy and robustness benchmarks; that DARPA's XAI program could not check whether explanations are faithful and that explanations "don't compose"; safety cases and GSN notation; and that ML systems "cannot reliably perform" counterfactual analysis. The paragraph now reports the hybrid agent talk, the report's mention of formal verification of ML models, its one sentence on DARPA's XAI agenda (quoted), its statement that conventional models focus on statistical associations (quoted), and its open question on unifying causal reasoning and verification. (4) What Would Unlock Progress. The four requirements (shared formal languages, interface verification, causal attribution tools, lifecycle integration as components are retrained) were presented as established needs; the report has them only partly, as open questions. The paragraph now lists the report's open questions, labels the compositional framing as this brief's own reading, and adds the seminar page's statement that the planned main outcome was a blueprint of such a framework. The text said the seminar "proposed" the framework and "acknowledged this is a research vision"; the report says the purpose was to initiate a debate with the aim of laying the foundations for such a framework, and that it obtained partial answers. (5) Source Notes. The note on EU AI Act conformity assessment and `temporal:worsening` was replaced with the places where the report touches regulation (executive summary, the Kester and Halpern talks, the breakout on regulation). The note on cardiac pacemakers and IEC 62304 was removed; neither is in the report. The note that DARPA's XAI program "inadvertently demonstrated" that interpretability and verification are disconnected was replaced with the report's one sentence. Confirmed in the source and left as written: the Source line (title, Dagstuhl Seminar 24121, Dagstuhl Reports Vol. 14, Issue 3, pp. 75–91, DOI 10.4230/DagRep.14.3.75; the report dates the seminar March 17–22, 2024); the mix of disciplines in the first Source Note. Not checked because they are this project's own suggestions and cross-references, not claims about the source: the student entry point, and the two related briefs named in the second Source Note. The two titles were left as they are; note that the long title states flatly that end-to-end checking of such systems is impossible, which is stronger than the report, which treats system-level trust as an open question. A second reader went over the repairs on the same day: re-read both sources in full, confirmed every quotation in the write-up against the report text, kept every removal (none of the removed material appears in either source), and tightened five sentences so they say no more than the report does (the hybrid-agent talk, the tools-for-regulators question, the problematic-components question, and the opening sentences of the last two sections). Genes untouched.