Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software

Abstract: The increasing availability of Machine Learning (ML) models, particularly foundation models, enables their use across a range of downstream applications, from scenarios with missing data to safety-critical contexts. This, in principle, may contravene not only the models' terms of use, but also governmental principles and regulations. This paper presents a preliminary investigation into the use of ML models by 173 open-source projects on GitHub, spanning 16 application domains. We evaluate whether models are used to make decisions, the scope of these decisions, and whether any post-processing measures are taken to reduce the risks inherent in fully autonomous systems. Lastly, we investigate the models' compliance with established terms of use. This study lays the groundwork for defining guidelines for developers and creating analysis tools that automatically identify potential regulatory violations in the use of ML models in software systems.

• The paper finds that 26% of high-risk projects use ML for autonomous decisions, frequently omitting the required human-in-the-loop safeguards.
• The paper demonstrates its empirical methodology through manual code review and analysis of post-processing patterns across diverse high-risk domains.
• The paper highlights significant non-compliance with model terms of use and EU AI regulations, urging the need for automated compliance auditing and improved safety measures.

Early Evidence of Non-Compliant ML Automation in Open-Source High-Risk Applications

Introduction

The study "Machine Learning in the Wild: Early Evidence of Non-Compliant ML-Automation in Open-Source Software" (2603.29698) provides a systematic investigation into the integration of ML models in high-risk open-source software and their compliance with both regulatory frameworks, such as the EU AI Act, and the models' own terms of use (ToU). The analysis focuses on the use of foundation models and their automation capabilities within mission- and safety-critical domains, revealing pervasive non-compliance and raising substantive safety, ethical, and legal concerns.

Study Design and Methodology

The authors conduct an empirical analysis of 173 open-source Python projects, selected from 16 application domains prioritized according to their classification as "high-risk" by the EU AI Act. High-risk markers include domains such as healthcare, autonomous driving, finance, cybersecurity, and legal compliance. The selection pipeline cross-references projects tagged by relevant application domain keywords on GitHub with direct dependencies on the Hugging Face transformers library, ensuring attention to current ML practices and foundation model adoption.

Three primary research questions guide the manual analysis:

1. Quantify the prevalence and characteristics of ML-based automated decision-making.
2. Characterize the post-processing safeguards layered atop ML-driven automation.
3. Identify the extent of violations against models' terms of use and legal requirements.

Manual code and documentation review, control flow analysis, and tracing of ML inference implementation were employed for extracting fine-grained detail regarding decision logic and compliance.

Automated Decision-Making in High-Risk Domains

Among the 173 high-risk projects, the investigation finds that 45 (26%) employ ML models as autonomous (or semi-autonomous) decision-support or control mechanisms. This cohort is predominantly concentrated within healthcare and autonomous systems. Within this group, 36 projects execute fully automated decisions, where model outputs directly instantiate actions or recommendations without enforced human intervention, while 9 projects implement a human-in-the-loop pattern, requiring active human oversight prior to action finalization.

The nature of automated decisions spans a spectrum from AI-based recommenders and control systems (e.g., driving function in autonomous vehicles) to financial advisory, threat detection, governance, compliance, and even direct generation of therapeutic interventions. Notably, a nontrivial portion of the systems move beyond benign recommendations—several projects directly drive actuation or resource allocation, thereby magnifying operational risk.

Post-Processing of ML Outputs

A significant finding is that—contrary to naïve end-to-end ML integration narratives—most projects employing ML-driven automation (40 of 45) introduce application-level post-processing to mitigate error propagation and non-deterministic model behavior. Classifiable post-processing patterns include:

• Output Structuring: Regular expression-based normalization to enforce output schema or correct model compliance errors.
• Output Consistency Checking: Domain-specific, expert-encoded rules ensuring outputs adhere to physical, legal, or operational constraints (e.g., medical drug interaction guards, control boundary conditions).
• Auto-Refinement Loops: System-level feedback and iterative re-evaluation of decisions, often with historical simulation or optimization-in-the-loop to enhance reliability (e.g., in trade automation and trajectory planning).
• Multi-Agent Coordination: Ensembling or arbitration among multiple ML agents for consensus-based decision selection, especially in critical domains (e.g., computer-aided diagnosis).
• Confidence Thresholding: Application of probabilistic cutoffs to gate downstream actuation.

However, a subset of systems (5 projects) deliberately bypass all post-processing, exposing themselves to the full spectrum of LLM/model failure modes.

Non-Compliance with Terms of Use and Regulation

The study highlights a pronounced disregard for both ML model terms of use (ToU) and statutory requirements exemplified by the EU AI Act. Of the 45 projects employing ML automation, 25 (~56%) are found to contravene explicit ToU clauses, especially prohibitions on automated high-stakes decisions without human validation (e.g., OpenAI GPT and Google Gemini restrictions), or the use of models in safety-related, healthcare, legal, or financial domains. All identified violations are associated with proprietary LLM usage; compliance issues are less pronounced (or less readily determinable) among open-weight models due to less stringent ToU.

Crucially, all 45 projects, by design, fall under "high-risk" according to the EU AI Act annex—thus requiring, as a matter of EU law, the implementation of human-in-the-loop (HITL) or comparable oversight mechanisms. The majority do not conform, implementing either incomplete or absent HITL gating.

Implications and Prospects

These observations have significant ramifications:

• Practical Risk Exposure: The current state exposes users and developers to substantial operational, ethical, and legal risks. Deployment of non-compliant ML-driven automation in real-world safety-critical environments could lead to regulatory sanctions, civil liability, and direct harm.
• Algorithmic Trustworthiness: Reliance on foundation models for high-risk automation without adequate safeguards undermines trust in open-source AI systems and reinforces the necessity of verifiable compliance layers in ML integration toolchains.
• Audit and Governance Needs: There is an emergent need for automated compliance auditing tools capable of tracking ToU conformance, regulatory obligations, and enforcing HITL requirements in ML system deployment, especially in open-source ecosystems.
• Model and Library Provider Responsibility: The findings challenge model providers and platforms (e.g., Hugging Face, OpenAI, Google) to embed compliance affordances, usage pattern detection, and up-to-date ToU guidance directly into their deployment APIs.
• Ecosystem Co-Evolution: As regulatory frameworks (EU AI Act, US NIST AI RMF, etc.) continue to mature and enforceability increases, the disconnect between permissive development practices and statutory constraints will necessitate both adaptation in development culture and advancements in automated legal validation technology.

Future Directions

Research directions stemming from the findings include: scalable static and dynamic analysis methods for compliance verification; automated detection of HITL gating in codebases; model card augmentation for enforceable ToU; and the development of developer-facing guidance and compliance templates, potentially leveraging the capabilities of LLMs themselves for semi-automated policy verification.

Conclusion

The systemic integration of ML automation in high-risk open-source projects proceeds largely without compliance to model-specific ToU or statutory regulation, particularly HITL mandates. While ad-hoc post-processing is widespread, it is insufficient to ensure regulatory fidelity or mitigate all operational threats. The study provides direct empirical evidence of the compliance gap, motivating the development of tools and frameworks for actionable, scalable, and enforceable adherence to evolving AI governance standards (2603.29698).