Key facts
- U.S. health agencies are rapidly deploying AI agents for therapy and prescription.
- Concerns exist regarding the safety and transparency of AI in healthcare.
- The FDA saw a 148% increase in AI use between FY 2024-2025.
- Generative AI use cases across federal agencies increased nine-fold from 2023 to 2024.
- Older rule-based algorithms were transparent and auditable, unlike newer AI systems.
- Programmer errors and data flaws in older AI systems led to improper disenrollments from Medicaid.
U.S. health agencies are rapidly expanding their use of artificial intelligence, with federal projects deploying AI agents capable of offering therapy and prescribing medicine. This accelerated adoption, driven by goals of boosting efficiency, accuracy, and cost savings, has raised significant concerns about safety, transparency, and the potential for harm, particularly as AI systems become more complex and opaque.
According to the U.S. Department of Health and Human Services (HHS) AI Use Case Inventory, the FDA saw a 148% surge in AI use between FY 2024 and 2025, while the CDC experienced an 87% increase, CMS a 78% rise, and the NIH a 51% jump. States are also increasing their AI adoption. A recent U.S. Government Accountability Office (GAO) report found that while overall AI use cases across 11 federal agencies doubled from 2023 to 2024, generative AI use cases increased nine-fold, with health-based agencies leading the trend.
Historically, health agencies relied on rule-based algorithms that were transparent and auditable. However, the shift towards more advanced AI, including machine-learning models and generative large language models (LLMs), presents challenges. Even simpler algorithms have led to issues; for instance, programmer errors and data flaws in Medicaid eligibility systems resulted in hundreds of thousands of improper disenrollments, leading to class-action lawsuits. The increasing complexity of AI makes errors harder to detect and correct, posing a greater risk in high-stakes domains like healthcare where even rare mistakes can have serious consequences for patients.