Introduction: Artificial intelligence (AI) and machine learning (ML) increasingly shape decision-making in high-impact domains such as healthcare, yet most secondary students lack opportunities to understand how these systems function or reproduce inequities. Advancing AI literacy in K–12 education requires context-rich curricula paired with valid assessments that measure students’ understanding of foundational ML concepts. This study developed and provided initial validation evidence for an assessment of foundational ML understanding while examining student learning within a context-based healthcare AI curriculum.

Methods: We conducted a multi-year study in which high school students engaged in an 18-lesson curriculum integrating data science, AI, and ML through authentic healthcare applications. Students analyzed real-world datasets, built and evaluated ML models, and participated in a collaborative datathon with healthcare and data science professionals. A curriculum-aligned assessment measuring foundational ML concepts was administered before and after instruction, and psychometric analyses, as well as multiple linear and multilevel regression models were used to evaluate assessment quality and changes in student learning.

Results: Students entered the curriculum with limited but non-random knowledge of ML concepts and demonstrated statistically significant gains following instruction. The assessment also demonstrated improved reliability and item discrimination following instruction, suggesting that students developed more coherent conceptual knowledge of ML. At the same time, the findings identified several issues with the assessment and some of its items that warrant further refinement.

Conclusion: This study provides initial validation evidence for one of the few assessments of foundational ML concepts developed for secondary students and provides evidence that a context-based healthcare curriculum can support foundational ML learning. The findings also highlight the importance of addressing both authentic contexts and continued assessment development to support the technical and societal dimensions of improving AI literacy.