AI Institute for Transforming Education for Children with Speech and Language Processing Challenges

Abstract: It is estimated that more than 3.4 million children need speech and language related services in the US school system, yet there are less than sixty-one thousand speech-language pathologists (SLPs) to serve them. The COVID-19 pandemic has further exacerbated this gap, making it almost impossible for SLPs to provide individualized services for children. The AI Institute for Transforming Education for Children with Speech and Language Processing Challenges aims to close this gap by developing advanced AI technologies to scale SLPs' availability and services such that no child in need of speech and language services is left behind. Towards this end, the Institute proposes to develop two novel AI solutions: (1) the AI Screener to enable universal early screening for all children, and (2) the AI Orchestrator to work with SLPs to provide individualized interventions for children with their formal Individualized Educational Plan (IEP). In developing these solutions, the Institute will advance foundational AI technologies, enhance understanding of children's speech and language development, serve as a nexus point for special education stakeholders, and represent a fundamental paradigm shift in how SLPs serve children in need of ability based speech and language services. The AI Screener will be initially deployed in early childhood classrooms and will analyze video and audio streams of children's classroom interactions, derive conventional speech and language measures used by SLPs, and assess novel and hard to obtain automaticity measures. The AI Orchestrator is a superset of the AI Screener with its main application in the public school classrooms. It will help SLPs to administer a wide range of evidence-based interventions and assess their effects on meeting children's individual IEP learning targets. At the core of the Orchestrator is a robust multi-agent reinforcement learning framework that can evaluate the potential benefits of different intervention practices and recommend those most appropriate for each child. Both solutions will push significant advances in self-supervised learning to address sparse and noisy data issues, multimodality perception, learning material rewriting and enrichment, and edge AI for real time processing. The Institute will develop human centered AI design methodologies to embody the solutions in a form appropriate for children’s learning. Education research and the learning sciences will inform the initial prototyping and validation, and will derive valuable insights from the field deployed solutions. The National Center for Special Education Research at the Institute of Education Sciences of the US Department of Education is partnering with NSF to provide funding for the Institute. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

Details

  • Organization: National Science Foundation
  • Award #: DRL-2229873
  • Amount: $20,000,000
  • Date: Jan. 15, 2023 - Dec. 31, 2027
  • PI: Venugopal Govindaraju
  • Co-PI: Dr. David Feil-Seifer

Supported Publications

  • Oliva, D., Knight, J., Becker, J. T., Amistani, H., Parajuli, S., Nicolescu, M., & Feil-Seifer, D. Design Activity for Personalized Robot Faces: Evaluating Child Responses To Customizable Expressive Faces. Submitted to International Conference on Human-Robot Interaction (HRI), Melbourne, Australia, IEEE. Oct 2024. IEEE. ( details )
  • Schmidt-Wolf, M., Becker, J. T., Oliva, D., Nicolescu, M., & Feil-Seifer, D. Enhancing Human-Robot Collaboration by Investigating Legible Motion Patterns Through a Human Study. To Appear in International Conference on Robot and Human Interactive Communication (RO-MAN), Pasadena, USA, IEEE. Aug 2024. IEEE. ( details )
  • Oliva, D., Dahan, R., Sultan, R., Lopez, J., Gulia-Nuss, M., Nuss, A., Teglas, M., Feil-Seifer, D., & Harris, C. F. TickTrax: A Mobile and Web-Based Application for Tick Monitoring and Analysis. Oct 2023. ( details )
  • Schmidt-Wolf, M., Folmer, E., & Feil-Seifer, D. Comprehensive Feedback Module Comparison for Autonomous Vehicle-Pedestrian Communication in Virtual Reality. In International Conference on Social Robotics (ICSR2023), Doha, Qatar, Oct 2023. ( details )
  • Schmidt-Wolf, M., Becker, T., Oliva, D., Feil-Seifer, D., & Nicolescu, M. Exploring Legibility for Robot Motion in Cluttered Environments. Submitted to 2024 IEEE International Conference on Robotics and Automation (ICRA2024), Yokohama, Japan, Sep 2023. ( details )
  • Oliva, D., Schmidt-Wolf, M., & Feil-Seifer, D. Autonomous Systems on the Road: Enhancing Interactions Between Autonomous Vehicles and Bicyclists. Sep 2023. ( details )