The “Other Me”: Human-Centered AI Assistance In Situ

Project Description

Imagine a situated AI assistant embedded in your smart glasses and collaborating with you to create a novel artifact, e.g., building an Arduino smart lamp. It communicates with you through the natural language to understand your goals and preferences. It works with you to develop and execute the solution. It sees what you see from a first-person view to understand your actions and offer useful tips. It connects with other AI assistants to scale up to complex tasks, e.g., assembling or disassembling an aircraft engine, which require a team solution. AI assistants become the “other me”, an external, but situated extension of our individual minds and senses.  The “Other Me” (TOM) is an integrative program of fundamental research to enable wearable, situated AI assistance for everyone in daily life and work. 

Research Technical Area

  • Decision-making
  • Planning
  • Man-machine Collaboration
  • Machine Learning
  • Natural Language Processing
  • Computer Vision
  • Multi-agent systems
  • Reinforcement learning

Benefits to the society

Our research enables a new generation of AI assistants that understand humans’ intent and actions in situ, communicates with humans through natural language, and collaborates with humans to accomplish complex tasks over many steps. By providing just-in-time, situated assistance, they will reinvent the relationship between the human and the device in our daily life and work, from cooking in the kitchen to aircraft engine maintenance in the hangar. 

Project’s Publications

–

Team’s Principal Investigator

David HSU

National University of Singapore (NUS)

Link to profile

David HSU is Provost’s Chair Professor in the Department of Computer Science, National University of Singapore, the director of Smart Systems Institute, and also the founding director of NUS Artificial Intelligence Laboratory (NUSAIL). He is an IEEE Fellow. His research interests span robotics, AI, and computational biology. In recent years, he has been working on robot planning and learning under uncertainty and human-robot collaboration.

 

Recent Notable Awards

  • IJCAI-JAIR Best Paper Prize, 2022
  • Robotics: Science & Systems Test of Time Award, 2021
  • Humanitarian Robotics and Automation Technology Challenge Award, International Conference on Robotics & Automation (ICRA), 2015

The Team

Principal Investigators

David HSU (Lead PI), National University of Singapore
Research Focus: Decision making, Planning, learning, human- robot interaction

Atreyi KANKANHALLI, National University of Singapore
Research Focus: Man-machine collaboration

Wee Sun LEE, National University of Singapore
Research Focus: Machine learning, planning, natural language processing

Pradeep VARAKANTHAM, National University of Singapore
Research Focus: Multi-agent systems, reinforcement learning, planning

Angela YAO, National University of Singapore
Research Focus: Computer vision, machine learning

Co-Principal Investigators

Basura FERNANDO, National University of Singapore
Research Focus: Computer vision, machine learning

Akshat KUMAR, Singapore Management University
Research Focus: Multi-agent systems, reinforcement learning, planning

Gim Hee LEE, National University of Singapore
Research Focus: Computer vision, machine learning

LEONG Tze Yun, National University of Singapore
Research Focus: Responsible AI, decision-theoretic AI, human-AI collaboration

Wei LU, Singapore University of Technology & Design
Research Focus: Natural language processing, machine learning

Malika MEGHJANI, Singapore University of Technology & Design
Research Focus: Planning, human-agent interaction

Desmond ONG, National University of Singapore
Research Focus: Affective Computing

Gergious PILIOURAS, Singapore University of Technology & Design
Research Focus: Multi-agent learning, decision theory, game theory

Harold SOH, National University of Singapore
Research Focus: Human-agent interaction, machine learning

Shengdong ZHAO, National University of Singapore
Research Focus: Interactive design