Memory-efficient Online Continuous Object Recognition on Video Streams

Project Description

The world is not stationary. To thrive in evolving environments, humans are capable of continual acquisition and transfer of new knowledge, from a continuous stream of highly correlated visual stimuli over multiple tasks, while retaining previously learned experiences. In contrast to human learning, most AI systems struggle to recognise new objects when presented with a video stream of new object classes incrementally available over time. Meanwhile, they also fail to retain good performance on earlier learned classes without repeated passes over the data, known as catastrophic forgetting.  Our aim is to design a continual learning framework that recognises incrementally available object classes on video streams without repeated visits of the old data in a memory-efficient manner. With multi-disciplinary approaches at the intersection of neuroscience, cognitive science, and AI, we hope to gain a better understanding of the continual learning mechanisms in biological and artificial intelligence.

Research Technical Area

  1. Computer Vision
  2. Cognitive Science
  3. Computational Neuroscience

Benefits to the society

The success of our unique and first-of-its-kind framework will bring profound impacts to AI basic research and help us better understand the continual learning mechanisms in both human brains and computational models. Our technology will also boost AI deployment in the real world, where the environment is dynamically evolving, and the data stream is always incrementally available over time. For example, new products hit store shelves in supermarkets every day. Storing all product information and daily re-training of AI models on all the products from the beginning is time-consuming, computationally expensive and memory inefficient. AI must learn on-the-fly to identify these new products without forgetting the old ones.

Project’s Publications

Team’s Principal Investigator

Mengmi ZHANG


Mengmi ZHANG is a research scientist and principal investigator of Deep NeuroCognition Lab in A*STAR. She is also an adjunct assistant professor from Department of Electrical and Computer Engineering in National University of Singapore (NUS). Prior to this, Dr. Zhang was a postdoc at Harvard-MIT Center for Brains Minds and Machines from 2019-2021. She obtained her PhD at the NUS (2015-2019) and was a visiting graduate student in KreimanLab at Harvard Medical School (2017-2018). Her research background is at the intersection of artificial intelligence and computational neuroscience. She has made contributions to understanding gaze anticipation, zero-shot visual search, contextual reasoning and continual learning.

The Team

Co-Principal Investigator

Research Focus: Computer Vision


Gabriel KREIMAN, Harvard Medical School
Research Focus: Computational Neuroscience, Neuroscience

Jie ZHENG, Harvard Medical School
Research Focus: Neuroscience