UCSC team wins third place in first-ever Amazon SimBot challenge

A team of UC Santa Cruz computer science and engineering (CSE) Ph.D. students won third place in the first-ever Amazon Alexa Prize SimBot Challenge, a university competition focused on advancing virtual assistant technology.

In this challenge, teams were charged with building virtual robots that execute real-world household tasks such as making coffee or cleaning a room in a virtual environment through conversational interactions with users. Machine learning techniques implemented by the team allowed the bots to continuously learn from their environment and improve their performance. Next-generation personal assistant bots stand to be especially impactful for older adults and people living with disabilities. 

“The skills and technologies developed for the SimBot Challenge have real-world applications,” said Jing Gu, a Ph.D. student and leader of team SlugJARVIS. “Our team's success could lead to opportunities in the fields of home automation, robotics, and AI. Participating in the challenge and reaching the finals is a valuable learning experience.”

The UCSC team, called SlugJARVIS, is advised by Assistant Professor of CSE Xin (Eric) Wang and includes seven CSE Ph.D. students who all study in Wang’s ERIC lab. The project aligns with areas of research exploration in Wang’s lab – embodied artificial intelligence (robots operating in 3D environments), natural language processing, and computer vision.

“The SimBot Challenge began concurrently with the arrival of our lab's first group of students. They have done an incredibly outstanding job in research, and securing the top 3 position in the finals marked an important milestone in our embodied AI journey. However, we view this achievement as just the beginning, with our sights firmly set on a future ambition: to bring embodied agents into reality across diverse applications such as robotics, smart home assistants, and beyond.”

The SlugJarvis team began working on this project in January 2022, and later won the public benchmark phase of the challenge. In April 2023, the team advanced to the finalist round, where they competed to make their bot best respond to commands and multimodal sensor inputs from within a virtual world.

Amazon Alexa customers interacted with the finalists’ SimBots by saying "Alexa, play with robot" on Echo Show or Fire TV devices. They then had the option to leave ratings and feedback, which played into the final rankings and helped the student teams improve their bots. 

The UCSC team was awarded a $50,000 prize for their third-place finish. Teams from the University of Michigan and UC Santa Barbara were awarded first and second place, respectively. The work of SlugJarvis along with other participating university teams is detailed in a series of research papers.