For the second year in a row, UC Santa Cruz’s Team Athena has advanced to the finals of the Amazon Alexa Prize Socialbot Grand Challenge, a university competition designed to advance the field of conversational artificial intelligence (AI). Though hundreds of teams from around the world applied to participate, Athena was one of only nine selected to compete in this year’s challenge and is one of five to advance to the finals.
Mentored by faculty advisor Marilyn Walker—a professor in the Department of Computer Science and Engineering (CSE) and director of the department’s new Natural Language Processing (NLP) master’s program—the 11-member team includes a mix of undergraduate and graduate students from CSE, including five members of the inaugural NLP class. For those five, serving on Team Athena allows them to compete in the Alexa Prize Socialbot Grand Challenge while also fulfilling the NLP program’s capstone project requirement.
“The capstone project is an opportunity to do a substantial team research project over three quarters to develop practical skills in the field of natural language processing,” said Walker, who has mentored the team all four years in which it competed, including when the team was known as SlugBot in 2017 and 2018. “The Alexa Socialbot Grand Challenge gives the NLP students an opportunity to try out their research ideas in a real time system that interacts in conversation with thousands of human users.”
For the other six members of Team Athena—a mix of undergraduate and graduate students in other CSE degree programs who work in Walker’s Natural Language and Dialogue Systems Lab—competing in the Socialbot Grand Challenge provides an opportunity to gain valuable experience solving real-world problems in the fields of natural language processing and conversational AI.
The Socialbot Grand Challenge tasks teams to build a socialbot capable of carrying out 20-minute coherent and engaging conversations on various current events and popular topics with users of Alexa-enabled devices. Amazon provides each team a $250,000 research grant, Alexa-enabled devices, and free Amazon Web Services (AWS) cloud computing services to support their research and development efforts.
The competition began November 9, 2020, when the socialbots were released into the Alexa ecosystem. One socialbot is chosen at random for testing when a user of an Alexa-enabled device says, “Alexa, let’s chat.” Once the conversation concludes, the user gets the chance to give a verbal rating of the socialbot’s performance and provide feedback, which is sent anonymously to the socialbot’s team. The team also receives a transcript of the conversation.
“You can tell a lot from the conversations with users, especially things that aren’t working,” Walker said.
The capstone project for the five NLP students on Team Athena aims to improve conversational AI by making it better at understanding a key facet of human language that the technology has historically had trouble with.
“Originally the socialbot did not have the capabilities of understanding different pronouns, so the NLP capstone team has been working on their own encapsulated project to develop new approaches to address this issue," said Walker.
Through the use of coreference resolution algorithms—the task of finding all expressions that refer to the same entity in a text—the NLP capstone team is developing a conversational model capable of capturing pronoun context. This approach aims to solve a language comprehension problem with socialbots, which may seem trivial to humans, but has been proven to be a major stumbling block for conversational AI.
“We noticed if we don’t provide context to different pronouns, our system won’t necessarily know how to continue the conversation, so we might lose out on a potential positive interaction by responding with information unrelated to what the user is talking about,” said NLP student Angela Ramirez. “This led us to focus on using coreference resolution to resolve ambiguous pronouns that exist in the user’s utterances.”
Ramirez and NLP team member Cecilia Li have both been working on the capstone project and Athena system. They often work closely together in reviewing edge cases—major issues that arise within the system that impact user experience—and annotating data, when needed, to improve the socialbot’s ability to understand pronoun context in edge cases.
“I’ve learned that interacting with users in a live system is usually far less predictable than textbook exercises,” said Li. “With Alexa Prize being an open-ended project, not only do we need to establish the problem on the fly and choose the correct tools to solve them, we also need to design the process every step of the way as we proceed. Understanding and learning NLP in this unique environment is truly an experience that will benefit us in our future careers.”
All five NLP students have been consistently working in tandem with the other six members of Team Athena, splitting up tasks, completing weekly goals, and finding innovative ways to improve the socialbot so they have a strong chance of placing in the top three of this year’s challenge.
Amazon Alexa Socialbot Grand Challenge winners will be announced in August 2021. Monetary prizes will be awarded to the top three teams. For more information about the challenge, visit the Amazon Alexa Prize page.
The members of this year’s Team Athena are Angela Ramirez, Cecilia Li, Eduardo Zamora, Phillip Lee, and Jeshwanth Bheemanpally from the new NLP master’s program; and Omkar Patil (team leader), Davan Harrison, Juraj Juraska, Lena Reed, Kevin Bowden, and Rohan Pandey from other CSE undergraduate and graduate programs.
