About the Challenge

The innovative use of AI, especially through Large Language Models (LLMs), presents both exciting opportunities and notable challenges. One such challenge is the ability of sophisticated jailbreaking attacks to circumvent existing safety mechanisms, highlighting the need for continued advancements in AI security.

AI Singapore, in collaboration with the CyberSG R&D Programme Office (CRPO), is excited to launch the Global Challenge for Safe and Secure LLMs, an 8-month competition divided into two tracks: Attack and Defense. This challenge aims to advance AI research in developing robust security measures and innovative approaches to mitigate jailbreaking attacks on LLMs. Our goal is to foster safer AI interactions worldwide, particularly in applications with significant societal impact.


In this challenge, participants can choose to join either or both tracks. Participation in Track 1 is not required for Track 2.

Track 1 (Attack): Participants will innovate by creating automated approaches for crafting test cases (prompts) designed to trigger a range of undesirable responses from a series of fine-tuned LLMs despite their training to prevent such outcomes. The focus will be on 75 predefined behaviours categorized under themes such as prejudice, violence, illegal activities, fraudulent schemes, and misinformation.

Track 2 (Defense): Participants will develop robust security measures to reinforce the resilience of LLMs against the advanced jailbreak attacks demonstrated in Track 1.


  • Researchers and Industry Professionals from around the world
  • Individuals interested in AI Security and Safety with experience in Natural Language Processing, Machine Learning, and AI Model Robustness


Submissions will be evaluated based on relevant metrics and ranked on public and private leaderboards.


Start of Track 1: 2 July 2024
Submission Deadline for Track 1A: 6 August 2024
Notification of Shortlisted Top 10 Teams to enter Track 1B: by 16 August 2024
Start of Track 1B Submission: 20 August 2024
Submission Deadline for Track 1B: 17 September 2024
Prize Ceremony: 16 October 2024


  • Any individual who is at least 18 years of age can form/join a team to participate in the challenge
  • A team can comprise 1 to 6 members
  • An individual may not be on more than one team


The top three teams in each track will be declared as the winners of the challenge and awarded the following cash prizes:

Track 1

  • First Place: USD 30,000
  • Second Place: USD 15,000
  • Third Place: USD 7,500

Track 2

  • First Place: USD 40,000
  • Second Place: USD 20,000
  • Third Place: USD 10,000

The top ten teams in each track will be provided a travel stipend of USD 2,000 for one representative per team to attend the respective tracks’ Prize Ceremonies.