%4a%75%6e%6d%6f%20%4b%69%6d
-
KAIST Develops AI That Finds Its Own Hidden Weaknesses, Paving the Way for Safer Generative AI Models
KAIST researchers have developed a safety verification technology that uncovers roughly seven times more hidden vulnerabilities in AI than existing methods. The technology is expected to serve as a foundation for developing safer, more trustworthy AI.
KAIST (President Choongsik Bae) announced on the 30th of July that a research team led by Professor Junmo Kim from the School of Electrical Engineering has developed a new framework called Stable-GFlowNet (S-GFN), which overcomes the limitations of red-teaming—a safety verification process that deliberately attacks large language models (LLMs) to expose hidden weaknesses.
Red-teaming for generative AI is the process of crafting attack prompts designed to probe an AI's vulnerabilities and induce the AI to produce harmful or dangerous responses before the program is deployed. Since discovering a wider variety of attack methods allows more vulnerabilities to be addressed in advance, both the success rate and diversity of attacks are critical.
Previous approaches primarily relied on reinforcement learning—an AI technique trained to maximize reward—to generate attack prompts. However, these methods frequently suffered from mode collapse—a phenomenon where the model repeatedly converges on a narrow set of high-reward attack prompts rather than generating diverse outputs, thereby limiting its ability to uncover various vulnerabilities.
Generative Flow Networks (GFlowNets)—an AI generation technique trained to produce diverse outputs in proportion to their reward—were proposed as a solution. Yet GFlowNet training is computationally complex and unstable, and noisy reward signals can assign high rewards even to meaningless sentences, often causing training to collapse.
To address these issues, the research team developed three core techniques that help the model learn effective attacks more reliably while filtering out flawed ones.
First, much like comparing several paths to choose the best one, the team introduced Contrastive Trajectory Balance (CTB), which reduces computational complexity and stabilizes training by directly comparing pairs of generated attack trajectories.
Second, akin to filtering out background noise to focus on a single voice, the team applied Noise Gradient Pruning (NGP) to eliminate minor reward fluctuations and ensure the model learns exclusively from meaningful signals.
Third, the team applied the Min-K Fluency Stabilizer (MKS), which guides the model to generate attack prompts resembling text that a real user would write—just as a human reader naturally prefers coherent sentences to gibberish.
As a result, Stable-GFlowNet discovered 134 unique attack types—about seven times more than the 17 unique attack types found by the existing GFlowNet-based technique—while maintaining a high attack success rate of 92%.
Defense models trained using attacks generated by Stable-GFlowNet also demonstrated strong generalization, effectively defending against a wide range of attacks in cross-attack tests, which evaluate performance using attack techniques different from those used during training.
The team further demonstrated that CTB and NGP achieve faster and more stable performance than existing methods—not only in AI safety verification, but also in other distribution-matching tasks such as molecular generation for drug discovery.
Professor Kim said, "This technology is significant in that it can reliably uncover a wide range of AI vulnerabilities even in realistic conditions with limited data and high noise." He added, "Because it allows a broader range of risk factors to be identified and defended against before generative AI is deployed in real-world services, we expect it to become a core foundational technology for developing safer, more trustworthy AI."
The study was led by first author Minchan Kwon, a Ph.D. candidate from the School of Electrical Engineering, and was selected as a Spotlight paper—placing it in the top 2.2% of submissions—at the International Conference on Machine Learning (ICML) 2026, one of the world's most prestigious AI conferences.
※ Paper title: Stable-GFlowNet: Toward Diverse and Robust LLM Red-Teaming via Contrastive Trajectory Balance
arXiv: https://arxiv.org/abs/2605.00553
This research was supported by the Institute of Information & Communications Technology Planning & Evaluation’s (IITP) SW Star Lab program, funded by the Ministry of Science and ICT.
2026.07.30 View 921 -
2018 KAIST Research Day Honors Outstanding Research Achievements
(KAIST President Sung-Chul Shin and Professor Jong-Hwan Kim) Professor Jong-Hwan Kim from the School of Electrical Engineering was recognized at the 2018 KAIST Research Day as the Research Grand Prize Awardee. The ten most distinguished research achievements of the past year were also recognized.
The Research Grand Prize recognizes the professor whose comprehensive research performance evaluation indicator was the highest over the past five years. The indicator combines the number of research contracts, IPR and royalty income.
During the May 25th ceremony, Professor Hyochoong Bang from the Department of Aerospace Engineering and Professor In so Kweon from the School of Electrical Engineering also won the Best Research Award prize.
This year, the Research Innovation Award went to Professor Dong Soo Han from the School of Computing. The Research Innovation Award combines scores in the categories of foreign patent registrations, contracts of technological transfer, and income from technology fees, technology consultations, and startups.
The Convergence Research Award was given to Professor Junmo Kim from the School of Electrical Engineering and Professor Hyun Myung from the Department of Civil & Environmental Engineering. The Convergence Research Award recognizes the most outstanding research team that created innovative research results over a one-year period.
President Sung-Chul Shin said, “KAIST has selected the ten most outstanding research achievements of 2017 conducted by our faculty and researchers. All of them demonstrated exceptional creativity, which opens new research paths in each field though their novelty, innovation, and impact.”
KAIST hosts Research Day every year to introduce major research performances at KAIST and share knowledge about the research and development.
During Research Day, KAIST also announced the ten most distinguished research achievements contributed by KAIST professors during the previous year. They are listed below.
▲ High-Speed Motion Core Technology for Magnetic Memory by Professor Kab-Jin Kim from the Department of Physics
▲ A Double Well Potential System by Professor Jaeyoung Byeon from the Department of Mathematical Sciences
▲ Cheap and Efficient Dehydrogenation of Alkanes by Professor Mu-Hyun Baik from the Department of Chemistry
▲ A Dynamic LPS Transfer Mechanism for Innate Immune Activation by Professor Ho Min Kim from the Graduate School of Medical Science and Engineering
▲ A Memristive Functional Device and Circuit on Fabric for Fibertronics by Professor Yang-Kyu Choi and Professor Sung-Yool Choi from the School of Electrical Engineering
▲ A Hippocampal Morphology Study Based on a Progressive Template Deformable Model by Professor Jinah Park from the School of Computing
▲ The Development of a 6-DOF Dynamic Response Measurement System for Civil Infrastructure Monitoring by Professor Hoon Sohn from the Department of Civil and Environmental Engineering
▲ Cooperative Tumour Cell Membrane Targeted Phototherapy by Professor Ji-Ho Park from the Department of Bio and Brain Engineering
▲ HUMICOTTA: A 3D-Printed Terracotta Humidifier by Professor Sangmin Bae from the Department of Industrial Design
▲ Ultrathin, Cross-Linked Ionic Polymer Thin Films by Professor Sung Gap Im from the Department of Chemical and Biomolecular Engineering
2018.05.28 View 18370