research
<(From Left) Hyun-Bin Oh, Takida Yuhta, Uesaka Toshimitsu, Tae-Hyun Oh, Mitsufuji Yuki>
When people watch a scene in the film Jurassic Park where a giant dinosaur walks toward them, they naturally imagine a heavy, rumbling sound, as if the ground were shaking. This is because humans predict sound by considering not only the shape of an object, but also physical properties such as its size, weight, and speed of movement. However, existing video-to-audio generation AI mainly generates sound based on the category of objects or scene information in the video, and has not sufficiently reflected physical properties that vary depending on weight or speed.
KAIST (President Kwang Hyung Lee) announced on the 26th of May that a collaborative research team involving Professor Tae-Hyun Oh of the School of Computing, KAIST, together with joint researchers from POSTECH (President Sung Keun Kim) and Sony AI, has developed “PAVAS (Physics-Aware Video-to-Audio Synthesis),” an artificial intelligence (AI) technology that understands the physical situation in a video and generates more realistic sound.
<Concept Diagram of PAVAS (Physics-Aware Video-to-Audio Synthesis) Technology>
The key feature of this technology is that it is designed so that AI can infer invisible physical information such as the mass and velocity of objects in a video on its own. Ordinary videos do not provide exact numerical values for an object’s weight or speed, but the research team enabled AI to estimate them by analyzing the surrounding environment and movement context, and to reflect the results in the sound generation process.
In other words, the AI was designed to go beyond simply recognizing “what is visible” and to understand the physical cause of “why this sound should occur.”
As a result of technical validation, the research team’s AI generated sounds very similar to real-world environments in scenes involving physical interactions such as collisions or impacts between objects. In particular, it produced more realistic audio in which loudness and tone naturally changed when the mass and velocity of objects varied.
Recently, generative AI technologies that simultaneously generate video and audio have been advancing rapidly. Representative examples include Google’s “Veo 3” and ByteDance’s “Seedance 2.0.” However, in actual film, advertising, and game production sites, there is far greater demand for post-production work that adds sound effects suited to existing video scenes or supplements audio than for generating entirely new videos.
While existing commercial AI models have focused on generating video and audio together, PAVAS is differentiated by its ability to analyze the movement and collision characteristics of objects in a video and generate realistic sound effects that precisely match the scene.
<Comparison of Spectrograms Generated by Conventional Video-to-Audio Models and PAVAS>
The research team explained that this technology presents new possibilities in the field of “Physical AI,” or physically consistent generative AI. Physically consistent generative AI refers to AI that goes beyond simply producing plausible results and understands the laws of physics and causal relationships in the real world.
In the future, this technology is expected to provide more immersive user experiences in a wide range of fields, including the automation of content sound production, augmented reality (AR) and virtual reality (VR) content, the metaverse, and robotics simulation.
Professor Tae-Hyun Oh stated, “While existing generative AI has developed by increasing the scale of data and models, this research is meaningful in that it was designed so that AI directly understands physical quantities and causal relationships,” adding, “In the future, it can be expanded into a core foundational technology for next-generation multimodal AI that simultaneously understands and processes diverse types of information, including text, video, and speech.”
This study was led by POSTECH integrated M.S.-Ph.D. student Hyun-Bin Oh as the first author, with KAIST Professor Tae-Hyun Oh and Sony AI researchers Yuhta Takida, Toshimitsu Uesaka, and Yuki Mitsufuji participating as co-authors. This research was selected as an Oral presentation paper at CVPR 2026 (Computer Vision and Pattern Recognition 2026), the world’s most prestigious academic conference in the field of computer vision (image-based artificial intelligence technology), where only the top 0.88% of all papers are selected for oral presentation, recognizing the excellence of the work. The presentation is scheduled to take place on June 6.
※ Paper title: “PAVAS: Physics-Aware Video-to-Audio Synthesis,” DOI: https://arxiv.org/abs/2512.08282
This research was supported by the Mid-Career Research Program under the Basic Research Program of the Ministry of Science and ICT, the Pioneer Research Program for Future Converging Technology of the Ministry of Science, ICT and Future Planning, the AGI Program of the Ministry of Science and ICT, and the KAIST InnoCORE Program.
-
research KAIST Automates the Search for “Dream Semiconductor” 2D Semiconductors
The era of researchers manually searching for two-dimensional semiconductors, which are drawing attention as next-generation AI semiconductors, is coming to an end. KAIST researchers have automated semiconductor screening and device fabrication, analyzed thousands of devices, and revealed the relationship between thickness and performance that had long been difficult to identify. This achievement is expected to shift next-generation semiconductor research toward a data-driven approach and acce
2026-07-09 -
research KAIST Identifies the “Hidden Energy Cost” of AI Agents for the First Time
As the era of AI agents—systems that can reason and act autonomously—begins, the power consumption of data centers is emerging as a critical challenge. A KAIST research team has, for the first time, analyzed the computational cost and energy consumption of AI agents, finding that they can consume up to 136.5 times energy per query than conventional generative AI. The study shows that competitiveness in the AI era is expanding beyond model performance to include the efficiency of d
2026-07-07 -
event KAIST Global Entrepreneurship Summer School Marks Fifth Consecutive Year of Cultivating Future Entrepreneurs in Silicon Valley
The 2026 Global Entrepreneurship Summer School (GESS), organized by the KAIST Office of Global Initiatives, has successfully concluded its fifth annual program. Now in its fifth year, GESS has become KAIST's flagship global entrepreneurship program, providing students with firsthand experience in Silicon Valley—the world's leading startup ecosystem—and equipping them with the entrepreneurial mindset and global competencies needed to launch ventures on the international stage.
2026-07-06 -
research KAIST: Dementia-Causing Substance Turns On a Therapeutic “Switch”
A substance that worsens dementia has become a “switch” that initiates treatment. KAIST researchers have developed a new therapeutic approach that uses hydrogen peroxide (H₂O₂), a reactive oxygen species that damages cells and increases in the brains of patients with Alzheimer’s disease, to activate a drug selectively in diseased brain tissue. The team also confirmed improvements in cognitive function through animal experiments, presenting a new possibility for next-generatio
2026-07-03 -
research KAIST Develops AI That Reads Animal Behavior Like Language
An artificial intelligence model capable of reading and interpreting animal behavior like language has been developed by researchers at KAIST. The team created BehaVERT, an AI model that learns behavioral data in a manner similar to natural language and was able to independently identify social behavioral deficits in an autism mouse model, opening a new avenue for interpretable neuroscience. KAIST (President Kwang-Hyung Lee) announced that a research team led by Professor Dae-Soo Kim from the
2026-07-01