BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:Asia/Tokyo
X-LIC-LOCATION:Asia/Tokyo
BEGIN:STANDARD
TZOFFSETFROM:+0900
TZOFFSETTO:+0900
TZNAME:JST
DTSTART:18871231T000000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260817T171532Z
LOCATION:Hall B7 (1)\, B Block\, Level 7
DTSTART;TZID=Asia/Tokyo:20241204T114300
DTEND;TZID=Asia/Tokyo:20241204T115400
UID:siggraphasia_SIGGRAPH Asia 2024_sess114_papers_1085@linklings.com
SUMMARY:Anim-Director: A Large Multimodal Model Powered Agent for Controll
 able Animation Video Generation
DESCRIPTION:Yunxin Li, Haoyuan Shi, and Baotian Hu (Harbin Institute of Te
 chnology); Longyue Wang (Alibaba Group); Jiashun Zhu and Jinyi Xu (Jilin U
 niversity); Zhen Zhao (Tencent AILab); and Min Zhang (Harbin Institute of 
 Technology)\n\nTraditional animation generation methods depend on training
  generative models with human-labelled data, entailing a sophisticated mul
 ti-stage pipeline that demands substantial human effort and incurs high tr
 aining costs. Due to limited prompting plans, these methods typically prod
 uce brief, information-poor, and context-incoherent animations. To overcom
 e these limitations and automate the animation process, we pioneer the int
 roduction of large multimodal models (LMMs) as the core processor to build
  an autonomous animation-making agent, named Anim-Director. This agent mai
 nly harnesses the advanced understanding and reasoning capabilities of LMM
 s and generative AI tools to create animated videos from concise narrative
 s or simple instructions. Specifically, it operates in three main stages: 
 Firstly, the Anim-Director generates a coherent storyline from user inputs
 , followed by a detailed director’s script that encompasses settings of ch
 aracter profiles and interior/exterior descriptions, and context-coherent 
 scene descriptions that include appearing characters, interiors or exterio
 rs, and scene events. Secondly, we employ LMMs with the image generation t
 ool to produce visual images of settings and scenes. These images are desi
 gned to maintain visual consistency across different scenes using a visual
 -language prompting method that combines scene descriptions and images of 
 the appearing character and setting. Thirdly, scene images serve as the fo
 undation for producing animated videos, with LMMs generating prompts to gu
 ide this process. The whole process is notably autonomous without manual i
 ntervention, as the LMMs interact seamlessly with generative tools to gene
 rate prompts, evaluate visual quality, and select the best one to optimize
  the final output. To assess the effectiveness of our framework, we collec
 t varied short narratives and incorporate various Image/video evaluation m
 etrics including visual consistency and video quality. The experimental re
 sults and case studies demonstrate the Anim-Director’s versatility and sig
 nificant potential to streamline animation creation.\n\nRegistration Categ
 ory: Full Access, Full Access Supporter\n\nLanguage Format: English Langua
 ge\n\nSession Chair: Kai Wang (Simon Fraser University, ShanghaiTech Unive
 rsity)\n\n
URL:https://asia.siggraph.org/2024/program/?id=papers_1085&sess=sess114
END:VEVENT
END:VCALENDAR