BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:Asia/Tokyo
X-LIC-LOCATION:Asia/Tokyo
BEGIN:STANDARD
TZOFFSETFROM:+0900
TZOFFSETTO:+0900
TZNAME:JST
DTSTART:18871231T000000
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20260817T171533Z
LOCATION:Hall B7 (1)\, B Block\, Level 7
DTSTART;TZID=Asia/Tokyo:20241205T110800
DTEND;TZID=Asia/Tokyo:20241205T111900
UID:siggraphasia_SIGGRAPH Asia 2024_sess129_papers_1250@linklings.com
SUMMARY:FürElise: Capturing and Physically Synthesizing Hand Motion of Pia
 no Performance
DESCRIPTION:Ruocheng Wang, Pei Xu, Haochen Shi, Elizabeth Schumann, and C.
  Karen Liu (Stanford University)\n\nPiano playing requires agile, precise,
  and coordinated hand control that stretches the limits of dexterity. Hand
  motion models with the sophistication to accurately recreate piano playin
 g have a wide range of applications in character animation, embodied AI, b
 iomechanics, and VR/AR. In this paper, we construct a first-of-its-kind la
 rge-scale dataset that contains approximately 10 hours of 3D hand motion a
 nd audio from 15 elite-level pianists playing 153 pieces of classical musi
 c. To capture natural performances, we designed a markerless setup in whic
 h motions are reconstructed from multi-view videos using state-of-the-art 
 pose estimation models. The motion data is further refined via inverse kin
 ematics using the high-resolution MIDI key-pressing data obtained from sen
 sors in a specialized Yamaha Disklavier piano. Leveraging the collected da
 taset, we developed a pipeline thatcan synthesize physically-plausible han
 d motions for musical scores outside of the dataset. Our approach employs 
 a combination of imitation learning and reinforcement learning to obtain p
 olicies for physics-based bimanual control involving the interaction betwe
 en hands and piano keys. To solve the sampling efficiency problem with the
  large motion dataset, we use a diffusion model to generate natural refere
 nce motions, which provide high-level trajectory and fingering (finger ord
 er and placement) information. However, the generated reference motion alo
 ne does not provide sufficient accuracy for piano performance modeling. We
  then further augmented the data by using musical similarity to retrieve s
 imilar motions from the captured dataset to boost the precision of the RL 
 policy. With the proposed method, our model generates natural, dexterous m
 otions that generalize to music from outside the training dataset.\n\nRegi
 stration Category: Full Access, Full Access Supporter\n\nLanguage Format: 
 English Language\n\nSession Chair: Yuting Ye (Reality Labs Research, Meta;
  Meta)\n\n
URL:https://asia.siggraph.org/2024/program/?id=papers_1250&sess=sess129
END:VEVENT
END:VCALENDAR