
I am currently a research scientist at Google Magenta, and I will join CMU as an assistant professor in Fall 2023. Previously, I was a postdoc at Stanford CS advised by Percy Liang. Before that, I completed a PhD at UCSD under Miller Puckette (music) and Julian McAuley (CS).
My research goal is to build music technology that enables a broader set of users to engage with music on a deeper level. To this end, I improve ML methods for generative modeling of music and other multimedia, and build interactive systems which allow anyone to leverage resultant models.
Selected Publications (Full List)
- Melody Transcription via Generative Pre-training In ISMIR, 2022. [pdf, arXiv, sound examples, code and data]
- It's Raw! Audio Generation with State-Space Models In ICML (Long Talk; Top 2%), 2022. [pdf, arXiv, sound examples, code]
- Codified Audio Language Modeling Learns Useful Representations for Music Information Retrieval In ISMIR (Best Paper Runner-up), 2021. [pdf, arXiv, code]
- Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music In ISMIR, 2021. [pdf, arXiv, sound examples, code]
- Swords: A Benchmark for Lexical Substitution with Improved Data Coverage and Quality In NAACL, 2021. [pdf, arXiv, code]
- Enabling Language Models to Fill in the Blanks In ACL, 2020. [pdf, arXiv, demo, code]
- LakhNES: Improving Multi-instrumental Music Generation with Cross-domain Pre-training In ISMIR, 2019. [pdf, arXiv, code, music examples]
- Expediting TTS Synthesis with Adversarial Vocoding In INTERSPEECH (Oral), 2019. [pdf, arXiv, code, sound examples]
- Piano Genie In ACM IUI, 2019. [pdf, arXiv, blog post, videos, demo, code]
- Adversarial Audio Synthesis In ICLR, 2019. [pdf, arXiv, code, demo, sound examples, notebook]
- GANSynth: Adversarial Neural Audio Synthesis In ICLR, 2019. [pdf, blog post, code, sound examples]
- The NES Music Database: A Multi-instrumental Dataset with Expressive Performance Attributes In ISMIR, 2018. [pdf, arXiv, dataset, code, notebook]
- Semantically Decomposing the Latent Spaces of Generative Adversarial Networks In ICLR, 2018. [pdf, arXiv, code, demo]
- Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition In ICASSP (Oral), 2018. [pdf, arXiv]
- Dance Dance Convolution In ICML, 2017. [pdf, arXiv, dataset, code, demo] * Indicates equal contribution
Work Experience
- (2022-Current) Research Scientist at Google Magenta.
- (Summer 2018) Internship at Google Magenta (Music generation w/ Ian Simon and Sander Dieleman)
- (Summer 2017) Internship at Google (Speech recognition w/ Bo Li and Rohit Prabhavalkar)
- (Summer 2016) Internship at Google Search
- (Summer 2015) Internship at Google Play Music (MIR w/ Nicolas Boulanger-Lewandowski)
- (2011-2014) Mentor for UT Freshman Research Initiative w/ Joel Lehman and Risto Miikkulainen
- (Summers 2011-2014) Internships at UT Applied Research Laboratories, Qualcomm, and two startups
Media Coverage
- UploadVR Beat Sage Update Adds 90 Degree Levels, Walls And Single Saber Mode
- UploadVR Get Rhythm: How Beat Sage Uses AI To Create Beat Saber Maps
- Road to VR This 'Beat Saber' Project Uses AI to Generate Custom Beat Maps for Any Song
- UploadVR New AI Tool Turns Any Song Into A Custom Beat Saber Map, And It Really Works
- Stereogum Watch The Flaming Lips Play A Bowl Of Fruit At Google I/O
- Business Insider A Google intern helped build an AI tool inspired by 'Guitar Hero' to let rookies play piano
- The Verge Google’s AI-powered Piano Genie lets anyone improvise perfectly by bashing buttons
- Evening Standard Piano Genie: Google's AI programme is like Guitar Hero for the piano world
- Engadget Google’s Piano Genie lets anyone improvise classical music
- MIT Tech Review Machine-Learning Algorithm Watches DDR, Then Creates Dances of Its Own
- The Verge Scientists have taught a neural network to choreograph Dance Dance Revolution levels
- The Register Yet another job menaced by AI! Uh, wait, it says here... Dance Dance Revolution designers
- Vice This Machine Learned to Choreograph by Watching Dance Dance Revolution
Other
- (2020) Released Beat Sage, a web service for automatically creating Beat Saber levels (link)
- (2020) Download files from Google Drive on the command line (link, code)
- (2019) PhD dissertation on music, AI, and interaction (pdf)
- (2018) Transcribe a batch of solo piano recordings to MIDI (link)
- (2017) PhD qualifying examination (pdf)
- (2016) Master's thesis on convolution-based cross-synthesis, (pdf, code, vst)
- (2015) Prototype for MOOC on computer music fundamentals using Web Audio API (link)
- (2015) Mobile-friendly, networked musical controller (demo)
- (2014) Multichannel convolution reverb plugin (screenshot, code, windows vst)
- (2013) Undergraduate thesis on musical instrument mimicry (pdf, examples, vst)
- (2013) OpenGL 3D spectrogram (page, code)
- (2012-2014) Played keyboard for Food Group
Last updated 2023/01/30