Skip to content

Xichen Pan

Office: 424, 60 5th Ave, New York, NY 10011

Work Email: xichenpan [at] nyu [dot] edu

Personal Email: xcpan [dot] mail [at] gmail [dot] com

Bio

I am a second-year Ph.D. student of Computer Science at NYU Courant, advised by Prof. Saining Xie. I am also a Visiting Researcher at Meta AI (2024-2025 AI Mentorship Program, 20% part-time), based on NYC office. I interned at Meta GenAI Emu team (2024 Summer with Dr. Ji Hou), Microsoft Research Asia (2022-2023 with Dr. Li Dong), Alibaba Group (2022 Fall with Dr. Pengda Qin), and Horizon Robotics (2021-2022 with Yichen Gong). Previously, I obtained my bachelor’s degree in Computer Science from Shanghai Jiao Tong University (SJTU) and won the Best Thesis Award. I was fortunately advised by and maintain a close connection with Prof. Zhouhan Lin at SJTU.

Research Interest

Generative Models

Designing more controlable and high fidelity methods for image, video, and 3D generation, with a focus on:

  • Preserving spatial and temporal consistency

  • Leveraging text-to-image priors for advanced applications

Multimodal Learning

Developing vision-language models for vision-centric applications, focusing on representation learning and self-supervised pre-training

News

[07/2024] I will join Meta AI New York office as a Visiting Researcher (2024-2025 AI Mentorship Program, 20% part-time) in 2024 Fall.

[02/2024] Our paper was accepted by CVPR 2024, check it out here. See you in Seattle!

[02/2024] I will join Meta GenAI as a Research Scientist Intern in 2024 Summer. See you in Menlo Park!

[01/2024] Our paper was accepted by ICLR 2024, check it out here.

[10/2023] πŸ– Our paper was accepted by WACV 2024 as Oral, check it out here.

[09/2023] πŸŽ‰ Excited to start my CS Ph.D. at NYU Courant advised by Prof. Saining Xie.

[12/2022] Glad to work with Dr. Li Dong and Dr. Furu Wei at Microsoft Research Asia for the upcoming year, leading up to Fall 2023.

[06/2022] My bachelor thesis won Best Thesis Award in SJTU! Thanks my advisor Prof. Zhouhan Lin, checkout the honor roll.

[02/2022] Our paper was accepted by ACL 2022 Main Conference, check out full paper.

Education

New York University Courant InstituteImage title

Sept. 2023 -- Present

Ph.D. in Computer Science, advised by Prof. Saining Xie

Shanghai Jiao Tong UniversityImage title

Sept. 2018 -- June 2022

B.Eng. in Computer Science (Outstanding Graduate of Class 2022), advised by Prof. Zhouhan Lin

Publications & Manuscripts

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Jairam Iyer, Xichen Pan, Ziteng Wang, Rob Fergus, Yann LeCun, Saining Xie

NeurIPS 2024 (Oral)   arXiv   Code   Project Page

Image Sculpting: Precise Object Editing with 3D Geometry Control

Jiraphon Yenphraphai, Xichen Pan, Sainan Liu, Daniele Panozzo, Saining Xie

CVPR 2024   arXiv   Code   Project Page

Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei

ICLR 2024   arXiv   Code   Project Page

Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models

Xichen Pan, Pengda Qin, Yuhong Li, Hui Xue, Wenhu Chen

WACV 2024 (Oral, Top 6% of accepted papers)   arXiv   Code

Multimodal Audio-Visual Speech Recognition System Based On Pre-trained Models

Xichen Pan

Bachelor thesis at Shanghai Jiao Tong University (Best Thesis Award, 1st/150)   News   Honor Roll

Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-visual Speech Recognition

Xichen Pan, Peiyu Chen, Yichen Gong, Helong Zhou, Xinbing Wang, Zhouhan Lin

ACL 2022 Main Conference   arXiv   Code

Experience

Meta AIImage title

Sept. 2024 – May. 2025

Visiting Researcher (2024-2025 AI Mentorship Program, 20% part-time)

Meta GenAIImage title

May. 2024 – Sept. 2024

Research Scientist Intern

Microsoft Research AsiaImage title

Dec. 2022 – Sept. 2023

StarBridge Program Research Assistant

Alibaba GroupImage title

Sept. – Dec. 2022

Research Intern

Horizon RoboticsImage title

Apr. 2021 – July 2022

Research Intern

John Hopcroft Center for Computer Science, Shanghai Jiao Tong UniversityImage title

Apr. 2021 – June 2022

Research Intern

NSF Center for Big Learning, University of FloridaImage title

July – Sept. 2020

Research Intern

Selected Projects

Open CS Application

An open-source GitHub page built for reference in selecting CS programs in north America. The page is powered by Material for MkDocs and supports collaboration through Pull Requests and GitHub Actions.

Media Exposures

Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models, Synced

Academic Service

ICML'24, ACL'24, ECCV'24, TMLR, IJCV

Some of My Friends

Cornell: Youming Deng, Gene Chou

CMU: Kexun Zhang

Georgia Tech: Haotian Xue

Neon, Inc.: Alex Chi

New York University: List of My Labmates, Hexu Zhao

Ohio State University: Kai Zhang

Oxford University: Junlin Han

Shanghai Jiao Tong University: Xinyu Xu

Stanford: Yanjie Ze

UC Berkeley: Junyi Zhang, Yichuan Wang

UMich: Yiming Dou

University of Washington: Zihan Li

USC: Di Chang

This Site Already Has Free Hit Counters Visitors