I am currently a first-year PhD student at UMass Amherst, where I am supervised by Prof. Chuang Gan and collaborate with Prof. Yilun Du. My research focuses on Building 3D Foundation Models and World Models. Total citations:

I received my Bachelor’s degree from Shanghai Jiao Tong University (SJTU), where I worked closely with Prof. Cewu Lu and Dr. Lixin Yang from the Machine Vision and Intelligence Group (MVIG). In addition, I have worked with Prof. Junchi Yan at the SJTU-ReThinkLab. Previously, I was a visiting student at the Massachusetts Institute of Technology and the MIT-IBM Watson AI Lab, guided by Prof. Josh Tenenbaum.

News

  • 2024.05:  One paper 3D-VLA is accepted by ICML 2024.
  • 2024.04:  Invited talk “Building Embodied 3D Foundation Models” at OpenDriveLab, Shanghai AI Lab and QingkeLab.
  • 2024.03:  Introduce 3D-VLA, a 3D Vision-Language-Action Generative World Model
  • 2023.10:  One paper Color-NeuS is accepted by 3DV 2024.
  • 2023.09:  Two papers were accepted by NeurIPS 2023. One spotlight and one poster.
  • 2023.07:  Introduce 3D-LLM, the first Large Language Model that could take 3D representations as inputs.
  • 2023.07:  One paper CHORD is accepted by ICCV 2023.
  • 2023.04:  One paper IOT-CL is accepted by ICML 2023.
  • 2022.04:  Build the website: AISJTU.ICU.

Publications

arXiv
sym

RapVerse: Coherent Vocals and Whole-Body Motions Generations from Text

Jiaben Chen, Xin Yan, Yihang Chen, Siyuan Cen, Qinwei Ma, Haoyu Zhen, Kaizhi Qian, Lie Lu, Chuang Gan

arXivProjectCode

GitHub Stars Demo views

ICML 2024

3D-VLA: 3D Vision-Language-Action Generative World Model

Haoyu Zhen, Xiaowen Qiu, Peihao Chen, Jincheng Yang, Xin Yan, Yilun Du, Yining Hong, Chuang Gan

arXivProjectCodeTwitter

ICML 2024

NeurIPS 2023

3D-LLM: Injecting the 3D World into Large Language Models

Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, Chuang Gan

arXivProjectCodeTwitter

NeurIPS 2023 (Spotlight)

NeurIPS 2023
sym

Relative Entropic Optimal Transport: a (Prior-aware) Matching Perspective to (Unbalanced) Classification.

Liangliang Shi, Haoyu Zhen, Gu Zhang, Junchi Yan

NeurIPS 2023

3DV 2024

Color-NeuS: Reconstructing Neural Implicit Surfaces with Color

Licheng Zhong, Lixin Yang, Kailin Li, Haoyu Zhen, Mei Han, Cewu Lu

arXivProjectCodeData

3DV 2024

ICCV 2023
sym

CHORD: Category-level in-Hand Object Reconstruction via Shape Deformation

Kailin Li, Lixin Yang, Haoyu Zhen, Zenan Lin, Xinyu Zhan, Licheng Zhong, Jian Xu, Kejian Wu, Cewu Lu

arXivProject

ICCV 2023

ICML 2023
sym

Understanding and Generalizing Contrastive Learning from the Inverse Optimal Transport Perspective

Liangliang Shi, Gu Zhang, Haoyu Zhen, Jintao Fan, Junchi Yan

OpenReviewSlides

ICML 2023

Projects

PyBlend
sym

PyBlend: A Python Library for Blender

pip install pyblendPyPIGithubGallery

PyBlend, a project I’ve passionately contributed to, is a Python library that provides a high-level interface for Blender scripting. It empowers artists and developers like me to effortlessly enhance Blender workflows with an array of dynamic functions.

AI-SJTU
sym

A Website For Courses of AI Pilot Class, SJTU

Founder and Primary Contributor: Haoyu Zhen

Project Source CodeHomepage

I built a community and developed a website for the AI pilot class at Shanghai Jiao Tong University. The website enables students to evaluate the quality of different courses and share learning materials.

Honors and Awards

  • 2024.04 Outstanding Graduate of Shanghai Jiao Tong University.
  • 2023.12 Rong Chang Scholarship of SJTU (10 recipients, 30000¥).
  • 2021/2022/2023.11 Academic Excellence Scholarship of SJTU (top 10%).
  • 2021.11 Huawei “Smart Base” Scholarship.
  • 2019.10 The first prize of the 36th CPHO semi-finals, Beijing Division (Rank: 17, top 1%).

Education

University of Massachusetts Amherst

PhD Student, 2024.09 - Present

Advisor: Chuang Gan

Massachusetts Institute of Technology

Visiting Student, 2023.06 - 2023.12

Advisor: Josh Tenenbaum

Shanghai Jiao Tong University

Undergraduate, Artificial Intelligence, 2020.09 - 2024.06

Advisor: Junchi Yan, Cewu Lu

Detailed Academic Performance [Projects and Lecture Notes]
  • GPA: 92.80/100, 4.03/4.3 (Rank: 4/88)
  • 4th semester: GPA:95.24/100, 4.20/4.3 (Rank 1/89).
  • Sophomore year: GPA:94.04/100, 4.12/4.3 (Rank 2/89).

Experiences

Meshy, Taichi Graphics

Intern, 2024.02 - 2024.08

Host: Tiantian Liu

Machine Vision and Intelligence Group

Research undergraduate, 2021.12 - 2024.08

Advisor: Cewu Lu

SJTU-ReThinkLab

Research undergraduate, 2022.09 - 2023.05

Advisor: Junchi Yan

Services

Reviewer:

  • ICLR 2025, ICRA 2025, AAAI 2024/2025, WACV 2025

Teaching Assistant:

Life

  • I spent a wonderful time at the Beijing National Day School from 2014 to 2020. I was a member of the Class of Physics Olympiad.
  • My best friend: Yilin Sun. In our sophomore year, we teamed up to finish our first project in AI2611 Machine Learning:
    GEFF sym
    We proposed a novel method for extracting gaze features from the face and eyes using the PIXIE model and contrastive learning (SimCLR) framework
    - Gaze Estimation with Fused Features.
  • Previously, I was a player in the
    Badminton Team of SEIEE sym
    Nov. 2021, School of Electronic Information and Electrical Engineering vs. School of Mechanical Engineering Badminton Match.
    and the
    SJTU frisbee team. sym
    Oct. 11, 2022, taken at the frisbee pickup game, Zhiyuan Football Field, SJTU.
    In addition to these sports, I also enjoy playing baseball, table tennis, football, and basketball.
  • I am a fan of DC movies, as well as the TV series Rick & Morty, Breaking Bad, and Better Call Saul.