Avatar

PhD Student

Carnegie Mellon University

Biography

I am a first year PhD student in Robotics Student at Carnegie Mellon University working with Prof. Katerina Fragkiadaki. Previously, I received M.S. in Robotics from Carnegie Mellon University advised by Prof. Katerina Fragkiadaki and B.E. in Computer Science from BITS Pilani, Pilani Campus where I spent one year working with Prof. Katerina Fragkiadaki as a research assistant in Machine Learning Department, Carnegie Mellon University.

I am interested in Computer Vision and its application to embodied learning. Lately, I have been interested in relationship between various visual modalities: how are 2D, 3D and video modalities connected together? How can one benefit the other? In my free time, I love listening to podcasts and lectures related to philosophy and ethics (My recent favourite). I also like reading fiction, watching (and playing) cricket and travelling.

Virtual MLC office hours (open to everyone): I dedicate two hours every week to mentoring newcomers in the fields of robotics, computer vision, and machine learning. If you're new to these areas and need help with research, grad school applications, or anything else, please feel free to book a slot.

Interests

  • Computer Vision
  • Robotics

Education

  • PhD in Robotics, 2023-Present

    Carnegie Mellon University

  • Masters in Robotics, 2021-23 ( Thesis PDF Thesis Talk )

    Carnegie Mellon University

  • B.E. in Computer Science, 2017-21

    BITS PILANI, Pilani Campus

News

[Sep 2023] Awarded best reviewer award in ICCV 2023!

[Apr 2023] I’ll be continuing in Robotics Institute at CMU for my PhD!

[Oct 2022] I’ll be presenting BUTD-DETR at ECCV 2022!

[Aug 2022] I’ll be attending Alexa Prize Simbot Bootcamp in Seattle!

[May 2022] I’ll be spending my summer as a research intern at Apple with Miguel and Navdeep!

[Dec 2021] Our team got selected for Alexa Prize Simbot Challenge. We will be working on advancing teachable embodied household agents.

[Aug 2021] I’ll be starting my Masters in Robotics at Carnegie Mellon University. I will also continue as a Research Assistant with Prof. Katerina Fragkiadaki.

[Aug 2020] I’ll be attending Google Research India’s AI Summer School (Computer Vision track) from August 20-22, 2020!

[May 2020] Joined Prof. Katerina Fragkiadaki’s Lab as Research Associate at Carnegie Mellon University.

Recent Publications

ODIN: A Single Model for 2D and 3D Perception

ODIN is a single model for 2D and 3D perception.
Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W. Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, Katerina Fragkiadaki
CVPR 2024
PDF Code Project

Diffusion-ES: Gradient-free Planning with Diffusion for Autonomous Driving and Zero-Shot Instruction Following

Diffusion-ES is a flexible test-time trajectory optimization method for arbitrary reward functions that combines generative trajectory models and sampling-based search.
Brian Yang, Huangyuan Su, Nikolaos Gkanatsios, Tsung-Wei Ke, Ayush Jain, Jeff Schneider, Katerina Fragkiadaki
CVPR 2024
PDF Code Project

Energy-based Models as Zero-Shot Planners for Compositional Scene Rearrangement

We propose a model that maps spatial rearrangement instructions to goal scene configurations via gradient descent on a set of relational energy functions
Nikolaos Gkanatsios*, Ayush Jain*, Zhou Xian, Yunchu Zhang, Christopher A. Atketson, Katerina Fragkiadaki (* Equal Contribution)
RSS 2023
PDF Code Project

Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds

Most models tasked to ground referential utterances in 2D and 3D scenes learn to select the referred …
Ayush Jain*, Nikolaos Gkanatsios*, Ishita Mediratta, Katerina Fragkiadaki (* Equal Contribution)
ECCV 2022.
PDF Code Project Blog

Move to See Better: Towards Self-Improving Embodied Object Detection

Humans learn to better understand the world by moving around their environment to get more …
Zhaoyuan Fang*, Ayush Jain*, Gabriel Sarch*, Adam Harley, Katerina Fragkiadaki (* Equal Contribution)
BMVC 2021.
PDF Code Project

AI-enabled Object Detection in Unmanned Aerial Vehicles for Edge Computing Applications

Unmanned Aerial Vehicles (UAVs) are emerging as a powerful tool for various industrial and smart …
Ayush Jain*, Rohit Ramaprasad*, Pratik Narang, Murari Mandal, Vinay Chamola, F. Richard Yu, Mohsen Guizani (* Equal Contribution)
IEEE Network. 2021.
PDF Code Project

VisDrone-DET2020: The Vision Meets Drone Object Detection in Image Challenge Results

The Vision Meets Drone Object Detection in Image Challenge (VisDrone-DET 2020) is the third annual …
Dawei Du, Longyin Wen, Pengfei Zhu, Heng Fan, Qinghua Hu, Haibin Ling, Mubarak Shah, Junwen Pan, Ayush Jain, Pratik Narang, et al.
ECCV 2020 Workshop.
PDF

NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain

Significant advances have been made in recent years on Natural Language Processing with machines …
Ayush Jain*, N.M. Meenachi, B. Venkatraman
Arxiv 2020.
PDF Code Project Slides

Experience

 
 
 
 
 

Research Intern

Apple

May 2022 – August 2022 Cupertino, California, USA
 
 
 
 
 

Graduate Research Assistant

Carnegie Mellon University

Aug 2021 – Present Pittsburgh, Pennsylvania, USA
With Prof. Katerina Fragkiadaki, I am working on language grounding in static 2D and 3D scenes, robot manipultation following language instructions, and instruction following in indoor household environment.
 
 
 
 
 

Research Associate

Carnegie Mellon University

May 2020 – Jul 2021 Pittsburgh, Pennsylvania, USA
Under the supervision of Prof. Katerina Fragkiadaki, I worked on enabling an embodied agent to learn about objects without ground truth supervision in an unseen 3D environment just by moving around.
 
 
 
 
 

Research Assistant

BITS PILANI

Aug 2019 – Apr 2020 Rajasthan, India
As a research assistant, I worked under the guidance of Prof. Pratik Narang in computer vision. My team and I proposed a novel architectural design that improved the performance of previous methods by a substantial margin.

Academic Service and Teaching

I reviewed for ICCV 2023 (best reviewer award), Neurips 2023, AAAI 2023, CVPR 2022, ECCV 2022, BMVC 2020-21, TPAMI 2021-22

I have served as a teaching assistant for the following courses at BITS Pilani:

  • CS F407 Artificial Intelligence, Fall 2020
  • CS F464 Machine Learning, Spring 2020
  • CS F111 Computer Programming, Spring 2019 and Fall 2019