Avatar

Graduate Student

Carnegie Mellon University

Biography

I am a second year Masters in Robotics Student at Carnegie Mellon University working with Prof. Katerina Fragkiadaki. During summers of 2022, I worked as a Research Intern at Apple with Miguel A. Bautista and Navdeep Jaitley. Previously, I received B.E. in Computer Science from BITS Pilani, Pilani Campus where I spent one year working with Prof. Katerina Fragkiadaki as a research assistant in Machine Learning Department, Carnegie Mellon University.

I am interested in Computer Vision, Natural Language Processing and its application to embodied learning. I envision developing robots that continually learn by actively moving around while interacting with objects and humans. In my free time, I love listening to podcasts and lectures related to philosophy and ethics (My recent favourite). I also like reading fiction, watching (and playing) cricket and travelling.

Inspired by Krishna, I dedicate two hours every week to mentoring newcomers in the fields of robotics, computer vision, and machine learning. If you're new to these areas and need help with research, grad school applications, or anything else, please feel free to fill out this form.

Interests

  • Embodied AI
  • Computer Vision
  • Natural Language Processing

Education

  • Masters in Robotics, 2021-Present

    Carnegie Mellon University

  • B.E. in Computer Science, 2017-21

    BITS PILANI, Pilani Campus

News

[Apr 2023] I’ll be continuing in Robotics Institute at CMU for my PhD!

[Oct 2022] I’ll be presenting BUTD-DETR at ECCV 2022!

[Aug 2022] I’ll be attending Alexa Prize Simbot Bootcamp in Seattle!

[May 2022] I’ll be spending my summer as a research intern at Apple with Miguel and Navdeep!

[Dec 2021] Our team got selected for Alexa Prize Simbot Challenge. We will be working on advancing teachable embodied household agents.

[Aug 2021] I’ll be starting my Masters in Robotics at Carnegie Mellon University. I will also continue as a Research Assistant with Prof. Katerina Fragkiadaki.

[Aug 2020] I’ll be attending Google Research India’s AI Summer School (Computer Vision track) from August 20-22, 2020!

[May 2020] Joined Prof. Katerina Fragkiadaki’s Lab as Research Associate at Carnegie Mellon University.

Recent Publications

Energy-based Models as Zero-Shot Planners for Compositional Scene Rearrangement

We propose a model that maps spatial rearrangement instructions to goal scene configurations via gradient descent on a set of relational energy functions
Nikolaos Gkanatsios*, Ayush Jain*, Zhou Xian, Yunchu Zhang, Christopher A. Atketson, Katerina Fragkiadaki (* Equal Contribution)
RSS 2023
PDF

Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Clouds

Most models tasked to ground referential utterances in 2D and 3D scenes learn to select the referred …
Ayush Jain*, Nikolaos Gkanatsios*, Ishita Mediratta, Katerina Fragkiadaki (* Equal Contribution)
ECCV 2022.
PDF Code Project

Move to See Better: Towards Self-Improving Embodied Object Detection

Humans learn to better understand the world by moving around their environment to get more …
Zhaoyuan Fang*, Ayush Jain*, Gabriel Sarch*, Adam Harley, Katerina Fragkiadaki (* Equal Contribution)
BMVC 2021.
PDF Code Project

AI-enabled Object Detection in Unmanned Aerial Vehicles for Edge Computing Applications

Unmanned Aerial Vehicles (UAVs) are emerging as a powerful tool for various industrial and smart …
Ayush Jain*, Rohit Ramaprasad*, Pratik Narang, Murari Mandal, Vinay Chamola, F. Richard Yu, Mohsen Guizani (* Equal Contribution)
IEEE Network. 2021.
PDF Code Project

VisDrone-DET2020: The Vision Meets Drone Object Detection in Image Challenge Results

The Vision Meets Drone Object Detection in Image Challenge (VisDrone-DET 2020) is the third annual …
Dawei Du, Longyin Wen, Pengfei Zhu, Heng Fan, Qinghua Hu, Haibin Ling, Mubarak Shah, Junwen Pan, Ayush Jain, Pratik Narang, et al.
ECCV 2020 Workshop.
PDF

NukeBERT: A Pre-trained language model for Low Resource Nuclear Domain

Significant advances have been made in recent years on Natural Language Processing with machines …
Ayush Jain*, N.M. Meenachi, B. Venkatraman
PDF Code Project Slides

Experience

 
 
 
 
 

Research Intern

Apple

May 2022 – Present Cupertino, California, USA
 
 
 
 
 

Graduate Research Assistant

Carnegie Mellon University

Aug 2021 – Present Pittsburgh, Pennsylvania, USA
With Prof. Katerina Fragkiadaki, I am working on language grounding in static 2D and 3D scenes, robot manipultation following language instructions, and instruction following in indoor household environment.
 
 
 
 
 

Research Associate

Carnegie Mellon University

May 2020 – Jul 2021 Pittsburgh, Pennsylvania, USA
Under the supervision of Prof. Katerina Fragkiadaki, I worked on enabling an embodied agent to learn about objects without ground truth supervision in an unseen 3D environment just by moving around.
 
 
 
 
 

Research Assistant

BITS PILANI

Aug 2019 – Apr 2020 Rajasthan, India
As a research assistant, I worked under the guidance of Prof. Pratik Narang in computer vision. My team and I proposed a novel architectural design that improved the performance of previous methods by a substantial margin.

Academic Service and Teaching

I reviewed for BMVC 2020, 2021, CVPR 2022, ECCV 2022, TPAMI 2021, 2022

I have served as a teaching assistant for the following courses at BITS Pilani:

  • CS F407 Artificial Intelligence, Fall 2020
  • CS F464 Machine Learning, Spring 2020
  • CS F111 Computer Programming, Spring 2019 and Fall 2019