Applied Scientist - Multimodal Foundation Models & Robotics

Share job

The Spatial AI Lab is part of the Applied Sciences Group, a Microsoft research and development organization dedicated to creating next-generation human-computer interaction technologies leveraging the most recent AI developments and exploring new hardware capabilities and device form-factors. Our team of scientists and engineers has strong expertise in computer vision and multi-modal AI, with a particular focus on spatial and embodied AI.

As part of our growing team, you will conduct research at the intersection of large-scale generative modeling and embodied AI, with a focus on robotics. Your primary focus will be on building the core intelligence for a new generation of agents, training the multimodal foundation models that empower them to perceive complex environments, reason about tasks, and act seamlessly across both the physical and digital worlds. This opportunity will allow you to deepen your expertise in training embodied foundation models, deploying algorithms on robotic hardware and large-scale AI systems, and contribute to our pioneering research through publications and collaborations with partners like ETH Zurich.

Microsoft’s mission is to empower every person and every organization on the planet to achieve more. As employees we come together with a growth mindset, innovate to empower others, and collaborate to realize our shared goals. Each day we build on our values of respect, integrity, and accountability to create a culture of inclusion where everyone can thrive at work and beyond.

Required Qualifications:
• A PhD in Robotics, Machine Learning, or a related field, OR 3+ years of relevant industry experience.
• Demonstrated coding, debugging, and engineering skills in programming languages such as Python or C++.
• Hands-on experience with modern deep learning frameworks (e.g. Pytorch/Tensorflow/Jax).
• Self-motivated team-player, problem solver, and keen to learn.
• Ability to present complex technical concepts to a diverse audience.

Preferred Qualifications:
Experience in one or more of the following areas:
• Foundation Models: hands-on training experience in at least one of the following topics: LLMs; Large vision-language models (VLMs); Video generative models and diffusion algorithms; or action-based transformers and Vision Language Action models (VLAs).
• Large-Scale ML Systems: Experience with large scale machine learning compute systems.
• Robotics:
o Hands-on training experience in robot learning techniques, such as reinforcement learning, imitation learning as well as classical control methods
o Solid understanding of robot kinematics, dynamics and sensors
o Familiarity with control algorithms such as PID, model predictive control (MPC), and whole-body control.

Track record of impact, either via first author research publications at top-tier machine learning or robotics conferences (CoRL, RSS, NeurIPS, ICML, ICLR, CVPR), or via contributions to successful industry initiatives.

#W+DJOBS

• Design and implement novel foundation models and algorithms for general-purpose embodied agents;
• Implement high-performance machine-learning pipelines and optimize data and learning stacks for scalability, efficiency, and performance.
• Optimize and deploy AI models on robot hardware;
• Collaborate across Microsoft research and engineering teams to transition cutting-edge research into real-world impact.
• Drive the team’s rapid progress, with success measured by both the advancement of internal capabilities and impactful contributions to the scientific community
• Opportunity to collaborate with top academic partners like ETH Zurich to advance pioneering research at scale, with opportunities to co-author work in top-tier venues, present at workshops, and mentor students.