Oct 1, 2024

Reality Promises has been recognized with the Best Paper Award at ACM UIST 2025 in Busan, South Korea.

Aug 24, 2025
Multiple outlets, including the Princeton Engineering blog and UploadVR, reported about Reality Promises.
Aug 22, 2025
Introducing Reality Promises that blend and bend reality to the user's needs–enabled by robotics and gaussian splatting. I will give a talk on our work at ACM UIST 2025 in Busan, South Korea. In the meantime, check-out the material on this project:

Older News

Dec 10, 2024
My first 100 days at Princeton University have passed, mostly spent on the next exciting research project! Some impressions of work and beyond:
Mo Kari guiding Bobby Murphy through an augmented reality experience while using a Quest 3 headset
demoing research prototypes
to Snap founder and CTO Bobby Murphy
a photo of Nassau Hall with fall vibes
enjoying walks over the beautiful Princeton campus
Mo on campus
taking photos
Prof. Yann LeCun giving a talk on AI and showing a slide that says to not work LLMs for progress in AI
drawing inspiration from Turing award winners (here: Prof. Yann LeCun) and aspirants thereof
Princeton Univesity Orchestra on stage
enjoying the Princeton University Orchestra
Mo working at the desk
however, research
looks like this 99% of the time
More photos of my first 100 days are on my blog.
Aug 19, 2024

I have obtained my PhD (Dr. rer. nat) in Computer Science after submitting and defending my dissertation on Situated and Semantics-Aware Mixed Reality.

My dissertation is available below. Find some Top 3 moments of my PhD time also below. A few thoughts about my PhD time can be found here.

Academic Experience

Princeton University
Department of Computer Science | Princeton HCI
Under a Presidential Postdoctoral Research Felllowship, I currently work on semantically controlled robots in Prof. Dr. Parastoo Abtahi's Situated Interactions lab.
ETH Zürich
Department of Computer Science | Institute for Intelligent Interactive Systems
I was a visiting PhD student to Prof. Dr. Christian Holz's Sensing, Interaction & Perception Lab to work on space-constrained input in mobile VR.
California State University, Los Angeles
During my Master's degree, I was awarded a Fulbright full grant for studying a term in the United States.
University of Duisburg-Essen
Faculty of Business Administration | Institute for Computer Science and Business Information Systems
Before joining Prof. Dr. Reinhard Schütte's Integrated Information Systems research group as a PhD student in Computer Science, I obtained a bachelor's and a master's degree in Information Systems, and had worked as a student assistant in Prof. Schütte's group and Prof. Dr. Stefan Eicker's Software Engineering research group.

Industry Experience

Apple
Hardware Tech | Sensor Software & Prototyping
Advised by Dr. Tom Runia and Dr. Vittorio Megaro in Dr. Brian Amberg's Zurich Vision Lab, I've worked on AI-based 3D computer vision for the future of FaceTime Personas on Apple Vision Pro.
Meta
Reality Labs | Research
Advised by Dr. Raj Sodhi in Dr. Hrvoje Benko's interaction research team, I worked on Mixed Reality experiences with semantic scene awareness.
Porsche
Finance & Technology

Advised by Dr. Tobias Grosse-Puppendahl and Jochen Gross, I co-founded Porsche Emerging Tech Research and made it successful within 18 months (incl. 2 first-author papers and > 15 patent filings). My work focused on context-aware in-car and around-car mixed reality systems. I hired and guided people, built the ML stack, and pushed organizational boundaries on the way.

Before my PhD, I was advised by Dr. Andrada Junge and Martin Mayer as a data science intern on vehicle quality analyics, authored my master's thesis on estimating remaining EV battery life from past usage in a fleet-learning approach, and joined as a freelance data engineer to build a parallelized terabyte-scale vehicle data processing pipeline.

Accenture
Strategy
I interned on a strategic post-merger integration initiative for an automotive client to develop a migration and integration plan towards a cloud-native microservice architecture, following the acquisition of multiple mobility startups.
Cinuru
I interned as a software engineer at Cinuru - a MediaTech startup from the ecosystem of the Hasso Plattner Institute and the Film University Babelsberg in Potsdam, working on FFT-based movie trailer recognizition on off-grid mobile phones.

Selected Publications

For a full list of publications, please refer to my Google Scholar profile.

Papers

Mohamed Kari, Parastoo Abtahi
Reality Promises: Virtual-Physical Decoupling Illusions in Mixed Reality via Invisible Mobile Robots
38th Annual ACM Symposium on User Interface Software and Technology (UIST), 2025.
 Best Paper Award
Humans incessantly manipulate objects in their environment. Yet, mixed reality systems fall short of enabling seamless manipulations of the physical scene, constraining experiences to virtual effects. In this paper, we present the concept of Reality Promises, the mixed-reality illusion of manipulating the scene in ways only virtuality affords while secretly propagating virtual manipulations to physical reality. By decoupling virtual modes of manipulations from the physical mode of manipulation, Reality Promises create the illusion of manipulating the physical scene instantaneously, using magical forces or fantastical creatures. Concealed from the user, our system directs a mobile robot that manipulates physical objects between dynamic virtual-physical decoupling and recoupling points without revealing itself to the user. To render the robot invisible and physical objects interactable, we introduce a robot-aware 3D Gaussian splat rasterization, shading, and animation system that renders splats co-aligned with the local space into the user's passthrough view where needed. We systematically derive interaction protocols that provide cohesive end-to-end user experiences, such as materializing objects out of thin air or applying user or character-induced virtual forces to physical objects.
Scene Responsiveness for Visuotactile Illusions in Mixed Reality
36th Annual ACM Symposium on User Interface Software and Technology (UIST), 2023.
 Honorable Mention Award
Manipulating their environment is one of the fundamental actions that humans, and actors more generally, perform. Yet, today's mixed reality systems enable us to situate virtual content in the physical scene but fall short of expanding the visual illusion to believable environment manipulations. In this paper, we present the concept and system of Scene Responsiveness, the visual illusion that virtual actions affect the physical scene. Using co-aligned digital twins for coherence-preserving just-in-time virtualization of physical objects in the environment, Scene Responsiveness allows actors to seemingly manipulate physical objects as if they were virtual. Based on Scene Responsiveness, we propose two general types of end to-end illusionary experiences that ensure visuotactile consistency through the presented techniques of object elusiveness and object rephysicalization. We demonstrate how our Daydreaming illusion enables virtual characters to enter the scene through a physically closed door and vandalize the physical scene, or users to enchant and summon far-away physical objects. In a user evaluation of our Copperfield illusion, we found that Scene Responsiveness can be rendered so convincingly that it lends itself to magic tricks. We present our system architecture and conclude by discussing the implications of scene-responsive mixed reality for gaming and telepresence.
Mohamed Kari, Christian Holz
HandyCast: Phone-based Bimanual Input for Virtual Reality in Mobile and Space-Constrained Settings via Pose-and-Touch Transfer
ACM Conference on Human Factors in Computing Systems (CHI), 2023.
Despite the potential of Virtual Reality as the next computing platform for general purposes, current systems are tailored to stationary settings to support expansive interaction in mid-air. However, in mobile scenarios, the physical constraints of the space surrounding the user may be prohibitively small for spatial interaction in VR with classical controllers. In this paper, we present HandyCast, a smartphone-based input technique that enables full-range 3D input with two virtual hands in VR while requiring little physical space, allowing users to operate large virtual environments in mobile settings. HandyCast defines a pose-and-touch transfer function that fuses the phone's position and orientation with touch input to derive two individual 3D hand positions. Holding their phone like a gamepad, users can thus move and turn it to independently control their virtual hands. Touch input using the thumbs fine-tunes the respective virtual hand position and controls object selection. We evaluated HandyCast in three studies, comparing its performance with that of Go-Go, a classic bimanual controller technique. In our open-space study, participants required significantly less physical motion using HandyCast with no decrease in completion time or body ownership. In our space-constrained study, participants achieved significantly faster completion times, smaller interaction volumes, and shorter path lengths with HandyCast compared to Go-Go. In our technical evaluation, HandyCast's fully standalone inside-out 6D tracking performance again incurred no decrease in completion time compared to an outside-in tracking baseline.
TransforMR: Pose-Aware Object Substitution for Composing Alternate Mixed Realities
20th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2021.
Despite the advances in machine perception, semantic scene understanding is still a limiting factor in mixed reality scene composition. In this paper, we present TransforMR, a video see-through mixed reality system for mobile devices that performs 3D-pose-aware object substitution to create meaningful mixed reality scenes. In real-time and for previously unseen and unprepared real-world environments, TransforMR composes mixed reality scenes so that virtual objects assume behavioral and environment-contextual properties of replaced real-world objects. This yields meaningful, coherent, and humaninterpretable scenes, not yet demonstrated by today's augmentation techniques. TransforMR creates these experiences through our novel pose-aware object substitution method building on different 3D object pose estimators, instance segmentation, video inpainting, and pose-aware object rendering. TransforMR is designed for use in the real-world, supporting the substitution of humans and vehicles in everyday scenes, and runs on mobile devices using just their monocular RGB camera feed as input. We evaluated TransforMR with eight participants in an uncontrolled city environment employing different transformation themes. Applications of TransforMR include real-time character animation analogous to motion capturing in professional film making, however without the need for preparation of either the scene or the actor, as well as narrative-driven experiences that allow users to explore fictional parallel universes in mixed reality. We make all of our source code and assets available
SoundsRide: Affordance-Synchronized Music Mixing for In-Car Audio Augmented Reality
34th Annual ACM Symposium on User Interface Software and Technology (UIST), 2021.
 Best Paper Award
Also presented at: ACM SIGGRAPH 2023. The Future of Advanced User Interfaces: Best of UIST session.
Music is a central instrument in video gaming to attune a player's attention to the current atmosphere and increase their immersion in the game. We transfer the idea of scene-adaptive music to car drives and propose SoundsRide, an in-car audio augmented reality system that mixes music in real-time synchronized with sound affordances along the ride. After exploring the design space of affordance-synchronized music, we design SoundsRide to temporally and spatially align high-contrast events on the route, e. g., highway entrances or tunnel exits, with high-contrast events in music, e. g., song transitions or beat drops, for any recorded and annotated GPS trajectory by a three-step procedure. In real-time, SoundsRide 1) estimates temporal distances to events on the route, 2) fuses these novel estimates with previous estimates in a cost-aware music-mixing plan, and 3) if necessary, re-computes an updated mix to be propagated to the audio output. To minimize user-noticeable updates to the mix, SoundsRide fuses new distance information with a filtering procedure that chooses the best updating strategy given the last music-mixing plan, the novel distance estimations, and the system parameterization. We technically evaluate SoundsRide and conduct a user evaluation with 8 participants to gain insights into how users perceive SoundsRide in terms of mixing, affordances, and synchronicity. We find that SoundsRide can create captivating music experiences and positively as well as negatively influence subjectively perceived driving safety, depending on the mix and user.

Dissertation

Mohamed Kari
Situated and Semantics-Aware Mixed Reality
University of Duisburg-Essen, Germany.
 with highest distinction

Computers moved from the basement onto the user's desk into their pocket and around their wrist. Not only did they physically converge to the user's space of action, but sensors such as CMOS, GPS, NFC, and LiDAR also began to provide access to the environment, thus situating them in the user's direct reality. However, the user and their computer exhibit fundamentally different perceptual processes, resulting in substantially different internal representations of reality.

Augmented and Mixed Reality systems, as the most advanced situated computing devices of today, are mostly centered around geometric representations of the world around them, such as planes, point clouds, or meshes. This way of thinking has proven useful in tackling classical problems including tracking or visual coherence. However, little attention has been directed toward a semantic and user-oriented understanding of the world and the specific situational parameters a user is facing. This is despite the fact that humans not only characterize their environment by geometries, but they also consider objects, the relationships between objects, their meanings, and the affordance of objects. Furthermore, humans observe other humans, how they interact with objects and other people, all together stimulating intent, desire, and behavior. The resulting gap between the human-perceived and the machine-perceived world impedes the computer's potential to seamlessly integrate with the user's reality. Instead, the computer continues to exist as a separate device, in its own reality, reliant on user input to align its functionality to the user's objectives.

This dissertation on Situated and Semantics-Aware Mixed Reality aims to get closer to Augmented and Mixed Reality experiences, applications, and interactions that address this gap and seamlessly blend with a user's space and mind. It aims to contribute to the integration of interactions and experiences with the physicality of the world based on creating an understanding of both the user and their environment in dynamic situations across space and time.

Method-wise, the research presented in this dissertation is concerned with the design, implementation, and evaluation of novel, distributed, semantics-aware, spatial, interactive systems, and, to this end, the application of state-of-the-art computer vision for scene reconstruction and understanding, together enabling Situated and Semantics-Aware Mixed Reality.

Blog and Projects

Find my blog on computing and ML infrastucture and other stuff here.

Projects

3D Coordinate Converter
BibTeX Cleaner

Blog Posts

Website-only Projects

Villa Reverie Homepage
Dünenhuus Homepage

PhD Top 3 Experiences

Top 3: Conference Talks

ACM CHI 2023 in Hamburg
ACM CHI 2023 in Hamburg
presenting HandyCast
ACM SIGGRAPH 2023 in Los Angeles
ACM SIGGRAPH 2023 in Los Angeles
presenting SoundsRide
ACM UIST 2023 in San Francisco
ACM UIST 2023 in San Francisco
presenting Scene Responsiveness

Top 3: Awesome Offices

Meta's 71st floor in #30HY in Manhattan
Meta's 71st floor in #30HY in Manhattan
working on Scene Responsivesness
A Porsche Taycan prototype
A Porsche Taycan prototype
working on SoundsRide
ETH Zurich's Main Campus
ETH Zurich's main campus
working on HandyCast

Top 3: Technology Pioneer Encounters (aka: Always chase meeting & talking to your heroes!)

Demoing and discussing Scene Responsiveness with Mike Abrash (Graphics Programming Pioneer, Oculus/Meta Chief Scientist, Head of Meta RL Research)
Conference encouter with David Holz (Founder of Midjourney)
Fulbright PULSE interview with Avery Wang (Inventor of the Shazam algorithm)

Top 3: Porsches

Porsche Carrera S
Porsche Cayman GT4
Porsche Taycan

Top 3: Retreats

UDE IIS in Amsterdam
IIS Retreat in Amsterdam 2021
Porsche Emerging Tech Research at the Grossglockner 2024
Porsche Emerging Tech Research at the Grossglockner 2024

Top 3: Moments in the Porsche PhD Community

PhD Podcast, here live-podcasting with Mattias Ulbrich, CIO Porsche, CEO Porsche Digital
Welcoming new PhD students in the community
Meet-up with Oli Blume, CEO Porsche & VW

Top 3: Lunch & Dinner Routines

SIPLAB (The name says it all!) lunch on the ETH Dozentenfoyer
Meta RL-R Computational Intelligence (Free food tastes double...)
Redmond & Bellevue Post-Work Dining

Top 3: Team Building Exercises

Inviting your PhD advisor to a Porsche Taycan factory tour
Beating your Meta supervisor in Whirlyball
Watching an Apple keynote where the team's feature is launched

Top 3: Mentors

Vittorio, Tom, & Masi helping me become the scientist I aspire to be
Raejoon listening to me
Jochen enabling me

Top 3: Press Reports


Top 3: Interactions with the University President (Prof. Dr. Ulrich Radtke)

Conferment of honors
Scholarship celebration
Alumni talk on the 10-year scholarship program anniversary

Top 5: Trajectory Changers

In spaceflight, planets can provide gravity assists to a spacecraft in order to alter its path. I was lucky enough to encounter, orbit, and swing-by multiple extraordinary individuals that changed my PhD (and life) trajectory.

My teacher Lukas Hillesheim
introducing me to programming
My PhD advisor Prof. Dr. Reinhard Schütte
shaping me in uncountable aspects and introducing me to science
My PhD peer -now Dr.- David Bethge
10x-ing my amibition
My Porsche advisor Dr. Tobias Grosse-Puppendahl
introducing me to research excellence in HCI
My ETH host & collaborator Prof. Dr. Christian Holz
introducing me to the tools of the trade in HCI

Top 3: Learnings on research and science

Scientists are story tellers. For a long time, I thought scienstists say what is (re-contextualizing Augstein) and the scientific method produces the output logically determined like a mechanical system. However, turns out the scientific method is a only tool to figure out a story as it unfolds. Still, the scientist needs to tell it. At least, the impactful scientist.

In fact, I think every impactful individual is a story teller. Only by telling a story do you convince reviewers, students, supervisors, and job candidates. And, most importantly, convince yourself.

Of course, there are differences between scientific stories and other stories. Why do you tell a story? How do you tell a story? To whom do you tell a story? How convincing is your story? How much does your story correspond with reality, how plausible is it, and how well is it grounded in scientifically tenable statements? I'd say, scientists are some of the best story tellers in the world as they tell a previously untold story. An optimistic story. As stringent as it gets. An easy-to-follow story for an often highly complex, non-fictional plot.

Thoughts inspired from discussions with Reinhard Schütte, Tobias Grosse-Puppendahl, and Bettina Luescher.

Science is the endeavour of solving difficult problems under uncertainty. These difficult problems are generally rooted in the lack of non-trivial knowledge.

That's why great science is closely related to great enterpreneurship. Both activities share that they are predominantly characterized by uncertainty. Sometimes, the whole problem and its boundaries are uncertain. Sometimes, the problem determinants are uncertain. And sometimes, "only" the solution to an already well-structured problem is uncertain.

Therefore, conducting research is mostly the process of eliminating uncertainties, e.g., through implementation, observation, or reasoning.

Thoughts inspired from discussions with Reinhard Schütte and Andreas Fender.

Individuals make the difference. Institutions have no talent, knowledge, skill, experience, or expertise. Individuals do.

Of course, institutions are better or worse in their tendency of attracting individuals with talent, etc. But talented people can be where you don't expect them and vice versa. Therefore, you need to be on the constant lookout for talent.

In science-and any creative role really, this specifically means if there is a great outcome, it is not due to the institution. It is because of one good individual. And, maybe they were renting out DVDs in the video rental store, only to write and direct Pulp Fiction a few years later.

Sometimes, one might think there are whole teams or departments that created something. But there is no next-level support when you are already at the edge. Nope, in fact, the better the outcome, the lower the number of people that are behind it. Either you solve whatever problem, or it remains a problem. Either you are the expert, or no one is. Either you develop the skill, or no one has it. Either you do it, or no one does.

PS: Especially in larger organizations, there is the tendency to develop the believe that good processes can replace good individuals. That's a fallacy. (I've never heard anyone talk of Agile Development in any good software company. Just get your code into CI.)

Thoughts inspired from discussions with Reinhard Schütte, Christian Holz, and Daniel Cremers.

Podcasts

At Porsche, I have initiated, produced, and together with my fellow head of the PhD network David Bethge, co-hosted the Porsche PhD Podcast where we interviewed Porsche PhD students company-internally on their research and Porsche executives on the role of PhD research for innovation.
As a Fulbright alumnus, I have initiated, and now produce and host the Fulbright PULSE podcast together with the German-American Fulbright Commission and the German Fulbright Alumni Association. Tune in on Apple Podcasts or Spotify on the first Monday of the next month or listen in on one of the following episodes!
Prof. Dr. Gloria Mark
Fulbright '05/06 @ Humboldt University of Berlin

Chancellor's Professor of Human-Computer Interaction at UC Irvine

Prof. Anett-Maud Joppien
Fulbright '85-87 @ UC Berkeley & University of Illinois, Chicago

Architect and Professor of Architecture at TU Darmstadt

Manuel Bewarder
Fulbright '08/09 @ American University, Washington DC

Investigative Journalist at WDR and NDR

Prof. Dr. Rainer Storb
Fulbright '65 @ University of Washington

Pioneer of the Stem Cell Transplantation

Vera Kostiuk Busch & Dr. Hanna Shvindina
Fulbright @ Middlebury College & Fulbright '18-'19 @ Purdue University

Charity/NGO Founders in Support of Ukraine

Alexander Graf Lambsdorff
Fulbright '93 @ Georgetown University

German Ambassador to Russia

Dr. Avery Wang
Fulbright '90-91 @ Ruhr University Bochum

Co-Founder and Former Chief Scientist at Shazam, Principal Research Scientist at Apple

Dr. Juliane Kronen
Fulbright '84-85 @ University of Missouri

Founder and Director of Innatura gGmbH, Jury Member of the Right Livelihood Award, and Former Partner at Boston Consulting Group

Dr. Philip Häusser
Fulbright '14 @ University of California, Santa Cruz

Science Communicator in TV and Web & Chief Technology Officer at Ablacon

Prof. Dr. Daniel Cremers
Fulbright '95 @ Stony Brook and Indiana State

Professor of Computer Vision and Artificial Intelligence at TU Munich

Bettina Luescher
Fulbright '83-85 @ University of Wisconsin-Madison

Former Chief Spokesperson of the UN World Food Programme and Host of CNN International

Camera Work

Aerial Videography

Travel Photography

Please see my travel photography website for more photos.