How We Built Eye Tracking and Semi-Physical Climbing for Reach The Clouds

Two ideas worth combining

At Carbon Studio we like pursuing ideas that make VR feel more like a place than a gadget. For Reach The Clouds — the Meta Quest Pro demo we announced earlier this year — two ideas we had been circling for a while finally ended up in the same prototype: using eye tracking as a real input method, and building a semi-physical climbing system that turned movement into gameplay rather than traversal.

On paper, both sound like features you tick a box for. In practice, both opened up a lot of design questions — and a handful of technical ones we did not quite expect.

Eye tracking in practice — OVREyeGaze + raycast

We used Meta OVREyeGaze to read the player’s gaze direction. That is the easy part — the SDK gives you a Unity Transform aligned with where the eyes are pointing. What it does not give you is what the player is actually looking at in the scene. Gaze is a direction, not a reference to an object.

So we added a second layer: from the gaze’s forward vector we cast a raycast into the world, and checked whether the ray hit anything interactive. If it did, the object on the other end handled the rest.

This matches how Meta’s own docs suggest wiring eye tracking into Unity scenes — OVREyeGaze as the source of direction, raycasting as the way to resolve that direction into a concrete target. The choice we had to make was how much logic lived in the raycaster, and how much lived on each object.

Eye Interactable — separating detection from response

We ended up splitting the work into two systems:

• Gaze detection — one component, responsible for reading OVREyeGaze, casting the ray, and deciding what the player is looking at.
• Eye Interactable — a custom component that lives on each targetable object in the scene and defines what should happen when that object is being gazed at.

That separation sounds obvious in retrospect, but it gave us a lot back. The detector did not need to know about branches, switches, or cloud layers — it just emitted “this object was gazed at” events. Each Eye Interactable was free to start a branch growing, light up, play a sound, or chain into any other mechanic, without touching the gaze code at all.

Gameplay code stayed close to the gameplay. Input code stayed close to input.

Raycast and LineRenderer — detection and debugging

Two tools did most of the heavy lifting inside the gaze detector.

The raycast itself was the detection backbone — the same Unity API you would use for any pointer-style selection, just fed by the eyes instead of a controller.

The LineRenderer was our debugging friend. Eye tracking is hard to debug by feel — you cannot easily see what your own eyes are doing. So we drew the ray in the scene during development and watched it move as we moved our gaze. That let us spot drift, misalignment, and edge cases that would have been almost impossible to catch otherwise.

By the end of the prototype the LineRenderer was off in the shipping build — but it stuck around in dev builds for a long time.

The hardest part — interpreting where someone is actually looking

Here is the problem nobody tells you about up front: a naive gaze ray does not hit what you are looking at.

In our first implementation we took the forward vector of the eye transform and shot a straight ray into the scene. It worked. It just did not feel right. The ray would punch “past” the object we were looking at, hit something far behind, and the game would react to the wrong thing.

The issue was a gap between the raw input (a direction) and the intent behind it (a point of focus). Eyes do not stare in a perfectly straight line at infinity — they converge on something, and that “something” is what the player is thinking about. Our simplistic ray ignored convergence entirely.

We spent a meaningful chunk of the prototype refining this. Without going too deep: the fix was to stop treating eye direction as a single vector into the void, and start treating it as a way to approximate the player’s real focus point in 3D space. Once that clicked, everything downstream — the Eye Interactable triggers, the LineRenderer visualisations, the play-test feedback — started feeling honest.

Production challenges — a new library on a custom player controller

Two things made the implementation harder than a clean integration would suggest.

First, Meta OVREyeGaze was relatively new while we were building with it. The official documentation existed, but examples were thin, and community knowledge even thinner. Every time we hit a weird edge case, we were also building the tooling to understand it.

Second, and more painfully: we were on a custom player controller, not Meta’s default. The docs — and frankly most eye-tracking tutorials out there — assume you are using OVRCameraRig with OVRManager, enabling Eye Tracking Support in the rig, and requesting Eye Tracking permissions on startup through Meta’s usual boilerplate. That is a perfectly sensible default. It also happens to be the one we had overridden.

So we had to re-do all of that by hand: wiring eye-tracking support into our own rig, requesting permissions through our own app-startup flow, making sure the gaze data ended up in the right transform at the right time. Nothing about it was technically hard. It was just a series of small decisions that each had to be re-made on top of our own player architecture.

Semi-physical climbing — movement that weighs something

The other pillar of Reach The Clouds was a semi-physical climbing system. The name is ugly; the idea is simple.

We wanted climbing in VR to preserve the physical feel of movement — you reach, you grab, your weight is somewhere — without becoming so physical that it got in its own way. Fully physical climbing is exhausting and unreadable. Purely animated climbing is lifeless. We wanted the middle.

The mechanic is built on the player physically grabbing environmental elements — branches, outcrops, handholds — with their hands. That grab is what strengthens presence: the contact is real, the release is deliberate, and the space between grabs carries weight.

More important than the grab itself was how the player launches toward the next hold. It is easy to let VR climbing turn into a noodly drift from point to point. We spent a long time tuning the launch: how much momentum a swing gives you, how the body responds, what “too far” and “just right” feel like. The goal was rhythm — movement that had a beat, not a trajectory.

Breakable branches and the falling mechanic

To push the mechanic from traversal to gameplay, we added two things.

Branches break. Not all of them, not at random — but a branch the player grabs can snap under weight, or after too much time, or under the wrong angle. Suddenly, which branch you reach for matters.

You can fall. Falling is not just loss of progress; it is a consequence that shapes every upcoming decision. A climb in our system is never just a walk up a wall. It is a sequence of small bets: can I hold this? can I reach from here? is that branch going to give?

Together, these two additions turned a movement system into a gameplay situation. That was the goal from the start.

Technology in service of immersion

Putting eye tracking and semi-physical climbing next to each other was not an accident. Looking back, they reinforce the same idea from two directions.

Eye tracking made the player’s attention a first-class input. Where you look is now something the world knows.

Climbing made the player’s body a first-class input. How you move, how you grab, how you commit — the world knows those too.

Neither feature, on its own, would make the experience feel immersive. Together, they raised the floor of what the demo could ask of the player, and what the player could expect back.

That is, we think, where the most interesting VR work lives: at the intersection of precise input and physical involvement.

What’s next

Prototypes like Reach The Clouds are not only about the specific demo. They are about building the muscle memory for more advanced VR interaction design — the kind of thing you cannot learn from a whitepaper, only from trying.

Eye tracking is getting cheaper, more accurate, and more broadly available. Semi-physical input (hands, bodies, haptics) is mature enough to ship. The two combined are still a fast-moving area — which is exactly why it is worth experimenting with now, before the conventions are set.

If you want to see how this all comes together in actual gameplay, the Reach The Clouds announcement covers what the demo is and what the experience is meant to feel like. This post is the companion piece for the people curious about the wires underneath.