︎

liminal window


Discipline
Interactive media
Installation art
Mixed reality


Timeline
Nov - Dec 2023
Team Member(s)
Lening Hou

Role
Creative developer
3D texture artist
Tools
Blender
Adobe Substance 3D
three.js


An interactive installation exploring the themes of liminal space and surrealism. Featuring a fabricated window with projection mapping, it presents an eerie environment that dynamically reacts to real-time head tracking. I was responsible primarily for digital aspects including 3D texturing and rendering, programming, and scene arrangement and interactivity.


Process


Ideation

We were tasked with creating an interactive project that takes inspiration from the physical limitations of a cornell box. After research and brainstorming, this was then narrowed down to creating an installation by building a window in which the scene changes based on the viewer’s gaze. 

Conceptually, the goal was to explore the themes of liminal space and surrealism, touching on the following ideas:

  • Time: the absence of time, the passing of time, and spaces of transition or lingering.

  • Comfort vs. discomfort: scenes with elements that are calming or pleasant but, at the same time, unsettling

  • Reality: places that appear familiar and seemingly normal, yet may not exist in the real world


Inspiration images

Using a window as a constraint limits the viewer’s perspective since they are not able to freely move through the scene. This serves our purpose since they could see a vast space with no end in sight but, at the same time, feel trapped because of this physical restriction.


Technical Approach

The biggest obstacle was figuring out the technical side around eye tracking, so this was addressed first. Some initial technologies that failed:

  • Tobii Eye Tracker + Unity: This combination is only compatible with Windows and primarily geared toward gaming applications, making it less portable and impractical in the long run.

  • Gaze detection with Unity using preexisting libraries: Most eye-tracking libraries were either outdated or required calibration, thus only working well for one specific user.

  • Arduino + Unity: This was attempted by 3D printing glasses and linking it with the Arduino’s built-in gyroscope. Connection via Bluetooth was unreliable, and a wired connection interfered with the immersion in the project.




Development

The final interaction was then developed using ml5.js, three.js, and A-Frame as they seemed to be the most flexible for integrating gaze detection. To avoid interruptions and ensure a smoother experience, I used the pose estimation model as a way to capture face movement (as opposed to face landmark estimation). The position is then used to determine camera perspective for the scene, and can also control movement or position of individual objects.




For the scene itself, models were mostly designed in Blender and textured in Substance 3D Painter, with the skybox being created with p5.js.

For the fabrication aspect, the window was built by combining 8 small frames to achieve the desired shape, adding hinges, and then putting an opaque sheet for the projection.