{"id":507822,"date":"2026-07-03T11:32:13","date_gmt":"2026-07-03T11:32:13","guid":{"rendered":"https:\/\/www.newsbeep.com\/nz\/507822\/"},"modified":"2026-07-03T11:32:13","modified_gmt":"2026-07-03T11:32:13","slug":"student-builds-virtual-reality-game-that-reads-emotions","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/nz\/507822\/","title":{"rendered":"Student Builds Virtual Reality Game That Reads Emotions"},"content":{"rendered":"

You\u2019re in a dance club bathed in a neon purple glow, surrounded by ghostly silhouettes that sway to the music. A stranger at the bar greets you with a flirty wave: \u201cHey, you look so cute!\u201d Sparks fly.\u00a0<\/p>\n

It\u2019s all part of a virtual reality game called Rekindle, but the narrative seems real enough that you feel giddy. You break into what\u2019s known as a genuine Duchenne smile \u2014 the kind where your mouth corners lift and your eyes squint a little. The biometric software in the headset you\u2019re wearing scans your face and checks and your muscle movements against the 70 that are involved in typical expressions. You\u2019re happy and your gaming PC knows it. You sink deeper into the story.<\/p>\n

Tapping into the emotional landscape and biometrics in gaming are both well-worn paths in the virtual reality, or VR, space, but Rekindle, a new first-person Interactive Digital Narrative (IDN) that is the brainchild of Interdisciplinary Design & Media PhD student Hector Fan<\/a> and Northeastern professors Mark Sivak<\/a> and Casper Harteveld<\/a>, gives them a fresh spin.<\/p>\n

Video excerpts with scenes from Rekindle. Courtesy of Hector Fan<\/p>\n

Set in a dystopian future where oppressive powers use memory manipulation to erase any trace of sexual identity that they view as being deviant, the fictional protagonist sets out to recover it, one memory at a time. All the while, the player\u2019s empathy for the LGBTQ+ community is being deepened by exploring the world through the eyes of the story\u2019s hero. The narrative may be fictional, but the struggle is all too real, says Fan who hopes the game can foster \u201cpositive social change.\u201d<\/p>\n

At one level, the \u201cmemories\u201d are collectibles that the player gathers to advance through the game. However, unlike your typical Mario coins or Pac-Man dots, they\u2019re meant to be relatable situations that either jog your personal memories or remind you of significant events that are \u201cculturally shared\u201d in the LGBTQ+ space, Sivak explains. This shared archive includes some dark chapters, such as the 2016 Pulse Nightclub shooting, when a gunman killed 49 people and wounded 58 at the Orlando venue\u2019s \u201cLatin Night\u201d event frequented by members of the gay community.\u00a0<\/p>\n

\"AScenes from Rekindle. Ten \u201cmemory fragments\u201d are tagged with emotions that the player is asked to display in order to collect the associated memory. Courtesy of Hector Fan<\/p>\n

The game\u2019s narrative is sprinkled with prompts that call on the user to actively display emotions corresponding to each collectible \u201cmemory fragment\u201d in order to advance through the narrative while building an emotional tie to the protagonist.\u00a0<\/p>\n

Once \u201cyou\u2019re in the helmet, you become that person,\u201d Harteveld explains.<\/p>\n

In the past, emotions have been treated as \u201cinput\u201d rather than \u201coutput,\u201d Fan, Sivak, and Harteveld write in a paper<\/a> that was presented recently at the Augmented World Expo<\/a>, an XR and spatial computing community meet-up that took place this June in Long Beach, Calif. Responding to emotions of other characters on the screen, the player is passively \u201cclicking through the game.\u201d\u00a0<\/p>\n

Rekindle turns the tables in favor of active engagement in a fictional world that echoes our own. At one point, a fight scene brings to mind the Stonewall Riots<\/a> of 1969 or the more recent West Hollywood Mass Brawls<\/a> this March. \u201cThe memories \u2026 are culturally generic,\u201d Fan explains. The emotional input you\u2019re asked to provide further boosts the sense of \u201cdramatic agency\u201d and makes you feel like a participant.\u00a0<\/p>\n

Video showing how the biometric technology is calibrated for each user. Courtesy of Hector Fan<\/p>\n

Graham Page, EVP of Commercial Research and Innovation at iMotions<\/a>, which pools multiple types of biometric data on a single platform, agrees that tracking user emotions adds an important dimension to the VR experience: \u201cOne of the frustrations of using, or interacting with tech, is that it knows what we click or say, but it doesn\u2019t know necessarily how we feel or how we say it.\u201d The ability to create tools sensitive to \u201cnon-verbal insight\u201d paves the way for more meaningful interaction, he suggests. Technology that reads the room.<\/p>\n

To do this, the biometric features of Rekindle\u2019s Samsung Galaxy XR headset track the user\u2019s facial expression and map them onto a set of 6 basic emotions, such as \u201chappy,\u201d \u201csad,\u201d or \u201csurprised,\u201d and 15 compound emotions, which include combos such as \u201chappily surprised,\u201d \u201csadly angry,\u201d or \u201cfearfully disgusted.\u201d\u00a0<\/p>\n

\"\"<\/p>\n

+1<\/p>\n

Rekindle ignites player emotions for next-level VR immersion\"\"\"\"\/<\/p>\n

The headset picks up muscle movements and translates them into \u201caction units\u201d (AUs): for example, raised eyebrows or narrowing eyes are possible AUs. That information is then fed into a system that ranks each of the 21 possible emotions with a 0 to 1 score that indicates their relative strength, with 0 signaling a total absence of a particular emotion in that moment.<\/p>\n

While the technology adds another dimension to the gaming experience, it also opens the door for new levels of complexity and possible tension. Be it the flirtation at a gay bar or encounters with violence related to sexual identity, the situations within the Rekindle narrative will sit differently with a member of the LGBTQ+ community than someone who doesn\u2019t know what the Pride Flag stands for. Those differences will naturally affect the degree to which the emotions displayed by the players will match the author\u2019s intent and fit with the game\u2019s progression.<\/p>\n

But that\u2019s also part of the point. Fan recognizes that there\u2019s value to being put in someone else\u2019s shoes, even if they pinch a little. \u201cI think that the goal is to \u2026 foster empathy or attitude change towards the LGBTQ+ community.\u201d By trying to enact the protagonist\u2019s emotion \u2014 one that Fan had in mind when building the game \u2014 the player becomes \u201calmost like an actor within a simulation,\u201d he says. Taking on the protagonist\u2019s quest \u201ccould help them to conduct perspective taking of other people\u2019s experience that they might not have access to.\u201d<\/p>\n

In the future, Fan plans to lean into the differences in how people view and react to the situations in the game as a way to expand narrative progression. For example, if that friendly greeting at the bar gets a response that\u2019s more \u201cEw!\u201d than \u201cHey there!\u201d from the user, the storyline and the interactions that follow could change accordingly. The result? An experience that feels even more immersive and realistic \u2014 one that also allows the player to grapple with their own biases and their possible effects in a way that feels safe, Fan hopes.\u00a0<\/p>\n

\"StylizedThe headset can detect 70 basic muscle movements involved in emotional expression. Illustration by Renee Zhang<\/p>\n

Ultimately, the technology has potential beyond the gaming world. \u201cBooks could also be interactive,\u201d Harteveld suggests, describing texts with a nonlinear format similar to the \u201cchoose your own adventure\u201d books that allow the reader to change the plot as they progress through the narrative.<\/p>\n

Sivak also mentions that virtual exposure therapy has been helping people grapple with phobias and post traumatic stress disorder, PTSD. Harteveld adds that an emotion tracking component could take it up a notch, for example by reading a patient\u2019s stress levels instead of relying on their words alone. An emotion tracking camera could also boost safety on the road by sounding the alarm if somebody is falling asleep at the wheel, he suggests.<\/p>\n

\tNortheastern Global News, in your inbox.<\/p>\n

Sign up for NGN\u2019s daily newsletter for news, discovery and analysis from around the world.<\/p>\n

\"\"<\/p>\n

Harteveld recently wore eye-tracking glasses on a paragliding adventure that stirred up first-time jitters. While they could only track his eye movement to measure his stress levels, he envisions a version that could one day also respond to emotions akin to a virtual tandem jumper at his side. Tech that could talk you through a difficult task after sensing your anxiety could be especially useful, Harteveld thinks.<\/p>\n

\u00a0\u201cWhen you get into emergency situations, maybe it\u2019s good that a system or a warning is trying to calm you down\u201d and walk you through the steps, he says.<\/p>\n

Katya Poltorak is a science reporter at Northeastern Global News.<\/p>\n","protected":false},"excerpt":{"rendered":"You\u2019re in a dance club bathed in a neon purple glow, surrounded by ghostly silhouettes that sway to…\n","protected":false},"author":2,"featured_media":507823,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[3185,27915,3447,111,139,69,145,360],"class_list":["post-507822","post","type-post","status-publish","format-standard","has-post-thumbnail","category-technology","tag-emotions","tag-empathy","tag-gaming","tag-new-zealand","tag-newzealand","tag-nz","tag-technology","tag-virtual-reality"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/507822","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/comments?post=507822"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/posts\/507822\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media\/507823"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/media?parent=507822"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/categories?post=507822"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/nz\/wp-json\/wp\/v2\/tags?post=507822"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}