The promise of emotionally aware systems comes with risks. Bera warned that many emotionally responsive interfaces are essentially simulations, and users may not always realize what is real and what is not.<\/p>\n
\u201cOne major risk is empathy theater,\u201d he said. \u201cThat is when an AI system mimics emotion convincingly without actually understanding or caring. Just because a machine sounds empathetic, does not mean it is.\u201d<\/p>\n
The gap between performance and intention can lead to harm, especially when people place trust in systems that are not designed to offer care. Bera is particularly concerned about affective systems being used in hiring, feedback or mental health contexts without clear guardrails. \u201cWe must ask: are we amplifying human understanding or automating misunderstanding?\u201d he said.<\/p>\n
Signal misinterpretation is another danger, Bera said\u2014especially when emotional expressions can vary so widely across cultures and individuals. \u201cThere is the concern of misread signals,\u201d Bera said. \u201cEspecially across cultural or neurodiverse expressions of emotion.\u201d<\/p>\n
Haber raised a separate concern: what happens when AI systems are too agreeable? \u201cIt is important to have a conversation partner that can engage in sometimes seemingly adversarial ways,\u201d he said. \u201cIf I am talking with a chatbot about how I treated someone horribly and am trying to justify my actions, it can be really harmful if it tells me I was totally justified in doing that.\u201d<\/p>\n
Many large language models (LLMs) are trained to be affirming, not challenging, Haber said. The result is what Haber described as \u201cemotional echo chambers,\u201d where \u00a0harmful ideas get reinforced through AI feedback loops.<\/p>\n
\u201cWe can become uncritical, unreflective and more isolated,\u201d he said. \u201cThere has been a lot of talk about how large language model-based chatbots are sycophantic. That is just a small part of a much larger and more complex issue.\u201d<\/p>\n
Seif El Nasr added that emotionally responsive systems can create unintended psychological dependencies. \u201cThese include psychological issues such as overreliance, loneliness and anxiety,\u201d she said. \u201cThey can also lead to isolation from society.\u201d<\/p>\n