Rethinking Emotion Detection

Emotion detection AI models are trained using predefined “ground truth” emotion labels, assuming a single correct interpretation for each piece of text. However, human emotion perception is subjective and context-dependent—people often interpret the same message in multiple ways based on personal experiences, cultural background, and emotional intelligence (EI). This raises a critical question: Do current AI models truly capture the complexity of human emotion, or are they oversimplified?

‍

This research investigates the alignment between human-labeled emotions and AI-generated “ground truth” labels using the GoEmotions dataset. By analyzing user annotations and their emotional intelligence scores, we aim to:

✔️ Assess how often human interpretations align with AI emotion labels.

✔️ Explore whether individuals with higher emotional intelligence (EI) perceive emotional nuances differently.

✔️ Provide insights to improve AI emotion detection models through more context-aware and pluralistic approaches.

‍

To evaluate the gap between AI-generated “ground truth” labels and human emotion perception, we conducted a mixed-methods user study combining qualitative insights and quantitative analysis. Participants labeled emotions in Reddit comments from the GoEmotions dataset, and their selections were compared against AI-generated labels using statistical alignment metrics. We also measured participants’ emotional intelligence (EI) scores using the Schutte Self-Report Emotional Intelligence Test (SSEIT) to analyze correlations between EI and labeling behaviour.

Quantitative analysis revealed only 45.7% alignment between human-selected labels and AI “ground truth,” while qualitative responses highlighted how participants relied on context, punctuation, and personal experience to interpret emotions. Those with higher EI scores showed a greater tendency to assign multiple emotions per text, emphasizing the complexity of human emotional perception—something current AI models struggle to capture. These findings suggest that AI emotion detection should shift toward pluralistic, context-aware models that support multi-label classifications rather than relying on rigid, single-label outputs.