Psychological distress in a social media content moderator: A case report

The expansion of social media platforms has given birth to a specific professional category in charge of « sanitizing » social media and keeping it free of “inappropriate” content. These content moderators, as a result of being constantly exposed to potentially traumatic images, may be at risk of developing stress-related symptoms. Here we present the case of a social media content moderator who experienced intrusive images, anxiety symptoms and insomnia.

Literature about this subject is very scarce, while the need is increasing to assess this phenomenon and set up strategies of monitoring, counseling and treating this category of workers.

Key points

The expansion of social media platforms has given birth to a specific professional category: content moderators.

As a result of being constantly exposed to potentially traumatic images, content moderators may be at risk of developing stress-related symptoms.

The need is increasing to assess this phenomenon and set up strategies of monitoring, counseling and treating this category of workers.

During the last few years, social media have become more and more popular. In 2020, an estimated 3.6 billion people are using social media worldwide, and almost 4.41 billion will be using it by 2025 [1]. This expansion of social media platforms has been fostering potential harm for internet users such as the exposition to violence, self-harm, or child pornography [2,3]. It has also given birth to a specific professional category in charge of « sanitizing » social media and keeping it free of any inappropriate content. These content moderators, as a result of being constantly exposed to potentially traumatic images, may be at risk of developing stress-related symptoms. However, and to our knowledge, this specific professional category has been poorly studied. Here we present the case of a social media content moderator who experienced intrusive images, anxiety symptoms and insomnia after two years of work.

A twenty seven years old male patient, married, without children and with no previous psychiatric history, presented to our general outpatient psychiatry clinic with insomnia and anxiety. When the patient sought help, he has been working as a social media content moderator for two years in a dedicated platform, and has been specifically in charge of viewing and suppressing undesirable pornographic content for almost six months. He usually works eight hours a day with sometimes night shifts. The patient explained that he was not allowed to disclose anything about the content he had to moderate, and that he had the obligation to view every visual material entirely before he decides to delete or ignore it. He also regularly had to watch in whole live videos that could contain non suicidal self-harm and/or suicide attempts. The patient, as well as his co-workers, had the possibility to see a coach at workplace and during working time, however the patient was reluctant to seek help from the company coach, not knowing if he was subject to duty of professional secrecy. The patient’s initial psychiatric evaluation found sleep onset insomnia, anxiety, low mood, irritability and intrusive images that he described as the repetition of the content he had to absorb during his working hours. He described himself as « haunted » by violent and/or inappropriate images, especially those with child pornography. The patient did not meet DSM-5 criteria nor for obsessive compulsive disorder neither for post-traumatic stress disorder. He was however diagnosed with adjustment disorder with mixed anxiety and depressed mood. The patient’s initial score on perceived stress scale (PSS) was 31. He was put on escitalopram (10 mg per day) with three weeks leave from work. The first follow up evaluation took place three weeks later. The patient reported that his mood was slightly better, and that he felt less irritable and less anxious. However, intrusive thoughts and images remained as persistent as before the treatment’s initiation. He was still perceiving his stress as high (PSS score up to 29). He refused the possibility to increase his treatment dosage and decided to quit his job. In addition, psychotherapy combining techniques of stress-management and coping skills to deal with stressful events was initiated with the patient. Two months after the start of treatment, and the absence of any exposure to stressors, the patient improved markedly, and resumed a different professional activity, with a significant reduction in intrusive images which no longer cause suffering or impact on his daily life.

Many studies have explored potential psychological harm in internet users exposed to violence, self-harm or pornography contents [4,5]. In contrast, literature about the impact of these contents on internet professionals is very scarce or almost nonexistent.

On social media, companies and people with different cultural, religious and political backgrounds interact. Thus, these platforms have to be kept safe from any violent, inappropriate, or hateful content. This is precisely the responsibility of content moderators who are asked to process hours of potentially inappropriate footages and apply several and codified operations such as discriminating different types of violation, checking profiles of users who shared problematic contents, erasing the contents and then reporting it to hierarchy.

Content moderators are not well known for the public; they are usually from low income countries and can be considered as cheap labor. According to Gillepsie, a possible reason why content moderators are kept hidden by social media platforms is that ordering, moderating and managing internet has to look objective, technical and not impacted by human subjectivity [6].

The patient we present in this case report experienced distress and intrusive thoughts that he perceived as directly linked to his job, with accompanying depressive and anxiety symptoms. This presentation corresponds to DSM-5 Adjustment disorder criteria, with mixed anxiety and depressed mood subtype, especially in the absence of PTSD, acute stress disorder and personality disorder criteria. The diagnosis was confirmed also by the improvement of all symptoms two months after the suppression of the stressor by resigning from this job. We choose to use antidepressants drugs to help patient with symptoms of depression and anxiety. Knowing that psychotherapy is the recommended treatment for adjustment disorder, we proposed techniques of stress-management and coping skills, even if the patient decided to quit his job, to prevent further recurrences and progression to post-traumatic stress disorder which remains to be monitored.

Such cases are rarely reported in medical literature, and there is no study that specifically evaluated the impact of such contents on internet workers. By analogy, internet and social media content moderators can be compared to other professions exposed to inappropriate or violent material with an impact on their mental health. Indeed, journalists that cover potentially war and non-war traumatic events, such as terrorism and natural disasters, are at risk of developing post-traumatic stress disorder or symptoms [7]. In the same vein, Bourke, et al. [8] yielded a survey among 600 people in charge of internet crimes against children (ICAC) including child exploitation and pornography. This study found that 25% of ICAC personnel experienced secondary traumatic stress named also compassion fatigue, which is defined by symptoms that mimic PTSD but that are caused by indirect exposure to traumatic material or events [9].

In 2015, Eiser, [10] described the « postmodern stress disorder » as a variation of post-traumatic stress disorder, where repetitive exposure to digital images of violence may have an impact similar to warfare or civilian assaults. Therefore, the more social media are going to expand, the more numerous will be content moderators exposed to post traumatic psychopathology and the bigger will be the need to assess this phenomenon and set up strategies of monitoring, counseling and treating this category of workers.

Patient gave his verbal consent to publish this article.

1. https://www.statista.com/statistics/278414/number-of-worldwide-social-network-users
2. Lewis SP, Heath NL, St Denis JM, et al. The scope of nonsuicidal self-injury on YouTube. Pediatrics. 2011; 127: e552–557. PubMed: https://pubmed.ncbi.nlm.nih.gov/21339269/
3. Temporini H. Child pornography and the internet. Psychiatr Clin North Am. 2012; 35: 821-835. PubMed: https://pubmed.ncbi.nlm.nih.gov/23107565/
4. Moreno MA, Ton A, Selkie E, Evans Y. Secret Society 123: Understanding the Language of Self-Harm on Instagram. J Adolesc Health. 2016; 58: 78-84. PubMed: https://pubmed.ncbi.nlm.nih.gov/26707231/
5. Fineberg NA, Demetrovics Z, Stein DJ, Ioannidis K, Potenza MN, et al. Manifesto for a European research network into Problematic Usage of the Internet. Eur Neuropsychopharmacol. 2018; 28: 1232-1246. PubMed: https://pubmed.ncbi.nlm.nih.gov/30509450/
6. Gillespie T. Custodians of the Internet. Amsterdam University Press. 2018.
7. Idås T, Backholm K, Korhonen J. Trauma in the newsroom: social support, post-traumatic stress and post-traumatic growth among journalists working with terror. Eur J Psychotraumatol. 2019; 10: 1620085. PubMed: https://pubmed.ncbi.nlm.nih.gov/31231480/
8. Bourke ML, Craun SW. Secondary traumatic stress among Internet Crimes Against Children task force personnel: impact, risk factors, and coping strategies. Sex Abuse. 2014; 26: 586-609. PubMed: https://pubmed.ncbi.nlm.nih.gov/24259539/
9. Adams RE, Boscarino JA, Figley CR. Compassion fatigue and psychological distress among social workers: a validation study. Am J Orthopsychiatry. 2006; 76: 103-108. PubMed: https://pubmed.ncbi.nlm.nih.gov/16569133/
10. Eiser AR. Postmodern Stress Disorder (PMSD): A Possible New Disorder. Am J Med. 2015; 128: 1178-1181. PubMed: https://pubmed.ncbi.nlm.nih.gov/26031889/