Using the repository: Emotion regulation items
Anu Hiekkaranta, November 2021
Many emotion regulation researchers are now expanding their investigation outside the laboratory to the study of naturally occurring emotion regulation. Ideally suited for such an endeavor of course, is the Experience Sampling Method (ESM). An ESM novice aiming to study emotion regulation in the wild may ask: Where do I start? We suggest you start by browsing the repository! You can explore existing items in the repository by using the search tools at https://www.esmitemrepository.com/. In this post, I will walk you through the search results for two emotion regulation strategies, rumination and acceptance, as well as how you might best make use of what you find in the repository.
Example case 1: Rumination
Let’s say you would like to study rumination in daily life. Rumination items are one of the most often included items in new contributions to the repository. One way to use the repository to find relevant items with a concept in mind is to look up items with a certain description. At the time of writing this blog post searching the repository for items with “rumination” in the description results in 14 hits. You will immediately be able to view the results and some details about them. For more information about each item, you can download the search results. Browsing the rumination items will give you an idea of how different researchers have adapted items from traditional self-report questionnaires to ESM items (e.g., I am thinking about my feelings, I am thinking about my problems (Moberly & Watkins, 2008), and I am ruminating (Snippe et al., 2019)). Moreover, your search results will also illustrate how others have developed ways to investigate ruminating by anchoring to different time frames, specific events, and specific thoughts. For example, some researchers first ask participants what they are thinking about at the moment they got the notification, followed by questions about the characteristics of their current thoughts, such as rating their agreement with statements like: This is a repetitive thought (ESM Item Repository, Kirtley et al., 2019, item 152). Alternatively, participants in a study may be asked to identify a specific event, such as the most important event in the past two hours. Then, participants can be asked questions related the event, such as I have thought about it a lot (most important event in the past 2 hours), (Kirtley et al., 2021). You may also note that researchers have anchored ruminating to different types of events: positive, negative, and important.
Per each search results with the description “rumination”, you can also see, among other details, what population the item has been used in, the response scale, and the number of times the item was presented per day. In addition, browsing for emotion regulation items in the repository reveals differences in how many items different researchers have used to study emotion regulation constructs within their experience sampling questionnaires. You will see for instance, that when studying rumination, some have opted for one item, while others use several.
Now, let’s consider what similarities we find when we search for another emotion regulation strategy in the repository, and take note some of the practical challenges in choosing the items for your study.
Example case 2: Acceptance
As with rumination, browsing the repository for acceptance-related items demonstrates the diversity in operationalization of emotion regulation constructs. For example, the description “acceptance” can be found in the repository for all of the following items:
-I let my thoughts be there without reacting to them (ESM Item Repository, Kirtley et al., 2019: item 307)
-Think about the most negative event today. I let it happen (Kirtley et al., 2021)
-Today, I could let go of my negative feelings without acting upon them (ESM Item Repository, Kirtley et al., 2019, item 341)
-(Most negative event) I just accepted my emotions (Kirtley et al., 2021)
Careful examination of these items reveals subtle but important differences. Acceptance, like rumination, can be anchored to e.g., thoughts, events, emotions, or emotions related to specific events. When deciding which items to use for any emotion regulation construct, it will be useful to consider not only one’s theoretical understanding of the construct of acceptance, but also practical concerns. Which of the items are easiest to respond to and how many times per day, given the population in which you want to study acceptance, and the timing of your study? For example, reflecting on one’s thoughts and reactivity to them several times a day requires a level of insight you may not expect from children or teenagers (and indeed, this poses a challenge to adults as well!). To tackle the issue, anchoring acceptance to events may help participants grasp the concept of acceptance better in the moment. However, persons living in quarantine or locked down cities as many currently are, may have few events to report. These concerns raise new questions. What will you consider an event? Do participants need to be briefed with specific examples of events that may take place for them, in their current circumstances? You may also conclude that existing items (gathered from the repository or elsewhere) do not sufficiently capture the type of emotion regulation you are interested in. In that case, we hope you return to the drawing board wiser and even a little more inspired!
Discover and develop
The field of emotion regulation in daily life is still in its infancy. Indeed, many fields in psychology are still new to experience sampling. Therefore, we invite you to use the repository to discover the existing items relevant to your research, be it emotion regulation or something else. We hope that what you learn may save you time and help you make decisions about which items to use. Moreover, we hope to inspire you to think about how to validate existing items and how to adapt and develop items to your research needs. As a final point, once you have collected some data, we warmly encourage you to make your own submission to the ESM Item Repository!
References:
Kirtley, O. J., Hiekkaranta, A. P., Kunkels, Y. K., Verhoeven, D., Van Nierop, M., & Myin-Germeys, I. (2019). The Experience Sampling Method (ESM) Item Repository. https://doi.org/10.17605/OSF.IO/KG376
Kirtley, O. J., Achterhof, R., Hagemann, N., Hermans, K. S. F. M., Hiekkaranta, A. P., Lecei, A., … Myin-Germeys, I. (2021, April 2). Initial cohort characteristics and protocol for SIGMA: An accelerated longitudinal study of environmental factors, inter- and intrapersonal processes, and mental health in adolescence. https://doi.org/10.31234/osf.io/jp2fk
Moberly, N. J., & Watkins, E, R. (2008). Ruminative self-focus and negative affect: An experience sampling study. Journal of Abnormal Psychology, 117(2), 314.
Snippe, E., Helmich, M., Kunkels, Y. K., Riese, H., Smit, A., & Wichers, M. (2019). Esm Item Documentation. Retrieved from https://osf.io/a8572/.
The struggle of selecting good experience sampling items
Gudrun Eisele, November 2020
I clearly remember the feeling when setting up my first experience sampling method (ESM) study. It felt like only few days had passed since the start of my PhD and the first meeting with my supervisors who said to “go ahead and have it ready by May”. So there I was, going ahead, kind of. The truth is, I was rather overwhelmed by the seemingly infinite number of decisions to make.
In this state of trying to figure out what I was doing and with May coming closer, I remember finding the item selection a particularly daunting task. Just quickly adding some items, “it can’t be that hard”, I kept telling myself. I spent most of my days scanning through ESM articles, which often did not report the exact items. When they did, these were usually not presented in the original language, and most of the time no information on item quality was provided whatsoever. How should we choose?
The broader problem
Almost 3 years later, after running this study (https://psyarxiv.com/zf4nm/), certainly making numerous mistakes, and seeing others set up studies, I know now that this is not an isolated experience. Choosing good ESM items is just very hard. There are many reasons for this: Frequently, exact ESM items are still not reported in journal articles and their quality remains extremely difficult to judge. Guidance on how to develop them is practically nonexistent within the ESM literature (there are exceptions, for example: Horstman & Ziegler, 2020; Palmier-Claus et al., 2010) and researchers selecting the items have often not received sufficient training in good measurement practices. Some labs may have elaborate procedures for the development or selection of ESM items, ranging from meetings, to pilot studies, to focus groups. However, the results of this work are rarely shared publicly, and all of this valuable information remains hidden away behind closed doors.
The ESM item repository
The ESM item repository is a first step in solving this issue. It provides a platform to collect and find existing ESM items. However, the repository does not currently help users choose from the large pool of items that are in it. You might find yourself scrolling through the increasingly long list of items (we now reached 756 item submissions!) and still have no idea how to select items for your study.
A Delphi study on ESM item quality criteria
This is why we decided to set up a Delphi study on item quality with ESM experts from different countries and fields of study. In our preregistered Delphi study, over the course of three rounds, we are hoping to develop an item quality assessment tool, specific to ESM, that can be used to evaluate existing items and develop new ones.
At the moment, 50 ESM experts who signed up or were nominated to participate are completing the first round of the study. During this round, we ask experts to name quality criteria that they use to evaluate the quality of ESM items. In the second round, experts will be asked to evaluate the resulting criteria. Finally, quality criteria that are agreed upon in the third round of the Delphi study will be used to develop the item quality assessment tool, which will be made publicly available.
We will keep you posted about the progress of the study here and on Twitter!
Horstmann, K. T., & Ziegler, M. (2020). Assessing Personality States: What to Consider when Constructing Personality State Measures. European Journal of Personality. https://doi.org/10.1002/per.2266
Palmier‐Claus, J. E., Myin‐Germeys, I., Barkus, E., Bentley, L., Udachina, A., Delespaul, P. A. E. G., ... & Dunn, G. (2011). Experience sampling research in individuals with mental illness: reflections and guidance. Acta Psychiatrica Scandinavica, 123(1), 12-20. https://doi.org/10.1111/j.1600-0447.2010.01596.x
The Experience Sampling Method Item Repository: What are we doing, why are we doing it and where are we going next?
Olivia Kirtley, October 2020
The Experience Sampling Method Item Repository is one year old! To celebrate our first birthday, we have launched a new blog where the rest of the repository team (Anu Hiekkaranta, Yoram Kunkels, Gudrun Eisele, Davinia Verhoeven, Martine van Nierop and Inez Myin-Germeys) and I will keep you up-to-date on the latest developments in the repository project, including highlighting the collection of items, and discussing ESM research more broadly. So let me start us off with our very first blog post, to tell you a bit about what we are doing, why we’re doing it and what lies ahead for the repository.
The Experience Sampling Method (ESM; Csikszentmihalyi & Larson, 1987), sometimes referred to as Ecological Momentary Assessment (EMA; Stone & Shiffman, 1994) brings myriad possibilities for gaining rich, valuable data about the context in which individuals’ thoughts, feelings and behaviours arise: their normal day-to-day lives (Myin-Germeys et al., 2018). Capturing these daily-life experiences, outside of the lab and the clinic brings us many new opportunities for increasing our understanding of phenomena, including individuals’ social experiences, emotion regulation, psychopathology symptoms and self-harm thoughts and behaviours. Over the years, ESM research has grown a lot and is a field full of exciting, cutting-edge statistical and technological developments. Perhaps surprisingly, however, some key questions around how we conduct ESM research, including about the measures that we use, remain unanswered.
We do not have a suite of validated ESM questionnaires, which have undergone a rigorous psychometric validation process, in the same way that we have validated measures for assessing depression or anxiety*. Consequently, many ESM items are constructed by researchers “on the fly”, and passed along from one researcher to another. Sometimes the same ESM items are used to assess different or related constructs and there is currently very little psychometric validation of ESM items (Horstmann & Ziegler, 2020; Wright & Zimmermann, 2019), with only around 30% of papers in a recent review reporting the psychometric properties of the ESM items used (Trull & Ebner-Priemer, 2020). Yet, measurement matters (Fried & Flake, 2018) and lack of transparency around measures, as well as the absence of solid psychometric validation, can threaten the transparency and methodological reproducibility of ESM research more broadly. Further, it may also limit the conclusions we can draw from our studies. Whilst conversations around the transparency, reproducibility and replicability of research have mainly focused on cognitive and social psychology, this does not mean that these issues do not apply to ESM research (Kirtley et al., 2020). So how do we go about addressing these issues?
A good first step is to get an overview of the items currently “out in the wild” of the ESM research landscape. That’s where we come in. The Experience Sampling Method (ESM) Item Repository (Kirtley et al., 2020) is an ongoing three-phase project in which we are building an open (publicly available) bank of ESM items, with the help of contributions from ESM researchers from around the world. Phase I of the project involves populating the repository and collecting as many items as possible. This first version of the repository is available to discover and search at www.esmitemrepository.com. At this stage, the repository has only undergone minor quality screening for formatting issues, spelling and language errors etc, so is very much still a “warts and all” list of ESM items. Items cover topics including mood, emotion regulation, psychotic experiences, quantity and quality of online and offline social experiences and self-harm thoughts and behaviours. This allows us and other researchers to gain insights regarding the range and content of ESM items being used, and to begin to identify patterns among items. Phase II of the project is just about to be launched and involves a quality assessment of the items in the repository. To accomplish this task, we need a quality assessment tool for ESM items, but because research is rather like Hal changing a lightbulb (https://youtu.be/AbSehcT19u0), we first need to develop a quality assessment tool. Gudrun Eisele, PhD student in the Center for Contextual Psychiatry (CCP), is leading this work with fellow CCP PhD student Anu Hiekkaranta and they’ll be telling you more about that in the next blog post. In Phase III of the project, we will begin the psychometric validation of items within the repository, but we don’t just want to keep all the psychometric fun to ourselves; a key goal of the repository is also to facilitate other scientists to conduct ESM research and to psychometrically validate available items. We’re busy with plans for this phase of the project and will share more details soon.
Our main goals in establishing the repository are to increase the discoverability of ESM items, facilitate ESM research and to improve the transparency and quality of ESM research. Given that ESM is increasingly used in clinical psychology and psychiatry research, we also hope to make a contribution to transparency and reproducibility within those areas, which have often not been part of conversations around open science practices. For more on that topic, please read the brilliant work of Jennifer Tackett and her colleagues (Tackett, Brandes, & Reardon, 2019; Tackett et al., 2019). The rest of the fantastic ESM Item Repository team and I are thrilled that our efforts to increase the openness of ESM research have been recognised by two awards this year: a commendation from the Society for the Improvement of Psychological Science (https://twitter.com/improvingpsych/status/1247246158432526336?s=20) and an Open Research Award from the University of Groningen (https://twitter.com/Bibliothecaris/status/1311299779591385089?s=20). We’re not done yet though! There are lots more exciting things to come. A vital and ongoing task is continuing to build the collection of items. After all, the success of the ESM Item Repository depends on researchers contributing their items, so please consider sending us your items to include in the repository! To find out more information about the project and how to contribute items, check out the contributors pack on our OSF page (https://osf.io/kg376/).
Notes:
* Unfortunately, in some cases, the “rigorous psychometric validation process” for traditional self-report questionnaires has also been treated as more of a serving suggestion. I could write a whole other blog post about this (and maybe I will!), but let’s stick to ESM for now. Interested readers should definitely check out all the wonderful things that Eiko Fried and Jessica Flake have written about these issues (see references below).
References:
Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the Experience Sampling Method. Journal of Nervous and Mental Disease, 175(9), 526–536.
Flake, J. K., & Fried, E. I. (In press). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science. Preprint: https://doi.org/10.31234/osf.io/hs7wm
Fried & Flake (2018). Measurement Matters. The Observer (Association for Psychological Science).
Horstmann, K. T., and Ziegler, M. (2020) Assessing Personality States: What to Consider when Constructing Personality State Measures. European Journal of Personality.
Kirtley, O. J., Hiekkaranta, A. P., Kunkels, Y. K., Eisele, G., Verhoeven, D., Van Nierop, M., & Myin-Germeys, I. (2020, October 1). The Experience Sampling Method (ESM) Item Repository. https://doi.org/10.17605/OSF.IO/KG376
Kirtley, O. J., Lafit, G., Achterhof, R., Hiekkaranta, A. P., & Myin-Germeys, I. (In press). Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM). Advances in Methods and Practices in Psychological Science. Preprint: https://doi.org/10.31234/osf.io/seyq7
Myin‐Germeys, I., Kasanova, Z., Vaessen, T., Vachon, H., Kirtley, O., Viechtbauer, W., & Reininghaus, U. (2018). Experience sampling methodology in mental health research: new insights and technical developments. World Psychiatry, 17(2), 123-132.
Stone, A.A., & Shiffman, S. (1994). Ecological momentary assessment (EMA) in behavioural medicine. Annals of Behavioral Medicine, 16, 199-202.
Tackett, J. L., Brandes, C. M., & Reardon, K. W. (2019). Leveraging the Open Science Framework in clinical psychological assessment research. Psychological Assessment. doi:10.1037/pas0000583
Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., . . . Shrout, P. E. (2017). It's Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science. Perspectives in Psychological Science, 12(5), 742-756. doi:10.1177/1745691617690042
Trull, T. J., & Ebner-Priemer, U. W. (2020). Ambulatory assessment in psychopathology research: A review of recommended reporting guidelines and current practices. Journal of Abnormal Psychology, 129(1), 56–63.
Wright, A. G. C., & Zimmermann, J. (2019). Applied ambulatory assessment: Integrating idiographic and nomothetic principles of measurement. Psychological Assessment, 31(12), 1467–1480. https://doi.org/10.1037/pas0000685
About the author: Olivia Kirtley is an FWO Senior Postdoctoral Research Fellow within the Center for Contextual Psychiatry at KU Leuven in Belgium, where she also leads “SIGMA”, a large-scale longitudinal study of adolescent mental health and development using experience sampling methods (ESM). Her current research uses ESM to investigate dynamic processes involved in ideation-to-action transitions in adolescents who self-harm. Olivia leads several projects aimed at increasing transparency and reproducibility in the ESM field, including designing a pre-registration template for ESM research and leading the ESM Item Repository.
