Factor analysis: Overrated, Misused, But Still Useful.

By Benjamin Kunc

March 2025

               

                    This blog post explores the role of Factor Analysis in psychological measurement. If you don’t have time to read the whole post, here is a handy summary:

 

• Building an argument about a scale validity solely on factor analysis can be misleading.
• A thorough conceptual work, followed by an investigation of response processes in the target population, is at least as important for scale validity as psychometric analyses. Probably more.
• Exploratory factor analysis is more useful for detecting empirical phenomena than as a validation method.
• Many things can go wrong with measurement in ESM/EMA designs, and you might want to double-check if the measurement process worked well. In such cases, confirmatory factor analysis is always your friend.
• At the same time, confirmatory factor analysis is more informative when you encounter null results than if you find results that support your hypotheses.
• You might dislike the previous statement. If so, you should also consider rejecting the idea of criterion validity.

 

 

                     Psychologists love to measure, and quantitative empirical research has played a crucial role in psychological science since its origins. The sheer volume of quantitative papers relying on measuring psychological constructs may lead to the impression that measurement is simple. After all, most of us passed some psychometric courses, and we all still vaguely remember learning about validity, reliability, factor analyses, and Cronbach’s alpha. So what’s the fuss about?

     While psychometricians and methodologists tend to strongly recommend assessing the validity and reliability of used scales (e.g., in Standards for Educational and Psychological Testing, American Educational Research Association, 2014), we rarely see this in research. The absence of evidence of scale validity even led some researchers to describe the current state of psychology as a Measurement crisis (Flake & Fried, 2019). It should not be surprising that the same tendency has also been observed in subfields using the Experience Sampling Methodology (ESM; Janssens et al., 2023; Trull & Ebner-Priemer, 2020).

     When it comes to validation, several potential sources of evidence can be used to test the assumptions^[1] of a successful measurement process. Standards mention the test content, response processes of respondents, internal structure of the scores, and relation of the test scores to other variables.

     The suggested sources of evidence can be used to test three methodological assumptions. First, the content of the developed items must align with all the relevant aspects of the studied construct. Second, respondents must understand the questions, and use the response options appropriately. Finally, if the first two assumptions hold, we may assume that responses to the items reflect the value of the investigated construct.

     Notably, psychologists seem to almost exclusively test the assumption about the structure of the response scores and relations of the scales to other variables, while avoiding investigating the scales’ content and response processes in participants (Zumbo et al., 2014).

As for testing the internal structure of response scores, exploratory (EFA) and confirmatory (CFA) factor analyses are the most popular methods (Zumbo et al., 2014). While factor analyses can be useful in exploring the relationship between items (via EFA), or testing specific assumptions about these relations (via CFA), they come with limitations.

     For example, EFA can indicate existing underlying constructs even for questionnaires based on meaningless items (Maul, 2017). Moreover, the current practices of CFA may lead to incorrect statistical estimation of the internal structure of the response scores (McNeish & Wolf, 2021; Savalei & Huang, 2025), and incorrect theoretical conclusions even in cases of accurate estimations (Borgstede & Eggert, 2023).

How to make the most of factor analyses?

                       While building the argument about scale validity solely on factor analyses is risky, these analyses may still play an important role in validation, under certain conditions. In an optimal scenario of scale development, the investigated construct should first be precisely defined. This would be followed by generating items whose content would be further investigated to ensure they cover all relevant aspects of the construct. A potentially useful way to complete the first two steps is to gather the opinions of relevant experts and find consensus on what the construct is, and how it can be measured. A Delphi study design can be particularly effective for this aim (see Herdman et al., 2002 for an example).

      Next, the scale should be presented to respondents from the target population to test whether they understand the items and use the response options as the experts expected. This can be tested purely qualitatively within cognitive interviews or by employing the Response Process Evaluation method (Wolf et al., 2023). The RPE incorporates qualitative assessment with quantitative metrics by counting the number of participants whose response behavior aligns with the researchers’ intentions. This approach allows for detecting items that are not understood by most participants, and iteratively adjusting them in several rounds of evaluation.

      Finally, we are getting to the point where factor analyses are most useful. Once solid evidence about the scale content and participants’ response processes exists, it is time to employ the EFA. At this point, evidence provided by the EFA would be at a much lower risk of finding conceptually meaningless factors. This can be quite useful, not necessarily for validating the scale, but rather as an empirical method that provides further information about the studied construct. In this context, EFA could become an important part of the theory construction cycle by allowing researchers to identify new empirical phenomena, which is necessary for developing sound proto-theories (see Borsboom et al., 2021; Fried, 2020 for a discussion).

Figure 1: When to use EFA

Alternatively, researchers might decide they already have enough information to specify hypotheses about the relationship between items’ scores and underlying factors after the previous steps of scale development. In that case, CFA can be used to test the final remaining assumption – do the relationships between scores and factors reflect the hypothesized structure of the construct?

Figure 2: When to use CFA

 

Do we need CFA?

                         But what exactly is the benefit of running a CFA at this point? One might argue that, based on the previous validation steps, we already know that the scale works well. The content of items is aligned with the investigated construct, and the response processes of participants reflect the value of the construct.

       Indeed, conducting CFA would not add much information about the scale itself. Instead, it could serve as a check of the whole measurement process. This is particularly useful when the measurement process may fail even if items work well, like in the ESM/EMA designs. For example, participants may respond more carelessly towards the end of the study, or the sampling design may systematically fail to capture an important subset of occasions (e.g. measuring momentary physical activity in professional swimmers). In cases of substantial disruptions to the measurement process, CFA would indicate that the measurement model did not accurately reflect the hypothesized structure.

         But if a measurement process may fail even when we use already tested items, does it mean we should conduct CFA in all papers that employ questionnaires? Well, maybe.

Imagine you are conducting a study built on a solid theory, in which you hypothesize that a certain phenomenon will be predicted by the value of some measured construct. You already know that the validity of the used scales is supported by evidence derived from their content and target population response processes.

         Now let’s say you find results supporting your hypothesis. Since a correct detection of an existing effect is conditioned upon the assumption that the measurement process worked well, the measurement process could have failed only if the results were falsely positive. In other words, it should be rather unlikely that the measurement process failed, and the CFA conducted would be informative only under rare circumstances.

         On the other hand, a CFA could provide highly informative evidence if your hypothesized effect was not detected. In such a case, it could either rule out that the negative results occurred due to the failure of the measurement process (thus making null results more informative) or show that the assumptions about the particular measurement process were not met (thus explicitly showing that the null results are not so informative).

If this is wrong, so are the Standards

                           Should we then test our methodological assumptions only if we encounter null results? That does not sound very reasonable either. However, we need to keep in mind that current recommendations propose using the relation of the test scores to some other variable as an acceptable piece of evidence about test validity. Personally, I fail to see how this is conceptually different from encountering positive results in hypothesis testing. Therefore, we should either agree that finding positive results supports the validity of the used scale, or we should reject the concept of criterion validity, and stop using it.

         Whether we need to do CFA when we find positive results ultimately depends on how much we can trust the positive findings in our work. Since most psychological research is still based on testing null hypotheses, our trust should reflect the expected likelihood of finding false positive results. At the moment, I am afraid that false positives are not as rare as we would like, and the need to thoroughly check all methodological assumptions will remain with us for some time. On a more positive note, I believe that psychological science can improve, and that one day we will not need to be so intellectually insecure about every assumption we dare to make. Until then, keep validating as much as you can.

 

References

- American Educational Research Association, American Psychological Association & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

- Borgstede, M., & Eggert, F. (2023). Squaring the circle: From latent variables to theory-based measurement. Theory & Psychology, 33(1), 118-137. https://doi.org/10.1177/09593543221127985  

- Borsboom, D., Van Der Maas, H. L., Dalege, J., Kievit, R. A., & Haig, B. D. (2021). Theory construction methodology: A practical framework for building theories in psychology. Perspectives on Psychological Science, 16(4), 756-766. https://doi.org/10.1177/174569162096964       

- Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in methods and practices in psychological science, 3(4), 456-465. https://doi.org/10.1177/251524592095239           

- Fried, E. I. (2020). Theories and models: What they are, what they are for, and what they are about. Psychological Inquiry, 31(4), 336-344. https://doi.org/10.1080/1047840X.2020.1854011          

- Herdman, M., Rajmil, L., Ravens‐Sieberer, U., Bullinger, M., Power, M., Alonso, J., & European Kidscreen and Disabkids groups. (2002). Expert consensus in the development of a European health‐related quality of life measure for children and adolescents: a Delphi study. Acta Paediatrica, 91(12), 1385-1390. https://doi.org/10.1111/j.1651-2227.2002.tb02838.x 

- Janssens, J., Kiekens, G., Jaeken, M., & Kirtley, O. J. (2023). A systematic review of interpersonal processes and their measurement within experience sampling studies of self-injurious thoughts and behaviors. PsyArXiv. https://doi.org/10.31234/osf.io/fmuc5  

- Maul, A. (2017). Rethinking traditional methods of survey validation. Measurement: Interdisciplinary research and perspectives, 15(2), 51-69. https://doi.org/10.1080/15366367.2017.1348108    

- McNeish, D., & Wolf, M. G. (2023). Dynamic fit index cutoffs for confirmatory factor analysis models. Psychological Methods, 28(1), 61–88. https://doi.org/10.1037/met0000425    

Savalei, V., & Huang, M. (2025). Fit indices are insensitive to multiple minor violations of perfect simple structure in confirmatory factor analysis. Psychological Methods. Advance online publication. https://doi.org/10.1037/met0000718   

- Trull, T. J., & Ebner-Priemer, U. W. (2020). Ambulatory assessment in psychopathology research: A review of recommended reporting guidelines and current practices. Journal of Abnormal Psychology, 129(1), 56–63. https://doi.org/10.1037/abn0000473 

- Wolf, M. G., Ihm, E., Maul, A., & Taves, A. (2023). The response process evaluation method. Preprint at PsyArXiv. https://doi.org/10.31234/osf.io/rbd2x  

- Zumbo, B. D., & Chan, E. K. (2014). Setting the stage for validity and validation in social, behavioral, and health sciences: Trends in validation practices. In Validity and validation in social, behavioral, and health sciences (pp. 3-8). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-07794-9

^[1] For brevity, I will ignore underlying philosophical assumptions, which are usually not considered during scale validation (e.g., Does the construct exist? Is its existence relevant for the theory? Can it be measured? Do researchers understand it well enough to be able to develop measurement tools? etc.)

 

 

 

The Everyday Measures of Temporal Emotions (EMOTE) database: An open platform to request and deposit ESM data

By Imogen Smith, Pete Koval, Elise Kalokerinos, & Daniel Russo-Batterham

Guest blog, February 2025

               

                    As you’re here, visiting the ESM Item Repository, you probably don’t need much convincing about the benefits of experience sampling methods (ESM). You’re almost certainly convinced that repeatedly assessing people’s feelings, thoughts, and behaviours as they go about their usual daily activities offers valuable insights into human psychology. However, you might also be aware that ESM data are time-intensive and expensive to collect. This might constrain you to obtain a smaller sample size than ideally required to address your research questions. Additionally, because of the burden ESM places on participants (i.e., requiring them to answer many, frequent surveys), you might be facing difficult decisions about which items to include in your next ESM study. What if we told you about a place where ESM data grows on trees; a land of intensive longitudinal milk and honey; where you could search for existing ESM datasets that include your variables of interest and access them for free? We have developed such a place: the Everyday Measures of Temporal Emotions, or EMOTE, database (https://emotedatabase.com/).  

 

EMOTE is our attempt to address some of the limitations of ESM research by facilitating the sharing of ESM data among researchers. The EMOTE database (emotedatabase.com) is a massive, open-access, searchable, and continuously growing repository of ESM data on daily emotional processes (although there are also many other psychological constructs included). Instead of ESM data gathering dust, EMOTE allows these data to be used again and again, contributing to cumulative scientific knowledge and saving time and money in the process

 

The EMOTE database has two primary use cases. First, researchers can search for and request ESM data currently available in the database for their own secondary data-analysis projects. EMOTE makes it easy to explore the thousands of variables stored on EMOTE by allowing searches by dataset, construct names, variable names, and other variable characteristics. Once a researcher has identified their required variables, they can easily add variables to a ‘shopping cart’ and complete an access request form, which is screened by the EMOTE team. After a data access request is approved, the requested variables are extracted into a custom dataset and are made available to the researcher. This allows researchers to efficiently obtain an ESM dataset tailored perfectly to their research needs. No need to waste valuable time and money designing and collecting a new ESM study — just shop for one (free-of-charge) on EMOTE! In addition, datasets housed in EMOTE are easily harmonized, enabling large-scale reanalysis across multiple datasets. 

 

Second, researchers can contribute to Open Science by sharing their own valuable ESM data with other scientists by uploading it to EMOTE. We endeavour to make the data deposit process relatively painless by allowing researchers to upload their raw data, and by providing a guided process to obtain background information about the study design and details of the variables in the data. This ensures that important information about the data is transmitted to data users. As of the date of this blog post, this feature is still in beta-testing, but if you’d like to share your data, you can leave your details on the ‘Share data’ page of our website, and we’ll be in touch. 

 

So far, EMOTE contains data from over 4,700 participants, sampled at over 190,000 measurement occasions in 36 individual studies. All datasets on EMOTE include some measure of daily emotion, but these datasets also contain a wealth of different constructs, with over 3,600 unique variables. These studies span a wide range of research focuses, with data collected from community adults, undergraduate students, clinical patients, and online samples. The studies hosted on EMOTE range from relatively low-frequency daily diary designs (one assessment per day), all the way up to high-frequency ESM designs (with up to 50 surveys per day), collected via a range of modes from paper-and-pencil via personal digital assistants (PDAs) through to smartphone apps. 

 

To help EMOTE grow, we would love your input! Currently, the EMOTE database accepts any form of intensive longitudinal data with human participants (e.g. experience sampling methods, ecological momentary assessment, diary methods, ambulatory assessment). We are interested in any dataset with at least one measurement occasion per day. If you're an experience sampling researcher with data you would like to contribute, or you would like to make a data request, we encourage you to visit https://emotedatabase.com/ .

 

Our platform is an initiative of the Functions of Emotion in Everyday Life (FEEL) Lab and the Melbourne Data Analytics Platform (MDAP), both at the University of Melbourne, as well as the Research Group of Quantitative Psychology and Individual Differences at KU Leuven.

 

8 lessons learnt from working on an Academic Start-up:
The ESM Item Repository

Yoram Kunkels, January 2025

                 Start-ups have been storming the limelight in the last few years. Mostly are active in the broader technology sector, such as Whatsapp, Netflix, and Airbnb. Though most of these start-ups are often mostly commercial in nature, one could easily forget that Academia is also more than capable to spawn groundbreaking startup projects. However, where more business-oriented start-ups might attract large investments and can thus quickly scale up and professionalise, academic startups are often more limited in that sense. So how to start, grow, and maintain an academic startup? Here I will give some pointers from my experience from the ESM Item Repository.

 

      1. Identify a need / goal

With the ESM Item Repository we tried to solve a problem that we identified as a broader need within our academic field, namely; addressing the need for a central database containing commonly used ESM items. Having such a clearly defined need to solve will help constrain your efforts and planning. Beware of adding or expanding to much in this stage; keep your goals simple, concise, and in touch with the practicalities of the issue you are trying to solve.

 

      2. Get a capable team together

A capable team is one of the most important ingredients to make your academic startup a success. Though academic startups often do not immediately have access to large sums of capital to attract the best and brightest, they do have an important advantage. That is, many of the best and brightest start their career in academia, through study or work, so your ideal starting team will often be right at your doorstep. Moreover, a team is a long-term investment, so skills that are not yet completely filled in the team, can be trained or found in future team members.

 

      3. Start thinking and planning

With your starting team complete, let's start thinking, talking, and brainstorming together. For example, an initial hack-a-thon session was the successful start of the ESM Item Repository. It helps make plans more concrete, and allow team members to give feedback on the plans and help develop them. It can also help to get your team structure clear early. Who is leading the project? How to match each team members best skills to the work? However, as the project is in its early stage it might be beneficial not to plan to much, as a lot can still change. That is, you might want to start focussing now on the next point.

 

      4. Get a prototype or minimum viable product (MVP) soon

While it might be tempting to try and completely plan out your product and it's development beforehand, it might be more efficient to start working on a prototype or minimum viable product (MVP) soon after the initial brainstorm session. This will allow you to build practical knowledge about the product you are creating and will highlight potential challenges early on. For example, with the ESM Item Repository we had a prototype running relatively quickly after our first brainstorm session. This first prototype was flawed and thus we decided on another option. However, without this first prototype we would not have this valuable practical experience, and allows others to give feedback. So if it is possible to fail cheaply and frequently, as with our software, it might actually be wise to do so. Also because it trains skills and experience in your team, which brings us to the next point.

 

      5. Build, manage, and maintain the team

While you might already be happy with your initial team, it is important to recognize that it is subject to change, as for example, people can start other jobs. Also, as an academic startup, many team members might only be available part-time, requiring extra management. I think the natural academic team process lends itself well to an academic startup, as it is naturally focussed on developing people's skills.

 

      6. Create awareness / sell your product

Now hopefully you have a working team and product, thus making it time to start pushing your product into the market. If for example, your product is an online platform, it might be as easy a putting your website online. However, even then making the right people find your website can still be difficult. A whole business field has developed around this, called Search Engine Optimalization (SEO).

Also be ready to engage your audience personally, and invite them to participate. As the ESM Item Repository depends on researcher collaboration (in contributing their ESM items), this already created positive engagement which motivates people to use and revisit our  site.

 

      7. Ask / listen to user feedback

Just as your team grows, your product might also have to grow and develop to keep up with customer requirements. To do this properly, user feedback is necessary. Think about what kind of feedback you want, and how to acquire it. While a short questionnaire or text field on your website can yield some basic information, a more thorough look at user experience might require some more in-depth focus groups. Adress any negative feedback you receive, and ask users for confirmation on new features.

 

      8. Professionalise your workflow and infrastructure

Early on during the academic startup lifecycle you might be dependent on free-to-use alternative to professional options. For example, an online app might be hosted for very low costs or even for free. However, these free plans are often throttled on important features. Look what would offer the most benefit for the lowest cost in the beginning. For example, for the ESM Item Repository we have free app hosting via shinyapps.io. The downside here is that it comes with an unsightly and very forgettable URL, e.g., “myname.shinyapps.io/My_New_App_v18”. This made it difficult for people to remember and visit organically. Hence a cheap and simple solution we used is just to rent a proper domain name (“www.esmitemrepository.com”) and redirect visitors from there to our app.

The same goes for (digital) infrastructure. There is often a plethora of free or inexpensive digital tools. For example, sites like GitHub allow not only for software version control, but also project planning tools like milestones.

 

The ESM Item Repository: Review of 2024 and looking ahead to 2025

Olivia Kirtley, January 2025

New Year is always a contemplative period and so it seems like a good moment to reflect on how 2024 was for the ESM Item Repository, and look ahead to what’s in store for 2025.

Last year marked a major milestone for the Repository: in March, we entered our 1000^th item into the Repository. Then, with thanks to our newly established Data Team (Benjamin Kunc, Steffie Schoefs, and Nian Kemme, as well as former interns Ben Símsa and Johan Le Grange), no sooner had we reached 1000 items, but by June, we had more than 3000 items in the Repository. Currently, there are 3,419 items in the ESM Item Repository, from researchers across 11 countries, and we have many more items being cleaned and processed ready for entry into the main Repository database.

In the last two years, the Repository team has been working hard to gather ESM items from “the wild”, i.e., harvesting items listed in published empirical articles and systematic reviews of ESM research. Whilst this may sound pretty simple — look at the methods section and find the ESM item details — this is painstaking, almost forensic work, as many empirical articles and reviews do not provide all the information that we are looking for about the ESM items, and this means we often have to contact authors or follow a citation paper trail to track down information about ESM items. For some, older, ESM studies, the exact wording of the items and their response options may be lost forever. This challenge only underscores the value of the ESM Item Repository and our mission to increase measurement transparency and methodological rigor in ESM research. To this end, alongside our work building an open bank of ESM items for researchers to use, we also conduct research and build tools to advance ESM measurement.

Another major achievement for the ESM Item Repository in 2024 was the release of the first quality assessment tool for ESM items, the ESM-Q. The development of the ESM-Q was led by Research Foundation Flanders Junior Postdoctoral Research Fellow, Dr Gudrun Eisele (Center for Contextual Psychiatry, KU Leuven) and Dr Anu Hiekkaranta (Center for Contextual Psychiatry, KU Leuven; University of Gothenburg, Sweden), and involved a Delphi study with 42 international ESM experts. The idea for the tool was born several years ago in a truly “Hal and the lightbulb” moment, common to ESM research: we need X to do Y, but as X doesn’t exist, I guess we better make X first. The result is a quality assessment tool with 10 core criteria and 15 additional criteria to help researchers judge “what makes a good ESM item?”. We hope that researchers will be able to use the tool to aid them in selecting and designing good ESM items, and potentially also as a tool for assessing item quality in systematic reviews of ESM studies.

Last year also saw the restructuring of the ESM Item Repository team into additional subteams: our existing coding team was joined by the scientific team and the data team. This enables us to better divide the different tasks that keep the Repository going and to work on building specialist expertise within these subteams. Especially important here is our new data team, led by Steffie Schoefs, as a constant stream of high-quality data (ESM items) is central to the continued growth of the Repository. We welcomed several new team members, Nieke Vermaelen and Martien Wampers, including masters student interns Beatriz Diaz (University of Algarve, Portugal), and Christina Stefani and Wikor Januszewski (KU Leuven, Belgium), and we also said goodbye to Nian Kemme (KU Leuven, Belgium), who left her Research Assistant position with us to pursue at PhD at the University of Nijmegen (Netherlands).

So what’s next for the Repository in 2025? For a little while now, we have been working on a pilot for a new workstream at the Repository called “Special Collections”. This builds on the idea of the Repository as a living museum of ESM items. In the same way that museums have special collections on a particular topic, the ESM Item Repository will soon start to display special collections of ESM items on different topics, to aid discoverability of items and facilitate ESM measurement work in specific domains. Watch this space for the first Repository Special Collection coming out in the first quarter of the year. Another development that we are very excited about is being led by CCP PhD student, Laura Van Heck, in collaboration with Martien Wampers and Steffie Schoefs, and that is a brand new item submission system for the Repository. We hope that this new system will make it easier for researchers to submit items to the Repository, and also reduce the potential for errors in the data that can cause back-end coding issues for the Repository search portal. You can expect to see this new system coming out later this year and some ESM Item Repository supporters may even get a sneak peek of the new system during testing.

As ever, we rely on researchers submitting their items to grow the Repository’s collection, so please consider making it your new year’s resolution to contribute to the ESM Item Repository and to help us build a better and more transparent science of daily life.

 

Using the repository: Emotion regulation items

Anu Hiekkaranta, November 2021

Many emotion regulation researchers are now expanding their investigation outside the laboratory to the study of naturally occurring emotion regulation. Ideally suited for such an endeavor of course, is the Experience Sampling Method (ESM). An ESM novice aiming to study emotion regulation in the wild may ask: Where do I start? We suggest you start by browsing the repository! You can explore existing items in the repository by using the search tools at https://www.esmitemrepository.com/. In this post, I will walk you through the search results for two emotion regulation strategies, rumination and acceptance, as well as how you might best make use of what you find in the repository.

Example case 1: Rumination

Let’s say you would like to study rumination in daily life. Rumination items are one of the most often included items in new contributions to the repository. One way to use the repository to find relevant items with a concept in mind is to look up items with a certain description. At the time of writing this blog post searching the repository for items with “rumination” in the description results in 14 hits. You will immediately be able to view the results and some details about them. For more information about each item, you can download the search results. Browsing the rumination items will give you an idea of how different researchers have adapted items from traditional self-report questionnaires to ESM items (e.g., I am thinking about my feelings, I am thinking about my problems (Moberly & Watkins, 2008), and I am ruminating (Snippe et al., 2019)). Moreover, your search results will also illustrate how others have developed ways to investigate ruminating by anchoring to different time frames, specific events, and specific thoughts. For example, some researchers first ask participants what they are thinking about at the moment they got the notification, followed by questions about the characteristics of their current thoughts, such as rating their agreement with statements like: This is a repetitive thought (ESM Item Repository, Kirtley et al., 2019, item 152). Alternatively, participants in a study may be asked to identify a specific event, such as the most important event in the past two hours. Then, participants can be asked questions related the event, such as I have thought about it a lot (most important event in the past 2 hours), (Kirtley et al., 2021). You may also note that researchers have anchored ruminating to different types of events: positive, negative, and important.

Per each search results with the description “rumination”, you can also see, among other details, what population the item has been used in, the response scale, and the number of times the item was presented per day. In addition, browsing for emotion regulation items in the repository reveals differences in how many items different researchers have used to study emotion regulation constructs within their experience sampling questionnaires. You will see for instance, that when studying rumination, some have opted for one item, while others use several.

Now, let’s consider what similarities we find when we search for another emotion regulation strategy in the repository, and take note some of the practical challenges in choosing the items for your study.

Example case 2: Acceptance

As with rumination, browsing the repository for acceptance-related items demonstrates the diversity in operationalization of emotion regulation constructs. For example, the description “acceptance” can be found in the repository for all of the following items:

-I let my thoughts be there without reacting to them (ESM Item Repository, Kirtley et al., 2019: item 307)

-Think about the most negative event today. I let it happen (Kirtley et al., 2021)

-Today, I could let go of my negative feelings without acting upon them (ESM Item Repository, Kirtley et al., 2019, item 341)

-(Most negative event) I just accepted my emotions (Kirtley et al., 2021)

 

Careful examination of these items reveals subtle but important differences. Acceptance, like rumination, can be anchored to e.g., thoughts, events, emotions, or emotions related to specific events. When deciding which items to use for any emotion regulation construct, it will be useful to consider not only one’s theoretical understanding of the construct of acceptance, but also practical concerns. Which of the items are easiest to respond to and how many times per day, given the population in which you want to study acceptance, and the timing of your study? For example, reflecting on one’s thoughts and reactivity to them several times a day requires a level of insight you may not expect from children or teenagers (and indeed, this poses a challenge to adults as well!). To tackle the issue, anchoring acceptance to events may help participants grasp the concept of acceptance better in the moment. However, persons living in quarantine or locked down cities as many currently are, may have few events to report. These concerns raise new questions. What will you consider an event? Do participants need to be briefed with specific examples of events that may take place for them, in their current circumstances? You may also conclude that existing items (gathered from the repository or elsewhere) do not sufficiently capture the type of emotion regulation you are interested in. In that case, we hope you return to the drawing board wiser and even a little more inspired!

 

Discover and develop

 

The field of emotion regulation in daily life is still in its infancy. Indeed, many fields in psychology are still new to experience sampling. Therefore, we invite you to use the repository to discover the existing items relevant to your research, be it emotion regulation or something else. We hope that what you learn may save you time and help you make decisions about which items to use. Moreover, we hope to inspire you to think about how to validate existing items and how to adapt and develop items to your research needs. As a final point, once you have collected some data, we warmly encourage you to make your own submission to the ESM Item Repository!

 

References:

 

Kirtley, O. J., Hiekkaranta, A. P., Kunkels, Y. K., Verhoeven, D., Van Nierop, M., & Myin-Germeys, I. (2019). The Experience Sampling Method (ESM) Item Repository. https://doi.org/10.17605/OSF.IO/KG376

Kirtley, O. J., Achterhof, R., Hagemann, N., Hermans, K. S. F. M., Hiekkaranta, A. P., Lecei, A., … Myin-Germeys, I. (2021, April 2). Initial cohort characteristics and protocol for SIGMA: An accelerated longitudinal study of environmental factors, inter- and intrapersonal processes, and mental health in adolescence. https://doi.org/10.31234/osf.io/jp2fk

 

Moberly, N. J., & Watkins, E, R. (2008). Ruminative self-focus and negative affect: An experience sampling study. Journal of Abnormal Psychology, 117(2), 314.

 

Snippe, E., Helmich, M., Kunkels, Y. K., Riese, H., Smit, A., & Wichers, M. (2019). Esm Item Documentation. Retrieved from https://osf.io/a8572/.

 

The Experience Sampling Method Item Repository: What are we doing, why are we doing it and where are we going next?

Olivia Kirtley, October 2020

The Experience Sampling Method Item Repository is one year old! To celebrate our first birthday, we have launched a new blog where the rest of the repository team (Anu Hiekkaranta, Yoram Kunkels, Gudrun Eisele, Davinia Verhoeven, Martine van Nierop and Inez Myin-Germeys) and I will keep you up-to-date on the latest developments in the repository project, including highlighting the collection of items, and discussing ESM research more broadly. So let me start us off with our very first blog post, to tell you a bit about what we are doing, why we’re doing it and what lies ahead for the repository.

    The Experience Sampling Method (ESM; Csikszentmihalyi & Larson, 1987), sometimes referred to as Ecological Momentary Assessment (EMA; Stone & Shiffman, 1994) brings myriad possibilities for gaining rich, valuable data about the context in which individuals’ thoughts, feelings and behaviours arise: their normal day-to-day lives (Myin-Germeys et al., 2018). Capturing these daily-life experiences, outside of the lab and the clinic brings us many new opportunities for increasing our understanding of phenomena, including individuals’ social experiences, emotion regulation, psychopathology symptoms and self-harm thoughts and behaviours. Over the years, ESM research has grown a lot and is a field full of exciting, cutting-edge statistical and technological developments. Perhaps surprisingly, however, some key questions around how we conduct ESM research, including about the measures that we use, remain unanswered.

    We do not have a suite of validated ESM questionnaires, which have undergone a rigorous psychometric validation process, in the same way that we have validated measures for assessing depression or anxiety*. Consequently, many ESM items are constructed by researchers “on the fly”, and passed along from one researcher to another. Sometimes the same ESM items are used to assess different or related constructs and there is currently very little psychometric validation of ESM items (Horstmann & Ziegler, 2020; Wright & Zimmermann, 2019), with only around 30% of papers in a recent review reporting the psychometric properties of the ESM items used (Trull & Ebner-Priemer, 2020). Yet, measurement matters (Fried & Flake, 2018) and lack of transparency around measures, as well as the absence of solid psychometric validation, can threaten the transparency and methodological reproducibility of ESM research more broadly. Further, it may also limit the conclusions we can draw from our studies. Whilst conversations around the transparency, reproducibility and replicability of research have mainly focused on cognitive and social psychology, this does not mean that these issues do not apply to ESM research (Kirtley et al., 2020). So how do we go about addressing these issues?

    A good first step is to get an overview of the items currently “out in the wild” of the ESM research landscape. That’s where we come in. The Experience Sampling Method (ESM) Item Repository (Kirtley et al., 2020) is an ongoing three-phase project in which we are building an open (publicly available) bank of ESM items, with the help of contributions from ESM researchers from around the world. Phase I of the project involves populating the repository and collecting as many items as possible. This first version of the repository is available to discover and search at www.esmitemrepository.com. At this stage, the repository has only undergone minor quality screening for formatting issues, spelling and language errors etc, so is very much still a “warts and all” list of ESM items. Items cover topics including mood, emotion regulation, psychotic experiences, quantity and quality of online and offline social experiences and self-harm thoughts and behaviours. This allows us and other researchers to gain insights regarding the range and content of ESM items being used, and to begin to identify patterns among items. Phase II of the project is just about to be launched and involves a quality assessment of the items in the repository. To accomplish this task, we need a quality assessment tool for ESM items, but because research is rather like Hal changing a lightbulb (https://youtu.be/AbSehcT19u0), we first need to develop a quality assessment tool. Gudrun Eisele, PhD student in the Center for Contextual Psychiatry (CCP), is leading this work with fellow CCP PhD student Anu Hiekkaranta and they’ll be telling you more about that in the next blog post. In Phase III of the project, we will begin the psychometric validation of items within the repository, but we don’t just want to keep all the psychometric fun to ourselves; a key goal of the repository is also to facilitate other scientists to conduct ESM research and to psychometrically validate available items. We’re busy with plans for this phase of the project and will share more details soon.

    Our main goals in establishing the repository are to increase the discoverability of ESM items, facilitate ESM research and to improve the transparency and quality of ESM research. Given that ESM is increasingly used in clinical psychology and psychiatry research, we also hope to make a contribution to transparency and reproducibility within those areas, which have often not been part of conversations around open science practices. For more on that topic, please read the brilliant work of Jennifer Tackett and her colleagues (Tackett, Brandes, & Reardon, 2019; Tackett et al., 2019). The rest of the fantastic ESM Item Repository team and I are thrilled that our efforts to increase the openness of ESM research have been recognised by two awards this year: a commendation from the Society for the Improvement of Psychological Science (https://twitter.com/improvingpsych/status/1247246158432526336?s=20) and an Open Research Award from the University of Groningen (https://twitter.com/Bibliothecaris/status/1311299779591385089?s=20). We’re not done yet though! There are lots more exciting things to come. A vital and ongoing task is continuing to build the collection of items. After all, the success of the ESM Item Repository depends on researchers contributing their items, so please consider sending us your items to include in the repository! To find out more information about the project and how to contribute items, check out the contributors pack on our OSF page (https://osf.io/kg376/).

 

Notes:

* Unfortunately, in some cases, the “rigorous psychometric validation process” for traditional self-report questionnaires has also been treated as more of a serving suggestion. I could write a whole other blog post about this (and maybe I will!), but let’s stick to ESM for now. Interested readers should definitely check out all the wonderful things that Eiko Fried and Jessica Flake have written about these issues (see references below).

 

References:

Csikszentmihalyi, M., & Larson, R. (1987). Validity and reliability of the Experience Sampling Method. Journal of Nervous and Mental Disease, 175(9), 526–536. 

Flake, J. K., & Fried, E. I. (In press). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Advances in Methods and Practices in Psychological Science. Preprint: https://doi.org/10.31234/osf.io/hs7wm

Fried & Flake (2018). Measurement Matters. The Observer (Association for Psychological Science).

Horstmann, K. T., and Ziegler, M. (2020) Assessing Personality States: What to Consider when Constructing Personality State Measures. European Journal of Personality.

Kirtley, O. J., Hiekkaranta, A. P., Kunkels, Y. K., Eisele, G., Verhoeven, D., Van Nierop, M., & Myin-Germeys, I. (2020, October 1). The Experience Sampling Method (ESM) Item Repository. https://doi.org/10.17605/OSF.IO/KG376

Kirtley, O. J., Lafit, G., Achterhof, R., Hiekkaranta, A. P., & Myin-Germeys, I. (In press). Making the black box transparent: A template and tutorial for (pre-)registration of studies using Experience Sampling Methods (ESM). Advances in Methods and Practices in Psychological Science. Preprint: https://doi.org/10.31234/osf.io/seyq7

Myin‐Germeys, I., Kasanova, Z., Vaessen, T., Vachon, H., Kirtley, O., Viechtbauer, W., & Reininghaus, U. (2018). Experience sampling methodology in mental health research: new insights and technical developments. World Psychiatry, 17(2), 123-132.

Stone, A.A., & Shiffman, S. (1994). Ecological momentary assessment (EMA) in behavioural medicine. Annals of Behavioral Medicine, 16, 199-202.

Tackett, J. L., Brandes, C. M., & Reardon, K. W. (2019). Leveraging the Open Science Framework in clinical psychological assessment research. Psychological Assessment. doi:10.1037/pas0000583

Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., . . . Shrout, P. E. (2017). It's Time to Broaden the Replicability Conversation: Thoughts for and From Clinical Psychological Science. Perspectives in Psychological Science, 12(5), 742-756. doi:10.1177/1745691617690042

Trull, T. J., & Ebner-Priemer, U. W. (2020). Ambulatory assessment in psychopathology research: A review of recommended reporting guidelines and current practices. Journal of Abnormal Psychology, 129(1), 56–63.

Wright, A. G. C., & Zimmermann, J. (2019). Applied ambulatory assessment: Integrating idiographic and nomothetic principles of measurement. Psychological Assessment, 31(12), 1467–1480. https://doi.org/10.1037/pas0000685

 

About the author: Olivia Kirtley is an FWO Senior Postdoctoral Research Fellow within the Center for Contextual Psychiatry at KU Leuven in Belgium, where she also leads “SIGMA”, a large-scale longitudinal study of adolescent mental health and development using experience sampling methods (ESM). Her current research uses ESM to investigate dynamic processes involved in ideation-to-action transitions in adolescents who self-harm. Olivia leads several projects aimed at increasing transparency and reproducibility in the ESM field, including designing a pre-registration template for ESM research and leading the ESM Item Repository.