objective of the study in research paper sample

21 Research Objectives Examples (Copy and Paste)

Research objectives refer to the definitive statements made by researchers at the beginning of a research project detailing exactly what a research project aims to achieve.

These objectives are explicit goals clearly and concisely projected by the researcher to present a clear intention or course of action for his or her qualitative or quantitative study. 

Research objectives are typically nested under one overarching research aim. The objectives are the steps you’ll need to take in order to achieve the aim (see the examples below, for example, which demonstrate an aim followed by 3 objectives, which is what I recommend to my research students).

Research Objectives vs Research Aims

Research aim and research objectives are fundamental constituents of any study, fitting together like two pieces of the same puzzle.

The ‘research aim’ describes the overarching goal or purpose of the study (Kumar, 2019). This is usually a broad, high-level purpose statement, summing up the central question that the research intends to answer.

Example of an Overarching Research Aim:

“The aim of this study is to explore the impact of climate change on crop productivity.” 

Comparatively, ‘research objectives’ are concrete goals that underpin the research aim, providing stepwise actions to achieve the aim.

Objectives break the primary aim into manageable, focused pieces, and are usually characterized as being more specific, measurable, achievable, relevant, and time-bound (SMART).

Examples of Specific Research Objectives:

1. “To examine the effects of rising temperatures on the yield of rice crops during the upcoming growth season.” 2. “To assess changes in rainfall patterns in major agricultural regions over the first decade of the twenty-first century (2000-2010).” 3. “To analyze the impact of changing weather patterns on crop diseases within the same timeframe.”

The distinction between these two terms, though subtle, is significant for successfully conducting a study. The research aim provides the study with direction, while the research objectives set the path to achieving this aim, thereby ensuring the study’s efficiency and effectiveness.

How to Write Research Objectives

I usually recommend to my students that they use the SMART framework to create their research objectives.

SMART is an acronym standing for Specific, Measurable, Achievable, Relevant, and Time-bound. It provides a clear method of defining solid research objectives and helps students know where to start in writing their objectives (Locke & Latham, 2013).

Each element of this acronym adds a distinct dimension to the framework, aiding in the creation of comprehensive, well-delineated objectives.

Here is each step:

• Specific : We need to avoid ambiguity in our objectives. They need to be clear and precise (Doran, 1981). For instance, rather than stating the objective as “to study the effects of social media,” a more focused detail would be “to examine the effects of social media use (Facebook, Instagram, and Twitter) on the academic performance of college students.”
• Measurable: The measurable attribute provides a clear criterion to determine if the objective has been met (Locke & Latham, 2013). A quantifiable element, such as a percentage or a number, adds a measurable quality. For example, “to increase response rate to the annual customer survey by 10%,” makes it easier to ascertain achievement.
• Achievable: The achievable aspect encourages researchers to craft realistic objectives, resembling a self-check mechanism to ensure the objectives align with the scope and resources at disposal (Doran, 1981). For example, “to interview 25 participants selected randomly from a population of 100” is an attainable objective as long as the researcher has access to these participants.
• Relevance : Relevance, the fourth element, compels the researcher to tailor the objectives in alignment with overarching goals of the study (Locke & Latham, 2013). This is extremely important – each objective must help you meet your overall one-sentence ‘aim’ in your study.
• Time-Bound: Lastly, the time-bound element fosters a sense of urgency and prioritization, preventing procrastination and enhancing productivity (Doran, 1981). “To analyze the effect of laptop use in lectures on student engagement over the course of two semesters this year” expresses a clear deadline, thus serving as a motivator for timely completion.

You’re not expected to fit every single element of the SMART framework in one objective, but across your objectives, try to touch on each of the five components.

Research Objectives Examples

1. Field: Psychology

Aim: To explore the impact of sleep deprivation on cognitive performance in college students.

• Objective 1: To compare cognitive test scores of students with less than six hours of sleep and those with 8 or more hours of sleep.
• Objective 2: To investigate the relationship between class grades and reported sleep duration.
• Objective 3: To survey student perceptions and experiences on how sleep deprivation affects their cognitive capabilities.

2. Field: Environmental Science

Aim: To understand the effects of urban green spaces on human well-being in a metropolitan city.

• Objective 1: To assess the physical and mental health benefits of regular exposure to urban green spaces.
• Objective 2: To evaluate the social impacts of urban green spaces on community interactions.
• Objective 3: To examine patterns of use for different types of urban green spaces. 

3. Field: Technology

Aim: To investigate the influence of using social media on productivity in the workplace.

• Objective 1: To measure the amount of time spent on social media during work hours.
• Objective 2: To evaluate the perceived impact of social media use on task completion and work efficiency.
• Objective 3: To explore whether company policies on social media usage correlate with different patterns of productivity.

4. Field: Education

Aim: To examine the effectiveness of online vs traditional face-to-face learning on student engagement and achievement.

• Objective 1: To compare student grades between the groups exposed to online and traditional face-to-face learning.
• Objective 2: To assess student engagement levels in both learning environments.
• Objective 3: To collate student perceptions and preferences regarding both learning methods.

5. Field: Health

Aim: To determine the impact of a Mediterranean diet on cardiac health among adults over 50.

• Objective 1: To assess changes in cardiovascular health metrics after following a Mediterranean diet for six months.
• Objective 2: To compare these health metrics with a similar group who follow their regular diet.
• Objective 3: To document participants’ experiences and adherence to the Mediterranean diet.

6. Field: Environmental Science

Aim: To analyze the impact of urban farming on community sustainability.

• Objective 1: To document the types and quantity of food produced through urban farming initiatives.
• Objective 2: To assess the effect of urban farming on local communities’ access to fresh produce.
• Objective 3: To examine the social dynamics and cooperative relationships in the creating and maintaining of urban farms.

7. Field: Sociology

Aim: To investigate the influence of home offices on work-life balance during remote work.

• Objective 1: To survey remote workers on their perceptions of work-life balance since setting up home offices.
• Objective 2: To conduct an observational study of daily work routines and family interactions in a home office setting.
• Objective 3: To assess the correlation, if any, between physical boundaries of workspaces and mental boundaries for work in the home setting.

8. Field: Economics

Aim: To evaluate the effects of minimum wage increases on small businesses.

• Objective 1: To analyze cost structures, pricing changes, and profitability of small businesses before and after minimum wage increases.
• Objective 2: To survey small business owners on the strategies they employ to navigate minimum wage increases.
• Objective 3: To examine employment trends in small businesses in response to wage increase legislation.

9. Field: Education

Aim: To explore the role of extracurricular activities in promoting soft skills among high school students.

• Objective 1: To assess the variety of soft skills developed through different types of extracurricular activities.
• Objective 2: To compare self-reported soft skills between students who participate in extracurricular activities and those who do not.
• Objective 3: To investigate the teachers’ perspectives on the contribution of extracurricular activities to students’ skill development.

10. Field: Technology

Aim: To assess the impact of virtual reality (VR) technology on the tourism industry.

• Objective 1: To document the types and popularity of VR experiences available in the tourism market.
• Objective 2: To survey tourists on their interest levels and satisfaction rates with VR tourism experiences.
• Objective 3: To determine whether VR tourism experiences correlate with increased interest in real-life travel to the simulated destinations.

11. Field: Biochemistry

Aim: To examine the role of antioxidants in preventing cellular damage.

• Objective 1: To identify the types and quantities of antioxidants in common fruits and vegetables.
• Objective 2: To determine the effects of various antioxidants on free radical neutralization in controlled lab tests.
• Objective 3: To investigate potential beneficial impacts of antioxidant-rich diets on long-term cellular health.

12. Field: Linguistics

Aim: To determine the influence of early exposure to multiple languages on cognitive development in children.

• Objective 1: To assess cognitive development milestones in monolingual and multilingual children.
• Objective 2: To document the number and intensity of language exposures for each group in the study.
• Objective 3: To investigate the specific cognitive advantages, if any, enjoyed by multilingual children.

13. Field: Art History

Aim: To explore the impact of the Renaissance period on modern-day art trends.

• Objective 1: To identify key characteristics and styles of Renaissance art.
• Objective 2: To analyze modern art pieces for the influence of the Renaissance style.
• Objective 3: To survey modern-day artists for their inspirations and the influence of historical art movements on their work.

14. Field: Cybersecurity

Aim: To assess the effectiveness of two-factor authentication (2FA) in preventing unauthorized system access.

• Objective 1: To measure the frequency of unauthorized access attempts before and after the introduction of 2FA.
• Objective 2: To survey users about their experiences and challenges with 2FA implementation.
• Objective 3: To evaluate the efficacy of different types of 2FA (SMS-based, authenticator apps, biometrics, etc.).

15. Field: Cultural Studies

Aim: To analyze the role of music in cultural identity formation among ethnic minorities.

• Objective 1: To document the types and frequency of traditional music practices within selected ethnic minority communities.
• Objective 2: To survey community members on the role of music in their personal and communal identity.
• Objective 3: To explore the resilience and transmission of traditional music practices in contemporary society.

16. Field: Astronomy

Aim: To explore the impact of solar activity on satellite communication.

• Objective 1: To categorize different types of solar activities and their frequencies of occurrence.
• Objective 2: To ascertain how variations in solar activity may influence satellite communication.
• Objective 3: To investigate preventative and damage-control measures currently in place during periods of high solar activity.

17. Field: Literature

Aim: To examine narrative techniques in contemporary graphic novels.

• Objective 1: To identify a range of narrative techniques employed in this genre.
• Objective 2: To analyze the ways in which these narrative techniques engage readers and affect story interpretation.
• Objective 3: To compare narrative techniques in graphic novels to those found in traditional printed novels.

18. Field: Renewable Energy

Aim: To investigate the feasibility of solar energy as a primary renewable resource within urban areas.

• Objective 1: To quantify the average sunlight hours across urban areas in different climatic zones. 
• Objective 2: To calculate the potential solar energy that could be harnessed within these areas.
• Objective 3: To identify barriers or challenges to widespread solar energy implementation in urban settings and potential solutions.

19. Field: Sports Science

Aim: To evaluate the role of pre-game rituals in athlete performance.

• Objective 1: To identify the variety and frequency of pre-game rituals among professional athletes in several sports.
• Objective 2: To measure the impact of pre-game rituals on individual athletes’ performance metrics.
• Objective 3: To examine the psychological mechanisms that might explain the effects (if any) of pre-game ritual on performance.

20. Field: Ecology

Aim: To investigate the effects of urban noise pollution on bird populations.

• Objective 1: To record and quantify urban noise levels in various bird habitats.
• Objective 2: To measure bird population densities in relation to noise levels.
• Objective 3: To determine any changes in bird behavior or vocalization linked to noise levels.

21. Field: Food Science

Aim: To examine the influence of cooking methods on the nutritional value of vegetables.

• Objective 1: To identify the nutrient content of various vegetables both raw and after different cooking processes.
• Objective 2: To compare the effect of various cooking methods on the nutrient retention of these vegetables.
• Objective 3: To propose cooking strategies that optimize nutrient retention.

The Importance of Research Objectives

The importance of research objectives cannot be overstated. In essence, these guideposts articulate what the researcher aims to discover, understand, or examine (Kothari, 2014).

When drafting research objectives, it’s essential to make them simple and comprehensible, specific to the point of being quantifiable where possible, achievable in a practical sense, relevant to the chosen research question, and time-constrained to ensure efficient progress (Kumar, 2019). 

Remember that a good research objective is integral to the success of your project, offering a clear path forward for setting out a research design , and serving as the bedrock of your study plan. Each objective must distinctly address a different dimension of your research question or problem (Kothari, 2014). Always bear in mind that the ultimate purpose of your research objectives is to succinctly encapsulate your aims in the clearest way possible, facilitating a coherent, comprehensive and rational approach to your planned study, and furnishing a scientific roadmap for your journey into the depths of knowledge and research (Kumar, 2019). 

Kothari, C.R (2014). Research Methodology: Methods and Techniques . New Delhi: New Age International.

Kumar, R. (2019). Research Methodology: A Step-by-Step Guide for Beginners .New York: SAGE Publications.

Doran, G. T. (1981). There’s a S.M.A.R.T. way to write management’s goals and objectives. Management review, 70 (11), 35-36.

Locke, E. A., & Latham, G. P. (2013). New Developments in Goal Setting and Task Performance . New York: Routledge.

Chris Drew (PhD)

Dr. Chris Drew is the founder of the Helpful Professor. He holds a PhD in education and has published over 20 articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education. [Image Descriptor: Photo of Chris]

• Chris Drew (PhD) https://helpfulprofessor.com/author/admin/ 5 Best Laminators for Teachers, Reviewed!
• Chris Drew (PhD) https://helpfulprofessor.com/author/admin/ 5 Best College Care Packages for Guys & Girls
• Chris Drew (PhD) https://helpfulprofessor.com/author/admin/ 7 Best Microwaves for College Student Dorms
• Chris Drew (PhD) https://helpfulprofessor.com/author/admin/ 5 Best Backpacks for Teachers

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

Educational resources and simple solutions for your research journey

What Are Research Objectives and How to Write Them (with Examples)

Table of Contents

Introduction

Research is at the center of everything researchers do, and setting clear, well-defined research objectives plays a pivotal role in guiding scholars toward their desired outcomes. Research papers are essential instruments for researchers to effectively communicate their work. Among the many sections that constitute a research paper, the introduction plays a key role in providing a background and setting the context. 1 Research objectives, which define the aims of the study, are usually stated in the introduction. Every study has a research question that the authors are trying to answer, and the objective is an active statement about how the study will answer this research question. These objectives help guide the development and design of the study and steer the research in the appropriate direction; if this is not clearly defined, a project can fail!

Research studies have a research question, research hypothesis, and one or more research objectives. A research question is what a study aims to answer, and a research hypothesis is a predictive statement about the relationship between two or more variables, which the study sets out to prove or disprove. Objectives are specific, measurable goals that the study aims to achieve. The difference between these three is illustrated by the following example:

• Research question : How does low-intensity pulsed ultrasound (LIPUS) compare with a placebo device in managing the symptoms of skeletally mature patients with patellar tendinopathy?
• Research hypothesis : Pain levels are reduced in patients who receive daily active-LIPUS (treatment) for 12 weeks compared with individuals who receive inactive-LIPUS (placebo).
• Research objective : To investigate the clinical efficacy of LIPUS in the management of patellar tendinopathy symptoms.

This article discusses the importance of clear, well-thought out objectives and suggests methods to write them clearly.

What is the introduction in research papers?

Research objectives are usually included in the introduction section. This section is the first that the readers will read so it is essential that it conveys the subject matter appropriately and is well written to create a good first impression. A good introduction sets the tone of the paper and clearly outlines the contents so that the readers get a quick snapshot of what to expect.

A good introduction should aim to: 2,3

• Indicate the main subject area, its importance, and cite previous literature on the subject
• Define the gap(s) in existing research, ask a research question, and state the objectives
• Announce the present research and outline its novelty and significance
• Avoid repeating the Abstract, providing unnecessary information, and claiming novelty without accurate supporting information.

Why are research objectives important?

Objectives can help you stay focused and steer your research in the required direction. They help define and limit the scope of your research, which is important to efficiently manage your resources and time. The objectives help to create and maintain the overall structure, and specify two main things—the variables and the methods of quantifying the variables.

A good research objective:

• defines the scope of the study
• gives direction to the research
• helps maintain focus and avoid diversions from the topic
• minimizes wastage of resources like time, money, and energy

Types of research objectives

Research objectives can be broadly classified into general and specific objectives . 4 General objectives state what the research expects to achieve overall while specific objectives break this down into smaller, logically connected parts, each of which addresses various parts of the research problem. General objectives are the main goals of the study and are usually fewer in number while specific objectives are more in number because they address several aspects of the research problem.

Example (general objective): To investigate the factors influencing the financial performance of firms listed in the New York Stock Exchange market.

Example (specific objective): To assess the influence of firm size on the financial performance of firms listed in the New York Stock Exchange market.

In addition to this broad classification, research objectives can be grouped into several categories depending on the research problem, as given in Table 1.

Table 1: Types of research objectives

Characteristics of research objectives

Research objectives must start with the word “To” because this helps readers identify the objective in the absence of headings and appropriate sectioning in research papers. 5,6

• A good objective is SMART (mostly applicable to specific objectives):
• Specific—clear about the what, why, when, and how
• Measurable—identifies the main variables of the study and quantifies the targets
• Achievable—attainable using the available time and resources
• Realistic—accurately addresses the scope of the problem
• Time-bound—identifies the time in which each step will be completed
• Research objectives clarify the purpose of research.
• They help understand the relationship and dissimilarities between variables.
• They provide a direction that helps the research to reach a definite conclusion.

How to write research objectives?

Research objectives can be written using the following steps: 7

• State your main research question clearly and concisely.
• Describe the ultimate goal of your study, which is similar to the research question but states the intended outcomes more definitively.
• Divide this main goal into subcategories to develop your objectives.
• Limit the number of objectives (1-2 general; 3-4 specific)
• Assess each objective using the SMART
• Start each objective with an action verb like assess, compare, determine, evaluate, etc., which makes the research appear more actionable.
• Use specific language without making the sentence data heavy.
• The most common section to add the objectives is the introduction and after the problem statement.
• Add the objectives to the abstract (if there is one).
• State the general objective first, followed by the specific objectives.

Formulating research objectives

Formulating research objectives has the following five steps, which could help researchers develop a clear objective: 8

• Identify the research problem.
• Review past studies on subjects similar to your problem statement, that is, studies that use similar methods, variables, etc.
• Identify the research gaps the current study should cover based on your literature review. These gaps could be theoretical, methodological, or conceptual.
• Define the research question(s) based on the gaps identified.
• Revise/relate the research problem based on the defined research question and the gaps identified. This is to confirm that there is an actual need for a study on the subject based on the gaps in literature.
• Identify and write the general and specific objectives.
• Incorporate the objectives into the study.

Advantages of research objectives

Adding clear research objectives has the following advantages: 4,8

• Maintains the focus and direction of the research
• Optimizes allocation of resources with minimal wastage
• Acts as a foundation for defining appropriate research questions and hypotheses
• Provides measurable outcomes that can help evaluate the success of the research
• Determines the feasibility of the research by helping to assess the availability of required resources
• Ensures relevance of the study to the subject and its contribution to existing literature

Disadvantages of research objectives

Research objectives also have few disadvantages, as listed below: 8

• Absence of clearly defined objectives can lead to ambiguity in the research process
• Unintentional bias could affect the validity and accuracy of the research findings

Key takeaways

• Research objectives are concise statements that describe what the research is aiming to achieve.
• They define the scope and direction of the research and maintain focus.
• The objectives should be SMART—specific, measurable, achievable, realistic, and time-bound.
• Clear research objectives help avoid collection of data or resources not required for the study.
• Well-formulated specific objectives help develop the overall research methodology, including data collection, analysis, interpretation, and utilization.
• Research objectives should cover all aspects of the problem statement in a coherent way.
• They should be clearly stated using action verbs.

Frequently asked questions on research objectives

Q: what’s the difference between research objectives and aims 9.

A: Research aims are statements that reflect the broad goal(s) of the study and outline the general direction of the research. They are not specific but clearly define the focus of the study.

Example: This research aims to explore employee experiences of digital transformation in retail HR.

Research objectives focus on the action to be taken to achieve the aims. They make the aims more practical and should be specific and actionable.

Example: To observe the retail HR employees throughout the digital transformation.

Q: What are the examples of research objectives, both general and specific?

A: Here are a few examples of research objectives:

• To identify the antiviral chemical constituents in Mumbukura gitoniensis (general)
• To carry out solvent extraction of dried flowers of Mumbukura gitoniensis and isolate the constituents. (specific)
• To determine the antiviral activity of each of the isolated compounds. (specific)
• To examine the extent, range, and method of coral reef rehabilitation projects in five shallow reef areas adjacent to popular tourist destinations in the Philippines.
• To investigate species richness of mammal communities in five protected areas over the past 20 years.
• To evaluate the potential application of AI techniques for estimating best-corrected visual acuity from fundus photographs with and without ancillary information.
• To investigate whether sport influences psychological parameters in the personality of asthmatic children.

Q: How do I develop research objectives?

A: Developing research objectives begins with defining the problem statement clearly, as illustrated by Figure 1. Objectives specify how the research question will be answered and they determine what is to be measured to test the hypothesis.

Q: Are research objectives measurable?

A: The word “measurable” implies that something is quantifiable. In terms of research objectives, this means that the source and method of collecting data are identified and that all these aspects are feasible for the research. Some metrics can be created to measure your progress toward achieving your objectives.

Q: Can research objectives change during the study?

A: Revising research objectives during the study is acceptable in situations when the selected methodology is not progressing toward achieving the objective, or if there are challenges pertaining to resources, etc. One thing to keep in mind is the time and resources you would have to complete your research after revising the objectives. Thus, as long as your problem statement and hypotheses are unchanged, minor revisions to the research objectives are acceptable.

Q: What is the difference between research questions and research objectives? 10

Q: are research objectives the same as hypotheses.

A: No, hypotheses are predictive theories that are expressed in general terms. Research objectives, which are more specific, are developed from hypotheses and aim to test them. A hypothesis can be tested using several methods and each method will have different objectives because the methodology to be used could be different. A hypothesis is developed based on observation and reasoning; it is a calculated prediction about why a particular phenomenon is occurring. To test this prediction, different research objectives are formulated. Here’s a simple example of both a research hypothesis and research objective.

Research hypothesis : Employees who arrive at work earlier are more productive.

Research objective : To assess whether employees who arrive at work earlier are more productive.

To summarize, research objectives are an important part of research studies and should be written clearly to effectively communicate your research. We hope this article has given you a brief insight into the importance of using clearly defined research objectives and how to formulate them.

• Farrugia P, Petrisor BA, Farrokhyar F, Bhandari M. Practical tips for surgical research: Research questions, hypotheses and objectives. Can J Surg. 2010 Aug;53(4):278-81.
• Abbadia J. How to write an introduction for a research paper. Mind the Graph website. Accessed June 14, 2023. https://mindthegraph.com/blog/how-to-write-an-introduction-for-a-research-paper/
• Writing a scientific paper: Introduction. UCI libraries website. Accessed June 15, 2023. https://guides.lib.uci.edu/c.php?g=334338&p=2249903
• Research objectives—Types, examples and writing guide. Researchmethod.net website. Accessed June 17, 2023. https://researchmethod.net/research-objectives/#:~:text=They%20provide%20a%20clear%20direction,track%20and%20achieve%20their%20goals .
• Bartle P. SMART Characteristics of good objectives. Community empowerment collective website. Accessed June 16, 2023. https://cec.vcn.bc.ca/cmp/modules/pd-smar.htm
• Research objectives. Studyprobe website. Accessed June 18, 2023. https://www.studyprobe.in/2022/08/research-objectives.html
• Corredor F. How to write objectives in a research paper. wikiHow website. Accessed June 18, 2023. https://www.wikihow.com/Write-Objectives-in-a-Research-Proposal
• Research objectives: Definition, types, characteristics, advantages. AccountingNest website. Accessed June 15, 2023. https://www.accountingnest.com/articles/research/research-objectives
• Phair D., Shaeffer A. Research aims, objectives & questions. GradCoach website. Accessed June 20, 2023. https://gradcoach.com/research-aims-objectives-questions/
• Understanding the difference between research questions and objectives. Accessed June 21, 2023. https://board.researchersjob.com/blog/research-questions-and-objectives

R Discovery is a literature search and research reading app that instantly creates personalized reading feeds based on your interests. Researchers can keep up with the latest, most relevant content from its growing library of 115M+ research articles sourced from top aggregators such as CrossRef, Unpaywall, PubMed, PubMed Central, Open Alex as well as reputed publishers like Springer Nature, Elsevier, Wiley, IOP, Taylor & Francis, NEJM, BMJ, Karger, Emerald Publishing and more. The top-rated app in its space, R Discovery’s carefully curated features give you the power to choose what, where, and how you read research.

Get the app for free or upgrade to R Discovery Prime, which unlocks unlimited access to app-only features that let you listen to research on the go, read in your language, add collaborators to your reading lists, and auto sync your library to reference managers. Experience the world of research at your fingertips, get R Discovery Prime now at just US$39 a year!

Related Posts

Introducing Ask R Discovery: Get Factual Answers, Validated by Scientific Research

Starting young: Acknowledging the Growing Interest in Science for Kids

• Privacy Policy
• SignUp/Login

Home » Research Objectives – Types, Examples and Writing Guide

Research Objectives – Types, Examples and Writing Guide

Table of Contents

Research Objectives

Research objectives refer to the specific goals or aims of a research study. They provide a clear and concise description of what the researcher hopes to achieve by conducting the research . The objectives are typically based on the research questions and hypotheses formulated at the beginning of the study and are used to guide the research process.

Types of Research Objectives

Here are the different types of research objectives in research:

• Exploratory Objectives: These objectives are used to explore a topic, issue, or phenomenon that has not been studied in-depth before. The aim of exploratory research is to gain a better understanding of the subject matter and generate new ideas and hypotheses .
• Descriptive Objectives: These objectives aim to describe the characteristics, features, or attributes of a particular population, group, or phenomenon. Descriptive research answers the “what” questions and provides a snapshot of the subject matter.
• Explanatory Objectives : These objectives aim to explain the relationships between variables or factors. Explanatory research seeks to identify the cause-and-effect relationships between different phenomena.
• Predictive Objectives: These objectives aim to predict future events or outcomes based on existing data or trends. Predictive research uses statistical models to forecast future trends or outcomes.
• Evaluative Objectives : These objectives aim to evaluate the effectiveness or impact of a program, intervention, or policy. Evaluative research seeks to assess the outcomes or results of a particular intervention or program.
• Prescriptive Objectives: These objectives aim to provide recommendations or solutions to a particular problem or issue. Prescriptive research identifies the best course of action based on the results of the study.
• Diagnostic Objectives : These objectives aim to identify the causes or factors contributing to a particular problem or issue. Diagnostic research seeks to uncover the underlying reasons for a particular phenomenon.
• Comparative Objectives: These objectives aim to compare two or more groups, populations, or phenomena to identify similarities and differences. Comparative research is used to determine which group or approach is more effective or has better outcomes.
• Historical Objectives: These objectives aim to examine past events, trends, or phenomena to gain a better understanding of their significance and impact. Historical research uses archival data, documents, and records to study past events.
• Ethnographic Objectives : These objectives aim to understand the culture, beliefs, and practices of a particular group or community. Ethnographic research involves immersive fieldwork and observation to gain an insider’s perspective of the group being studied.
• Action-oriented Objectives: These objectives aim to bring about social or organizational change. Action-oriented research seeks to identify practical solutions to social problems and to promote positive change in society.
• Conceptual Objectives: These objectives aim to develop new theories, models, or frameworks to explain a particular phenomenon or set of phenomena. Conceptual research seeks to provide a deeper understanding of the subject matter by developing new theoretical perspectives.
• Methodological Objectives: These objectives aim to develop and improve research methods and techniques. Methodological research seeks to advance the field of research by improving the validity, reliability, and accuracy of research methods and tools.
• Theoretical Objectives : These objectives aim to test and refine existing theories or to develop new theoretical perspectives. Theoretical research seeks to advance the field of knowledge by testing and refining existing theories or by developing new theoretical frameworks.
• Measurement Objectives : These objectives aim to develop and validate measurement instruments, such as surveys, questionnaires, and tests. Measurement research seeks to improve the quality and reliability of data collection and analysis by developing and testing new measurement tools.
• Design Objectives : These objectives aim to develop and refine research designs, such as experimental, quasi-experimental, and observational designs. Design research seeks to improve the quality and validity of research by developing and testing new research designs.
• Sampling Objectives: These objectives aim to develop and refine sampling techniques, such as probability and non-probability sampling methods. Sampling research seeks to improve the representativeness and generalizability of research findings by developing and testing new sampling techniques.

How to Write Research Objectives

Writing clear and concise research objectives is an important part of any research project, as it helps to guide the study and ensure that it is focused and relevant. Here are some steps to follow when writing research objectives:

• Identify the research problem : Before you can write research objectives, you need to identify the research problem you are trying to address. This should be a clear and specific problem that can be addressed through research.
• Define the research questions : Based on the research problem, define the research questions you want to answer. These questions should be specific and should guide the research process.
• Identify the variables : Identify the key variables that you will be studying in your research. These are the factors that you will be measuring, manipulating, or analyzing to answer your research questions.
• Write specific objectives: Write specific, measurable objectives that will help you answer your research questions. These objectives should be clear and concise and should indicate what you hope to achieve through your research.
• Use the SMART criteria: To ensure that your research objectives are well-defined and achievable, use the SMART criteria. This means that your objectives should be Specific, Measurable, Achievable, Relevant, and Time-bound.
• Revise and refine: Once you have written your research objectives, revise and refine them to ensure that they are clear, concise, and achievable. Make sure that they align with your research questions and variables, and that they will help you answer your research problem.

Example of Research Objectives

Examples of research objectives Could be:

Research Objectives for the topic of “The Impact of Artificial Intelligence on Employment”:

• To investigate the effects of the adoption of AI on employment trends across various industries and occupations.
• To explore the potential for AI to create new job opportunities and transform existing roles in the workforce.
• To examine the social and economic implications of the widespread use of AI for employment, including issues such as income inequality and access to education and training.
• To identify the skills and competencies that will be required for individuals to thrive in an AI-driven workplace, and to explore the role of education and training in developing these skills.
• To evaluate the ethical and legal considerations surrounding the use of AI for employment, including issues such as bias, privacy, and the responsibility of employers and policymakers to protect workers’ rights.

When to Write Research Objectives

• At the beginning of a research project : Research objectives should be identified and written down before starting a research project. This helps to ensure that the project is focused and that data collection and analysis efforts are aligned with the intended purpose of the research.
• When refining research questions: Writing research objectives can help to clarify and refine research questions. Objectives provide a more concrete and specific framework for addressing research questions, which can improve the overall quality and direction of a research project.
• After conducting a literature review : Conducting a literature review can help to identify gaps in knowledge and areas that require further research. Writing research objectives can help to define and focus the research effort in these areas.
• When developing a research proposal: Research objectives are an important component of a research proposal. They help to articulate the purpose and scope of the research, and provide a clear and concise summary of the expected outcomes and contributions of the research.
• When seeking funding for research: Funding agencies often require a detailed description of research objectives as part of a funding proposal. Writing clear and specific research objectives can help to demonstrate the significance and potential impact of a research project, and increase the chances of securing funding.
• When designing a research study : Research objectives guide the design and implementation of a research study. They help to identify the appropriate research methods, sampling strategies, data collection and analysis techniques, and other relevant aspects of the study design.
• When communicating research findings: Research objectives provide a clear and concise summary of the main research questions and outcomes. They are often included in research reports and publications, and can help to ensure that the research findings are communicated effectively and accurately to a wide range of audiences.
• When evaluating research outcomes : Research objectives provide a basis for evaluating the success of a research project. They help to measure the degree to which research questions have been answered and the extent to which research outcomes have been achieved.
• When conducting research in a team : Writing research objectives can facilitate communication and collaboration within a research team. Objectives provide a shared understanding of the research purpose and goals, and can help to ensure that team members are working towards a common objective.

Purpose of Research Objectives

Some of the main purposes of research objectives include:

• To clarify the research question or problem : Research objectives help to define the specific aspects of the research question or problem that the study aims to address. This makes it easier to design a study that is focused and relevant.
• To guide the research design: Research objectives help to determine the research design, including the research methods, data collection techniques, and sampling strategy. This ensures that the study is structured and efficient.
• To measure progress : Research objectives provide a way to measure progress throughout the research process. They help the researcher to evaluate whether they are on track and meeting their goals.
• To communicate the research goals : Research objectives provide a clear and concise description of the research goals. This helps to communicate the purpose of the study to other researchers, stakeholders, and the general public.

Advantages of Research Objectives

Here are some advantages of having well-defined research objectives:

• Focus : Research objectives help to focus the research effort on specific areas of inquiry. By identifying clear research questions, the researcher can narrow down the scope of the study and avoid getting sidetracked by irrelevant information.
• Clarity : Clearly stated research objectives provide a roadmap for the research study. They provide a clear direction for the research, making it easier for the researcher to stay on track and achieve their goals.
• Measurability : Well-defined research objectives provide measurable outcomes that can be used to evaluate the success of the research project. This helps to ensure that the research is effective and that the research goals are achieved.
• Feasibility : Research objectives help to ensure that the research project is feasible. By clearly defining the research goals, the researcher can identify the resources required to achieve those goals and determine whether those resources are available.
• Relevance : Research objectives help to ensure that the research study is relevant and meaningful. By identifying specific research questions, the researcher can ensure that the study addresses important issues and contributes to the existing body of knowledge.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Assignment – Types, Examples and Writing Guide

References in Research – Types, Examples and...

Figures in Research Paper – Examples and Guide

Delimitations in Research – Types, Examples and...

Research Findings – Types Examples and Writing...

Research Paper – Structure, Examples and Writing...

Research Aims, Objectives & Questions

The “Golden Thread” Explained Simply (+ Examples)

By: David Phair (PhD) and Alexandra Shaeffer (PhD) | June 2022

The research aims , objectives and research questions (collectively called the “golden thread”) are arguably the most important thing you need to get right when you’re crafting a research proposal , dissertation or thesis . We receive questions almost every day about this “holy trinity” of research and there’s certainly a lot of confusion out there, so we’ve crafted this post to help you navigate your way through the fog.

Overview: The Golden Thread

• What is the golden thread
• What are research aims ( examples )
• What are research objectives ( examples )
• What are research questions ( examples )
• The importance of alignment in the golden thread

What is the “golden thread”?  

The golden thread simply refers to the collective research aims , research objectives , and research questions for any given project (i.e., a dissertation, thesis, or research paper). These three elements are bundled together because it’s extremely important that they align with each other, and that the entire research project aligns with them.

Importantly, the golden thread needs to weave its way through the entirety of any research project , from start to end. In other words, it needs to be very clearly defined right at the beginning of the project (the topic ideation and proposal stage) and it needs to inform almost every decision throughout the rest of the project. For example, your research design and methodology will be heavily influenced by the golden thread (we’ll explain this in more detail later), as well as your literature review.

The research aims, objectives and research questions (the golden thread) define the focus and scope ( the delimitations ) of your research project. In other words, they help ringfence your dissertation or thesis to a relatively narrow domain, so that you can “go deep” and really dig into a specific problem or opportunity. They also help keep you on track , as they act as a litmus test for relevance. In other words, if you’re ever unsure whether to include something in your document, simply ask yourself the question, “does this contribute toward my research aims, objectives or questions?”. If it doesn’t, chances are you can drop it.

Alright, enough of the fluffy, conceptual stuff. Let’s get down to business and look at what exactly the research aims, objectives and questions are and outline a few examples to bring these concepts to life.

Research Aims: What are they?

Simply put, the research aim(s) is a statement that reflects the broad overarching goal (s) of the research project. Research aims are fairly high-level (low resolution) as they outline the general direction of the research and what it’s trying to achieve .

Research Aims: Examples  

True to the name, research aims usually start with the wording “this research aims to…”, “this research seeks to…”, and so on. For example:

“This research aims to explore employee experiences of digital transformation in retail HR.”   “This study sets out to assess the interaction between student support and self-care on well-being in engineering graduate students”  

As you can see, these research aims provide a high-level description of what the study is about and what it seeks to achieve. They’re not hyper-specific or action-oriented, but they’re clear about what the study’s focus is and what is being investigated.

Need a helping hand?

Research Objectives: What are they?

The research objectives take the research aims and make them more practical and actionable . In other words, the research objectives showcase the steps that the researcher will take to achieve the research aims.

The research objectives need to be far more specific (higher resolution) and actionable than the research aims. In fact, it’s always a good idea to craft your research objectives using the “SMART” criteria. In other words, they should be specific, measurable, achievable, relevant and time-bound”.

Research Objectives: Examples  

Let’s look at two examples of research objectives. We’ll stick with the topic and research aims we mentioned previously.  

For the digital transformation topic:

To observe the retail HR employees throughout the digital transformation. To assess employee perceptions of digital transformation in retail HR. To identify the barriers and facilitators of digital transformation in retail HR.

And for the student wellness topic:

To determine whether student self-care predicts the well-being score of engineering graduate students. To determine whether student support predicts the well-being score of engineering students. To assess the interaction between student self-care and student support when predicting well-being in engineering graduate students.

  As you can see, these research objectives clearly align with the previously mentioned research aims and effectively translate the low-resolution aims into (comparatively) higher-resolution objectives and action points . They give the research project a clear focus and present something that resembles a research-based “to-do” list.

Research Questions: What are they?

Finally, we arrive at the all-important research questions. The research questions are, as the name suggests, the key questions that your study will seek to answer . Simply put, they are the core purpose of your dissertation, thesis, or research project. You’ll present them at the beginning of your document (either in the introduction chapter or literature review chapter) and you’ll answer them at the end of your document (typically in the discussion and conclusion chapters).  

The research questions will be the driving force throughout the research process. For example, in the literature review chapter, you’ll assess the relevance of any given resource based on whether it helps you move towards answering your research questions. Similarly, your methodology and research design will be heavily influenced by the nature of your research questions. For instance, research questions that are exploratory in nature will usually make use of a qualitative approach, whereas questions that relate to measurement or relationship testing will make use of a quantitative approach.  

Let’s look at some examples of research questions to make this more tangible.

Research Questions: Examples  

Again, we’ll stick with the research aims and research objectives we mentioned previously.  

For the digital transformation topic (which would be qualitative in nature):

How do employees perceive digital transformation in retail HR? What are the barriers and facilitators of digital transformation in retail HR?  

And for the student wellness topic (which would be quantitative in nature):

Does student self-care predict the well-being scores of engineering graduate students? Does student support predict the well-being scores of engineering students? Do student self-care and student support interact when predicting well-being in engineering graduate students?  

You’ll probably notice that there’s quite a formulaic approach to this. In other words, the research questions are basically the research objectives “converted” into question format. While that is true most of the time, it’s not always the case. For example, the first research objective for the digital transformation topic was more or less a step on the path toward the other objectives, and as such, it didn’t warrant its own research question.  

So, don’t rush your research questions and sloppily reword your objectives as questions. Carefully think about what exactly you’re trying to achieve (i.e. your research aim) and the objectives you’ve set out, then craft a set of well-aligned research questions . Also, keep in mind that this can be a somewhat iterative process , where you go back and tweak research objectives and aims to ensure tight alignment throughout the golden thread.

The importance of strong alignment 

Alignment is the keyword here and we have to stress its importance . Simply put, you need to make sure that there is a very tight alignment between all three pieces of the golden thread. If your research aims and research questions don’t align, for example, your project will be pulling in different directions and will lack focus . This is a common problem students face and can cause many headaches (and tears), so be warned.

Take the time to carefully craft your research aims, objectives and research questions before you run off down the research path. Ideally, get your research supervisor/advisor to review and comment on your golden thread before you invest significant time into your project, and certainly before you start collecting data .  

Recap: The golden thread

In this post, we unpacked the golden thread of research, consisting of the research aims , research objectives and research questions . You can jump back to any section using the links below.

As always, feel free to leave a comment below – we always love to hear from you. Also, if you’re interested in 1-on-1 support, take a look at our private coaching service here.

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

28 Comments

Thank you very much for your great effort put. As an Undergraduate taking Demographic Research & Methodology, I’ve been trying so hard to understand clearly what is a Research Question, Research Aim and the Objectives in a research and the relationship between them etc. But as for now I’m thankful that you’ve solved my problem.

Well appreciated. This has helped me greatly in doing my dissertation.

An so delighted with this wonderful information thank you a lot.

so impressive i have benefited a lot looking forward to learn more on research.

I am very happy to have carefully gone through this well researched article.

Infact,I used to be phobia about anything research, because of my poor understanding of the concepts.

Now,I get to know that my research question is the same as my research objective(s) rephrased in question format.

I please I would need a follow up on the subject,as I intends to join the team of researchers. Thanks once again.

Thanks so much. This was really helpful.

i found this document so useful towards my study in research methods. thanks so much.

This is my 2nd read topic in your course and I should commend the simplified explanations of each part. I’m beginning to understand and absorb the use of each part of a dissertation/thesis. I’ll keep on reading your free course and might be able to avail the training course! Kudos!

Thank you! Better put that my lecture and helped to easily understand the basics which I feel often get brushed over when beginning dissertation work.

This is quite helpful. I like how the Golden thread has been explained and the needed alignment.

This is quite helpful. I really appreciate!

The article made it simple for researcher students to differentiate between three concepts.

Very innovative and educational in approach to conducting research.

A very helpful piece. thanks, I really appreciate it .

Very well explained, and it might be helpful to many people like me.

Wish i had found this (and other) resource(s) at the beginning of my PhD journey… not in my writing up year… 😩 Anyways… just a quick question as i’m having some issues ordering my “golden thread”…. does it matter in what order you mention them? i.e., is it always first aims, then objectives, and finally the questions? or can you first mention the research questions and then the aims and objectives?

Thank you for a very simple explanation that builds upon the concepts in a very logical manner. Just prior to this, I read the research hypothesis article, which was equally very good. This met my primary objective.

My secondary objective was to understand the difference between research questions and research hypothesis, and in which context to use which one. However, I am still not clear on this. Can you kindly please guide?

In research, a research question is a clear and specific inquiry that the researcher wants to answer, while a research hypothesis is a tentative statement or prediction about the relationship between variables or the expected outcome of the study. Research questions are broader and guide the overall study, while hypotheses are specific and testable statements used in quantitative research. Research questions identify the problem, while hypotheses provide a focus for testing in the study.

Exactly what I need in this research journey, I look forward to more of your coaching videos.

This helped a lot. Thanks so much for the effort put into explaining it.

What data source in writing dissertation/Thesis requires?

What is data source covers when writing dessertation/thesis

This is quite useful thanks

I’m excited and thankful. I got so much value which will help me progress in my thesis.

where are the locations of the reserch statement, research objective and research question in a reserach paper? Can you write an ouline that defines their places in the researh paper?

Thank you so much for making research aim, research objectives and research question so clear. This will be helpful to me as i continue with my thesis.

Thanks much for this content. I learned a lot. And I am inspired to learn more. I am still struggling with my preparation for dissertation outline/proposal. But I consistently follow contents and tutorials and the new FB of GRAD Coach. Hope to really become confident in writing my dissertation and successfully defend it.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Writing the Research Objectives: 5 Straightforward Examples

The research objective of a research proposal or scientific article defines the direction or content of a research investigation. Without the research objectives, the proposal or research paper is in disarray. It is like a fisherman riding on a boat without any purpose and with no destination in sight. Therefore, at the beginning of any research venture, the researcher must be clear about what he or she intends to do or achieve in conducting a study.

How do you define the objectives of a study? What are the uses of the research objective? How would a researcher write this essential part of the research? This article aims to provide answers to these questions.

Definition of a Research Objective

A research objective describes, in a few words, the result of the research project after its implementation. It answers the question,

“ What does the researcher want or hope to achieve at the end of the research project.”  

The research objective provides direction to the performance of the study.

What are the Uses of the Research Objective?

The uses of the research objective are enumerated below:

• serves as the researcher’s guide in identifying the appropriate research design,
• identifies the variables of the study, and
• specifies the data collection procedure and the corresponding analysis for the data generated.

The research design serves as the “blueprint” for the research investigation. The University of Southern California describes the different types of research design extensively. It details the data to be gathered, data collection procedure, data measurement, and statistical tests to use in the analysis.

The variables of the study include those factors that the researcher wants to evaluate in the study. These variables narrow down the research to several manageable components to see differences or correlations between them.

Specifying the data collection procedure ensures data accuracy and integrity . Thus, the probability of error is minimized. Generalizations or conclusions based on valid arguments founded on reliable data strengthens research findings on particular issues and problems.

In data mining activities where large data sets are involved, the research objective plays a crucial role. Without a clear objective to guide the machine learning process, the desired outcomes will not be met.

How is the Research Objective Written?

A research objective must be achievable, i.e., it must be framed keeping in mind the available time, infrastructure required for research, and other resources.

Before forming a research objective, you should read about all the developments in your area of research and find gaps in knowledge that need to be addressed. Readings will help you come up with suitable objectives for your research project.

5 Examples of Research Objectives

The following examples of research objectives based on several published studies on various topics demonstrate how the research objectives are written:

• This study aims to find out if there is a difference in quiz scores between students exposed to direct instruction and flipped classrooms (Webb and Doman, 2016).
• This study seeks to examine the extent, range, and method of coral reef rehabilitation projects in five shallow reef areas adjacent to popular tourist destinations in the Philippines (Yeemin et al ., 2006).
• This study aims to investigate species richness of mammal communities in five protected areas over the past 20 years (Evans et al ., 2006).
• This study aims to clarify the demographic, epidemiological, clinical, and radiological features of 2019-nCoV patients with other causes of pneumonia (Zhao et al ., 2020).
• This research aims to assess species extinction risks for sample regions that cover some 20% of the Earth’s terrestrial surface.

Finally, writing the research objectives requires constant practice, experience, and knowledge about the topic investigated. Clearly written objectives save time, money, and effort.

Once you have a clear idea of your research objectives, you can now develop your conceptual framework which is a crucial element of your research paper as it guides the flow of your research. The conceptual framework will help you develop your methodology and statistical tests.

I wrote a detailed, step-by-step guide on how to develop a conceptual framework with illustration in my post titled “ Conceptual Framework: A Step by Step Guide on How to Make One. “

Evans, K. L., Rodrigues, A. S., Chown, S. L., & Gaston, K. J. (2006). Protected areas and regional avian species richness in South Africa.  Biology letters ,  2 (2), 184-188.

Thomas, C. D., Cameron, A., Green, R. E., Bakkenes, M., Beaumont, L. J., Collingham, Y. C., … & Hughes, L. (2004). Extinction risk from climate change. Nature, 427(6970), 145-148.

Webb, M., & Doman, E. (2016). Does the Flipped Classroom Lead to Increased Gains on Learning Outcomes in ESL/EFL Contexts?. CATESOL Journal, 28(1), 39-67.

Yeemin, T., Sutthacheep, M., & Pettongma, R. (2006). Coral reef restoration projects in Thailand.  Ocean & Coastal Management ,  49 (9-10), 562-575.

Zhao, D., Yao, F., Wang, L., Zheng, L., Gao, Y., Ye, J., Guo, F., Zhao, H. & Gao, R. (2020). A comparative study on the clinical features of COVID-19 pneumonia to other pneumonias, Clinical Infectious Diseases , ciaa247, https://doi.org/10.1093/cid/ciaa247

© 2020 March 23 P. A. Regoniel Updated 17 November 2020

Related Posts

Infodemic During the Pandemic: 10 Ways to Discern Fact from Fake News

A Case Analysis of Nell’s Language Acquisition

Science blogs: a way to disseminate scientific information, about the author, patrick regoniel.

Dr. Regoniel is a multipotentialite whose skills, knowledge, and interests span many areas of life.

This is excellent

thank you for clarification

Leave a Reply Cancel Reply

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

How to Write Research Objectives

3-minute read

• 22nd November 2021

Writing a research paper, thesis, or dissertation ? If so, you’ll want to state your research objectives in the introduction of your paper to make it clear to your readers what you’re trying to accomplish. But how do you write effective research objectives? In this post, we’ll look at two key topics to help you do this:

• How to use your research aims as a basis for developing objectives.
• How to use SMART criteria to refine your research objectives.

For more advice on how to write strong research objectives, see below.

Research Aims and Objectives

There is an important difference between research aims and research objectives:

• A research aim defines the main purpose of your research. As such, you can think of your research aim as answering the question “What are you doing?”
• Research objectives (as most studies will have more than one) are the steps you will take to fulfil your aims. As such, your objectives should answer the question “How are you conducting your research?”

For instance, an example research aim could be:

This study will investigate the link between dehydration and the incidence of urinary tract infections (UTIs) in intensive care patients in Australia.

To develop a set of research objectives, you would then break down the various steps involved in meeting said aim. For example:

This study will investigate the link between dehydration and the incidence of urinary tract infections (UTIs) in intensive care patients in Australia. To achieve this, the study objectives w ill include:

• Replicat ing a small Singaporean study into the role of dehydration in UTIs in hospital patients (Sepe, 2018) in a larger Australian cohort.
• Trialing the use of intravenous fluids for intensive care patients to prevent dehydration.
• Assessing the relationship between the age of patients and quantities of intravenous fluids needed to counter dehydration.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

Note that the objectives don’t go into any great detail here. The key is to briefly summarize each component of your study. You can save details for how you will conduct the research for the methodology section of your paper.

Make Your Research Objectives SMART

A great way to refine your research objectives is to use SMART criteria . Borrowed from the world of project management, there are many versions of this system. However, we’re going to focus on developing specific, measurable, achievable, relevant, and timebound objectives.

In other words, a good research objective should be all of the following:

• S pecific – Is the objective clear and well-defined?
• M easurable – How will you know when the objective has been achieved? Is there a way to measure the thing you’re seeking to do?
• A chievable – Do you have the support and resources necessary to undertake this action? Are you being overly ambitious with this objective?
• R elevant – Is this objective vital for fulfilling your research aim?
• T imebound – Can this action be realistically undertaken in the time you have?

If you follow this system, your research objectives will be much stronger.

Expert Research Proofreading

Whatever your research aims and objectives, make sure to have your academic writing proofread by the experts!

Our academic editors can help you with research papers and proposals , as well as any other scholarly document you need checking. And this will help to ensure that your academic writing is always clear, concise, and precise.

Submit a free sample document today to trial our services and find out more.

Share this article:

Post A New Comment

Get help from a language expert. Try our proofreading services for free.

Scientific notation: definition and examples.

In the world of science, numbers reign supreme. Metrics, data, and statistics are used to...

What Is a Monograph?

A monograph is a comprehensive piece of writing that provides an intensive in-depth analysis of...

Book Proofreading Services: How to Ensure Your Self-Published Book Is Error-Free

Self-publishing is becoming increasingly popular with authors, but that also means that there’s increasing competition....

An Introduction to Interrogative Pronouns – With Examples

Pronouns are versatile words that help simplify language. To avoid repetition, we use them instead...

Why Every Blogger Should Use Proofreading Services

If you write and publish your own blog, then you know that accuracy and presentation...

How to Write a Business Plan

A business plan is a document that outlines the objectives of a particular business and...

Make sure your writing is the best it can be with our expert English proofreading and editing.

Thesis and Purpose Statements

Use the guidelines below to learn the differences between thesis and purpose statements.

In the first stages of writing, thesis or purpose statements are usually rough or ill-formed and are useful primarily as planning tools.

A thesis statement or purpose statement will emerge as you think and write about a topic. The statement can be restricted or clarified and eventually worked into an introduction.

As you revise your paper, try to phrase your thesis or purpose statement in a precise way so that it matches the content and organization of your paper.

Thesis statements

A thesis statement is a sentence that makes an assertion about a topic and predicts how the topic will be developed. It does not simply announce a topic: it says something about the topic.

Good: X has made a significant impact on the teenage population due to its . . . Bad: In this paper, I will discuss X.

A thesis statement makes a promise to the reader about the scope, purpose, and direction of the paper. It summarizes the conclusions that the writer has reached about the topic.

A thesis statement is generally located near the end of the introduction. Sometimes in a long paper, the thesis will be expressed in several sentences or an entire paragraph.

A thesis statement is focused and specific enough to be proven within the boundaries of the paper. Key words (nouns and verbs) should be specific, accurate, and indicative of the range of research, thrust of the argument or analysis, and the organization of supporting information.

Purpose statements

A purpose statement announces the purpose, scope, and direction of the paper. It tells the reader what to expect in a paper and what the specific focus will be.

Common beginnings include:

“This paper examines . . .,” “The aim of this paper is to . . .,” and “The purpose of this essay is to . . .”

A purpose statement makes a promise to the reader about the development of the argument but does not preview the particular conclusions that the writer has drawn.

A purpose statement usually appears toward the end of the introduction. The purpose statement may be expressed in several sentences or even an entire paragraph.

A purpose statement is specific enough to satisfy the requirements of the assignment. Purpose statements are common in research papers in some academic disciplines, while in other disciplines they are considered too blunt or direct. If you are unsure about using a purpose statement, ask your instructor.

This paper will examine the ecological destruction of the Sahel preceding the drought and the causes of this disintegration of the land. The focus will be on the economic, political, and social relationships which brought about the environmental problems in the Sahel.

Sample purpose and thesis statements

The following example combines a purpose statement and a thesis statement (bold).

The goal of this paper is to examine the effects of Chile’s agrarian reform on the lives of rural peasants. The nature of the topic dictates the use of both a chronological and a comparative analysis of peasant lives at various points during the reform period. . . The Chilean reform example provides evidence that land distribution is an essential component of both the improvement of peasant conditions and the development of a democratic society. More extensive and enduring reforms would likely have allowed Chile the opportunity to further expand these horizons.

For more tips about writing thesis statements, take a look at our new handout on Developing a Thesis Statement.

Writing Process and Structure

This is an accordion element with a series of buttons that open and close related content panels.

Getting Started with Your Paper

Interpreting Writing Assignments from Your Courses

Generating Ideas for Your Paper

Creating an Argument

Thesis vs. Purpose Statements

Developing a Thesis Statement

Architecture of Arguments

Working with Sources

Quoting and Paraphrasing Sources

Using Literary Quotations

Citing Sources in Your Paper

Drafting Your Paper

Introductions

Paragraphing

Developing Strategic Transitions

Conclusions

Revising Your Paper

Peer Reviews

Reverse Outlines

Revising an Argumentative Paper

Revision Strategies for Longer Projects

Finishing Your Paper

Twelve Common Errors: An Editing Checklist

How to Proofread your Paper

Writing Collaboratively

Collaborative and Group Writing

• Chapter 1: Home
• Narrowing Your Topic
• Problem Statement

Purpose Statement Overview

Best practices for writing your purpose statement, writing your purpose statement, sample purpose statements.

• Student Experience Feedback Buttons
• Conceptual Framework
• Theoretical Framework
• Quantitative Research Questions This link opens in a new window
• Qualitative Research Questions This link opens in a new window
• Qualitative & Quantitative Research Support with the ASC This link opens in a new window
• Library Research Consultations This link opens in a new window

Jump to DSE Guide

The purpose statement succinctly explains (on no more than 1 page) the objectives of the research study. These objectives must directly address the problem and help close the stated gap. Expressed as a formula:

Good purpose statements:

• Flow from the problem statement and actually address the proposed problem
• Are concise and clear
• Answer the question ‘Why are you doing this research?’
• Match the methodology (similar to research questions)
• Have a ‘hook’ to get the reader’s attention
• Set the stage by clearly stating, “The purpose of this (qualitative or quantitative) study is to ...

In PhD studies, the purpose usually involves applying a theory to solve the problem. In other words, the purpose tells the reader what the goal of the study is, and what your study will accomplish, through which theoretical lens. The purpose statement also includes brief information about direction, scope, and where the data will come from.

A problem and gap in combination can lead to different research objectives, and hence, different purpose statements. In the example from above where the problem was severe underrepresentation of female CEOs in Fortune 500 companies and the identified gap related to lack of research of male-dominated boards; one purpose might be to explore implicit biases in male-dominated boards through the lens of feminist theory. Another purpose may be to determine how board members rated female and male candidates on scales of competency, professionalism, and experience to predict which candidate will be selected for the CEO position. The first purpose may involve a qualitative ethnographic study in which the researcher observes board meetings and hiring interviews; the second may involve a quantitative regression analysis. The outcomes will be very different, so it’s important that you find out exactly how you want to address a problem and help close a gap!

The purpose of the study must not only align with the problem and address a gap; it must also align with the chosen research method. In fact, the DP/DM template requires you to name the  research method at the very beginning of the purpose statement. The research verb must match the chosen method. In general, quantitative studies involve “closed-ended” research verbs such as determine , measure , correlate , explain , compare , validate , identify , or examine ; whereas qualitative studies involve “open-ended” research verbs such as explore , understand , narrate , articulate [meanings], discover , or develop .

A qualitative purpose statement following the color-coded problem statement (assumed here to be low well-being among financial sector employees) + gap (lack of research on followers of mid-level managers), might start like this:

In response to declining levels of employee well-being, the purpose of the qualitative phenomenology was to explore and understand the lived experiences related to the well-being of the followers of novice mid-level managers in the financial services industry. The levels of follower well-being have been shown to correlate to employee morale, turnover intention, and customer orientation (Eren et al., 2013). A combined framework of Leader-Member Exchange (LMX) Theory and the employee well-being concept informed the research questions and supported the inquiry, analysis, and interpretation of the experiences of followers of novice managers in the financial services industry.

A quantitative purpose statement for the same problem and gap might start like this:

In response to declining levels of employee well-being, the purpose of the quantitative correlational study was to determine which leadership factors predict employee well-being of the followers of novice mid-level managers in the financial services industry. Leadership factors were measured by the Leader-Member Exchange (LMX) assessment framework  by Mantlekow (2015), and employee well-being was conceptualized as a compound variable consisting of self-reported turnover-intent and psychological test scores from the Mental Health Survey (MHS) developed by Johns Hopkins University researchers.

Both of these purpose statements reflect viable research strategies and both align with the problem and gap so it’s up to the researcher to design a study in a manner that reflects personal preferences and desired study outcomes. Note that the quantitative research purpose incorporates operationalized concepts  or variables ; that reflect the way the researcher intends to measure the key concepts under study; whereas the qualitative purpose statement isn’t about translating the concepts under study as variables but instead aim to explore and understand the core research phenomenon.  

Always keep in mind that the dissertation process is iterative, and your writing, over time, will be refined as clarity is gradually achieved. Most of the time, greater clarity for the purpose statement and other components of the Dissertation is the result of a growing understanding of the literature in the field. As you increasingly master the literature you will also increasingly clarify the purpose of your study.

The purpose statement should flow directly from the problem statement. There should be clear and obvious alignment between the two and that alignment will get tighter and more pronounced as your work progresses.

The purpose statement should specifically address the reason for conducting the study, with emphasis on the word specifically. There should not be any doubt in your readers’ minds as to the purpose of your study. To achieve this level of clarity you will need to also insure there is no doubt in your mind as to the purpose of your study.

Many researchers benefit from stopping your work during the research process when insight strikes you and write about it while it is still fresh in your mind. This can help you clarify all aspects of a dissertation, including clarifying its purpose.

Your Chair and your committee members can help you to clarify your study’s purpose so carefully attend to any feedback they offer.

The purpose statement should reflect the research questions and vice versa. The chain of alignment that began with the research problem description and continues on to the research purpose, research questions, and methodology must be respected at all times during dissertation development. You are to succinctly describe the overarching goal of the study that reflects the research questions. Each research question narrows and focuses the purpose statement. Conversely, the purpose statement encompasses all of the research questions.

Identify in the purpose statement the research method as quantitative, qualitative or mixed (i.e., “The purpose of this [qualitative/quantitative/mixed] study is to ...)

Avoid the use of the phrase “research study” since the two words together are redundant.

Follow the initial declaration of purpose with a brief overview of how, with what instruments/data, with whom and where (as applicable) the study will be conducted. Identify variables/constructs and/or phenomenon/concept/idea. Since this section is to be a concise paragraph, emphasis must be placed on the word brief. However, adding these details will give your readers a very clear picture of the purpose of your research.

Developing the purpose section of your dissertation is usually not achieved in a single flash of insight. The process involves a great deal of reading to find out what other scholars have done to address the research topic and problem you have identified. The purpose section of your dissertation could well be the most important paragraph you write during your academic career, and every word should be carefully selected. Think of it as the DNA of your dissertation. Everything else you write should emerge directly and clearly from your purpose statement. In turn, your purpose statement should emerge directly and clearly from your research problem description. It is good practice to print out your problem statement and purpose statement and keep them in front of you as you work on each part of your dissertation in order to insure alignment.

It is helpful to collect several dissertations similar to the one you envision creating. Extract the problem descriptions and purpose statements of other dissertation authors and compare them in order to sharpen your thinking about your own work.  Comparing how other dissertation authors have handled the many challenges you are facing can be an invaluable exercise. Keep in mind that individual universities use their own tailored protocols for presenting key components of the dissertation so your review of these purpose statements should focus on content rather than form.

Once your purpose statement is set it must be consistently presented throughout the dissertation. This may require some recursive editing because the way you articulate your purpose may evolve as you work on various aspects of your dissertation. Whenever you make an adjustment to your purpose statement you should carefully follow up on the editing and conceptual ramifications throughout the entire document.

In establishing your purpose you should NOT advocate for a particular outcome. Research should be done to answer questions not prove a point. As a researcher, you are to inquire with an open mind, and even when you come to the work with clear assumptions, your job is to prove the validity of the conclusions reached. For example, you would not say the purpose of your research project is to demonstrate that there is a relationship between two variables. Such a statement presupposes you know the answer before your research is conducted and promotes or supports (advocates on behalf of) a particular outcome. A more appropriate purpose statement would be to examine or explore the relationship between two variables.

Your purpose statement should not imply that you are going to prove something. You may be surprised to learn that we cannot prove anything in scholarly research for two reasons. First, in quantitative analyses, statistical tests calculate the probability that something is true rather than establishing it as true. Second, in qualitative research, the study can only purport to describe what is occurring from the perspective of the participants. Whether or not the phenomenon they are describing is true in a larger context is not knowable. We cannot observe the phenomenon in all settings and in all circumstances.

It is important to distinguish in your mind the differences between the Problem Statement and Purpose Statement.

The Problem Statement is why I am doing the research

The Purpose Statement is what type of research I am doing to fit or address the problem

The Purpose Statement includes:

• Method of Study
• Specific Population

Remember, as you are contemplating what to include in your purpose statement and then when you are writing it, the purpose statement is a concise paragraph that describes the intent of the study, and it should flow directly from the problem statement.  It should specifically address the reason for conducting the study, and reflect the research questions.  Further, it should identify the research method as qualitative, quantitative, or mixed.  Then provide a brief overview of how the study will be conducted, with what instruments/data collection methods, and with whom (subjects) and where (as applicable). Finally, you should identify variables/constructs and/or phenomenon/concept/idea.

Qualitative Purpose Statement

Creswell (2002) suggested for writing purpose statements in qualitative research include using deliberate phrasing to alert the reader to the purpose statement. Verbs that indicate what will take place in the research and the use of non-directional language that do not suggest an outcome are key. A purpose statement should focus on a single idea or concept, with a broad definition of the idea or concept. How the concept was investigated should also be included, as well as participants in the study and locations for the research to give the reader a sense of with whom and where the study took place. 

Creswell (2003) advised the following script for purpose statements in qualitative research:

“The purpose of this qualitative_________________ (strategy of inquiry, such as ethnography, case study, or other type) study is (was? will be?) to ________________ (understand? describe? develop? discover?) the _________________(central phenomenon being studied) for ______________ (the participants, such as the individual, groups, organization) at __________(research site). At this stage in the research, the __________ (central phenomenon being studied) will be generally defined as ___________________ (provide a general definition)” (pg. 90).

Quantitative Purpose Statement

Creswell (2003) offers vast differences between the purpose statements written for qualitative research and those written for quantitative research, particularly with respect to language and the inclusion of variables. The comparison of variables is often a focus of quantitative research, with the variables distinguishable by either the temporal order or how they are measured. As with qualitative research purpose statements, Creswell (2003) recommends the use of deliberate language to alert the reader to the purpose of the study, but quantitative purpose statements also include the theory or conceptual framework guiding the study and the variables that are being studied and how they are related. 

Creswell (2003) suggests the following script for drafting purpose statements in quantitative research:

“The purpose of this _____________________ (experiment? survey?) study is (was? will be?) to test the theory of _________________that _________________ (compares? relates?) the ___________(independent variable) to _________________________(dependent variable), controlling for _______________________ (control variables) for ___________________ (participants) at _________________________ (the research site). The independent variable(s) _____________________ will be generally defined as _______________________ (provide a general definition). The dependent variable(s) will be generally defined as _____________________ (provide a general definition), and the control and intervening variables(s), _________________ (identify the control and intervening variables) will be statistically controlled in this study” (pg. 97).

• The purpose of this qualitative study was to determine how participation in service-learning in an alternative school impacted students academically, civically, and personally.  There is ample evidence demonstrating the failure of schools for students at-risk; however, there is still a need to demonstrate why these students are successful in non-traditional educational programs like the service-learning model used at TDS.  This study was unique in that it examined one alternative school’s approach to service-learning in a setting where students not only serve, but faculty serve as volunteer teachers.  The use of a constructivist approach in service-learning in an alternative school setting was examined in an effort to determine whether service-learning participation contributes positively to academic, personal, and civic gain for students, and to examine student and teacher views regarding the overall outcomes of service-learning.  This study was completed using an ethnographic approach that included observations, content analysis, and interviews with teachers at The David School.
• The purpose of this quantitative non-experimental cross-sectional linear multiple regression design was to investigate the relationship among early childhood teachers’ self-reported assessment of multicultural awareness as measured by responses from the Teacher Multicultural Attitude Survey (TMAS) and supervisors’ observed assessment of teachers’ multicultural competency skills as measured by the Multicultural Teaching Competency Scale (MTCS) survey. Demographic data such as number of multicultural training hours, years teaching in Dubai, curriculum program at current school, and age were also examined and their relationship to multicultural teaching competency. The study took place in the emirate of Dubai where there were 14,333 expatriate teachers employed in private schools (KHDA, 2013b).
• The purpose of this quantitative, non-experimental study is to examine the degree to which stages of change, gender, acculturation level and trauma types predicts the reluctance of Arab refugees, aged 18 and over, in the Dearborn, MI area, to seek professional help for their mental health needs. This study will utilize four instruments to measure these variables: University of Rhode Island Change Assessment (URICA: DiClemente & Hughes, 1990); Cumulative Trauma Scale (Kira, 2012); Acculturation Rating Scale for Arabic Americans-II Arabic and English (ARSAA-IIA, ARSAA-IIE: Jadalla & Lee, 2013), and a demographic survey. This study will examine 1) the relationship between stages of change, gender, acculturation levels, and trauma types and Arab refugees’ help-seeking behavior, 2) the degree to which any of these variables can predict Arab refugee help-seeking behavior.  Additionally, the outcome of this study could provide researchers and clinicians with a stage-based model, TTM, for measuring Arab refugees’ help-seeking behavior and lay a foundation for how TTM can help target the clinical needs of Arab refugees. Lastly, this attempt to apply the TTM model to Arab refugees’ condition could lay the foundation for future research to investigate the application of TTM to clinical work among refugee populations.
• The purpose of this qualitative, phenomenological study is to describe the lived experiences of LLM for 10 EFL learners in rural Guatemala and to utilize that data to determine how it conforms to, or possibly challenges, current theoretical conceptions of LLM. In accordance with Morse’s (1994) suggestion that a phenomenological study should utilize at least six participants, this study utilized semi-structured interviews with 10 EFL learners to explore why and how they have experienced the motivation to learn English throughout their lives. The methodology of horizontalization was used to break the interview protocols into individual units of meaning before analyzing these units to extract the overarching themes (Moustakas, 1994). These themes were then interpreted into a detailed description of LLM as experienced by EFL students in this context. Finally, the resulting description was analyzed to discover how these learners’ lived experiences with LLM conformed with and/or diverged from current theories of LLM.
• The purpose of this qualitative, embedded, multiple case study was to examine how both parent-child attachment relationships are impacted by the quality of the paternal and maternal caregiver-child interactions that occur throughout a maternal deployment, within the context of dual-military couples. In order to examine this phenomenon, an embedded, multiple case study was conducted, utilizing an attachment systems metatheory perspective. The study included four dual-military couples who experienced a maternal deployment to Operation Iraqi Freedom (OIF) or Operation Enduring Freedom (OEF) when they had at least one child between 8 weeks-old to 5 years-old.  Each member of the couple participated in an individual, semi-structured interview with the researcher and completed the Parenting Relationship Questionnaire (PRQ). “The PRQ is designed to capture a parent’s perspective on the parent-child relationship” (Pearson, 2012, para. 1) and was used within the proposed study for this purpose. The PRQ was utilized to triangulate the data (Bekhet & Zauszniewski, 2012) as well as to provide some additional information on the parents’ perspective of the quality of the parent-child attachment relationship in regards to communication, discipline, parenting confidence, relationship satisfaction, and time spent together (Pearson, 2012). The researcher utilized the semi-structured interview to collect information regarding the parents' perspectives of the quality of their parental caregiver behaviors during the deployment cycle, the mother's parent-child interactions while deployed, the behavior of the child or children at time of reunification, and the strategies or behaviors the parents believe may have contributed to their child's behavior at the time of reunification. The results of this study may be utilized by the military, and by civilian providers, to develop proactive and preventive measures that both providers and parents can implement, to address any potential adverse effects on the parent-child attachment relationship, identified through the proposed study. The results of this study may also be utilized to further refine and understand the integration of attachment theory and systems theory, in both clinical and research settings, within the field of marriage and family therapy.

Was this resource helpful?

• << Previous: Problem Statement
• Next: Alignment >>
• Last Updated: Apr 24, 2023 1:37 PM
• URL: https://resources.nu.edu/c.php?g=1006886

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Velentgas P, Dreyer NA, Nourjah P, et al., editors. Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Jan.

Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide.

• Hardcopy Version at Agency for Healthcare Research and Quality

Chapter 1 Study Objectives and Questions

Scott R Smith , PhD.

The steps involved in the process of developing research questions and study objectives for conducting observational comparative effectiveness research (CER) are described in this chapter. It is important to begin with identifying decisions under consideration, determining who the decisionmakers and stakeholders in the specific area of research under study are, and understanding the context in which decisions are being made. Synthesizing the current knowledge base and identifying evidence gaps is the next important step in the process, followed by conceptualizing the research problem, which includes developing questions that address the gaps in existing evidence. Understanding the stage of knowledge that the study is designed to address will come from developing these initial questions. Identifying which questions are critical to reduce decisional uncertainty and minimize gaps in the current knowledge base is an important part of developing a successful framework. In particular, it is beneficial to look at what study populations, interventions, comparisons, outcomes, timeframe, and settings (PICOTS framework) are most important to decisionmakers in weighing the balance of harms and benefits of action. Some research questions are easier to operationalize than others, and study limitations should be recognized and accepted from an early stage. The level of new scientific evidence that is required by the decisionmaker to make a decision or to take action must be recognized. Lastly, the magnitude of effect must be specified. This can mean defining what is a clinically meaningful difference in the study endpoints from the perspective of the decisionmaker and/or defining what is a meaningful difference from the patient's perspective.

The foundation for designing a new research protocol is the study's objectives and the questions that will be investigated through its implementation. All aspects of study design and analysis are based on the objectives and questions articulated in a study's protocol. Consequently, it is exceedingly important that a study's objectives and questions be formulated meticulously and written precisely in order for the research to be successful in generating new knowledge that can be used to inform health care decisions and actions.

An important aspect of CER 1 and other forms of translational research is the potential for early involvement and inclusion of patients and other stakeholders to collaborate with researchers in identifying study objectives, key questions, major study endpoints, and the evidentiary standards that are needed to inform decisionmaking. The involvement of stakeholders in formulating the research questions increases the applicability of the study to the end-users and facilitates appropriate translation of the results into health care practice and use by patient communities. While stakeholders may be defined in multiple ways, for the purposes of this User's Guide , a broad definition will be used. Hence, stakeholders are defined as individuals or organizations that use scientific evidence for decisionmaking and therefore have an interest in the results of new research. Implicit in this definition of stakeholders is the importance for stakeholders to understand the scientific process, including considerations of bioethics and the limitations of research, particularly with regard to studies involving human subjects. Ideally, stakeholders also should express commitment to using objective scientific evidence to inform their decisionmaking and recognize that disregarding sound scientific methods often will undermine decisionmaking. For stakeholder organizations, it is also advantageous if the organization has well-established processes for transparently reviewing and incorporating research findings into decisions as well as organized channels for disseminating research results.

There are at least seven essential steps in the conceptualization and development of a research question or set of questions for an observational CER protocol. These steps are presented as a general framework in Table 1.1 below and elaborated upon in the subsequent sections of this chapter. The framework is based on the principle that researchers and stakeholders will work together to objectively lay out the research problems, research questions, study objectives, and key parameters for which scientific evidence is needed to inform decisionmaking or health care actions. The intent of this framework is to facilitate communication between researchers and stakeholders in conceptualizing the research problem and the design of a study (or a program of research involving a series of studies) in order to maximize the potential that new knowledge will be created from the research with results that can inform decisionmaking. To do this, research results must be relevant, applicable, unbiased and sufficient to meet the evidentiary threshold for decisionmaking or action by stakeholders. In order for the results to be valid and credible, all persons involved must be committed to protecting the integrity of the research from bias and conflicts of interest. Most importantly, the study must be designed to protect the rights, welfare, and well-being of subjects involved in the research.

Framework for developing and conceptualizing a CER protocol.

• Identifying Decisions, Decisionmakers, Actions, and Context

In order for research findings to be useful for decisionmaking, the study protocol should clearly articulate the decisions or actions for which stakeholders seek new scientific evidence. While only some studies may be sufficiently robust for making decisions or taking action, statements that describe the stakeholders' decisions will help those who read the protocol understand the rationale for the study and its potential for informing decisions or for translating the findings into changes in health care practices. This information also improves the ability of protocol readers to understand the purpose of the study so they can critically review its design and provide recommendations for ways it may be potentially improved. If stakeholders have a need to make decisions within a critical time frame for regulatory, ethical, or other reasons, this interval should be expressed to researchers and described in the protocol. In some cases, the time frame for decisionmaking may influence the choice of outcomes that can be studied and the study designs that can be used. For some stakeholders' questions, research and decisionmaking may need to be divided into stages, since it may take years for outcomes with long lag times to occur, and research findings will be delayed until they do.

In writing this section of the protocol, investigators should ask stakeholders to describe the context in which the decision will be made or actions will be taken. This context includes the background and rationale for the decision, key areas of uncertainty and controversies surrounding the decision, ways scientific evidence will be used to inform the decision, the process stakeholders will use to reach decisions based on scientific evidence, and a description of the key stakeholders who will use or potentially be affected by the decision. By explaining these contextual factors that surround the decision, investigators will be able to work with stakeholders to determine the study objectives and other major parameters of the study. This work also provides the opportunity to discuss how the tools of science can be applied to generate new evidence for informing stakeholder decisions and what limits may exist in those tools. In addition, this initial step begins to clarify the number of analyses necessary to generate the evidence that stakeholders need to make a decision or take other actions with sufficient certainty about the outcomes of interest. Finally, the contextual information facilitates advance planning and discussions by researchers and stakeholders about approaches to translation and implementation of the study findings once the research is completed.

• Synthesizing the Current Knowledge Base

In designing a new study, investigators should conduct a comprehensive review of the literature, critically appraise published studies, and synthesize what is known related to the research objectives. Specifically, investigators should summarize in the protocol what is known about the efficacy, effectiveness, and safety of the interventions and about the outcomes being studied. Furthermore, investigators should discuss measures used in prior research and whether these measures have changed over time. These descriptions will provide background on the knowledge base for the current protocol. It is equally important to identify which elements of the research problem are unknown because evidence is absent, insufficient, or conflicting.

For some research problems, systematic reviews of the literature may be available and can be useful resources to guide the study design. The AHRQ Evidence-based Practice Centers 2 and the Cochrane Collaboration 3 are examples of established programs that conduct thorough systematic reviews, technology assessments, and specialized comparative effectiveness reviews using standardized methods. When available, systematic reviews and technology assessments should be consulted as resources for investigators to assess the current knowledge base when designing new studies and working with stakeholders.

When reviewing the literature, investigators and stakeholders should identify the most relevant studies and guidelines about the interventions that will be studied. This will allow readers to understand how new research will add to the existing knowledge base. If guidelines are a source of information, then investigators should examine whether these guidelines have been updated to incorporate recent literature. In addition, investigators should assess the health sciences literature to determine what is known about expected effects of the interventions based on current understanding of the pathophysiology of the target condition. Furthermore, clinical experts should be consulted to help identify gaps in current knowledge based on their expertise and interactions with patients. Relevant questions to ask to assess the current knowledge base for development of an observational CER study protocol are:

• What are the most relevant studies and guidelines about the interventions, and why are these studies relevant to the protocol (e.g., because of the study findings, time period conducted, populations studied, etc.)?
• Are there differences in recommendations from clinical guidelines that would indicate clinical equipoise?
• What else is known about the expected effects of the interventions based on current understanding of the pathophysiology of the targeted condition?
• What do clinical experts say about gaps in current knowledge?
• Conceptualizing the Research Problem

In designing studies for addressing stakeholder questions, investigators should engage multiple stakeholders in discussions about how the research problem is conceptualized from the stakeholders' perspectives. These discussions will aid in designing a study that can be used to inform decisionmaking. Together, investigators and stakeholders should work collaboratively to determine the major objectives of the study based on the health care decisions facing stakeholders. As pointed out by Heckman, 4 research objectives should be formalized outside considerations of available data and the inferences that can be made from various statistical estimation approaches. Doing so will allow the study objectives to be determined by stakeholder needs rather than the availability of existing data. A thorough discussion of these considerations is beyond the scope of this chapter, but some important considerations are summarized in supplement 1 of this User's Guide.

In order to conceptualize the problem, stakeholders and other experts should be asked to describe the potential relationships between the intervention and important health outcomes. This description will help researchers develop preliminary hypotheses about the stated relationships. Likewise, stakeholders, researchers, and other experts should be asked to enumerate all major assumptions that affect the conceptualization of the research problem, but will not be directly examined in the study. These assumptions should be described in the study protocol and in reporting final study results. By clearly stating the assumptions, protocol reviewers will be better able to assess how the assumptions may influence the study results.

Based on the conceptualization of the research problem, investigators and stakeholders should make use of applicable scientific theory in designing the study protocol and developing the analytic plan. Research that is designed using a validated theory has a higher potential to reach valid conclusions and improve the overall understanding of a phenomenon. In addition, theory will aid in the interpretation of the study findings, since these results can be put in context with the theory and with past research. Depending on the nature of the inquiry, theory from specific disciplines such as health behavior, sociology, or biology could be the basis for designing the study. In addition, the research team should work with stakeholders to develop a conceptual model or framework to guide the implementation of the study. The protocol should also contain one or more figures that summarize the conceptual model or framework as it applies to the study. These figures will allow readers to understand the theoretical or conceptual basis for the study and how the theory is operationalized for the specific study. The figures should diagram relationships between study variables and outcomes to help readers of the protocol visualize relationships that will be examined in the study.

For research questions about causal associations between exposures and outcomes, causal models such as directed acyclic graphs (DAGs) may be useful tools in designing the conceptual framework for the study and developing the analytic plan. The value of DAGs in the context of refining study questions is that they make assumptions explicit in ways that can clarify gaps in knowledge. Free software such as DAGitty is available for creating, editing, and analyzing causal models. A thorough discussion of DAGs is beyond the scope of this chapter, but more information about DAGs is available in supplement 2 of this User's Guide.

The following list of questions may be useful for defining and describing a study's conceptual framework in a CER protocol:

• What are the main objectives of the study, as related to specific decisions to be made?
• What are the major assumptions of decisionmakers, investigators, and other experts about the problem or phenomenon being studied?
• What relationships, if any, do experts hypothesize exist between interventions and outcomes?

What is known about each element of the model?

Can relationships be expressed by causal diagrams?

• Determining the Stage of Knowledge Development for the Study Design

The scientific method is a process of observation and experimentation in order for the evidence base to be expanded as new knowledge is developed. Therefore, stakeholders and investigators should consider whether a program of research comprising a sequential or concurrent series of studies, rather than a single study, is needed to adequately make a decision. Staging the research into multiple studies and making interim decisions may improve the final decision and make judicious use of scarce research resources. In some cases, the results of preliminary studies, descriptive epidemiology, or pilot work may be helpful in making interim decisions and designing further research. Overall, a planned series of related studies or a program of research may be needed to adequately address stakeholders' decisions.

An example of a structured program of research is the four phases of clinical studies used by the Food and Drug Administration (FDA) to reach a decision about whether or not a new drug is safe and efficacious for market approval in the United States. Using this analogy, the final decision about whether a drug is efficacious and safe to be marketed for specific medical indications is based upon the accumulation of scientific evidence from a series of studies (i.e., not from any individual study), which are conducted in multiple sequential phases. The evidence generated in each phase is reviewed to make interim decisions about the safety and efficacy of a new pharmaceutical until ultimately all the evidence is reviewed to make a final decision about drug approval.

Under the FDA model for decisionmaking, initial research involves laboratory and animal tests. If the evidence generated in these studies indicates that the drug is active and not toxic, the sponsor submits an application to the FDA for an “investigational new drug.” If the FDA approves, human testing for safety and efficacy can begin. The first phase of human testing is usually conducted in a limited number of healthy volunteers (phase 1). If these trials show evidence that the product is safe in healthy volunteers, then the drug is further studied in a small number of volunteers who have the targeted condition (phase 2). If phase 2 studies show that the drug has a therapeutic effect and lacks significant adverse effects, trials with large numbers of people are conducted to determine the drug's safety and efficacy (phase 3). Following these trials, all relevant scientific studies are submitted to the FDA for a decision about whether the drug should be approved for marketing. If there are additional considerations like special safety issues, observational studies may be required to assess the safety of the drug in routine clinical care after the drug is approved for marketing (phase 4). Overall, the decisionmaking and research are staged so that the cumulative findings from all studies are used by the FDA to make interim decisions until the final decision is made about whether a medical product will be approved for marketing.

While most decisions about the comparative effectiveness of interventions will not need such extensive testing, it still may be prudent to stage research in a way that allows for interim decisions and sequentially more rigorous studies. On the other hand conditional approval or interim decisions may risk confusing patients and other stakeholders about the extent to which current evidence indicates that a treatment is effective and safe for all individuals with a health condition. For instance, under this staged approach new treatments could rapidly diffuse into a market even when there is limited evidence of long-term effectiveness and safety for all potential users. An illustrative example of this is the case of lung-volume reduction surgery, which was increasingly being used to treat severe emphysema despite limited evidence supporting its safety and efficacy until new research raised questions about the safety of the procedure. 6

Below is one potential categorization for the stages of knowledge development as related to informing decisions about questions of comparative effectiveness:

• Descriptive analysis
• Hypothesis generation
• Feasibility studies/proof of concept
• Hypothesis supporting
• Hypothesis testing

The first stages (i.e., descriptive analysis, hypothesis generation, and feasibility studies) are not mutually exclusive and usually are not intended to provide conclusive results for most decisions. Instead these stages provide preliminary evidence or feasibility testing before larger, more resource-intensive studies are launched. Results from these categories of studies may allow for interim decisionmaking (e.g., conditional approval for reimbursement of a treatment while further research is conducted). While a phased approach to research may postpone the time when a conclusive decision can be reached it does help to conserve resources such as those that may be consumed in launching a large multicenter study when a smaller study may be sufficient. Investigators will need to engage stakeholders to prioritize what stage of research may be most useful for the practical range of decisions that will be made.

Investigators should discuss in the protocol what stage of knowledge the current study will fulfill in light of the actions available to different stakeholders. This will allow reviewers of the protocol to assess the degree to which the evidence generated in the study holds the potential to fill specific knowledge gaps. For studies that are described in the protocol as preliminary, this may also help readers understand other tradeoffs that were made in the design of the study, in terms of methodological limitations that were accepted a priori in order to gather preliminary information about the research questions.

• Defining and Refining Study Questions Using PICOTS Framework

As recommended in other AHRQ methods guides, 7 investigators should engage stakeholders in a dialogue in order to understand the objectives of the research in practical terms, particularly so that investigators know the types of decisions that the research may affect. In working with stakeholders to develop research questions that can be studied with scientific methods, investigators may ask stakeholders to identify six key components of the research questions that will form the basis for designing the study. These components are reflected in the PICOTS typology and are shown below in Table 1.2 . These components represent the critical elements that will help investigators design a study that will be able to address the stakeholders' needs. Additional references that expand upon how to frame research questions can be found in the literature. 8 - 9

PICOTS typology for developing research questions.

The PICOTS typology outlines the key parts of the research questions that the study will be designed to address. 10 As a new research protocol is developed these questions can be presented in preliminary form and refined as other steps in the process are implemented. After the preliminary questions are refined, investigators should examine the questions to make sure that they will meet the needs of the stakeholders. In addition, they should assess whether the questions can be answered within the timeframe allotted and with the resources that are available for the study.

Since stakeholders ultimately determine effectiveness, it is important for investigators to ensure that the study endpoints and outcomes will meet their needs. Stakeholders need to articulate to investigators the health outcomes that are most important for a particular stakeholder to make decisions about treatment or take other health care actions. The endpoints that stakeholders will use to determine effectiveness may vary considerably. Unlike efficacy trials, in which clinical endpoints and surrogate measures are frequently used to determine efficacy, effectiveness may need to be determined based on several measures, many of which are not biological. These endpoints may be categorized as clinical endpoints, patient-reported outcomes and quality of life, health resource utilization, and utility measures. Types of measures that could be used are mortality, morbidity and adverse effects, quality of life, costs, or multiple outcomes. Chapter 6 gives a more extensive discussion of potential outcome measures of effectiveness.

The reliability, validity, and accuracy of study instruments to validly measure the concepts they purport to measure will also need to be acceptable to stakeholders. For instance, if stakeholders are interested in quality of life as an outcome, but do not believe there is an adequate measure of quality of life, then measurement development may need to be done prior to study initiation or other measures will need to be identified by stakeholders.

• Discussing Evidentiary Need and Uncertainty

Investigators and stakeholders should discuss the tradeoffs of different study designs that may be used for addressing the research questions. This dialogue will help researchers design a study that will be relevant and useful to the needs of stakeholders. All study designs have strengths and weaknesses, the latter of which may limit the conclusiveness of the final study results. Likewise, some decisions may require evidence that cannot be obtained from certain designs. In addition to design weaknesses, there are also practical tradeoffs that need to be considered in terms of research resources, like the time needed to complete the study, the availability of data, investigator expertise, subject recruitment, human subjects protection, research budget, difference to be detected, and lost-opportunity costs of doing the research instead of other studies that have priority for stakeholders. An important decision that will need to be made is whether or not randomization is needed for the questions being studied. There are several reasons why randomization might be needed, such as determining whether an FDA-approved drug can be used for a new use or indication that was not studied as part of the original drug approval process. A paper by Concato includes a thorough discussion of issues to consider when deciding whether randomization is necessary. 11

In discussing the tradeoffs of different study designs, researchers and stakeholders may wish to discuss the principal goals of research and ensure that researchers and stakeholders are aligned in their understanding of what is meant by scientific evidence. Fundamentally, research is a systematic investigation that uses scientific methods to measure, collect, and analyze data for the advancement of knowledge. This advancement is through the independent peer review and publication of study results, which are collectively referred to as scientific evidence. One definition of scientific evidence has been proposed by Normand and McNeil 12 as:

… the accumulation of information to support or refute a theory or hypothesis. … The idea is that assembling all the available information may reduce uncertainty about the effectiveness of the new technology compared to existing technologies in a setting where we believe particular relationships exist but are uncertain about their relevance …

While the primary aim of research is to produce new knowledge , the Normand and McNeil concept of evidence emphasizes that research helps create knowledge by reducing uncertainty about outcomes. However, rarely, if at all, does research eliminate all uncertainty around most decisions. In some cases, successful research will answer an important question and reduce uncertainty related to that question, but it may also increase uncertainty by leading to more, better informed questions regarding unknowns. As a result, nearly all decisions face some level of uncertainty even in a field where a body of research has been completed. This distinction is also critical because it helps to separate the research and subsequent actions that decisionmakers may take based on their assessment of the research results. Those subsequent actions may be informed by the research findings but will also be based on stakeholders' values and resources. Hence, as the definition by Normand and McNeil implies, research generates evidence but stakeholders decide whether to act on the evidence. Scientific evidence informs decisions to the extent it can adequately reduce the uncertainty about the problem for the stakeholder. Ultimately, treatment decisions are only guided by an assessment of the certainty that a course of therapy will lead to the outcomes of interest and the likelihood that this conclusion will be affected by the results of future studies.

In conceptualizing a study design, it is important for investigators to understand what constitutes sufficient and valid evidence from the stakeholder's perspective. In other words, what is the type of evidence that will be required to inform the stakeholder's decision to act or make a conscious decision not to take action? Evidence needed for action may vary by type of stakeholder and the scope of decisions that the stakeholder is making. For instance, a stakeholder who is making a population-based decision such as whether to provide insurance coverage for a new medical device with many alternatives may need substantially robust research findings in order to take action and provide that insurance coverage. In this example, the stakeholder may only accept as evidence a study with strong internal validity and generalizability (i.e., one conducted in a nationally representative sample of patients with the disease). On the other hand a patient who has a health condition where there are few treatments may be willing to accept lower-quality evidence in order to make a decision about whether to proceed with treatment despite a higher level of uncertainty about the outcome.

In many cases, there may exist a gradient of actions that can be taken based on available evidence. Quanstrum and Hayward 13 have discussed this gradient and argued that health care decisionmaking is changing, partly because more information is available to patients and other stakeholders about treatment options. As shown in the upper panel (A) in Figure 1.1 , many people may currently believe that health care treatment decisions are basically uniform for most people and under most circumstances. Panel A represents a hypothetical treatment whereby there is an evidentiary threshold or a point at which treatment is always beneficial and should be recommended. On the other hand below this threshold care provides no benefits and treatment should be discouraged. Quanstrum and Hayward argue that increasingly health care decisions are more like the lower panel (B). This panel portrays health care treatments as providing a large zone of discretion where benefits may be low or modest for most people. While above this zone treatment may always be recommended, individuals who fall within the zone may have questionable health benefits from treatment. As a result, different decisionmakers may take different actions based on their individual preferences.

Conceptualization of clinical decisionmaking. See Quanstrum KH, Hayward RA (Reference #). This figure is copyrighted by the Massachusetts Medical Society and reprinted with permission.

In light of this illustration, the following questions are suggested for discussion with stakeholders to help elicit the amount of uncertainty that is acceptable so that the study design can reach an appropriate level of evidence for the decision at hand:

• What level of new scientific evidence does the decisionmaker need to make a decision or take action?
• What quality of evidence is needed for the decisionmaker to act?
• What level of certainty of the outcome is needed by the decisionmaker(s)?
• How specific does the evidence need to be?
• Will decisions require consensus of multiple parties?

Additional Considerations When Considering Evidentiary Needs

As mentioned earlier, different stakeholders may disagree on the usefulness of different research designs, but it should be pointed out that this disagreement may be because stakeholders have different scopes of decisions to make. For example, high-quality research that is conclusive may be needed to make a decision that will affect the entire nation. On the other hand, results with more uncertainty as to the magnitude of the effect estimate(s) may be acceptable in making some decisions such as those affecting fewer people or where the risks to health are low. Often this disagreement occurs when different stakeholders debate whether evidence is needed from a new randomized controlled trial or whether evidence can be obtained from an analysis of an existing database. In this debate, both sides need to clarify whether they are facing the same decision or the decisions are different, particularly in terms of their scope.

Groups committed to evidence-based decisionmaking recognize that scientific evidence is only one component of the process of making decisions. Evidence generation is the goal of research, but evidence alone is not the only facet of evidence-based decisionmaking. In addition to scientific evidence, decisionmaking involves the consideration of (a) values, particularly the values placed on benefits and harms, and (b) resources. 14 Stakeholder differences in values and resources may mean that different decisions are made based on the same scientific evidence. Moreover, differences in values may create conflict in the decisionmaking process. One stakeholder may believe a particular study outcome is most important from their perspective, while another stakeholder may believe a different outcome is the most important for determining effectiveness.

Likewise, there may be inherent conflicts in values between individual decisionmaking and population decisionmaking, even though these decisions are often interrelated. For example, an individual may have a higher tolerance for treatment risk in light of the expected treatment benefits for him or her. On the other hand a regulatory health authority may determine that the population risk is too great without sufficient evidence that treatment provides benefits to the population. An example of this difference in perspective can be seen with how different decisionmakers responded to evidence about the drug Avastin ® (bevacizumab) for the treatment of metastatic breast cancer. In this case, the FDA revoked their approval of the breast cancer indication for Avastin after concluding that the drug had not been shown to be safe and effective for that use. Nonetheless, Medicare, the public insurance program for the elderly and disabled continued to allow coverage when a physician prescribes the drug, even for breast cancer. Likewise, some patient groups were reported to be concerned by the decision since it presumably would deny some women access to Avastin treatment. For a more thorough discussion of these issues around differences in perspective, the reader is referred to an article by Atkins 15 and the examples in Table 1.3 below.

Examples of individual versus population decisions (Adapted from Atkins, 2007).

• Specifying Magnitude of Effect

In order for decisions to be objective, it is important for there to be an a priori discussion with stakeholders about the magnitude of effect that stakeholders believe represents a meaningful difference between treatment options. Researchers will be familiar with the basic tenet that statistically significant differences do not always represent clinically meaningful differences. Hence, researchers and stakeholders will need to have knowledge of the instruments that are used to measure differences and the accuracy, limitations, and properties of those instruments. Three key questions are recommended to use when eliciting from stakeholders the effect sizes that are important to them for making a decision or taking action:

• How do patients and other stakeholders define a meaningful difference between interventions?
• How do previous studies and reviews define a meaningful difference?
• Are patients and other stakeholders interested in superiority or noninferiority as it relates to decisionmaking?
• Challenges to Developing Study Questions and Initial Solutions

In developing CER study objectives and questions, there are some potential challenges that face researchers and stakeholders. The involvement of patients and other stakeholders in determining study objectives and questions is a relatively new paradigm, but one that is consistent with established principles of translational research. A key principle of translational research is that users need to be involved in research at the earliest stages for the research to be adopted. 16 In addition, most research is currently initiated by an investigator, and traditionally there have been few incentives (and some disincentives) to involving others in designing a new research study. Although the research paradigm is rapidly shifting, 17 there is little information about how to structure, process, and evaluate outcomes from initiatives that attempt to engage stakeholders in developing study questions and objectives with researchers. As different approaches are taken to involve stakeholders in the research process, researchers will learn how to optimize the process of stakeholder involvement and improve the applicability of research to the end-users.

The bringing together of stakeholders may create some general challenges to the research team. For instance, it may be difficult to identify, engage, or manage all stakeholders who are interested in developing and using scientific evidence for addressing a problem. A process that allows for public commenting on research protocols through Internet postings may be helpful in reaching the widest network of interested stakeholders. Nevertheless, finding stakeholders who can represent all perspectives may not always be practical or available to the study team. In addition, competing interests among stakeholders may make prioritization of research questions challenging. Different stakeholders have different needs and this may make prioritization of research difficult. Nonetheless, as the science of translational research evolves, the collaboration of researchers with stakeholders will likely become increasingly the standard of practice in designing new research.

To assist researchers and stakeholders with working together, AHRQ has published several online resources to facilitate the involvement of stakeholders in the research process. These include a brief guide for stakeholders that highlights opportunities for taking part in AHRQ's Effective Health Care Program, a facilitation primer with strategies for working with diverse stakeholder groups, a table of suggested tasks for researchers to involve stakeholders in the identification and prioritization of future research, and learning modules with slide presentations on engaging stakeholders in the Effective Health Care Program. 18 - 19 In addition, AHRQ supports the Evidence-based Practice Centers in working with various stakeholders to further develop and prioritize decisionmakers' future research needs, which are published in a series of reports on AHRQ's Web site and on the National Library of Medicine's open-access Bookshelf. 20

Likewise, AHRQ supports the active involvement of patients and other stakeholders in the AHRQ DEcIDE program, in which different models of engagement have been used. These models include hosting in-person meetings with stakeholders to create research agendas; 21 - 22 developing research based on questions posed by public payers such as Centers for Medicare and Medicaid Services; addressing knowledge gaps that have been identified in AHRQ systematic reviews through new research; and supporting five research consortia, each of which involves researchers, patients, and other stakeholders working together to develop, prioritize, and implement research studies.

• Summary and Conclusion

This chapter provides a framework for formulating study objectives and questions, for a research protocol on a CER topic. Implementation of the framework involves collaboration between researchers and stakeholders in conceptualizing the research objectives and questions and the design of the study. In this process, there is a shared commitment to protect the integrity of the research results from bias and conflicts of interest, so that the results are valid for informing decisions and health care actions. Due to the complexity of some health care decisions, the evidence needed for decisionmaking or action may need to be developed from multiple studies, including preliminary research that becomes the underpinning for larger studies. The principles described in this chapter are intended to strengthen the writing of research protocols and enhance the results from the emanating studies, for informing the important decisions facing patients, providers, and other stakeholders about health care treatments and new technologies. Subsequent chapters in this User's Guide provide specific principles for operationalizing the study objectives and research questions in writing a complete study protocol that can be executed as new research.

Checklist: Guidance and key considerations for developing study objectives and questions for observational CER protocols

View in own window

Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide is copyrighted by the Agency for Healthcare Research and Quality (AHRQ). The product and its contents may be used and incorporated into other materials on the following three conditions: (1) the contents are not changed in any way (including covers and front matter), (2) no fee is charged by the reproducer of the product or its contents for its use, and (3) the user obtains permission from the copyright holders identified therein for materials noted as copyrighted by others. The product may not be sold for profit or incorporated into any profitmaking venture without the expressed written permission of AHRQ.

• Cite this Page Smith SR. Study Objectives and Questions. In: Velentgas P, Dreyer NA, Nourjah P, et al., editors. Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide. Rockville (MD): Agency for Healthcare Research and Quality (US); 2013 Jan. Chapter 1.
• PDF version of this title (5.8M)

In this Page

Other titles in these collections.

• AHRQ Methods for Effective Health Care
• Health Services/Technology Assessment Text (HSTAT)

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Study Objectives and Questions - Developing a Protocol for Observational Compara... Study Objectives and Questions - Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• How to Write a Strong Hypothesis | Steps & Examples

How to Write a Strong Hypothesis | Steps & Examples

Published on May 6, 2022 by Shona McCombes . Revised on August 15, 2023.

A hypothesis is a statement that can be tested by scientific research. If you want to test a relationship between two or more variables, you need to write hypotheses before you start your experiment or data collection .

Example: Hypothesis

Daily apple consumption leads to fewer doctor’s visits.

Table of contents

What is a hypothesis, developing a hypothesis (with example), hypothesis examples, other interesting articles, frequently asked questions about writing hypotheses.

A hypothesis states your predictions about what your research will find. It is a tentative answer to your research question that has not yet been tested. For some research projects, you might have to write several hypotheses that address different aspects of your research question.

A hypothesis is not just a guess – it should be based on existing theories and knowledge. It also has to be testable, which means you can support or refute it through scientific research methods (such as experiments, observations and statistical analysis of data).

Variables in hypotheses

Hypotheses propose a relationship between two or more types of variables .

• An independent variable is something the researcher changes or controls.
• A dependent variable is something the researcher observes and measures.

If there are any control variables , extraneous variables , or confounding variables , be sure to jot those down as you go to minimize the chances that research bias  will affect your results.

In this example, the independent variable is exposure to the sun – the assumed cause . The dependent variable is the level of happiness – the assumed effect .

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Step 1. Ask a question

Writing a hypothesis begins with a research question that you want to answer. The question should be focused, specific, and researchable within the constraints of your project.

Step 2. Do some preliminary research

Your initial answer to the question should be based on what is already known about the topic. Look for theories and previous studies to help you form educated assumptions about what your research will find.

At this stage, you might construct a conceptual framework to ensure that you’re embarking on a relevant topic . This can also help you identify which variables you will study and what you think the relationships are between them. Sometimes, you’ll have to operationalize more complex constructs.

Step 3. Formulate your hypothesis

Now you should have some idea of what you expect to find. Write your initial answer to the question in a clear, concise sentence.

4. Refine your hypothesis

You need to make sure your hypothesis is specific and testable. There are various ways of phrasing a hypothesis, but all the terms you use should have clear definitions, and the hypothesis should contain:

• The relevant variables
• The specific group being studied
• The predicted outcome of the experiment or analysis

5. Phrase your hypothesis in three ways

To identify the variables, you can write a simple prediction in  if…then form. The first part of the sentence states the independent variable and the second part states the dependent variable.

In academic research, hypotheses are more commonly phrased in terms of correlations or effects, where you directly state the predicted relationship between variables.

If you are comparing two groups, the hypothesis can state what difference you expect to find between them.

6. Write a null hypothesis

If your research involves statistical hypothesis testing , you will also have to write a null hypothesis . The null hypothesis is the default position that there is no association between the variables. The null hypothesis is written as H 0 , while the alternative hypothesis is H 1 or H a .

• H 0 : The number of lectures attended by first-year students has no effect on their final exam scores.
• H 1 : The number of lectures attended by first-year students has a positive effect on their final exam scores.

If you want to know more about the research process , methodology , research bias , or statistics , make sure to check out some of our other articles with explanations and examples.

• Sampling methods
• Simple random sampling
• Stratified sampling
• Cluster sampling
• Likert scales
• Reproducibility

 Statistics

• Null hypothesis
• Statistical power
• Probability distribution
• Effect size
• Poisson distribution

Research bias

• Optimism bias
• Cognitive bias
• Implicit bias
• Hawthorne effect
• Anchoring bias
• Explicit bias

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

A hypothesis is not just a guess — it should be based on existing theories and knowledge. It also has to be testable, which means you can support or refute it through scientific research methods (such as experiments, observations and statistical analysis of data).

Null and alternative hypotheses are used in statistical hypothesis testing . The null hypothesis of a test always predicts no effect or no relationship between variables, while the alternative hypothesis states your research prediction of an effect or relationship.

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics. It is used by scientists to test specific predictions, called hypotheses , by calculating how likely it is that a pattern or relationship between variables could have arisen by chance.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. (2023, August 15). How to Write a Strong Hypothesis | Steps & Examples. Scribbr. Retrieved November 17, 2023, from https://www.scribbr.com/methodology/hypothesis/

Is this article helpful?

Shona McCombes

Other students also liked, construct validity | definition, types, & examples, what is a conceptual framework | tips & examples, operationalization | a guide with examples, pros & cons, what is your plagiarism score.

How to Write a Research Paper Introduction (with Examples)

The research paper introduction section, along with the Title and Abstract, can be considered the face of any research paper. The following article is intended to guide you in organizing and writing the research paper introduction for a quality academic article or dissertation.

The research paper introduction aims to present the topic to the reader. A study will only be accepted for publishing if you can ascertain that the available literature cannot answer your research question. So it is important to ensure that you have read important studies on that particular topic, especially those within the last five to ten years, and that they are properly referenced in this section. 1 What should be included in the research paper introduction is decided by what you want to tell readers about the reason behind the research and how you plan to fill the knowledge gap. The best research paper introduction provides a systemic review of existing work and demonstrates additional work that needs to be done. It needs to be brief, captivating, and well-referenced; a well-drafted research paper introduction will help the researcher win half the battle.

The introduction for a research paper is where you set up your topic and approach for the reader. It has several key goals:

• Present your research topic
• Capture reader interest
• Summarize existing research
• Position your own approach
• Define your specific research problem and problem statement
• Highlight the novelty and contributions of the study
• Give an overview of the paper’s structure

The research paper introduction can vary in size and structure depending on whether your paper presents the results of original empirical research or is a review paper. Some research paper introduction examples are only half a page while others are a few pages long. In many cases, the introduction will be shorter than all of the other sections of your paper; its length depends on the size of your paper as a whole.

Table of Contents

What is the introduction for a research paper, why is the introduction important in a research paper, what are the parts of introduction in the research, 1. introduce the research topic:, 2. determine a research niche:, 3. place your research within the research niche:, frequently asked questions on research paper introduction, key points to remember.

The introduction in a research paper is placed at the beginning to guide the reader from a broad subject area to the specific topic that your research addresses. They present the following information to the reader

• Scope: The topic covered in the research paper
• Context: Background of your topic
• Importance: Why your research matters in that particular area of research and the industry problem that can be targeted

The research paper introduction conveys a lot of information and can be considered an essential roadmap for the rest of your paper. A good introduction for a research paper is important for the following reasons:

• It stimulates your reader’s interest: A good introduction section can make your readers want to read your paper by capturing their interest. It informs the reader what they are going to learn and helps determine if the topic is of interest to them.
• It helps the reader understand the research background: Without a clear introduction, your readers may feel confused and even struggle when reading your paper. A good research paper introduction will prepare them for the in-depth research to come. It provides you the opportunity to engage with the readers and demonstrate your knowledge and authority on the specific topic.
• It explains why your research paper is worth reading: Your introduction can convey a lot of information to your readers. It introduces the topic, why the topic is important, and how you plan to proceed with your research.
• It helps guide the reader through the rest of the paper: The research paper introduction gives the reader a sense of the nature of the information that will support your arguments and the general organization of the paragraphs that will follow. It offers an overview of what to expect when reading the main body of your paper.

A good research paper introduction section should comprise three main elements: 2

• What is known: This sets the stage for your research. It informs the readers of what is known on the subject.
• What is lacking: This is aimed at justifying the reason for carrying out your research. This could involve investigating a new concept or method or building upon previous research.
• What you aim to do: This part briefly states the objectives of your research and its major contributions. Your detailed hypothesis will also form a part of this section.

How to write a research paper introduction?

The first step in writing the research paper introduction is to inform the reader what your topic is and why it’s interesting or important. This is generally accomplished with a strong opening statement. The second step involves establishing the kinds of research that have been done and ending with limitations or gaps in the research that you intend to address. Finally, the research paper introduction clarifies how your own research fits in and what problem it addresses. If your research involved testing hypotheses, these should be stated along with your research question. The hypothesis should be presented in the past tense since it will have been tested by the time you are writing the research paper introduction.

The following key points, with examples, can guide you when writing the research paper introduction section:

• Highlight the importance of the research field or topic
• Describe the background of the topic
• Present an overview of current research on the topic

Example: The inclusion of experiential and competency-based learning has benefitted electronics engineering education. Industry partnerships provide an excellent alternative for students wanting to engage in solving real-world challenges. Industry-academia participation has grown in recent years due to the need for skilled engineers with practical training and specialized expertise. However, from the educational perspective, many activities are needed to incorporate sustainable development goals into the university curricula and consolidate learning innovation in universities.

• Reveal a gap in existing research or oppose an existing assumption
• Formulate the research question

Example: There have been plausible efforts to integrate educational activities in higher education electronics engineering programs. However, very few studies have considered using educational research methods for performance evaluation of competency-based higher engineering education, with a focus on technical and or transversal skills. To remedy the current need for evaluating competencies in STEM fields and providing sustainable development goals in engineering education, in this study, a comparison was drawn between study groups without and with industry partners.

• State the purpose of your study
• Highlight the key characteristics of your study
• Describe important results
• Highlight the novelty of the study.
• Offer a brief overview of the structure of the paper.

Example: The study evaluates the main competency needed in the applied electronics course, which is a fundamental core subject for many electronics engineering undergraduate programs. We compared two groups, without and with an industrial partner, that offered real-world projects to solve during the semester. This comparison can help determine significant differences in both groups in terms of developing subject competency and achieving sustainable development goals.

The purpose of the research paper introduction is to introduce the reader to the problem definition, justify the need for the study, and describe the main theme of the study. The aim is to gain the reader’s attention by providing them with necessary background information and establishing the main purpose and direction of the research.

The length of the research paper introduction can vary across journals and disciplines. While there are no strict word limits for writing the research paper introduction, an ideal length would be one page, with a maximum of 400 words over 1-4 paragraphs. Generally, it is one of the shorter sections of the paper as the reader is assumed to have at least a reasonable knowledge about the topic. 2 For example, for a study evaluating the role of building design in ensuring fire safety, there is no need to discuss definitions and nature of fire in the introduction; you could start by commenting upon the existing practices for fire safety and how your study will add to the existing knowledge and practice.

When deciding what to include in the research paper introduction, the rest of the paper should also be considered. The aim is to introduce the reader smoothly to the topic and facilitate an easy read without much dependency on external sources. 3 Below is a list of elements you can include to prepare a research paper introduction outline and follow it when you are writing the research paper introduction. Topic introduction: This can include key definitions and a brief history of the topic. Research context and background: Offer the readers some general information and then narrow it down to specific aspects. Details of the research you conducted: A brief literature review can be included to support your arguments or line of thought. Rationale for the study: This establishes the relevance of your study and establishes its importance. Importance of your research: The main contributions are highlighted to help establish the novelty of your study Research hypothesis: Introduce your research question and propose an expected outcome. Organization of the paper: Include a short paragraph of 3-4 sentences that highlights your plan for the entire paper

Cite only works that are most relevant to your topic; as a general rule, you can include one to three. Note that readers want to see evidence of original thinking. So it is better to avoid using too many references as it does not leave much room for your personal standpoint to shine through. Citations in your research paper introduction support the key points, and the number of citations depend on the subject matter and the point discussed. If the research paper introduction is too long or overflowing with citations, it is better to cite a few review articles rather than the individual articles summarized in the review. A good point to remember when citing research papers in the introduction section is to include at least one-third of the references in the introduction.

The literature review plays a significant role in the research paper introduction section. A good literature review accomplishes the following: Introduces the topic – Establishes the study’s significance – Provides an overview of the relevant literature – Provides context for the study using literature – Identifies knowledge gaps However, remember to avoid making the following mistakes when writing a research paper introduction: Do not use studies from the literature review to aggressively support your research Avoid direct quoting Do not allow literature review to be the focus of this section. Instead, the literature review should only aid in setting a foundation for the manuscript.

Remember the following key points for writing a good research paper introduction: 4

• Avoid stuffing too much general information: Avoid including what an average reader would know and include only that information related to the problem being addressed in the research paper introduction. For example, when describing a comparative study of non-traditional methods for mechanical design optimization, information related to the traditional methods and differences between traditional and non-traditional methods would not be relevant. In this case, the introduction for the research paper should begin with the state-of-the-art non-traditional methods and methods to evaluate the efficiency of newly developed algorithms.
• Avoid packing too many references: Cite only the required works in your research paper introduction. The other works can be included in the discussion section to strengthen your findings.
• Avoid extensive criticism of previous studies: Avoid being overly critical of earlier studies while setting the rationale for your study. A better place for this would be the Discussion section, where you can highlight the advantages of your method.
• Avoid describing conclusions of the study: When writing a research paper introduction remember not to include the findings of your study. The aim is to let the readers know what question is being answered. The actual answer should only be given in the Results and Discussion section.

To summarize, the research paper introduction section should be brief yet informative. It should convince the reader the need to conduct the study and motivate him to read further.

1. Jawaid, S. A., & Jawaid, M. (2019). How to write introduction and discussion. Saudi Journal of Anaesthesia, 13(Suppl 1), S18.

2. Dewan, P., & Gupta, P. (2016). Writing the title, abstract and introduction: Looks matter!. Indian pediatrics, 53, 235-241.

3. Cetin, S., & Hackam, D. J. (2005). An approach to the writing of a scientific Manuscript1. Journal of Surgical Research, 128(2), 165-167.

4. Bavdekar, S. B. (2015). Writing introduction: Laying the foundations of a research paper. Journal of the Association of Physicians of India, 63(7), 44-6.

Paperpal is an AI writing assistant that help academics write better, faster with real-time suggestions for in-depth language and grammar correction. Trained on millions of research manuscripts enhanced by professional academic editors, Paperpal delivers human precision at machine speed.

Try it for free or upgrade to  Paperpal Prime , which unlocks unlimited access to premium features like academic translation, paraphrasing, contextual synonyms, consistency checks and more. It’s like always having a professional academic editor by your side! Go beyond limitations and experience the future of academic writing.  Get Paperpal Prime now at just US$19 a month!

Related Reads:

• Scientific writing style guides explained

5 Reasons for Rejection After Peer Review

Ethical research practices for research with human subjects.

• How to write a research paper title

Practice vs. practise: Learn the difference

Academic paraphrasing: why paperpal’s rewrite should be your first choice , you may also like, paraphrasing in academic writing: answering top author queries, chemistry terms: 7 commonly confused words in chemistry..., 7 ways to improve your academic writing process, how to write a conclusion for research papers..., paperpal copilot is live: experience the generative ai..., life sciences papers: 9 tips for authors writing..., what is peer review: importance and types of..., self-plagiarism in research: what it is and how....

Peer Reviewed

What do we study when we study misinformation? A scoping review of experimental research (2016-2022)

Article metrics.

CrossRef Citations

Altmetric Score

PDF Downloads

We reviewed 555 papers published from 2016–2022 that presented misinformation to participants. We identified several trends in the literature—increasing frequency of misinformation studies over time, a wide variety of topics covered, and a significant focus on COVID-19 misinformation since 2020. We also identified several important shortcomings, including overrepresentation of samples from the United States and Europe and excessive emphasis on short-term consequences of brief, text-based misinformation. Most studies examined belief in misinformation as the primary outcome. While many researchers identified behavioural consequences of misinformation exposure as a pressing concern, we observed a lack of research directly investigating behaviour change. 

School of Applied Psychology, University College Cork, Ireland

School of Psychology, University College Dublin, Ireland

Research Question

• What populations, materials, topics, methods, and outcomes are common in published misinformation research from 2016–2022?

Essay Summary

• The goal of this review was to identify the scope of methods and measures used in assessing the impact of real-world misinformation. 
• We screened 8,469 papers published between 2016 and 2022, finding 555 papers with 759 studies where participants were presented with misinformation. 
• The vast majority of studies included samples from the United States or Europe, used brief text-based misinformation (1–2 sentences), measured belief in the misinformation as a primary outcome, and had no delay between misinformation exposure and measurement of the outcome. 
• The findings highlight certain elements of misinformation research that are currently underrepresented in the literature. In particular, we note the need for more diverse samples, measurement of behaviour change in response to misinformation, and assessment of the longer-term consequences of misinformation exposure. 
• Very few papers directly examined effects of misinformation on behaviour (1%) or behavioural intentions (10%), instead measuring proxy outcomes such as belief or attitudes. Nevertheless, many papers draw conclusions regarding the consequences of misinformation for real-world behaviour.
• We advise caution in inferring behavioural consequences unless behaviours (or behavioural intentions) are explicitly measured. 
• We recommend that policymakers reflect on the specific outcomes they hope to influence and consider whether extant evidence indicates that their efforts are likely to be successful.  

Implications

In this article, we report a scoping review of misinformation research from 2016-2022. A scoping review is a useful evidence synthesis approach that is particularly appropriate when the purpose of the review is to identify knowledge gaps or investigate research conduct across a body of literature (Munn et al., 2018). Our review investigates the methods used in misinformation research since interest in so-called “fake news” spiked in the wake of the 2016 U.S. presidential election and the Brexit referendum vote. While previous publications have reflected critically on the current focus and future pathways for the field (Camargo & Simon, 2022), here we address a simple question: what do we study when we study misinformation? We are interested in the methods, outcomes, and samples that are commonly used in misinformation research and what that might tell us about our focus and blind spots.         

Our review covers studies published from January 2016 to July 2022 and includes any studies where misinformation was presented to participants by researchers. The misinformation had to be related to real-world information (i.e., not simple eyewitness misinformation effects), and the researchers had to measure participants’ response to the misinformation as a primary outcome. As expected, we found an increase in misinformation research over time, from just three studies matching our criteria in 2016 to 312 published in 2021. As the number of studies has grown, so too has the range of topics covered. The three studies published in 2016 all assessed political misinformation, but by 2021, just 35% of studies addressed this issue, while the remainder examined other topics, including climate change, vaccines, nutrition, immigration, and more. COVID-19 became a huge focus for misinformation researchers in late 2020, and our review includes over 200 studies that used COVID-related materials. Below we discuss some implications and recommendations for the field based on our findings.

Call for increased diversity & ecological validity

It has been previously noted that the evidence base for understanding misinformation is skewed by pragmatic decisions affecting the topics that researchers choose to study. For example, Altay and colleagues (2023) argue that misinformation researchers typically focus on social media because it is methodologically convenient, and that this can give rise to the false impression that misinformation is a new phenomenon or one solely confined to the internet. Our findings highlight many other methodological conveniences that affect our understanding of misinformation relating to samples, materials, and experimental design. 

Our findings clearly show that certain populations and types of misinformation are well-represented in the literature. In particular, the majority of studies (78%) drew on samples from the United States or Europe. Though the spread of misinformation is widely recognised as a global phenomenon (Lazer, 2018), countries outside the United States and Europe are underrepresented in misinformation research. We recommend more diverse samples for future studies, as well as studies that assess interventions across multiple countries at once (e.g., Porter & Wood, 2021). Before taking action, policymakers should take note of whether claims regarding the spread of misinformation or the effectiveness of particular interventions have been tested in their jurisdictions and consider whether effects are likely to generalise to other contexts.  

There are growing concerns about large-scale disinformation campaigns and how they may threaten democracies (Nagasko, 2020; Tenove, 2020). For example, research has documented elaborate Russian disinformation campaigns reaching individuals via multiple platforms, delivery methods, and media formats (Erlich & Garner, 2023; Wilson & Starbird, 2020). Our review of the misinformation literature suggests that most studies don’t evaluate conditions that are relevant to these disinformation campaigns. Most studies present misinformation in a very brief format, comprising a single presentation of simple text. Moreover, most studies do not include a delay between presentation and measurement of the outcome. This may be due to ethical concerns, which are, of course, of crucial importance when conducting misinformation research (Greene et al., 2022). Nevertheless, this has implications for policymakers, who may draw on research that does not resemble the real-world conditions in which disinformation campaigns are likely to play out. For example, there is evidence to suggest that repeated exposure can increase the potency of misinformation (Fazio et al., 2022; Pennycook et al., 2018), and some studies have found evidence of misinformation effects strengthening over time (Murphy et al., 2021). These variables are typically studied in isolation, and we,therefore, have an incomplete understanding of how they might interact in large-scale campaigns in the real world. This means that policymakers may make assumptions about which messages are likely to influence citizens based on one or two variables—for example, a news story’s source or the political congeniality of its content—without considering the impact of other potentially interacting variables, such as the delay between information exposure and the target action (e.g., voting in an election) or the number of times an individual is likely to have seen the message. In sum, we would recommend a greater focus on ecologically valid methods to assess misinformation that is presented in multiple formats, across multiple platforms, on repeated occasions, and over a longer time interval. We also encourage future research that is responsive to public and policy-maker concerns with regard to misinformation. For example, a misinformation-related topic that is frequently covered in news media is the looming threat of deepfake technology and the dystopian future it may herald (Broinowski, 2022). However, deepfakes were very rarely examined in the studies we reviewed (nine studies in total). 

Our review contributes to a growing debate as to how we should measure the effectiveness of misinformation interventions. Some have argued that measuring discernment (the ability to distinguish true from false information) is key (Guay et al., 2023). For example, in assessing whether an intervention is effective, we should consider the effects of the intervention on belief in fake news (as most studies naturally do) but also consider effects on belief in  true  news—that is, news items that accurately describe true events. This reflects the idea that while believing and sharing misinformation can lead to obvious dangers, not believing or not sharing true information may also be costly. Interventions that encourage skepticism towards media and news sources might cause substantial harm if they undercut trust in real news, particularly as true news is so much more prevalent than fake news (Acerbi et al., 2022). In our review, less than half of the included studies presented participants with both true and false information. Of those that did present true information, just 15% reported a measure of discernment (7% of all included studies), though there was some indication that this outcome measure has been more commonly reported in recent years. We recommend that future studies consider including both true items and a measure of discernment, particularly when assessing susceptibility to fake news or evaluating an intervention. Furthermore, policymakers should consider the possibility of unintended consequences if interventions aiming to reduce belief in misinformation are employed without due consideration of their effects on trust in news more generally.   

Is misinformation likely to change our behaviour?

 Many of our most pressing social concerns related to misinformation centre on the possibility of false information inciting behaviour change—for example, that political misinformation might have a causal effect on how we decide to vote, or that health misinformation might have a causal effect in refusal of vaccination or treatment. In the current review, we found the most common outcome measure was belief in misinformation (78% of studies), followed by attitudes towards the target of the misinformation (18% of studies). While it is, of course, of interest to examine how misinformation can change beliefs and attitudes, decades of research have shown that information provision is often ineffective at meaningfully changing attitudes (Albarracin & Shavitt, 2018) and even where such an intervention is successful, attitude change is not always sufficient to induce behavioural change (Verplanken & Orbell, 2022). 

When assessing whether misinformation can affect behaviour, previous research has reported mixed results. Loomba et al. (2021) found that exposure to COVID vaccine misinformation reduced intentions to get vaccinated, but other studies have reported null or inconsistent effects (Aftab & Murphy, 2022; de Saint Laurent et al., 2022; Greene & Murphy, 2021; Guess et al., 2020). The current review highlights the small number of studies that have examined offline behavioural intentions (10% of papers reviewed) or offline behaviour itself (< 1% of studies) as an outcome of misinformation exposure. Our findings reveal a mismatch between the stated goals and methodology of research, where many papers conceive of misinformation as a substantial problem and may cite behavioural outcomes (such as vaccine refusal) as the driver of this concern, but the studies instead measure belief. We acknowledge that studying real-world effects of misinformation presents some significant challenges, both practical (we cannot follow people into the voting booth or doctor’s office) and ethical (e.g., if experimental presentation of misinformation has the potential to cause real-world harm to participants or society). Moreover, it can be exceptionally difficult to identify causal links between information exposure and complex behaviours such as voting (Aral & Eckles, 2019). Nevertheless, we recommend that where researchers have an interest in behaviour change, they should endeavour to measure that as part of their study. Where a study has only measured beliefs, attitudes, or sharing intentions, we should refrain from drawing conclusions with regard to behaviour. 

From a policy perspective, those who are concerned about misinformation, such as governments and social media companies, ought to clearly specify whether these concerns relate to beliefs or behaviour, or both. Behaviour change is not the only negative outcome that may result from exposure to misinformation—confusion and distrust in news sources are also significant outcomes that many policymakers may wish to address. We recommend that policymakers reflect on the specific outcomes they hope to influence and consider whether extant evidence indicates that their efforts are likely to be successful. For example, if the goal is to reduce belief in or sharing of misinformation, there may be ample evidence to support a particular plan of action. On the other hand, if the goal is to affect a real-world behaviour such as vaccine uptake, our review suggests that the jury is still out. Policymakers may, therefore, be best advised to lend their support to new research aiming to explicitly address the question of behaviour change in response to misinformation. Specifically, we suggest that funding should be made available by national and international funding bodies to directly evaluate the impacts of misinformation in the real world. 

Finding 1: Studies assessing the effects of misinformation on behaviour are rare.

As shown in Table 1, the most commonly recorded outcome by far was belief in the misinformation presented (78% of studies), followed by attitudes (18.31%). Online behavioural intentions, like intention to share (18.05%) or intention to like or comment on a social media post (5.01%), were more commonly measured than offline behavioural intentions (10.94%), like planning to get vaccinated. A tiny proportion of studies (1.58%) measured actual behaviour and how it may change as a result of misinformation exposure. Even then, just one study (0.13%) assessed real-world behaviour—speed of tapping keys in a lab experiment (Bastick, 2021)—all other studies assessed online behaviour such as sharing of news articles or liking social media posts. Thus, no studies in this review assessed the kind of real-world behaviour targeted by misinformation, such as vaccine uptake or voting behaviour.

Finding 2: Studies in this field  overwhelmingly use short, text-based misinformation. 

The most common format for presenting misinformation was text only (62.71% of included studies), followed by text accompanied by an image (32.41%). Use of other formats was rare; video only (1.84%), text and video (1.32%), images only (< 1%), and audio only (< 1%). 

Of the studies that used textual formats (with or without additional accompanying media), the majority (62.72%) presented between one and two sentences of text. An additional 17.50% presented misinformation in a longer paragraph (more than two sentences), 12.92% presented a page or more, and 6.86% did not specify the length of misinformation text presented.

The most frequent framing for the misinformation presented was news stories (44.27%), misinformation presented with no context (33.47%), and Facebook posts (16.47%). Other less frequent misinformation framing included Twitter posts (7.64%), other social media posts (8.04%), other types of webpages (2.11%), fact checks of news stories (1.58%), and government and public communications (0.26%). 

Very few studies presented doctored media to participants; a small number (1.19%) presented deepfake videos and 1.05% presented other forms of doctored media.

Fin ding 3: Most studies assess outcomes instantly. 

Fewer than 7% of studies reported any delay between exposure to the misinformation and the measurement of outcome ( n  = 52). While many did not specify exactly how long the delay was ( n  = 30), most were less than a week; 1–2 minutes ( n  = 4), 5–10 minutes ( n  = 4), 1 day–1 week ( n  = 13), 3 weeks ( n  = 1) and 1–6 months ( n  = 2).

Finding 4: Most participants were from the USA or Europe. 

The majority of participants sampled were from the United States (49.93%), followed by Europe (28.19%) (see Table 2). All other regions each accounted for 6% or less of the total number of participants sampled, such as East Asia (5.53%), Africa (5.27%) and the Middle East (4.74%). Furthermore, 102 studies (13.26%) did not specify from where they sampled participants.

Finding 5: COVID-19 became a major focus of misinformation research. 

Political misinformation was the most commonly studied topic until 2021, when COVID-related misinformation research became the dominant focus of the field (see Table 3 for a full breakdown of the topics included in the selected studies). Overall, experimental misinformation research is on the rise. Our review included one paper from 2016, 12 papers from 2017, 18 papers from 2018, 48 papers from 2019, 123 papers from 2020, 231 papers from 2021, and 122 papers for the first half of 2022. 

Finding 6: Most studies do not report discernment between true and false misinformation.

In total, 340 studies (45.12%) presented participants with both true and false information. Of these studies, 52 (15.29%) reported a measure of discernment based on participants’ ability to discriminate between true and false information (e.g., the difference in standardised sharing intention scores between true and false items). Across the entire review then, fewer than 7% of studies report discernment between true and false information as an outcome. There was some indication that measurement of discernment is becoming more common over time; no studies included in the review reported a measure of discernment prior to 2019, and 48 out of the 52 studies that did measure discernment were published between 2020 and 2022.

A search was conducted to identify studies that presented participants with misinformation and measured their responses (e.g., belief in misinformation) after participants were exposed to misinformation. All studies must have been published since January 2016, with an English-language version available in a peer-reviewed journal. The final search for relevant records was carried out on the 16th of July, 2022. Searches were carried out in three databases (Scopus, Web of Science, and PsychINFO) using the search terms “misinformation” OR “fake news” OR “disinformation” OR “fabricated news” OR “false news.” The search strategy, inclusion criteria, and extraction templates were preregistered at  https://osf.io/d5hrj/ . 

Inclusion criteria

There were two primary criteria for inclusion in the current scoping review. A study was eligible for inclusion if it (i) presented participants with misinformation with any potential for real-world consequences and (ii) measured participants’ responses to this misinformation (e.g., belief in the misinformation, intentions to share the misinformation) as a main outcome. 

Exclusion criteria

Studies were excluded if they presented participants with misinformation of no real-world consequence (e.g., misinformation about a simulated crime, fabricated stories about fictitious plane crashes, misinformation about fictional persons that were introduced during the course of an experiment). If the misinformation was only relevant within the narrow confines of an experiment, we considered the paper ineligible. Furthermore, studies were excluded if they presented participants with general knowledge statements (e.g., trivia statements) or if they presented participants with misleading claims that were not clearly inaccurate (e.g., a general exaggeration of the benefits of a treatment). Studies were also excluded if the misinformation was only presented in the context of a debunking message, as were studies where the misinformation was presented as a hypothetical statement (e.g., “imagine if we told you that …”, “how many people do you think believe that…”). Studies of eyewitness memory were excluded, as were any studies not published in English. Finally, opinion pieces, commentaries, systematic reviews, or observational studies were excluded.

Originally, only experimental studies were to be included in the review. However, upon screening the studies, it became apparent that distinguishing between experiments, surveys, and intervention-based research was sometimes difficult—for example, cross-sectional studies exploring individual differences in fake news susceptibility might not be classified as true experiments (as they lack control groups and measure outcomes at only one time point), but they were clearly relevant to our aims. To avoid arbitrary decisions, we decided to drop this requirement and instead included all articles that met the inclusion criteria.

Screening and selection process

The search strategy yielded a total of 18,333 records (see Figure 2 for a summary of the screening process). Curious readers may note that a Google Scholar search for the search terms listed above produces a substantially different number of hits, though the number will vary from search to search. This lack of reproducibility in Google Scholar searches is one of many reasons why Google Scholar is not recommended for use in systematic reviews, and the three databases employed here are preferred (Gusenbauer & Haddaway, 2020; also see Boeker et al., 2013; Bramer et al., 2016). Following the removal of duplicates ( n  = 9,864), a total of 8,469 records were eligible to be screened. The titles and abstracts of the 8,469 eligible records were screened by six reviewers, in pairs of two, with a seventh reviewer resolving conflicts where they arose (weighted Cohen’s κ = 0.81). A total of 7,666 records were removed at this stage, as the records did not meet the criteria of the scoping review. 

The full texts of the remaining 803 records were then screened by four reviewers in pairs of two, with conflicts resolved by discussion among the pair with the conflict (weighted Cohen’s κ = 0.68). Among the 803 records, 248 records were excluded (see Figure 2 for reasons for exclusion). Thus, there were 555 papers included for extraction, with a total of 759 studies included therein. An alphabetical list of all included articles is provided in the Appendix, and the full data file listing all included studies and their labels is available at  https://osf.io/3apkt/ .

• / Psychology

Cite this Essay

Murphy, G., de Saint Laurent, C., Reynolds, M., Aftab, O., Hegarty, K. Sun, Y. & Greene, C. M. (2023). What do we study when we study misinformation? A scoping review of experimental research (2016-2022). Harvard Kennedy School (HKS) Misinformation Review . ttps://doi.org/10.37016/mr-2020-130

Bibliography

Acerbi, A., Altay, S., & Mercier, H. (2022). Fighting misinformation or fighting for information?. Harvard Kennedy School (HKS) Misinformation Review, 3 (1). https://doi.org/10.37016/mr-2020-87

Aftab, O., & Murphy, G. (2022). A single exposure to cancer misinformation may not significantly affect related behavioural intentions. HRB Open Research, 5 (82), 82. https://doi.org/10.12688/hrbopenres.13640.1

Albarracin, D., & Shavitt, S. (2018). Attitudes and attitude change. Annual Review of Psychology, 69 , 299–327. https://psycnet.apa.org/doi/10.1146/annurev-psych-122216-011911

Altay, S., Berriche, M., & Acerbi, A. (2023). Misinformation on misinformation: Conceptual and methodological challenges.  Social Media+ Society ,  9 (1), 20563051221150412. https://doi.org/10.1177/20563051221150412

Aral, S., & Eckles, D. (2019). Protecting elections from social media manipulation. Science , 365 (6456), 858–861. https://doi.org/doi:10.1126/science.aaw8243

Bastick, Z. (2021). Would you notice if fake news changed your behavior? An experiment on the unconscious effects of disinformation.  Computers in Human Behavior ,  116 , 106633. https://doi.org/10.1016/j.chb.2020.106633

Bramer, W. M., Giustini, D., & Kramer, B. M. R. (2016). Comparing the coverage, recall, and precision of searches for 120 systematic reviews in Embase, MEDLINE, and Google Scholar: A prospective study. Systematic Reviews , 5 (1), 39. https://doi.org/10.1186/s13643-016-0215-7

Boeker, M., Vach, W., & Motschall, E. (2013). Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Medical Research Methodology , 13 (1), 131. https://doi.org/10.1186/1471-2288-13-131

Broinowski, A. (2022). Deepfake nightmares, synthetic dreams: A review of dystopian and utopian discourses around deepfakes, and why the collapse of reality may not be imminent—yet.  Journal of Asia-Pacific Pop Culture ,  7 (1), 109–139. https://doi.org/10.5325/jasiapacipopcult.7.1.0109

Camargo, C. Q., & Simon, F. M. (2022). Mis- and disinformation studies are too big to fail: Six suggestions for the field’s future. Harvard Kennedy School (HKS) Misinformation Review , 3 (5). https://doi.org/10.37016/mr-2020-106

de Saint Laurent, C., Murphy, G., Hegarty, K., & Greene, C. M. (2022). Measuring the effects of misinformation exposure and beliefs on behavioural intentions: A COVID-19 vaccination study. Cognitive Research: Principles and Implications , 7 (1), 87. https://doi.org/10.1186/s41235-022-00437-y

Erlich, A., & Garner, C. (2023). Is pro-Kremlin disinformation effective? Evidence from Ukraine. The International Journal of Press/Politics , 28( 1), 5–28. https://doi.org/10.1177/19401612211045221

Fazio, L. K., Pillai, R. M., & Patel, D. (2022). The effects of repetition on belief in naturalistic settings.  Journal of Experimental Psychology: General, 151 (10), 2604–2613. https://doi.org/10.1037/xge0001211

Greene, C. M., de Saint Laurent, C., Murphy, G., Prike, T., Hegarty, K., & Ecker, U. K. (2022). Best practices for ethical conduct of misinformation research: A scoping review and critical commentary. European Psychologist, 28 (3), 139–150. https://doi.org/10.1027/1016-9040/a000491

Greene, C. M., & Murphy, G. (2021). Quantifying the effects of fake news on behavior: Evidence from a study of COVID-19 misinformation. Journal of Experimental Psychology: Applied , 27 (4), 773–784. https://doi.org/10.1037/xap0000371

Guay, B., Berinsky, A. J., Pennycook, G., & Rand, D. (2023). How to think about whether misinformation interventions work. Nature Human Behaviour, 7 , 1231–1233. https://doi.org/10.1038/s41562-023-01667-w

Guess, A. M., Lockett, D., Lyons, B., Montgomery, J. M., Nyhan, B., & Reifler, J. (2020). “Fake news” may have limited effects beyond increasing beliefs in false claims. Harvard Kennedy School (HKS) Misinformation Review , 1 (1). https://doi.org/10.37016/mr-2020-004

Gusenbauer, M., & Haddaway, N. R. (2020). Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Research Synthesis Methods , 11 (2), 181–217. https://doi.org/https://doi.org/10.1002/jrsm.1378

Loomba, S., de Figueiredo, A., Piatek, S. J., de Graaf, K., & Larson, H. J. (2021). Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA. Nature Human Behaviour , 5 (3), 337–348. https://doi.org/10.1038/s41562-021-01056-1

Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology , 18 . https://doi.org/10.1186/s12874-018-0611-x

Murphy, G., Lynch, L., Loftus, E., & Egan, R. (2021). Push polls increase false memories for fake new stories.  Memory ,  29 (6), 693–707. https://doi.org/10.1080/09658211.2021.1934033

Nagasako, T. (2020). Global disinformation campaigns and legal challenges. International Cybersecurity Law Review, 1 (1–2), 125–136. https://doi.org/10.1365/s43439-020-00010-7

Lazer, D. M. J., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., Metzger, M. J., Nyhan, B., Pennycook, G., Rothschild, D., Schudson, M., Sloman, S. A., Sunstein, C. R., Thorson, E. A., Watts, D. J., & Zittrain, J. L. (2018). The science of fake news. Science , 359 (6380), 1094-1096. https://doi.org/doi:10.1126/science.aao2998

Pennycook, G., Cannon, T. D., & Rand, D. G. (2018). Prior exposure increases perceived accuracy of fake news.  Journal of Experimental Psychology: General ,  147 (12), 1865–1880. https://doi.org/10.1037/xge0000465

Porter, E., & Wood, T. J. (2021). The global effectiveness of fact-checking: Evidence from simultaneous experiments in Argentina, Nigeria, South Africa, and the United Kingdom. Proceedings of the National Academy of Sciences , 118 (37), e2104235118. https://doi.org/10.1073/pnas.2104235118

Tenove, C. (2020). Protecting democracy from disinformation: Normative threats and policy responses. The International Journal of Press/Politics , 25 (3), 517–537. https://doi.org/10.1177/1940161220918740

Verplanken, B., & Orbell, S. (2022). Attitudes, habits, and behavior change. Annual Review of Psychology , 73 , 327–352. https://doi.org/10.1146/annurev-psych-020821-011744

Wilson, T., & Starbird, K. (2020). Cross-platform disinformation campaigns: lessons learned and next steps. Harvard Kennedy School (HKS) Misinformation Review , 1 (1) . https://doi.org/10.37016/mr-2020-002

This project was funded by the Health Research Board of Ireland – COV19-2020-030. The funding body had no role in the design, interpretation, or reporting of the research.

Competing Interests

The authors declare no competing interests.

This review protocol was exempt from ethics approval.

This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided that the original author and source are properly credited.

Data Availability

All materials needed to replicate this study are available via the Harvard Dataverse at  https://doi.org/10.7910/DVN/X1YH6S and OSF at https://osf.io/3apkt/ .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 16 November 2023

A study of generative large language model for medical research and healthcare

• Cheng Peng   ORCID: orcid.org/0000-0002-1994-893X 1 ,
• Xi Yang 1 , 2 ,
• Aokun Chen 1 , 2 ,
• Kaleb E. Smith 3 ,
• Nima PourNejatian 3 ,
• Anthony B. Costa 3 ,
• Cheryl Martin 3 ,
• Mona G. Flores   ORCID: orcid.org/0000-0002-7362-3044 3 ,
• Ying Zhang   ORCID: orcid.org/0000-0003-4210-2104 4 ,
• Tanja Magoc 5 ,
• Gloria Lipori   ORCID: orcid.org/0000-0001-5616-2701 5 , 6 ,
• Duane A. Mitchell   ORCID: orcid.org/0000-0001-6049-213X 6 ,
• Naykky S. Ospina 7 ,
• Mustafa M. Ahmed 8 ,
• William R. Hogan   ORCID: orcid.org/0000-0002-9881-1017 1 ,
• Elizabeth A. Shenkman   ORCID: orcid.org/0000-0003-4903-1804 1 ,
• Yi Guo   ORCID: orcid.org/0000-0003-0587-4105 1 , 2 ,
• Jiang Bian   ORCID: orcid.org/0000-0002-2238-5429 1 , 2 &
• Yonghui Wu   ORCID: orcid.org/0000-0002-6780-6135 1 , 2  

npj Digital Medicine volume  6 , Article number:  210 ( 2023 ) Cite this article

19 Altmetric

Metrics details

• Health care
• Translational research

There are enormous enthusiasm and concerns in applying large language models (LLMs) to healthcare. Yet current assumptions are based on general-purpose LLMs such as ChatGPT, which are not developed for medical use. This study develops a generative clinical LLM, GatorTronGPT, using 277 billion words of text including (1) 82 billion words of clinical text from 126 clinical departments and approximately 2 million patients at the University of Florida Health and (2) 195 billion words of diverse general English text. We train GatorTronGPT using a GPT-3 architecture with up to 20 billion parameters and evaluate its utility for biomedical natural language processing (NLP) and healthcare text generation. GatorTronGPT improves biomedical natural language processing. We apply GatorTronGPT to generate 20 billion words of synthetic text. Synthetic NLP models trained using synthetic text generated by GatorTronGPT outperform models trained using real-world clinical text. Physicians’ Turing test using 1 (worst) to 9 (best) scale shows that there are no significant differences in linguistic readability ( p  = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) and clinical relevance ( p  = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human) and that physicians cannot differentiate them ( p  < 0.001). This study provides insights into the opportunities and challenges of LLMs for medical research and healthcare.

Introduction

Generative large language models (LLMs) such as the ChatGPT 1 have surprised the world by answering questions conversationally and generating textual content such as emails, articles, and even computer codes, triggering enormous enthusiasm in applying LLMs to healthcare 2 , 3 , 4 . People are enthusiastic about LLMs in the potential to facilitate documentation of patient reports (e.g., a progress report) 3 , 4 , improving diagnostic accuracy 5 , and assisting in various clinical care 6 , 7 , while at the same time concerning the hallucinations and fabrications 7 , 8 , bias and stereotype 9 , and risks of patient privacy and ethics 10 . Yet, this enthusiasm and concerns are based on ChatGPT, which is not designed for healthcare use 1 . Until now, it is unclear how this disruptive technology can help medical research and potentially improve the quality of healthcare.

Language model is a simple statistical distribution used in natural language processing (NLP) to formulate the probability of a sequence of words or the next word in a sequence. Surprisingly, when it is used as a learning objective to train a specific neural network architecture named transformer, and when the model size is very large such as billions or hundreds of billions of parameters, important artificial intelligence (AI) emerges. For example, LLMs can learn knowledge from one task and apply it to another task (i.e., transfer learning), learn from very few labeled samples (i.e., few-shot learning), and learn without human-labeled samples (i.e., zero-shot learning) 11 , 12 , 13 . The LLM pretrained using decoder-only transformer such as GPT-3 is known as generative LLM as it can generate human-like text. The conversational ability of LLMs is achieved using prompt-based text generation 14 , the key technology guiding LLMs to generate reasonable answers and contextual contents.

This study aims to develop a generative LLM using real-world clinical text and evaluate its utility for medical research and healthcare. We train GatorTronGPT using 82 billion words of de-identified clinical text 15 from University of Florida (UF) Health and 195 billion diverse English words from the Pile 16 dataset. We train GatorTronGPT from scratch using the GPT-3 17 architecture. We formulate biomedical relation extraction and question answering using a unified text generation architecture 18 to evaluate how GatorTronGPT could benefit medical research using 6 benchmark datasets. To examine the utility of text generation in the clinical domain, we apply GatorTronGPT to generate 20 billion words of synthetic clinical text, which are used to train synthetic NLP models using BERT 19 architecture, denoted as GatorTronS (‘S’ stands for synthetic). We compare GatorTronS models with GatorTron 15 , a clinical NLP model trained using real-world 90 billion words of text, to test the hypothesis that generative clinical LLMs can be used to generate synthetic clinical text for medical research. To test if LLMs could be used in healthcare, two internal medicine subspecialists from endocrinology (NSO) and cardiology (MMA) manually evaluate clinical paragraphs written by GatorTronGPT compared with real-world paragraphs written by UF Health physicians. Figure 1 shows an overview of the study design. This study provides valuable insights into the opportunities and challenges of LLMs for medical research and healthcare.

a Train GatorTronGPT from scratch using GPT-3 architecture with up to 20 billion parameters. b Solve biomedical relation extraction and question answering using a unified P-tuning base text generation architecture. c Apply GatorTronGPT to generate 20 billion words of synthetic clinical text, which was used to train synthetic natural language processing model, GatorTronS. d Turing evaluation of 30 paragraphs of text written by GatorTronGPT mixed with 30 real-world paragraphs written by UF Health physicians. TrM transformer unit; B billion.

Training of GatorTronGPT from scratch

Training the 5 billion GatorTronGPT model used approximately 6 days and the 20 billion model used about 20 days on 560 A100 80 G GPUs from 70 NVIDIA DGX nodes using the NVIDIA SuperPOD reference cluster architecture. Figure 2 shows the training and validation loss. Table 1 compares GatorTronGPT with GatorTronS and GatorTron on model architecture, training dataset, parameter size, and whether the model is a generative LLM, to help differentiate the three LLMs.

a Training loss. b Validation loss.

GatorTronGPT for Biomedical natural language processing

Table 2a compares GatorTronGPT with four existing biomedical transformer models on end-to-end relation extraction of drug-drug interaction, chemical-disease relation, and drug-target interaction. GatorTronGPT outperformed all existing models, with the best F1-score of 0.500, 0.494, and 0.419, respectively. GatorTronGPT improved state-of-the-art by 3–10% compared with the second-best BioGPT 18 model. We consistently observed performance improvement when scaling up the size of GatorTronGPT. Table 2b compares GatorTronGPT with six existing biomedical transformers using three benchmark datasets for biomedical question answering. The GatorTronGPT model with 20 billion parameters tied with BioLinkBERT on the MedQA dataset achieving the best performance of 0.451. GatorTronGPT also achieved the second-best performance of 0.776 for the PubMedQA dataset compared with the best performance of 0.782 from BioGPT. The performance of GatorTronGPT on the MedMCQA dataset was lower than a much larger LLM, Galactica, with 120 billion parameters.

Evaluation of GatorTronS

Tables 3 and 4 compare GatorTronS trained with different sizes of synthetic clinical text with ClinicalBERT and GatorTron 15 . For clinical concept extraction, GatorTronS, trained using 20 billion and 5 billion synthetic clinical text, achieved the best F1-score for the three benchmark datasets. GatorTronS outperformed the original GatorTron model by >1% F1-score on all three benchmark datasets. For medical relation extraction, the GatorTronS trained using 10 billion synthetic clinical text achieved the best F1-score of 0.962 on the 2018 n2c2 challenge benchmark dataset, which is comparable with the original GatorTron model (0.960). For semantic textual similarity and natural language inference, GatorTronS achieved the best evaluation scores, outperforming the original GatorTron by >1%. For question answering using emrQA dataset, GatorTronS outperformed the original GatorTron model trained using real-world clinical text by >1%. The comparison results show that a minimum of 5 billion words of synthetic clinical text are required to train a synthetic model with comparable performance to GatorTron, a transformer trained using 82 billion words of real-world UF Health clinical text. Figure 3 compares GatorTronS models trained with different sizes of synthetic text using line plots. We observed consistent performance improvements from all eight datasets by increasing the size of synthetic text from 1 billion to 5 billion words. The improvements are not consistent when increasing the data size from 5 billion up to 20 billion words.

B billion words of text.

Physicians’ Turing test

The Turing test results show that, on average, less than half (49.2%) of the clinical notes were identified correctly, including 36.7% of the synthetic notes and 61.7% of the human notes (Table 5a ). Among the 30 synthetic notes written by GatorTronGPT, 9 (30.0%) and 13 (43.4%) were correctly labeled as ‘AI’ by the two physicians, respectively. Among the 30 human notes written by physicians, 17 (56.7%) and 20 (66.7%) were correctly labeled as ‘Human’, respectively. Considering GatorTronGPT was considered as a human for more than 30% of the instances (the criteria from Turing test) 20 , GatorTronGPT passed the Turing test ( p  < 0.001). Table 5b summarizes the means and standard deviations of the linguistic readability and clinical relevance and consistency. Statistical tests show that there is no significant difference between notes written by GatorTronGPT and human physicians in both linguistic readability ( p  = 0.22) and clinical relevance and consistency ( p  = 0.91). Table 5c shows two examples written by GatorTronGPT; more examples are provided in Supplementary Table S1 . Percent agreement and interrater reliability were found to be good or excellent, as summarized in Supplementary Tables S2 and S3 .

This study develops a generative clinical LLM, GatorTronGPT, using the GPT-3 architecture 13 with 277 billion words of mixed clinical and English text. GatorTronGPT achieves state-of-the-art performance for four out of six biomedical NLP benchmark datasets. Our previous GatorTron 15 model, trained using an encoder-only BERT architecture with 8.9 billion parameters, also achieved state-of-the-art performance on six clinical NLP benchmark datasets. The two studies demonstrate the benefit of LLMs for biomedical and clinical research. GatorTronGPT can generate synthetic clinical text for developing synthetic clinical NLP models (i.e., GatorTronS), which achieve better or comparable performance to GatorTron, an NLP model trained using real-world clinical text, demonstrating the utility of synthetic clinical text generation. The physicians’ Turing test show that GatorTronGPT can generate clinical text with comparable linguistic readability and clinical relevance to real-world clinical notes. This study provides valuable insights into the opportunities and challenges of generative LLMs for medical research and healthcare.

We discover an important utility of synthetic clinical text generation. To date, there has been a gap in accessing and sharing large-scale clinical text and clinical LLMs due to the sensitive nature of clinical text and the fact that automatic de-identification systems cannot remove 100% protected health information (PHI). Not surprisingly, a recent study 21 on clinical foundation models point out that most LLMs in the medical domain are trained using “small, narrowly-scoped” clinical dataset with limited note types (e.g., MIMIC 22 ) or “broad, public” biomedical literature (e.g., PubMed) that has limited insights to healthcare. Generative LLMs can provide large-scale synthetic clinical text to fill the gap. We compare the synthetic text with real-world clinical text to examine why GatorTronS, a transformer model trained using a much smaller (e.g., 5 billion words) synthetic clinical text corpus, could achieve better or comparable performance to GatorTron 15 , a transformer model trained using a much larger (90 billion words) real-world clinical text corpus. We identify potential reasons including (1) real-world clinical text has significant redundancies, which is a well-known characteristic of clinical narratives 23 , and (2) GatorTronGPT generates more diverse synthetic clinical text. We randomly sample a subset of real-world clinical notes with number of words comparable to the synthetic text (i.e., 20 billion words) to compare the coverage of unigrams (i.e., individual tokens) and bigrams (i.e., two consecutive tokens). The comparison results show that the synthetic text generated by GatorTronGPT contain remarkably more diverse unigrams (40.43 million : 4.82 million, ratios are reported as “synthetic” : “real notes”) and bigrams (416.35 million : 62.51 million); the synthetic text also has higher entropy than the real-world clinical text (4.97: 4.95). Supplementary Table S4 provides detailed comparison results and examples. A previous study 24 has reported that by augmenting real-world clinical training data using additional human annotated synthetic text generated by a smaller generative LLM, GPT-2, NLP models can achieve better performance. Our study further demonstrates that, without additional human annotation and augmentation of training data, a larger clinical GPT-3 model can generate synthetic clinical text to train synthetic NLP models outperforming NLP models trained using real-world clinical text. Text generation using generative LLMs could mitigate the risk of exposing patient privacy and improve accessing and sharing of large-scale clinical text and NLP models, thus enabling the next generation of clinical text analytics using synthetic clinical text.

Generative LLMs aspire to become a “Unified Field Theory” to unify most fundamental NLP tasks using a single model architecture. It might be still early to judge if LLMs will become the one and only foundation model 12 for NLP, but it looks like we are closer than ever. Generative LLMs have the potential to impact medical research in many aspects. In addition to performance improvement demonstrated in this study, generative LLMs provide a unified solution using prompt-based text generation 25 , which leads to a new paradigm of “one model for all NLP tasks” and has better few-shot learning and transfer learning ability to deliver portable clinical NLP systems 13 , 26 . The evaluation of GatorTronGPT shows that clinical LLMs can be used to generate clinical-relevant content with the potential to help document 3 and code patient information in EHR systems, thus reducing the extensively onerous documentation burden for clinicians 27 , 28 , 29 . The prompt-based text generation of LLMs can potentially help compose treatment plans by integrating instructions from clinical guidelines and patients’ historical records in EHRs. The conversational ability of LLMs provides opportunities to develop intelligent EHR systems with human-like communication 2 , where healthcare providers, patients, and other stakeholders can communicate in an intelligent electronic health record (EHR) system. Industry stakeholders such as Epic and Nuance have been reported to be exploring these potentials 30 , 31 .

Our Turing test focuses on (1) linguistic readability; (2) clinical relevance; and (3) physicians’ ability to differentiate synthetic and human notes. The statistical tests show that there are no significant differences in linguistic readability ( p  = 0.22; 6.57 of GatorTronGPT compared with 6.93 of human) or clinical relevance ( p  = 0.91; 7.0 of GatorTronGPT compared with 6.97 of human). Further, physicians cannot differentiate them ( p  < 0.001), suggesting the potential utility of GatorTronGPT for text generation in healthcare. Two physician evaluators find that the texts written by GatorTronGPT generally lack clinical logic, indicating that more research and development are needed to make this technology mature for healthcare. Our Turing test focuses on statistical differences not utility in real-world clinical practice, which should be examined in future studies when this technology matures. A recent study 32 examined an LLM developed at New York University, i.e., NYUTron, and our previously developed GatorTron 15 for prediction of readmission, in-hospital mortality, comorbidity, length of stay, and insurance denial, demonstrating the potential utility of LLMs in healthcare.

While LLMs are promising for healthcare applications, much more research and development are needed to achieve this goal. Current general-purpose LLMs are designed for conversation as a chatbot outside of healthcare. Therefore, the current use of ChatGPT for healthcare is more like a typical case of intended use versus actual use as described in the medical device regulation 33 . Domain-specific LLMs are needed for clinical applications. Due to the noisy data and probabilistic nature of text generation, LLMs are prone to confabulation or hallucination, which is dangerous for healthcare. In this study, we adopted robust decoding strategies (e.g., nucleus sampling) to alleviate potential off-target text generation. Researchers are exploring solutions such as reinforcement learning from human feedback (RLHF) 34 to reduce hallucinations, but it is still a not yet solved limitation of current LLMs. Future studies should explore strategies to better control the hallucinations at a minimal level to ensure the safety of using LLMs in healthcare. The security and risk of LLMs must be carefully examined in healthcare settings. We applied a de-identification system to remove PHIs from UF Health notes before training GatorTronGPT, future studies should carefully examine if GatorTronGPT has potential risk of speaking out PHIs and quantify the potential risk of re-identify real-world patients. Synthetic data, though generated by AI models, may still mirror the characteristics of its source material (e.g., UF health clinical notes). For example, ChatGPT has been reported to accidentally leak sensitive business data from a private company 35 . In addition, people are increasingly aware of the potential bias of AI applications in healthcare. Bias inherited from the original training data may be imitated and sometimes even amplified by AI models, which may cause systematic bias to specific patient groups 36 . Future studies should explore strategies to mitigate potential bias and ensure fairness of LLM applications. Like any medical AI applications, it is necessary to carefully examine this disruptive new technology to guide its application and make it “approved ” AI-enabled medical tool 37 .

We developed GatorTronGPT using 82 billion words of de-identified clinical text 15 from the University of Florida (UF) Health and 195 billion diverse English words from the Pile 16 dataset. We trained GatorTronGPT from scratch using the GPT-3 17 architecture (used by ChatGPT). We formulated biomedical relation extraction and question answering using a unified text generation architecture 18 and evaluated GatorTronGPT using 6 biomedical benchmark datasets. To examine the utility of text generation, we applied GatorTronGPT to generate 20 billion words of synthetic clinical text, which were used to train synthetic NLP models, denoted as GatorTronS (“S” stands for synthetic). We compared GatorTronS with GatorTron 15 , a clinical NLP model trained with the same architecture but using real-world clinical text. To test if LLMs could generate text for healthcare settings, two internal medicine subspecialists from endocrinology (NSO) and cardiology (MMA) manually evaluated 60 clinical paragraphs including 30 paragraphs written by GatorTronGPT randomly mixed with 30 real-world paragraphs written by UF Health physicians. Figure 1 shows an overview of the study design.

Data source

This study used 82 billion words of clinical narratives from UF Health Integrated Data Repository (IDR) and 195 billion words of diverse English words from the Pile 16 corpus. This study was approved by the University of Florida Institutional Review Board under IRB202102223; the need for patient consent was waived. At UF Health, we collected approximately 290 million clinical notes from 2011–2021 from over 126 departments, approximately 2 million patients and 50 million encounters from inpatient, outpatient, and emergency settings 15 . We merged the UF Health clinical corpus with the Pile 16 dataset to generate a large corpus with 277 billion words. We performed minimal preprocessing for the Pile dataset and applied a de-identification system to remove 18 PHI categories defined in the Health Insurance Portability and Accountability Act (HIPAA) from the UF Health notes.

Preprocessing and de-identification of clinical text

Following our previous study 15 , we performed a minimal preprocessing procedure. First, we removed all empty notes and the notes with less than 10 characters followed by performing a deduplication at the note level using the exact string match strategy. Then, we leveraged an internally developed preprocessing tool ( https://github.com/uf-hobi-informatics-lab/NLPreprocessing ) to normalize the clinical text. The normalization processing consists of three steps including (1) unifying all text into UTF-8 encoding, removing illegal UTF-8 strings, and removing HTML/XML tags if any; (2) sentence boundary detection where we normalize the clinical notes into sentences; (3) word tokenization where we used heuristic rules to separate punctuation and special symbols (e.g., slash, parenthesis) from words (e.g., converting “(HbA1c)” to “(HbA1c)” and “excision/chemo” to “excision/chemo”) and fixing concatenations (e.g., missing white space like converting “CancerScreening ” to “Cancer Screening”). After preprocessing, we performed another deduplication at the sentence level using the exact string match strategy.

To de-identified the UF Health clinical notes, we adopted an internally developed de-identification system which consists of an LSTM-CRFs based model and a postprocessing module replacing system-detected protected health information (PHI) entities with dummy strings (e.g., replace patients’ names with [**NAME**]). We adopted the safe-harbor method to identify 18 PHI categories defined in the Health Insurance Portability and Accountability Act (HIPAA). The LSTM-CRFs model for PHI detection was trained using the publicly available 2014 i2b2 de-identification datasets and an internal dataset with over 1100 clinical notes from UF Health annotated for PHI removal (named as UF-deid-dataset; not publicly available due to IRB restrictions). After three years of continuous customization and improvement at UF Health, the current model achieved an overall F1 score of 97.98% (precision of 96.27% and recall of 99.76%) on the UF-deid-dataset test set, which means our de-identification system can remove 99.76% of all PHIs. Detailed information about the development of the de-identification system can be accessed from our previous paper 38 .

Train GatorTronGPT from scratch

We trained GatorTronGPT using 5 billion parameters and 20 billion parameters and determined the number of layers, hidden sizes, and number of attention heads according to the guidelines for optimal depth-to-width parameter allocation proposed by ref. 39 as well as our previous experience in developing GatorTron 15 . The 5 billion model has 24 layers, hidden size of 4,096, and number of attention heads of 32; the 20 billion model has 44 layers, hidden size of 6144, and number of attention heads of 48. We trained the 5 billion model using a 2-way tensor model parallel with a batch size of 1120 and learning rate of 1.200E-05. We trained the 20 billion model using an 8-way tensor model parallel with a batch size of 560 and a learning rate of 1.000E-05. We adopted a dropout rate of 0.1. We inherited the GPT-3 architecture implemented in the MegaTron-LM 40 and trained GatorTronGPT models from scratch with the default GPT-3 loss function 13 . We used a total number of 560 NVIDIA DGX A100 GPUs from 70 superPOD nodes at UF’s HiPerGator-AI cluster to train GatorTronGPT by leveraging both data-level and model-level parallelisms implemented by the Megatron-LM package 40 . (See https://github.com/NVIDIA/Megatron-LM for more details) We monitored the training progress by training loss and validation loss using 3% of the data and stopped the training when there was no improvement.

GatorTronGPT for biomedical relation extraction and question answering

End-to-end relation extraction is an NLP task to identify the triplets < concept1, concept2, relation > from biomedical text. Question answering is to identify the answer for a given question and the context . Following previous studies 18 , 41 , we approached the two tasks using a unified prompt-based text generation architecture. Specifically, we adopted a fixed-LLM prompt-tuning strategy 42 to attach a continuous embedding (i.e., virtue tokens) to the input sequence [ virtual tokens; x; y ] as a soft prompt to control the text generation; the LLM was not changed during training. We provide details in the Supplement.

End-to-end biomedical relation extraction

We compared the two GatorTronGPT models with four existing transformer models including GPT-2 43 , REBEL, REBEL-pt 25 , and BioGPT 18 on three biomedical tasks for end-to-end relation extraction using three benchmark datasets including drug-drug interaction 44 (DDI), BioCreative V chemical-disease relation 45 (BC5CDR), and drug-target interaction 46 (KD-DTI).

GPT-2 was trained using text data from 8 million webpages with 1.5 billion parameters, which is a scale-up of the first generation of GPT45 model. The GPT model outperformed previous transformer models on 9 out of 12 NLP tasks, whereas, the GPT-2 model further demonstrated text generation ability, which laid foundation for complex NLP tasks such as machine reading comprehension and question answering.

REBEL and REBEL-pt

REBEL is a transformer model based on the BART architecture designed for end-to-end relation extraction using sequence-to-sequence modeling, which outperformed previous relation extraction models based on classifications. REBEL-pt is an enhanced version of REBEL by further fine-tuning it using the triplets derived using Wikipedia hyperlinks.

BioGPT is a domain-specific generative transformer-based LLM developed using the GPT-2 architecture and the Pubmed biomedical literature, which achieved good performance in NLP tasks including relation extraction and question answering in the biomedical domain.

Following the previous study 18 , we formulated both biomedical relation extraction and question answering as a prompt-based text generation model and applied prompt-tuning (p-tuning) algorithms. We concatenate learnable soft prompts (also called virtual prompt embeddings) with the word embeddings from the context (i.e., input sentence). The sample sequence is constructed as [ prompt , context , relation ], where the prompt is generated using a LSTM model and the relation is the gold standard label including the head entity, tail entity, and their relation type. During the inference, the context and the prompt are used as the input for our GatorTronGPT model to condition and let the model generate the relations. We converted the original relation triplets into a sequence representation. For example, there is an “ agonist ” relation between a drug - “ Igmesine ” and a target “ Opioid receptor sigma 1 ”, which was converted as: “the relation between [ Igmesine ] and [ Opioid receptor sigma 1 ] is [ agonist ] ” . Thus, the relation extraction can be solved as a text generation. During inference, we converted the generated text back to triplets for evaluation. We fine-tuned and evaluated our GatorTronGPT on the end-to-end relation extraction task across four biomedical datasets: BC5CDR (chemical–disease–relation extraction), KD-DTI (drug–target–interaction extraction), DDI (drug–drug–interaction extraction) and 2018 n2c2 (Drug-ADE-relation extraction). The precision, recall, and F1 score were used for evaluation.

Biomedical question answering

We compared GatorTronGPT with six existing transformer models using three widely used benchmark dataset including PubMedQA 47 —a biomedical question answering dataset collected from PubMed abstracts, which requires answering questions with ‘ yes/no/maybe ’ ; MedMCQA 48 —a large-scale multi-choice question answering dataset designed to address real world medical entrance exam questions covering 2400 healthcare topics and 21 medical subjects; and MedQA-USMLE 49 —a multi-choice dataset collected from the professional medical board exams. These datasets have been widely used to evaluate LLMs 18 , 47 , 48 , 49 .

Given a question, a context, and candidate answers, we concatenated the context and the candidate answers into a source sequence and compose the target sequence as: “the answer to the question given possible options is:”, “answer”: “C”. Then, we adopted soft prompts instead of hard prompts (manually designed clear text phrases) in p-tuning. Specifically, we used a randomly initiated continuous embedding as soft prompts, which were fine-tuned in the training. For the PubMedQA dataset, we explored the provided artificially generated text data. Specifically, we automatically labeled the generated text using our p-tuning model developed using the training set and experimented to feedback different proportion of auto-labeled data into training. The best performance was achieved by using 5% of the auto-labeled artificially generated text data. For p-tuning, we used the implementation in NVIDIA NeMo 50 , which is optimized for LLMs. We used the following parameters in our p-tuning: a global batch size of 32, virtual tokens for p-tuning 15, encoder MLP with encoder hidden size of 2048, max sequence length of 4096 for PubMedQA (long abstracts), 2048 for MedMCQA and MedQA-USMLE, and a fused Adam optimizer with a learning rate of 1e-4 and a weight decay of 0·01, betas of 0·9 and 0·98, a cosine annealing scheduler monitoring validation loss with a 50 step warm up. For example, the below is a prompt we used for MedQA-USMLE.

{“taskname”: “usmle-qa”, “prompt”: “QUESTION: A 23-year-old man comes to the physician for evaluation of decreased hearing, dizziness, and ringing in his right ear for the past 6 months. Physical examination shows multiple soft, yellow plaques and papules on his arms, chest, and back. There is sensorineural hearing loss and weakness of facial muscles bilaterally. His gait is unsteady. An MRI of the brain shows a 3-cm mass near the right internal auditory meatus and a 2-cm mass at the left cerebellopontine angle. The abnormal cells in these masses are most likely derived from which of the following embryological structures?\nMULTIPLE CHOICES: (A) Neural tube\n(B) Surface ectoderm\n(C) Neural crest\n(D) Notochord\nTARGET: the answer to the question given possible options is: “, “answer”: “C”}

GatorTronGPT for synthetic clinical text generation

We sought to test the hypothesis that LLMs can generate synthetic clinical text to train synthetic NLP models useful for medical research. We applied GatorTronGPT to generate synthetic clinical text according to a set of seeds without any fine-tuning, which is a typical zero-shot learning setting. Then, using the generated synthetic clinical text, we trained synthetic transformer-based NLP models using our previous BERT-based GatorTron architecture 15 , denoted as GatorTronS (‘S’ stands for synthetic). We trained GatorTronS models using different sizes of synthetic clinical text and compared them with the original GatorTron model trained using UF Health clinical text. To make it comparable, we trained GatorTronS using the same architecture and number of parameters (i.e., 345 million) as GatorTron 15 . We provide detailed information in the Supplement.

Synthetic clinical text generation

Following previous studies 51 , we approached synthetic clinical text generation using an iterative sampling algorithm and applied top-p (i.e., nucleus sampling) sampling and temperature sampling to balance the diversity and quality of text generation 51 . We approached the synthetic clinical text generation as an open-ended text-to-text generation task 52 , 53 , where the generated clinical text is restricted by the context (e.g., the prompts). Specifically, given a sequence of \(m\) tokens \({{X}_{{pre}}=x}_{1}{x}_{2}...{x}_{m}\) as input context, the task is to generate the next \(n\) continuation tokens \({{X}_{{cont}}=x}_{m+1}{x}_{m+2}...{x}_{m+n}\) until reaching the max length of 512 tokens. We generate text through iteratively sampling from the pre-trained language model GatorTronGPT one token at a time by conditioning on the preceding context:

where \(P({x}_{i}|{x}_{1}\ldots {x}_{i-1})\) is the next token distribution. We adopt Top-p (nucleus) sampling 54 during sampling to select words whose cumulative probability exceeds a predefined threshold p .

where \({V}^{(p)}\) is the top-p vocabulary used to sample the next word. This approach dynamically adapts the number of words considered at each step based on their probabilities, balancing diversity and coherence of the generated text.

We set the parameter of top-p sampling at 0.9 and the parameter for temperature sampling at 1.2 according to our empirical assessment. We sampled the beginning 15 tokens from all sections of the de-identified notes from the MIMIC III database 22 and generated approximately 8 million prompts. We also tried several random seeds in GatorTronGPT to generate multiple documents from one prompt. We controlled GatorTronGPT to generate a maximum length of 512 tokens.

Synthetic NLP model development

We applied GatorTronGPT to generate different sizes of synthetic clinical text including 1 billion, 5 billion, 10 billion, and 20 billion words of clinical text and developed corresponding synthetic NLP models, denoted as GatorTronS. Following our previous study 15 , we trained GatorTronS using the same architecture of GatorTron – a BERT architecture with 345 million parameters.

Comparison with existing transformer models

We compared GatorTronS models with ClinicalBERT 55 —an existing clinical transformer model and GatorTron 15 , the current largest clinical transformer model trained using >90 billion words of text, using 5 clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference, and question answering.

Turing test of text generation for healthcare settings

We randomly sampled 30 narrative sections from real-world UF Health clinical notes, including “past medical history”, “history of present illness”, “assessment/plan”, and “chief complaint”. For each of the 30 sections, we extracted the beginning 15 tokens as a seed for GatorTronGPT to generate a synthetic paragraph up to 512 tokens. We cut off the 30 real-world clinical sections to 512 tokens, removed all format information, and randomly mixed them with 30 synthetic sections written by GatorTronGPT. Two UF Health physicians (NSO, MMA) manually reviewed the 60 paragraphs of notes to evaluate: (1) linguistic readability on a 1(worst) to 9 (best) scale, (2) clinical relevance and consistency on a 1 to 9 scale, (3) determine if it was written by a human physician or GatorTronGPT. Percent agreement and Gwet’s AC 1 were calculated to evaluate interrater reliability 56 .

Data availability

The benchmark datasets that support the findings of this study are available from the official websites of natural language processing challenges with Data Use Agreements. More specifically: 1. i2b2 2010, 2012 datasets and n2c2 2018, 2019 datasets: https://portal.dbmi.hms.harvard.edu/projects/n2c2-nlp/ . 2. MedNLI dataset: https://physionet.org/content/mednli/1.0.0/ . 3. emrQA dataset: https://github.com/panushri25/emrQA#download-dataset . 4. The Pile dataset: https://pile.eleuther.ai/ . 5. UF Health IDR clinical notes are not open to the public due to patient privacy information. The GatorTronS, and GatorTron models are available as open-source resources. The synthetic clinical transformer model, GatorTronS, is available from: https://huggingface.co/UFNLP/gatortronS . The GatorTron model trained using real-world clinical text is available: https://huggingface.co/UFNLP/gatortron-base .

Code availability

The computer codes to train GatorTronGPT models are available from: https://github.com/NVIDIA/Megatron-LM/blob/main/pretrain_gpt.py . The scripts used for data preprocessing, vocabulary training and other utilities are available from: https://github.com/uf-hobi-informatics-lab/GatorTronGPT . The computer codes to train GatorTronS models are available from: https://github.com/NVIDIA/Megatron-LM and https://github.com/NVIDIA/NeMo . The computer codes for preprocessing of text data are available from: https://github.com/uf-hobi-informatics-lab/NLPreprocessing .

Introducing ChatGPT. https://openai.com/blog/chatgpt .

Lee, P., Bubeck, S. & Petro, J. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. N. Engl. J. Med. 388 , 1233–1239 (2023).

Article   PubMed   Google Scholar  

Patel, S. B. & Lam, K. ChatGPT: the future of discharge summaries? Lancet Digit Health 5 , e107–e108 (2023).

Article   CAS   PubMed   Google Scholar  

Ali, S. R., Dobbs, T. D., Hutchings, H. A. & Whitaker, I. S. Using ChatGPT to write patient clinic letters. Lancet Digit Health 5 , e179–e181 (2023).

Hirosawa, T. et al. Diagnostic accuracy of differential-diagnosis lists generated by generative pretrained transformer 3 chatbot for clinical vignettes with common chief complaints: a pilot study. Int. J. Environ. Res. Public Health 20 , 3378 (2023).

Grünebaum, A., Chervenak, J., Pollet, S. L., Katz, A. & Chervenak, F. A. The Exciting Potential for ChatGPT in Obstetrics and Gynecology. Am. J. Obstet. Gynecol . https://doi.org/10.1016/j.ajog.2023.03.009 (2023).

Cascella, M., Montomoli, J., Bellini, V. & Bignami, E. Evaluating the feasibility of ChatGPT in healthcare: an analysis of multiple clinical and research scenarios. J. Med. Syst. 47 , 33 (2023).

Article   PubMed   PubMed Central   Google Scholar  

Azamfirei, R., Kudchadkar, S. R. & Fackler, J. Large language models and the perils of their hallucinations. Crit. Care 27 , 120 (2023).

Straw, I. & Callison-Burch, C. Artificial Intelligence in mental health and the biases of language based models. PLoS One 15 , e0240376 (2020).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Li, H. et al. Ethics of large language models in medicine and medical research. Lancet Digital Health https://doi.org/10.1016/S2589-7500(23)00083-3 (2023).

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large Language Models are Zero-Shot Reasoners. Adv. Neural Inf. Process. Syst . 35 , 22199–213 (2022).

Bommasani, R. et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 (2021).

Brown, T., Mann, B. & Ryder, N. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33 , 1877–1901 (2020).

Google Scholar  

Liu, P. et al. Pre-train, prompt, and predict: a systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55 , 1–35 (2023).

CAS   Google Scholar  

Yang, X. et al. A large language model for electronic health records. NPJ Digit. Med. 5 , 194 (2022).

Gao, L. et al. The Pile: an 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00027 (2020).

Floridi, L. & Chiriatti, M. GPT-3: its nature, scope, limits, and consequences. Minds Mach. 30 , 681–694 (2020).

Article   Google Scholar  

Luo, R. et al. BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief. Bioinform . 23 , bbac409 (2022).

Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) 4171–4186 (Association for Computational Linguistics, 2019). https://doi.org/10.18653/v1/N19-1423 .

Mohammed, M., Khan, M. B. & Bashier, E. B. M. Machine Learning (CRC Press, 2016). https://doi.org/10.1201/9781315371658 .

Wornow, M. et al. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit Med. 6 , 135 (2023).

Johnson, A. E. W. et al. MIMIC-III, a freely accessible critical care database. Sci. Data 3 , 160035 (2016).

Searle, T., Ibrahim, Z., Teo, J. & Dobson, R. Estimating redundancy in clinical text. J. Biomed. Inform. 124 , 103938 (2021).

Li, J. et al. Are synthetic clinical notes useful for real natural language processing tasks: a case study on clinical entity recognition. J. Am. Med. Inform. Assoc. 28 , 2193–2201 (2021).

Huguet Cabot, P.-L. & Navigli, R. REBEL: relation extraction by end-to-end language generation. in Findings of the Association for Computational Linguistics: EMNLP 2021 2370–2381 (Association for Computational Linguistics, 2021). https://doi.org/10.18653/v1/2021.findings-emnlp.204 .

Peng, C. et al. Clinical concept and relation extraction using prompt-based machine reading comprehension. J. Am. Med. Inform. Assoc . https://doi.org/10.1093/jamia/ocad107 (2023).

Gaffney, A. et al. Medical documentation burden among US office-based physicians in 2019: a national study. JAMA Intern. Med. 182 , 564–566 (2022).

Downing, N. L., Bates, D. W. & Longhurst, C. A. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann. Intern. Med. 169 , 50 (2018).

Kroth, P. J. et al. Association of electronic health record design and use factors with clinician stress and burnout. JAMA Netw. Open 2 , e199609 (2019).

Diaz, N. Epic to use Microsoft’s GPT-4 in EHRs. https://www.beckershospitalreview.com/ehrs/epic-to-use-microsofts-open-ai-in-ehrs.html .

Trang, B. We’re getting much more aggressive’: Microsoft’s Nuance adds GPT-4 AI to its medical note-taking tool. https://www.statnews.com/2023/03/20/microsoft-nuance-gpt4-dax-chatgpt/ .

Jiang, L. Y. et al. Health system-scale language models are all-purpose prediction engines. Nature 619 , 357–362 (2023).

Kleesiek, J., Wu, Y., Stiglic, G., Egger, J. & Bian, J. An opinion on ChatGPT in health care-written by humans only. J. Nucl. Med . https://doi.org/10.2967/jnumed.123.265687 (2023).

Ouyang, L. et al. Training language models to follow instructions with human feedback. arXiv [cs.CL] (2022).

Ray, S. Samsung bans ChatGPT among employees after sensitive code leak. Forbes Magazine (2023).

Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 356 , 183–186 (2017).

Center for Devices & Radiological Health. Artificial Intelligence and Machine Learning in Software as a Medical Device. U.S. Food and Drug Administration https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device .

Yang, X. et al. A study of deep learning methods for de-identification of clinical notes in cross-institute settings. BMC Med. Inform. Decis. Mak. 19 , 232 (2019).

Levine, Y., Wies, N., Sharir, O., Bata, H. & Shashua, A. The depth-to-width interplay in self-attention. arXiv [cs.LG] (2020).

Shoeybi, M. et al. Megatron-LM: training multi-billion parameter language models using model parallelism. arXiv [cs.CL] (2019).

Li, X. L. & Liang, P. Prefix-tuning: optimizing continuous prompts for generation. in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) 4582–4597 (Association for Computational Linguistics, 2021). https://doi.org/10.18653/v1/2021.acl-long.353 .

Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H. & Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys. 59 , 1–35 (2023).

Radford A., Wu J., Child R., Luan D. & Amodei D. Language models are unsupervised multitask learners. OpenAI, 1 , (2019)

The ddi corpus: An annotated corpus with pharmacological sub-stances and drug-drug interactions . J. Biomed. Inform . 46 , 914–920 (2013).

Li, J. et al. BioCreative V CDR task corpus: a resource for chemical disease relation extraction. Database (Oxf.) 2016 , baw068 (2016).

Hou, Y. et al. Discovering drug–target interaction knowledge from biomedical literature. Bioinformatics 38 , 5100–5107 (2022).

Jin, Q., Dhingra, B., Liu, Z., Cohen, W. & Lu, X. PubMedQA: a dataset for biomedical research question answering. in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (Association for Computational Linguistics, 2019). https://doi.org/10.18653/v1/d19-1259 .

Singhal, K. et al. Large language models encode clinical knowledge. arXiv [cs.CL] (2022).

Jin, D. et al. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. NATO Adv. Sci. Inst. E Appl. Sci. 11 , 6421 (2021).

NeMo: NeMo: a toolkit for conversational AI. (NVIDIA GitHub).

Holtzman A., Buys J., Forbes M. & Choi Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019).

Clark, E., Ji, Y. & Smith, N. A. Neural text generation in stories using entity representations as context. in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) 2250–2260 (Association for Computational Linguistics, 2018). https://doi.org/10.18653/v1/N18-1204 .

Celikyilmaz, A., Clark, E. & Gao, J. Evaluation of text generation: a survey. arXiv preprint arXiv:2006.14799 (2020).

Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019).

Huang, K., Altosaar, J. & Ranganath, R. ClinicalBERT: modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342 (2019).

Wongpakaran, N., Wongpakaran, T., Wedding, D. & Gwet, K. L. A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: a study conducted with personality disorder samples. BMC Med. Res. Methodol. 13 , 61 (2013).

Download references

Acknowledgements

This study was partially supported by a Patient-Centered Outcomes Research Institute® (PCORI®) Award (ME-2018C3-14754), a grant from the National Cancer Institute, 1R01CA246418, grants from the National Institute on Aging, NIA R56AG069880 and 1R01AG080624, and the Cancer Informatics and eHealth core jointly supported by the UF Health Cancer Center and the UF Clinical and Translational Science Institute. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding institutions. We would like to thank the UF Research Computing team, led by Dr. Erik Deumens, for providing computing power through UF HiperGator-AI cluster.

Author information

Authors and affiliations.

Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, FL, USA

Cheng Peng, Xi Yang, Aokun Chen, William R. Hogan, Elizabeth A. Shenkman, Yi Guo, Jiang Bian & Yonghui Wu

Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, FL, USA

Xi Yang, Aokun Chen, Yi Guo, Jiang Bian & Yonghui Wu

NVIDIA, Santa Clara, CA, USA

Kaleb E. Smith, Nima PourNejatian, Anthony B. Costa, Cheryl Martin & Mona G. Flores

Research Computing, University of Florida, Gainesville, FL, USA

Integrated Data Repository Research Services, University of Florida, Gainesville, FL, USA

Tanja Magoc & Gloria Lipori

Lillian S. Wells Department of Neurosurgery, Clinical and Translational Science Institute, University of Florida, Gainesville, FL, USA

Gloria Lipori & Duane A. Mitchell

Division of Endocrinology, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA

Naykky S. Ospina

Division of Cardiovascular Medicine, Department of Medicine, College of Medicine, University of Florida, Gainesville, FL, USA

Mustafa M. Ahmed

You can also search for this author in PubMed   Google Scholar

Contributions

Y.W., J.B., X.Y., N.P., A.B.C., and M.G.F. were responsible for the overall design, development, and evaluation of this study. X.Y., C.P., A.C., and K.E.S. had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Y.G. and Y.W. designed the Turing evaluation of synthetic clinical text generated by GatorTronGPT. N.S.O. and M.M.A. are the two human physicians who performed Turing test. Y.W., X.Y., K.E.S., C.P., Y.G., and J.B. did the bulk of the writing, W.H., E.A.S., D.A.M., T.M., C.A.H., A.B.C., and G.L. also contributed to writing and editing of this manuscript. All authors reviewed the manuscript critically for scientific content, and all authors gave final approval of the manuscript for publication.

Corresponding author

Correspondence to Yonghui Wu .

Ethics declarations

Competing interests.

K.E.S., N.P.N., A.B.C., C.M., and M.G.F. are employed by NVIDIA. There are no other competing financial or non-financial interests. The work presented in this study was conducted exclusively within the University of Florida Health.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Xi Yang finished this work when he was a full-time employee at the University of Florida.

Supplementary information

Supplementary information, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and Permissions

About this article

Cite this article.

Peng, C., Yang, X., Chen, A. et al. A study of generative large language model for medical research and healthcare. npj Digit. Med. 6 , 210 (2023). https://doi.org/10.1038/s41746-023-00958-w

Download citation

Received : 05 June 2023

Accepted : 01 November 2023

Published : 16 November 2023

DOI : https://doi.org/10.1038/s41746-023-00958-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Translational Research newsletter — top stories in biotechnology, drug discovery and pharma.

This paper is in the following e-collection/theme issue:

Published on 17.11.2023 in Vol 25 (2023)

Evaluating Web-Based Care for Mental Health and Substance Use Issues for Lesbian, Gay, Bisexual, Transgender, Queer, Questioning, and 2-Spirit Youths in the Context of the COVID-19 Pandemic: Community-Based Participatory Research Study

Authors of this article:

Original Paper

• Michael Chaiton 1, 2, 3 , PhD   ; 
• Rachel Thorburn 3, 4 , MA   ; 
• Emily Chan 3, 5 * , MSW   ; 
• Ilana Copeland 3, 5 * , MSW   ; 
• Chieng Luphuyong 6 * , MDes   ; 
• Patrick Feng 6, 7 , PhD  

1 Ontario Tobacco Research Unit, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

2 Institute of Medical Science, Temerty Faculty of Medicine, University of Toronto, Toronto, ON, Canada

3 Institute for Mental Health Policy Research, Centre for Addiction and Mental Health, Toronto, ON, Canada

4 Applied Psychology and Human Development, Ontario Institute for Studies in Education, University of Toronto, Toronto, ON, Canada

5 Factor Inwentash School of Social Work, University of Toronto, Toronto, ON, Canada

6 Ontario College of Art & Design, Toronto, ON, Canada

7 Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada

*these authors contributed equally

Corresponding Author:

Rachel Thorburn, MA

Applied Psychology and Human Development

Ontario Institute for Studies in Education

University of Toronto

252 Bloor St W

Toronto, ON, M5S 1V6

Phone: 1 519 702 4000

Email: [email protected]

Background: Mental health (MH) and substance use (SU) care supports are often difficult to access for the lesbian, gay, bisexual, transgender, queer, questioning, and 2-spirit (LGBTQ2S+) population. There is little known on how the shift to web-based care has affected and changed the experiences of LGBTQ2S+ youths within the MH care system.

Objective: This study sought to examine how web-based care modalities have affected access to care and quality of care for LGBTQ2S+ youths seeking MH and SU services.

Methods: Researchers used a web-based co-design method to explore this population’s relationship with MH and SU care supports, focusing on the experiences of 33 LGBTQ2S+ youths and their relationship with MH and SU supports during the COVID-19 pandemic. A participatory design research method was used to gain experiential knowledge of LGBTQ2S+ youths’ lived experience with accessing MH and SU care. Thematic analysis was used to examine the resulting audio-recorded data transcripts and create themes.

Results: Themes related to web-based care included accessibility, web-based communication, provision of choice, and provider relationship and interactions. Barriers to care were identified in particular for disabled youths, rural youths, and other participants with marginalized intersecting identities. Unexpected benefits of web-based care were also found and emphasize the idea that this modality is beneficial for some LGBTQ2S+ youths.

Conclusions: During the COVID-19 pandemic, a time where MH- and SU-related problems have increased, programs need to reevaluate current measures so that the negative effects of web-based care modalities can be reduced for this population. Implications for practice encourage service providers to be more empathetic and transparent when providing services for LGBTQ2S+ youths. It is suggested that LGBTQ2S+ care should be provided by LGBTQ2S+ folks or organizations or service providers who are trained by LGBTQ2S+ community members. Additionally, hybrid models of care should be established in the future so that LGBTQ2S+ youths have the option to access in-person services, web-based ones, or both as there can be benefits to web-based care once it has been properly developed. Implications for policy also include moving away from a traditional health care team model and developing free and lower-cost services in remote areas.

Introduction

Lesbian, gay, bisexual, transgender, queer, questioning, and 2-spirit (LGBTQ2S+) youths and young adults are at higher risk of developing several mental health (MH) conditions, including suicidality and substance use (SU)–related risks and harms [ 1 , 2 ]. Prepandemic, sexual and gender minority youths have been shown to experience higher rates of emotional distress and discrimination due to the heteronormativity of medical and social service systems along with stigma and perceived homophobia from their familial and social support networks [ 1 ]. The effects of this discrimination have been exacerbated by the recent COVID-19 pandemic. Craig et al [ 3 ] found that “pandemics create a perfect storm for vulnerable populations by exacerbating existing stressors and eliminating access to services that are urgently needed.” A Canadian study conducted by Egale Canada and INNOVATIVE research group reported that a higher proportion (42%) of LGBTQ2S+ participants was experiencing significant negative impacts on their physical health, MH, and quality of life during the pandemic, as compared to the general public (30%) [ 4 ].

During this time, LGBTQ2S+ youths and young adults are especially vulnerable because they are having to quarantine in potentially hostile family environments, which can increase rates of domestic abuse and decrease access to web-based services [ 5 - 8 ]. Physical distancing measures or “stay-at-home” orders to reduce community transmission of COVID-19 have worsened distal minority stress for LGBTQ2+ youths, meaning external stressors created through discriminatory events such as family rejection due to one’s gender or sexual identity [ 7 ]. The Trevor Project’s third annual survey taken during the course of the pandemic found that “42% of LGBTQ2S+ respondents seriously considered attempting suicide in the past year, including more than half of transgender and non-binary youth” [ 9 ].

During the COVID-19 pandemic, in-person MH and SU services for LGBTQ2S+ youths shifted to offer web-based care [ 3 ]. Anonymous text-based platforms such as TrevorSpace and Discord Servers have also been providing web-based support to LGBTQ2S+ youths, both before and during the pandemic [ 6 ]. Some interventions such as cognitive behavioral therapy and supportive peer groups have shown to be effective when delivered on the web [ 1 , 10 ]. However, other treatment modalities have proven unsuccessful when delivered in a web-based format [ 5 ]. More research is needed to understand the impact of web-based care in this population.

There exists a large body of research that calls on policy makers and service providers to improve LGBTQ2S+ youths’ and young adults’ access to web-based care because of the unique gaps that have arisen from the COVID-19 pandemic [ 11 ]. Our research seeks to improve access to MH and SU care services for LGBTQ2S+ youths and young adults in the context of the pandemic. This paper examines how web-based care modalities introduce or exacerbate challenges for LGBTQ2S+ youths to access appropriate MH and SU care supports.

Study Design

This study is embedded in a larger research program entitled “Sexual & Gender Minority Youth Access to Services in Mental Health During the COVID-19 Pandemic” (S.M.A.S.H COVID), a project based out of the Centre for Addiction and Mental Health, a psychiatric teaching hospital located in Ontario, Canada. A previous analysis by the research team using survey data from 1404 Canadian LGBTQ2S+ youths indicated that 51.8% of respondents identified a need for MH or SU support during the COVID-19 pandemic but experienced barriers to accessing care [ 12 ]. S.M.A.S.H COVID was a community-based participatory research project that sought to better understand the experiences of LGBTQ2S+ youths who are facing barriers to accessing MH and SU care services in the context of the COVID-19 pandemic. The project was guided by the conceptual framework of grounded theory. Grounded theory involves the construction of hypotheses or theories through the collection or analysis of data [ 13 ]. It is typically used in qualitative social sciences research to understand social processes from the perspective of individuals involved in those processes [ 13 ]. Grounded theory was chosen for S.M.A.S.H COVID because the research team wanted to understand the barriers LGBTQ2S+ youths were experiencing from the perspectives of the youths themselves and to listen and learn from their experiences and ideas rather than come in with preconceived ideas about the issues participants faced. Every step of the research process, from conception to analysis, included LGBTQ2S+ youths on the research and facilitation team.

S.M.A.S.H COVID used a design charrette model or a series of intensely focused, multiday sessions with activities designed to collaboratively elucidate barriers to care and create realistic and achievable solutions [ 14 ]. The design charrette model was chosen for this project because it is a collaborative approach to generating sustainable solutions to complex problems [ 14 ]. The specific research activities within the design charrette were designed by graduate students from OCAD University who identified as LGBTQ2S+ youths or allies. Eight facilitators led the sessions, including 5 LGBTQ2S+ youths and 3 allies. Facilitators were members of the research team, OCAD graduate students, or youth engagement facilitators from Centre for Addiction and Mental Health.

In total, 33 LGBTQ2S+ youths aged 16-29 years and located in Ontario or Quebec participated in the program, held digitally on Webex, over 3 sessions in June 2021. In total, 33 was chosen as our sample size based on recommendations for design charrettes and thematic analysis that suggest picking a sample size that is small enough to manage the data and large enough to demonstrate patterns, with some sources indicating the ideal size should be between approximately 30 and 40 [ 14 - 16 ]. See Table 1 for participant demographics collected through a short survey completed by participants prior to beginning the sessions. Note that this sample was previously discussed and described in [ 17 ]. All participants had accessed or tried to access MH or SU care during the COVID-19 pandemic. Participants were not asked to disclose information about the specific types of support they had sought. Participants were recruited through email invitation to the Public Health Agency of Canada 2SLGBTQI+ Campaign on Commercial Tobacco Use and Its Culture cohort network and by referent recruitment through research team members.

This analysis focuses on the first session, where participants engaged in an activity called “journey mapping.” Journey mapping is a method for visualizing the process that a user goes through in the course of accomplishing a goal [ 18 ]. Originally developed for user-experience and customer-based research, it has been adopted in other sectors, including health care [ 18 ]. Journey mapping was used as a safe and low-threshold engagement activity for participants, where conversation was a mechanism for group knowledge sharing.

Participants were split into 4 groups for a guided hour of discussion on the research question: what barriers exist for LGBTQ2S+ youths and young adults to access identity-affirming mental health care during the COVID-19 pandemic? As participants spoke, a graphic illustrator visually mapped discussed journeys, creating a tangible representation of participants’ shared experiences for each of the 4 groups (for an example of one of the journey maps created, see Figure 1 ).

All journey mapping sessions were audio-recorded. Braun and Clarke’s [ 19 ] thematic analysis was used to analyze the recorded data. Together, 5 members of the research team listened to the audio recordings, including 2 team members who identify as LGBTQ2S+ youths. The researchers then used thematic analysis to identify prominent themes in the data. The audio recordings were then transcribed, and the data were reviewed again by 3 research team members (including 1 LGBTQ2S+ youth) to reassess the appropriateness of each theme and identify relevant quotes. The final draft of this paper was edited and approved by an LGBTQ2S+ youth member of the research team.

Based on this analysis, research team members also created a vignette to illustrate key themes from the journey mapping exercise. The vignette is presented below, as an introduction to this study’s results.

Ethics Approval

This study was approved by The Centre for Addiction and Mental Health Research Ethics Board (approval 039/2021).

Participants spoke about their experiences in accessing MH and SU care during the pandemic. While the research team initially expected participants to describe their journeys accessing care, participants described their experiences more as a series of disconnected events. While each participant’s experience was unique, common themes from the discussion were identified by the research team, which are synthesized in Textbox 1 .

Overall, 4 themes specific to LGBTQ2S+ youths seeking web-based care were identified. In addition to the 4 web-based themes, we identified other general themes that were not specifically related by participants to web-based care but were prominent points of discussion that participants identified as important barriers to accessing MH and SU support. We elaborate on these themes below.

John is an 18-year-old university freshman who started school in 2019. Due to the COVID-19 pandemic in March 2020, all of John’s classes became web-based. John had always struggled with feelings of anxiety and depression in the past; however, these feelings intensified during the pandemic. John decided to speak with a counselor about his experience, though he was nervous due to being misgendered by counselors he had seen in the past. When he called the counseling office, John was not able to speak with anyone directly and was told to leave his contact information and that someone would get back to him (theme: web-based communication). After waiting for 2 weeks with no callback, John emailed the office to follow-up. A week later, the receptionist from the counseling office finally returned John’s email. At this point, John had waited 3 weeks for an initial response from the clinic (themes: web-based communication and accessibility). They told that there would be a 3-month wait for services, that they would not be able to choose their counselor, and that the appointments would be held digitally (theme: provision of choice). John was nervous because his good counseling experiences in the past were due to the relationship he was able to build with his counselor in person. However, he was relieved that it would be easy for him to attend web-based sessions since the counseling office would be difficult for him to access without a car (theme: accessibility).

After waiting for 3 months, John attended their appointment and was repeatedly misgendered by his counselor. The internet also cut out multiple times during the appointment, and John lost 10 minutes of his 50-minute session due to technical issues (themes: web-based communication and accessibility). John felt that he was not able to connect with his counselor over the screen in the same way he had previously with his in-person counselor (theme: provider relationship and interactions). John felt defeated and did not end up making another appointment.

Themes Specific to Web-Based Care

Theme 1: accessibility.

Overall, participants reported that web-based care led to better accessibility to services than that before the COVID-19 pandemic, both in the sense of being physically easier to access and also because many services reduced or waived their fees for web-based care options:

There's these kinds of things that are online now, although they're more accessible, which is wonderful. And some, some of the fees are even being waived, which is so wonderful, too.

LGBTQ2S+ youths from rural communities shared that they traditionally experience reduced access to in-person care and often have to travel far due to a lack of services in their area. Rural youths reported that the move to web-based care reduced barriers associated with traveling into the city for in-person services, allowing them more choices in providers, more options for specialized services, and greater access to service providers who were adequately trained to work with gender and sexually diverse youths.

Disabled youths also reported that web-based care resulted in better access to services than they had experienced before the pandemic. A facilitator noted the following comments made during the conversation by a participant through text chat:

They've been doing online stuff for all their life…so saying something that's online they are accustomed to. And…that worked for them… they were saying it's also been way more exhausting to not have online options, and COVID is honestly the most accessible than it’s ever been in (their) life.

Despite the overall benefit of web-based care for accessibility, disabled youths also noted that web-based accessibility tools need to work for those relying on them. For example, web-based modalities often offer closed captions; however, this service is only helpful when the captions are clear and coherent. For some neurodivergent youths, web-based care also posed unique barriers. Particularly, phone and email services made communication challenging for participants who rely heavily on visual social cues.

Finally, participants noted that web-based services are not accessible to all youths. Specifically, they mentioned that some youths do not have access to the internet or technology devices to access web-based services and that many LGBTQ2S+ youths and young adults do not have a safe or supportive space to access web-based care:

Due to COVID they have just begun doing counseling virtual…but like some folks have moved back home with their parents or are living in rooming houses. For me, I live with my parents, and I'd say that like privacy is a big concern for me. So in some ways, yes, things are more accessible, but there's also a lot of limitations and like concerns regarding how you access things.

Overall, the move to web-based care increased the accessibility of services compared to in-person care, especially for specific subpopulations that face barriers to in-person care, such as rural youths and disabled youths. However, barriers to web-based care were also identified, including challenges navigating less social cues, lack of internet access, and lack of safe or supportive spaces to access web-based care.

Theme 2: Web-Based Communication

Participants reported difficulty communicating and building rapport on the web or over the phone. During email communication, many participants did not feel heard over email and expressed issues with timely responses from providers. When email communication did occur, it felt more formal and impersonal and not as organic or effective as a “natural conversation.” Participants noted that the nature of email as opposed to an in-person interaction meant that they had to self-motivate more to continue to reach out and did not receive feedback right away, which some found challenging.

There's no feedback immediately about what you're talking about. You have to like, think about everything while you're writing that email or while you're getting ready for something. Because when you're with somebody and talking to someone, like naturally, you think of things as it happens. Whereas if you're writing an email, you have to kind of get everything out, and then like, go over it, make sure you've gotten all your points down.

However, some participants benefited from email communication. For example, when having gender-affirming surgery, a participant reported that they preferred being able to email their provider about the surgery and “just being able to disconnect, write it down, and send it off” without worrying about what the provider thought about them.

Phone anxiety was identified as an obstacle to effective care for services that required phone conversations for registration or service provision. Regarding both email and phone conversation, some participants noted that they rely on body language as a communication aid, and the ability to read facial expressions and establish eye contact is not present in the forms of web-based modalities such as emails and phone calls. This was especially challenging for participants who identified as neurodivergent.

With video service modalities, participants felt it was harder to be vulnerable on Zoom. Screen fatigue and exhaustion were common issues that arose, especially for participants who also had work or school on the web. The exhaustion and stress of web-based communication resulted in low motivation to access web-based services for some participants, who reported that the web-based format did not feel as effective as in-person services. However, other participants noted that in-person services could be much more exhausting in different ways and that web-based services were much more accessible to them. In general, participants felt that technology-mediated communication was less personal and identified a lack of human connection with provider interactions during the pandemic as compared to in-person interactions.

Theme 3: Provision of Choice

Another recurring theme noted by participants was the lack of care options presented to them. Participants described the lack of options, especially during the COVID-19 pandemic, as inflexible and unaccommodating and stressed that this one-size-fits-all approach did not work in the context of many of their unique circumstances:

There's like this whole one size fit all kind of thing or, you know, we think this will work for all these people. And then within gender and sexual diversity, there's a whole other, you know, what is affirming care? To who? why?, like, for which identities does that make sense?

Youths described that it was important for the success of their care to be given options such as being able to choose their therapy format (eg, web-based or in-person), to choose their own care provider who could best meet their individual needs, and to be given options about what treatment modalities they felt fit their situation best. Overall, they felt care providers should be taking greater care to consult them in their own treatment process, in order to meet their individual needs.

I wish there would have been, they would have asked me what I think would have helped me or based on like, my culture or my personality, they would have asked me what I think would have been best, instead of just telling you this is what I'm going to do and it will make them better when it did not help.

Theme 4: Provider Relationship and Interactions

Participants experienced several issues while accessing care related to provider relationships and interactions including a perceived lack of provider warmth or compassion from web-based care. Participants described a lack of basic empathy or compassion from providers, which made youths feel like just a number or “someone’s problem” when trying to access care. They felt that providers did not care about them and “like they don't want to know your problems,” and their concerns went unheard as a result. This was reflected in situations such as providers being slow to respond to urgent situations and showing up late or not adequately preparing for client sessions.

I've had contacted with therapists and you know, other support people, via emails via texting, phone calls, and it just feels impersonal. And it's almost like they don't really hear you will have time for you. Because you're just another person on a screen, you're just another email.

Another issue identified was a need for a consistent provider relationship, a challenge made even more difficult by the shift to web-based care. Multiple participants said building relationships with providers was important because it was harder to share sensitive information with service providers without mutual trust and established rapport—a process that was more difficult digitally. Participants said that being “tossed around” from one provider to another made it hard to build relationships and trust and that this made the care-seeking process inefficient. Participants felt that their interactions should be a bond-building process, but instead often felt like web-based interactions were transactional, and that providers were trying to quickly end sessions. They wanted service providers to take the time to build a genuine connection despite the challenges of web-based care. This was further complicated by issues such as a limit to the number of interactions they could have with a provider.

Um, one thing that I know is that when you do connect with people during the pandemic, it's very professional, and it's not as it's not as like, vulnerable as it would be in person because you don't have that bond and connection with people on like, a base level. It's, it feels less because you know, that both people are just sitting, looking at the screen and you know, you, you lose something in that.

Systemic Issues not Specifically Related to Web-Based Care

Participants also spoke to longstanding systemic problems in the provision of MH and SU care for LGBTQ2S+ youths and young adults that affected their journey through the MH system that was independent of web-based care. Although these issues are not pandemic-specific, they are worth noting as ongoing barriers being experienced by gender and sexually diverse youths and young adults who seek affirming care. Furthermore, responses from participants suggest the pandemic may have worsened many of these access-to-care issues.

One issue was provider competency. This included the need for informed, affirming, and educated providers from diverse backgrounds who were understanding of the needs of the LGBTQ2S+ community and intersecting identities. Training should be required for providers to LGBTQ2S+ youths to have a level of care that is consistent with the needs of service users, particularly LGBTQ2S+ and Black, Indigenous, and people of color youths. Service providers in rural areas were identified as being in particular need of this kind of competency, although it was identified as an issue across geographic locations. Participants identified that a lack of provider competency resulted in experiences of discrimination when trying to access care. This was particularly salient in the discussion for trans participants, who shared painful experiences of being misgendered or deadnamed by providers. Participants reported that these experiences of discrimination were detrimental to their MH. Relatedly, participants also identified the need for safer opportunities to provide feedback to providers and for training programs to be followed up with systems of accountability to ensure providers were incorporating their learnings into their practice. Youths wanted to be at the center of their own care and for the therapy process to be one of continuous feedback and consent.

Wait times were reported as a significant access barrier for many participants, especially within rural communities. Participants mentioned the lack of support and transparency during wait times as causing major delays that create disconnects in care and hinder progress along their care journeys. It was noted that the pandemic created further delays to existing waitlists.

Navigating services that were web-based or not can be overwhelming for LGBTQ2S+ youths. MH and SU care often involve multiple providers, and this can be challenging to manage. With limited options, it is hard for youths to know where to start and what is available. For example, one participant mentioned ambiguous labeling like “youth wellness” as not indicative of what kind of services are offered and another said that they were looking for recovery support but were not sure what that should look like (eg, abstinence vs harm reduction). Other difficulties brought up by participants included the need to “self-motivate” to access care, the inability to be present and show up for oneself during a therapy session, trying to meet various learning needs within group settings, and the anxiety of making phone calls after discharge to get follow-up support.

Lack of continuity of care was a significant theme. Participants do not experience a journey—instead, they experience isolated moments of care that are not connected. Despite Ontario’s transition into a “health care team” model, participants viewed the care for LGBTQ2S+ youths as “siloed.” The lack of communication among providers (even in the same care team) burdened participants who had to build new relationships and re-explain their personal situation with strangers.

One participant noted on systems navigation:

I'd also say like, like referrals, or just like setup in general…that whole process can be very discouraging, debilitating, exhausting, and lengthy. Right, and you have to get a GP before you're able to get a psychiatrist, you have to go through all these steps…., and sometimes these things take months and months at a time. And so by the point that you have one step completed, so much has happened to you, and that time, and it's like yeah, I needed your help then and it's almost like I don't even I don't even want anymore.

Principal Findings

The purpose of this study was to answer the question: how do web-based care modalities introduce or exacerbate challenges for LGBTQ2S+ youths to access appropriate MH and SU care supports? Using a web-based co-design method, researchers were able to engage with youths to better understand the barriers to web-based care. The primary set of themes was specific to web-based care and included accessibility, web-based communication, provision of choice, and finally provider relationship and interactions. More general barriers were also identified, including disparities in access, wait times, systems navigation, and continuity of care.

This study is one of the first to highlight the implications of COVID-19 on access to care for this population. Although all youths are affected by many of these barriers, previous research has established that existing social-structural inequalities are being exacerbated by the COVID-19 pandemic, and findings show that LGBTQ2S+ youths are disproportionately affected by this shift to the pandemic-related reliance on web-based care [ 5 ]. Supporting this, participants reported that the nature of informal and formal support has changed leading to consequences such as low motivation, increased screen fatigue, and phone anxiety. Furthermore, participants suggested that some LGBTQ2S+ youths might not have access to technological devices and that the ones who could access them may not have a safe space to do so, leading to further disparities when attempting to access MH and SU services.

Previous research indicates that using web-based care modalities and telehealth can be beneficial if implemented appropriately [ 20 , 21 ], and it was evident that participants did have more access to opportunities and resources through technology and digitally accessible services. The outcomes of this study add to a long-standing body of literature that advocates for the development of web-based care, enabling care seekers to choose which method (web-based, in-person, or hybrid format), treatment (affirmative CBT and peer-support groups), and provider best align with their personal values and perspectives [ 20 , 21 ]. Providers should invest in both web-based care and in-person services so both choices of care are available.

There are a few limitations of this study that are important to note. Because this study follows a unique qualitative design charrette research design, the findings were very experiential. Therefore, there should be discretion when generalizing this study’s findings to broader LGBTQ2S+ populations, as the sample size was limited. A related limitation is that although the sample used was appropriate for the research question, there could be more opportunities to increase the diversity of ethnocultural knowledge and experience. For instance, this study only engaged with participants who spoke English, causing a language barrier for potential participants. While the research team did maximize diversity among its English-speaking participants, more could have been done to engage with non-English speakers and LGBTQ2S+ youths who speak English as a second language.

Another complication is that this study was conducted on a web-based platform, and the sample excludes those who were not able to access technological devices and the internet. This normally includes LGBTQ2S+ youths who are homeless, in foster care, or live in poverty [ 7 ]. As this is an especially vulnerable subpopulation of LGBTQ2S+ youths (especially during the COVID-19 pandemic), including these perspectives would have made the sample size more inclusive. Furthermore, researchers and participants identified internet connection issues as another technological barrier. Some participants were not able to fully access the program due to tenuous internet connections, interfering with their ability to contribute. In the future, a hybrid model of research (web-based and in-person design) could be used to include more youths and increase the accessibility of the study.

Areas of future research could include more community stakeholders, such as service providers, so that more intersectional research on LGBTQ2S+ youths can be carried out. Involving stakeholders can lead to heightened knowledge mobilization into broader society so that mainstream MH and SU organizations can begin to work more with and for this population. Using community-based participatory action research for future studies could use community service providers and young people as key stakeholders, encouraging a greater understanding of the issues and more direct policy implementation.

Additionally, research should be conducted on barriers to access for LGBTQ2S+ adults during the COVID-19 pandemic. This would allow researchers to contrast the results between LGBTQ2S+ youths and adults and observe age, generational factors (eg, millennials vs baby boomers), and potentially technological literacy as factors that can influence these barriers. Research on LGBTQ2S+ adults can contribute to developing shared realities between the entire LGBTQ2S+ population during the pandemic and suggest cross-generational themes that arise when accessing web-based care.

As mentioned earlier, this study has an evident bias that neglected to recruit LGBTQ2S+ youths who did not have access to the internet, technological devices, a safe environment to go to, and those who did not speak English. Future designs need to incorporate the perspectives of youths outside these parameters as it is likely that these subpopulations are experiencing this shift to web-based care more so than the one that was just studied. Web-based interventions and services need to be able to accommodate all LGBTQ2S+ youths as we continue to shift into the era of web-based care.

Conclusions

The purpose of this study was to examine the experiences of LGBTQ2S+ youths accessing web-based MH and SU care during the COVID-19 pandemic. Results suggest that although some youths benefited from the increased accessibility of web-based care, many found web-based care to introduce barriers of its own, such as issues communicating and connecting with service providers digitally. Additionally, challenges such as internet connection and needing a safe space at home to access care were identified. Existing issues such as gender and sexual identity discrimination, systems navigation, and waitlists for care were described as being exacerbated by the abrupt shift to web-based care. Participants stressed the need for more choices in their own care to individualize it to their needs, including choice of provider, modality (web-based or in person), and type of therapy.

The implications of this work are that service providers and the organizations they work for should strive to offer as many options as possible to clients to suit their needs, especially for groups such as LGBTQ2S+ youths who are at increased risk of experiencing systemic barriers and discrimination. Although web-based care has benefits and can increase accessibility for some youths, it presents barriers for others. Offering both in-person and web-based modalities can help clients choose the format that works best for them. Additionally, providers should strive to educate themselves on best practices when working with LGBTQ2S+ youths, in order to prevent issues such as discrimination that are exacerbated by the shift to web-based care.

Conflicts of Interest

None declared.

• Almeida J, Johnson RM, Corliss HL, Molnar BE, Azrael D. Emotional distress among LGBT youth: the influence of perceived discrimination based on sexual orientation. J Youth Adolesc 2009;38(7):1001-1014 [ https://europepmc.org/abstract/MED/19636742 ] [ CrossRef ] [ Medline ]
• Russell ST, Fish JN. Mental health in lesbian, gay, bisexual, and transgender (LGBT) youth. Annu Rev Clin Psychol 2016;12:465-487 [ https://europepmc.org/abstract/MED/26772206 ] [ CrossRef ] [ Medline ]
• Craig SL, Iacono G, Pascoe R. The delivery of technology-mediated affirmative cognitive behavioural therapy groups to LGBTQ+ youth during a pandemic: a practice innovation. Can J Commun Ment Health 2020;39(3):79-83 [ https://doi.org/10.7870/cjcmh-2020-020 ] [ CrossRef ]
• Impact of COVID-19: Canada’s LGBTQI2S community in focus. Egale Canada & Innovative Research Group. 2020. URL: https://tinyurl.com/4rhth5nv [accessed 2023-06-20]
• DeMulder J, Kraus-Perrotta C, Zaidi H. Sexual and gender minority adolescents must be prioritised during the global COVID-19 public health response. Sex Reprod Health Matters 2020;28(1):1804717 [ https://europepmc.org/abstract/MED/32799759 ] [ CrossRef ] [ Medline ]
• Fish JN, McInroy LB, Paceley MS, Williams ND, Henderson S, Levine DS, et al. "I'm kinda stuck at home with unsupportive parents right now": LGBTQ youths' experiences with COVID-19 and the importance of online support. J Adolesc Health 2020;67(3):450-452 [ https://europepmc.org/abstract/MED/32591304 ] [ CrossRef ] [ Medline ]
• Salerno JP, Devadas J, Pease M, Nketia B, Fish JN. Sexual and gender minority stress amid the COVID-19 pandemic: implications for LGBTQ young persons' mental health and well-being. Public Health Rep 2020;135(6):721-727 [ https://europepmc.org/abstract/MED/33026972 ] [ CrossRef ] [ Medline ]
• Bielski Z. The struggles faced by LGBTQ youth during COVID-19 lockdowns: when schools closed abruptly in March, some LGBTQ youth were forced to move back in with unaccepting and homophobic relatives. The Globe and Mail. 2021. URL: https://www.proquest.com/docview/2478740847/9CC80EACA3FD49EFPQ/1?accountid=14771 [accessed 2023-06-20]
• The Trevor Project. New research underscores mental health disparities faced by diverse LGBTQ youth mental health and beyond: the Trevor project's third annual national survey on LGBTQ youth mental health emphasizes the need for greater investment in mental health care and intersectional policy solutions to confront the public health crisis of suicide. PR Newswire. URL: https://www.proquest.com/docview/2528773513/9CC80EACA3FD49EFPQ/5?accountid=14771 [accessed 2023-06-20]
• Craig SL, Austin A. The AFFIRM open pilot feasibility study: a brief affirmative cognitive behavioral coping skills group intervention for sexual and gender minority youth. Child Youth Serv Rev 2016;64:136-144 [ https://doi.org/10.1016/j.childyouth.2016.02.022 ] [ CrossRef ]
• Drabble LA, Eliason MJ. Introduction to special issue: impacts of the COVID-19 pandemic on LGBTQ+ health and well-being. J Homosex 2021;68(4):545-559 [ CrossRef ] [ Medline ]
• Chaiton M, Musani I, Pullman M, Logie CH, Abramovich A, Grace D, et al. Access to mental health and substance use resources for 2SLGBTQ+ youth during the COVID-19 pandemic. Int J Environ Res Public Health 2021 Oct 28;18(21):11315 [ https://www.mdpi.com/resolver?pii=ijerph182111315 ] [ CrossRef ] [ Medline ]
• Noble H, Mitchell G. What is grounded theory? Evid Based Nurs 2016;19(2):34-35 [ https://ebn.bmj.com/content/19/2/34 ] [ CrossRef ] [ Medline ]
• Roggema R. The design charrette. In: Roggema R, editor. The Design Charrette: Ways to Envision Sustainable Futures. Netherlands: Springer; 2014:15-34
• Baker S, Edwards R. How many qualitative interviews is enough. National Centre for Research Methods. 2012. URL: https://eprints.ncrm.ac.uk/id/eprint/2273/ [accessed 2023-02-10]
• Sandelowski M. Sample size in qualitative research. Res Nurs Health 1995;18(2):179-183 [ CrossRef ] [ Medline ]
• Chaiton M, Thorburn R, Sutton M, Feng P. LGBTQ2S+ youth perspectives on mental healthcare provider bias, standards of care, and accountability. Youth 2023 Jan 12;3(1):93-106 [ http:// ] [ CrossRef ] [ Medline ]
• Howard T. Journey mapping: a brief overview. Commun Des Q Rev 2014;2(3):10-13 [ CrossRef ]
• Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3(2):77-101 [ CrossRef ]
• Wosik J, Fudim M, Cameron B, Gellad ZF, Cho A, Phinney D, et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc 2020;27(6):957-962 [ https://europepmc.org/abstract/MED/32311034 ] [ CrossRef ] [ Medline ]
• Hollander JE, Carr BG. Virtually perfect? telemedicine for COVID-19. N Engl J Med 2020;382(18):1679-1681 [ https://www.nejm.org/doi/10.1056/NEJMp2003539 ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T McCall; submitted 14.11.22; peer-reviewed by E Layland, U Aneni; comments to author 29.01.23; revised version received 21.02.23; accepted 11.03.23; published 17.11.23

©Michael Chaiton, Rachel Thorburn, Emily Chan, Ilana Copeland, Chieng Luphuyong, Patrick Feng. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.11.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.