literature based research methodology example

Dissertations & projects: Literature-based projects

• Research questions
• The process of reviewing
• Project management
• Literature-based projects

On these pages:

“As a general rule, the introduction is usually around 5 to 10 per cent of the word limit; each chapter around 15 to 25 per cent; and the conclusion around 5 per cent.” Bryan Greetham, How to Write Your Undergraduate Dissertation

This page gives guidance on the structure of a literature-based project.   That is, a project where the data is found in existing literature rather than found through primary research. They may also include information from primary sources such as original documents or other sources.

How to structure a literature-based project

The structure of a literature-based dissertation is usually thematic, but make sure to check with your supervisor to make sure you are abiding by your department’s project specifications. A typical literature-based dissertation will be broken up into the following sections:

Abstract or summary

Acknowledgments, contents page, introduction, themed chapters.

• Bibliography/Reference list

Use this basic structure as your document plan . Remember that you do not need to write it in the order it will finally be written in. 

For more advice on managing the order of your project, see our section on Project Management.   

If you use the template provided on our Formatting page, you will see that it already has a title page included. You just need to fill in the appropriate boxes by typing or choosing from the drop-down-lists. The information you need to provide is: 

• Type of assignment (thesis, dissertation or independent project)
• Partial or full fulfilment information
• Subject area
• Your name (and previous qualifications if applicable)
• Month and year of submission

This may not always be required - check with your tutor.

• It is  independent  of the rest of the report - it is a mini-report, which needs to make sense completely on its own.
• References should  not  be included.
• Nothing should appear in the abstract that is not in the rest of the report.
• Usually between 200-300 words.
• Write as a  single  paragraph.

It is recommended that you write your abstract  after  your report.

If you choose not to use the template, then you will need to go through the document after it is written and create a list showing which heading is on which page of your document.

Purpose: To thank those who were directly involved in your work .

• Do not confuse the acknowledgements section with a dedication - this is not where you thank your friends and relatives unless they have helped you with your manuscript.
• Acknowledgments are about courtesy, where you thank those who were directly involved in your work, or were involved in supporting your work (technicians, tutors, other students, financial support etc).
• This section tends to be  very brief , a few lines at the most. Identify those who provided you with the most support, and thank them appropriately.
• At the very least, make sure you acknowledge your supervisor!!

Purpose: To state the research problem and give a brief introduction to the background literature, provide justification for your research questions and explain your methodology and main findings.

• Explain what the problem you will be addressing is, what your research questions are, and why they will help address the issue.
• Explain (and justify) your methodology - where you searched, what your keywords were, what your inclusion and exclusion criteria were,
• Define the scope of the dissertation, explaining any limitations.
• Lay out the structure of the dissertation, taking the reader through each section and providing any key definitions.
• Very briefly describe what your main findings are - but leave the detail for the sections below.

It is good practice to come back to the introduction after you have finished writing up the rest of the document to ensure it sets the appropriately scene for subsequent sections.

Should you have a separate literature review chapter?

Not usually , as your project is basically a big literature review, it isn't necessary to have a separate chapter. You would normally introduce background literature in your introduction instead.

However, if your supervisor suggests a separate chapter then it could go at this point, after the main introduction (which would then not include background literature). 

For more advice on writing a literature review see the Literature Review pages on this guide.

Purpose: To present the themes you have identified in your research and explain how they contribute to answering your research questions

You will typically have 3-5 themed chapters. Each one should contain:

• An introduction to the theme - what things it means and what it incorporates.
• How the theme was addressed within the literature - this should be analytical not just descriptive.
• A conclusion which shows how the theme relates to the research question(s).

Ensuring your themed chapters flow

Choosing the order of your theme chapters is an important part of the structure to your project. For example, if you study History and your project covers a topic that develops over a large time period, it may be best to order each chapter chronologically. Other subjects may have a natural narrative running through the themes. Think about how your reader will be able to follow along with your overall argument.

Although each chapter must be dedicated to a particular theme, it must link back to previous chapters and flow into the following chapter. You need to ensure they do not seem like they are unrelated to each other. There will be overlaps, mention these.

Some literature-based projects will focus on primary sources. If yours does, make sure primary sources are at the core of your paragraphs and chapters, and use secondary sources to expand and explore the theme further. 

Purpose: To present the conclusion that you have reached as a result of both the background literature review and the analysis in your thematic chapters

A conclusion summarises all the points you have previously made and it  should not  include any evidence or topics you have not included in your introduction or themed chapters. There should be no surprises.

It should be about 5-10% of your word limit so make sure you leave enough words to do it justice. There will be marks in the marking scheme specifically allocated to the strength of your conclusion which cannot be made up elsewhere.

Some conclusions will also include recommendations for practice or ideas for further research. Check with your supervisor to see if they are expecting either or both of these.

Reference list

It is good practice to develop a reference list whilst  writing the project, rather than leaving it until the end. This prevents a lot of searching around trying to remember where you accessed a particular source. If using primary sources, it also allows you to monitor the balance between primary and secondary sources included in the project. There is software available to help manage your references and the university officially supports RefWorks and EndNote. 

For more advice on reference management, see our Skills Guide: Referencing Software

• Transcriptions
• Correspondence
• Ethical approval forms

If you have information that you would like to include but are finding it disrupts the main body of text as its too cumbersome, or would distract from the main arguments of your dissertation, the information can be included in the appendix section. Each appendix should be focused on one item. 

Appendices  should not include any information that is key to your topic or overall argument. 

• << Previous: Structure
• Next: Scientific >>
• Last Updated: Apr 24, 2024 1:09 PM
• URL: https://libguides.hull.ac.uk/dissertations
• Login to LibApps
• Library websites Privacy Policy
• University of Hull privacy policy & cookies
• Website terms and conditions
• Accessibility
• Report a problem

Methodological Approaches to Literature Review

• Living reference work entry
• First Online: 09 May 2023
• Cite this living reference work entry

• Dennis Thomas 2 ,
• Elida Zairina 3 &
• Johnson George 4  

570 Accesses

1 Citations

The literature review can serve various functions in the contexts of education and research. It aids in identifying knowledge gaps, informing research methodology, and developing a theoretical framework during the planning stages of a research study or project, as well as reporting of review findings in the context of the existing literature. This chapter discusses the methodological approaches to conducting a literature review and offers an overview of different types of reviews. There are various types of reviews, including narrative reviews, scoping reviews, and systematic reviews with reporting strategies such as meta-analysis and meta-synthesis. Review authors should consider the scope of the literature review when selecting a type and method. Being focused is essential for a successful review; however, this must be balanced against the relevance of the review to a broad audience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Institutional subscriptions

Akobeng AK. Principles of evidence based medicine. Arch Dis Child. 2005;90(8):837–40.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66.

Article   PubMed   PubMed Central   Google Scholar  

Arksey H, O’Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.

Article   Google Scholar  

Aromataris E MZE. JBI manual for evidence synthesis. 2020.

Google Scholar  

Aromataris E, Pearson A. The systematic review: an overview. Am J Nurs. 2014;114(3):53–8.

Article   PubMed   Google Scholar  

Aromataris E, Riitano D. Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. Am J Nurs. 2014;114(5):49–56.

Babineau J. Product review: covidence (systematic review software). J Canad Health Libr Assoc Canada. 2014;35(2):68–71.

Baker JD. The purpose, process, and methods of writing a literature review. AORN J. 2016;103(3):265–9.

Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med. 2010;7(9):e1000326.

Bramer WM, Rethlefsen ML, Kleijnen J, Franco OH. Optimal database combinations for literature searches in systematic reviews: a prospective exploratory study. Syst Rev. 2017;6(1):1–12.

Brown D. A review of the PubMed PICO tool: using evidence-based practice in health education. Health Promot Pract. 2020;21(4):496–8.

Cargo M, Harris J, Pantoja T, et al. Cochrane qualitative and implementation methods group guidance series – paper 4: methods for assessing evidence on intervention implementation. J Clin Epidemiol. 2018;97:59–69.

Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997;126(5):376–80.

Article   CAS   PubMed   Google Scholar  

Counsell C. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med. 1997;127(5):380–7.

Cummings SR, Browner WS, Hulley SB. Conceiving the research question and developing the study plan. In: Cummings SR, Browner WS, Hulley SB, editors. Designing Clinical Research: An Epidemiological Approach. 4th ed. Philadelphia (PA): P Lippincott Williams & Wilkins; 2007. p. 14–22.

Eriksen MB, Frandsen TF. The impact of patient, intervention, comparison, outcome (PICO) as a search strategy tool on literature search quality: a systematic review. JMLA. 2018;106(4):420.

Ferrari R. Writing narrative style literature reviews. Medical Writing. 2015;24(4):230–5.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane qualitative and implementation methods group guidance series – paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Grant MJ, Booth A. A typology of reviews: an analysis of 14 review types and associated methodologies. Health Inf Libr J. 2009;26(2):91–108.

Green BN, Johnson CD, Adams A. Writing narrative literature reviews for peer-reviewed journals: secrets of the trade. J Chiropr Med. 2006;5(3):101–17.

Gregory AT, Denniss AR. An introduction to writing narrative and systematic reviews; tasks, tips and traps for aspiring authors. Heart Lung Circ. 2018;27(7):893–8.

Harden A, Thomas J, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. J Clin Epidemiol. 2018;97:70–8.

Harris JL, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 2: methods for question formulation, searching, and protocol development for qualitative evidence synthesis. J Clin Epidemiol. 2018;97:39–48.

Higgins J, Thomas J. In: Chandler J, Cumpston M, Li T, Page MJ, Welch VA, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.3, updated February 2022). Available from www.training.cochrane.org/handbook.: Cochrane; 2022.

International prospective register of systematic reviews (PROSPERO). Available from https://www.crd.york.ac.uk/prospero/ .

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Landhuis E. Scientific literature: information overload. Nature. 2016;535(7612):457–8.

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, Loveday H, Carrier J, Stannard D. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from https://synthesismanual.jbi.global . https://doi.org/10.46658/JBIMES-20-03 .

Chapter   Google Scholar  

Lorenzetti DL, Topfer L-A, Dennett L, Clement F. Value of databases other than medline for rapid health technology assessments. Int J Technol Assess Health Care. 2014;30(2):173–8.

Moher D, Liberati A, Tetzlaff J, Altman DG, the PRISMA Group. Preferred reporting items for (SR) and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;6:264–9.

Mulrow CD. Systematic reviews: rationale for systematic reviews. BMJ. 1994;309(6954):597–9.

Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol. 2018;18(1):143.

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):1–13.

Murphy CM. Writing an effective review article. J Med Toxicol. 2012;8(2):89–90.

NHMRC. Guidelines for guidelines: assessing risk of bias. Available at https://nhmrc.gov.au/guidelinesforguidelines/develop/assessing-risk-bias . Last published 29 August 2019. Accessed 29 Aug 2022.

Noyes J, Booth A, Cargo M, et al. Cochrane qualitative and implementation methods group guidance series – paper 1: introduction. J Clin Epidemiol. 2018b;97:35–8.

Noyes J, Booth A, Flemming K, et al. Cochrane qualitative and implementation methods group guidance series – paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol. 2018a;97:49–58.

Noyes J, Booth A, Moore G, Flemming K, Tunçalp Ö, Shakibazadeh E. Synthesising quantitative and qualitative evidence to inform guidelines on complex interventions: clarifying the purposes, designs and outlining some methods. BMJ Glob Health. 2019;4(Suppl 1):e000893.

Peters MD, Godfrey CM, Khalil H, McInerney P, Parker D, Soares CB. Guidance for conducting systematic scoping reviews. Int J Evid Healthcare. 2015;13(3):141–6.

Polanin JR, Pigott TD, Espelage DL, Grotpeter JK. Best practice guidelines for abstract screening large-evidence systematic reviews and meta-analyses. Res Synth Methods. 2019;10(3):330–42.

Article   PubMed Central   Google Scholar  

Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7(1):1–7.

Shea BJ, Reeves BC, Wells G, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Brit Med J. 2017;358

Sterne JA, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J. 2016;355

Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting. JAMA. 2000;283(15):2008–12.

Tawfik GM, Dila KAS, Mohamed MYF, et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health. 2019;47(1):1–9.

The Critical Appraisal Program. Critical appraisal skills program. Available at https://casp-uk.net/ . 2022. Accessed 29 Aug 2022.

The University of Melbourne. Writing a literature review in Research Techniques 2022. Available at https://students.unimelb.edu.au/academic-skills/explore-our-resources/research-techniques/reviewing-the-literature . Accessed 29 Aug 2022.

The Writing Center University of Winconsin-Madison. Learn how to write a literature review in The Writer’s Handbook – Academic Professional Writing. 2022. Available at https://writing.wisc.edu/handbook/assignments/reviewofliterature/ . Accessed 29 Aug 2022.

Thompson SG, Sharp SJ. Explaining heterogeneity in meta-analysis: a comparison of methods. Stat Med. 1999;18(20):2693–708.

Tricco AC, Lillie E, Zarin W, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):15.

Tricco AC, Lillie E, Zarin W, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Yoneoka D, Henmi M. Clinical heterogeneity in random-effect meta-analysis: between-study boundary estimate problem. Stat Med. 2019;38(21):4131–45.

Yuan Y, Hunt RH. Systematic reviews: the good, the bad, and the ugly. Am J Gastroenterol. 2009;104(5):1086–92.

Download references

Author information

Authors and affiliations.

Centre of Excellence in Treatable Traits, College of Health, Medicine and Wellbeing, University of Newcastle, Hunter Medical Research Institute Asthma and Breathing Programme, Newcastle, NSW, Australia

Dennis Thomas

Department of Pharmacy Practice, Faculty of Pharmacy, Universitas Airlangga, Surabaya, Indonesia

Elida Zairina

Centre for Medicine Use and Safety, Monash Institute of Pharmaceutical Sciences, Faculty of Pharmacy and Pharmaceutical Sciences, Monash University, Parkville, VIC, Australia

Johnson George

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Johnson George .

Section Editor information

College of Pharmacy, Qatar University, Doha, Qatar

Derek Charles Stewart

Department of Pharmacy, University of Huddersfield, Huddersfield, United Kingdom

Zaheer-Ud-Din Babar

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this entry

Cite this entry.

Thomas, D., Zairina, E., George, J. (2023). Methodological Approaches to Literature Review. In: Encyclopedia of Evidence in Pharmaceutical Public Health and Health Services Research in Pharmacy. Springer, Cham. https://doi.org/10.1007/978-3-030-50247-8_57-1

Download citation

DOI : https://doi.org/10.1007/978-3-030-50247-8_57-1

Received : 22 February 2023

Accepted : 22 February 2023

Published : 09 May 2023

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-50247-8

Online ISBN : 978-3-030-50247-8

eBook Packages : Springer Reference Biomedicine and Life Sciences Reference Module Biomedical and Life Sciences

• Publish with us

Policies and ethics

• Find a journal
• Track your research

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Writing a Literature Review

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

A literature review is a document or section of a document that collects key sources on a topic and discusses those sources in conversation with each other (also called synthesis ). The lit review is an important genre in many disciplines, not just literature (i.e., the study of works of literature such as novels and plays). When we say “literature review” or refer to “the literature,” we are talking about the research ( scholarship ) in a given field. You will often see the terms “the research,” “the scholarship,” and “the literature” used mostly interchangeably.

Where, when, and why would I write a lit review?

There are a number of different situations where you might write a literature review, each with slightly different expectations; different disciplines, too, have field-specific expectations for what a literature review is and does. For instance, in the humanities, authors might include more overt argumentation and interpretation of source material in their literature reviews, whereas in the sciences, authors are more likely to report study designs and results in their literature reviews; these differences reflect these disciplines’ purposes and conventions in scholarship. You should always look at examples from your own discipline and talk to professors or mentors in your field to be sure you understand your discipline’s conventions, for literature reviews as well as for any other genre.

A literature review can be a part of a research paper or scholarly article, usually falling after the introduction and before the research methods sections. In these cases, the lit review just needs to cover scholarship that is important to the issue you are writing about; sometimes it will also cover key sources that informed your research methodology.

Lit reviews can also be standalone pieces, either as assignments in a class or as publications. In a class, a lit review may be assigned to help students familiarize themselves with a topic and with scholarship in their field, get an idea of the other researchers working on the topic they’re interested in, find gaps in existing research in order to propose new projects, and/or develop a theoretical framework and methodology for later research. As a publication, a lit review usually is meant to help make other scholars’ lives easier by collecting and summarizing, synthesizing, and analyzing existing research on a topic. This can be especially helpful for students or scholars getting into a new research area, or for directing an entire community of scholars toward questions that have not yet been answered.

What are the parts of a lit review?

Most lit reviews use a basic introduction-body-conclusion structure; if your lit review is part of a larger paper, the introduction and conclusion pieces may be just a few sentences while you focus most of your attention on the body. If your lit review is a standalone piece, the introduction and conclusion take up more space and give you a place to discuss your goals, research methods, and conclusions separately from where you discuss the literature itself.

Introduction:

• An introductory paragraph that explains what your working topic and thesis is
• A forecast of key topics or texts that will appear in the review
• Potentially, a description of how you found sources and how you analyzed them for inclusion and discussion in the review (more often found in published, standalone literature reviews than in lit review sections in an article or research paper)
• Summarize and synthesize: Give an overview of the main points of each source and combine them into a coherent whole
• Analyze and interpret: Don’t just paraphrase other researchers – add your own interpretations where possible, discussing the significance of findings in relation to the literature as a whole
• Critically Evaluate: Mention the strengths and weaknesses of your sources
• Write in well-structured paragraphs: Use transition words and topic sentence to draw connections, comparisons, and contrasts.

Conclusion:

• Summarize the key findings you have taken from the literature and emphasize their significance
• Connect it back to your primary research question

How should I organize my lit review?

Lit reviews can take many different organizational patterns depending on what you are trying to accomplish with the review. Here are some examples:

• Chronological : The simplest approach is to trace the development of the topic over time, which helps familiarize the audience with the topic (for instance if you are introducing something that is not commonly known in your field). If you choose this strategy, be careful to avoid simply listing and summarizing sources in order. Try to analyze the patterns, turning points, and key debates that have shaped the direction of the field. Give your interpretation of how and why certain developments occurred (as mentioned previously, this may not be appropriate in your discipline — check with a teacher or mentor if you’re unsure).
• Thematic : If you have found some recurring central themes that you will continue working with throughout your piece, you can organize your literature review into subsections that address different aspects of the topic. For example, if you are reviewing literature about women and religion, key themes can include the role of women in churches and the religious attitude towards women.
• Qualitative versus quantitative research
• Empirical versus theoretical scholarship
• Divide the research by sociological, historical, or cultural sources
• Theoretical : In many humanities articles, the literature review is the foundation for the theoretical framework. You can use it to discuss various theories, models, and definitions of key concepts. You can argue for the relevance of a specific theoretical approach or combine various theorical concepts to create a framework for your research.

What are some strategies or tips I can use while writing my lit review?

Any lit review is only as good as the research it discusses; make sure your sources are well-chosen and your research is thorough. Don’t be afraid to do more research if you discover a new thread as you’re writing. More info on the research process is available in our "Conducting Research" resources .

As you’re doing your research, create an annotated bibliography ( see our page on the this type of document ). Much of the information used in an annotated bibliography can be used also in a literature review, so you’ll be not only partially drafting your lit review as you research, but also developing your sense of the larger conversation going on among scholars, professionals, and any other stakeholders in your topic.

Usually you will need to synthesize research rather than just summarizing it. This means drawing connections between sources to create a picture of the scholarly conversation on a topic over time. Many student writers struggle to synthesize because they feel they don’t have anything to add to the scholars they are citing; here are some strategies to help you:

• It often helps to remember that the point of these kinds of syntheses is to show your readers how you understand your research, to help them read the rest of your paper.
• Writing teachers often say synthesis is like hosting a dinner party: imagine all your sources are together in a room, discussing your topic. What are they saying to each other?
• Look at the in-text citations in each paragraph. Are you citing just one source for each paragraph? This usually indicates summary only. When you have multiple sources cited in a paragraph, you are more likely to be synthesizing them (not always, but often
• Read more about synthesis here.

The most interesting literature reviews are often written as arguments (again, as mentioned at the beginning of the page, this is discipline-specific and doesn’t work for all situations). Often, the literature review is where you can establish your research as filling a particular gap or as relevant in a particular way. You have some chance to do this in your introduction in an article, but the literature review section gives a more extended opportunity to establish the conversation in the way you would like your readers to see it. You can choose the intellectual lineage you would like to be part of and whose definitions matter most to your thinking (mostly humanities-specific, but this goes for sciences as well). In addressing these points, you argue for your place in the conversation, which tends to make the lit review more compelling than a simple reporting of other sources.

Research Methods

• Getting Started
• Literature Review Research
• Research Design
• Research Design By Discipline
• SAGE Research Methods
• Teaching with SAGE Research Methods

Literature Review

• What is a Literature Review?
• What is NOT a Literature Review?
• Purposes of a Literature Review
• Types of Literature Reviews
• Literature Reviews vs. Systematic Reviews
• Systematic vs. Meta-Analysis

Literature Review  is a comprehensive survey of the works published in a particular field of study or line of research, usually over a specific period of time, in the form of an in-depth, critical bibliographic essay or annotated list in which attention is drawn to the most significant works.

Also, we can define a literature review as the collected body of scholarly works related to a topic:

• Summarizes and analyzes previous research relevant to a topic
• Includes scholarly books and articles published in academic journals
• Can be an specific scholarly paper or a section in a research paper

The objective of a Literature Review is to find previous published scholarly works relevant to an specific topic

• Help gather ideas or information
• Keep up to date in current trends and findings
• Help develop new questions

A literature review is important because it:

• Explains the background of research on a topic.
• Demonstrates why a topic is significant to a subject area.
• Helps focus your own research questions or problems
• Discovers relationships between research studies/ideas.
• Suggests unexplored ideas or populations
• Identifies major themes, concepts, and researchers on a topic.
• Tests assumptions; may help counter preconceived ideas and remove unconscious bias.
• Identifies critical gaps, points of disagreement, or potentially flawed methodology or theoretical approaches.
• Indicates potential directions for future research.

All content in this section is from Literature Review Research from Old Dominion University 

Keep in mind the following, a literature review is NOT:

Not an essay 

Not an annotated bibliography  in which you summarize each article that you have reviewed.  A literature review goes beyond basic summarizing to focus on the critical analysis of the reviewed works and their relationship to your research question.

Not a research paper   where you select resources to support one side of an issue versus another.  A lit review should explain and consider all sides of an argument in order to avoid bias, and areas of agreement and disagreement should be highlighted.

A literature review serves several purposes. For example, it

• provides thorough knowledge of previous studies; introduces seminal works.
• helps focus one’s own research topic.
• identifies a conceptual framework for one’s own research questions or problems; indicates potential directions for future research.
• suggests previously unused or underused methodologies, designs, quantitative and qualitative strategies.
• identifies gaps in previous studies; identifies flawed methodologies and/or theoretical approaches; avoids replication of mistakes.
• helps the researcher avoid repetition of earlier research.
• suggests unexplored populations.
• determines whether past studies agree or disagree; identifies controversy in the literature.
• tests assumptions; may help counter preconceived ideas and remove unconscious bias.

As Kennedy (2007) notes*, it is important to think of knowledge in a given field as consisting of three layers. First, there are the primary studies that researchers conduct and publish. Second are the reviews of those studies that summarize and offer new interpretations built from and often extending beyond the original studies. Third, there are the perceptions, conclusions, opinion, and interpretations that are shared informally that become part of the lore of field. In composing a literature review, it is important to note that it is often this third layer of knowledge that is cited as "true" even though it often has only a loose relationship to the primary studies and secondary literature reviews.

Given this, while literature reviews are designed to provide an overview and synthesis of pertinent sources you have explored, there are several approaches to how they can be done, depending upon the type of analysis underpinning your study. Listed below are definitions of types of literature reviews:

Argumentative Review      This form examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. The purpose is to develop a body of literature that establishes a contrarian viewpoint. Given the value-laden nature of some social science research [e.g., educational reform; immigration control], argumentative approaches to analyzing the literature can be a legitimate and important form of discourse. However, note that they can also introduce problems of bias when they are used to to make summary claims of the sort found in systematic reviews.

Integrative Review      Considered a form of research that reviews, critiques, and synthesizes representative literature on a topic in an integrated way such that new frameworks and perspectives on the topic are generated. The body of literature includes all studies that address related or identical hypotheses. A well-done integrative review meets the same standards as primary research in regard to clarity, rigor, and replication.

Historical Review      Few things rest in isolation from historical precedent. Historical reviews are focused on examining research throughout a period of time, often starting with the first time an issue, concept, theory, phenomena emerged in the literature, then tracing its evolution within the scholarship of a discipline. The purpose is to place research in a historical context to show familiarity with state-of-the-art developments and to identify the likely directions for future research.

Methodological Review      A review does not always focus on what someone said [content], but how they said it [method of analysis]. This approach provides a framework of understanding at different levels (i.e. those of theory, substantive fields, research approaches and data collection and analysis techniques), enables researchers to draw on a wide variety of knowledge ranging from the conceptual level to practical documents for use in fieldwork in the areas of ontological and epistemological consideration, quantitative and qualitative integration, sampling, interviewing, data collection and data analysis, and helps highlight many ethical issues which we should be aware of and consider as we go through our study.

Systematic Review      This form consists of an overview of existing evidence pertinent to a clearly formulated research question, which uses pre-specified and standardized methods to identify and critically appraise relevant research, and to collect, report, and analyse data from the studies that are included in the review. Typically it focuses on a very specific empirical question, often posed in a cause-and-effect form, such as "To what extent does A contribute to B?"

Theoretical Review      The purpose of this form is to concretely examine the corpus of theory that has accumulated in regard to an issue, concept, theory, phenomena. The theoretical literature review help establish what theories already exist, the relationships between them, to what degree the existing theories have been investigated, and to develop new hypotheses to be tested. Often this form is used to help establish a lack of appropriate theories or reveal that current theories are inadequate for explaining new or emerging research problems. The unit of analysis can focus on a theoretical concept or a whole theory or framework.

* Kennedy, Mary M. "Defining a Literature."  Educational Researcher  36 (April 2007): 139-147.

All content in this section is from The Literature Review created by Dr. Robert Larabee USC

Robinson, P. and Lowe, J. (2015),  Literature reviews vs systematic reviews.  Australian and New Zealand Journal of Public Health, 39: 103-103. doi: 10.1111/1753-6405.12393

What's in the name? The difference between a Systematic Review and a Literature Review, and why it matters . By Lynn Kysh from University of Southern California

Systematic review or meta-analysis?

A  systematic review  answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria.

A  meta-analysis  is the use of statistical methods to summarize the results of these studies.

Systematic reviews, just like other research articles, can be of varying quality. They are a significant piece of work (the Centre for Reviews and Dissemination at York estimates that a team will take 9-24 months), and to be useful to other researchers and practitioners they should have:

• clearly stated objectives with pre-defined eligibility criteria for studies
• explicit, reproducible methodology
• a systematic search that attempts to identify all studies
• assessment of the validity of the findings of the included studies (e.g. risk of bias)
• systematic presentation, and synthesis, of the characteristics and findings of the included studies

Not all systematic reviews contain meta-analysis. 

Meta-analysis is the use of statistical methods to summarize the results of independent studies. By combining information from all relevant studies, meta-analysis can provide more precise estimates of the effects of health care than those derived from the individual studies included within a review.  More information on meta-analyses can be found in  Cochrane Handbook, Chapter 9 .

A meta-analysis goes beyond critique and integration and conducts secondary statistical analysis on the outcomes of similar studies.  It is a systematic review that uses quantitative methods to synthesize and summarize the results.

An advantage of a meta-analysis is the ability to be completely objective in evaluating research findings.  Not all topics, however, have sufficient research evidence to allow a meta-analysis to be conducted.  In that case, an integrative review is an appropriate strategy. 

Some of the content in this section is from Systematic reviews and meta-analyses: step by step guide created by Kate McAllister.

• << Previous: Getting Started
• Next: Research Design >>
• Last Updated: Aug 21, 2023 4:07 PM
• URL: https://guides.lib.udel.edu/researchmethods

• Privacy Policy

Home » Literature Review – Types Writing Guide and Examples

Literature Review – Types Writing Guide and Examples

Table of Contents

Literature Review

Definition:

A literature review is a comprehensive and critical analysis of the existing literature on a particular topic or research question. It involves identifying, evaluating, and synthesizing relevant literature, including scholarly articles, books, and other sources, to provide a summary and critical assessment of what is known about the topic.

Types of Literature Review

Types of Literature Review are as follows:

• Narrative literature review : This type of review involves a comprehensive summary and critical analysis of the available literature on a particular topic or research question. It is often used as an introductory section of a research paper.
• Systematic literature review: This is a rigorous and structured review that follows a pre-defined protocol to identify, evaluate, and synthesize all relevant studies on a specific research question. It is often used in evidence-based practice and systematic reviews.
• Meta-analysis: This is a quantitative review that uses statistical methods to combine data from multiple studies to derive a summary effect size. It provides a more precise estimate of the overall effect than any individual study.
• Scoping review: This is a preliminary review that aims to map the existing literature on a broad topic area to identify research gaps and areas for further investigation.
• Critical literature review : This type of review evaluates the strengths and weaknesses of the existing literature on a particular topic or research question. It aims to provide a critical analysis of the literature and identify areas where further research is needed.
• Conceptual literature review: This review synthesizes and integrates theories and concepts from multiple sources to provide a new perspective on a particular topic. It aims to provide a theoretical framework for understanding a particular research question.
• Rapid literature review: This is a quick review that provides a snapshot of the current state of knowledge on a specific research question or topic. It is often used when time and resources are limited.
• Thematic literature review : This review identifies and analyzes common themes and patterns across a body of literature on a particular topic. It aims to provide a comprehensive overview of the literature and identify key themes and concepts.
• Realist literature review: This review is often used in social science research and aims to identify how and why certain interventions work in certain contexts. It takes into account the context and complexities of real-world situations.
• State-of-the-art literature review : This type of review provides an overview of the current state of knowledge in a particular field, highlighting the most recent and relevant research. It is often used in fields where knowledge is rapidly evolving, such as technology or medicine.
• Integrative literature review: This type of review synthesizes and integrates findings from multiple studies on a particular topic to identify patterns, themes, and gaps in the literature. It aims to provide a comprehensive understanding of the current state of knowledge on a particular topic.
• Umbrella literature review : This review is used to provide a broad overview of a large and diverse body of literature on a particular topic. It aims to identify common themes and patterns across different areas of research.
• Historical literature review: This type of review examines the historical development of research on a particular topic or research question. It aims to provide a historical context for understanding the current state of knowledge on a particular topic.
• Problem-oriented literature review : This review focuses on a specific problem or issue and examines the literature to identify potential solutions or interventions. It aims to provide practical recommendations for addressing a particular problem or issue.
• Mixed-methods literature review : This type of review combines quantitative and qualitative methods to synthesize and analyze the available literature on a particular topic. It aims to provide a more comprehensive understanding of the research question by combining different types of evidence.

Parts of Literature Review

Parts of a literature review are as follows:

Introduction

The introduction of a literature review typically provides background information on the research topic and why it is important. It outlines the objectives of the review, the research question or hypothesis, and the scope of the review.

Literature Search

This section outlines the search strategy and databases used to identify relevant literature. The search terms used, inclusion and exclusion criteria, and any limitations of the search are described.

Literature Analysis

The literature analysis is the main body of the literature review. This section summarizes and synthesizes the literature that is relevant to the research question or hypothesis. The review should be organized thematically, chronologically, or by methodology, depending on the research objectives.

Critical Evaluation

Critical evaluation involves assessing the quality and validity of the literature. This includes evaluating the reliability and validity of the studies reviewed, the methodology used, and the strength of the evidence.

The conclusion of the literature review should summarize the main findings, identify any gaps in the literature, and suggest areas for future research. It should also reiterate the importance of the research question or hypothesis and the contribution of the literature review to the overall research project.

The references list includes all the sources cited in the literature review, and follows a specific referencing style (e.g., APA, MLA, Harvard).

How to write Literature Review

Here are some steps to follow when writing a literature review:

• Define your research question or topic : Before starting your literature review, it is essential to define your research question or topic. This will help you identify relevant literature and determine the scope of your review.
• Conduct a comprehensive search: Use databases and search engines to find relevant literature. Look for peer-reviewed articles, books, and other academic sources that are relevant to your research question or topic.
• Evaluate the sources: Once you have found potential sources, evaluate them critically to determine their relevance, credibility, and quality. Look for recent publications, reputable authors, and reliable sources of data and evidence.
• Organize your sources: Group the sources by theme, method, or research question. This will help you identify similarities and differences among the literature, and provide a structure for your literature review.
• Analyze and synthesize the literature : Analyze each source in depth, identifying the key findings, methodologies, and conclusions. Then, synthesize the information from the sources, identifying patterns and themes in the literature.
• Write the literature review : Start with an introduction that provides an overview of the topic and the purpose of the literature review. Then, organize the literature according to your chosen structure, and analyze and synthesize the sources. Finally, provide a conclusion that summarizes the key findings of the literature review, identifies gaps in knowledge, and suggests areas for future research.
• Edit and proofread: Once you have written your literature review, edit and proofread it carefully to ensure that it is well-organized, clear, and concise.

Examples of Literature Review

Here’s an example of how a literature review can be conducted for a thesis on the topic of “ The Impact of Social Media on Teenagers’ Mental Health”:

• Start by identifying the key terms related to your research topic. In this case, the key terms are “social media,” “teenagers,” and “mental health.”
• Use academic databases like Google Scholar, JSTOR, or PubMed to search for relevant articles, books, and other publications. Use these keywords in your search to narrow down your results.
• Evaluate the sources you find to determine if they are relevant to your research question. You may want to consider the publication date, author’s credentials, and the journal or book publisher.
• Begin reading and taking notes on each source, paying attention to key findings, methodologies used, and any gaps in the research.
• Organize your findings into themes or categories. For example, you might categorize your sources into those that examine the impact of social media on self-esteem, those that explore the effects of cyberbullying, and those that investigate the relationship between social media use and depression.
• Synthesize your findings by summarizing the key themes and highlighting any gaps or inconsistencies in the research. Identify areas where further research is needed.
• Use your literature review to inform your research questions and hypotheses for your thesis.

For example, after conducting a literature review on the impact of social media on teenagers’ mental health, a thesis might look like this:

“Using a mixed-methods approach, this study aims to investigate the relationship between social media use and mental health outcomes in teenagers. Specifically, the study will examine the effects of cyberbullying, social comparison, and excessive social media use on self-esteem, anxiety, and depression. Through an analysis of survey data and qualitative interviews with teenagers, the study will provide insight into the complex relationship between social media use and mental health outcomes, and identify strategies for promoting positive mental health outcomes in young people.”

Reference: Smith, J., Jones, M., & Lee, S. (2019). The effects of social media use on adolescent mental health: A systematic review. Journal of Adolescent Health, 65(2), 154-165. doi:10.1016/j.jadohealth.2019.03.024

Reference Example: Author, A. A., Author, B. B., & Author, C. C. (Year). Title of article. Title of Journal, volume number(issue number), page range. doi:0000000/000000000000 or URL

Applications of Literature Review

some applications of literature review in different fields:

• Social Sciences: In social sciences, literature reviews are used to identify gaps in existing research, to develop research questions, and to provide a theoretical framework for research. Literature reviews are commonly used in fields such as sociology, psychology, anthropology, and political science.
• Natural Sciences: In natural sciences, literature reviews are used to summarize and evaluate the current state of knowledge in a particular field or subfield. Literature reviews can help researchers identify areas where more research is needed and provide insights into the latest developments in a particular field. Fields such as biology, chemistry, and physics commonly use literature reviews.
• Health Sciences: In health sciences, literature reviews are used to evaluate the effectiveness of treatments, identify best practices, and determine areas where more research is needed. Literature reviews are commonly used in fields such as medicine, nursing, and public health.
• Humanities: In humanities, literature reviews are used to identify gaps in existing knowledge, develop new interpretations of texts or cultural artifacts, and provide a theoretical framework for research. Literature reviews are commonly used in fields such as history, literary studies, and philosophy.

Role of Literature Review in Research

Here are some applications of literature review in research:

• Identifying Research Gaps : Literature review helps researchers identify gaps in existing research and literature related to their research question. This allows them to develop new research questions and hypotheses to fill those gaps.
• Developing Theoretical Framework: Literature review helps researchers develop a theoretical framework for their research. By analyzing and synthesizing existing literature, researchers can identify the key concepts, theories, and models that are relevant to their research.
• Selecting Research Methods : Literature review helps researchers select appropriate research methods and techniques based on previous research. It also helps researchers to identify potential biases or limitations of certain methods and techniques.
• Data Collection and Analysis: Literature review helps researchers in data collection and analysis by providing a foundation for the development of data collection instruments and methods. It also helps researchers to identify relevant data sources and identify potential data analysis techniques.
• Communicating Results: Literature review helps researchers to communicate their results effectively by providing a context for their research. It also helps to justify the significance of their findings in relation to existing research and literature.

Purpose of Literature Review

Some of the specific purposes of a literature review are as follows:

• To provide context: A literature review helps to provide context for your research by situating it within the broader body of literature on the topic.
• To identify gaps and inconsistencies: A literature review helps to identify areas where further research is needed or where there are inconsistencies in the existing literature.
• To synthesize information: A literature review helps to synthesize the information from multiple sources and present a coherent and comprehensive picture of the current state of knowledge on the topic.
• To identify key concepts and theories : A literature review helps to identify key concepts and theories that are relevant to your research question and provide a theoretical framework for your study.
• To inform research design: A literature review can inform the design of your research study by identifying appropriate research methods, data sources, and research questions.

Characteristics of Literature Review

Some Characteristics of Literature Review are as follows:

• Identifying gaps in knowledge: A literature review helps to identify gaps in the existing knowledge and research on a specific topic or research question. By analyzing and synthesizing the literature, you can identify areas where further research is needed and where new insights can be gained.
• Establishing the significance of your research: A literature review helps to establish the significance of your own research by placing it in the context of existing research. By demonstrating the relevance of your research to the existing literature, you can establish its importance and value.
• Informing research design and methodology : A literature review helps to inform research design and methodology by identifying the most appropriate research methods, techniques, and instruments. By reviewing the literature, you can identify the strengths and limitations of different research methods and techniques, and select the most appropriate ones for your own research.
• Supporting arguments and claims: A literature review provides evidence to support arguments and claims made in academic writing. By citing and analyzing the literature, you can provide a solid foundation for your own arguments and claims.
• I dentifying potential collaborators and mentors: A literature review can help identify potential collaborators and mentors by identifying researchers and practitioners who are working on related topics or using similar methods. By building relationships with these individuals, you can gain valuable insights and support for your own research and practice.
• Keeping up-to-date with the latest research : A literature review helps to keep you up-to-date with the latest research on a specific topic or research question. By regularly reviewing the literature, you can stay informed about the latest findings and developments in your field.

Advantages of Literature Review

There are several advantages to conducting a literature review as part of a research project, including:

• Establishing the significance of the research : A literature review helps to establish the significance of the research by demonstrating the gap or problem in the existing literature that the study aims to address.
• Identifying key concepts and theories: A literature review can help to identify key concepts and theories that are relevant to the research question, and provide a theoretical framework for the study.
• Supporting the research methodology : A literature review can inform the research methodology by identifying appropriate research methods, data sources, and research questions.
• Providing a comprehensive overview of the literature : A literature review provides a comprehensive overview of the current state of knowledge on a topic, allowing the researcher to identify key themes, debates, and areas of agreement or disagreement.
• Identifying potential research questions: A literature review can help to identify potential research questions and areas for further investigation.
• Avoiding duplication of research: A literature review can help to avoid duplication of research by identifying what has already been done on a topic, and what remains to be done.
• Enhancing the credibility of the research : A literature review helps to enhance the credibility of the research by demonstrating the researcher’s knowledge of the existing literature and their ability to situate their research within a broader context.

Limitations of Literature Review

Limitations of Literature Review are as follows:

• Limited scope : Literature reviews can only cover the existing literature on a particular topic, which may be limited in scope or depth.
• Publication bias : Literature reviews may be influenced by publication bias, which occurs when researchers are more likely to publish positive results than negative ones. This can lead to an incomplete or biased picture of the literature.
• Quality of sources : The quality of the literature reviewed can vary widely, and not all sources may be reliable or valid.
• Time-limited: Literature reviews can become quickly outdated as new research is published, making it difficult to keep up with the latest developments in a field.
• Subjective interpretation : Literature reviews can be subjective, and the interpretation of the findings can vary depending on the researcher’s perspective or bias.
• Lack of original data : Literature reviews do not generate new data, but rather rely on the analysis of existing studies.
• Risk of plagiarism: It is important to ensure that literature reviews do not inadvertently contain plagiarism, which can occur when researchers use the work of others without proper attribution.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Thesis – Structure, Example and Writing Guide

Research Process – Steps, Examples and Tips

Research Report – Example, Writing Guide and...

Data Collection – Methods Types and Examples

Context of the Study – Writing Guide and Examples

Appendices – Writing Guide, Types and Examples

Chapter Four: Theory, Methodologies, Methods, and Evidence

Research Methods

You are viewing the first edition of this textbook. a second edition is available – please visit the latest edition for updated information..

This page discusses the following topics:

Research Goals

Research method types.

Before discussing research   methods , we need to distinguish them from  methodologies  and  research skills . Methodologies, linked to literary theories, are tools and lines of investigation: sets of practices and propositions about texts and the world. Researchers using Marxist literary criticism will adopt methodologies that look to material forces like labor, ownership, and technology to understand literature and its relationship to the world. They will also seek to understand authors not as inspired geniuses but as people whose lives and work are shaped by social forces.

Example: Critical Race Theory Methodologies

Critical Race Theory may use a variety of methodologies, including

• Interest convergence: investigating whether marginalized groups only achieve progress when dominant groups benefit as well
• Intersectional theory: investigating how multiple factors of advantage and disadvantage around race, gender, ethnicity, religion, etc. operate together in complex ways
• Radical critique of the law: investigating how the law has historically been used to marginalize particular groups, such as black people, while recognizing that legal efforts are important to achieve emancipation and civil rights
• Social constructivism: investigating how race is socially constructed (rather than biologically grounded)
• Standpoint epistemology: investigating how knowledge relates to social position
• Structural determinism: investigating how structures of thought and of organizations determine social outcomes

To identify appropriate methodologies, you will need to research your chosen theory and gather what methodologies are associated with it. For the most part, we can’t assume that there are “one size fits all” methodologies.

Research skills are about how you handle materials such as library search engines, citation management programs, special collections materials, and so on.

Research methods  are about where and how you get answers to your research questions. Are you conducting interviews? Visiting archives? Doing close readings? Reviewing scholarship? You will need to choose which methods are most appropriate to use in your research and you need to gain some knowledge about how to use these methods. In other words, you need to do some research into research methods!

Your choice of research method depends on the kind of questions you are asking. For example, if you want to understand how an author progressed through several drafts to arrive at a final manuscript, you may need to do archival research. If you want to understand why a particular literary work became a bestseller, you may need to do audience research. If you want to know why a contemporary author wrote a particular work, you may need to do interviews. Usually literary research involves a combination of methods such as  archival research ,  discourse analysis , and  qualitative research  methods.

Literary research methods tend to differ from research methods in the hard sciences (such as physics and chemistry). Science research must present results that are reproducible, while literary research rarely does (though it must still present evidence for its claims). Literary research often deals with questions of meaning, social conventions, representations of lived experience, and aesthetic effects; these are questions that reward dialogue and different perspectives rather than one great experiment that settles the issue. In literary research, we might get many valuable answers even though they are quite different from one another. Also in literary research, we usually have some room to speculate about answers, but our claims have to be plausible (believable) and our argument comprehensive (meaning we don’t overlook evidence that would alter our argument significantly if it were known).

A literary researcher might select the following:

Theory: Critical Race Theory

Methodology: Social Constructivism

Method: Scholarly

Skills: Search engines, citation management

Wendy Belcher, in  Writing Your Journal Article in 12 Weeks , identifies two main approaches to understanding literary works: looking at a text by itself (associated with New Criticism ) and looking at texts as they connect to society (associated with Cultural Studies ). The goal of New Criticism is to bring the reader further into the text. The goal of Cultural Studies is to bring the reader into the network of discourses that surround and pass through the text. Other approaches, such as Ecocriticism, relate literary texts to the Sciences (as well as to the Humanities).

The New Critics, starting in the 1940s,  focused on meaning within the text itself, using a method they called “ close reading .” The text itself becomes e vidence for a particular reading. Using this approach, you should summarize the literary work briefly and q uote particularly meaningful passages, being sure to introduce quotes and then interpret them (never let them stand alone). Make connections within the work; a sk  “why” and “how” the various parts of the text relate to each other.

Cultural Studies critics see all texts  as connected to society; the critic  therefore has to connect a text to at least one political or social issue. How and why does  the text reproduce particular knowledge systems (known as discourses) and how do these knowledge systems relate to issues of power within the society? Who speaks and when? Answering these questions helps your reader understand the text in context. Cultural contexts can include the treatment of gender (Feminist, Queer), class (Marxist), nationality, race, religion, or any other area of human society.

Other approaches, such as psychoanalytic literary criticism , look at literary texts to better understand human psychology. A psychoanalytic reading can focus on a character, the author, the reader, or on society in general. Ecocriticism  look at human understandings of nature in literary texts.

We select our research methods based on the kinds of things we want to know. For example, we may be studying the relationship between literature and society, between author and text, or the status of a work in the literary canon. We may want to know about a work’s form, genre, or thematics. We may want to know about the audience’s reading and reception, or about methods for teaching literature in schools.

Below are a few research methods and their descriptions. You may need to consult with your instructor about which ones are most appropriate for your project. The first list covers methods most students use in their work. The second list covers methods more commonly used by advanced researchers. Even if you will not be using methods from this second list in your research project, you may read about these research methods in the scholarship you find.

Most commonly used undergraduate research methods:

• Scholarship Methods:  Studies the body of scholarship written about a particular author, literary work, historical period, literary movement, genre, theme, theory, or method.
• Textual Analysis Methods:  Used for close readings of literary texts, these methods also rely on literary theory and background information to support the reading.
• Biographical Methods:  Used to study the life of the author to better understand their work and times, these methods involve reading biographies and autobiographies about the author, and may also include research into private papers, correspondence, and interviews.
• Discourse Analysis Methods:  Studies language patterns to reveal ideology and social relations of power. This research involves the study of institutions, social groups, and social movements to understand how people in various settings use language to represent the world to themselves and others. Literary works may present complex mixtures of discourses which the characters (and readers) have to navigate.
• Creative Writing Methods:  A literary re-working of another literary text, creative writing research is used to better understand a literary work by investigating its language, formal structures, composition methods, themes, and so on. For instance, a creative research project may retell a story from a minor character’s perspective to reveal an alternative reading of events. To qualify as research, a creative research project is usually combined with a piece of theoretical writing that explains and justifies the work.

Methods used more often by advanced researchers:

• Archival Methods: Usually involves trips to special collections where original papers are kept. In these archives are many unpublished materials such as diaries, letters, photographs, ledgers, and so on. These materials can offer us invaluable insight into the life of an author, the development of a literary work, or the society in which the author lived. There are at least three major archives of James Baldwin’s papers: The Smithsonian , Yale , and The New York Public Library . Descriptions of such materials are often available online, but the materials themselves are typically stored in boxes at the archive.
• Computational Methods:  Used for statistical analysis of texts such as studies of the popularity and meaning of particular words in literature over time.
• Ethnographic Methods:  Studies groups of people and their interactions with literary works, for instance in educational institutions, in reading groups (such as book clubs), and in fan networks. This approach may involve interviews and visits to places (including online communities) where people interact with literary works. Note: before you begin such work, you must have  Institutional Review Board (IRB)  approval “to protect the rights and welfare of human participants involved in research.”
• Visual Methods:  Studies the visual qualities of literary works. Some literary works, such as illuminated manuscripts, children’s literature, and graphic novels, present a complex interplay of text and image. Even works without illustrations can be studied for their use of typography, layout, and other visual features.

Regardless of the method(s) you choose, you will need to learn how to apply them to your work and how to carry them out successfully. For example, you should know that many archives do not allow you to bring pens (you can use pencils) and you may not be allowed to bring bags into the archives. You will need to keep a record of which documents you consult and their location (box number, etc.) in the archives. If you are unsure how to use a particular method, please consult a book about it. [1] Also, ask for the advice of trained researchers such as your instructor or a research librarian.

• What research method(s) will you be using for your paper? Why did you make this method selection over other methods? If you haven’t made a selection yet, which methods are you considering?
• What specific methodological approaches are you most interested in exploring in relation to the chosen literary work?
• What is your plan for researching your method(s) and its major approaches?
• What was the most important lesson you learned from this page? What point was confusing or difficult to understand?

Write your answers in a webcourse discussion page.

• Introduction to Research Methods: A Practical Guide for Anyone Undertaking a Research Project  by Catherine, Dr. Dawson
• Practical Research Methods: A User-Friendly Guide to Mastering Research Techniques and Projects  by Catherine Dawson
• Qualitative Inquiry and Research Design: Choosing Among Five Approaches  by John W. Creswell  Cheryl N. Poth
• Qualitative Research Evaluation Methods: Integrating Theory and Practice  by Michael Quinn Patton
• Research Design: Qualitative, Quantitative, and Mixed Methods Approaches  by John W. Creswell  J. David Creswell
• Research Methodology: A Step-by-Step Guide for Beginners  by Ranjit Kumar
• Research Methodology: Methods and Techniques  by C.R. Kothari

Strategies for Conducting Literary Research Copyright © 2021 by Barry Mauer & John Venecek is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Library Guides

Dissertations 4: methodology: methods.

• Introduction & Philosophy
• Methodology

Primary & Secondary Sources, Primary & Secondary Data

When describing your research methods, you can start by stating what kind of secondary and, if applicable, primary sources you used in your research. Explain why you chose such sources, how well they served your research, and identify possible issues encountered using these sources.  

Definitions  

There is some confusion on the use of the terms primary and secondary sources, and primary and secondary data. The confusion is also due to disciplinary differences (Lombard 2010). Whilst you are advised to consult the research methods literature in your field, we can generalise as follows:  

Secondary sources 

Secondary sources normally include the literature (books and articles) with the experts' findings, analysis and discussions on a certain topic (Cottrell, 2014, p123). Secondary sources often interpret primary sources.  

Primary sources 

Primary sources are "first-hand" information such as raw data, statistics, interviews, surveys, law statutes and law cases. Even literary texts, pictures and films can be primary sources if they are the object of research (rather than, for example, documentaries reporting on something else, in which case they would be secondary sources). The distinction between primary and secondary sources sometimes lies on the use you make of them (Cottrell, 2014, p123). 

Primary data 

Primary data are data (primary sources) you directly obtained through your empirical work (Saunders, Lewis and Thornhill 2015, p316). 

Secondary data 

Secondary data are data (primary sources) that were originally collected by someone else (Saunders, Lewis and Thornhill 2015, p316).   

Comparison between primary and secondary data   

Use  

Virtually all research will use secondary sources, at least as background information. 

Often, especially at the postgraduate level, it will also use primary sources - secondary and/or primary data. The engagement with primary sources is generally appreciated, as less reliant on others' interpretations, and closer to 'facts'. 

The use of primary data, as opposed to secondary data, demonstrates the researcher's effort to do empirical work and find evidence to answer her specific research question and fulfill her specific research objectives. Thus, primary data contribute to the originality of the research.    

Ultimately, you should state in this section of the methodology: 

What sources and data you are using and why (how are they going to help you answer the research question and/or test the hypothesis. 

If using primary data, why you employed certain strategies to collect them. 

What the advantages and disadvantages of your strategies to collect the data (also refer to the research in you field and research methods literature). 

Quantitative, Qualitative & Mixed Methods

The methodology chapter should reference your use of quantitative research, qualitative research and/or mixed methods. The following is a description of each along with their advantages and disadvantages. 

Quantitative research 

Quantitative research uses numerical data (quantities) deriving, for example, from experiments, closed questions in surveys, questionnaires, structured interviews or published data sets (Cottrell, 2014, p93). It normally processes and analyses this data using quantitative analysis techniques like tables, graphs and statistics to explore, present and examine relationships and trends within the data (Saunders, Lewis and Thornhill, 2015, p496). 

Qualitative research  

Qualitative research is generally undertaken to study human behaviour and psyche. It uses methods like in-depth case studies, open-ended survey questions, unstructured interviews, focus groups, or unstructured observations (Cottrell, 2014, p93). The nature of the data is subjective, and also the analysis of the researcher involves a degree of subjective interpretation. Subjectivity can be controlled for in the research design, or has to be acknowledged as a feature of the research. Subject-specific books on (qualitative) research methods offer guidance on such research designs.  

Mixed methods 

Mixed-method approaches combine both qualitative and quantitative methods, and therefore combine the strengths of both types of research. Mixed methods have gained popularity in recent years.  

When undertaking mixed-methods research you can collect the qualitative and quantitative data either concurrently or sequentially. If sequentially, you can for example, start with a few semi-structured interviews, providing qualitative insights, and then design a questionnaire to obtain quantitative evidence that your qualitative findings can also apply to a wider population (Specht, 2019, p138). 

Ultimately, your methodology chapter should state: 

Whether you used quantitative research, qualitative research or mixed methods. 

Why you chose such methods (and refer to research method sources). 

Why you rejected other methods. 

How well the method served your research. 

The problems or limitations you encountered. 

Doug Specht, Senior Lecturer at the Westminster School of Media and Communication, explains mixed methods research in the following video:

LinkedIn Learning Video on Academic Research Foundations: Quantitative

The video covers the characteristics of quantitative research, and explains how to approach different parts of the research process, such as creating a solid research question and developing a literature review. He goes over the elements of a study, explains how to collect and analyze data, and shows how to present your data in written and numeric form.

Link to quantitative research video

Some Types of Methods

There are several methods you can use to get primary data. To reiterate, the choice of the methods should depend on your research question/hypothesis. 

Whatever methods you will use, you will need to consider: 

why did you choose one technique over another? What were the advantages and disadvantages of the technique you chose? 

what was the size of your sample? Who made up your sample? How did you select your sample population? Why did you choose that particular sampling strategy?) 

ethical considerations (see also tab...)  

safety considerations  

validity  

feasibility  

recording  

procedure of the research (see box procedural method...).  

Check Stella Cottrell's book  Dissertations and Project Reports: A Step by Step Guide  for some succinct yet comprehensive information on most methods (the following account draws mostly on her work). Check a research methods book in your discipline for more specific guidance.  

Experiments 

Experiments are useful to investigate cause and effect, when the variables can be tightly controlled. They can test a theory or hypothesis in controlled conditions. Experiments do not prove or disprove an hypothesis, instead they support or not support an hypothesis. When using the empirical and inductive method it is not possible to achieve conclusive results. The results may only be valid until falsified by other experiments and observations. 

For more information on Scientific Method, click here . 

Observations 

Observational methods are useful for in-depth analyses of behaviours in people, animals, organisations, events or phenomena. They can test a theory or products in real life or simulated settings. They generally a qualitative research method.  

Questionnaires and surveys 

Questionnaires and surveys are useful to gain opinions, attitudes, preferences, understandings on certain matters. They can provide quantitative data that can be collated systematically; qualitative data, if they include opportunities for open-ended responses; or both qualitative and quantitative elements. 

Interviews  

Interviews are useful to gain rich, qualitative information about individuals' experiences, attitudes or perspectives. With interviews you can follow up immediately on responses for clarification or further details. There are three main types of interviews: structured (following a strict pattern of questions, which expect short answers), semi-structured (following a list of questions, with the opportunity to follow up the answers with improvised questions), and unstructured (following a short list of broad questions, where the respondent can lead more the conversation) (Specht, 2019, p142). 

This short video on qualitative interviews discusses best practices and covers qualitative interview design, preparation and data collection methods. 

Focus groups   

In this case, a group of people (normally, 4-12) is gathered for an interview where the interviewer asks questions to such group of participants. Group interactions and discussions can be highly productive, but the researcher has to beware of the group effect, whereby certain participants and views dominate the interview (Saunders, Lewis and Thornhill 2015, p419). The researcher can try to minimise this by encouraging involvement of all participants and promoting a multiplicity of views. 

This video focuses on strategies for conducting research using focus groups.  

Check out the guidance on online focus groups by Aliaksandr Herasimenka, which is attached at the bottom of this text box. 

Case study 

Case studies are often a convenient way to narrow the focus of your research by studying how a theory or literature fares with regard to a specific person, group, organisation, event or other type of entity or phenomenon you identify. Case studies can be researched using other methods, including those described in this section. Case studies give in-depth insights on the particular reality that has been examined, but may not be representative of what happens in general, they may not be generalisable, and may not be relevant to other contexts. These limitations have to be acknowledged by the researcher.     

Content analysis 

Content analysis consists in the study of words or images within a text. In its broad definition, texts include books, articles, essays, historical documents, speeches, conversations, advertising, interviews, social media posts, films, theatre, paintings or other visuals. Content analysis can be quantitative (e.g. word frequency) or qualitative (e.g. analysing intention and implications of the communication). It can detect propaganda, identify intentions of writers, and can see differences in types of communication (Specht, 2019, p146). Check this page on collecting, cleaning and visualising Twitter data.

Extra links and resources:  

Research Methods  

A clear and comprehensive overview of research methods by Emerald Publishing. It includes: crowdsourcing as a research tool; mixed methods research; case study; discourse analysis; ground theory; repertory grid; ethnographic method and participant observation; interviews; focus group; action research; analysis of qualitative data; survey design; questionnaires; statistics; experiments; empirical research; literature review; secondary data and archival materials; data collection. 

Doing your dissertation during the COVID-19 pandemic  

Resources providing guidance on doing dissertation research during the pandemic: Online research methods; Secondary data sources; Webinars, conferences and podcasts; 

• Virtual Focus Groups Guidance on managing virtual focus groups

5 Minute Methods Videos

The following are a series of useful videos that introduce research methods in five minutes. These resources have been produced by lecturers and students with the University of Westminster's School of Media and Communication. 

Case Study Research

Research Ethics

Quantitative Content Analysis 

Sequential Analysis 

Qualitative Content Analysis 

Thematic Analysis 

Social Media Research 

Mixed Method Research 

Procedural Method

In this part, provide an accurate, detailed account of the methods and procedures that were used in the study or the experiment (if applicable!). 

Include specifics about participants, sample, materials, design and methods. 

If the research involves human subjects, then include a detailed description of who and how many participated along with how the participants were selected.  

Describe all materials used for the study, including equipment, written materials and testing instruments. 

Identify the study's design and any variables or controls employed. 

Write out the steps in the order that they were completed. 

Indicate what participants were asked to do, how measurements were taken and any calculations made to raw data collected. 

Specify statistical techniques applied to the data to reach your conclusions. 

Provide evidence that you incorporated rigor into your research. This is the quality of being thorough and accurate and considers the logic behind your research design. 

Highlight any drawbacks that may have limited your ability to conduct your research thoroughly. 

You have to provide details to allow others to replicate the experiment and/or verify the data, to test the validity of the research. 

Bibliography

Cottrell, S. (2014). Dissertations and project reports: a step by step guide. Hampshire, England: Palgrave Macmillan.

Lombard, E. (2010). Primary and secondary sources.  The Journal of Academic Librarianship , 36(3), 250-253

Saunders, M.N.K., Lewis, P. and Thornhill, A. (2015).  Research Methods for Business Students.  New York: Pearson Education. 

Specht, D. (2019).  The Media And Communications Study Skills Student Guide . London: University of Westminster Press.  

• << Previous: Introduction & Philosophy
• Next: Ethics >>
• Last Updated: Sep 14, 2022 12:58 PM
• URL: https://libguides.westminster.ac.uk/methodology-for-dissertations

CONNECT WITH US

Research Methods: Literature Reviews

• Annotated Bibliographies
• Literature Reviews
• Scoping Reviews
• Systematic Reviews
• Scholarship of Teaching and Learning
• Persuasive Arguments
• Subject Specific Methodology

A literature review involves researching, reading, analyzing, evaluating, and summarizing scholarly literature (typically journals and articles) about a specific topic. The results of a literature review may be an entire report or article OR may be part of a article, thesis, dissertation, or grant proposal. A literature review helps the author learn about the history and nature of their topic, and identify research gaps and problems.

Steps & Elements

Problem formulation

• Determine your topic and its components by asking a question
• Research: locate literature related to your topic to identify the gap(s) that can be addressed
• Read: read the articles or other sources of information
• Analyze: assess the findings for relevancy
• Evaluating: determine how the article are relevant to your research and what are the key findings
• Synthesis: write about the key findings and how it is relevant to your research

Elements of a Literature Review

• Summarize subject, issue or theory under consideration, along with objectives of the review
• Divide works under review into categories (e.g. those in support of a particular position, those against, those offering alternative theories entirely)
• Explain how each work is similar to and how it varies from the others
• Conclude which pieces are best considered in their argument, are most convincing of their opinions, and make the greatest contribution to the understanding and development of an area of research

Writing a Literature Review Resources

• How to Write a Literature Review From the Wesleyan University Library
• Write a Literature Review From the University of California Santa Cruz Library. A Brief overview of a literature review, includes a list of stages for writing a lit review.
• Literature Reviews From the University of North Carolina Writing Center. Detailed information about writing a literature review.
• Undertaking a literature review: a step-by-step approach Cronin, P., Ryan, F., & Coughan, M. (2008). Undertaking a literature review: A step-by-step approach. British Journal of Nursing, 17(1), p.38-43

Literature Review Tutorial

• << Previous: Annotated Bibliographies
• Next: Scoping Reviews >>
• Last Updated: Feb 29, 2024 12:00 PM
• URL: https://guides.auraria.edu/researchmethods

1100 Lawrence Street Denver, CO 80204 303-315-7700 Ask Us Directions

Types of Literature Review

There are many types of literature review. The choice of a specific type depends on your research approach and design. The following types of literature review are the most popular in business studies:

Narrative literature review , also referred to as traditional literature review, critiques literature and summarizes the body of a literature. Narrative review also draws conclusions about the topic and identifies gaps or inconsistencies in a body of knowledge. You need to have a sufficiently focused research question to conduct a narrative literature review

Systematic literature review requires more rigorous and well-defined approach compared to most other types of literature review. Systematic literature review is comprehensive and details the timeframe within which the literature was selected. Systematic literature review can be divided into two categories: meta-analysis and meta-synthesis.

When you conduct meta-analysis you take findings from several studies on the same subject and analyze these using standardized statistical procedures. In meta-analysis patterns and relationships are detected and conclusions are drawn. Meta-analysis is associated with deductive research approach.

Meta-synthesis, on the other hand, is based on non-statistical techniques. This technique integrates, evaluates and interprets findings of multiple qualitative research studies. Meta-synthesis literature review is conducted usually when following inductive research approach.

Scoping literature review , as implied by its name is used to identify the scope or coverage of a body of literature on a given topic. It has been noted that “scoping reviews are useful for examining emerging evidence when it is still unclear what other, more specific questions can be posed and valuably addressed by a more precise systematic review.” [1] The main difference between systematic and scoping types of literature review is that, systematic literature review is conducted to find answer to more specific research questions, whereas scoping literature review is conducted to explore more general research question.

Argumentative literature review , as the name implies, examines literature selectively in order to support or refute an argument, deeply imbedded assumption, or philosophical problem already established in the literature. It should be noted that a potential for bias is a major shortcoming associated with argumentative literature review.

Integrative literature review reviews , critiques, and synthesizes secondary data about research topic in an integrated way such that new frameworks and perspectives on the topic are generated. If your research does not involve primary data collection and data analysis, then using integrative literature review will be your only option.

Theoretical literature review focuses on a pool of theory that has accumulated in regard to an issue, concept, theory, phenomena. Theoretical literature reviews play an instrumental role in establishing what theories already exist, the relationships between them, to what degree existing theories have been investigated, and to develop new hypotheses to be tested.

At the earlier parts of the literature review chapter, you need to specify the type of your literature review your chose and justify your choice. Your choice of a specific type of literature review should be based upon your research area, research problem and research methods.  Also, you can briefly discuss other most popular types of literature review mentioned above, to illustrate your awareness of them.

[1] Munn, A. et. al. (2018) “Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach” BMC Medical Research Methodology

  John Dudovskiy

Skills for Learning : Dissertations & Literature Reviews

Dissertations  are extended projects in which you choose, research and write about a specific topic. They provide an opportunity to explore an aspect of your subject in detail. You are responsible for managing your dissertation, though you will be assigned a supervisor. Dissertations are typically empirical (based on your own research) or theoretical (based on others’ research/arguments).

The  Dissertation IT Kit  contains information about formatting your dissertation document in Word.

Look at the  Library Subject Guides  for your area. These have information on finding high quality resources for your dissertation. 

We run interactive workshops to help you prepare for your dissertation. Find out more on the  Skills for Learning Workshops  page.

We have online academic skills modules within MyBeckett for all levels of university study. These modules will help your academic development and support your success at LBU. You can work through the modules at your own pace, revisiting them as required. Find out more from our FAQ  What academic skills modules are available?  

Dissertation proposals

What are dissertation proposals.

A dissertation proposal is an outline of your proposed research project. It is what you imagine your dissertation might look like before you start. Consider it a temporary document which might change during the negotiation process between you and your dissertation supervisor.  The proposal can help you clarify exactly what you want to cover in your dissertation. It can also outline how you are going to approach it. Your dissertation plan and structure might change throughout this process as you develop your ideas. Your proposal is the first step towards your goal: a completed dissertation.

Structuring your dissertation proposal

The structure, content, and length of your dissertation proposal will depend on your course requirements. Some courses may require that your aims and objectives are separate from the main body of the proposal. You might be expected to write a literature review, and/or provide a detailed methodology. You might also be asked to include an extensive context for your proposed study. Consult your module handbook or assignment brief for the specific requirements of your course. 

Give each section of your proposal a heading You can also experiment with giving your proposed dissertation a title. Both of these approaches may help you focus and stay on topic. Most dissertation proposals will have a fairly standard structure, under the following headings:

Sections of a dissertation proposal

• Aims and objectives
• Rationale for your study
• Methodology
• Brief literature review
• Benefits of your research

Describe what you plan to investigate. You could write a statement of your topic, a research question(s), or a hypothesis.

• Explain why you want to do this research.
• Write a justification as to why the project is worth undertaking.
• Reasons might include: a gap in existing research; questioning or extending the findings of earlier research; replicating a piece of research to test its reliability.
• Describe and justify how you plan to do the research.
• You might be reviewing the work of others, which mainly involves secondary, or desk-based, research. Or you might plan to collect data yourself, which is primary research. It is common for undergraduate dissertations to involve a mixture of these.
• If you are doing secondary research, describe how you will select your sources. For primary research, describe how you will collect your data. This might include using questionnaires, interviews, archival research, or other methods. 
• Others will have researched this topic before, or something similar.
• The literature review allows you to outline what they have found and where your project fits in. For example, you could highlight disagreements or discrepancies in the existing research.

Outline who might potentially gain from your research and what you might find out or expand upon. For example, there could be implications for practice in a particular profession.

Dissertation style and language

A dissertation is a logical, structured, argument-based exploration of a topic. The style of your writing may vary slightly in each chapter. For example, your results chapter should display factual information, whereas your analysis chapter might be more argument-based. Make sure your language, tone and abbreviations are consistent within each section. Your language should be formal and contain terminology relevant to your subject area. Dissertations have a large word count. It is important to structure your work with headings and a contents page. Use signposting language to help your reader understand the flow of your writing. Charts, tables or images may help you communicate specific information. 

Top tip!  To signpost in your dissertation, use the ‘Signalling Transition’ section of the  Manchester Academic Phrasebank .

Download the Dissertation Project Checklist Worksheet to help with planning your dissertation work. 

• Dissertation Project Checklist Worksheet

The  Dissertation IT Kit  also contains information about formatting your dissertation document in Microsoft Word.

Past dissertations

Exploring past dissertations within your academic field can give you an idea as to how to structure your dissertation and find similar research methodologies. You can access dissertations and theses completed by students at Leeds Beckett and other universities. To find external dissertations, look at our FAQ answer ' Are there other dissertations I can look at?' . To find dissertations completed by Leeds Beckett students, use the FAQ answer ' Can I find copies of past dissertations in the Library? '

Sections of a dissertation

Not all dissertations will follow the same structure.  Your style can change depending on your school. Check your module handbook, assignment brief or speak with your course tutor for further guidance.

To decide what to include:

• Think about your project from an outsider’s perspective. What do they need to know and in what order? What is the most clear and logical way for you to present your research?  
• Discuss your project with your supervisor. Be open about ideas or concerns you have around the structure and content. 

Each section of a dissertation has a different purpose. Think about whether you're doing an empirical or theoretical dissertation and use the headings below to find out what you should be including.

You can also use the Leeds Beckett Dissertation Template to help you understand what your dissertation should look like. 

• Leeds Beckett Dissertation Template

Empirical (research-based)

• 1. Abstract
• 2. Contents Page
• 3. Introduction
• 4. Literature Review
• 5. Methodology
• 6. Findings / Results
• 7. Discussion
• 8. Conclusion
• 9. Reference List / Bibliography
• 10. Appendices

Abstract : provides a brief summary of your whole dissertation.

The abstract outlines the purpose of your research and your methodology (where necessary). You should summarise your main findings and conclusion.

Top tips! Give the reader a sense of why your project is interesting and valuable. Write in the past tense. Aim for about half a page.

Contents page : lists all the sections of your dissertation with the page numbers. Do this last by using the automatic function in Word.

Introduction: introduces the reader to your research project.

Provide context to the topic and define key terms. Ensure that the scope of your investigation is clear. Outline your aims and objectives, and provide a brief description of your research methods. Finally, give an indication of your conclusion/findings.

Top tips! Start broad (background information) and get more specific (your research aims and findings). Try writing the introduction after the literature review and methodology chapters. This way, you will have a better idea of your research aims.

Literature Review : positions your research in relation to what has come before it.

The literature review will summarise prior research on the topic, such as journal articles, books, government reports and data. You should introduce key themes, concepts, theories or methods that provide context for your own research. Analyse and evaluate the literature by drawing comparisons and highlighting strengths and weaknesses. Download the Critical Analysis Questions and Evidence Matrix Worksheets to help you with this process and for more information on literature searching see Finding Information .

• Critical Analysis Questions Worksheet
• Evidence Matrix Worksheet

The literature review should justify the need for your research and highlight areas for further investigation. Avoid introducing your own ideas at this point; instead, compare and comment on existing ideas.

Top tips! Your literature review is not a descriptive summary of various sources. You need to synthesise (bring together) and critically analyse prior research. Sophisticated use of reporting verbs is important for this process. Download our Reporting Verbs Worksheet to help you with this.

• Reporting Verbs Worksheet

Find out more about literature reviews elsewhere on this topic page.

Find out more about critical thinking.

Methodology : provides a succinct and accurate record of the methodology used and justifies your choice of methods.

In this section, you describe the qualitative and/or quantitative methods* used to carry out your research/experiment. You must justify your chosen research methodology and explain how it helps you answer your research question. Where appropriate, explain the rationale behind choices such as procedures, equipment, participants and sample size. You may need to reference specific guidelines that you have used, especially in subjects such as healthcare. If your research involves people, you may also need to demonstrate how it fulfils ethical guidelines.

Top tips! Your account should be sufficiently detailed so that someone else could replicate your research. Write in the passive voice. Remember, at this point you are not reporting any findings.

*Qualitative research is based on opinions and ideas, while quantitative research is based on numerical data.

Find out more about the research process.

Findings/Results : presents the data collected from your research in a suitable format.

Provide a summary of the results of your research/experiment. Consider the most effective methods for presenting your data, such as charts, graphs or tables. Present all your findings honestly. Do not change any data, even if it is not what you expected to find.

Top tips! Whilst you might acknowledge trends or themes in the data, at this stage, you won’t be analysing it closely. If you are conducting qualitative research, this section may be combined with the discussion section. Important additional documents, such as transcriptions or questionnaires, can be added to your appendices.

Discussion : addresses your research aims by analysing your findings.

In this chapter, you interpret and discuss your results and draw conclusions. Identify trends, themes or issues that arise from the findings and discuss their significance in detail. These themes can also provide the basis for the structure of this section. You can draw upon information and concepts from your literature review to help interpret your findings. For example, you can show how your findings build upon or contradict earlier research.

Top tips! Ensure that the points you make are backed up with evidence from your findings. Refer back to relevant information from your literature review to discuss and interpret your findings.

Conclusion : summarises your main points.

Provide an overview of your main findings and demonstrate how you have met your research objectives. Set your research into a wider context by showing how it contributes to current academic debates. Discuss the implications of your research and put forward any recommendations.

Top tips! Do not introduce any new information in this section. Your conclusion should mirror the content of your introduction but offer more conclusive answers.

Reference List / Bibliography : a complete list of all sources used.

List all the sources that you have consulted in the process of your research. Your Reference List or Bibliography must follow specific guidelines for your discipline (e.g. Harvard or OSCOLA). Look through your module handbook or speak to your supervisor for more information.

Find out more about referencing and academic integrity .

Appendix (single) or Appendices (plural):  presents raw data and/or transcripts that aren’t in the main body of your dissertation.

You may have to be selective in the data you present in your findings section. If this is the case, you may choose to present the raw data/extended version in an appendix. If you conduct qualitative research, such as interviews, you will include the transcripts in your appendix. Appendices are not usually included in the word count.

Top tips! Discuss with your supervisor whether you will need an appendix and what to include.

Theoretical (argument based)

• Contents page
• Introduction
• Literature Review
• Main body (divided into chapters)
• Reference list / Bibliography

Provides a brief summary of your whole dissertation.

The abstract outlines the purpose of your research and your methodology (where necessary). You should summarise your main findings and conclusion.

Top tip!  Give the reader a sense of why your project is interesting and valuable. Write in the past tense. Aim for about half a page.

Contents page : lists all the sections of your dissertation with the page numbers. Using the automatic table of contents feature in Microsoft Word can help you format this.

The  Dissertation IT kit provides guidance on how to use these tools. 

Introduces the reader to your research project.

Provide context to the topic and define key terms. Ensure that the scope of your investigation is clear. Outline your aims and objectives, and provide a brief description of your research methods. Introduce your argument and explain why your research topic is important. Finally, give an indication of your conclusion/findings.

Top tip!  Start broad (background information) and get more specific (your research aims and findings). Try writing the introduction after the literature review and methodology chapters. This way, you will have a better idea of your research aims.

Summarises prior research on the topic, such as journal articles, books, and other information sources. You should introduce key themes, concepts, theories or methods that provide context for your own research. You should also analyse and evaluate the literature by drawing comparisons and highlighting strengths and weaknesses. 

Many (although not all) theoretical dissertations will include a separate literature review. You may decide to include this as a separate chapter. Otherwise, you can integrate it into your introduction or first themed chapter.

Find out more about literature reviews on the  Literature Reviews  page.

Divide the main body of your research into chapters organised by chronology or themes. Each chapter should be like a mini-essay that helps you answer your research questions. Like an essay, each chapter should have an introduction, main body and conclusion. Develop your argument and demonstrate critical thinking by drawing on relevant sources. Compare and contrast ideas, and make suggestions or recommendations where relevant. Explain how each chapter helps answer your main research question.

Top tip! Divide each chapter into chunks and use subheadings where necessary to structure your work.

Find out more on the  Critical Thinking  pages. 

Top tip!  Do not introduce any new information in this section. Your conclusion should mirror the content of your introduction but offer more conclusive answers.

List all the sources that you have consulted in the process of your research. Your Reference List or Bibliography must follow specific guidelines for your discipline (Harvard, APA or OSCOLA). Look through your module handbook or speak to your supervisor for more information.

Find out more about  referencing and academic integrity .

Appendix (single) or Appendices (plural):  presents any data, such as images or tables, that aren’t in the main body of your dissertation.

You may have to be selective about the information you include in the main body of your dissertation. If this is the case, you may place data such as images or tables in the appendix. Appendices are not usually included in the word count.

Top tip!  Discuss with your supervisor whether you will need any appendices and what to include.

Your browser does not support iframes.

Skills for Learning home

Artificial intelligence tools

Before using any generative artificial intelligence or paraphrasing tools in your assessments, you should check if this is permitted on your course.

If their use is permitted on your course, you must  acknowledge any use of generative artificial intelligence tools  such as ChatGPT or paraphrasing tools (e.g., Grammarly, Quillbot, etc.), even if you have only used them to generate ideas for your assignment or for proofreading.

• Academic Integrity Module in MyBeckett
• Assignment Calculator
• Building on Feedback
• Disability Advice
• Essay X-ray tool
• International Students' Academic Introduction
• Manchester Academic Phrasebank
• Quote, Unquote
• Skills and Subject Suppor t
• Turnitin Grammar Checker

{{You can add more boxes below for links specific to this page [this note will not appear on user pages] }}

Topic Links

• Academic Integrity Module
• Dissertation IT Kit
• Dissertation Process and Planning Video
• Writing your Literature Review Video
• Dissertations Methodology Chapter Video
• Dissertations Results and Analysis Chapter Video
• Dissertations Editing Video

Resources & Worksheets

• Annotated Bibliographies Worksheet
• CRAAP Test Worksheet
• Literature Review Planning Worksheet
• Paraphrasing and Summarising Information
• Synthesising Sources in Writing Worksheet

Skills for Learning FAQs

0113 812 1000

• University Disclaimer
• Accessibility

Review of Related Literature (RRL)

Ai generator.

The Review of Related Literature (RRL) is a crucial section in research that examines existing studies and publications related to a specific topic. It summarizes and synthesizes previous findings, identifies gaps, and provides context for the current research. RRL ensures the research is grounded in established knowledge, guiding the direction and focus of new studies.

What Is Review of Related Literature (RRL)?

The Review of Related Literature (RRL) is a detailed analysis of existing research relevant to a specific topic. It evaluates, synthesizes, and summarizes previous studies to identify trends, gaps, and conflicts in the literature. RRL provides a foundation for new research, ensuring it builds on established knowledge and addresses existing gaps.

Format of Review of Related Literature (RRL)

The Review of Related Literature (RRL) is a critical part of any research paper or thesis . It provides an overview of existing research on your topic and helps to establish the context for your study. Here is a typical format for an RRL:

1. Introduction

• Purpose : Explain the purpose of the review and its importance to your research.
• Scope : Define the scope of the literature reviewed, including the time frame, types of sources, and key themes.

2. Theoretical Framework

• Concepts and Theories : Present the main theories and concepts that underpin your research.
• Relevance : Explain how these theories relate to your study.

3. Review of Empirical Studies

• Sub-theme 1 : Summarize key studies, including methodologies, findings, and conclusions.
• Sub-theme 2 : Continue summarizing studies, focusing on different aspects or variables.
• Sub-theme 3 : Include any additional relevant studies.

4. Methodological Review

• Approaches : Discuss the various methodologies used in the reviewed studies.
• Strengths and Weaknesses : Highlight the strengths and weaknesses of these methodologies.
• Gaps : Identify gaps in the existing research that your study aims to address.

5. Synthesis and Critique

• Integration : Integrate findings from the reviewed studies to show the current state of knowledge.
• Critique : Critically evaluate the literature, discussing inconsistencies, limitations, and areas for further research.

6. Conclusion

• Summary : Summarize the main findings from the literature review.
• Research Gap : Clearly state the research gap your study will address.
• Contribution : Explain how your study will contribute to the existing body of knowledge.

7. References

• Citation Style : List all the sources cited in your literature review in the appropriate citation style (e.g., APA, MLA, Chicago).

Review of Related Literature (RRL) 1. Introduction This review examines research on social media’s impact on mental health, focusing on anxiety and depression across various demographics over the past ten years. 2. Theoretical Framework Anchored in Social Comparison Theory and Uses and Gratifications Theory, this review explores how individuals’ social media interactions affect their mental health. 3. Review of Empirical Studies Adolescents’ Mental Health Instagram & Body Image : Smith & Johnson (2017) found Instagram use linked to body image issues and lower self-esteem among 500 high school students. Facebook & Anxiety : Brown & Green (2016) showed Facebook use correlated with higher anxiety and depressive symptoms in a longitudinal study of 300 students. Young Adults’ Mental Health Twitter & Stress : Davis & Lee (2018) reported higher stress levels among heavy Twitter users in a survey of 400 university students. LinkedIn & Self-Esteem : Miller & White (2019) found LinkedIn use positively influenced professional self-esteem in 200 young professionals. Adult Mental Health General Social Media Use : Thompson & Evans (2020) found moderate social media use associated with better mental health outcomes, while excessive use correlated with higher anxiety and depression in 1,000 adults. 4. Methodological Review Studies used cross-sectional surveys, longitudinal designs, and mixed methods. Cross-sectional surveys provided large data sets but couldn’t infer causation. Longitudinal studies offered insights into long-term effects but were resource-intensive. Mixed methods enriched data through qualitative insights but required careful integration. 5. Synthesis and Critique The literature shows a complex relationship between social media and mental health, with platform-specific and demographic-specific effects. However, reliance on self-reported data introduces bias, and many cross-sectional studies limit causal inference. More longitudinal and experimental research is needed. 6. Conclusion Current research offers insights into social media’s mental health impact but leaves gaps, particularly regarding long-term effects and causation. This study aims to address these gaps through comprehensive longitudinal analysis. 7. References Brown, A., & Green, K. (2016). Facebook Use and Anxiety Among High School Students . Psychology in the Schools, 53(3), 257-264. Davis, R., & Lee, S. (2018). Twitter and Psychological Stress: A Study of University Students . Journal of College Student Development, 59(2), 120-135. Miller, P., & White, H. (2019). LinkedIn and Its Effect on Professional Self-Esteem . Journal of Applied Psychology, 104(1), 78-90. Smith, J., & Johnson, L. (2017). The Impact of Instagram on Teen Body Image . Journal of Adolescent Health, 60(5), 555-560. Thompson, M., & Evans, D. (2020). The Relationship Between Social Media Use and Mental Health in Adults . Cyberpsychology, Behavior, and Social Networking, 23(4), 201-208.

Review of Related Literature (RRL) Examples

Review of related literature in research, review of related literature in research paper, review of related literature qualitative research.

Review of Related Literature Quantitative Research

More Review of Related Literature (RRL) Examples

• Impact of E-learning on Student Performance
• Effectiveness of Mindfulness in Workplace
• Green Building and Energy Efficiency
• Impact of Technology on Healthcare Delivery
• Effects of Nutrition on Cognitive Development in Children
• Impact of Employee Training Programs on Productivity
• Effects of Climate Change on Biodiversity
• Impact of Parental Involvement on Student Achievement
• Effects of Mobile Learning on Student Engagement
• Effects of Urban Green Spaces on Mental Health

Purpose of the Review of Related Literature (RRL)

The Review of Related Literature (RRL) serves several critical purposes in research:

• Establishing Context : It situates your research within the broader field, showing how your study relates to existing work.
• Identifying Gaps : It highlights gaps, inconsistencies, and areas needing further exploration in current knowledge, providing a clear rationale for your study.
• Avoiding Duplication : By reviewing what has already been done, it helps ensure your research is original and not a repetition of existing studies.
• Building on Existing Knowledge : It allows you to build on the findings of previous research, using established theories and methodologies to inform your work.
• Theoretical Foundation : It provides a theoretical basis for your research, grounding it in existing concepts and theories.
• Methodological Insights : It offers insights into the methods and approaches used in similar studies, helping you choose the most appropriate methods for your research.
• Establishing Credibility : It demonstrates your familiarity with the field, showing that you are well-informed and have a solid foundation for your research.
• Supporting Arguments : It provides evidence and support for your research questions, hypotheses, and objectives, strengthening the overall argument of your study.

How to Write Review of Related Literature (RRL)

Writing a Review of Related Literature (RRL) involves several key steps. Here’s a step-by-step guide:

1. Define the Scope and Objectives

• Determine the Scope : Decide on the breadth of the literature you will review, including specific themes, time frame, and types of sources.
• Set Objectives : Clearly define the purpose of the review. What do you aim to achieve? Identify gaps, establish context, or build on existing knowledge.

2. Search for Relevant Literature

• Identify Keywords : Use keywords and phrases related to your research topic.
• Use Databases : Search academic databases like Google Scholar, PubMed, JSTOR, etc., for relevant articles, books, and papers.
• Select Sources : Choose sources that are credible, recent, and relevant to your research.

3. Evaluate and Select the Literature

• Read Abstracts and Summaries : Quickly determine the relevance of each source.
• Assess Quality : Consider the methodology, credibility of the authors, and publication source.
• Select Key Studies : Choose studies that are most relevant to your research questions and objectives.

4. Organize the Literature

• Thematic Organization : Group studies by themes or topics.
• Chronological Organization : Arrange studies in the order they were published to show the development of ideas over time.
• Methodological Organization : Categorize studies by the methods they used.

5. Write the Review

• State the purpose and scope of the review.
• Explain the importance of the topic.
• Theoretical Framework : Present and discuss the main theories and concepts.
• Summarize key studies, including their methodologies, findings, and conclusions.
• Organize by themes or other chosen organizational methods.
• Methodological Review : Discuss the various methodologies used, highlighting their strengths and weaknesses.
• Synthesis and Critique : Integrate findings, critically evaluate the literature, and identify gaps or inconsistencies.
• Summarize the main findings from the literature review.
• Highlight the research gaps your study will address.
• State how your research will contribute to the existing knowledge.

6. Cite the Sources

• Use Appropriate Citation Style : Follow the required citation style (e.g., APA, MLA, Chicago).
• List References : Provide a complete list of all sources cited in your review.

What is an RRL?

An RRL summarizes and synthesizes existing research on a specific topic to identify gaps and guide future studies.

Why is RRL important?

It provides context, highlights gaps, and ensures new research builds on existing knowledge.

How do you write an RRL?

Organize by themes, summarize studies, evaluate methodologies, identify gaps, and conclude with relevance to current research.

What sources are used in RRL?

Peer-reviewed journals, books, conference papers, and credible online resources.

How long should an RRL be?

Length varies; typically 10-20% of the total research paper.

What are common RRL mistakes?

Lack of organization, insufficient synthesis, over-reliance on outdated sources, and failure to identify gaps.

Can an RRL include non-scholarly sources?

Primarily scholarly, but reputable non-scholarly sources can be included for context.

What is the difference between RRL and bibliography?

RRL synthesizes and analyzes the literature, while a bibliography lists sources.

How often should an RRL be updated?

Regularly, especially when new relevant research is published.

Can an RRL influence research direction?

Yes, it identifies gaps and trends that shape the focus and methodology of new research.

Text prompt

• Instructive
• Professional

10 Examples of Public speaking

20 Examples of Gas lighting

• Open access
• Published: 04 June 2024

Comparison of Bayesian approaches for developing prediction models in rare disease: application to the identification of patients with Maturity-Onset Diabetes of the Young

• Pedro Cardoso   ORCID: orcid.org/0000-0002-1014-9058 1 ,
• Timothy J. McDonald   ORCID: orcid.org/0000-0003-3559-6660 1 ,
• Kashyap A. Patel   ORCID: orcid.org/0000-0002-9240-8104 1 ,
• Ewan R. Pearson   ORCID: orcid.org/0000-0001-9237-8585 2 ,
• Andrew T. Hattersley   ORCID: orcid.org/0000-0001-5620-473X 1 ,
• Beverley M. Shields   ORCID: orcid.org/0000-0003-3785-327X 1   na1 &
• Trevelyan J. McKinley   ORCID: orcid.org/0000-0002-9485-3236 1   na1  

BMC Medical Research Methodology volume  24 , Article number:  128 ( 2024 ) Cite this article

21 Accesses

1 Altmetric

Metrics details

Clinical prediction models can help identify high-risk patients and facilitate timely interventions. However, developing such models for rare diseases presents challenges due to the scarcity of affected patients for developing and calibrating models. Methods that pool information from multiple sources can help with these challenges.

We compared three approaches for developing clinical prediction models for population screening based on an example of discriminating a rare form of diabetes (Maturity-Onset Diabetes of the Young - MODY) in insulin-treated patients from the more common Type 1 diabetes (T1D). Two datasets were used: a case-control dataset (278 T1D, 177 MODY) and a population-representative dataset (1418 patients, 96 MODY tested with biomarker testing, 7 MODY positive). To build a population-level prediction model, we compared three methods for recalibrating models developed in case-control data. These were prevalence adjustment (“offset”), shrinkage recalibration in the population-level dataset (“recalibration”), and a refitting of the model to the population-level dataset (“re-estimation”). We then developed a Bayesian hierarchical mixture model combining shrinkage recalibration with additional informative biomarker information only available in the population-representative dataset. We developed a method for dealing with missing biomarker and outcome information using prior information from the literature and other data sources to ensure the clinical validity of predictions for certain biomarker combinations.

The offset, re-estimation, and recalibration methods showed good calibration in the population-representative dataset. The offset and recalibration methods displayed the lowest predictive uncertainty due to borrowing information from the fitted case-control model. We demonstrate the potential of a mixture model for incorporating informative biomarkers, which significantly enhanced the model’s predictive accuracy, reduced uncertainty, and showed higher stability in all ranges of predictive outcome probabilities.

We have compared several approaches that could be used to develop prediction models for rare diseases. Our findings highlight the recalibration mixture model as the optimal strategy if a population-level dataset is available. This approach offers the flexibility to incorporate additional predictors and informed prior probabilities, contributing to enhanced prediction accuracy for rare diseases. It also allows predictions without these additional tests, providing additional information on whether a patient should undergo further biomarker testing before genetic testing.

Peer Review reports

Clinical prediction models can be useful in rare diseases to aid earlier diagnosis and more appropriate management. However, developing these models can be challenging as suitable data sources for model development may be difficult to acquire. The prevalence of a rare disease in a population-of-interest can be informed by population cohorts [ 1 ], but low numbers of cases in these datasets limit the ability to identify risk factors and produce robust predictive models for disease risk in the general population [ 2 ]. Case-control studies [ 3 ] enrich the study population with more disease cases than a random sample from the population, facilitating more robust estimates of associations between patient features and disease risk using measures such as odds ratios. Furthermore, the rise of rare disease registries [ 4 ] makes recruiting larger case numbers for these studies easier. However, from a clinical perspective, disease risk probabilities are more natural metrics than odds ratios for diagnosis or screening purposes, but estimated risk probabilities from case-control data will be overestimated as they are not based on random samples from the general population [ 5 , 6 ]. A key challenge is, therefore, how to produce well-calibrated estimates of individual disease risk probabilities for rare diseases in the general population, utilising information from different data sources.

Various methods have been developed that borrow information from one population and recalibrate their outputs to be valid in another population [ 7 , 8 , 9 , 10 , 11 , 12 , 13 ]. These approaches include simple methods such as adjustments of the likelihood ratio based on the sensitivity and specificity of the test at various thresholds [ 14 ] or offset updating to adjust the overall model probabilities according to a more appropriate population prevalence [ 10 , 11 ]. However, these approaches are limited and would not account for differences in patient characteristics that may occur in different datasets, which could be a particular problem in case-control studies when enriching for a particular disease or when only collecting specific controls, which would ignore more “grey-area” patients that may be seen in a population setting. More complex techniques are available, such as shrinkage methods to adjust the intercept and model coefficients [ 7 , 12 ], or previous studies could be used to inform the prior belief of model parameters in Bayesian modelling [ 15 ]. Although more sophisticated, these approaches would need data from multiple sources that may not be available for rare diseases. In addition, datasets may not always contain the same information for rare diseases, and specific testing or features may only be available to a subset of patients. More flexible approaches are needed that would allow modelling in these situations.

We use a specific motivating example of developing a prediction model for a rare form of diabetes called Maturity-Onset Diabetes of the Young (MODY) that can be used to inform referral decisions for genetic screening for the condition. In this study, we 1) evaluate a range of approaches for appropriately recalibrating model probabilities in prediction models for rare diseases utilising different data sources (including case-control data, prevalence estimates, and population datasets) and 2) develop a Bayesian hierarchical mixture modelling approach which can combine a clinical features risk model with additional informative biomarker test information, utilising prior information from other data sources to account for missing data and ensure that the recalibrated probabilities are clinically plausible given specific test results. This latter approach also allows for predictions for new individuals who do not have biomarker test results (since these are not currently routinely collected for MODY), which greatly facilitates using such a prediction model in clinical practice. We also show how the model can help inform on the utility of additional biomarker testing before making a final screening decision for MODY.

Motivating example

Our motivating disease system in this manuscript is a rare young-onset genetic form of diabetes called Maturity-Onset Diabetes of the Young (MODY) [ 16 ], which is estimated to account for 1–2% of all diabetes cases [ 17 , 18 ]. MODY is challenging to identify and is estimated to be misdiagnosed in up to 77% of cases [ 19 ]. Diagnostic genetic testing is expensive; however, it is crucial to properly diagnose as these patients do not require treatment with insulin injections [ 20 ], unlike the most common young-onset form of diabetes, type 1 diabetes (T1D).

Statistical models that use patient characteristics to predict the probability of having MODY can aid decisions regarding which patients to refer for diagnostic MODY testing. One such set of models is routinely used in clinical practice via an online calculator [ 14 ] (found at: https://www.diabetesgenes.org/exeter-diabetes-app/ModyCalculator ) and has been shown to improve positive test rates of new MODY cases [ 19 ]. These prediction models for MODY were developed using case-control data and recalibrated to population prevalences using conversion tables derived from the sensitivities and specificities at different probability thresholds [ 14 ]. There are several consequences of this approach for prevalence adjustment: i) the recalibrated probabilities end up being grouped; ii) individuals cannot have a recalibrated probability that is lower than the estimated prevalence in the general population; and iii) the recalibrated probabilities can be sensitive to the choice of grouping used. Addressing these limitations would be important, but the most appropriate approach for adjusting for the prevalence is unclear.

In addition, since the original model development, biomarker screening tests (C-peptide and islet autoantibodies [ 21 , 22 ]) have become routinely available clinically, and the results of these tests could significantly alter the probability of MODY. C-peptide is a measure of endogenous insulin secretion, and islet autoantibodies are markers of the autoimmune process in T1D. MODY is characterised by non-insulin dependence, so these patients produce significant amounts of their own endogenous insulin (have positive C-peptide), and they do not have the autoimmune process associated with T1D (negative islet autoantibodies), whereas being C-peptide negative (i.e. insulin deficient) or having positive islet autoantibodies is characteristic of T1D. Finding approaches to build these test results into the recalibration would have considerable advantages.

The diagnosis of MODY requires expensive genetic testing. Currently, patients are referred for diagnostic genetic testing on an ad-hoc basis when the clinician considers a MODY diagnosis. In line with guidelines (ISPAD [ 23 ] and NHS genomic testing criteria [ 24 ]), criteria for referring can include:

Clinical presentation and patient features (including age at diagnosis, BMI, treatment, measures of glucose control (HbA 1c ) and family history of diabetes);

Results of biomarker testing (C-peptide and islet autoantibodies);

The use of prediction models in the form of the MODY calculator (can be found at: https://www.diabetesgenes.org/exeter-diabetes-app/ModyCalculator ).

Study population

For model development and recalibration, we used data from two sources comprising patients with confirmed MODY and insulin-treated patients with T1D, the predominant alternative diagnosis in young-onset patients:

Case-control dataset (Fig. 1 a)

This dataset was used to develop the original MODY prediction model [ 14 ]. All participants were diagnosed with diabetes between the ages of 1 and 35. T1D was defined as occurring in patients treated with insulin within 6 months of diagnosis [ 14 ]. The dataset includes 278 patients with T1D and 177 probands with a genetic diagnosis of MODY obtained from referrals to the Molecular Genetics Laboratory at the Royal Devon and Exeter NHS Foundation Trust, UK. The dataset comprises the following variables: sex, age-at-diagnosis, age-at-recruitment, BMI, parents affected with diabetes and HbA 1c (%). No biomarker data are available.

Population-representative dataset (UNITED – Fig. 1 b)

The UNITED study [ 25 ] was a population-representative cohort that recruited 62% of all patients with diabetes diagnosed between the ages of 1 and 30 in two regions of the UK (Exeter and Tayside) ( n =1418). Due to the expense of genetic testing, a screening strategy with C-peptide and islet autoantibody testing was used to narrow down the cohort eligible for MODY testing (Fig. 1 b).

Structure of a ) case-control and b ) UNITED (population) datasets. \({\text{MODY}}^{+}\) corresponds to a positive test when genetically tested for MODY and \({\text{MODY}}^{-}\) corresponds to a negative test when genetically tested for MODY. C + = C-peptide positive, C - = C-peptide negative, A + = Antibody positive, A - = Antibody negative

For this model, consistent with the original model [ 14 ], we analysed all patients insulin-treated within 6 months of diagnosis, corresponding to 1171 patients, of which 96 were tested for MODY (given that they were C-peptide positive and antibody negative) and 7 MODY cases were diagnosed (Fig. 1 b). The dataset is comprised of the following variables: sex, age-at-diagnosis, age-at-recruitment, BMI, parents affected with diabetes and HbA 1c (%), with additional C-peptide and islet autoantibodies test results.

Approaches for recalibration

The analysis in this paper was split into three different scenarios to enable population-appropriate probabilities to be calculated with and without the additional biomarker information:

Scenario a) Clinical features model ignoring biomarker information. For this analysis, we used all patients in the population-representative dataset (UNITED). This scenario assumes all those not MODY tested are \({\text{MODY}}^{-}\) in the population cohort, i.e. 7 MODY positive patients and 1,164 MODY negative (of which 1,075 were not tested for MODY but are assumed to be MODY negative for the analysis since the biomarker results are inconsistent with MODY).

Scenario b) Clinical features model in only those pre-screened to be at increased probability of MODY based on the biomarkers. This included 96 patients, of which 7 are MODY positive and 89 MODY negative. This scenario only analyses patients in the population cohort who had genetic testing (i.e. tested C-peptide positive and autoantibody negative), so it provides more appropriate model probabilities in patients with these test results indicating a higher risk of MODY, but simply rules out MODY (does not provide a probability) in those who are C-peptide negative or antibody positive.

Scenario c) Model fully incorporating both clinical features and biomarker information. We analysed all patients in the population cohort and included biomarker information. For this analysis, we included 96 patients who had testing for MODY (7 MODY positive and 89 MODY negative) and 1,075 patients who did not have testing for MODY. The biomarker information of those not MODY tested was used to more appropriately adjust the model probabilities (151 C-peptide positive and autoantibody positive, 924 C-peptide negative) (Fig.  1 b).

We explored six approaches for producing predictions using different degrees of data availability, which fall into three groups:

Approaches that only utilise case-control data and adjust to a known population prevalence: Original and Offset .

Approaches that utilise a case-control dataset and additional calibration dataset (e.g. the UNITED population dataset in this study): Re-estimation and Recalibration .

Approaches that utilise additional data on informative diagnostic tests and provide biologically plausible constraints: mixture model approaches (for both Re-estimation and Recalibration ). This mixture model splits individuals into two groups according to their diagnostic test information (a C-peptide negative or antibody positive group: \({C}^{-}\cup {A}^{+}\) ; and a C-peptide positive and antibody negative group: \({C}^{+}\cap {A}^{-}\) ). We use an informative prior distribution to constrain the probability of having MODY in the \({C}^{-}\cup {A}^{+}\) group and use one of the other recalibration methods in the \({C}^{+}\cap {A}^{-}\) group.

The Supplementary Materials Notation section contains a glossary of mathematical symbols used throughout the article. We fit these models using the package NIMBLE [ 26 , 27 ] (version 1.0.1) in the software R [ 28 ] (version 4.3.2).

1. Training dataset only approaches

Training data model.

Let \({M}_{j}^{C}\) be a binary variable denoting whether an individual \(j\) in the case-control data set has MODY or not, such that

We then model:

where the log odds are given by:

with \({X}_{jv}^{C} (v=1,\dots ,p)\) a set of \(p\) covariates for individual \(j\) . We put independent vague \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) priors on the regression parameters.

The posterior for this model then takes the form:

where \(\theta =\left({\beta }_{v}^{C};v=0,\dots ,p\right)\) , with \(\pi \left(\cdot \right)\) denoting the relevant probability (density) mass functions derived above for the model and joint prior distribution.

Original approach

This method was implemented by Shields et al . (2012) [ 14 ] during the development of the original MODY prediction model. The approach fitted a model to a case-control dataset using the patients’ characteristics and used the relationship:

where \({M}^{+}\) is the event that the patient has MODY, and \({R}^{+}\) is whether a hypothetical “test” is positive (and similarly for \({M}^{-}\) and \({R}^{-}\) ). In this case, \({R}^{+}\) is derived by applying a threshold, \({p}^{*}\) , to the predicted probabilities \({p}_{j}^{C}\) obtained from a training model (see eq. ( 2 )) for a given individual \(j\) , such that an individual is classed as positive if \({p}_{j}^{C}>{p}^{*}\) and negative otherwise.

Therefore, for a given choice of \({p}^{*}\) , estimates of the sensitivity, \(P\left({R}^{+}|{M}^{+}\right)\) , and specificity, \(P\left({R}^{-}|{M}^{-}\right)\) , of these classifications at a range of thresholds were calculated using the case-control data. \(P\left({M}^{+}\right)\) is then chosen as an estimate of the prevalence of MODY in the general population, which in the original model was given by 0.7% [ 14 ], which assumed no knowledge of biomarker test results. In this paper, we adjusted slightly differently depending on scenario a) or scenario b). In scenario a), we estimated the pre-test probability to be 0.6% (informed by the prevalence of MODY in the UNITED dataset). For scenario b), we estimated the pre-test probability to be 7.3% (informed by the prevalence of MODY in those who were C-peptide positive and antibody negative in UNITED).

For a new individual in the general population, with covariates \({X}_{i}^{*}\) say, then one can derive an estimate for \({p}_{i}^{C}\) (based on eq. ( 2 )) as

before using Table 1 to map their predicted \({p}_{i}^{C}\) from the case-control model to a recalibrated probability of having MODY in the general population.

Albert Offset approach

This approach was proposed by Albert (1982) [ 10 ] and similarly to the method above, leverages the relationship:

where \(X\) is a set of explanatory variables. In words:

For the training data (C – case-control dataset), we use the same model for \({M}_{j}^{C}\) and \({p}_{j}^{C}\) as before (see eqs. ( 1 ) and ( 2 )), and then the idea is that if we know the disease odds in the training data, then we can re-write eq. ( 2 ) as:

hence the original \({\beta }_{0}^{C}={\eta }_{0}^{C}+\mathrm{log }(\text{pre-test odds training})\) . Therefore, under the assumption that the likelihood ratio for any given set of covariates is the same in the training and calibration datasets, then for a new individual \(i\) in the general population, with covariates \({X}_{i}^{*}\) , we can recalibrate as:

This approach gives individual-level recalibration probabilities that do not rely on thresholding. The Albert Offset approach requires a training dataset for fitting the original model and an estimate of the disease odds in both the training data and the population-of-interest. For this cohort, as before, you could adapt the offset based on the prevalence of MODY of 0.6% based on scenario a) or 7.3% based on scenario b). We also explore an example where the likelihood ratio assumption is not maintained between datasets for illustrative purposes. We put independent vague \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) priors on the regression parameters.

2. Population-representative dataset approaches

Re-estimation approach.

This approach fits a new model directly to the population-representative dataset (UNITED), ignoring the case-control dataset entirely. Given sufficient cases and controls in a given dataset, this model fitted using, e.g. maximum likelihood, will give asymptotically unbiased estimates for the odds ratios and probabilities. However, for rare diseases, one would have to have very large sample sizes to get sufficient numbers of cases to develop an entirely new model. As a comparison, we use the model structure developed in the case-control dataset and then refit the model to the population-representative dataset (UNITED). Here, we denote the MODY status for individual \(i\) in the UNITED dataset as \({M}_{i}^{U}\) , and model this as

with \({X}_{iv}^{U}\) \(\left(v=1,\dots ,p\right)\) a set of \(p\) covariates for individual \(i\) . We place independent vague \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) priors on the regression parameters.

where \(\theta ={(\beta }_{v}^{U}; v=1,\dots ,p)\) , with \(\pi (\cdot)\) denoting the relevant probability (density) mass functions derived above for the model and joint prior distribution.

Recalibration approach

In the context of the models developed here, the Recalibration approach [ 7 ] uses a model fitted to the case-control dataset to generate predictions of the linear predictor for each individual in the population-representative data set (UNITED). In the training data, for individual \(j\) , \({M}_{j}^{C}\) and \({p}_{j}^{C}\) are modelled as before (see eqs. ( 1 ) and ( 2 )), and then for each individual \(i\) in the calibration dataset (UNITED), with predictors \({X}_{i}^{U}\) , the linear predictors \({z}_{i}=\widehat{{\upbeta }_{0}^{\text{C}}}+\widehat{{\upbeta }_{1}^{\text{C}}}{X}_{i1}^{U}+\dots +\widehat{{\upbeta }_{p}^{\text{C}}}{X}_{ip}^{U}\) are calculated, where \(\widehat{{\beta }_{v}^{C}}\) is a point estimate of the \(v\) th regression parameter from the case-control model. These \({z}_{i}\) terms are then used as covariates in a second (shrinkage) model:

This approach [ 7 ] can have the effect of scaling the odds ratios and intercept terms where necessary, and a side-effect is that if no recalibration is required, then \({\gamma }_{0}=0\) and \({\gamma }_{1}=1\) . Again, these approaches could be built using scenarios a) and b), dependent on the assumptions we are willing to make with UNITED. The method used by Steyerberg et al. (2004) [ 7 ] uses the point predictions for \({z}_{i}\) based on the maximum likelihood estimates from the case-control data, which ignores the uncertainty in the estimations of \({z}_{i}\) . Instead, we develop a joint Bayesian hierarchical model where we simultaneously fit both models and propagate the uncertainties directly from the case-control model to the recalibration model [ 29 ]. We put independent vague \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) prior distributions on the regression parameters, with a \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) prior for \({\gamma }_{0}\) and a \({\text{Normal}}\left(\mu =1, {\text{sd}}=10\right)\) prior for \({\gamma }_{1}\) .

The posterior for this joint model then takes the form:

where \(\theta =\left({\theta }^{U},{\theta }^{C}\right)\) corresponds to the full vector of parameters, with \({\theta }^{U}=\left({\gamma }_{0},{\gamma }_{1}\right)\) and \({\theta }^{C}=\left({\beta }_{v}^{C};v=1,\dots ,p\right)\) , with \(\pi \left(\cdot \right)\) denoting the relevant probability (density) mass functions derived above for the different component models and joint prior distribution.

3. Mixture model approach

One area of development in this manuscript is how to incorporate biomarker test information into the model when the biomarker tests can place very strong constraints on the post-recalibration probabilities depending on their specific values. For example, a simple way to include a binary test result would be to add another covariate into the linear predictor in one of the previous methods. In the analysis, the biomarker data only exists in the calibration data (UNITED) but not the training data (case-control), so this approach would only use information from the calibration data to estimate the parameters relating to the biomarkers. Since there are few cases in the calibration data, this would necessarily result in large standard errors for the estimated effects and could lead to biologically implausible estimates. For example, an individual who is C-peptide negative or antibody positive can be considered to have a very low probability of having MODY, justified through prior data and biological plausibility (C-peptide negativity means that an individual is producing negligible amounts of their own insulin, which defines T1D). In clinical practice, an individual with these biomarker results would be treated as having T1D, which is equivalent to assuming that the probability of having MODY given these results is exactly zero. However, this approach does not allow for the rare (but possible) event that an individual has a positive genetic MODY test but is antibody positive or C-peptide negative (which would ideally also allow for imperfect sensitivities and specificities of the biomarker tests).

Using the mixture model approach in scenario c), it is possible to incorporate a non-zero prior probability of having MODY in these cases, where we use independent data sets to inform the prior distribution for this probability. We note that the mixture model allows for different prior constraints to be used for different subsets of the data: here the prior probability of having MODY is very low for C-peptide negative or antibody positive individuals [ 21 , 22 ], but is not similarly constrained for C-peptide positive and antibody negative individuals. Similar ideas could be used for other diseases where the prior information may not be as strong.

For the UNITED data, we let

We then set:

Letting \({X}_{i}^{U}\) be a vector of additional covariates for individual \(i\) , we can model \({M}_{i}^{U}\) as

We model \({p}_{{M}^{+}|{C}^{-}\cup {A}^{+}}\) using a \({\text{Beta}}(\alpha = 2.2,\beta =7361.3)\) prior probability distribution (see Supplementary Materials Prior elicitation section for a justification of this choice). We then model \({p}_{{M}^{+}|{C}^{+}\cap { A}^{-},{X}_{i}}\) differently, depending on whether we use the Re-estimation or Recalibration approaches (see below).

For the Re-estimation approach we model

and to finalise, we put independent vague \({\text{Normal}}\,\left(\mu =0, {\text{sd}}=10\right)\) priors on the regression parameters.

For the Recalibration approach we also utilise the case-control data. If we let \({M}_{j}^{C}\) be the MODY status for individual \(j\) in the case-control dataset, with vector of covariates \({X}_{j}^{C}\) , then \({M}_{j}^{C}\) and \({p}_{j}^{C}\) are modelled as before (see eqs. ( 1 ) and ( 2 )). Then, for individual \(i\) in the UNITED dataset (with \({C}_{i}^{+}\cap {A}_{i}^{-}\) ), we model

Incorporating biomarker test results

To allow for predictions in the absence of biomarker test results (which are not routinely collected in clinical practice), we model

with \({X}^{*U}\) comprised of the variables BMI, age-of-diagnosis, age-of-recruitment and parents affected with diabetes (here we use restricted cubic splines with 3 knots to model the continuous variables). In this case the predicted probability of MODY for an individual with unknown test results will be a weighted average of the \({p}_{{M}^{+}|{C}^{-}\cup {A}^{+}}\) and \({p}_{{M}^{+}|{C}^{+}\cap { A}^{-},{X}_{i}}\) , weighted by the probability of being \({C}^{-}\cup {A}^{+}\) based on suitable individual-level characteristics. We place independent vague \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) prior distributions on the regression parameters, with a \({\text{Normal}}\left(\mu =0, {\text{sd}}=10\right)\) prior on \({\gamma }_{0}\) and a \({\text{Normal}}\left(\mu =1, {\text{sd}}=10\right)\) prior on \({\gamma }_{1}\) .

For the Re-estimation mixture , the posterior then takes the form:

where \(\theta =\left({\theta }^{U},{\theta }^{T},{p}_{M^+|{C}^{-}\cup {A}^{+}}\right)\) corresponds to the full vector of parameters, with \({\theta }^{U}=\left({\beta }_{v}^{U};v=1,\dots ,p\right)\) and \({\theta }^{T}=({\beta }_{v}^{*};v=1,\dots ,r)\) with \(\pi \left(\cdot \right)\) denoting the relevant probability (density) mass functions derived above for the different component models and joint prior distribution.

For the Recalibration mixture , the posterior then takes the form:

where \(\theta =({\theta }^{U},{\theta }^{T},{p}_{M^+|{C}^{-}\cup {A}^{+}},{\theta }^{C})\) corresponds to the full vector of parameters, with \({\theta }^{U}=\left({\gamma }_{0},{\gamma }_{1}\right)\) , \({\theta }^{C}=\left({\theta }_{v}^{C};v=1,\dots ,p\right)\) and \({\theta }^{T}=\left({\beta }_{v}^{*};v=1,\dots ,r\right)\) , with \(\pi \left(\cdot \right)\) denoting the relevant probability (density) mass functions derived above for the different component models and joint prior distributions.

Assessment of model performance, calibration and stability analysis

In scenario a), we validate fitted probabilities for all patients in UNITED (setting those with missing MODY testing to \({\text{MODY}}^{-}\) ). In scenarios b) and c), we only validate fitted probabilities on \({C}^{-}\cup {A}^{+}\) patients as these were the only patients who had pre-screening based on biomarkers and had genetic testing of MODY genes. The area under the receiver operating characteristic (AUROC) curve was used as a measure of overall discrimination performance. Calibration curves were plotted to visualise how well the predicted probabilities were calibrated against the observed data. For the calibration curves, predicted probabilities were grouped by quintiles and plotted against the observed probability of positive individuals within each quintile. To assess convergence, we monitored the available parameters for evidence of convergence and Gelman-Rubin \(\widehat{R}\) values [ 30 ]. Further validation procedures are explained in the Supplementary Materials Stability analysis section.

Comparing datasets

In the case-control dataset, 177 out of 455 patients had MODY, leading to an enriched proportion with MODY of 40%. In contrast, in the recalibration population (UNITED) cohort, 7 out of 1171 patients (0.6%) had MODY, much more consistent with the prevalence of MODY in the population. The characteristics of patients in the two datasets were broadly similar (sFig. 1).

Models and their recalibration from the 6 different approaches

All models in this study converged quickly, so we ran four chains of 500,000 iterations, with the first 300,000 discarded for burn-in in each case (sFig. 2-4).

The first, recalibration approach, the Original approach achieved an \(\widehat{R}=1.0\) for all parameters. As expected, the choice of prevalence used for recalibration affected the conversion probabilities. Table 1 shows the different post-recalibration probabilities of having MODY using the different prevalences for both scenarios a) and b), with post-recalibration probabilities more appropriately higher in scenario b) to allow for the biomarker results in those who were C-peptide positive and antibody negative.

Table 2 describes the model parameter estimates for the Albert Offset and Re-estimation approaches in scenarios a) and b). The Albert Offset approach achieved an \(\widehat{R}=1.0\) , and the Re-estimation approach achieved an \(\widehat{R}=1.0\) for all parameters. Coefficients were quite different in the various approaches.

The Recalibration approach achieved an \(\widehat{R}=1.0\) for all parameters. In scenario a), estimates were \({\gamma }_{0}=-4.39\) (95%CI -5.41; -3.56) and \({\gamma }_{1}=0.96\) (95%CI 0.49; 1.54) and in scenario b), the estimates were \({\gamma }_{0}=-2.26\) (95%CI -3.33; -1.37) and \({\gamma }_{1}=0.86\) (95%CI 0.31; 1.57).

For scenario c), fully incorporating the biomarker information into the model, probabilities could be obtained using the Recalibration and Re-estimation mixture approaches. The Recalibration mixture approach achieved an \(\widehat{R}<1.01\) for all parameters, with an estimated \({\gamma }_{0}=-2.26\) (95%CI -3.33; -1.37) and \({\gamma }_{1}=0.86\) (95%CI 0.32; 1.58). The Re-estimation mixture approach achieved an \(\widehat{R}<1.01\) . The model that estimates \(T\) achieved an AUROC of 0.76 (95%CI 0.75; 0.77) in both mixture approaches, with model parameters described in sTable 1.

Discrimination and calibration of the models developed using the 6 different approaches

All approaches led to good model discrimination, with the Re-estimation approaches having the highest AUROC (Table 3 ).

In scenario a), for the approaches that used only the case-control dataset and adjusted for a known prevalence, the Original approach overestimated the observed probability of MODY in the UNITED population and had large uncertainty at higher percentages. In contrast, the Albert Offset approach slightly underestimated the observed probability of MODY in the UNITED population. Looking at the approaches that used the population-representative dataset (UNITED), both the Re-estimation and Recalibration approaches slightly underestimated the observed probability of MODY with slightly more uncertainty in the predictions from the Re-estimation approach (Fig. 2 ).

Calibration of scenario a) in UNITED. Scenario a): assume all not MODY tested are \({\text{MODY}}^{-}\) (based on strong clinical knowledge ( n =1,171)

In scenario b), for the approaches that used the case-control dataset alone, with adjustment for known prevalence, the Original approach overestimated the observed probability of MODY in the UNITED population. In contrast, the Albert Offset approach was able to calibrate well. Looking at the approaches that used an additional calibration dataset (UNITED), both the Re-estimation and Recalibration approaches calibrated well, but the Re-estimation approach demonstrated more uncertainty in the probability predictions (Fig. 3 ). In scenario c), the Re-estimation and Recalibration mixture approaches demonstrated similar performance to the equivalent models that did not use a mixture model approach, with similar levels of uncertainty in probability predictions (Fig. 3 ). In this case, the Albert Offset method worked well, but it relies on the assumption that the likelihood ratio is the same in the two populations. For illustrative purposes, we also provide an example setting where the likelihood ratio is different between the training and calibration datasets (violating the assumption). In this latter example, the  Albert Offset  approach fails to calibrate well. In contrast, the  Recalibration  approach can scale the odds ratios and calibrates well (sFig. 5), so this method would be preferred if a recalibration dataset is available.

Calibration of scenario b) and c) in UNITED. Scenario b): only analyse patients which tested C-peptide positive and autoantibody negative ( n =96) – Original, Albert Offset, Re-estimation and Recalibration approaches. In scenario c): analyse all patients ( n =1,171, validated in n =96) – Re-estimation mixture and Recalibration mixture approaches

Stability plots for the mixture models in scenario c)

The bootstrap stability test was made for both mixture approaches. Both mixture approaches were ran 50,000 iterations with the first 30,000 discarded for burn-in, with an average \(\widehat{R}=1.02\) (95% 1.0; 2.1) for the Re-estimation mixture approach and an average \(\widehat{R}=1.01\) (95% 1.0; 1.6) for the Recalibration mixture approach (the higher \(\widehat{R}\) values occurred in bootstrapped datasets with less than 3 positive MODY cases, but this only occurred in 8/1000 datasets and made no difference to the plots in Fig.  4 , and so we left these runs in). Both recalibration approaches showed some variability in the estimated probabilities, with the Re-estimation mixture approach demonstrating higher uncertainty across all estimated probability levels (Fig. 4 ). However, we can see that because the Recalibration mixture approach borrows weight from the case-control data, the estimates were more stable than the Re-estimation mixture method. We also noted that by using the hierarchical modelling approach, the Recalibration mixture model uncertainty included the uncertainty in the case-control predictions. Thus, these uncertainty estimates are larger than a model where this additional predictive uncertainty is ignored.

Stability plots for Re-estimation mixture and Recalibration mixture approaches. Estimations of MODY probability from bootstrapped models are plotted against estimated MODY probabilities from the developed model

Final recalibrated probabilities

The approach chosen for our final models was the Recalibration mixture approach, which incorporated the most information with the lowest uncertainty in probability predictions. The mixture model ensures that those with biomarkers consistent with T1D (the \({C}^{-}\cup {A}^{+}\) individuals) are predicted to have a very low probability of MODY, consistent with independent prior information. Fig. 5 shows the predicted probability of MODY in the remaining \({C}^{+}\cap {A}^{-}\) individuals. Considering only 0.6% of the cohort had MODY, the model produced a wide range of probabilities. Most non-MODY cases were predicted to have a low probability of MODY, with 97.2% (1,132/1,164) of individuals having an upper 95% CI probability of MODY under 10%. In contrast, 7 out of the 7 MODY cases had an upper 95% CI probability >10%. This would mean that if using a >10% threshold to initiate MODY testing for the population, 39 patients would be tested, giving a positive predictive value of 17.9% (Fig. 5 ), equivalent to the Original approach.

Estimated probabilities of MODY from the Recalibration mixture model in C-peptide positive, antibody negative patients, split by whether patients tested positive for MODY or not. All patients with negative C-peptide or positive antibodies ( n =1,075) had probabilities close to 0 and are not shown

This paper explored recalibration methods for adapting a statistical model from case-control data to the general population for rare disease prediction.

We have shown that the calibration of disease risk probabilities can be improved via various methods, and in particular, our results show the added benefits of utilising a secondary (recalibration) dataset that corresponds to a random sample from the general population despite there being few cases in the latter. In addition, the recalibration data contains additional information on biomarker tests, which are highly informative about disease risk, but only for certain subsets of test results; because of this, some biomarker information is only available for subsets of individuals. Our Recalibration mixture model allows the inclusion of (incomplete) biomarker information and informative prior information (derived from previous studies) about disease risk for specific subsets of test results to ensure clinically valid risk probabilities in those cases.

The Recalibration mixture model has several other advantages. It allows for predictions to be made in clinical practice even if the biomarker information is not available. We can also propagate parameter uncertainties from the case-control model to the recalibrated population predictions by utilising the Bayesian framework. This gives a more robust estimate of the underlying predictive uncertainty than classical models that ignore this uncertainty. Furthermore, the predictions for individuals without biomarker test information also propagate the uncertainties from the missing information. Finally, since this model is used to help inform which individuals should be screened for MODY using expensive genetic testing, for those individuals who have missing biomarker information, we show how the mixture model can also be used to inform clinicians about the added utility of performing a biomarker test before making a final decision of whether to send individuals for genetic testing. Although highlighted with a specific application, these ideas could be adapted to other rare diseases.

We compared several approaches for recalibrating probabilities when developing prediction models for rare diseases. We showed that the Original method tends to overestimate the probabilities in the general population, but that the Albert Offset [ 10 ], Re-estimation and Recalibration [ 7 ] approaches achieve good calibration of MODY probability predictions in both the model of the overall population and also in the model examining only the subset who were \({C}^{+}\cap {A}^{-}\) (those genetically tested for MODY). The Albert Offset [ 10 ] and Recalibration [ 7 ] approaches achieved the smallest uncertainty around the observed probability of MODY. The Recalibration mixture model showed stability in our study and was the only approach that appropriately constrained the probability of MODY in \({C}^{-}\cup {A}^{+}\) individuals to be consistent with the strong prior information available in this setting. When developing prediction models for rare diseases in practice, different approaches will be plausible in different scenarios based on the available data sources. Table 4 provides an overview of the advantages and disadvantages of all modelling approaches explored in the manuscript.

When only a training dataset (case-control dataset in our setting) is available, and the aim is to adjust probabilities based on population prevalence, then the Albert Offset approach was the preferred method as it estimated the probability of MODY well in both our scenarios, with reasonable uncertainty in the predictions. In contrast, the Original approach [ 14 ] relies on thresholding probabilities and overestimates the probability of MODY in both scenarios. The Albert Offset approach has also been compared to other recalibration methods in other studies. Chan et al . (2008) had similar findings and deemed the Albert Offset the best approach [ 11 ]. In contrast, Grill et al . (2016) described this approach as the worst-performing one in their study [ 12 ]. These differences may relate to the Albert Offset approach's strong assumption that the covariate distribution is the same in the training dataset as in the population for which the probabilities are adjusted [ 10 , 11 ], and as we showed in sFig. 5, the Albert Offset approach can perform poorly for datasets where the covariate distribution is different. When recalibrating models for different settings, the likelihood ratio assumption could become harder to justify depending on the specific setting, and particular caution would be required in populations where the clinical characteristics of the patients differ substantially from the case-control dataset used for original model development. Establishing whether the similarities between covariate distributions are sufficient for pre-assessing the performance of the Albert Offset method is of interest for future research.

When only a population-representative dataset is available, the Re-estimation approach would be necessary. The  Re-estimation  approach calibrates well in the general population dataset for both scenarios but demonstrates high uncertainty surrounding the model predictions. In contrast, Grill et al. (2016) describe the Re-estimation approach as having equivalent performance to the Albert Offset , with both performing worse than all other approaches [ 12 ]. The high uncertainty in our analysis can be attributed to the fact that there are only 7 positive MODY cases in the general population dataset (prevalence of 0.6%) and that the distribution of predicted probabilities is skewed towards zero. This highlights the problem with fitting models for rare diseases to population data [ 31 ], where the low prevalence means very large sample sizes would be required to reduce the uncertainty around the predictions, and it would be important to assess the adequacy of the sample size prior to model fitting [ 32 ]. When both training and population-representative datasets are available (as in our study), we showed that the  Recalibration  approach demonstrates good calibration in the population-representative dataset for both scenarios. This approach combines the information captured from the training data with information from the calibration dataset [ 7 , 33 ], producing relatively low uncertainty in the model predictions compared to other approaches explored in this paper.

We also explored a scenario where additional biomarker testing was available but performed only on a limited subset of patients. Screening using biomarkers is common in clinical practice and often used in rare diseases where universal testing is not cost-effective or could be invasive (e.g. screening for chromosomal defects in pregnancy [ 34 ]). We developed a Bayesian hierarchical mixture model to follow the referral process involved in MODY testing and, therefore, utilise the additional biomarker tests for further refinement in the prediction of MODY probabilities. As a Bayesian model, we can incorporate additional information from other studies into the prior distributions for certain parameters, something previously explored by Boonstra et al. [ 15 ] in a different setting where additional information is only present for a subset of individuals. This approach has a further advantage in that predictions can still be made for patients with missing additional biomarkers, which are modelled using patient characteristics. This is important for our setting in which instead of ignoring the biomarker results altogether, the model has used this information to improve predictions so that even when biomarker information is missing, the MODY probabilities are a weighted sum across the latent biomarker test results, where the weights are informed by a model relating potential biomarker test outcomes conditional on a set of clinical features. We also combined the mixture model with the  Re-estimation  and the Recalibration approaches for just \({C}^{+}\cap {A}^{-}\) individuals. Both approaches showed uncertainty levels in the probability predictions consistent with the previously observed uncertainty estimates. Furthermore, both approaches were tested for stability using bootstrapped versions of the population-representative dataset [ 35 ], demonstrating that the  Recalibration mixture  approach was more stable with the predicted probabilities of MODY than the  Re-estimation mixture  approach.

Other approaches for recalibration have been considered in previous work. Chan et al . (2008) [ 11 ] compared three methods to update pre-test probability with information on a new test: the Albert ( Albert Offset in our study) [ 10 ], Spiegelhalter and Knill-Jones (SKJ) [ 36 ] and Knottnerus [ 37 ] approaches. The SKJ represents an alternative to the Albert Offset approach, with similar performance in their paper. The Knottnerus approach was more suited to cases with sequential biomarker testing, which was not appropriate for our work since we did not have data on some combinations of tests, instead we grouped biomarkers into a composite measure \(T\) . The Knottnerus approach could be compared to the mixture model approaches (allowing for non-independence between both biomarkers), and examining these approaches when considering sequential testing of more than one biomarker could be considered in future work. Grill et al . (2016) compared several methods for incorporating new information into existing risk prediction models: logistic-new (equivalent to Re-estimation ), LR-joint, LR-offset ( Albert Offset ), and LR-shrink (equivalent to SKJ from reference [ 11 , 36 ]). In contrast to our study, their original models were built in population data, and the new datasets with additional features were either cohort or case-control data [ 12 ]. In the context of rare diseases, case-control data is likely to provide the best dataset for initial model development since this gives the most power for estimating model parameters, and the population-representative model can then borrow information from the case-control model. In cases where the additional data are only available in the case-control setting, and original models were built on population data, the joint model approach ( Recalibration ) could be adapted to this scenario.

The model we recommend for the available MODY data is the Recalibration mixture  approach. A major strength of this procedure is that it allows predictions for patients with missing biomarker testing, and this weighted prediction of MODY probability can be used to inform whether a patient should be referred for further testing [ 38 ]. This model allows for the incorporation of strong prior information regarding the probability of having MODY for \({C}^{-}\cup {A}^{+}\) individuals, propagates uncertainties regarding the missing data in the UNITED study, and borrows weight from the case-control model through the recalibration procedure [ 7 ], thus improving the stability of predictions [ 35 ]. This model provides sensible predictions for the probability of MODY for patients with/without additional testing for C-peptide and antibodies. Patients with missing MODY testing (i.e. \({C}^{-}\cup {A}^{+}\) ) could have been set as a negative result test for all approaches due to the strong clinical knowledge of these tests being consistent with a T1D diagnosis. However, this may not be the case for other settings, where patients with missing outcomes could be believed to have a higher probability of the outcome, and therefore, assuming that the outcome is negative may be less justifiable.

There are some limitations to the  Recalibration Mixture  approach. We currently use biomarker tests as binary (positive/negative) results; in practice, biomarkers may be on a continuous scale. As such, the model could be adapted to include the biomarker results as additional covariates, which could be numerically integrated out if predicting to an individual that was missing this information in practice [ 38 ]. We are also limited by the small sample sizes in rare diseases [ 39 ], and even with our final model utilising two datasets, model predictions still have some uncertainty. However, we still saw good separation between MODY and T1D, and even accounting for the uncertainty, probability thresholds could be defined that rule out clear non-MODY cases and can be used to determine positive test rates at different probabilities in practice. These thresholds would balance the amount of testing to be carried out against the potential for missing genuine MODY cases, depending on how conservative the choice of threshold is. The model has yet to be validated in a hold-out dataset, but the stability plots using bootstrapped datasets provide some insight into the stability of model predictions [ 35 ]. Although the 95% credible interval of bootstrapped probabilities is relatively wide at higher values, the 50% credible interval is narrow for all probabilities around the equal line.

This paper provides a comparison of several recalibration approaches. The development of our recalibration approach uses established methodologies, and we have shown how it could apply to identifying patients with a high probability of MODY to allow more targeted diagnostic testing, but these ideas could be applied to other diseases. In practice, other settings could benefit from a similar Bayesian hierarchical model structure where informative biomarkers or additional testing information are available but only in a subset of patients due to its invasive nature or high cost of testing. With this structure, the model can be used to consider whether additional testing should be carried out when the individual already has a low probability (not on the cusp of referral), something explored previously in treatment selection for Type 2 diabetes [ 38 ]. Furthermore, this modelling structure could be particularly useful in other rare diseases with low sample sizes since it borrows weight from multiple datasets through recalibration, improving predictions.

We have compared several approaches to developing prediction models for rare diseases. We found the Recalibration mixture model approach to be the best approach, combining case-control and population-representative data sources. This approach allows the incorporation of additional data on biomarkers and appropriate prior probabilities.

Availability of data and materials

Data are available through application to the Genetic Beta Cell Research Bank https://www.diabetesgenes.org/current-research/genetic-beta-cell-research-bank/ and the Peninsula Research Bank https://exetercrfnihr.org/about/exeter-10000-prb/ . Example R code for fitting the approaches used in this study is available on GitHub within the repository “Exeter-Diabetes/Pedro-MODY_recal_approaches”.

Johnson SR, Ellis JJ, Leo PJ, Anderson LK, Ganti U, Harris JE, Curran JA, McInerney-Leo AM, Paramalingam N, Song X, Conwell LS, Harris M, Jones TW, Brown MA, Davis EA, Duncan EL. Comprehensive genetic screening: the prevalence of maturity-onset diabetes of the young gene variants in a population-based childhood diabetes cohort. Pediatr Diabetes. 2018;20(1):57–64.

Article   PubMed   Google Scholar  

Mitani AA, Haneuse S. Small data challenges of studying rare diseases. Diabetes Endocrinol. 2020;3(3):e201965.

Google Scholar  

Schulz KF, Grimes DA. Case-control studies: research in reverse. Epidemiology. 2002;359(9304):431–4.

Kölker S, Gleich F, Mütze U, Opladen T. Rare disease registries are key to evidence-basec personalized medicine: highlighting the european experience. Front Endocrinol. 2022;13:832063.

Article   Google Scholar  

Greenland S. Model-based estimation of relative risks and other epidemiologic measures in studies of common outcomes and in case-control studies. Am J Epidemiol. 2004;160(4):301–5.

Rothman KJ, Greenland S. Modern Epidemiology. Philadelphia: Lippincott-Raven; 1998.

Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JDF. Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med. 2004;23(16):2567–86.

Steyerberg E. Clinical Prediction Models: a practical approach to development, validation and updating. Springer International P. 2009.

Schuetz P, Koller MT, Christ-Crain M, Steyerberg EW, Stolz D, Müller CA, Bucher HC, Bingisser RM, Tamm M, Müller B. Predicting mortality with pneumonia severity scores: importance of model recalibration to local settings. Epidemiol Infect. 2008;136(12):1628–37.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Albert A. On the use and computation of likelihood ratios in clinical chemistry. Clin Chem. 1982;28(5):1113–9.

Article   CAS   PubMed   Google Scholar  

Chan SF, Deeks JJ, Macaskill P, Irwig L. Three methods to construct predictive models using logistic regression and likelihood ratios to facilitate adjustment for pretest probability give similar results. J Clin Epidemiol. 2008;61(1):52–63.

Grill S, Ankerst DP, Gail MH, Chatterjee N, Pfeiffer RM. Comparison of approaches for incorporating new information into existing risk prediction models. Stat Med. 2016;36(7):1134–56.

Cheng W, Taylor JM, Gu T, Tomlins SA, Mukherjee B. “Informing a risk prediction model for binary outcomes with external coefficient information”, Journal of the Royal Statistical Society. Series C Appl Stat. 2019;68(1):121–39.

Shields BM, McDonald TJ, Campbell MJ, Hyde C, Hattersley AT. The development and validation of a clinical prediction model to determine the probability of MODY in patients with young-onset diabetes. Diabetologia. 2012;55:1265–72.

Boonstra PS, Barbaro RP. Incorporating historical models with adaptive Bayesian updates. Biostatistics. 2020;21(2):e47–64.

Colclough K, Patel K. How do I diagnose maturity onset diabetes of the young in my patients? Clin Endocrinol. 2022;97(4):436–47.

Gardner D, Tai E-S. Clinical features and treatment of maturity onset diabetes of the young (MODY). Diabetes Metab Syndr Obes. 2012;2012(5):101–8.

Naylor R, Johnson A, Gaudio D, Adam M, Feldman J, Mirzaa G, Pagon R, Wallace S, Bean L, Gripp K and Amemiya A. Maturity-onset diabetes of the young overview, University of Washington; Seattle, 1993-2023.

Pang L, Colclough KC, Shepherd MH, McLean J, Pearson ER, Ellard S, Hattersley AT, Shields BM. Improvements in awareness and testing have led to a threefold increase over 10 years in the identification of monogenic diabetes in the U.K. Diabetes Care. 2022;45(3):642–9.

Shepherd M, Shields B, Hudson M, Pearson E, Hyde C, Ellard S, Hattersley A, Patel K. A UK nationwide prospective study of treatment change in MODY: genetic subtype and clinical characteristics predict optimal glycaemic control after discontinuing insulin and metformin. Diabetologia. 2018;61(12):2520–7.

Article   PubMed   PubMed Central   Google Scholar  

Thanabalasingham G, Pal A, Selwood MP, Dudley C, Fisher K, Bingley PJ, Ellard S, Farmer AJ, McCarthy MI, Owen KR. Systematic assessment of etiology in adults with a clinical diagnosis of young-onset type 2 diabetes is a successful strategy for identifying maturity-onset diabetes of the young. Diabetes Care. 2012;35(6):1206–12.

Besser RE, Shepherd MH, McDonald TJ, Shields BM, Knight BA, Ellard S, Hattersley AT. Urinary C-peptide creatinine ration is a practical outpatient tool for identifying hepatocyte nuclear factor 1-α/hepatocyte nuclear factor 4-α maturity-onset diabetes of the young from long-duration type 1 diabetes. Diabetes Care. 2011;34(2):286–91.

Greeley SA, Polak M, Njølstad PR, Barbetti F, Williams R, Castano L, Raile K, Chi DV, Habeb A, Hattersley AT, Codner E. ISPAD clinical practice consensus guidelines 2022: the diagnosis and management of monogenic diabetes in children and adolescents. Pediatr Diabetes. 2022;23(8):1188–211.

National Health Service. National Genomic Test Directory: testing criteria for rare and inherited disease.,” [Online]. Available: https://www.england.nhs.uk/wp-content/uploads/2018/08/rare-and-inherited-disease-eligibility-criteria-v2.pdf . Accessed 6 Aug 2023.

Shields B, Shepherd M, Hudson M, McDonald T, Colclough K, Peters J, Knight B, Hyde C, Ellard S, Pearson E, Hattersley A and UNITED study team. Population-based assessment of a biomarker-based screening pathway to aid diagnosis of monogenic diabetes in young-onset patients. Diabetes Care. 2017; 40(8): 1017-1025, 2017.

de Valpine P, Turek D, Paciorek C, Anderson-Bergman C, Temple Lang D, Bodik R. Programming with models: writing statistical algorithms for general model structures with NIMBLE. J Comput Graph Stat. 2017;26(2):403–13.

de Valpine P, Paciorek C, Turek D, Michaud N, Anderson-Bergman C, Obermeyer F and et al., “NIMBLE: MCMC, particle filtering, and programmable hierarchical modeling,” 2022. [Online]. Available: https://cran.r-project.org/package=nimble .

R Core Team, “R: a language and environment for statistical computing,” 2021. [Online]. Available: https://www.R-project.org/ .

Gelman A, Carlin J, Stern H, Rubin D. Bayesian data analysis. New York: Chapman and Hall/CRC; 1995.

Book   Google Scholar  

Gelman A, Rubin D. Inference from iterative simulation using multiple sequences. Stat Sci. 1992;7(4):457–72.

Griggs R, Batshaw M, Dunkle M, Gopal-Srivastava R, Kaye E, Krischer J, Nguyen T, Paulus K, Merkel P. Clinical research for rare disease: opportunities, challenges, and solutions. Mol Genet Metab. 2009;96(1):20–6.

Mitani A, Haneuse S. Small data challenges of studying rare diseases. JAMA Network Open. 2020;3(3):e201965.

Moons K, Kengne A, Grobbee D, Royston P, Vergouwe Y, Altman D, Woodward M. Risk prediction models: II. external validation, model updating, and impact assessment. Heart. 2012;98:691–8.

Wright D, Kagan K, Molina F, Gazzoni A, Nicolaides K. A mixture model of nuchal translucency thickness in screening for chromosomal defects. Ultrasound Obstet Gynecol. 2008;31(4):376–83.

Riley RD and Collins GS. Stability of clinical prediction models developed using statistical or machine learning methods. Biometric J. 2023;65(8):2200302.

Spiegelhalter D, Knill-Jones R. Statistical and knowledge-based approaches to clinical decision-support systems, with an application in gastroenterology. J R Statl Soc Series A. 1984;147(1):35–77.

Knottnerus J. Application of logistic regression to the analysis of diagnostic data: exact modeling of a probability tree of multiple binary varibles. Med Decis Mak. 1992;12(2):93–108.

Article   CAS   Google Scholar  

Cardoso P, Dennis JM, Bowden J, Shields BM and McKinley TJ. Dirichlet process mixture models to impute missing predictor data in counterfactual prediction models: an application to predict optimal type 2 diabetes therapy. BMC Med Inform Decis Mak. 2024; 24(12).  https://doi.org/10.1186/s12911-023-02400-3 .

Riley RD, Snell KI, Burke DL, Harrel FE Jr, Moons KG, Collins GS. Minimum samples size for developing a multivariate prediction model: part II - binary and time-to-event outcomes. Stat Med. 2019;38:1276–96.

Download references

Acknowledgements

For the purpose of open access, the author has applied a CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.

PC and TJM (McKinley) are funded by Research England’s Expanding Excellence in England (E3) fund. KAP is a Wellcome Trust fellow (219606/Z/19/Z). This work was further supported by Diabetes UK (reference 21/0006328), the National Institute for Health and Care Research (NIHR) Exeter Biomedical Research Centre (BRC) and the National Institute for Health and Care Research Exeter Clinical Research Facility. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, the Wellcome Trust or the Department of Health and Social Care.

Author information

Beverley M. Shields and Trevelyan J. McKinley are joint senior.

Authors and Affiliations

University of Exeter Medical School. Address: Clinical and Biomedical Sciences, RILD Building, Royal Devon & Exeter Hospital, Barrack Road, Exeter, EX2 5DW, UK

Pedro Cardoso, Timothy J. McDonald, Kashyap A. Patel, Andrew T. Hattersley, Beverley M. Shields & Trevelyan J. McKinley

University of Dundee. Address: Division of Population Health & Genomics, Ninewells Hospital and Medical School, University of Dundee, Dundee, DD1 9SY, UK

Ewan R. Pearson

You can also search for this author in PubMed   Google Scholar

Contributions

PC, BMS and TJM (McKinley) conceived and designed the study. PC, BMS and TJM (McKinley) analysed the data and developed the code. KAP and TJM (McDonald) provided the cohort data partly used to estimate the prior distribution of a positive MODY test result, into those with C-peptide negative or autoantibody positive test results. ATH and ERP led the UNITED population study. All authors contributed to the writing of the article, provided support for the analysis and interpretation of results, critically revised the article, and approved the final article.

Corresponding author

Correspondence to Trevelyan J. McKinley .

Ethics declarations

Ethics approval and consent to participate.

For the UNITED study, ethics approval was granted by the NRES Committee South West–Central Bristol (REC no. 10/H0106/03). For the case-control data, this study was approved by the Genetic Beta Cell Research Bank, Exeter, UK (ethical approval was provided by the North Wales Research Ethics Committee, UK (IRAS project ID 231760)) and the Peninsula Research Bank (Devon & Torbay Research Ethics Committee, ref: 2002/7/118).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1. , rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Cardoso, P., McDonald, T.J., Patel, K.A. et al. Comparison of Bayesian approaches for developing prediction models in rare disease: application to the identification of patients with Maturity-Onset Diabetes of the Young. BMC Med Res Methodol 24 , 128 (2024). https://doi.org/10.1186/s12874-024-02239-w

Download citation

Received : 12 January 2024

Accepted : 06 May 2024

Published : 04 June 2024

DOI : https://doi.org/10.1186/s12874-024-02239-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Bayesian modelling
• Rare diseases
• Prior elicitation
• Recalibration

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

2024 Theses Doctoral

Statistically Efficient Methods for Computation-Aware Uncertainty Quantification and Rare-Event Optimization

He, Shengyi

The thesis covers two fundamental topics that are important across the disciplines of operations research, statistics and even more broadly, namely stochastic optimization and uncertainty quantification, with the common theme to address both statistical accuracy and computational constraints. Here, statistical accuracy encompasses the precision of estimated solutions in stochastic optimization, as well as the tightness or reliability of confidence intervals. Computational concerns arise from rare events or expensive models, necessitating efficient sampling methods or computation procedures. In the first half of this thesis, we study stochastic optimization that involves rare events, which arises in various contexts including risk-averse decision-making and training of machine learning models. Because of the presence of rare events, crude Monte Carlo methods can be prohibitively inefficient, as it takes a sample size reciprocal to the rare-event probability to obtain valid statistical information about the rare-event. To address this issue, we investigate the use of importance sampling (IS) to reduce the required sample size. IS is commonly used to handle rare events, and the idea is to sample from an alternative distribution that hits the rare event more frequently and adjusts the estimator with a likelihood ratio to retain unbiasedness. While IS has been long studied, most of its literature focuses on estimation problems and methodologies to obtain good IS in these contexts. Contrary to these studies, the first half of this thesis provides a systematic study on the efficient use of IS in stochastic optimization. In Chapter 2, we propose an adaptive procedure that converts an efficient IS for gradient estimation to an efficient IS procedure for stochastic optimization. Then, in Chapter 3, we provide an efficient IS for gradient estimation, which serves as the input for the procedure in Chapter 2. In the second half of this thesis, we study uncertainty quantification in the sense of constructing a confidence interval (CI) for target model quantities or prediction. We are interested in the setting of expensive black-box models, which means that we are confined to using a low number of model runs, and we also lack the ability to obtain auxiliary model information such as gradients. In this case, a classical method is batching, which divides data into a few batches and then constructs a CI based on the batched estimates. Another method is the recently proposed cheap bootstrap that is constructed on a few resamples in a similar manner as batching. These methods could save computation since they do not need an accurate variability estimator which requires sufficient model evaluations to obtain. Instead, they cancel out the variability when constructing pivotal statistics, and thus obtain asymptotically valid t-distribution-based CIs with only few batches or resamples. The second half of this thesis studies several theoretical aspects of these computation-aware CI construction methods. In Chapter 4, we study the statistical optimality on CI tightness among various computation-aware CIs. Then, in Chapter 5, we study the higher-order coverage errors of batching methods. Finally, Chapter 6 is a related investigation on the higher-order coverage and correction of distributionally robust optimization (DRO) as another CI construction tool, which assumes an amount of analytical information on the model but bears similarity to Chapter 5 in terms of analysis techniques.

• Operations research
• Stochastic processes--Mathematical models
• Mathematical optimization
• Bootstrap (Statistics)
• Sampling (Statistics)

More About This Work

• DOI Copy DOI to clipboard