How to Estimate Your Needed Sample Size
A key aspect of any study is determining how many participants you will enroll. For this post, I will discuss the importance of estimating how many people you need in your study and how to make this calculation using STATA and EpiTools.
The first question I get is “why do we need to estimate our needed sample size?” This boils down to statistics. In layman’s terms, we need to know the number of people our study requires in order to minimize false positives and false negatives. In other words, we want to minimize Type I and Type II error. Type I error is essentially the probability of a “false positive,” or the probability that your results appear significant, when in fact they are not. Type II error is the probability of “false negatives,” or the probability that we are unable to detect a significant result, even though there is one. These are represented by alpha and beta in statistical analysis. Check out Khan Academy's explanation for more details:
The next, and perhaps most practical question, is “how do I calculate my needed sample size?” There are several ways to do this. Although I am proficient in multiple programs, I enjoy STATA’s succinct and fluid coding over some of the clunkier coding and drop-down menus from other statistical programs. Therefore, I will focus on how to estimate sample size in STATA.
Another website that does this for free is EpiTools. I have always had consistent results with STATA and EpiTools. However, if you want to use a “publishable” and respectable program, I would use a statistical software package.
Estimating sample size of two proportions
A common study aims to assess whether the proportion of folks with your outcome differs between two groups. For example, is the proportion of people passing a coding assessment for a potential job higher in those that went to college, versus those that did not? In this case, the outcome is “Passing the Test” and the exposure is “Going to College.” The general code in STATA 14 and above for estimating this sample size is:
power twoproportions P1 P2, alpha(A) beta(B)
P1= proportion people passing the test in the group that did not go to college (unexposed group)
P2= proportion of people passing the test in the group that did go to college (exposed)
A= Probability of type I error
B= Probability of type 2 error
Let’s break this down further. How do we decide what P1 and P2 will be if (as good researchers do) we are estimating our needed sample size before we start our study, and before we even know the results? P1 is usually an estimate of what we think based on prior studies, empirical observation, or educated guesses. It is always good to have data to back up our assumptions. For example, we saw a study that said that in the general population, only 25% of people pass the test. It would be reasonable to assume that P1=0.25. If we want to take it further, we might say to ourselves that the general population includes folks with college degrees. Even better-- we saw a study that says folks with only a high school education pass the test 18% of the time. Perhaps a more reasonable estimate would be P1=0.18, or that in our unexposed group the proportion of folks with the outcome is 0.18.
A trickier question is, how do we know the proportion of people passing the test in those that had a college degree? Or in more general terms, how do we know the proportion of people with the outcome in the exposed group? We can tackle this question a few ways. (1) We can base it on prior literature. Oftentimes, we are conducting studies that are unique but have parallel results from prior studies. (2) We can base it off of empirical observation or pilot studies. Perhaps we did a qualitative pilot study and saw that it was a certain value, or we have noticed a certain percentage of people coming to our office that seem to get an A. (3) We base it off what is important for our study. For example, we may be doing the study because we are creating a fund for college students, and it is important that students funded by our program are able to pass the assessment at least 50% of the time. In this case, P2=0.50.
So, with this information:
power twoproportions 0.18 0.50, alpha(A) beta(B)
Now, how do we decide what A and B are? As we discussed earlier, alpha is the probability of type 1 error or a false positive result. Often, alpha is set to 0.05 as standard practice, since most published research will rely on a p-value of 0.05 as a “cut-off” mark. As previously discussed, beta is the probability of type 2 error or a false negative result. The value assigned for beta usually varies between 0.20 and 0.10. The lower your beta, the more likely you can detect a difference if there actually is one. Oftentimes, however, folks will choose a beta of 0.20 simply because it gives them more flexibility in terms of achieving the needed sample size.
So, with this information:
power twoproportions 0.18 0.50, alpha(0.05) beta(0.20)
This code is saying “find me the sample size I need to detect a significant result if I my folks without a college degree pass the test 18% of the time and my folks with a college degree pass the test 50% of the time. I accept that my change of a false positive result is 5% and my chance of a false negative result is 20%.”
A word on other sample size calculations
There are of course many ways to measure the outcome. Below is a list of codes that can be used in STATA to replace “twoproportions.”
While it is true that sample size often depends on what data you have available and the willingness of potential participants to enroll in your study, best practices suggest that calculating your needed sample size occur prior to undertaking your research study. Surprisingly, estimating your sample size is an often-overlooked step in the research process—many clients do not take this into consideration, but are often appreciative and impressed when we are able to provide them with a number.