Welcome to your final presentation for Psych 515. We will review some major concepts and discuss where to go from here. We will be expanding on topics inherent to research methods. You’ve been exposed to all of them, but I will be defining some of them again and adding in more details and things to consider. The objectives of this presentation are provided to focus our goals and guide your note-taking. We will be covering null-hypothesis testing and what this means for research practices in our field. We will be reviewing some key terms and defining them all. Sample size, effect size, power, alpha level, type one and type two errors. And introducing alpha and beta. We will discuss the two important issues in our field and the five steps to estimate an appropriate sample size. Talk about why these topics matter and then end with some closing thoughts for psych by 15. Most of the statistics we’ve covered had been concerned with null-hypothesis testing, which culminates in a conclusion about whether the null hypothesis is rejected or failed to be rejected. Remember we reject the null hypothesis. P value, also known as the alpha value, falls below 0.05, as this is a probability of making a type one error. We’ve used this p or alpha value of 0.05 for both Psych 51015, because we want to focus on the range of statistical tools available to you. However, remember, this is a somewhat arbitrary decision where p equals 0.049999 would be a publishable finding a statistical significance. But p equals 0.05001 would not be a statistical significance. This is just where that arbitrary line in the sand has been drawn in our field. But obviously raises some concern for many researchers. Namely, this probability value goes down as the size of the effect goes up, and as the size of the sample size goes up. So in and of itself, reporting a P-value doesn’t provide a complete picture. Now that you know how to select, Run, and interpret an appropriate statistical tests for a variety of situations. With time to expose you to the darker side of research. Exposing how an estimated 50% of published research conclusions may not be replicable in our field. What’s the basis of the claim I just made will come when in 1962. Now classic paper on statistical power reviewed all 70 articles from one psychology journal in 1960. They contain sufficient data for him to analyze. Most of those papers reported statistically significant findings. Yet when he ran power analyses, he found that the majority of studies had less than a 50% chance of detecting an effect that truly exists in the population. Unfortunately, that trend has persisted in more recent meta-analyses, where another researcher found that of over 3,801 cognitive neuroscience and psychological papers recently published, the false report probability was still likely to exceed 50% That was done by six in the Ioannidis in 2017. How can this be? It appears many published research studies in our field suffer from low statistical power, primarily due to inadequate sample sizes. A sample size is a subset of a target population. It’s the group of people who participate in a study. Let’s go ahead and review some of these concepts. Effect size is the estimate of the effect to the variables of interest have on one another. The size of association or difference regardless of sample size. In Psych 51015, we’ve already covered how to calculate and interpret effect size. We’ve used our Cohen’s D and partial eta squared. Cohen’s d is the most popular in our field. Remember a Cohen’s D effect size? If it’s 0.2, it’s considered small. Up to 0.5 is considered medium, and 0.8 is considered a large effect. Let’s review the four possibilities and statistical decision-making. First, the researcher may reject the null hypothesis and conclude that there is an effect. This is typically what we’re hoping for. And if our decision is in fact correct, we’ve made a correct decision. However, if the researcher concludes there is an effect when in fact there is not, that’s a type one error. In other words, a type one error is if the null hypothesis is true, but we, as the researcher decide to reject it. Now another decision or research can make is to fail to reject the null hypothesis. In essence, say there’s no effect. If this is true, it is a correct decision. However, a type two error would be if we say that there was no effect when in reality there is an effect. Please make sure you have your handle on these thoughts before moving on to the next slide is ongoing to expand on this. The alpha level, also known as our p-value, has always been set at 0.05 and Psych 510515. This means we’re willing to make a type one error 5% of the time. If the alpha level had been set at 0.01, then the researcher is willing to make a type one error only 1% of the time. See the relationship between alpha level and type one error. Type two error is also known as beta. And this previously reviewed it’s been one concludes there is no effect when there actually is now power. It’s the probability of correctly rejecting the null hypothesis. In other words, it means if you’ve concluded that there was an effect, when in fact there really is. So if the researcher concludes that there is an effect, they can either make the correct decision which is reflected in power, or they can make a type two error. Now, the typical type two error rate is considered to be 0.2 or 20%. This is the percent of time people in our field are willing to make such an error. Power is one minus beta. So power is typically set at 0.8 or 80% in the, as the minimum in our field. This is important. I haven’t exposed you to this earlier, but we will be re-examining these ideas in a few more slides. So getting back to the meta-analyses that revealed nearly half of the studies published may not be replicable in our field. How can we use this information to better evaluate published findings? And how can we avoid designing studies that would result in this error ourselves? Statistical power of a study depends on three variables. The level at which the alpha significance a set which we voice use 0.05, the effect size, and the sample size. The relationships among these four factors underlies several important issues in our field, of which I want to emphasize as you depart from Psych 515. Let’s discuss two important issues and research design when we’re talking about null-hypothesis testing. The first one is if your null hypothesis test reveals a p-value below 0.05, you absolutely need to also report the effect size. Remember, statistical significance means that a differences real and not just due to sampling variability or chance. That is, we are saying the difference would persist if the study were repeated with new random samples. However, the effect size helps people determined that clinical relevance, which is like the magnitude of the difference, is large enough to be useful. So as it would warrant a change in the operating procedure. This is important because in effect, can be statistically significant, but not clinically relevant. And if an effect is not statistically significant as clinical relevance cannot be assessed. That’s why we do not report an effect size for something that was not significant. Let me say that in another way. Statistical significance is a necessary precondition for clinical relevance. But it says nothing about the actual magnitude of the effect. That’s why the inclusion of the effect size is so important. It’s a recommendation in your publication manual. If you want to see Section 2.07. And we also have repeatedly said this in Psych 51015 intact, you’ve been reporting the effect size on all statistically significant findings in your homework. So hopefully this review is merely a good reminder of why. The second important issue is that when you’re designing a study, you really should calculate an appropriate sample size. Now, why should we know how to calculate an appropriate sample size? First, you’re required to do so when you apply for nearly any grant. And many institutional review boards will require an estimate of an adequate sample size to detect the effects hypothesized in your study. So if you want to conduct research, you’re going to have to do this at some point. But most importantly, it can also save a great deal on resources and ensure the results you conclude from your study are meaningful. Sample size calculations are often called power calculations, which tells us how crucial the concept of power is to the spinal concepts we’re covering today. Why undersized studies can’t find real results as noted than the previously reviewed meta-analyses. An oversized studies find that even and substantial ones. So both undersized in oversized studies, waste time, energy, and money. The former by using resources without finding results, and the latter by using more resources than necessary. And both exposed and unnecessary number participants to experimental risk. That trait to good study design is to size a study so that you have just large enough of an effect to detect anything that’s a scientific importance. If you’re effect turns out to be bigger so much the better. First you have to gather some information about your study to help base your estimates. Once you’ve gathered this information, you can calculate by hand using formula found in many textbooks, or you can use one of many specialized software packages are calculators available online. But in essence, if you’re expecting a bigger effect size thing, you’re going to be allowed to use a smaller required sample. Whereas if there’s a smaller expected effect and Pate needing a larger sample size. Please note, there are a lot of options out there for calculating sample size. You can use confidence interval calculations or more sophisticated equations. There are free software programs you can download to the these calculations or you can purchase an add-on for SPSS. However, for most of these calculations, there are going to be five steps. Step one is to specify your hypothesis test. This is really if you have more than one hypothesis. It’s rather straightforward, but most studies have many hypotheses. When you’re running a sample size calculation, you should choose your one main hypothesis and explicitly state the null and alternative hypothesis. But have them be two-tailed regardless of whether there are one or two tilting your actual study. This just allows you to be most conservative in estimating your sample size. That way you can focus on one set of variables when estimating or Sample Size. Step two is to specify the significance level of the test. Now we’ve always used 0.05 in this class, but it doesn’t have to be for your thesis or dissertation or anything outside of this class. 0.01.001, or even if you want to go higher up into 0.1 had been successfully argued. Remember these are really somewhat arbitrary lines drawn in the sand for a while will be deemed insignificant. Step three, specify the smallest effect size that is of scientific interest. Now some people just use the medium effect size of 0.5 based on Cohen’s conventions for sample size determinations. And you can do that. But be aware it’s not always appropriate. If you want to be more precise, you can determine the most appropriate effect size by knowing your scale really well. Which is why some people consider this step to be the most difficult. The point here is not to specify the effect size that you expect to find or that others have found. But the smallest effect size of scientific interest in the particular domain for you are examining. For example, if your therapy lowered children’s anxiety scores by three points, is that a big enough change to meet to meaningfully improve the children’s lives. How big would that drop have to be for it to matter? Step four, estimate the values of other perimeters necessary to compute the power function, typically standard error. Most statistical tests have a format of effect divided by standard error. Remember, standard error is generally the standard deviation divided by N. And we’re solving for n, which is the point of figuring out your needed sample size. So you absolutely need a value of standard deviation to be able to solve for this. Now, there are two primary ways we can estimate standard error. The first one is to conduct a pilot study and use that data like the study you design here in this class site by fitting that similar nature to what would be expected for a pilot study. Or you could use historical data. Another study that uses same deepen a variable. If you have more than one study that you found in the literature even better, you can average their standard deviations for a more reliable estimate. But usually you have to use one of these two methods to be able to estimate your standard error. And the final step, number five, specify the intended power of the test. The power of a test is the probability of finding significance in that being the true conclusion. Remember, beta is the likelihood of making a type two error. And the maximum Type two error rate is typically considered to be 0.2, as I stated a few slides ago. Remember also I stated on that same side the power is one minus beta. So a power of 0.8 is the minimum. And typically used in standard or sample size estimate calculations. By gathering the information for these five steps, you would then have everything you need to calculate an appropriate sample size. However, at the end of the day, realize are several ways to calculate an appropriate sample size. But again, having the answers to these five steps should make the calculations a breeze. Beyond having a better chance at being awarded a grant or being able to publish your findings. You can also save time and provide more meaningful results to our field. It’s been stated that unfortunately, many published studies have very low power and our bad sources for basing your sample size. Remember how I previously stated two meta-analyses found Power published studies to be at approximately 50% of power is 50% for a steady, it basically means that a steady had a 50% chance of finding significance for a real effect given the sample size, effect size, and statistical test. On the flip side of that, at means or it may have been just as many other studies that never got published. They didn’t have adequate power, but they truly do have an effect. If you now attempt to build on the study and you use the same sample size, you might still only have a 50% chance of replicating that significant result. Sets knowledge will hopefully help you be more wary of taking research studies at face value. And hopefully we’ll encourage you to learn more about statistics so that you can better evaluate claims in our field. In fact, a huge benefit of knowing the relationships between power, effect size, alpha level, and sample size. Is it by knowing any three of these you can solve for the fourth, this is most critical at the research design stage, are priori, right before you conduct your research. As you can determine the most appropriate sample size based on your estimates of power, an effect size. If it reveals an unfeasible sample size, you can adjust your research design as necessary. For instance, by raising the sample size, narrowing your target population. Or you can add precision or controls to help reduce random error. Or you can even switch your design to something that will give you more power. However, relationships among these four can also provide post-hoc or after the fact insight. For instance, you can use a given sample size, alpha significance level, and effect size in a publication to determine the power of that study, to assess whether or not a published statistical tests in fact had a fair chance of rejecting an incorrect null hypothesis. Now, in your study for site 55 ting, you were told to use a sample size of at least 20. The study was for educational purposes only. But with small sample size, statistical tests might not have an adequate enough to detect a difference. That in reality might be there. The smaller the sample or the smaller the true difference exists, the greater the probability of failing to reject the null hypothesis and the error. For us. That’s OK though, because as researchers, we should be able to speculate and interpret any statistical findings, not just those consistent with our ideas. So this might have greatly limited the true interpretation of your results, but again, it was for educational practices and just developing skills of research. So recognize the limitation of your small sample size and how it limits the true generalizability of your lab project. But rest assured, you’ve gained hopefully many skills in this exercise. As previously stated, there are numerous calculators, software, some are free, and programs that can help you with these calculations. But there are many formulas based on the design of your study and its goals. And some sub-fields within psychology have their own preferred methods. Covering all of these possibilities is beyond the scope of the current course. But your thesis advisor may have you estimate your sample size or report the power of your findings. How to do so will depend on a number of factors related to your research design. Your advisor should be able to help guide you through the process if you have the basic understanding of these principles as outlined here. I’ve also included some articles relevant to developmental and industrial organizational psychology that may be a personal interest to you. The last article is about using the measures we’ve covered to do post-hoc calculations on sample size. A very intriguing topic in and of itself. These are all optional though. Finally, here are the references used to create this presentation since its content is not addressed in the assigned readings for the course. Now I end with some closing thoughts for Psych 515, the ultimate goal in research is to uncover truth. The goal drives researchers to improve methods in our field so that we can improve the lives of others. Not only should this be a professional proceed, but it speaks to God’s design of man. I believe this quote by Johann Kepler sums it up perfectly. It is it, right? Yes, a duty to searching cautious manner for the norms of everything God has created. For he himself has let man take part in knowledge of these things. For these secrets are not of the kind whose research should be forbidden for whether they are set before our eyes like a mirror. So that by examining them, we observe to some extent the goodness and wisdom of the Creator. As we part, we reviewed some major concepts of research design, including limitations inherent to our field related to using null-hypothesis testing, Power, effect size, sample size, and alpha values, and things to consider when designing research and when reading articles in our field. I pray as Illi, the sequence of classes that God used it to equip you with knowledge to change the world for the better. And this concludes your final presentation for Psych 515.