**Understanding Hypothesis Testing and Resampling Techniques**

In this article, we will delve into the world of hypothesis testing and resampling techniques, specifically focusing on permutation testing and its application in hypothesis testing. We will explore the concept of null hypotheses, test statistics, and p-values, as well as discuss the challenges and considerations involved in designing effective hypothesis tests.

**Simulating Deltas**

To begin, let's consider a scenario where we have simulated 10,000 deltas, each representing a sample size of 50 from two classes. We observe that 2,000 of these samples have a height difference greater than or equal to 10 centimeters. This implies that approximately 20% of the times, given our null hypothesis being true, we would expect to observe a height difference of this magnitude.

The concept of delta refers to the simulated data, and each delta represents a single sample size. By observing the distribution of these deltas, we can infer the probability of a certain outcome occurring under the assumption that the null hypothesis is true. In this case, our null hypothesis being true means that there is no class difference in terms of height.

**P-Values and Resampling**

Now, let's consider the p-value, which is a measure of the probability of observing a result as extreme or more extreme than the one we observed, assuming that the null hypothesis is true. In this scenario, our p-value would be 1%, indicating that the probability of observing a height difference greater than or equal to 10 centimeters by chance alone is extremely low.

However, if our p-value is not small, we fail to reject the null hypothesis, meaning that there is insufficient evidence to suggest that there is a class difference in terms of height. Conversely, if our p-value is small, we reject the null hypothesis, and conclude that there is a statistically significant class difference.

**Designing Hypothesis Tests**

The design of a hypothesis test involves several key components: the test statistic, the null hypothesis, and the simulation method. The test statistic is a measure of the data that helps us determine whether the observed effect is statistically significant. The null hypothesis is a statement about the population parameter that we want to test for.

In this case, our null hypothesis is that there is no class difference in terms of height. Designing an effective hypothesis test requires careful consideration of these components and the simulation method used to estimate the p-value.

**Permutation Testing**

Permutation testing is a resampling technique that involves randomly permuting the data and recomputing the test statistic for each permutation. This process generates a distribution of test statistics under the null hypothesis, from which we can infer the probability of observing a result as extreme or more extreme than the one we observed.

Permutation testing offers several advantages over traditional statistical methods, including robustness to non-normality and independence assumptions, and flexibility in designing custom tests. However, it also requires careful consideration of the test statistic and simulation method used.

**Challenges and Considerations**

Designing an effective hypothesis test involves several challenges and considerations. Firstly, one must carefully define the null hypothesis and test statistic, ensuring that they are well-defined and meaningful. Additionally, the simulation method used to estimate the p-value must be robust and reliable.

Furthermore, it is essential to consider the assumptions underlying the test, such as independence and normality of the data. Failure to address these assumptions can lead to incorrect conclusions about the significance of the results.

**Conclusion**

In conclusion, hypothesis testing and resampling techniques offer powerful tools for inferring statistical significance in real-world data. By understanding permutation testing and its application in hypothesis testing, researchers can design effective tests that accurately detect differences between groups or trends in the data. However, designing an effective hypothesis test requires careful consideration of several key components, including the null hypothesis, test statistic, and simulation method used to estimate the p-value.

"WEBVTTKind: captionsLanguage: enone of the concepts in our course is hypothesis testing and how to use resampling and permutation test to do facilitate hypothesis testing so there are lots of questions and there is a lot of confusion especially about hypothesis testing and how we employ resampling and permutation tests to compute or to perform hypothesis testing so what I will do here is we'll take the same example we'll take the same example that we discuss in the course and I'll explain the same example in more detail now stressing on the points stressing stressing more stressing on the point that most students forget because the flow of arguments that we often do in hypothesis testing is very easy to miss out the details so I will stress on the key points so that most of the questions that are asked on this topic are easily solved by just viewing or read by redoing the same example by stressing on the key points where people miss out or people do not pay enough attention to right so the first step first okay let's first see what is the what is the setting like so we have two classrooms of students classroom one and classroom though right we have taken we have taken 50 observationally have taken 50 observations of students in class 1 and we have taken 50 observations of students in class 2 right so this is this is already observation this is already an observation or a set of observations that we have already made remember we already made these observations we already made these observations and collected this data of 50 students from class 1 and 50 students from class 2 right this is the empirical or the actual data this is the actual data that is collected right now once you collect this data right so if you compute the mean of all the students in class 1 let's call it uni 1 let's take let's take the mean of all the students in class 2 let's call it mu 2 all right so these two are also you can compute new one as the average value or the mean value of all the students in the 50 all the 50 students that we have sampled from class 1 mu 2 is the sample mean of all the students in class 2 right now I will define a variable called X which is mu 2 minus mu 1 right remember mu 1 and mu 2 are computed from the actual observations right so in the video we call this as X or we call this as capital Delta all right so what the better I refer to it as capital Delta or X it's basically the difference between the observed remember I'm using the word observed the observed means or observed sample means of students 50 students in class 2 and 50 students in class 1 right now let's assume this value is 10 centimeters now the very key important thing here is this is an observed value this is an observation because we got this value of capital Delta or X from a series of observations right this observation has already been made now right this is very important we call this a step zero of hypothesis testing okay just just to keep track of the steps and the flow of argument so first we have already made this observation of course instead of 50 if I have if I can take larger samples yes I will get more rigorous data but that's okay so my sample size here here my sample size here my sample size is 50 because I have taken 50 observations of students in class 1 and 50 observations of students in class 2 there could be more students the population of class 1 may be 500 the population of class two maybe 300 but I am just taking a sample here and this what we get here is basically the difference of sample means of class 2 and class 1 and this is an observation this is a ground truth this is a truth because we already made this observation because this is computed from real actual data let's not forget that fact right number one number two now once I get so once I get this observation or once I make this observation the next question I'm going to ask is what is the probability the question this is very important what is the probability of observing of observing a value of X which is greater than equal to 10 centimeters if if there was if there was no difference if there was no difference in actual class Heights in class Heights now let's break this up into pieces it's very very important to break it up into pieces okay so not what we are saying here is remember we already made the observation here we already made the observation of 10 centimeters now the question we are asking is imagine if there was no difference in class Heights if there was no height difference in class Heights if there was if there is a big if here if there was no difference in class Heights what is the probability of still observing a difference of 10 centimeters between sample means then sample size equals 250 so I'll repeat it again very very important this is what we compute as our p-value right what is the probability what is the probability of observing a value of greater than 10 centimeters observing making this observation right what is the probability of making this observation remember we already made an observation which is exactly equal to 10 centimeters now I am asking what is the probability of making an observation and what is your observation your observation is when you take sample sizes of 50 each and when you take the sample means and when you subtract the mean of class 2 the sample 2 from sample 1 if you get what is the probability of getting that value so I'll repeat it what is the probability of getting a difference of greater than 10 centimeters for sample means of class 2 and class 1 when the sample size equals 250 right we are writing that whole thing as just this because of the way we define X what is X here X is the difference of sample means when sample size is equals to 50 right so what is a probability of making this observation if given or if we have a hypothesis that we are making and what is that hypothesis this hypothesis is that there is no difference in class heights right so what are we doing we are computing the probability of making this observation given or if this this is given right so probability of X greater than equal to 10 centimeters given our hypothesis this hypothesis is that there is no difference in class Heights and this hypothesis is called the null hypothesis this is what we compute and we take this and we call this as the p-value right it's very very important so I have written it in English very carefully p-value is for this for this specific example p-value is the probability of observing a sample mean a sample mean between class 2 and class 1 difference of sample means which is mu 2 minus mu 1 when the sample size is fixed to 50 right what is the probability that that difference or that variable or that statistic this is called the test statistic this is called the test statistic what is the probability that your test statistic is greater than 10 centimeters and remember this is coming from an observation so what is the probability of the observation given a null hypothesis or given that there is no difference in class Heights this is what is your p-value now let's say this is a very important piece of argument like if p value is small if p value is small so let's say p value is equal to point 0 1 or 1 percent right by small what I mean by this is it means less than 5 percent and you might wonder why are we taking less than 5 percent because this is customary we typically take less than 5 percent as a customary value so if p value is small what happens let us go through the flow what does it mean p-value is small if p-value is small let's write it mathematically it means that the probability of X greater than equal to 10 centimeters given h0 is small or it is what 1% now right this is what it means right because this is a definition of thievery this is what p-value means if p-value is equal to point zero 1 or 1% or small what does it mean it means this probability this exact probability spot now here here comes a fun fact here from this what can i what can i perceive here see if this is small then your X but while this is small we already observed that x equals to say 10 centimeters or we observe that X is greater than equal to 10 centimeters because if it is equal to it it is also same as greater than equal to we already made that observation don't forget that fact when very first observation we made was that X or capital Delta is equal to 10 centimeters so I have already made this observation which means this is true this part is true this part is an observation or this is certainly true because it comes from an observation if this is true and if this probability is small right so this part is true this part is true because it is coming from an observation but this probability this overall property spoke then what does that imply because there only two parts in this probability there is a that this part or this part right so this part is absolutely true because that's an observation we made we already made this observation so it can't be incorrect right given this observation now if this value is small then what does it entail what this implies that because we have already made this observation this implies that H 0 is less probable H 0 is less probable or in other words H 0 may not be true may not be true which means we reject our null hypothesis ok I'll go through the flow of arguments it's very very important to get through the flow of arguments okay first of all if p-value is small like 1% we typically take anything less than 5% it's customary in statistics that if anything is less than or equal to 5% we considered small now you might say why 5% it's customary 5% is considered small enough of course there are cases where you lose where we will use less than 1% or less than 2% or less than 5% or less than 3% depends on typically it's customary to take a value of less than 5% to be small now what does it mean when I say p-value is small it means this whole probability is small probability of X greater than equal to 10 centimeters given the null hypothesis is small now if this is small there are two components here there is a that this part or this part now this part is certainly true because we already made the observation what did we do in step zero we actually made the observation of 10 centimeters which means this part is certainly true because in X greater than equal to 10 centimeters that is also equal to and this is our observation which means X greater than equal to 10 centimeters is certainly true and that's the only observation we have now between these two if this is certainly true because we already made the argument we already made the observation what might go wrong then why is this probability small this probability small because this must be less probable or this must be incorrect or this must not be true or this may not be true so this implies because this overall probability is small this implies that our heck zero is less probable or a h0 may not be true if X 0 is not true then we reject h0 what does rejecting h0 mean rejecting h0 means that our hypothesis that there is no difference in class Heights is incorrect which means there is actually a difference in classmates right and by the way for this whole analysis for this whole thought process we only have these hundred points that's very important for this whole for this whole analysis to perform this whole analysis including computing you might wonder how do we compute p-value I'll come to that in a way that's done using permutation test and resampling will come to that in a while but in this whole process the only thing that we have are these hundred observations right 50 samples from class 1 and 50 samples from class 2 that's all we have we don't have anything else yet right now just by basing on these hundred observations we have successfully concluded that there is no difference in class sites very very interesting idea in statistics just by using this 100 100 sample points we've made a huge leap right and we concluded using the whole hypothesis testing paradigm that there is no difference in class sense now that comes the question so the flow of arguments is very important oftentimes what students forget is that this observation is a ground truth people forget students forget that that's why I have stressed it a lot here this part is the core of hypothesis testing and p-values right this is an observation that's already made right and the whole probability being small basically means that this must be improbable or less probable or maybe not true which means it we should reject it and take the alternative hypothesis which is nothing but the opposite of h0 now comes the interesting question how do we compute our p-value so what is a p-value in a nutshell our p-value is probability of X greater than equal to 10 centimeters given h0 this is what we have to compute right now remember to compute this whole thing we just have our class 1 and class 2 samples by 50 samples of class 1 and we have 50 samples of plastic that's all we have we don't have anything else using just these hundred values we have to compute this probability and we do that using the ideas from resampling and permutation test right so i'll take you through step by step again now the first step here is since we have plus 1 right we have class one and class two right we have 50 values in class one and we have 50 values in class two now given these 250 values what do I have to computer how to compute probability of X greater than equal to 10 centimeters given H 0 and what is X here X is nothing but mu-2 minus mu-1 when the sample sizes are 50 50 each but remember I can't again sample them from classes the only data I have is this 50 50 I can't again sample from the classes for whatever reason given just these hundred values how do i compute it that's what your permutation test and resampling helps you answer first thing they say is this you have 100 values let's combine all the values here and create a large set called s which has this hundred values right imagine if imagine if my null hypothesis was true what is my null hypothesis being true mean it means there is no difference there is no difference in class Heights right what you have to compute you have to compute probability of X greater than equal to 0 or in other words probability of mu 2 minus mu 1 greater than equal to 10 centimeters given head 0 which means I somehow need to now simulate remember I somehow have to simulate the null hypothesis I somehow have to simulate the null hypothesis just using these hundred values I don't have anything else all I have is just these hundred values and using these 100 values I need to simulate man'll hypothesis and what is the null hypothesis null hypothesis that there is no difference in class Heights so if I take this large set of 100 values where did I have taken all the values from the class 1 and class 2 just the heights which are given there 50 observations here and 50 observation I combine all of them and a jumbled up right now I get hundred observations now if I break them randomly now if I sample 50 randomly from here and 50 randomly from here there will be some heights in this from class 1 there will be some heights from this from class 2 similarly in this set so I'm randomly splitting this 100 into two sets right let's call this set s 1 and s 2 right let's call this a test right now when I randomly split it there will be some points here in this set from classroom there will be some points from class two similarly there will be some points here roughly 50% of points here from class 1 and 50% of points from class 2 roughly roughly because because of randomization now if I compute the average of all the heights in class 1 in this sample one average of all the heights in the sample - and if I do YouTube if I perform u2 minus mu-1 let's say you might get a value which is Delta one small Delta one now remember what is this Delta one this Delta one this Delta one is basically new to minus mu-1 it is the again repeated I will read it in English it is the difference of sample means remember it is a difference of sample means when my sample sizes are 50 right and when I assumed that there is no difference in class Heights how do you see to simulate this no difference in class heights we are taking this random sample that that that logical step is very important how do you now simulate hit zero you simulate hell zero I am saying take the 50 students here and take the 50 students here put them into a large bucket of hundred students now randomly pick 50 students here 50 students here when you randomly pick 50 students and 50 students from the pool of hundred students there is going to be no difference in class heads because you have randomly picked it this randomly picking 50 and 50 here is a way to simulate your dnal hypothesis right so your Delta 1 is basically or simulated is basically a simulated difference in class heights in class Heights right with sample size with sample size equal to 50 and with null hypothesis equals to true and why am I saying our hypothesis true because I randomly picked by randomly picking I am ensuring that there is no difference in Classifieds is but randomly picking up measuring that this randomization or resampling is the key especially for this given problem now just the way I have randomly splitted into two sets s1 and s2 and I computed Delta one I can redo this whole randomization multiple times if I redo it and I get that set s1 s2 again which means I get mu1 mu2 again and if i take this difference between them I can get Delta two again if I just repeat this randomization once more because this is trivial to do in a computer if you had given hundred points I can split it into two sets of 50 50 points randomly by using a simple coin toss right so if I if I compute lots of these Delta 1 Delta 2 Delta 3 so on so forth let's assume Delta 10 K or 10,000 so I'm performing this simulation 10,000 times and what is each of these values each of these Delta eyes each of these deltas are simulated differences in class Heights with a sample size of 50 and while simulating with hypothesis being true right so I've computed 10,000 simulated values these are all simulated simulated XS because what is X X is mu 2 minus mu 1 so these are simulated XS simulated differences with h0 being true which has 0 being true now if I sort them all these values may not be sorted right if I sort them in let's say you might get Delta 1 to Delta 2 - Delta 3 - so on Delta 10k - sorry these are see I just sort them because these are simulated values what do I want to compute now I want to compute probability of X greater than equal to 10 centimeters given H 0 or in other words I want to compute probability of okay because what is each of them each of them is an X so what is the probability of Y or Delta is that you have computed because what is each of them each of them is a simulated X right so all of them if I sort them such that if I sort them in increasing order and I've sorted them in increasing order let's say whom my 10 centimeter value is here it says you my 10 centimeter value is here which we call as a capital Delta in our videos it says you're at my 10 centimeter values here and let's say zoom there are 2 K values here and there are 8 K values here then what does it mean it means it means that when I simulate it means here look at this I've simulated head 0 I assumed that head 0 is true and what is each of my eggs each of my deltas is a simulated value of x and of all the deltas that I submitted of all the 10,000 deltas I've simulated on 10,000 X's that I simulated 2,000 of them are greater than equal to 10 centimeters right so this is equal to 2,000 or 2k of how many simulations of 10 K simulations so this is 20% if there are 2 K values here remember these are all in increasing order so this is less than equal to less than equal to less than equal to less than equal to less than equal to so and so so the 10th there are 2000 values of XS that are greater than 10 centimeters because what is X Y or each of your deltas or simulations of capital X that's what I wrote here right each of your Delta is that simulated excise with null hypothesis being true so which means what does this mean it means 20% of your deltas or 20% of the times whenever your null hypothesis is true 20% of the times right you observe a height difference of greater than equal to 10 centimeters with a sample size of 50 don't forget the sample size of 50 now with this value so now let's let's do it again now I have Delta 1 - let's say Delta 2 - Delta 3 - so on so forth Delta 10k - now if my 10 centimeters lies here and if I have only let's say a total of 10,000 points right right if I have only 100 points here right the rest of the 9900 points are here then my probability of X greater than equal to 10 centimeters given H 0 equals 200 by 10 K which is equal to 1 percent this is my p-value right what I am computing here is my p-value when my p-value is small when my p-value is small again let's go back to the previous argument that we saw if my B value is small then I reject my null hypothesis if my p-value is not small let's look at this right here my p-value is not small p-value is not small p-value is not small hence we do not reject h0 we accept h0 here right so there are two parts to this argument first part that you have to really get it the very first part is what is hypothesis testing which I've explained in the first part you have made an observation this observation is the ground truth it's absolutely true given the observation now if you're you're trying to build this probability that you want to compute right now if that probability or p-value is small then you reject your null hypothesis or else you accept it now how to compute that p-value is what you do using Li sampling and permutation testing right and for this we do all of this and remember in this whole exercise all you have is the 50 samples from class 1 and 50 samples from cluster that's all you have you don't have anything else more than this it's very very important to understand that now the other typical question that comes up is how do you design what is the null hypothesis we pick the null hypothesis which is easier to simulate which is easier and possible to simulate because remember in the permutation testing or to simulate h0 so some people ask why why why did we choose that there is no class difference as the as the as the null hypothesis what if I chose my null hypothesis as having a class difference then the challenge here is how do you do the simulation the the actual problem the actual challenge in designing hypothesis tests I Potts in designing a hypothesis testing is to design what is a test statistic which is like our x2 design what is our H 0 this is non-trivial parts these are the these are the problem so when you have to design a hypothesis testing you have to design what is the test statistic design what is a null hypothesis and you to design how to simulate the null hypothesis so you choose Darkman hypothesis which is easier to simulate if your null hypothesis is hard to simulate or impossible to simulate you can't go ahead with hypothesis testing using permutation test and resampling right so actually designing a hypothesis test is non-trivial you need to be you need to really get into the groove and understand how to design the right test statistic how to design the right null hypothesis and how to design the right simulation because if you mess up any of them your whole be then you can't compute your p-value if you can't compute your p-value your whole hypothesis testing goes for a toss right so lot of questions a lot of questions that are typically asked for resampling and permutation testing or for p-values or for hypothesis testing I hope I've covered all of them in this in this half an hour long video where have we revisited focusing mostly or stressing mostly on those areas which people overlook and get confused with hypothesis testing and resampling and permutation testone of the concepts in our course is hypothesis testing and how to use resampling and permutation test to do facilitate hypothesis testing so there are lots of questions and there is a lot of confusion especially about hypothesis testing and how we employ resampling and permutation tests to compute or to perform hypothesis testing so what I will do here is we'll take the same example we'll take the same example that we discuss in the course and I'll explain the same example in more detail now stressing on the points stressing stressing more stressing on the point that most students forget because the flow of arguments that we often do in hypothesis testing is very easy to miss out the details so I will stress on the key points so that most of the questions that are asked on this topic are easily solved by just viewing or read by redoing the same example by stressing on the key points where people miss out or people do not pay enough attention to right so the first step first okay let's first see what is the what is the setting like so we have two classrooms of students classroom one and classroom though right we have taken we have taken 50 observationally have taken 50 observations of students in class 1 and we have taken 50 observations of students in class 2 right so this is this is already observation this is already an observation or a set of observations that we have already made remember we already made these observations we already made these observations and collected this data of 50 students from class 1 and 50 students from class 2 right this is the empirical or the actual data this is the actual data that is collected right now once you collect this data right so if you compute the mean of all the students in class 1 let's call it uni 1 let's take let's take the mean of all the students in class 2 let's call it mu 2 all right so these two are also you can compute new one as the average value or the mean value of all the students in the 50 all the 50 students that we have sampled from class 1 mu 2 is the sample mean of all the students in class 2 right now I will define a variable called X which is mu 2 minus mu 1 right remember mu 1 and mu 2 are computed from the actual observations right so in the video we call this as X or we call this as capital Delta all right so what the better I refer to it as capital Delta or X it's basically the difference between the observed remember I'm using the word observed the observed means or observed sample means of students 50 students in class 2 and 50 students in class 1 right now let's assume this value is 10 centimeters now the very key important thing here is this is an observed value this is an observation because we got this value of capital Delta or X from a series of observations right this observation has already been made now right this is very important we call this a step zero of hypothesis testing okay just just to keep track of the steps and the flow of argument so first we have already made this observation of course instead of 50 if I have if I can take larger samples yes I will get more rigorous data but that's okay so my sample size here here my sample size here my sample size is 50 because I have taken 50 observations of students in class 1 and 50 observations of students in class 2 there could be more students the population of class 1 may be 500 the population of class two maybe 300 but I am just taking a sample here and this what we get here is basically the difference of sample means of class 2 and class 1 and this is an observation this is a ground truth this is a truth because we already made this observation because this is computed from real actual data let's not forget that fact right number one number two now once I get so once I get this observation or once I make this observation the next question I'm going to ask is what is the probability the question this is very important what is the probability of observing of observing a value of X which is greater than equal to 10 centimeters if if there was if there was no difference if there was no difference in actual class Heights in class Heights now let's break this up into pieces it's very very important to break it up into pieces okay so not what we are saying here is remember we already made the observation here we already made the observation of 10 centimeters now the question we are asking is imagine if there was no difference in class Heights if there was no height difference in class Heights if there was if there is a big if here if there was no difference in class Heights what is the probability of still observing a difference of 10 centimeters between sample means then sample size equals 250 so I'll repeat it again very very important this is what we compute as our p-value right what is the probability what is the probability of observing a value of greater than 10 centimeters observing making this observation right what is the probability of making this observation remember we already made an observation which is exactly equal to 10 centimeters now I am asking what is the probability of making an observation and what is your observation your observation is when you take sample sizes of 50 each and when you take the sample means and when you subtract the mean of class 2 the sample 2 from sample 1 if you get what is the probability of getting that value so I'll repeat it what is the probability of getting a difference of greater than 10 centimeters for sample means of class 2 and class 1 when the sample size equals 250 right we are writing that whole thing as just this because of the way we define X what is X here X is the difference of sample means when sample size is equals to 50 right so what is a probability of making this observation if given or if we have a hypothesis that we are making and what is that hypothesis this hypothesis is that there is no difference in class heights right so what are we doing we are computing the probability of making this observation given or if this this is given right so probability of X greater than equal to 10 centimeters given our hypothesis this hypothesis is that there is no difference in class Heights and this hypothesis is called the null hypothesis this is what we compute and we take this and we call this as the p-value right it's very very important so I have written it in English very carefully p-value is for this for this specific example p-value is the probability of observing a sample mean a sample mean between class 2 and class 1 difference of sample means which is mu 2 minus mu 1 when the sample size is fixed to 50 right what is the probability that that difference or that variable or that statistic this is called the test statistic this is called the test statistic what is the probability that your test statistic is greater than 10 centimeters and remember this is coming from an observation so what is the probability of the observation given a null hypothesis or given that there is no difference in class Heights this is what is your p-value now let's say this is a very important piece of argument like if p value is small if p value is small so let's say p value is equal to point 0 1 or 1 percent right by small what I mean by this is it means less than 5 percent and you might wonder why are we taking less than 5 percent because this is customary we typically take less than 5 percent as a customary value so if p value is small what happens let us go through the flow what does it mean p-value is small if p-value is small let's write it mathematically it means that the probability of X greater than equal to 10 centimeters given h0 is small or it is what 1% now right this is what it means right because this is a definition of thievery this is what p-value means if p-value is equal to point zero 1 or 1% or small what does it mean it means this probability this exact probability spot now here here comes a fun fact here from this what can i what can i perceive here see if this is small then your X but while this is small we already observed that x equals to say 10 centimeters or we observe that X is greater than equal to 10 centimeters because if it is equal to it it is also same as greater than equal to we already made that observation don't forget that fact when very first observation we made was that X or capital Delta is equal to 10 centimeters so I have already made this observation which means this is true this part is true this part is an observation or this is certainly true because it comes from an observation if this is true and if this probability is small right so this part is true this part is true because it is coming from an observation but this probability this overall property spoke then what does that imply because there only two parts in this probability there is a that this part or this part right so this part is absolutely true because that's an observation we made we already made this observation so it can't be incorrect right given this observation now if this value is small then what does it entail what this implies that because we have already made this observation this implies that H 0 is less probable H 0 is less probable or in other words H 0 may not be true may not be true which means we reject our null hypothesis ok I'll go through the flow of arguments it's very very important to get through the flow of arguments okay first of all if p-value is small like 1% we typically take anything less than 5% it's customary in statistics that if anything is less than or equal to 5% we considered small now you might say why 5% it's customary 5% is considered small enough of course there are cases where you lose where we will use less than 1% or less than 2% or less than 5% or less than 3% depends on typically it's customary to take a value of less than 5% to be small now what does it mean when I say p-value is small it means this whole probability is small probability of X greater than equal to 10 centimeters given the null hypothesis is small now if this is small there are two components here there is a that this part or this part now this part is certainly true because we already made the observation what did we do in step zero we actually made the observation of 10 centimeters which means this part is certainly true because in X greater than equal to 10 centimeters that is also equal to and this is our observation which means X greater than equal to 10 centimeters is certainly true and that's the only observation we have now between these two if this is certainly true because we already made the argument we already made the observation what might go wrong then why is this probability small this probability small because this must be less probable or this must be incorrect or this must not be true or this may not be true so this implies because this overall probability is small this implies that our heck zero is less probable or a h0 may not be true if X 0 is not true then we reject h0 what does rejecting h0 mean rejecting h0 means that our hypothesis that there is no difference in class Heights is incorrect which means there is actually a difference in classmates right and by the way for this whole analysis for this whole thought process we only have these hundred points that's very important for this whole for this whole analysis to perform this whole analysis including computing you might wonder how do we compute p-value I'll come to that in a way that's done using permutation test and resampling will come to that in a while but in this whole process the only thing that we have are these hundred observations right 50 samples from class 1 and 50 samples from class 2 that's all we have we don't have anything else yet right now just by basing on these hundred observations we have successfully concluded that there is no difference in class sites very very interesting idea in statistics just by using this 100 100 sample points we've made a huge leap right and we concluded using the whole hypothesis testing paradigm that there is no difference in class sense now that comes the question so the flow of arguments is very important oftentimes what students forget is that this observation is a ground truth people forget students forget that that's why I have stressed it a lot here this part is the core of hypothesis testing and p-values right this is an observation that's already made right and the whole probability being small basically means that this must be improbable or less probable or maybe not true which means it we should reject it and take the alternative hypothesis which is nothing but the opposite of h0 now comes the interesting question how do we compute our p-value so what is a p-value in a nutshell our p-value is probability of X greater than equal to 10 centimeters given h0 this is what we have to compute right now remember to compute this whole thing we just have our class 1 and class 2 samples by 50 samples of class 1 and we have 50 samples of plastic that's all we have we don't have anything else using just these hundred values we have to compute this probability and we do that using the ideas from resampling and permutation test right so i'll take you through step by step again now the first step here is since we have plus 1 right we have class one and class two right we have 50 values in class one and we have 50 values in class two now given these 250 values what do I have to computer how to compute probability of X greater than equal to 10 centimeters given H 0 and what is X here X is nothing but mu-2 minus mu-1 when the sample sizes are 50 50 each but remember I can't again sample them from classes the only data I have is this 50 50 I can't again sample from the classes for whatever reason given just these hundred values how do i compute it that's what your permutation test and resampling helps you answer first thing they say is this you have 100 values let's combine all the values here and create a large set called s which has this hundred values right imagine if imagine if my null hypothesis was true what is my null hypothesis being true mean it means there is no difference there is no difference in class Heights right what you have to compute you have to compute probability of X greater than equal to 0 or in other words probability of mu 2 minus mu 1 greater than equal to 10 centimeters given head 0 which means I somehow need to now simulate remember I somehow have to simulate the null hypothesis I somehow have to simulate the null hypothesis just using these hundred values I don't have anything else all I have is just these hundred values and using these 100 values I need to simulate man'll hypothesis and what is the null hypothesis null hypothesis that there is no difference in class Heights so if I take this large set of 100 values where did I have taken all the values from the class 1 and class 2 just the heights which are given there 50 observations here and 50 observation I combine all of them and a jumbled up right now I get hundred observations now if I break them randomly now if I sample 50 randomly from here and 50 randomly from here there will be some heights in this from class 1 there will be some heights from this from class 2 similarly in this set so I'm randomly splitting this 100 into two sets right let's call this set s 1 and s 2 right let's call this a test right now when I randomly split it there will be some points here in this set from classroom there will be some points from class two similarly there will be some points here roughly 50% of points here from class 1 and 50% of points from class 2 roughly roughly because because of randomization now if I compute the average of all the heights in class 1 in this sample one average of all the heights in the sample - and if I do YouTube if I perform u2 minus mu-1 let's say you might get a value which is Delta one small Delta one now remember what is this Delta one this Delta one this Delta one is basically new to minus mu-1 it is the again repeated I will read it in English it is the difference of sample means remember it is a difference of sample means when my sample sizes are 50 right and when I assumed that there is no difference in class Heights how do you see to simulate this no difference in class heights we are taking this random sample that that that logical step is very important how do you now simulate hit zero you simulate hell zero I am saying take the 50 students here and take the 50 students here put them into a large bucket of hundred students now randomly pick 50 students here 50 students here when you randomly pick 50 students and 50 students from the pool of hundred students there is going to be no difference in class heads because you have randomly picked it this randomly picking 50 and 50 here is a way to simulate your dnal hypothesis right so your Delta 1 is basically or simulated is basically a simulated difference in class heights in class Heights right with sample size with sample size equal to 50 and with null hypothesis equals to true and why am I saying our hypothesis true because I randomly picked by randomly picking I am ensuring that there is no difference in Classifieds is but randomly picking up measuring that this randomization or resampling is the key especially for this given problem now just the way I have randomly splitted into two sets s1 and s2 and I computed Delta one I can redo this whole randomization multiple times if I redo it and I get that set s1 s2 again which means I get mu1 mu2 again and if i take this difference between them I can get Delta two again if I just repeat this randomization once more because this is trivial to do in a computer if you had given hundred points I can split it into two sets of 50 50 points randomly by using a simple coin toss right so if I if I compute lots of these Delta 1 Delta 2 Delta 3 so on so forth let's assume Delta 10 K or 10,000 so I'm performing this simulation 10,000 times and what is each of these values each of these Delta eyes each of these deltas are simulated differences in class Heights with a sample size of 50 and while simulating with hypothesis being true right so I've computed 10,000 simulated values these are all simulated simulated XS because what is X X is mu 2 minus mu 1 so these are simulated XS simulated differences with h0 being true which has 0 being true now if I sort them all these values may not be sorted right if I sort them in let's say you might get Delta 1 to Delta 2 - Delta 3 - so on Delta 10k - sorry these are see I just sort them because these are simulated values what do I want to compute now I want to compute probability of X greater than equal to 10 centimeters given H 0 or in other words I want to compute probability of okay because what is each of them each of them is an X so what is the probability of Y or Delta is that you have computed because what is each of them each of them is a simulated X right so all of them if I sort them such that if I sort them in increasing order and I've sorted them in increasing order let's say whom my 10 centimeter value is here it says you my 10 centimeter value is here which we call as a capital Delta in our videos it says you're at my 10 centimeter values here and let's say zoom there are 2 K values here and there are 8 K values here then what does it mean it means it means that when I simulate it means here look at this I've simulated head 0 I assumed that head 0 is true and what is each of my eggs each of my deltas is a simulated value of x and of all the deltas that I submitted of all the 10,000 deltas I've simulated on 10,000 X's that I simulated 2,000 of them are greater than equal to 10 centimeters right so this is equal to 2,000 or 2k of how many simulations of 10 K simulations so this is 20% if there are 2 K values here remember these are all in increasing order so this is less than equal to less than equal to less than equal to less than equal to less than equal to so and so so the 10th there are 2000 values of XS that are greater than 10 centimeters because what is X Y or each of your deltas or simulations of capital X that's what I wrote here right each of your Delta is that simulated excise with null hypothesis being true so which means what does this mean it means 20% of your deltas or 20% of the times whenever your null hypothesis is true 20% of the times right you observe a height difference of greater than equal to 10 centimeters with a sample size of 50 don't forget the sample size of 50 now with this value so now let's let's do it again now I have Delta 1 - let's say Delta 2 - Delta 3 - so on so forth Delta 10k - now if my 10 centimeters lies here and if I have only let's say a total of 10,000 points right right if I have only 100 points here right the rest of the 9900 points are here then my probability of X greater than equal to 10 centimeters given H 0 equals 200 by 10 K which is equal to 1 percent this is my p-value right what I am computing here is my p-value when my p-value is small when my p-value is small again let's go back to the previous argument that we saw if my B value is small then I reject my null hypothesis if my p-value is not small let's look at this right here my p-value is not small p-value is not small p-value is not small hence we do not reject h0 we accept h0 here right so there are two parts to this argument first part that you have to really get it the very first part is what is hypothesis testing which I've explained in the first part you have made an observation this observation is the ground truth it's absolutely true given the observation now if you're you're trying to build this probability that you want to compute right now if that probability or p-value is small then you reject your null hypothesis or else you accept it now how to compute that p-value is what you do using Li sampling and permutation testing right and for this we do all of this and remember in this whole exercise all you have is the 50 samples from class 1 and 50 samples from cluster that's all you have you don't have anything else more than this it's very very important to understand that now the other typical question that comes up is how do you design what is the null hypothesis we pick the null hypothesis which is easier to simulate which is easier and possible to simulate because remember in the permutation testing or to simulate h0 so some people ask why why why did we choose that there is no class difference as the as the as the null hypothesis what if I chose my null hypothesis as having a class difference then the challenge here is how do you do the simulation the the actual problem the actual challenge in designing hypothesis tests I Potts in designing a hypothesis testing is to design what is a test statistic which is like our x2 design what is our H 0 this is non-trivial parts these are the these are the problem so when you have to design a hypothesis testing you have to design what is the test statistic design what is a null hypothesis and you to design how to simulate the null hypothesis so you choose Darkman hypothesis which is easier to simulate if your null hypothesis is hard to simulate or impossible to simulate you can't go ahead with hypothesis testing using permutation test and resampling right so actually designing a hypothesis test is non-trivial you need to be you need to really get into the groove and understand how to design the right test statistic how to design the right null hypothesis and how to design the right simulation because if you mess up any of them your whole be then you can't compute your p-value if you can't compute your p-value your whole hypothesis testing goes for a toss right so lot of questions a lot of questions that are typically asked for resampling and permutation testing or for p-values or for hypothesis testing I hope I've covered all of them in this in this half an hour long video where have we revisited focusing mostly or stressing mostly on those areas which people overlook and get confused with hypothesis testing and resampling and permutation test\n"