---
title: "Sampling and uncertainty 4: Confidence intervals for the difference between two means"
output: 
  learnr::tutorial:
    theme: default
    css: http://research.sbcs.qmul.ac.uk/r.knell/learnr_data/test2.css
runtime: shiny_prerendered
author: Rob Knell
description: >
  How to calculate the confidence interval for the difference between two means and how to interpret its
---

```{r setup, include=FALSE}
library(learnr)
knitr::opts_chunk$set(echo = TRUE)
```



## Calculating the confidence interval of the difference between means

So far we've only looked at the confidence interval of the mean. You can calculate confidence intervals for lots of other measures as well: proportions, the slopes and intercepts of lines and so on. Here we'll look at one useful example, the confidence interval of the difference between two means. Continuing with our penguin example, we've seen that we can get a rough indication of whether the mean weights of the penguins at our two colonies are different by looking at the confidence intervals of the two means. These show us the likely locations of the population means, and we've concluded that since the CIs don't overlap we can be reasonably sure that the population means are different. Another approach to this is to think about the difference between the two means, rather than the two means themselves. Just as we can think about the sampling distribution of the mean and calculate a standard error for the mean, so too can we think about the sampling distribution of the difference between two means and calculate a standard error for that. The confidence interval of a mean is the region where the population mean $\mu$ would lie 95% of the time if you repeatedly sampled from the population: similarly, the confidence interval for the difference between two sample means, $\bar{x}_1 -\bar{x}_2$ is the region where the difference between the two population means, $\mu_1 - \mu_2$ would lie 95% of the time if you were to repeatedly sample both means and calculate their difference.

The formula for the 95% confidence interval of the difference between two means is a bit more complicated than that for a single mean. I'm going to give you the maths but it's not necessary to understand it in detail. Fortunately it follows the same principle: you add or subtract the *standard error of the difference* ($SE_{diff}$), multiplied by a *t* value, from the difference between the sample means. In order to calculate the $SE_{diff}$, however, you have to calculate something called the *pooled standard deviation* or $s_p$ which is calculated as:

$$s_p = \sqrt{ \frac{\left( n_1 -1 \right) s^2_1 + \left( n_2 -1 \right) s^2_2}{n_1 + n_2 -2}}$$

Once we know this then (assuming the two population variances are roughly equal)

$$SE_{diff} = s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}$$

Once you know this the calculation is the same. The *df* for the calculation of *t* are now $n_1 + n_2 -2$.

$$ \textrm{Upper} \: 95\% \: \textrm{CI} = \bar{x}_1 -\bar{x}_2 + t \times SE_{diff}$$


$$ \textrm{Lower} \: 95\% \: \textrm{CI} = \bar{x}_1 -\bar{x}_2 - t \times SE_{diff}$$
Here, $\bar{x}_1$ is the mean of sample 1 and $\bar{x}_2$ is the mean for sample 2. Similarly, $s^2 _1$ is the variance of sample 1 and $s^2 _2$ is that of sample 2, and $n_1$ and $n_2$ are the sample sizes for samples 1 and 2 respectively.

The value of t is the appropriate value on $n_1 + n_2 -2$ degrees of freedom.

See if you can complete the code here to calculate the upper and lower confidence intervals for the difference between the mean weights for our two penguin samples.

```{r CIdiff1, exercise = TRUE, exercise.lines = 45}

penguins1 <- c(6.0, 5.8, 4.5, 3.7, 
              6.0, 4.3, 5.0, 6.0, 
              5.8, 3.5, 5.5, 5.7, 
              6.2, 4.7, 5.7)

penguins2 <- c(3.5, 4.7, 3.7, 4.4, 
               3.3, 4.8, 3.7, 3.7, 
               3.1, 3.2, 3.4, 3.7, 
               3.9, 5.2, 6.3, 3.6, 
               4.1)

# Calculate means
mean1 <- mean(penguins1)
mean2 <- mean(penguins2)

# Calculate variances
var1 <- var(penguins1)
var2 <- var(penguins2)

# Calculate sample sizes
n1 <- length(penguins1)
n2 <- length(penguins2)

# Value of t
t1 <- qt(0.975, df = n1 + n2 -2)

# Pooled standard deviation

sp <- sqrt(((n1 - 1)*var1 + (n2 - 1)*var2)/
             (n1 + n2 -2))

# Standard error of the difference
SEdiff <- 


# Calculate the confidence interval

upperCI <-

lowerCI <-

```
  

```{r CIdiff1-hint-1}
# To calculate SEdiff you need to use the formula given above
```

```{r CIdiff1-hint-2}
# To calculate SEdiff you need to use the formula given above

SEdiff <- sp * sqrt((1/n1) + (1/n2))
```

```{r CIdiff1-hint-3}
# For the lower and the upper confidence intervals
# you need to add in the calculation as given in the formula above

# Make sure all your arguments are separated by commas and
# that your brackets all match. 
```

```{r CIdiff1-hint-4}
# For the lower and the upper confidence intervals
# you need to add in the calculation as given in 
# the formula above

# You also need to ask R to print out the values
# calculated, maybe use cat() for a nice output
```

```{r CIdiff1-hint-5}
# For the lower and the upper confidence intervals
# you need to add in the calculation as given in 
# the formula above

# For the upper CI

upperCI <- mean1 - mean2 + t1 * SEdiff

# You also need to ask R to print out the values
# calculated, maybe use cat() for a nice output

cat("Upper 95% CI =", upperCI)
```

```{r CIdiff1-hint-6}
# For the lower and the upper confidence intervals
# you need to add in the calculation as given in 
# the formula above

# For the upper CI

upperCI <- mean1 - mean2 + t1 * SEdiff

# For the lower CI

lowerCI <- mean1 - mean2 - t1 * SEdiff

# You also need to ask R to print out the values
# calculated, maybe use cat() for a nice output

cat("Upper 95% CI =", upperCI)
cat("Lower 95% CI =", lowerCI)
```

```{r CIdiff1-hint-7}
# This is the complete solution

penguins1 <- c(6.0, 5.8, 4.5, 3.7, 
              6.0, 4.3, 5.0, 6.0, 
              5.8, 3.5, 5.5, 5.7, 
              6.2, 4.7, 5.7)

penguins2 <- c(3.5, 4.7, 3.7, 4.4, 
               3.3, 4.8, 3.7, 3.7, 
               3.1, 3.2, 3.4, 3.7, 
               3.9, 5.2, 6.3, 3.6, 
               4.1)

# Calculate means
mean1 <- mean(penguins1)
mean2 <- mean(penguins2)

# Calculate variances
var1 <- var(penguins1)
var2 <- var(penguins2)

# Calculate sample sizes
n1 <- length(penguins1)
n2 <- length(penguins2)

# Value of t
t1 <- qt(0.975, df = n1 + n2 -2)

# Pooled standard deviation

sp <- sqrt(((n1 - 1)*var1 + (n2 - 1)*var2)/
             (n1 + n2 -2))

# Standard error of the difference
SEdiff <- sp * sqrt((1/n1) + (1/n2))


# Calculate the confidence intervals

upperCI <- mean1 - mean2 + t1 * SEdiff

lowerCI <- mean1 - mean2 - t1 * SEdiff

# You also need to ask R to print out the values
# calculated, maybe use cat() for a nice output

cat("Upper 95% CI =", upperCI)
cat("Lower 95% CI =", lowerCI)

```

You should get the answer that the confidence intervals for the difference between means are from 0.59 to 1.83. This means that if you were to repeat this exercise many times, 95% of the time the difference in mean weight between the two penguin colonies would be between 0.59 and 1.83 Kg. There are two important things to notice here: firstly the 95% confidence intervals for this difference do not overlap zero. This tells us that there is a less than 5% chance that the true value is zero or close to it, and this means that we have a *statistically significant* difference between means. The second, arguably even more important thing to think about is that we have quantified the *effect size* --- the difference in mean weights between the two colonies --- and we have quantified how certain we are in of the accuracy of that number. That's really important in all sorts of areas of science.

In the example above we calculated the confidence intervals of the difference between the means from first principles, but fortunately we don't have to do this every time we wish to know this. The `t.test()` function in R carries out a t-test for the statistical significance of the difference between a pair of means, among other things. If you've not met the t-test yet don't worry, what you need to know for our present purposes is that by default it will give the 95% confidence intervals for the difference between two means. Here are our chinstrap penguin means compared using `t.test()`. I'm using the argument `var.equal = TRUE` because otherwise R will assume that we are dealing with means from populations with differing variances and will adjust the output accordingly.

```{r echo = FALSE}
penguins1 <- c(6.0, 5.8, 4.5, 3.7, 
              6.0, 4.3, 5.0, 6.0, 
              5.8, 3.5, 5.5, 5.7, 
              6.2, 4.7, 5.7)

penguins2 <- c(3.5, 4.7, 3.7, 4.4, 
               3.3, 4.8, 3.7, 3.7, 
               3.1, 3.2, 3.4, 3.7, 
               3.9, 5.2, 6.3, 3.6, 
               4.1)
```

```{r}
t.test(penguins1, penguins2, var.equal = TRUE)
```
The "95 percent confidence interval:" in this output gives the 95% confidence intervals for the difference between the means, and you can see that the numbers given here are the same as the numbers we calculated above.

As a final exercise, here are some more penguin weights from a third colony. 

4.0, 2.9, 4.4, 4.8, 6.5, 3.0, 2.2, 7.4

What is the mean weight of these penguins?

```{r}
penguins3 <- c(4.0, 2.9, 4.4, 4.8, 
               6.5, 3.0, 2.2, 7.4)

mean(penguins3)
```

The mean of these weights is 4.4Kg, which is rather low compared to the mean weight of the penguins from the first colony, which is 5.2Kg. How confident are we that the difference we are seeing likely reflects a true difference between the population means, rather than just arising from sampling error?

Use the `t.test()` function to compare the mean weight of these animals with the mean weight of the animals from your third colony. Don't worry about the rest of the output, just look at the 95% CIs for the difference between means. Don't forget to set var.equal = TRUE.

```{r CIdiff2-setup, echo = FALSE}

penguins1 <- c(6.0, 5.8, 4.5, 3.7, 
              6.0, 4.3, 5.0, 6.0, 
              5.8, 3.5, 5.5, 5.7, 
              6.2, 4.7, 5.7)

penguins3 <- c(4.0, 2.9, 4.4, 4.8, 
               6.5, 3.0, 2.2, 7.4)
```

```{r CIdiff2, exercise = TRUE}

```



```{r CIdiff2-hint-1}

# Use t.test() with the names of both
# vectors as arguments and the var.equal
# argument set to TRUE
```

```{r CIdiff2-hint-2}

# Use t.test() with the names of both
#vectors as arguments and the var.equal
# argument set to TRUE

t.test(penguins1, penguins3, var.equal = TRUE)
```

What you can see is that in this case the confidence intervals for the difference overlap zero: the lower value is negative and the upper value is positive. It's quite plausible that the actual value of the difference between the population means is zero or close to zero, so we have little confidence that the apparent difference between these two colonies isn't simply a consequence of sampling error. That's also relfected in the p-value which we obtain for the t-test and which is greater than 0.05 but that is something for another day.

## Final quiz

Here's a quiz which should test how well you've understood all of the subjects we've covered in this group of tutorials. All questions have a single correct answer.


```{r echo = FALSE, fig.height = 4, fig.width = 5, fig.cap = "Means and 95% confidence intervals for the sample of 15 penguins from colony 1 and 8 penguins from colony 2"}

penguins <- c(6.0, 5.8, 4.5, 3.7, 
              6.0, 4.3, 5.0, 6.0, 
              5.8, 3.5, 5.5, 5.7, 
              6.2, 4.7, 5.7)

# Calculate the mean
mean1 <- mean(penguins)

# Calculate the SE
SE1 <- sd(penguins)/sqrt(14)

# Calculute the value of t
t1 <- qt(0.95, df = 14)

# Calculate the CIs
lowerCI <- mean1 - SE1 * t1
upperCI <- mean1 + SE1 * t1


penguins2 <- c(4.0, 2.9, 4.4, 4.8, 
               6.5, 3.0, 2.2, 7.4)

# Calculate the mean
mean2 <- mean(penguins2)

# Calculate the SE
SE2 <- sd(penguins2)/sqrt(17)

# Calculute the value of t
t2 <- qt(0.975, df = 16)

# Calculate the CIs
lowerCI2 <- mean2 - SE2 * t2
upperCI2 <- mean2 + SE2 * t2

# Dummy variable for x-axis
X1 <- c(1,2)

# Make vector of the means
Y1 <- c(mean1, mean2)

# Plot graph with means. No x-axis
plot(Y1 ~ X1,
     xlim = c(0.5, 2.5),
     ylim = c(3.5, 6),
     xaxt = "n",
     pch = 16,
     cex = 1.5,
     col = "aquamarine4",
     ylab = "Weight (Kg)",
     xlab = "")


# Add error bars using arrows
arrows(x0 = X1, y0 = c(lowerCI, lowerCI2),
       x1 = X1, y1 = c(upperCI, upperCI2),
       code = 3,
       angle = 90,
       length = 0.05,
       lwd = 2,
       col = "aquamarine4")

# Draw in x-axis
axis(side = 1,
     at = c(1,2),
     labels = c("Colony 1", "Colony 3"))
```

```{r quiz2, echo = FALSE}
quiz(
  question("The figure above shows the means and 95% confidence intervals for the weights of penguins from the first and third penguin colonies. Why might the confidence interval for colony 3 be wider than that for colony 1?",
    answer("The sample size for colony 3 is much smaller", correct = TRUE),
    answer("The mean is closer to zero"),
    answer("The frequency distribution of weights at colony 3 has strong negative skew"),
    answer("The standard error for colony 3 is negative"),
    answer("The difference between the means is not statistically significant")
  ),
    question("Which of these is the correct calculation for the standard error of a mean?",
    answer("Variance divided by the degrees of freedom"),
    answer("Standard deviation divided by the square root of the sample size", correct = TRUE),
    answer("Standard deviation divided by the degrees of freedom"),
    answer("Sum of squared deviations from the mean divided by the degrees of freedom"),
    answer("qt(0.975, df = 15) * SE")
  ),
    question("As the sample size increases, what happens to the standard error?",
    answer("It becomes a more accurate estimate of the population standard error"),
    answer("It tends towards the population mean"),
    answer("It becomes more accurate"),
    answer("It will tend to decrease", correct = TRUE),
    answer("It approaches a normal distribution")
  ),
    question("If the 95% confidence intervals for the difference between two means are -0.18 and 5.4, and the difference between the means is 2.8, what can you conclude?",
    answer("Repeating the study with a larger sample size would probably find a significant difference"),
    answer("The true value of the difference between the population means is zero"),
    answer("If we were to calculate the confidence intervals for each mean separately they would not overlap"),
    answer("There is a reasonable probability that the true difference between the population means is close to zero", correct = TRUE),
    answer("There is a statistically significant effect")
  ),
    question("Under what circumstances will the sampling distribution of the mean not tend towards a normal distribution as the number of samples increases?",
    answer("When sampling from a bimodal distribution"),
    answer("When the standard deviation of the population is high"),
    answer("When the population being sampled has strong positive skew"),
    answer("When the number of individuals sampled for each mean is small"),
    answer("None of the above: it will always tend towards a normal distribution", correct = TRUE)
  )
)
```

<br><br><hr>

## License

This content is licensed under a [https://www.gnu.org/licenses/gpl-3.0.en.html](GPL-3) license
