Introduction to statistical tests

Goal

The goal of this course is to give students the basics of statistical testing. This is a PhD-level course for all students that have to perform hypothesis testing or solution comparison in their own research. The main idea is to give better scientific grounds to the experimental analysis they have to carry out for their study. The course is implemented using exercises in R.

At the end of the course, students would be able to:

Interpret a statistical test.
Be able to tell with a given confidence if solution A is better than solution B.
Determine how many experiments are required to achieve a given confidence interval.
Know how to do a Student t-test (and its variants).

Organization

This is a crash course in 2 days organized as follows:

First part
- Basics on probability.
- Basics on random variables (mean, variance, probability density, distribution function).
- Compute functions on probability and random variables in R (random generation, quantile, etc.).
- Law of large numbers, central-limit theorem.
Second part
- Introduction to R.
- Exercises.
Third part
- Confidence interval of a mean:
  - Normal law case.
  - General case (Student t-test).
- Comparing paired or unpaired observations.
- Confidence interval for one or two proportions.
- Confidence interval for linear regression.
- Hypothesis testing and p-value.
- Determining the number of experiments.
Fourth part
- Exercises on applications taken from the real world.

Material

The course material is:

Course slides for first and third part.
Slides for introduction to R (from Louis-Claude Canon).
R exercises and solutions.

Credits

Several exercises, examples, and ideas of this course have been taken from:

The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, April 1991, by Raj Jain.
MOOC Statistical Inference on Coursera.
Statistical inference for data science (a companion to the Coursera Statistical Inference Course) by Brian Caffo.