2.3: Measures of Spread
Introduction
So far we have learned about different ways to quantify the center of a distribution. A measure of center by itself is not enough, though, to describe a distribution. Consider the following two distributions of exam scores. Both distributions are centered at 70 (the median of both distributions is approximately 70), but the distributions are quite different. The first distribution has a much larger variability in scores compared to the second one.
In order to describe the distribution, we therefore need to supplement the graphical display not only with a measure of center, but also with a measure of the variability (or spread) of the distribution.
In this section, we will discuss the three most commonly used measures of spread:
- Range
- Inter-quartile range (IQR)
- Standard deviation
Like the different measures of center, these measures provide different ways to quantify the variability of the distribution.
Range
The range covered by the data is the most intuitive measure of variability. The range is exactly the distance between the smallest data point (min) and the largest one (Max).
- Range = Max – min
Note: When we first looked at the histogram, and tried to get a first feel for the spread of the data, we were actually approximating the range, rather than calculating the exact range.
Example
Best Actress Oscar Winners
We will continue with the Best Actress Oscar winners example (To see the full dataset, click here.)
34 34 27 37 42 41 36 32 41 33 31 74 33 49 38 61 21 41 26 80 42 29 33 36 45 49 39 34 26 25 33 35 35 28 30 29 61 32 33 45 29 62 22 44 |
In this example:
- min = 21 (Marlee Matlin for Children of a Lesser God, 1986)
- Max = 80 (Jessica Tandy for Driving Miss Daisy, 1989)
The range covered by all the data is 80 – 21 = 59 years.
Inter-Quartile Range (IQR)
While the range quantifies the variability by looking at the range covered by ALL the data, the IQR measures the variability of a distribution by giving us the range covered by the MIDDLE 50% of the data.
The following picture illustrates this idea: (Think about the horizontal line as the data ranging from the min to the Max).
Here is how the IQR is actually found:
- Arrange the data in increasing order, and find the median M. Recall that the median divides the data, so that 50% of the data points are below the median, and 50% of the data points are above the median.
- Find the median of the lower 50% of the data. This is called the first quartile of the distribution, and the point is denoted by Q1. Note from the picture that Q1 divides the lower 50% of the data into two halves, containing 25% of the data points in each half. Q1 is called the first quartile, since one quarter of the data points fall below it.
- Repeat this again for the top 50% of the data. Find the median of the top 50% of the data. This point is called the third quartile of the distribution, and is denoted by Q3. Note from the picture that Q3 divides the top 50% of the data into two halves, with 25% of the data points in each. Q3 is called the third quartile, since three quarters of the data points fall below it.
- The middle 50% of the data falls between Q1 and Q3, and therefore:
IQR = Q3 – Q1
Comments
-
The last picture shows that Q1, M, and Q3 divide the data into four quarters with 25% of the data points in each, where the median is essentially the second quartile. The use of IQR = Q3 – Q1 as a measure of spread is therefore particularly appropriate when the median M is used as a measure of center.
-
We can define a bit more precisely what is considered the bottom or top 50% of the data. The bottom (top) 50% of the data is all the observations whose position in the ordered list is to the left (right) of the location of the overall median M. The following picture will visually illustrate this for the simple cases of n = 7 and n = 8.
Note that when n is odd (as in n = 7 above), the median is not included in either the bottom or top half of the data; When n is even (as in n = 8 above), the data are naturally divided into two halves.
Example
Best Actress Oscar Winners
To find the IQR of the Best Actress Oscar winners distribution, it will be convenient to use the stemplot.
Q1 is the median of the bottom half of the data. Since there are 22 observations in that half, Q1 is the mean of the 11th and 12th ranked observations in that half:
[latex]Q1=\frac{(30+31)}{2}=30.5[/latex]
Similarly, Q3 is the median of the top half of the data, and since there are 22 observations in that half, Q3 is the mean of the 11th and 12th ranked observations in that half:
[latex]Q3=\frac{(42+42)}{2}=42[/latex]
IQR = (42 −30.5 ) = 11.5
Note that in this example, the range covered by all the ages is 59 years, while the range covered by the middle 50% of the ages is only 11.5 years. While the whole dataset is spread over a range of 59 years, the middle 50% of the data is packed into only 11.5 years. Looking again at the histogram will illustrate this:
Comment
Software packages use different formulas to calculate the quartiles Q1 and Q3. This should not worry you, as long as you understand the idea behind these concepts. For example, here are the quartile values provided by three different software packages for the age of best actress Oscar winners:
R:
Minitab:
Excel:
Note that Q1 and Q3 as reported by the various software packages differ from each other and are also slightly different from the ones we found here. There are different acceptable ways to find the median and the quartiles. These can give different results occasionally, especially for datasets where n (the number of observations) is fairly small. As long as you know what the numbers mean, and how to interpret them in context, it doesn’t really matter much what method you use to find them, since the differences are really negligible.
Using the IQR to Detect Outliers
So far we have quantified the idea of center, and we are in the middle of the discussion about measuring spread, but we haven’t really talked about a method or rule that will help us classify extreme observations as outliers. The IQR is used as the basis for a rule of thumb for identifying outliers.
The 1.5(IQR) Criterion for Outliers
An observation is considered a suspected outlier if it is:
- below Q1 – 1.5(IQR) or
- above Q3 + 1.5(IQR)
The following picture illustrates this rule:
Example
Best Actress Oscar Winners
We will continue with the Best Actress Oscar winners example (To see the data, click here).
34 34 27 37 42 41 36 32 41 33 31 74 33 49 38 61 21 41 26 80 42 29 33 36 45 49 39 34 26 25 33 35 35 28 30 29 61 32 33 45 29 62 22 44 |
Recall that when we first looked at the histogram of ages of Best Actress Oscar winners, there were 5 observations that looked like possible outliers:
We can now use the 1.5(IQR) criterion to check whether the 5 observations should indeed be classified as outliers:
- For this example we found that Q1=30.5andQ3=42.5⇒IQR=11.5
- Q1−1.5(IQR)=30.5−(1.5)(11.5)=13.25
- Q3+1.5(IQR)=42.5+(1.5)(11.5)=59.25
The 1.5(IQR) criterion tells us that any observation that is below 13.25 or above 59.25 is considered a suspected outlier.
We therefore conclude that the observations 61, 61, 62, 74 and 80 should be flagged as suspected outliers in the distribution of ages. Note that since the smallest observation is 21, there are no suspected low outliers in this distribution.
Did I get this?
Understanding Outliers
We just practiced one way to ‘flag’ possible outliers. Why is it important to identify possible outliers, and how should they be dealt with? The answers to these questions depend on the reasons for the outlying values. Here are several possibilities:
- Even though it is an extreme value, if an outlier can be understood to have been produced by essentially the same sort of physical or biological process as the rest of the data, and if such extreme values are expected to eventually occur again, then such an outlier indicates something important and interesting about the process you’re investigating, and it should be kept in the data.
-
If an outlier can be explained to have been produced under fundamentally different conditions from the rest of the data (or by a fundamentally different process), such an outlier can be removed from the data if your goal is to investigate only the process that produced the rest of the data.
-
An outlier might indicate a mistake in the data (like a typo, or a measuring error), in which case it should be corrected if possible or else removed from the data before calculating summary statistics or making inferences from the data (and the reason for the mistake should be investigated).
Here are examples of each of these types of outliers:
-
The following histogram displays the magnitude of 460 earthquakes in California, occurring in the year 2000, between August 28 and September 9:
Identifying the outlier:
On the very far right edge of the display (beyond 4.8), we see a low bar; this represents one earthquake (because the bar has height of 1) that was much more severe than the others in the data.
Understanding the outlier:
In this case, the outlier represents a much stronger earthquake, which is relatively rarer than the smaller quakes that happen more frequently in California.
How to handle the outlier:
For many purposes, the relatively severe quakes represented by the outlier might be the most important (because, for instance, that sort of quake has the potential to do more damage to people and infrastructure). The smaller-magnitude quakes might not do any damage, or even be felt at all. So, for many purposes it could be important to keep this outlier in the data.
- The following histogram displays the monthly percent return on the stock of Phillip Morris (a large tobacco company) from July 1990 to May 1997:
Identifying the outlier:
On the display, we see a low bar far to the left of the others; this represents one month’s return (because the bar has height of 1), where the value of Phillip Morris stock was unusually low.
Understanding the outlier:
The explanation for this particular outlier is that, in the early 1990s, there were highly-publicized federal hearings being conducted regarding the addictiveness of smoking, and there was growing public sentiment against the tobacco companies. The unusually low monthly value in the Phillip Morris dataset was due to public pressure against smoking, which negatively affected the company’s stock for that particular month.
How to handle the outlier:
In this case, the outlier was due to unusual conditions during one particular month that aren’t expected to be repeated, and that were fundamentally different from the conditions that produced the values in all the other months. So in this case, it would be reasonable to remove the outlier, if we wanted to characterize the ‘typical’ monthly return on Phillip Morris stock.
-
When archaeologists dig up objects such as pieces of ancient pottery, chemical analysis can be performed on the artifacts. The chemical content of pottery can vary depending on the type of clay as well as the particular manufacturing technique. The following histogram displays the results of one such actual chemical analysis, performed on 48 ancient Roman pottery artifacts from archaeological sites in Britain:
Identifying the outlier:
On the display, we see a low bar far to the right of the others; this represents one piece of pottery (because the bar has a height of 1), which has a suspiciously high manganous oxide value.
Understanding the outlier:
Based on comparison with other pieces of pottery found at the same site, and based on expert understanding of the typical content of this particular compound, it was concluded that the unusually high value was most likely a typo that was made when the data were published in the original 1980 paper (it was typed as “.394” but it was probably meant to be “.094”).
How to handle the outlier:
In this case, since the outlier was judged to be a mistake, it should be removed from the data before further analysis. In fact, removing the outlier is useful not only because it’s a mistake, but also because doing so reveals important structure that was otherwise hidden. This feature is evident on the next display:
When the outlier is removed, the display is re-scaled so that now we can see the set of 10 pottery pieces that had almost no manganous oxide. These 10 pieces might have been made with a different potting technique, so identifying them as different from the rest is historically useful. This feature was only evident after the outlier was removed.
Let’s Summarize
-
The range covered by the data is the most intuitive measure of spread and is exactly the distance between the smallest data point (min) and the largest one (Max).
-
Another measure of spread is the inter-quartile range (IQR), which is the range covered by the middle 50% of the data.
-
IQR = Q3 – Q1, the difference between the third and first quartiles. The first quartile (Q1) is the value such that one quarter (25%) of the data points fall below it, or the median of the bottom half of the data. The third quartile is the value such that three quarters (75%) of the data points fall below it, or the median of the top half of the data.
-
The IQR should be used as a measure of spread of a distribution only when the median is used as a measure of center.
-
The IQR can be used to detect outliers using the 1.5(IQR) criterion. Outliers are observations that fall below Q1 – 1.5(IQR) or above Q3 + 1.5(IQR).
Introduction
Before we move on to the third measure of spread (standard deviation), we’ll summarize what we’ve learned so far about measuring spread and use it to introduce another graphical display of the distribution of a quantitative variable, the boxplot.
The Five Number Summary
So far, in our discussion about measures of spread, the key players were:
-
the extremes (min and Max), which provide the range covered by all the data; and
-
the quartiles (Q1, M and Q3), which together provide the IQR, the range covered by the middle 50% of the data.
The combination of all five numbers (min, Q1, M, Q3, Max) is called the five number summary, and provides a quick numerical description of both the center and spread of a distribution.
Example
Best Actress Oscar Winners
We will continue with the Best Actress Oscar winners example (To see the data, click here.)
34 34 27 37 42 41 36 32 41 33 31 74 33 49 38 61 21 41 26 80 42 29 33 36 45 49 39 34 26 25 33 35 35 28 30 29 61 32 33 45 29 62 22 44 The five number summary of the age of Best Actress Oscar winners (1970-2013) is:
Min: 21
Q1: 30.5
M: 34.5
Q3: 42
Max: 80
Now that you understand what each of the five numbers means, you can appreciate how much information about the distribution is packed into the five-number summary. All this information can also be represented visually by using the boxplot.
The Boxplot
The boxplot graphically represents the distribution of a quantitative variable by visually displaying the five-number summary and any observation that was classified as a suspected outlier using the 1.5(IQR) criterion.
There are several ways to plot the whiskers on a boxplot. One convention is to plot whiskers down to the minimum and up to the maximum value. We use the 1.5(IQR criterion), also known as the Tukey method for plotting whiskers. First, calculate the IQR, the difference between the 75th and 25th percentiles (or Q3 – Q1). Multiply the IQR by 1.5. Add this value to the 75th percentile. If the value is greater than (or equal to) the maximum value in the dataset, draw the upper whisker to the maximum value. Otherwise, stop the whisker at the largest value that is less than 75th percentile + 1.5 * IQR. Plot any values that are greater than this as individual points that are outliers. Similarly, subtract 1.5 * IQR from the 25th percentile. If this value is smaller than the minimum value in the dataset, draw the lower whisker to the minimum value. Otherwise, stop the whisker at the lowest value that is greater than 25th percentile – 1.5 * IQR. Plot any values that are smaller than this as individual points that are outliers.
Using the Best Actress dataset, here is how we determine where to draw the whiskers:
-
Q3 = 42
-
Q1 = 30.5
-
IQR: 42 – 30.5 = 11.5
-
1.5 * IQR = 1.5 * 11.5 = 17.25
-
Q3 + 1.5 * IQR = 42 + 17.25 = 59.25
The largest observation that is less than or equal to 59.25 is 49 so we draw the upper whisker up to 49. All points above 49 are considered outliers (61, 61, 62, 74, 80).
Q1 – 1.5 * IQR = 30.5 – 17.25 = 13.25
The smallest observation that is greater than or equal to 13.5 is 21 so we draw the lower whisker down to 21, which is also the minimum. There are no outliers.
Here is how a boxplot is constructed: (this is for the “Best Actress” dataset—to see the dataset, click here.)
GeoGebra Group offers a simulation activity where you can practice calculating the median, Q1, Q3, IQR, and outliers and drawing a boxplot. Note that you can edit the data in the chart to see different results.