Which branch of statistics is about population based on samples?

Inferential Statistics

Inferential statistics are often used to compare the differences between the treatment groups. Inferential statistics use measurements from the sample of subjects in the experiment to compare the treatment groups and make generalizations about the larger population of subjects.

There are many types of inferential statistics and each is appropriate for a specific research design and sample characteristics. Researchers should consult the numerous texts on experimental design and statistics to find the right statistical test for their experiment. However, most inferential statistics are based on the principle that a test-statistic value is calculated on the basis of a particular formula. That value along with the degrees of freedom, a measure related to the sample size, and the rejection criteria are used to determine whether differences exist between the treatment groups. The larger the sample size, the more likely a statistic is to indicate that differences exist between the treatment groups. Thus, the larger the sample of subjects, the more powerful the statistic is said to be.

Virtually all inferential statistics have an important underlying assumption. Each replication in a condition is assumed to be independent. That is each value in a condition is thought to be unrelated to any other value in the sample. This assumption of independence can create a number of challenges for animal behavior researchers.

Inferential Statistics

Inferential statistics helps to suggest explanations for a situation or phenomenon. It allows you to draw conclusions based on extrapolations, and is in that way fundamentally different from descriptive statistics that merely summarize the data that has actually been measured. Let us go back to our party example. Say comparative statistics suggest that parties hosted by your friend Sophia are very successful (e.g., the average number of attendees and the median duration of her parties are greater than those of other parties). Your next questions may be: Why are her parties so successful? Is it the food she serves, the size of her social network, the prestige of her job, the number of men or women she knows, her physical attractiveness, the alcohol she provides, or the location and size of her residence? Inferential statistics may help you answer these questions. Finding that less well-attended parties had on average fewer drinks served would suggest that your friend Sophia's drinks might be the important factor. The differences in attendance and drinks served between her parties and other parties would have to be large enough to draw any conclusions.

Note that the inferential statistics usually suggest but cannot absolutely prove an explanation or cause-and-effect relationship. Inferential comes from the word infer. To infer is to conclude or judge from premises or evidence (American Heritage Dictionary) and not to prove. Often inferential statistics help to draw conclusions about an entire population by looking at only a sample of the population. Inferential statistics frequently involves estimation (i.e., guessing the characteristics of a population from a sample of the population) and hypothesis testing (i.e., finding evidence for or against an explanation or theory).

Statistics describe and analyze variables. We discuss measures and variables in greater detail in Chapter 4. A variable is a measured characteristic or attribute that may assume different values. A variable may be quantitative (e.g., height) or categorical (e.g., eye color). Variables may be independent (the value it assumes is not affected by any other variables) or dependent (the value it assumes is pre-determined by other variables). Variables are not inherently independent or dependent. An independent variable in one statistical model may be dependent on another. For example, assume that we have a statistical model to identify the cause of heart disease. Independent variables would be risk factors for heart disease: cigarettes smoked per day, drinks per day, and cholesterol level. The presence of heart disease would be a dependent value. The risk factor variables affect the presence of heart disease.

Statistical methods can analyze one variable at a time (i.e., univariate analysis) or more than one variable together at the same time (i.e., multivariate analysis). Bivariate analysis is analyzing two variables together. An example of a univariate analysis would be simply looking at the death rate (mortality) in different countries. An example of a bivariate analysis would be analyzing the relationship between alcoholism and mortality.

SAMPLING DISTRIBUTIONS

One of the advantages of working with samples is that the investigator does not have to observe each member of the population to get the answer to the question being asked. A sample, when taken at random, represents the population. The sample can be studied and conclusions drawn about the population from which it was taken.

Let us focus on the group that makes up the sample. We are not as interested in an individual's response as we are in the group's response. The individual values, of course, are accounted for in the group, but the way to compare outcomes is by looking at an overall response. The data from the groups are used to estimate a parameter. As you recall, these are values that represent an average of a collection of values, such as average age or standard (which is really “average”) deviation from the mean age.

To see how this is done, let us first look at a hypothetical situation. Consider what would happen if we were to study a population variable with a normal distribution. If we were to take multiple samples from this population, each sample theoretically would have a slightly different mean and standard deviation. When all sample means (

The problem, of course, is that we don't know with certainty how close we are by looking at just one sample. The mean of any given sample (

Figure 11-1 illustrates what could happen when we take a sample from the population. This is a very clever idea that was used to illustrate the concept of sampling distributions in the Primer of Biostatistics by Stanton Glantz.2 In this case, the population is Martians (as in those who are from Mars). Figure 11-1A shows the frequency distribution for the heights in every member of the population, assuming this could be measured. But since we really cannot actually measure this, notice the various samples that were taken in graphs B, C, and D. The dots are the sample means; they are close but not exact. The bars on either side of the mean dots represent the samples' standard deviations.

Now cover up graph A. The laws of random sampling tell us that each of these samples was equally likely to be picked. They are pretty good estimators for the population parameter of mean height. It is highly unlikely to pick a sample comprised of only members at one end of the curve. In this case, the estimate would be way off the mark.

Inferential statistics does not focus on “What is the true parameter?” Instead, we ask “How likely is it that we are within a certain distance from the true parameter?” What we really need to know is the degree of variability among the samples that could happen by chance, and the possibility of obtaining an aberrant or unusual sample. The method we use depends on the sampling distribution of the test statistic. Every statistical test relies on this. It is the basis of the entire theory of inference.

The sampling distribution of a statistic is the distribution of all values of that statistic for every sample of a particular size from the same population.

The statistic in this case has a special meaning—it is the result of the data analysis that is used to estimate the population parameter. The sampling distribution is an abstract concept. The distribution of the sample statistic is the result of what would happen if every sample of a particular size were studied. Even though we study a single, random sample, it is only one of an incredibly large number of possible samples—each with its own statistic.

The rest of the chapter discusses how sampling distributions for different types of test statistics are generated. This is not an actual step in this process of inference testing. We do not create a distribution because we have only one sample to work with. The statisticians look at the sample size and the type and variability of the data to see which distribution to use.

In the above example, we talked about using a sample mean, designated as

For any sample of a given size, we can calculate the mean,

Figure 11-2 is a graph of the means of 25 samples of Martian heights. When these means are plotted, a normal distribution emerges and forms a predictable pattern. Note that the mean of all of these samples, designated as

Theoretically, if we took the means for a given variable from every possible sample of the same size, we could plot these in a frequency distribution. Computer simulation actually can demonstrate this process. What emerges is a pattern that falls into the normal distribution, even if the original distribution of the values was not normal. Surprised? So was I at first. But it makes sense that sample means would tend to approach the true parameter, with equal chances of under- or overestimating the true mean.

Figure 11-3 is a computer-generated sampling distribution of means. The original population (graph A) had a mean of 50 and a standard deviation of 29. It was actually a rectangular distribution. All values between 0 and 100 were equally frequent.

In graphs B and C, each dot represents a sample mean. The means of the samples have a wider distribution for a smaller sample size of 5 (graph B), with an approximately normal distribution. When the sample size is increased to 30 (graph C), the distribution of the means is narrower.

It turns out that samples act in a predictable fashion. This predictability happens not only with sample means, but with other parameters as well. Recall that some parameters can be quite abstract, such as “risk of an accident.” For all possible samples of the same size from a population, the risk calculated will form a predictable collection of values. The actual risk in the population is fixed and the sample provides you with an estimate of that risk.

Think of sampling distributions as predictable collections of numbers that form a pattern. The rest of the process then falls into place: if it is too unlikely that our sample statistic came from the predictable collection that we would expect if the treatment had no effect, we reject the null hypothesis and declare one treatment to be significantly better. The logic behind all the statistical tests is based on this method.

3.5.1 Errors in Statistical Hypothesis Testing—Type I and Type II Errors

Inferential statistics provide a quantitative method to decide if the null hypothesis (H0) should be rejected. Since H0 must be either true or false, there are only two possible correct outcomes in an inferential test; correct rejection of H0 when it is false, and retaining H0 when it is true. Therefore, there are two possible errors that can be made which have been termed Type I and Type II errors. A type I error occurs when H0 is incorrectly rejected. This is commonly termed a false positive. A type II error occurs when H0 is retained when it is in fact false. This error is commonly termed a false negative. From the standpoint of reproducibility, knowing the probability of making a type I or type II error is essential. This probability depends on experimental design and execution, and on the sample size, once again highlighting the importance of power analysis

CLINICAL STUDIES

Most clinical studies require more of statistical analysis than just a summary of the data. Researchers want to quantify the differences between study groups and draw conclusions from those differences, if present. A full discussion of study design is beyond the scope of this chapter. However, there are several steps involved: (i) formulating a hypothesis, (ii) deciding on the study population, (iii) designing a data collection method that minimizes errors and avoids bias, (iv) estimating sample size for adequate statistical power, (v) and analyzing data and making inferences based on this statistical analysis.

QUANTIFYING CHANCE OCCURRENCES

Inferential statistics is based on the probability of a certain outcome happening by chance. In probability theory, the word outcome refers to the result observed. It does not necessarily reflect quality-adjusted life-years (QUALY) like the outcome variable we see in clinical trials. It is simply the result of an event. The range of probabilities varies between 0 (no probability of the event happening) and 1 (the outcome will always happen.) It is rare to find circumstances in nature where the probability of occurrence is equal to 0 or 1. If that were the case, there would be no need to apply probability theory. All events that are studied in medicine have a probability of occurrence between 0 and 1. This is expressed as a decimal, such as 0.35. The simplest, most informative interpretation of probability converts these values to percentages to express the chance of something happening. An outcome with a probability of 0.35 is said to have a 35% chance of occurrence. On average, it will happen 35 times out of 100 opportunities. It follows that an outcome with 100% probability means there is no possibility that the outcome will not happen (but this never happens!).

A p value is really a probability that a given outcome could occur by chance. It is usually expressed as a decimal, such as 0.07. A p value, when multiplied by 100, is a percentage. In the above example, the p value of 0.07 means that there is a 7% probability that the observed outcome could happen by chance alone. (This is based on an underlying assumption that certain conditions have been met, which we will look into later.) Another way of stating this is: If the study were repeated hundreds of times under the same circumstances, using members of the same population, an average of only seven of these studies out of 100 would give the result we observe based on chance alone. The reason why each study does not give identical results in these situations is because different samples are used, which results in different estimates of the parameter. We will discuss this concept again but, for now, just realize that the p value represents a probability, which can be expressed as a percentage.

4 Inductive and Transductive Inference: Sample and Population Statistics

Inferential statistics is a way of making inferences about populations based on samples. Given information about a subset of examples, how do we draw conclusions about the full set (including other specific examples in that full set)? Inferences based on principles of evidence use sample statistics. We suggest that matching and similarity-based inferences are based on population statistics. That is, if one has sample statistics then the inferential problem is to treat those statistics as evidence. If one has population statistics then the inferential problem is to assess a degree of match. Psychologists are very familiar with inferential statistics and evidence evaluation: Our studies tend to draw conclusions about populations based on samples. Rarely, if ever, do we have information about the whole population. What are inferences based on population statistics? Fortunately, there is an account of population-statistical inferences, and even a label for such inferences: Transductive.

Developmental psychologists are familiar with the term “transductive inference” from Piaget (1959). Piaget used “transductive” to describe a preoperational form of reasoning that connects specific cases with no general rule or underlying mechanism. Events that are experienced together are taken to be causally related. A car horn honks and the lights go off: The horn caused the darkness. The inference goes from one particular event (honking) to another (darkness) with no notion of a general mechanism. That is, how can honking horns cause darkness? How could this relation be generally sustained? Piaget's transductive inference seems to be a version of the “principle of association” in sympathetic magic (Frazer, 1894; events perceived together are taken to be causally related). Such a principle is also characteristic of a Humean or associative view of causation and causal inference: A causal relation just is the perception of association. Once a person associates horns and darkness, they will come to expect one given the other. Transductive inference (horn → darkness), therefore, seems to be familiar associative learning and similarity-based induction. Perhaps, Piaget was illustrating some special characteristics of young children's similarity-based inferences (e.g., “one-trial” associations) but the general process of inference is familiar. In particular, it is not clear how Piaget's transductive inference addresses the point that similarity-based inferences rely on population statistics. There is another account of transductive inference that does, however.

Apparently independent of Piaget, the term “transductive inference” was introduced into the machine learning/statistical learning theory literatures by Vladimir Vapnik (1998). Vapnik also uses “transductive” to refer to inferences from particular examples to particular examples. For Vapnik, though, transductive inferences are a kind of simplifying assumption, a way of approaching complex learning problems. Vapnik argues that (standard, evidential) statistical inference is an attempt to solve a general problem: From experience with a set of examples how can the learner construct a rule or pattern that can then be applied to new examples? The key feature of evidential inference is that the class of potential new examples is infinite. The point of transductive inference is that often the class of potential new examples is finite. For example, an inspector might encounter the problem of predicting which widgets in a batch of 100 are defective (see Fig. 1.1). Solving the problem for the particular batch will usually be much simpler than solving the problem of identifying defective widgets in general. For Vapnik, transductive inference is the strategy of limiting focus to the specific examples that the learner will actually encounter. Here is the link to Piaget's transductive inference. If the child's conclusion (horns cause darkness) is restricted just to that particular observed situation, then it seems less problematic: It is only when generalized that it falls apart. Similarly, the link to sample and population statistics becomes more apparent. In (standard, evidential) statistical inference, the 100 widgets are a sample. The learner's task is to use this sample to form representation of the class or population of widgets. In transductive inference, the 100 widgets are the population. The learner's task is to learn and use statistics to make predictions about this population.

An example will help illustrate the relation between transductive-population and evidential-sample inferences. Consider the 100 widgets. Suppose an inspector has examined all 100 and discovered the following: There are 10 black widgets and 90 white widgets moreover 5 of the black widgets are defective and 9 of the white are defective. These are descriptive statistics. The inspector's challenge is to use these descriptive statistics to make some inferences. We distinguish two kinds of inferences the inspector could be called upon to make: transductive or evidential (what we have been calling “standard statistical inference”). If the inference concerns the 100 widgets, the statistics observed are population statistics. For example, suppose the inspector is shown one of the widgets and told that it is white. The inspector recalls that only 10% of white widgets were defective and predicts that it will be fine. This is clearly a kind of inductive inference in that it is not guaranteed to be correct: The inspector's past experience makes the conclusion probable but not certain. But it is a special kind of inductive inference, a transductive inference. The inspector's problem is relatively simple. After having calculated the descriptive statistic, p(defective|white) = 0.1, there is really very little work to be done. The inspector can be confident using the descriptive statistic to guide his inferences because the statistic was calculated based on the examples he is making inferences about. In a certain respect, the inspector is not even making an inference, just reporting a description of the population. To move from “9 of these 90 white widgets are defective” to “one of these white widgets has a 10% chance of being defective” to “a white widget selected at random is probably not defective” hardly seems like much of an inductive leap at all. Put slightly differently, once the inspector has solved the descriptive problem (what is p(defective|white) among the 100 widgets?) the inferential problem of making a prediction about a randomly selected widget is easy.

The inspector faces a more difficult problem when making inferences about a widget not in the “training” set, widget 101. In this case, the descriptive statistics (e.g., p(defective|white) = 0.1) are characteristics of a sample and the inspector must calculate inferential statistics to make predictions. In this case, the inspector must consider the evidential relation between his sample (of 100 widgets) and the general population (from which the new widget was drawn). Is the sample biased? Recognizing and adjusting for sample bias is a specific problem of evidential inference. It is this inferential problem that distinguishes transductive inference from evidential inference. Sample bias does not matter when making a transductive inference to one of the 100 original widgets.

Consider the problem of predicting the color of a widget identified as defective. If the defective widget was one of the 100, the prediction is clear: It is probably white. Nine of the 14 defective widgets encountered were white. If the defective widget is new, widget 101, the prediction is less clear. The 100 original widgets were mostly white. Is that true of the general population widgets or is that a bias in the sample? Unless the inspector knows about the relation between his sample and the population he cannot use the former to make predictions about the latter. However, figuring out that relation, solving this inferential problem, is irrelevant for a transductive inference. If the 100 widgets are considered the population, there is no sampling bias. In this way, transductive inference can be used as a kind of simplifying assumption for inductive inference. Transductive inference is inductive inference where sample-population relations are ignored, where sample statistics are treated as population statistics. The challenge of transductive inference is limited to developing useful descriptions (characterizing the patterns in the available data).

3.02.3.4 Use of Statistics with Single-subject Data

For the most part, inferential statistics were designed for use in between-group comparisons. The assumptions underlying the widely accepted classical model of statistics are usually violated when statistical tests based on the model are applied to single-subject data. To begin with, presentation of conditions is not generally random in single-subject designs, and randomization is a necessary prerequisite to statistical analysis. More importantly (and more constantly), the independence of data required in classical statistics is generally not achieved when statistical analyses are applied to time-series data from a single subject (Sharpley & Alavosius, 1988). Busk and Marascuilo (1988) found, in a review of 101 baselines and 125 intervention phases from various single-subject experiments, that autocorrelations between data, in most cases, were significantly greater than zero and detectable even in cases of low statistical power. Several researchers have suggested using analyses based on a randomization task to circumvent the autocorrelation problem (Edgington, 1980; Levin, Marascuilo, & Hubert, 1978; Wampold & Furlong, 1981). For example, data from an alternating treatment design or extended complex phase change design, where the presentation of each phase is randomly determined, could be statistically analyzed by a procedure based on a randomization task. Some controversy surrounds the issue (Huitema, 1988), but the consensus seems to be that classical statistical analyses are too risky to use in individual time-series data unless at least 35–40 data points per phase are gathered (Horne, Yang, & Ware, 1982). Very few researchers have the good fortune to collect so much data.

Time-series analyses where collected data is simply used to predict subsequent behavior (Gottman, 1981; Gottman & Glass, 1978) can also be used, and is useful when such predictions are desired. However, such an analysis is not suitable for series with less than 20 points, as serial dependence and other factors will contribute to an overinflated alpha in such cases (Greenwood & Matyas, 1990). In cases where statistical analysis indicates the data is not autocorrelated, basic inferential statistical procedures such as a t-test may be used. Finally, the Box-Jenkins procedure (Box & Jenkins, 1976) can technically be used to determine the presence of a main effect based on the departure of observed data from an established pattern. However, this procedure would require a minimum of about 50 data points per phase, and thus is impractical for all but a few single-subject analyses.

In addition, most statistical procedures are of unknown utility when used with single-subject data. As most statistical procedures and interpretations of respective statistical results were derived from between-group studies, use of these procedures in single-subject designs yields ambiguous results. The meaning of a statistically significant result with a lone subject does not mean the same thing as a statistically significant result with a group, and the assumptions and evidentiary base supporting classical statistics simply dose not tell us what a significant result with a single subject means. Beyond the technical incorrectness of using nomethetic statistical approaches to ideographic data, it is apparent that such use of these statistics is of extremely limited use in guiding further research and bolstering confidence about an intervention's efficacy with an individual subject. If, for example, a statistically significant result were to be obtained in the treatment of a given client, this would tell us nothing about that treatment's efficacy with other potential clients. Moreover, data indicating a clinically significant change in a single client would be readily observable in a well-conducted and properly graphed single-subject experiment. Statistics—so necessary in detecting an overall positive effect in a group of subjects where some improved, some worsened, and some remained unchanged—would not be necessary in the case of one subject exhibiting one trend at any given time.

9.3.7 Point and Interval Estimation

One of the main tasks of inferential statistics is to estimate the population parameters from those of a sample. For example, when we calculate a certain biodistance between two samples, we make the (silent) assumption that this biodistance is, in fact, the biodistance between the populations from which the samples come. This approach is called point estimation of the properties of a population and it is strictly valid when the parameter under consideration is an unbiased estimator. By this term we mean that the sample parameter has the following property. If we take a plethora of samples from the same population, compute the value of this parameter in each sample, and then average all these values, this average value tends to the parameter value in the population. The mean value and the sample variance are unbiased estimators, whereas the sample standard deviation is a biased estimator of the population standard deviation. In regard to biodistances, unbiased estimators of population distances are the corrected squared Mahalanobis distance (CMD) and the MMD, whereas the squared Mahalanobis distance (MD) and the Mahalanobis-type distances TMD and RMD, as defined in Chapter 5, are biased estimators (Nikita, 2015; Sjøvold, 1975, 1977).

An alternative approach to estimating population parameters from samples is by means of confidence intervals. The P% confidence interval for a parameter is the range of values that includes the unknown population parameter with a probability equal to P%. The P% probability can also be written as P = 100(1 − α), where α is, in fact, the significance level, defined in the previous section, and signifies the probability that the population parameter lies outside the estimated confidence interval. For instance, when P = 95, then α = .05. In this case, the population parameter under study has a 95% probability of lying within the confidence interval and 5% probability of lying outside this interval.

Inferential Statistics

The other method for analyzing data is through inferential statistics. Used to make interpretations about a set of data, specifically to determine the likelihood that a conclusion about a sample is true, inferential statistics identify differences between two groups or an association of two groups; the former is more common in the pharmaceutical literature. Inferential statistics requires the performance of statistical tests to see if a conclusion is correct compared with the probability that conclusion is due to chance. These tests calculate a P-value that is then compared with the probability that the results are due to chance. This is alpha (α), which is most often 0.05; therefore, a P-value less than 0.05 is typically considered statistically significant.