Chapter 5 – Econometrics

The final chapter of this script gives a brief introduction into the realm of econometrics. Put simply, econometrics can be described as the study of methods that allow statistical analysis of economic issues. Such analyses are of central importance to the profession: no matter how sophisticated or convincing a theoretical model may be, the strongest proof of a theory is always its consistency with observations made in the real world. Therefore, economists need a set of tools allowing them to contrast their theoretical considerations against the real world. Conversely, econometric methods also allow for more exploratory analyses of empirical relationships, the results of which may themselves give rise to hypotheses that form the bases of interesting theoretical studies. Finally, independent of any economic theory, econometric methods can be used to assess the effectiveness of different political measures (policy evaluation), typically by treating the policy intervention as an exogenous shock to the economy, the causal effect of which is relatively easily identified through the use of appropriate methods.

The mathematics of econometrics are somewhat distinct from what we have seen thus far. It relies on new concepts, specifically random variables and their moments (expected values, variance, etc.), but at the same time, especially the concepts of differentiation, integration and optimization are also of central importance, which should facilitate our familiarization with econometrics. This chapter tries to serve as a refresher of/introduction to the very basics of econometrics, without going too deep into any specific direction. As such, it covers an introductory discussion of correlation and causation, the concept of random variables, and the linear regression model and its estimation.

Correlation does not mean or imply Causation

When engaging in an empirical analysis, our aim is usually to identify causal relationship, that is, we seek to arrive at conclusions such as “X causes Y to increase” or “Productivity growth has slowed down because the rate of technology diffusion has decreased”. In other sciences such as medicine, such statements are readily proved or disproved using controlled experiments. Put simply, to understand the effect of some treatment, you split a test population into two identical groups, apply the treatment to one group but not to the other, and observe how outcomes differ on average between the groups. While the outcomes will never be exactly identical across groups, we can use statistical methods that allow to quantify how likely it is to obtain the observed difference under the hypothesis that there is no treatment effect. If the likelihood is sufficiently small (usually: the difference of outcomes is sufficiently large), then we may claim to have identified a treatment effect.

In economics, however, things are rarely as simple. First, selecting two groups of identical economies/economic regions is already a vastly difficult task, as there are a multitude of economic characteristics along which regions may differ, such as demographics, political orientation, trade integration, and many more. Especially if not all relevant characteristics are observable, defining two comparable groups may be difficult. Much more importantly, however, economic experiments would usually be (i) very costly, and (ii) potentially unethical. Considering the example of slower technology diffusion and lower productivity growth from before: if you artificially banned some firms from adopting technologies that could improve their performance, these firms may fall behind in competition, and you could cause, among others, worker layoffs, firm bankruptcies, weakened competition, and lower government income from revenue tax. Further, if your hypothesis is indeed true (and existing research seems to show that it is), you would weaken aggregate productivity growth, a central determinant of overall economic well-being.

Sometimes, researchers find themselves in the situation that nature takes over the role of the unethical interventionist, e.g. when natural disasters destroy a relevant share of economic infrastructure in some regions, but does not in other neighboring and thus comparable regions. Here, quasi-experimental methods can be applied to obtain causal conclusions. The more common scenario, however, is that economists use methods of correlation analysis, and need to argue that the identified correlational relationships are indeed causal. Verbally, correlation refers to the co-movement of certain quantities. A perfect correlation is when quantities co-move in a one-to-one fashion without any deviation; a positive perfect correlation would, for instance, observed for the sale of left-hand and right-hand gloves, if gloves are sold in pairs, and a perfect negative correlation can be assumed between the inventory stock and the quantity of sales, so long as there is no restocking. Less obvious real-world correlations are rarely perfect. Coming back to medical treatments, some patients may not respond as much to a treatment as others, but as long as the treatment is effective in some way, we should observe a positive correlation between the intensity of the treatment and the health outcome.

For economists that mainly analyze correlations, a critical conceptual circumstance is the following: correlation does not imply causation, or put differently, correlation is not a sufficient condition for causation. Two simple examples help illustrate this point.

1. Over the past decades, in the US, energy consumption has increased while the marriage rate (share of unmarried persons in legal marrying age getting married in a given year) has decreased.
2. Over a given year, in months with higher ice cream sales, the number of shark attacks is higher.

Example 1 describes a negative correlation between energy consumption and the marriage rate, while example 2 describes a positive correlation between ice cream sales and shark attacks over time. So, does energy consumption decrease people’s willingness to get married, and do ice cream sales cause shark attacks? Clearly not. While the two quantities of example 1 are likely completely independent, in example 2, the quantities have a common determinant, but are not directly related. This common determinant is seasonality, or respectively the weather. On warm, sunny days, more people buy ice cream, and go to the beach swimming where they can be attacked by sharks. Therefore, the weather directly determines ice cream sales, and indirectly determines shark attacks.

A further limitation of correlations is that they only describe co-movement, but they also do not allow to infer on the direction of a potential causality between quantities. For instance, if employment and GDP are positively correlated, even if we are told there is a causal relationship between the two quantities, we don’t know whether higher GDP causes employment to increase, or whether increasing employment causes higher GDP.Crucially, as correlations may be in part driven by forces entirely unrelated to the causal relationship, it could even be the case that higher GDP causes lower employment, despite the quantities being positively correlated. That is, a positive correlation does not preclude a negative causal relationship, and vice versa, see also the example below. Such issues, unfortunately, can ex ante also be a concern in the more sophisticated models that we use. If we want to study how affects , but may also affect , we call this issue reverse causality.

The following example of a fictional economy describes how it may potentially be very harmful to base economic actions on insights from correlations. Consider an economy of a “saturated labor market” where strong population growth leads to only modest increases in the employment level:

For simplicity, production is assumed to follow , where is employment. Suppose we were interested in understanding the effect of the unemployment rate UR on GDP growth . In this example, the higher the unemployment rate, the higher GDP growth. If a policy maker would infer causality from this correlation, they could conclude that forcing workers into unemployment could boost GDP growth, possibly a desirable policy target. However, in the example, the unemployment rate and GDP growth are jointly determined by the total population (or population growth) and the induced effect on employment , similar to the example of the seasonality origin of the correlation between ice cream sales and shark attacks. Indeed, if you increased the unemployment rate at a given population level, i.e. holding this quantity constant, you would reduce employment, and thus and , so that the unemployment rate has a negative effect on GDP growth, despite the positive correlation.

In this example, the true effect of the unemployment rate on the outcome of interest, GDP growth, could be uncovered from considering the relationship holding constant a third variable that drives the correlation but is unrelated to the causal relationship. This is precisely the idea of the methods we use in practice: they allow to “hold constant”, or control for a set of observable quantities, and thereby enable us to zoom in on the residual relationship of quantities of interest unexplained by controlled quantities. By doing so, we are able to isolate the “direct” correlation between the two variables that abstracts from indirect dependence through third variables, and if we account for all such third variables, this direct correlation can be interpreted causally. The practical issue then is to know which third variables to account for, what to do if some of them are unobserved and/or unmeasurable, and how to identify the direction of the causality, that is not addressed by the mere correlation. To this end, a common tool, especially in macroeconometrics where we work with data that entail a time dimension, is to study dynamic relationships, i.e. the correlation between and , exploiting that causation works in only one direction through time.

To wrap up, you should take away that correlations are not always useful in arguing for causal relationships, as they can occur entirely at random (e.g. energy consumption and marriage rate), or due to joint dependence on a third variable (e.g. ice cream sales and shark attacks). Further, the sign of a correlation is not always informative about the sign of the causal relationship even if there is one (e.g. unemployment rate and GDP growth) if third variables play a role. Even if a correlation does describe a causal relationship (i.e., there are no third variables, or all relevant third variables are “held constant” by an appropriate model), it does not tell us the direction of the causal relationship, which needs to be argued for separately.

Probability Spaces and Random Variables

Before moving to our methods of empirical analysis, as per usual, a key step is to have all ingredients and our vocabulary straight. In the world of statistics, in which econometrics is a sub-discipline, the foundation of everything is probability theory, or stochastics. We will touch on this issue very briefly, focusing on the core concepts relevant to economists. The concepts we discuss here, especially random variables and their moments, are also key in many macroeconomic models that feature some uncertainty, and where agents have to form expectations about the future and adapt their behavior accordingly.

Usually, when engaging in empirical analysis, we try to learn something about the future, either directly or indirectly. A direct empirical question related to the future could be: “Will increasing university budgets augment GDP growth, and if so, by how much?” More indirectly, one may ask “Has increasing university budgets augmented GDP growth over a recent period, and if so, by how much?” The relevance of this question, however, also lies in the future: by looking at the (recent) past, we want to know whether one could expect similar relationships also in the future. Generally, we can learn about the future from the past if conditions remain similar enough. In this case, we exploit the circumstance that the events in the past used to be the future at some point, and outcomes were yet to materialize. In other words, as the outcomes were associated with some uncertainty, they were, to some degree, random (or: unpredictable), just as the future’s outcomes are somewhat random from today’s perspective. So, if conditions do not change much, we should be able to learn from the realizations of uncertainty in the past about what to expect for uncertainty in the future, and we can also quantify how accurate these expectations are, based on how accurate they would have been for the period on which we already have data.

This describes the intuition of why probability theory is relevant to economics, and especially econometrics: it allows to model the future as an environment associated with uncertainty, and we are able to describe this uncertainty and form expectations based on the assumption that the past was characterized by the same model, with the only difference that the uncertainty has already materialized. Therefore, a crucial thing to keep in mind is that all predictions for the future obtained from econometric analysis, including the effectiveness of policy measures and the identification of structural economic relationships, inform about the future only to the degree that framework conditions do not change significantly relative to the period covered by analyzed data. This is why well-published empirical papers tend to use data that is as recent as possible, and it is also why economic crises such as the global financial crisis around 2008-2012, but also more recently the Covid pandemic and Russia’s war on Ukraine and the ensuing economic repercussions, constitute considerable challenges for economists: these times of very unique economic conditions are unprecedented in history, and analyzing data from the past will only be partially able to guide effective policy in response to these crises.

Probability Space and Probability Measure

Definition: Probability Space.
A probability space is a triple , where

• is the sample space, the set of all possible outcomes,
• is the event space, the set of all possible events, and
• is the probability measure that assigns events a probability.

The above definition defines the probability space. It is best illustrated at a simple example: consider the case of rolling a regular dice. Before the dice is rolled, the set of possible outcomes is . One example of an event that could occur is that a specific number is rolled, e.g. or . However, it could also be that an even number is rolled, , or that a 6 is not rolled, . So, any combination of elements in may constitute an event, and more generally, also beyond this simple example, we tend to consider , i.e. the power set of the sample space, as the set of possible events.

It remains to define the probability measure that assigns events a probability between 0 and 1. Clearly, the measure should assign , the event that any number between 1 and 6 is rolled, a probability of 1. Further, for any specific number, i.e. , , the probability should be . More generally, as the dice is fair (i.e., all numbers have the same probability), we can define , where is the cardinality of the set: the probability of an event is the amount of different numbers it allows, divided by the amount of all possible numbers.

Definition: Event Space, -Algebra.
Consider a sample space . Then, a set is called an event space, or -algebra if

• and ,
• for any , for , it holds that ,
• for any , it holds that .

Definition: Probability Measure.
Consider a sample space and an event space . Then, a function is called a probability measure on if

• and is the sample space, the set of all possible outcomes,
• if are disjoint, i.e. , then .

While is always the “base set” of possible outcomes and requires little further characterization to allow generalization to richer stochastic environments, the other two components of the probability space do. These characterizations are given in the definitions above. Verbally, the event space requires that we consider that “anything can happen” (), that for any event , the opposite can happen, and that for any two events and that can happen, the event that either one of them can happen, , is also accounted for. The simplest way to achieve this is , which is the default case for everything to follow.

The definition of the probability measure may seem surprisingly general. Next to the intuitive characteristic of assuming only numbers between 0 and 1, we only require that “anything will happen” with probability 1 () while “nothing happens” with probability zero (), and that further, the probability that either one of two distinct events occurs is the sum of their probabilities. To see the latter, imagine the dice roll with and for which , and . The focus on disjoint sets is crucial: if some elements would realize both and , for instance modifying , then additivity need not be assumed. Note that by the second property, it is usually sufficient to know the probability of events referring to individual elements of , as any event can be decomposed into a disjoint union of such events: , . For notational simplicity, we define for .

The reason that the definition of a probability measure is so broad is that we want to allow ourselves to adapt the concept flexibly to concrete contexts. While we are allowed to call many functions a probability measure that describes the experiment of a dice roll in accordance with the definition of the probability measure, as we have seen above, only one of them suits the specific context of rolling a fair dice. Similar to the metric concept, the definition gives us a range of functions that we may attribute the label of a probability measure, and our exercise in concrete contexts is to find the most useful one. As an exercise, try to define the probability measure that characterizes rolling a rigged dice for which it is twice as likely to roll a 6 than it is to roll a 1, but all other numbers remain as likely as with a fair dice.

We know that for any , so that . By , we get , or , and .

Random Variable

In most practical scenarios, we care less about the realized events themselves, but rather about (numeric) outcomes that they imply. For instance, when you plan on going to the beach, you do not want to know all the meteorological conditions in the space of possible conditions, but you care about the functions and about that measures the temperature in degrees Celsius implied by the conditions . In other words, you care about two random variables: variables on the real line that are determined by (possibly much richer) outcomes in the event set of a probability space. As the beach example illustrates, the appeal of random variables is two-fold: on one hand, they can be defined to address directly what we are interested in, and they allow to reduce dimensionality and possibly reduce the prediction problem. To see the latter point, if we build a model to forecast exact meteorological conditions, this model will need a lot of data and may even then perform poorly, and if different conditions imply a similar temperature, we may be better off forecasting the temperature directly.

Definition: Random Variable, Random Vector.
Consider a probability space . Then, a function is called a random variable, and a vector where , are random variables, is called a random vector.

This definition gives a broad definition of the random variable concept. Those with a stronger background in statistics or econometrics may excuse that it is very vague and simplistic; the concept is mathematically not straightforward to define, and for our purposes, it suffices to focus on the characteristics stated in the definition.

Continuing the example of the fair dice roll, let’s consider a game where I give you 100 Euros if you roll a 6, but you get nothing otherwise. Can you define the function that describes your profit from this game, assuming for now I let you play for free?

The function is , with for and for .

A key concept related to random variables is their distribution, i.e. the description of probabilities characterizing the possible realizations of . Our leading example of the dice roll, and the game where you can win 100 Euros, is an example of a discrete random variable, characterized by a discrete probability distribution. For such random variables, there are a finite number of realizations (in our case: two, either 0 Euros or 100 Euros), and every realization may occur with a strictly positive probability (5/6 vs. 1/6). The other case is a continuous probability distribution, with an infinite number of realizations that can not be attributed positive probability: for any . One such example is a different game of dice: say I generate a random real number between 0 and 25, and then I let you roll a dice with result , giving you a profit , so that the result of the dice roll is multiplied by this random number. The first stage of this game, i.e. randomly choosing a number from , is characterized by the probability distribution for with . More commonly, we set as the minimal possible value (sometimes ) and characterize the distribution as , in our case . The probability density function is the derivative of this function: , in our case .

Definition: Expected Value.
Consider a probability space and a random variable on this space, with probability density function . Then, the expected value of is defined as the integral

and for a transformation of ,

Definition: Variance, Standard Deviation.
Consider a probability space and a random variable on this space. Then, the variance of is defined as , and its square root is called the standard deviation of , denoted . The variance can generally be computed as .

Before moving to economic models, we need to introduce two more key concepts, given by the definitions above. The expected value gives us the best possible prediction of given its distribution. Crucially, you always use the probabilities referring to , even if you consider a transformation of . For discrete distributions, when is the set of values that can take with positive probability, the expression simplifies to . This shows the intuition of the expected value: you expect the value with probability , and you expectation is then simply the sum of all these terms. While the expected value gives you a “mean” of what to expect for , it is silent on how good this prediction is (note that the expected value need not be a possible realization of . In the case of the dice roll, you can convince yourself that , which is not a number that the dice can show). This is the purpose of the variance, measures the expected degree of deviation of from the expected value. As the variance is based on a square term, we usually re-normalize it by taking the square root when talking about deviations from the expected value, which we call the standard deviation.

To close this section, you may test your understanding of the concepts thus far by answering two questions: First, if you were neutral to risk (i.e., you only care about your expected profit), would you play the game where you can win 100 Euros if you roll a 6 if you needed to buy in for 15 Euros (also think about whether you would spontaneously agree to this game without doing the math)? And second, what game do you prefer in terms of expectations, the one where you get a 100 Euros for a 6, or the one where you get Euros, and is drawn randomly from ? And which of these games has the more certain/less volatile outcome? For the latter game, you may use that the dice role is independent of the randomly drawn number , so that for any . (If you get stuck, feel free to open and read through the solution.)

The expected value of game 1 is

So you should want to pay no more than 16.67 Euros to play this game. If I offer this to you for a buy-in of 15 Euros, if you are risk-neutral, you should do it.

On a side note, in practice, most people would probably not take this game for 15 Euros, as people are risk-averse (they don’t assess games based on expected values) and they dislike losses more than they like gains. Behavioral economists argue that with these conditions, it is rational for people not to play such a game. Things were to change only if we played this game repeatedly, say a 1000 times, where you would expect to eventually be close to the expected profit on average. This is indeed a central result of probability theory, the law of large numbers, as introduced later: as for any fixed , if the realizations of are independent and follow the same distribution (independent and identically distributed, i.i.d.). Verbally, by repeating a game/an experiment often enough, the average of realizations approaches the expected value with arbitrary precision.

For game 2, the expected value is

Therefore, this game is much preferable to game 1 in terms of the expected profit. To address the volatility of outcomes, we need to compute the variance. Since , it remains to compute the expectations of the squared terms. First, for game 1,

so that

with .

Next,

and in analogy to above,

so that

with .

Therefore, not only is game 2 better in terms of the expected value, but it is also slightly less uncertain/volatile. The more risk-averse an agent is, the more they value certainty, and the uncertainty comparison could therefore be an additional practical reason to opt for game 2 rather than game 1.

Linear Regression Model

Having laid the foundation of stochastic concepts, this final section proceeds to discuss basic econometric models used in empirical practice. Recalling the introduction to this chapter, these models aim to do the first step from correlations to causality by allowing to “control”/”hold constant” third confounding variables. We will thoroughly keep this intuition in mind when discussing their definition and estimation.

Specification

One final concept that we need to understand the relationship between random variables, as we intend to model in the following, is the one of a conditional expectation – the value we expect a random variable to take knowing the value of . This quantity we call , the expected value of given . To understand this concept in simple terms, recall the game 2 in dice roll experiment: in the first step, we drew a random number from with a uniform distribution, i.e. all parts of equal length within were equally likely. For the purpose of notational consistency, let’s for now call this variable rather than , as we did before. The expected value, as calculated earlier, was . More generally, if we draw randomly with uniform distribution from , then the expected value is . But what if we draw the upper bound at random in a first step? Then things become a lot more complicated, and depend on the distribution from which we draw the upper bound . Still, if is the random variable that gives us the upper bound , we can say that . Note that is still a concrete realization of . However, we do not need to condition on concrete realizations: no matter what will be, given that we know , we know that the expected value of will be . Therefore, we write . This object, now, is a random variable that is a function of : with .

This simple example illustrates an important point: when we condition on concrete values of , the conditional expectation will be a real number, as the “unconditional” expectation is, too (note that this statement refers to the type of object; in general, it will not be the case that ). More generally, the conditional expectation of given is a random variable itself, and functionally depends on : .

Now it is time to put these concepts to use. Recall that initially, we set out to come up with a model that allows to causally address how a variable affects another variable . Since we want to speak to the future where the realizations of and are not known, we treat these quantities as random variables, and exploit that we have data that records existing realizations of these random variables in the past. The simplest way to describe a relationship between and using the conditional expectation is to define , where is the deviation of from what we would expect for it given , or the error made from predicting using . Re-arranging terms, one obtains

We know that generally, for an unknown function . While we do not know itself, we do know Taylor’s theorem, which tells us that a polynomial approximation to may be “good”. The simple, linear model assumes that a first order approximation fares sufficiently well. It should be mentioned that there are a number of tests for linearity, but they are considered beyond the purpose of this course. Under this assumption of linearity, , and the equation becomes

In this model, if we observe that is higher by units, we expect to be higher by units. Note that this does not assert anything about causality: a non-zero coefficient simply tells us that is useful in predicting , similar to the way ice cream sales are useful in predicting shark attacks. To move closer to a causal interpretation, let us consider a random vector of third variables of length that may describe an indirect relationship between and , such as seasonality in the ice cream/sharks example, or the population and employment level in the unemployment rate/GDP growth example. Similar to before, we can write

Then, measures the predictive potential of for conditional on , i.e. holding constant . To see this more explicitly, consider the equation in concrete realizations,

where the marginal effect of increasing the realization of on can be obtained as

In this view, is the effect that has on , holding the control variables constant at the levels . By the linear nature of the model, the levels at which these variables are held constant do not affect the marginal effect of on . Therefore, coming back to the example of unemployment rate and GDP growth, an appropriate empirical model is

In this model, describes the relationship between and GDP growth, holding constant the levels of population and employment. Still, the sign and magnitude of only tells you how useful is in predicting GDP growth at constant population and employment, and this relationship could, in practice, also arise due to further omitted variables that are not yet included in the model, or because GDP growth affects the unemployment rate. This aspect is discussed further in the last section of this chapter.

Estimation

Assume that you have arrived at a model that you think is useful in describing the relationship of interest, of the form

(1)

This model will tell you something interesting only if you manage to know/get a plausible estimate of the coefficient of interest, . For what follows, we return to the simpler case of only one random variable on the right hand side of the model, but the methods easily extend (with some slight complexification of the algebra) to the multivariate case. So, for now, we return to

(2)

Assume that you observe units of data on , . The intuition behind the approach to estimating the coefficients and is relatively simple. We know that the expected value of given , , which we assume to be linear, i.e. , is the best prediction that can be obtained for given . Therefore, we when calculating the estimate of from the data, we proceed to choose such that it gives the best linear prediction of the data points given . If we slightly rephrase the issue as optimizing the quality of prediction, things begin to sound very familiar. Indeed, the only part that we need yet to specify is what we mean by “best” prediction. As per the economist’s preference for the Euclidean space (recall that the Euclidean norm captures the direct distance), it seems natural to minimize the distance of the vectors of data points for and , where

is the data matrix for the equation’s right hand side (RHS) that stacks the observations of RHS variables in its columns. To make sure that you understand the structure of this matrix, think about how it would look like if you had the more general model of the first estimation equation with , i.e. only one control variable .

The matrix with one control variable is

where are the observations for the random variable . The vector would in this case be .

With this in mind, we can express the optimization problem as:

With the tools the previous chapters have equipped us with, solving this should be a relatively simple task: there are no constraints, and the objective function entails a polynomial of the solution. One source of complication could be the square root, but we can easily circumvent it by solving the equivalent problem

(3)

This formulation gives the most common estimators of the linear model their name: least-squares estimators. To see this name more intuitively, note that
is the error in predicting using the coefficient vector for prediction by , which we also call the residual. Therefore, the solution , provided that it exists, minimizes the residual sum of squares , i.e. the sum of squared prediction errors.

We are now set up to derive the (hopefully unique) solution for the estimator . To test your understanding of Chapter 4, you can try to derive it yourself before reading on. You may use without proof two helpful properties of differential calculus with matrices that are investigated on the problem sets of the course:

1. For and , , and
2. For and ,

Further, it may be helpful to recall the discussion of positive definiteness and the rank; you may assume that has full column rank. To help you get started, things remain more tractable if you stick with the vector notation, and start from the problem

If you have tried to solve it, or you want to read on directly, let’s get to the solution of this problem. The first step is to multiply out the objective. Doing so, the problem becomes

A first useful observation is that is a scalar, so that , and the problem simplifies to

In obtaining the first order condition, the two properties of differential calculus with matrices mentioned above are very useful. Applying them, we know that

Putting these together, the first order condition is

so that the solution must satisfy

While it looks like we are almost there, there are two things that we need to address: (i) can we invert to obtain a unique solution? (ii) If we can, does this solution constitute a (strict) local minimum? (And is this local minimum global?)

We know that matrices of the form for are always positive semi-definite: for , for . Now, recalling Chapter 2, this matrix is indeed positive definite if for any , , which is the case if has full column rank, i.e. no column of can be written as a linear combination of the remaining columns. In our case, has just two columns, and one of them is the vector containing only ones. Therefore, the matrix does not have full column rank if there exists such that for all , in which case the second column of , , can be expressed as .

In less abstract terms, this occurs if the random variable exhibits no variation across the observations in our data, i.e. for every observed unit , we observe the same realization . In this case, we would say that is collinear to the constant. However, in practice, the variables we usually consider do vary across observations, and the issue does not occur. Therefore, a key assumption in least-squares estimation is that has full column rank, or that the RHS random variables (including the “constant” that is always equal to one) are not linearly dependent. This assumption is usually met unless one adds variables that essentially give the same information. In practice, statistical softwares will usually give you an error message if this condition is violated in the data you analyze.

While collinearity to the constant is a relatively trivial issue that is easily avoided, in models with more RHS variables, the issue of collinearity can be less direct to spot. If you have vectors and of observations for random variables and , respectively, then collinearity occurs if for some . A simple example is and when units refer to persons, another one is and years until 100th birthday, where and . Intuitively, however, the issue of collinearity is still similar to the univariate case: you are adding at least one variable that does not give you new information beyond those you already have, and if you have two (sets of) variables that measure essentially the same thing, a statistical model will be unable to choose either one over the other in predicting the outcome variable . On this, recall also our discussions of non-full ranks and their interpretation as “redundant equations” in the analysis of linear equation systems in Chapter 2: even in the narrow mathematical sense, the full rank condition can be interpreted as the model not containing any redundant variable.

Returning to the optimization problem, if the observations for , vary across individuals, then the unique solution is

and it constitutes a strict local minimum by positive definiteness of , the second derivative of the objective function. This local solution is also global, as for any , , so that the objective vanishes to in any asymptotic direction.

This solution , called the ordinary least squares (OLS) estimator, can be shown to have very strong theoretical properties: it is unbiased, i.e. , meaning that it is expected to estimate the true parameter , and within the class of all estimators that are unbiased, it has the lowest variance, i.e. it is expected to provide the estimate closest to for any given sample. This estimator always estimates the coefficients of the linear conditional expectations function , and as the excursion below discusses briefly, even if the conditional expectation function is not linear, the coefficients estimated by the OLS estimator recover the best linear prediction of given . This is a crucial point: when you estimate OLS, you always estimate the best linear prediction, which corresponds to the conditional mean if it can be described by a linear function. Any discussion of “misspecification” or “estimation bias” does (or at least: should!) not argue that we are unable to recover the linear prediction coefficients, but that we are unable to interpret these coefficients in a causal/desired way. The last section addresses this issue to more detail.

Excursion. Non-linearity and best linear prediction.

A crucial question concerns the quality of the OLS estimator when the true conditional expectation function is not linear, i.e. there exist no parameters such that . A first case that can generally be easily handled by the concepts introduced thus far is the one of linearity in parameters. One such example is . Here, even though the conditional mean is not linear in , the inclusion of achieves a setup that is consistent with the linear model we introduced. Note that generally, for any given , so that inclusion of the squared term does not create a collinearity issue. However, in this case the marginal effect of on depends on the level of : . Another example is , the case of log-linearity. Also this setup is consistent with our model. If , then we have a log-log model where can be interpreted as an elasticity capturing a relationship in percentages: if increases by 1%, then increases by %.

However, linearity in parameters may often be a restrictive assumption. Without taking a stance on the functional form of , we can still interpret the linear model

and its multivariate extensions in a useful way. To see this, note that the OLS estimator recovers the best linear prediction of given in the data, i.e. given the sample of observations of pairs of and . It relies only on minimizing the residual sum of squares, and not on linearity of the conditional mean for the underlying random variables. Thus, the question at hand is how to interpret this linear prediction if not through the conditional mean. For this, consider the linear forecast for using an arbitrary vector . Then, the forecast for is good if

1. , and
2. is as small as possible.

Equality in point 2 follows from and . Therefore, such forecast would be right “on average”, and have the minimum amount of volatility. Let’s see how the optimum looks like:

where the second equality uses linearity of the mean. By this property of linearity, we can also isolate non-random vectors at the beginning and end of expressions from the expected value, specifically the vector . Writing , the expression becomes

Optimizing this expression with respect to is analogous to the OLS problem, and provided that is invertible, one obtains the unique solution

This looks very familiar to the OLS solution. This is no accident: while OLS recovers the best linear prediction in the data, the estimated parameter is the best linear prediction in expectation, or in the population. To reiterate, regardless of the functional form of the conditional expectation , the vector of parameters estimated by the OLS estimator gives as the best linear prediction of given , called the linear projection of onto . If the conditional mean is indeed linear, it coincides with the linear projection of onto . Still, the linear projection is the more general concept, as it always exists, regardless of the functional form of the conditional mean.

In terms of interpretation, without relying on the conditional mean and its intuition of marginal effects, the linear projection coefficients tell you how much, on average, can be expected to be higher if is observed to be higher by one unit. If you can argue that the model appropriately absorbs third variables, you can therefore claim that the linear projection model recovers average effects.

Inference

One thing we have only insufficiently addressed thus far is the quality of estimation. While the OLS estimator is not biased, and it is the “most efficient” estimator (lowest variance among the set of unbiased estimators), in practice, its variance (i.e., the expected squared error) can still be very high – just not as high as the one you would obtain with another estimator. Therefore, if you have a simple model and I tell you that OLS has been applied to obtain based on 10 data points, would you be confident to preclude that the true value of is 0? And would you rule out that it is 1?

To answer such questions, we need to address the uncertainty associated with our estimation. Only once we can quantify this uncertainty we will be able to perform meaningful inference on the estimates we obtained. To do so, we rely on the key concepts of convergence in probability and convergence in distribution.

Definition: Convergence in Probability.
Let be a sequence of random variables, and let be random variable. Then, we say that converges to in probability if for any ,

We write .

Definition: Convergence in Distribution.
Let be a sequence of random variables, and let be random variable. Then, we say that converges to in distribution if for any ,

We write . If the distribution of is known, e.g. is normally distributed with mean and variance , i.e. , we write .

To put it more simply, converges to in probability if for large , and are as good as equal (deviation smaller than an arbitrarily small ) with probability 1. The fairly abstract definition of convergence in distribution has a relatively simple interpretation: the sequence of distribution functions associated with the sequence of random variables approaches the distribution function of pointwise, i.e. at any given location (this criterion is indeed weaker than the one of uniform convergence, where we would require convergence across locations ). In simpler words, if is large enough, then the distribution of should be as good as identical to the one of . This is good news especially if the distribution of is difficult to characterize for finite , but the one of is easily obtained. Note that in distinction to convergence in probability, like you can draw two highly distinct numbers from the same normal distribution, specific realizations of can still be very different from those of , but their probability distributions will coincide. Therefore, this concept is weaker than convergence in probability. To link the two concepts, note that convergence in probability implies convergence of to the “random variable” with expectation and , i.e. the variable that is equal to 0 with probability 1. Indeed, most probability limits you will come across are real numbers/matrices, rather than stochastic random variables. This gives the following convention: if , where is a non-random object (e.g. real number or matrix), then this implies , where is the “random variable” equal to with probability 1.

In our context, the random variable will be the OLS estimator. Treating the observations as realizations of independent and identically distributed random variables , the OLS estimator without concrete realizations of data points is

Adopting the convention , we can more compactly represent it as

To address the error we make in estimation, it is useful to consider the quantity . To express it in a useful way, we plug in with to the expression above, and we obtain

This is the quantity that we need to study in order to characterize the error we make in estimating using . The final ingredients are given in the Theorems below. For the sake of compactness, they are not proven here or in the script, but excellent resources exist online or in textbooks. These results are fundamental enough that you will find comprehensive proofs on Wikipedia. However, for our purposes it is much more important to apply them properly, rather than to understand their theoretical justification.

Theorem: Law of large numbers.
Consider a set of independently and identically distributed variables with . Then,

Theorem: Central limit theorem.
Consider a set of independently and identically distributed variables with . Then, for , it holds that

Theorem: Slutsky’s theorem.
Let and be sequences of random variables with and and . Then,

and if is invertible with probability 1, then also

One last detail will help us put everything together: note that

For a derivation of , see the excursion above. With this, we proceed as follows:

Step 1. Applying the central limit theorem,

The variance of the asymptotic distribution is

so that compactly,

Step 2. By the law of large numbers,

Moreover, the law of large numbers gives

Step 3. We have two components that correspond to the setup of Slutsky’s theorem. First, putting together the two results of step 2, we obtain the following from Slutsky’s theorem (recalling that convergence in probability to a constant implies convergence in distribution to the same constant):

Corollary: OLS consistency.
For the OLS estimator , it holds that

and thus also . Therefore, we say that is a consistent estimator of .

This informs us that deviations of from will asymptotically be arbitrarily small with probability 1. However, it does not inform the distribution of for concrete sample sizes as we investigate in practice. For this, we consider instead the distribution of :

We define as the asymptotic variance of . With this result, for sufficiently large samples, the distribution of is approximately equal to the one of a random variable with distribution . This is useful because it also means that the distribution of is approximately equal to (by ), and finally that

Here, is used to indicate that asymptotically, the distribution of can be described as . With some oversimplification, for large enough , the distribution of is as good as indistinguishable from a normal distribution with mean and variance
.

The important bit of information obtained from this is that we now have an approximation to the finite sample variance of that is “good” if the sample is “large”. In practice, a few hundred observations are usually considered to suffice for simple models and a few thousand for more complex models, i.e. longer vectors . The matrix is straightforward to estimate using averages instead of expectations, and replacing the unobserved with its sample counterpart :

In the matrix , the (1,1) element gives the estimated variance of , and the element the one of , denoted as and , respectively. One can show that

This circumstance can be used to test hypotheses such as “” or “” as mentioned in the introduction of this subsection. The interested reader is referred to test theory, but in a nutshell, the procedure is as follows: you assume that your hypothesized value, let’s say , is indeed the true value of . Under this hypothesis, should be normally distributed with zero mean and unit variance. You then investigate what is the probability of observing a deviation of at least from zero under this hypothesis, i.e. in a scenario where is indeed distributed : if , then for ,

By symmetry of the normal distribution, , and

where is the distribution function of the standard normal distribution . Therefore, the probability of observing a deviation from zero of at least
is

This quantity is called the p-value, and is typically reported in regression output tables. If this probability is sufficiently large, you can not reject the hypothesis that . If it is sufficiently low, however, you reject the hypothesis. If your model is specified in a way that allows for a marginal effect interpretation, this rejection is required to claim that the model identifies a non-zero effect of on .

To conclude the discussion of the linear model, we have introduced the concept of the conditional expectation, and the linear conditional expectation regression model. This model is also useful if the true conditional expectation is not linear, among others due to a Taylor approximation justification. In this model, coefficients can be interpreted as marginal effects of one variable on the outcome , holding the other variables constant. Therefore, it allows to describe the partial relationship between and that is unexplained by third variables . This model can be estimated using the OLS estimator, which could be derived from a simple unconstrained optimization problem. This estimator has powerful theoretical properties, and always estimates the best linear prediction of given . The practical exercise of the econometrician is usually not to allow for “consistent” estimation of the linear prediction (which is guaranteed), but to write down a linear prediction that is useful for studying a given practical issue at hand. Beyond estimation, we have covered inference, i.e. methods of quantifying estimation uncertainty, as well as their justification through large sample asymptotics. These methods allow to test for, and especially to reject certain values for the estimated parameters in the model.

Correlation and Causality Revisited

The mathematics of econometric models is one side of the coin the empirical economist has to understand, their practical application is the other. While this script intends to give an introduction to the mathematical foundation of the economic profession, the latter aspect is still important enough to deserve a brief discussion also here. As mentioned in the first sections of this chapter, our goal is usually to arrive at a causal conclusion (“X causes Y”) while our methods measure only a correlational relationship (“X predicts Y”). While the linear model goes one step from correlation to causality allowing to hold fixed some third variables, its interpretation is still inherently correlational: the most we can say is that “holding constant Z, X predicts Y”, or equivalently “X predicts Y given Z”. So, how do we argue for causality, and what needs to be kept in mind in these arguments?

The typical approach is very simple: you assume that there exists a true, causal model of the form

i.e. that has some baseline level , and is linearly driven by () and other factors . This is an assumption that you can not test in practice, just like you can not test any assumption you impose on a theoretical model. Your research questions are typically:

1. Does affect , i.e. does the hypothesis hold?
2. If , is the causal effect economically relevant?

For the latter point, not any non-zero effect concerns/is considered relevant the economist. If you find that the introduction of 1000 robots to an industry reduces employment in this industry by 0.0001%, and there are no industries in your sample that introduce more than 1000 robots in a given year, then you would not claim that robot adoption reduces employment. The threshold to relevance, however, is subjective and depends also on the context.

There are two possible reasons that prevent you from estimating the causal -coefficients through the predictive model:

1. Omitted variables: there are third variables that cause and are correlated with .
2. Reverse causality: causes .

While reverse causality is not a problem in the causal model, it is an issue in the predictive model: suppose you are a firm that sells backpacks at 50 Euros a piece. An employee working in marketing wants to convince you that you should spend more on marketing, as he thinks business expansion could greatly boost the number of backpacks you sell. He shows you some estimates from a linear model he estimated:

where is the number of backpacks sold in month and is your firm’s revenue in this month. He has also learned about inference and shows you that his estimates are highly statistically significant. Would you, then, believe that if you managed to grow as a firm in terms of revenue, this would attract more new customers?

Well, if you re-write his estimates, you obtain

Given that your revenue comes from sales of backpacks which you sell at 50 Euros a piece, this equation does not look very surprising. This issue captures the essence of reverse causality: just because is a good predictor of , even if and are causally related, it does not mean that causes , but the predictive quality could also be because causes . Indeed, his estimates from the predictive equation are perfectly consistent with causal equations where does not cause , i.e. , but causes with coefficient 50.

So, how do we manage to estimate the causal coefficients using the predictive model? For omitted variables, since they cause , they should be contained in the remaining factors. If they enter linearly, we can write , so that the causal equation becomes

Intuitively, this equation explicitly accounts for all variables that would induce to predict for non-causal reasons. Indeed, we can show that estimation of this model allows to recover the causal coefficient , the formalities of this investigation are however beyond the purpose of this course. Note, however, that we design the model to estimate the causal coefficient for , . The causal coefficient(s) for may not be estimated consistently if there are third variables that determine and are correlated with . Thus, always limit your interpretations as much as possible to the coefficients corresponding to variable(s) for which you construct the model, and abstain from speculating on the coefficients of control variables.

Control variables can also be used to address issues of reverse causality: usually, if we are in the comfortable situation to have data for several periods, the first step to circumvent reverse causality is to adjust the timing, exploiting that causation can only work forward in time, not backwards. Continuing the example of the backpack firm, as this month’s sales of backpacks can not be caused by last month’s revenues, this step would suggest to specify the model

However, a second step may still be necessary if backpack sales are serially dependent, i.e. if depends on . For instance, if customers are satisfied with your product and recommend it to others, then if sales numbers are high in one month, they may also be higher in following months because more people recommend your product. In your model, this means that causes (not directly, but indirectly through increased mouth-to-mouth advertising). This is an issue because is clearly correlated with your explanatory variable of interest, . This discussion suggests that you should include as a control variable. Your model becomes

Whether it is sufficient to include just one lag for the dependent (i.e., left-hand side) variable is not a trivial question, but an issue of central interest to the field of time series analysis. Intuitively, a second lag would be needed if it explains today’s values even conditional on the first lag, which may occur for more complex dynamic processes. In practice, a simple but efficient approach is to estimate the model once with and once without the second lag, and compare the estimates obtained for the coefficient of interest. If it is different across models, the second lag should be included, and an analogous test should be performed on whether or not to include a third lag, and the process should be iterated until the coefficient of interest converges.

To summarize, we have briefly reviewed the two most common issues in empirical analysis, omitted variables and reverse causality, and we have seen how they can be addressed using control variables and the appropriate use of variable timing. In practice, especially omitted variables are tricky for two reasons: (i) you do not know ex-ante which third variables may be relevant for the relationship of interest, and (ii) not all of the variables you think are relevant are measured in the data available for your analysis. To address the former point, we usually estimate relatively rich models with many variables, as the cost of missing one important variable are much larger than including a few irrelevant ones, especially with large datasets. Next to specific variables, we also include fixed effects aimed at absorbing unmeasured variables that vary at higher levels; the interested reader can look up the standard fixed effects and the two-way fixed effects model. To address the latter point, several techniques have been developed to estimate causal parameters in environments where (we can) not (be sure that) all relevant third variables are observed. To this end, the interested reader is referred to instrumental variables estimation, difference-in-difference estimation, regression discontinuity design and propensity score matching. All these techniques are frequently used in the economic literature, where the former two have been most prominent.