Econometrics by emasterymw

VIEWS: 48 PAGES: 391

More Info
									Econometrics
Mathematical Economics




     PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information.
                               PDF generated at: Sat, 23 Feb 2013 22:26:53 UTC
Contents
Articles
   Econometrics                               1
   Mathematical economics                     9
   Statistics                                30
   Bayesian inference                        39
   Statistical hypothesis testing            51
   Estimation theory                         71
   Semiparametric model                      77
   Non-parametric statistics                 78
   Simulation                                80
   Probability distribution                 101
   Single-equation methods (econometrics)   107
   Cross-sectional study                    108
   Spatial analysis                         110
   Effect size                              120
   Time series                              130
   Panel data                               138
   Truncated regression model               140
   Censored regression model                141
   Poisson regression                       142
   Discrete choice                          145
   Simultaneous equations model             156
   Survival analysis                        159
   Survey methodology                       164
   Index (economics)                        169
   Aggregate demand                         171
   Operations research                      176
   Decision theory                          185
   Neural network                           189
   Econometric model                        197
   Statistical model                        199
   Model selection                          200
   Economic forecasting                     202
   Mathematical optimization                203
   Programming paradigm                     215
   Mathematical model                         218
   Simulation modeling                        224
   Dynamical system                           226
   Existence theorem                          235
   Stability theory                           236
   Economic equilibrium                       239
   Computational economics                    245
   Input-output model                         247
   Computable general equilibrium             252
   Game theory                                255
   Bargaining                                 274
   Cooperative game                           278
   Non-cooperative game                       286
   Stochastic game                            286
   Sequential game                            288
   Evolutionary game theory                   289
   Matching                                   310
   Economic data                              310
   Computer program                           314
   Survey sampling                            318
   Official statistics                        321
   Comparison of statistical packages         330
   Software                                   342
   Experimental economics                     348
   Experimental psychology                    355
   Social psychology                          364
   Field experiment                           376


References
   Article Sources and Contributors           379
   Image Sources, Licenses and Contributors   386


Article Licenses
   License                                    388
Econometrics                                                                                                                          1



    Econometrics
    Econometrics is the application of mathematics , statistical methods and , more recently, computer science , to
    economic data and described as the branch of economics that aims to give empirical content to economic relations.
    [1]
        More precisely, it is "the quantitative analysis of actual economic phenomena based on the concurrent
    development of theory and observation, related by appropriate methods of inference."[2] An influential introductory
    economics textbook describes econometrics as allowing economists "to sift through mountains of data to extract
    simple relationships."[3] The first known use of the term "econometrics" (in cognate form) was by Paweł Ciompa in
    1910. Ragnar Frisch is credited with coining the term in the sense that it is used today.[4]
    Econometrics is the unification of economics, mathematics, and statistics. This unification produces more than the
    sum of its parts.[5] Econometrics adds empirical content to economic theory allowing theories to be tested and used
    for forecasting and policy evaluation.[6]


    Basic econometric models: linear regression
    The basic tool for econometrics is the linear regression model. In modern econometrics, other statistical tools are
    frequently used, but linear regression is still the most frequently used starting point for an analysis.[7] Estimating a
    linear regression on two variables can be visualized as fitting a line through data points representing paired values of
    the independent and dependent variables.
    For example, consider Okun's law, which relates GDP growth to the
    unemployment rate. This relationship is represented in a linear
    regression where the change in unemployment rate (
                          ) is a function of an intercept ( ), a given
    value of GNP growth multiplied by a slope coefficient        and an error
    term,   :


    The unknown parameters          and     can be estimated. Here         is
                                                                                   Okun's law representing the relationship between
    estimated to be -1.77 and     is estimated to be 0.83. This means that if
                                                                                    GDP growth and the unemployment rate. The
    GNP grew one point faster, the unemployment rate would be predicted             fitted line is found using regression analysis.
    to drop by .94 points (-1.77*1+0.83). The model could then be tested
    for statistical significance as to whether an increase in growth is
    associated with a decrease in the unemployment, as hypothesized. If the estimate of                 were not significantly
    different from 0, we would fail to find evidence that changes in the growth rate and unemployment rate were related.


    Theory
    Econometric theory uses statistical theory to evaluate and develop econometric methods. Econometricians try to find
    estimators that have desirable statistical properties including unbiasedness, efficiency, and consistency. An estimator
    is unbiased if its expected value is the true value of the parameter; It is consistent if it converges to the true value as
    sample size gets larger, and it is efficient if the estimator has lower standard error than other unbiased estimators for
    a given sample size. Ordinary least squares (OLS) is often used for estimation since it provides the BLUE or "best
    linear unbiased estimator" (where "best" means most efficient, unbiased estimator) given the Gauss-Markov
    assumptions. When these assumptions are violated or other statistical properties are desired, other estimation
    techniques such as maximum likelihood estimation, generalized method of moments, or generalized least squares are
    used. Estimators that incorporate prior beliefs are advocated by those who favor Bayesian statistics over traditional,
    classical or "frequentist" approaches.
Econometrics                                                                                                                    2


    Gauss-Markov theorem
    The Gauss-Markov theorem shows that the OLS estimator is the best (minimum variance), unbiased estimator
    assuming the model is linear, the expected value of the error term is zero, errors are homoskedastic and not
    autocorrelated, and there is no perfect multicollinearity.

    Linearity
    The dependent variable is assumed to be a linear function of the variables specified in the model. The specification
    must be linear in its parameters. This does not mean that there must be a linear relationship between the independent
    and dependent variables. The independent variables can take non-linear forms as long as the parameters are linear.
    The equation                   qualifies as linear while                , does not.
    Data transformations can be used to convert an equation into a linear form. For example, the Cobb-Douglas
    equation—often used in economics—is nonlinear:


    But   it    can   be   expressed   in   linear   form   by    taking   the   natural   logarithm    of   both   sides:[8]


    This assumption also covers specification issues: assuming that the proper functional form has been selected and
    there are no omitted variables.

    Expected error is zero


    The expected value of the error term is assumed to be zero. This assumption can be violated if the measurement of
    the dependent variable is consistently positive or negative. The miss-measurement will bias the estimation of the
    intercept parameter, but the slope parameters will remain unbiased.[9]
    The intercept may also be biased if there is a logarithmic transformation. See the Cobb-Douglas equation above. The
    multiplicative error term will not have a mean of 0, so this assumption will be violated.[10]
    This assumption can also be violated in limited dependent variable models. In such cases, both the intercept and
    slope parameters may be biased.[11]

    Spherical errors


    Error terms are assumed to be spherical otherwise the OLS estimator is inefficient. The OLS estimator remains
    unbiased, however. Spherical errors occur when errors have both uniform variance (homoscedasticity) and are
    uncorrelated with each other.[12] The term "spherical errors" will describe the multivariate normal distribution: if
                         in the multivariate normal density, then the equation f(x)=c is the formula for a “ball” centered
    at μ with radius σ in n-dimensional space.[13]
    Heteroskedacity occurs when the amount of error is correlated with an independent variable. For example, in a
    regression on food expenditure and income, the error is correlated with income. Low income people generally spend
    a similar amount on food, while high income people may spend a very large amount or as little as low income people
    spend. Heteroskedacity can also be caused by changes in measurement practices. For example, as statistical offices
    improve their data, measurement error decreases, so the error term declines over time.
    This assumption is violated when there is autocorrelation. Autocorrelation can be visualized on a data plot when a
    given observation is more likely to lie above a fitted line if adjacent observations also lie above the fitted regression
    line. Autocorrelation is common in time series data where a data series may experience "inertia."[14] If a dependent
    variable takes a while to fully absorb a shock. Spatial autocorrelation can also occur geographic areas are likely to
    have similar errors. Autocorrelation may be the result of misspecification such as choosing the wrong functional
Econometrics                                                                                                                        3


    form. In these cases, correcting the specification is the preferred way to deal with autocorrelation.
    In the presence of non-spherical errors, the generalized least squares estimator can be shown to be BLUE.[15]

    Exogeneity of independent variables


    This assumption is violated if the variables are endogenous. Endogeneity can be the result of simultaneity, where
    causality flows back and forth between both the dependent and independent variable. Instrumental variable
    techniques are commonly used to address this problem.

    Full rank
    The sample data matrix must have full rank or OLS cannot be estimated. There must be at least one observation for
    every parameter being estimated and the data cannot have perfect multicollinearity.[16] Perfect multicollinearity will
    occur in a "dummy variable trap" when a base dummy variable is not omitted resulting in perfect correlation between
    the dummy variables and the constant term.
    Multicollinearity (as long as it is not "perfect") can be present resulting in a less efficient, but still unbiased estimate.


    Methods
    Applied econometrics uses theoretical econometrics and real-world data for assessing economic theories, developing
    econometric models, analyzing economic history, and forecasting.[17]
    Econometrics may use standard statistical models to study economic questions, but most often they are with
    observational data, rather than in controlled experiments. In this, the design of observational studies in econometrics
    is similar to the design of studies in other observational disciplines, such as astronomy, epidemiology, sociology and
    political science. Analysis of data from an observational study is guided by the study protocol, although exploratory
    data analysis may by useful for generating new hypotheses.[18] Economics often analyzes systems of equations and
    inequalities, such as supply and demand hypothesized to be in equilibrium. Consequently, the field of econometrics
    has developed methods for identification and estimation of simultaneous-equation models. These methods are
    analogous to methods used in other areas of science, such as the field of system identification in systems analysis
    and control theory. Such methods may allow researchers to estimate models and investigate their empirical
    consequences, without directly manipulating the system.
    One of the fundamental statistical methods used by econometricians is regression analysis. For an overview of a
    linear implementation of this framework, see linear regression. Regression methods are important in econometrics
    because economists typically cannot use controlled experiments. Econometricians often seek illuminating natural
    experiments in the absence of evidence from controlled experiments. Observational data may be subject to
    omitted-variable bias and a list of other problems that must be addressed using causal analysis of
    simultaneous-equation models.[19]
Econometrics                                                                                                                   4


    Experimental economics
    In recent decades, econometricians have increasingly turned to use of experiments to evaluate the often-contradictory
    conclusions of observational studies. Here, controlled and randomized experiments provide statistical inferences that
    may yield better empirical performance than do purely observational studies.[20]


    Data
    Data sets to which econometric analyses are applied can be classified as time-series data, cross-sectional data, panel
    data, and multidimensional panel data. Time-series data sets contain observations over time; for example, inflation
    over the course of several years. Cross-sectional data sets contain observations at a single point in time; for example,
    many individuals' incomes in a given year. Panel data sets contain both time-series and cross-sectional observations.
    Multi-dimensional panel data sets contain observations across time, cross-sectionally, and across some third
    dimension. For example, the Survey of Professional Forecasters contains forecasts for many forecasters
    (cross-sectional observations), at many points in time (time series observations), and at multiple forecast horizons (a
    third dimension).


    Instrumental variables
    In many econometric contexts, the commonly-used ordinary least squares method may not recover the theoretical
    relation desired or may produce estimates with poor statistical properties, because the assumptions for valid use of
    the method are violated. One widely-used remedy is the method of instrumental variables (IV). For an economic
    model described by more than one equation, simultaneous-equation methods may be used to remedy similar
    problems, including two IV variants, Two-Stage Least Squares (2SLS), and Three-Stage Least Squares (3SLS).[21]


    Computational methods
    Computational concerns are important for evaluating econometric methods and for use in decision making.[22] Such
    concerns include mathematical well-posedness: the existence, uniqueness, and stability of any solutions to
    econometric equations. Another concern is the numerical efficiency and accuracy of software.[23] A third concern is
    also the usability of econometric software.[24]


    Example
    A simple example of a relationship in econometrics from the field of labor economics is:


    This example assumes that the natural logarithm of a person's wage is a linear function of (among other things) the
    number of years of education that person has acquired. The parameter    measures the increase in the natural log of
    the wage attributable to one more year of education. The term is a random variable representing all other factors
    that may have direct influence on wage. The econometric goal is to estimate the parameters,                under
    specific assumptions about the random variable . For example, if is uncorrelated with years of education, then
    the equation can be estimated with ordinary least squares.
    If the researcher could randomly assign people to different levels of education, the data set thus generated would
    allow estimation of the effect of changes in years of education on wages. In reality, those experiments cannot be
    conducted. Instead, the econometrician observes the years of education of and the wages paid to people who differ
    along many dimensions. Given this kind of data, the estimated coefficient on Years of Education in the equation
    above reflects both the effect of education on wages and the effect of other variables on wages, if those other
    variables were correlated with education. For example, people born in certain places may have higher wages and
    higher levels of education. Unless the econometrician controls for place of birth in the above equation, the effect of
    birthplace on wages may be falsely attributed to the effect of education on wages.
Econometrics                                                                                                                                       5


    The most obvious way to control for birthplace is to include a measure of the effect of birthplace in the equation
    above. Exclusion of birthplace, together with the assumption that is uncorrelated with education produces a
    misspecified model. Another technique is to include in the equation additional set of measured covariates which are
    not instrumental variables, yet render   identifiable.[25] An overview of econometric methods used to study this
    problem can be found in Card (1999).[26]


    Journals
    The main journals which publish work in econometrics are Econometrica, the Journal of Econometrics, the Review
    of Economics and Statistics, Econometric Theory, the Journal of Applied Econometrics, Econometric Reviews, the
    Econometrics Journal,[27] Applied Econometrics and International Development, the Journal of Business &
    Economic Statistics, and the Journal of Economic and Social Measurement [28].


    Limitations and criticisms
                   See also Criticisms of econometrics
    Like other forms of statistical analysis, badly specified econometric models may show a spurious correlation where
    two variables are correlated but causally unrelated. In a study of the use of econometrics in major economics
    journals, McCloskey concluded that economists report p values (following the Fisherian tradition of tests of
    significance of point null-hypotheses), neglecting concerns of type II errors; economists fail to report estimates of the
    size of effects (apart from statistical significance) and to discuss their economic importance. Economists also fail to
    use economic reasoning for model selection, especially for deciding which variables to include in a regression.[29][30]
    In some cases, economic variables cannot be experimentally manipulated as treatments randomly assigned to
    subjects.[31] In such cases, economists rely on observational studies, often using data sets with many strongly
    associated covariates, resulting in enormous numbers of models with similar explanatory ability but different
    covariates and regression estimates. Regarding the plurality of models compatible with observational data-sets,
    Edward Leamer urged that "professionals ... properly withhold belief until an inference can be shown to be
    adequately insensitive to the choice of assumptions".[32]
    Economists from the Austrian School argue that aggregate economic models are not well suited to describe
    economic reality because they waste a large part of specific knowledge. Friedrich Hayek in his The Use of
    Knowledge in Society argued that "knowledge of the particular circumstances of time and place" is not easily
    aggregated and is often ignored by professional economists.[33][34]


    Notes
    [1] M. Hashem Pesaran (1987). "Econometrics," The New Palgrave: A Dictionary of Economics, v. 2, p. 8 [pp. 8-22]. Reprinted in J. Eatwell et
        al., eds. (1990). Econometrics: The New Palgrave, p. 1 (http:/ / books. google. com/ books?id=gBsgr7BPJsoC& dq=econometrics&
        printsec=find& pg=PA1=false#v=onepage& q& f=false) [pp. 1-34]. Abstract (http:/ / www. dictionaryofeconomics. com/
        article?id=pde2008_E000007& edition=current& q=Econometrics& topicid=& result_number=2) (2008 revision by J. Geweke, J. Horowitz,
        and H. P. Pesaran).
    [2] P. A. Samuelson, T. C. Koopmans, and J. R. N. Stone (1954). "Report of the Evaluative Committee for Econometrica," Econometrica 22(2),
        p. 142. [p p. 141 (http:/ / www. jstor. org/ pss/ 1907538)-146], as described and cited in Pesaran (1987) above.
    [3] Paul A. Samuelson and William D. Nordhaus, 2004. Economics. 18th ed., McGraw-Hill, p. 5.
    [4] • H. P. Pesaran (1990), "Econometrics," Econometrics: The New Palgrave, p. 2 (http:/ / books. google. com/ books?id=gBsgr7BPJsoC&
        dq=econometrics& printsec=find& pg=PA2=false#v=onepage& q& f=false), citing Ragnar Frisch (1936), "A Note on the Term
        'Econometrics'," Econometrica, 4(1), p. 95.
           • Aris Spanos (2008), "statistics and economics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_S000502& edition=current& q=statistics& topicid=& result_number=1)
    [5] Greene, 1.
    [6] Geweke, Horowitz & Pesaran 2008
    [7] Greene (2012), 12.
Econometrics                                                                                                                                           6

    [8] Kennedy 2003, p. 110
    [9] Kennedy 2003, p. 129
    [10] Kennedy 2003, p. 131
    [11] Kennedy 2003, p. 130
    [12] Kennedy 2003, p. 133
    [13] Greene 2012, p. 23-note
    [14] Greene 2010, p. 22
    [15] Kennedy 2003, p. 135
    [16] Kennedy 2003, p. 205
    [17] Clive Granger (2008). "forecasting," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_F000161& edition=current& q=forecast& topicid=& result_number=7)
    [18] Herman O. Wold (1969). "Econometrics as Pioneering in Nonexperimental Model Building," Econometrica, 37(3), pp. 369 (http:/ / www.
        jstor. org/ pss/ 1912787)-381.
    [19] Edward E. Leamer (2008). "specification problems in econometrics," The New Palgrave Dictionary of Economics. Abstract. (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_S000200& edition=current& q=Specification problems in econometrics& topicid=&
        result_number=1)
    [20] • H. Wold 1954. "Causality and Econometrics," Econometrica, 22(2), p p. 162 (http:/ / www. jstor. org/ pss/ 1907540)-177.
           • Kevin D. Hoover (2008). "causality in economics and econometrics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract
        (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000569& q=experimental methods in economics& topicid=&
        result_number=11) and galley proof. (http:/ / econ. duke. edu/ ~kdh9/ Source Materials/ Research/ Palgrave_Causality_Final. pdf)
    [21] Peter Kennedy (economist) (2003). A Guide to Econometrics, 5th ed. Description (http:/ / mitpress. mit. edu/ catalog/ item/ default.
        asp?ttype=2& tid=9577), preview (http:/ / books. google. com/ books?id=B8I5SP69e4kC& printsec=find& pg=PR5=gbs_atb#v=onepage&
        q& f=false), and TOC (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?ttype=2& tid=9577& mode=toc), ch. 9, 10, 13, and 18.
    [22] • Keisuke Hirano (2008). "decision theory in econometrics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ /
        www. dictionaryofeconomics. com/ article?id=pde2008_D000244& edition=current& q=Computational economics& topicid=&
        result_number=19).
           • James O. Berger (2008). "statistical decision theory," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_S000251& edition=& field=keyword& q=statistical decision theory& topicid=&
        result_number=1)
    [23] B. D. McCullough and H. D. Vinod (1999). "The Numerical Reliability of Econometric Software," Journal of Economic Literature, 37(2),
        pp. 633-665 (http:/ / www. pages. drexel. edu/ ~bdm25/ jel. pdf).
    [24] • Vassilis A. Hajivassiliou (2008). "computational methods in econometrics," The New Palgrave Dictionary of Economics, 2nd Edition.
        Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000559& edition=current& q=& result_number=1)
           • Richard E. Quandt (1983). "Computational Problems and Methods," ch. 12, in Handbook of Econometrics, v. 1, pp. 699 (http:/ / www.
        sciencedirect. com/ science/ article/ pii/ S1573441283010168)-764.
           • Ray C. Fair (1996). "Computational Methods for Macroeconometric Models," Handbook of Computational Economics, v. 1, pp. (http:/ /
        www. sciencedirect. com/ science/ article/ pii/ S1574002196010052143)-169.
    [25] Judea Pearl (2000). Causality: Model, Reasoning, and Inference, Cambridge University Press.
    [26] David Card (1999) "The Causal Effect of Education on Earning," in Ashenfelter, O. and Card, D., (eds.) Handbook of Labor Economics, pp
        1801-63.
    [27] http:/ / www. wiley. com/ bw/ journal. asp?ref=1368-4221
    [28] http:/ / www. iospress. nl/ html/ 07479662. php
    [29] McCloskey (May 1985). "The Loss Function has been mislaid: the Rhetoric of Significance Tests". American Economic Review 75 (2).
    [30] Stephen T. Ziliak and Deirdre N. McCloskey (2004). "Size Matters: The Standard Error of Regressions in the American Economic Review,"
        Journal of Socio-economics, 33(5), pp. 527-46 (http:/ / faculty. roosevelt. edu/ Ziliak/ doc/ Size Matters Journal of Socio-Economics Ziliak
        and McCloskey. pdf) (press +).
    [31] Leamer, Edward (March 1983). "Let's Take the Con out of Econometrics" (http:/ / www. jstor. org/ pss/ 1803924). American Economic
        Review 73 (1): 34. .
    [32] Leamer, Edward (March 1983). "Let's Take the Con out of Econometrics" (http:/ / www. jstor. org/ pss/ 1803924). American Economic
        Review 73 (1): 43. .
    [33] Robert F. Garnett. What Do Economists Know? New Economics of Knowledge. Routledge, 1999. ISBN 978-0-415-15260-0. p. 170
    [34] G. M. P. Swann. Putting Econometrics in Its Place: A New Direction in Applied Economics. Edward Elgar Publishing, 2008. ISBN
        978-1-84720-776-0. p. 62-64
Econometrics                                                                                                             7


    References
    • Handbook of Econometrics Elsevier. Links to volume chapter-preview links:
      Zvi Griliches and Michael D. Intriligator, ed. (1983). v. 1 (http://www.sciencedirect.com/science/handbooks/
      15734412/1); (1984), v. 2 (http://www.sciencedirect.com/science/handbooks/15734412/2); (1986),
      description (http://www.elsevier.com/wps/find/bookdescription.cws_home/601080/
      description#description), v. 3 (http://www.sciencedirect.com/science/handbooks/15734412/3); (1994),
      description (http://www.elsevier.com/wps/find/bookdescription.cws_home/601081/
      description#description), v. 4 (http://www.sciencedirect.com/science/handbooks/15734412/4)
      Robert F. Engle and Daniel L. McFadden, ed. (2001). Description (http://www.elsevier.com/wps/find/
      bookdescription.cws_home/601082/description#description), v. 5 (http://www.sciencedirect.com/science/
      handbooks/15734412/5)
      James J. Heckman and Edward E. Leamer, ed. (2007). Description (http://www.elsevier.com/wps/find/
      bookdescription.cws_home/712946/description#description), v. 6A (http://www.sciencedirect.com/science/
      handbooks/15734412/6/part/PA) & v. 6B (http://www.sciencedirect.com/science/handbooks/15734412/6/
      part/PB)
    • Handbook of Statistics, v. 11, Econometrics (1993), Elsevier. Links to first-page chapter previews. (http://www.
      sciencedirect.com/science/handbooks/01697161/11)
    • International Encyclopedia of the Social & Behavioral Sciences (2001), Statistics, "Econometrics and Time
      Series," links (http://www.sciencedirect.com/science?_ob=RefWorkIndexURL&_idxType=SC&
      _cdi=23486&_refWorkId=21&_explode=151000377,151000380&_alpha=&_acct=C000050221&
      _version=1&_userid=10&md5=10d43da5ed3104bf3d8bb99f72c80e11&refID=151000380#151000380) to
      first-page previews of 21 articles.
    • Angrist, Joshua & Pischke, Jörn‐Steffen (2010). "The Credibility Revolution in Empirical Economics: How
      Better Research Design Is Taking the Con out of Econometrics], 24(2), , pp. 3–30. Abstract. (http://www.
      ingentaconnect.com/content/aea/jep/2010/00000024/00000002/art00001)
    • Eatwell, John, et al., eds. (1990). Econometrics: The New Palgrave. Article-preview links (http://books.google.
      com/books?id=gBsgr7BPJsoC&dq=econometrics&printsec=find&pg=PR5=false#v=onepage&q&f=false)
      (from The New Palgrave: A Dictionary of Economics, 1987).
    • Geweke, John; Horowitz, Joel; Pesaran, Hashem (2008). Durlauf, Steven N.; Blume, Lawrence E.. eds.
      "Econometrics" (http://www.dictionaryofeconomics.com.proxyau.wrlc.org/article?id=pde2008_E000007).
      The New Palgrave Dictionary of Economics (Palgrave Macmillan). doi:10.1057/9780230226203.0425.
    • Greene, William H. (1999, 4th ed.) Econometric Analysis, Prentice Hall.
    • Hayashi, Fumio. (2000) Econometrics, Princeton University Press. ISBN 0-691-01018-8 Description and contents
      links. (http://press.princeton.edu/titles/6946.html)
    • Hamilton, James D. (1994) Time Series Analysis, Princeton University Press. Description (http://press.
      princeton.edu/titles/5386.html) and preview. (http://books.google.com/books/p/
      princeton?id=B8_1UBmqVUoC&printsec=frontcover&cd=1&source=gbs_ViewAPI&hl=en#v=onepage&q&
      f=false)
    • Hughes Hallett, Andrew J. "Econometrics and the Theory of Economic Policy: The Tinbergen-Theil
      Contributions 40 Years On," Oxford Economic Papers (1989) 41#1 pp 189-214
    • Kelejian, Harry H., and Wallace E. Oates (1989, 3rd ed.) Introduction to Econometrics.
    • Kennedy, Peter (2003). A guide to econometrics. Cambridge, Mass: MIT Press. ISBN 978-0-262-61183-1.
    • Russell Davidson and James G. MacKinnon (2004). Econometric Theory and Methods. New York: Oxford
      University Press. Description. (http://www.oup.com/us/catalog/general/subject/Economics/Econometrics/
      ~~/dmlldz11c2EmY2k9OTc4MDE5NTEyMzcyMg==?view=usa&ci=9780195123722#Description)
    • Mills, Terence C., and Kerry Patterson, ed. Palgrave Handbook of Econometrics:
Econometrics                                                                                                              8


          (2007) v. 1: Econometric Theoryv. 1. Links (http:/ / www. palgrave. com/ products/ title. aspx?pid=269866) to
          description and contents.
          (2009) v. 2, Applied Econometrics. Palgrave Macmillan. ISBN 978-1-4039-1799-7 Links (http:/ / www.
          palgrave.com/products/title.aspx?PID=267962) to description and contents.
    • Pearl, Judea (2009, 2nd ed.). Causality: Models, Reasoning and Inference, Cambridge University Press,
      Description (http://books.google.com/books?id=wnGU_TsW3BQC&source=gbs_navlinks_s), TOC (http://
      bayes.cs.ucla.edu/BOOK-09/book-toc-final.pdf), and preview, ch. 1-10 (http://books.google.com/
      books?id=wnGU_TsW3BQC&printsec=find&pg=PR7=gbs_atb#v=onepage&q&f=false) and ch. 11 (http://
      bayes.cs.ucla.edu/BOOK-09/ch11-toc-plus-p331-final.pdf). 5 economics-journal reviews (http://bayes.cs.
      ucla.edu/BOOK-2K/), including Kevin D. Hoover, Economics Journal.
    • Pindyck, Robert S., and Daniel L. Rubinfeld (1998, 4th ed.). Econometric Methods and Economic Forecasts,
      McGraw-Hill.
    • Studenmund, A.H. (2011, 6th ed.). Using Econometrics: A Practical Guide. Contents (chapter-preview) links.
      (http://www.coursesmart.com/9780131367760/chap01)
    • Wooldridge, Jeffrey (2003). Introductory Econometrics: A Modern Approach. Mason: Thomson South-Western.
      ISBN 0-324-11364-1 Chapter-preview links in brief (http://books.google.com/books?id=64vt5TDBNLwC&
      printsec=find&pg=PR3=gbs_atb#v=onepage&q&f=false) and detail. (http://books.google.com/
      books?id=64vt5TDBNLwC&printsec=find&pg=PR4=gbs_atb#v=onepage&q&f=false)


    Further reading
    • Econometric Theory book on Wikibooks
    • Giovanini, Enrico Understanding Economic Statistics (http://www.oecd.org/statistics/
      understandingeconomicstatistics), OECD Publishing, 2008, ISBN 978-92-64-03312-2


    External links
    • Econometric Society (http://www.econometricsociety.org)
    • The Econometrics Journal (http://www.ectj.org)
    • Econometric Links (http://www.econometriclinks.com)
    • Teaching Econometrics (http://www.economicsnetwork.ac.uk/subjects/econometrics.htm) (Index by the
      Economics Network (UK))
    • Applied Econometric Association (http://www.aea-eu.com)
    • The Society for Financial Econometrics (http://sofie.stern.nyu.edu/)
Mathematical economics                                                                                                         9



    Mathematical economics
    Mathematical economics is the application of mathematical methods to represent theories and analyze problems in
    economics. An advantage claimed for the approach is its allowing formulation of theoretical relationships with rigor,
    generality, and simplicity.[1] By convention, the applied methods refer to those beyond simple geometry, such as
    differential and integral calculus, difference and differential equations, matrix algebra, mathematical programming,
    and other computational methods.[2][3]
    It is argued that mathematics allows economists to form meaningful, testable propositions about wide-ranging and
    complex subjects which could less easily be expressed informally. Further, the language of mathematics allows
    economists to make specific, positive claims about controversial or contentious subjects that would be impossible
    without mathematics.[4] Much of economic theory is currently presented in terms of mathematical economic models,
    a set of stylized and simplified mathematical relationships asserted to clarify assumptions and implications.[5]
    Broad applications include:
    • optimization problems as to goal equilibrium, whether of a household, business firm, or policy maker
    • static (or equilibrium) analysis in which the economic unit (such as a household) or economic system (such as a
      market or the economy) is modeled as not changing
    • comparative statics as to a change from one equilibrium to another induced by a change in one or more factors
    • dynamic analysis, tracing changes in an economic system over time, for example from economic growth.[3][6][7]
    Formal economic modeling began in the 19th century with the use of differential calculus to represent and explain
    economic behavior, such as utility maximization, an early economic application of mathematical optimization.
    Economics became more mathematical as a discipline throughout the first half of the 20th century, but introduction
    of new and generalized techniques in the period around the Second World War, as in game theory, would greatly
    broaden the use of mathematical formulations in economics.[8][7]
    This rapid systematizing of economics alarmed critics of the discipline as well as some noted economists. John
    Maynard Keynes, Robert Heilbroner, Friedrich Hayek and others have criticized the broad use of mathematical
    models for human behavior, arguing that some human choices are irreducible to mathematics.


    History
    The use of mathematics in the service of social and economic analysis dates back to the 17th century. Then, mainly
    in German universities, a style of instruction emerged which dealt specifically with detailed presentation of data as it
    related to public administration. Gottfried Achenwall lectured in this fashion, coining the term statistics. At the same
    time, a small group of professors in England established a method of "reasoning by figures upon things relating to
    government" and referred to this practice as Political Arithmetick.[9] Sir William Petty wrote at length on issues that
    would later concern economists, such as taxation, Velocity of money and national income, but while his analysis was
    numerical, he rejected abstract mathematical methodology. Petty's use of detailed numerical data (along with John
    Graunt) would influence statisticians and economists for some time, even though Petty's works were largely ignored
    by English scholars.[10]
    The mathematization of economics began in earnest in the 19th century. Most of the economic analysis of the time
    was what would later be called classical economics. Subjects were discussed and dispensed with through algebraic
    means, but calculus was not used. More importantly, until Johann Heinrich von Thünen's The Isolated State in 1826,
    economists did not develop explicit and abstract models for behavior in order to apply the tools of mathematics.
    Thünen's model of farmland use represents the first example of marginal analysis.[11] Thünen's work was largely
    theoretical, but he also mined empirical data in order to attempt to support his generalizations. In comparison to his
    contemporaries, Thünen built economic models and tools, rather than applying previous tools to new problems.[12]
Mathematical economics                                                                                                                    10


    Meanwhile a new cohort of scholars trained in the mathematical methods of the physical sciences gravitated to
    economics, advocating and applying those methods to their subject,[13] and described today as moving from
    geometry to mechanics.[14] These included W.S. Jevons who presented paper on a "general mathematical theory of
    political economy" in 1862, providing an outline for use of the theory of marginal utility in political economy.[15] In
    1871, he published The Principles of Political Economy, declaring that the subject as science "must be mathematical
    simply because it deals with quantities." Jevons expected the only collection of statistics for price and quantities
    would permit the subject as presented to become an exact science.[16] Others preceded and followed in expanding
    mathematical representations of economic problems.


    Marginalists and the roots of neoclassical economics
    Augustin Cournot and Léon Walras built the tools
    of the discipline axiomatically around utility,
    arguing that individuals sought to maximize their
    utility across choices in a way that could be
    described mathematically.[17] At the time, it was
    thought that utility was quantifiable, in units
    known as utils.[18] Cournot, Walras and Francis
    Ysidro Edgeworth are considered the precursors
    to modern mathematical economics.[19]


    Augustin Cournot

    Cournot, a professor of Mathematics, developed a
    mathematical treatment in 1838 for duopoly—a
    market condition defined by competition between
    two sellers.[19] This treatment of competition, first
    published in Researches into the Mathematical
    Principles of Wealth,[20] is referred to as Cournot
                                                             Equilibrium quantities as a solution to two reaction functions in Cournot
    duopoly. It is assumed that both sellers had equal      duopoly. Each reaction function is expressed as a linear equation dependent
    access to the market and could produce their                                    upon quantity demanded.
    goods without cost. Further, it assumed that both
    goods were homogeneous. Each seller would vary her output based on the output of the other and the market price
    would be determined by the total quantity supplied. The profit for each firm would be determined by multiplying
    their output and the per unit Market price. Differentiating the profit function with respect to quantity supplied for
    each firm left a system of linear equations, the simultaneous solution of which gave the equilibrium quantity, price
    and profits.[21] Cournot's contributions to the mathematization of economics would be neglected for decades, but
    eventually influenced many of the marginalists.[21][22] Cournot's models of duopoly and Oligopoly also represent one
    of the first formulations of non-cooperative games. Today the solution can be given as a Nash equilibrium but
    Cournot's work preceded modern Game theory by over 100 years.[23]


    Léon Walras
    While Cournot provided a solution for what would later be called partial equilibrium, Léon Walras attempted to
    formalize discussion of the economy as a whole through a theory of general competitive equilibrium. The behavior
    of every economic actor would be considered on both the production and consumption side. Walras originally
    presented four separate models of exchange, each recursively included in the next. The solution of the resulting
    system of equations (both linear and non-linear) is the general equilibrium.[24] At the time, no general solution could
    be expressed for a system of arbitrarily many equations, but Walras's attempts produced two famous results in
Mathematical economics                                                                                                         11


    economics. The first is Walras' law and the second is the principle of tâtonnement. Walras' method was considered
    highly mathematical for the time and Edgeworth commented at length about this fact in his review of Éléments
    d'économie politique pure (Elements of Pure Economics).[25]
    Walras' law was introduced as a theoretical answer to the problem of determining the solutions in general
    equilibrium. His notation is different from modern notation but can be constructed using more modern summation
    notation. Walras assumed that in equilibrium, all money would be spent on all goods: every good would be sold at
    the market price for that good and every buyer would expend their last dollar on a basket of goods. Starting from this
    assumption, Walras could then show that if there were n markets and n-1 markets cleared (reached equilibrium
    conditions) that the nth market would clear as well. This is easiest to visualize with two markets (considered in most
    texts as a market for goods and a market for money). If one of two markets has reached an equilibrium state, no
    additional goods (or conversely, money) can enter or exit the second market, so it must be in a state of equilibrium as
    well. Walras used this statement to move toward a proof of existence of solutions to general equilibrium but it is
    commonly used today to illustrate market clearing in money markets at the undergraduate level.[26]
    Tâtonnement (roughly, French for groping toward) was meant to serve as the practical expression of Walrasian
    general equilibrium. Walras abstracted the marketplace as an auction of goods where the auctioneer would call out
    prices and market participants would wait until they could each satisfy their personal reservation prices for the
    quantity desired (remembering here that this is an auction on all goods, so everyone has a reservation price for their
    desired basket of goods).[27]
    Only when all buyers are satisfied with the given market price would transactions occur. The market would "clear" at
    that price—no surplus or shortage would exist. The word tâtonnement is used to describe the directions the market
    takes in groping toward equilibrium, settling high or low prices on different goods until a price is agreed upon for all
    goods. While the process appears dynamic, Walras only presented a static model, as no transactions would occur
    until all markets were in equilibrium. In practice very few markets operate in this manner.[28]

    Francis Ysidro Edgeworth
    Edgeworth introduced mathematical elements to Economics explicitly in Mathematical Psychics: An Essay on the
    Application of Mathematics to the Moral Sciences, published in 1881.[29] He adopted Jeremy Bentham's felicific
    calculus to economic behavior, allowing the outcome of each decision to be converted into a change in utility.[30]
    Using this assumption, Edgeworth built a model of exchange on three assumptions: individuals are self-interested,
    individuals act to maximize utility, and individuals are "free to recontract with another independently of...any third
    party."[31]
Mathematical economics                                                                                                                     12


    Given two individuals, the set of
    solutions where the both individuals
    can maximize utility is described by
    the contract curve on what is now
    known as an Edgeworth Box.
    Technically, the construction of the
    two-person solution to Edgeworth's
    problem was not developed graphically
    until 1924 by Arthur Lyon Bowley.[33]
    The contract curve of the Edgeworth
    box (or more generally on any set of
    solutions to Edgeworth's problem for
    more actors) is referred to as the core
    of an economy.[34]

    Edgeworth devoted considerable effort
                                                    An Edgeworth box displaying the contract curve an economy with two participants.
    to insisting that mathematical proofs         Referred to as the "core" of the economy in modern parlance, there are infinitely many
    were appropriate for all schools of                                                                                   [32]
                                                            solutions along the curve for economies with two participants
    thought in economics. While at the
    helm of The Economic Journal, he published several articles criticizing the mathematical rigor of rival researchers,
    including Edwin Robert Anderson Seligman, a noted skeptic of mathematical economics.[35] The articles focused on
    a back and forth over tax incidence and responses by producers. Edgeworth noticed that a monopoly producing a
    good that had jointness of supply but not jointness of demand (such as first class and economy on an airplane, if the
    plane flies, both sets of seats fly with it) might actually lower the price seen by the consumer for one of the two
    commodities if a tax were applied. Common sense and more traditional, numerical analysis seemed to indicate that
    this was preposterous. Seligman insisted that the results Edgeworth achieved were a quirk of his mathematical
    formulation. He suggested that the assumption of a continuous demand function and an infinitesimal change in the
    tax resulted in the paradoxical predictions. Harold Hotelling later showed that Edgeworth was correct and that the
    same result (a "diminution of price as a result of the tax") could occur with a discontinuous demand function and
    large changes in the tax rate).[36]


    Modern mathematical economics
    From the later-1930s, an array of new mathematical tools from the differential calculus and differential equations,
    convex sets, and graph theory were deployed to advance economic theory in a way similar to new mathematical
    methods earlier applied to physics.[8][37] The process was later described as moving from mechanics to
    axiomatics.[38]


    Differential calculus
    Vilfredo Pareto analyzed microeconomics by treating decisions by economic actors as attempts to change a given
    allotment of goods to another, more preferred allotment. Sets of allocations could then be treated as Pareto efficient
    (Pareto optimal is an equivalent term) when no exchanges could occur between actors that could make at least one
    individual better off without making any other individual worse off.[39] Pareto's proof is commonly conflated with
    Walrassian equilibrium or informally ascribed to Adam Smith's Invisible hand hypothesis.[40] Rather, Pareto's
    statement was the first formal assertion of what would be known as the first fundamental theorem of welfare
    economics.[41] These models lacked the inequalities of the next generation of mathematical economics.
Mathematical economics                                                                                                       13


    In the landmark treatise Foundations of Economic Analysis (1947), Paul Samuelson identified a common paradigm
    and mathematical structure across multiple fields in the subject, building on previous work by Alfred Marshall.
    Foundations took mathematical concepts from physics and applied them to economic problems. This broad view (for
    example, comparing Le Chatelier's principle to tâtonnement) drives the fundamental premise of mathematical
    economics: systems of economic actors may be modeled and their behavior described much like any other system.
    This extension followed on the work of the marginalists in the previous century and extended it significantly.
    Samuelson approached the problems of applying individual utility maximization over aggregate groups with
    comparative statics, which compares two different equilibrium states after an exogenous change in a variable. This
    and other methods in the book provided the foundation for mathematical economics in the 20th century.[7][42]


    Linear models
    Restricted models of general equilibrium were formulated by John von Neumann in 1937.[43] Unlike earlier versions,
    the models of von Neumann had inequality constraints. For his model of an expanding economy, von Neumann
    proved the existence and uniqueness of an equilibrium using his generalization of Brouwer's fixed point theorem.
    Von Neumann's model of an expanding economy considered the matrix pencil  A - λ B with nonnegative matrices A
    and B; von Neumann sought probability vectors p and q and a positive number λ that would solve the
    complementarity equation
          pT (A - λ B) q = 0,
    along with two inequality systems expressing economic efficiency. In this model, the (transposed) probability vector
    p represents the prices of the goods while the probability vector q represents the "intensity" at which the production
    process would run. The unique solution λ represents the rate of growth of the economy, which equals the interest
    rate. Proving the existence of a positive growth rate and proving that the growth rate equals the interest rate were
    remarkable achievements, even for von Neumann.[44][45][46] Von Neumann's results have been viewed as a special
    case of linear programming, where von Neumann's model uses only nonnegative matrices.[47] The study of von
    Neumann's model of an expanding economy continues to interest mathematical economists with interests in
    computational economics.[48][49][50]

    Input-output economics
    In 1936, the Russian–born economist Wassily Leontief built his model of input-output analysis from the 'material
    balance' tables constructed by Soviet economists, which themselves followed earlier work by the physiocrats. With
    his model, which described a system of production and demand processes, Leontief described how changes in
    demand in one economic sector would influence production in another.[51] In practice, Leontief estimated the
    coefficients of his simple models, to address economically interesting questions. In production economics, "Leontief
    technologies" produce outputs using constant proportions of inputs, regardless of the price of inputs, reducing the
    value of Leontief models for understanding economies but allowing their parameters to be estimated relatively
    easily. In contrast, the von Neumann model of an expanding economy allows for choice of techniques, but the
    coefficients must be estimated for each technology.[52][53]
Mathematical economics                                                                                                             14


    Mathematical optimization
    In mathematics, mathematical optimization (or optimization or
    mathematical programming) refers to the selection of a best element
    from some set of available alternatives.[54] In the simplest case, an
    optimization problem involves maximizing or minimizing a real
    function by selecting input values of the function and computing the
    corresponding values of the function. The solution process includes
    satisfying general necessary and sufficient conditions for optimality.
    For optimization problems, specialized notation may be used as to the
    function and its input(s). More generally, optimization includes finding
                                                                                Red dot in z direction as maximum for paraboloid
    the best available element of some function given a defined domain
                                                                                             function of (x, y) inputs
    and may use a variety of different computational optimization
    techniques.[55]

    Economics is closely enough linked to optimization by agents in an economy that an influential definition relatedly
    describes economics qua science as the "study of human behavior as a relationship between ends and scarce means"
    with alternative uses.[56] Optimization problems run through modern economics, many with explicit economic or
    technical constraints. In microeconomics, the utility maximization problem and its dual problem, the expenditure
    minimization problem for a given level of utility, are economic optimization problems.[57] Theory posits that
    consumers maximize their utility, subject to their budget constraints and that firms maximize their profits, subject to
    their production functions, input costs, and market demand.[58]

    Economic equilibrium is studied in optimization theory as a key ingredient of economic theorems that in principle
    could be tested against empirical data.[7][59] Newer developments have occurred in dynamic programming and
    modeling optimization with risk and uncertainty, including applications to portfolio theory, the economics of
    information, and search theory.[58]
    Optimality properties for an entire market system may be stated in mathematical terms, as in formulation of the two
    fundamental theorems of welfare economics[60] and in the Arrow–Debreu model of general equilibrium (also
    discussed below).[61] More concretely, many problems are amenable to analytical (formulaic) solution. Many others
    may be sufficiently complex to require numerical methods of solution, aided by software.[55] Still others are complex
    but tractable enough to allow computable methods of solution, in particular computable general equilibrium models
    for the entire economy.[62]
    Linear and nonlinear programming have profoundly affected microeconomics, which had earlier considered only
    equality constraints.[63] Many of the mathematical economists who received Nobel Prizes in Economics had
    conducted notable research using linear programming: Leonid Kantorovich, Leonid Hurwicz, Tjalling Koopmans,
    Kenneth J. Arrow, and Robert Dorfman, Paul Samuelson, and Robert Solow.[64] Both Kantorovich and Koopmans
    acknowledged that George B. Dantzig deserved to share their Nobel Prize for linear programming. Economists who
    conducted research in nonlinear programming also have won the Nobel prize, notably Ragnar Frisch in addition to
    Kantorovich, Hurwicz, Koopmans, Arrow, and Samuelson.
Mathematical economics                                                                                                        15


    Linear optimization
    Linear programming was developed to aid the allocation of resources in firms and in industries during the 1930s in
    Russia and during the 1940s in the United States. During the Berlin airlift (1948), linear programming was used to
    plan the shipment of supplies to prevent Berlin from starving after the Soviet blockade.[65][66]

    Nonlinear programming
    Extensions to nonlinear optimization with inequality constraints were achieved in 1951 by Albert W. Tucker and
    Harold Kuhn, who considered the nonlinear optimization problem:
          Minimize           (       ) subject to    (   ) ≤ 0 and      (     ) = 0 where
                                                    i                  j
                .
            ( ) is the function to be minimized
                 .
             ()(         = 1, ...,       ) are the functions of the         inequality constraints
            i
                     .
             j
                ()(      = 1, ...,     ) are the functions of the     equality constraints.
    In allowing inequality constraints, the Kuhn–Tucker approach generalized the classic method of Lagrange
    multipliers, which (until then) had allowed only equality constraints.[67] The Kuhn–Tucker approach inspired further
    research on Lagrangian duality, including the treatment of inequality constraints.[68][69] The duality theory of
    nonlinear programming is particularly satisfactory when applied to convex minimization problems, which enjoy the
    convex-analytic duality theory of Fenchel and Rockafellar; this convex duality is particularly strong for polyhedral
    convex functions, such as those arising in linear programming. Lagrangian duality and convex analysis are used
    daily in operations research, in the scheduling of power plants, the planning of production schedules for factories,
    and the routing of airlines (routes, flights, planes, crews).[69]

    Variational calculus and optimal control
    Economic dynamics allows for changes in economic variables over time, including in dynamic systems. The problem
    of finding optimal functions for such changes is studied in variational calculus and in optimal control theory. Before
    the Second World War, Frank Ramsey and Harold Hotelling used the calculus of variations to that end.
    Following Richard Bellman's work on dynamic programming and the 1962 English translation of L. Pontryagin et
    al.'s earlier work,[70] optimal control theory was used more extensively in economics in addressing dynamic
    problems, especially as to economic growth equilibrium and stability of economic systems,[71] of which a textbook
    example is optimal consumption and saving.[72] A crucial distinction is between deterministic and stochastic control
    models.[73] Other applications of optimal control theory include those in finance, inventories, and production for
    example.[74]

    Functional analysis
    It was in the course of proving of the existence of an optimal equilibrium in his 1937 model of economic growth that
    John von Neumann introduced functional analytic methods to include topology in economic theory, in particular,
    fixed-point theory through his generalization of Brouwer's fixed-point theorem.[8][43][75] Following von Neumann's
    program, Kenneth Arrow and Gérard Debreu formulated abstract models of economic equilibria using convex sets
    and fixed–point theory. In introducing the Arrow–Debreu model in 1954, they proved the existence (but not the
    uniqueness) of an equilibrium and also proved that every Walras equilibrium is Pareto efficient; in general, equilibria
    need not be unique.[76] In their models, the ("primal") vector space represented quantitites while the "dual" vector
    space represented prices.[77]
    In Russia, the mathematician Leonid Kantorovich developed economic models in partially ordered vector spaces,
    that emphasized the duality between quantities and prices.[78] Oppressed by communism, Kantorovich renamed
    prices as "objectively determined valuations" which were abbreviated in Russian as "o. o. o.", alluding to the
    difficulty of discussing prices in the Soviet Union.[77][79][80]
Mathematical economics                                                                                                       16


    Even in finite dimensions, the concepts of functional analysis have illuminated economic theory, particularly in
    clarifying the role of prices as normal vectors to a hyperplane supporting a convex set, representing production or
    consumption possibilities. However, problems of describing optimization over time or under uncertainty require the
    use of infinite–dimensional function spaces, because agents are choosing among functions or stochastic
    processes.[77][81][82][83]


    Differential decline and rise
    John von Neumann's work on functional analysis and topology in broke new ground in mathematics and economic
    theory.[43][84] It also left advanced mathematical economics with fewer applications of differential calculus. In
    particular, general equilibrium theorists used general topology, convex geometry, and optimization theory more than
    differential calculus, because the approach of differential calculus had failed to establish the existence of an
    equilibrium.
    However, the decline of differential calculus should not be exaggerated, because differential calculus has always
    been used in graduate training and in applications. Moreover, differential calculus has returned to the highest levels
    of mathematical economics, general equilibrium theory (GET), as practiced by the "GET-set" (the humorous
    designation due to Jacques H. Drèze). In the 1960s and 1970s, however, Gérard Debreu and Stephen Smale led a
    revival of the use of differential calculus in mathematical economics. In particular, they were able to prove the
    existence of a general equilibrium, where earlier writers had failed, because of their novel mathematics: Baire
    category from general topology and Sard's lemma from differential topology. Other economists asssociated with the
    use of differential analysis include Egbert Dierker, Andreu Mas-Colell, and Yves Balasko.[85][86] These advances
    have changed the traditional narrative of the history of mathematical economics, following von Neumann, which
    celebrated the abandonment of differential calculus.


    Game theory
    John von Neumann, working with Oskar Morgenstern on the theory of games, broke new mathematical ground in
    1944 by extending functional analytic methods related to convex sets and topological fixed-point theory to economic
    analysis.[8][84] Their work thereby avoided the traditional differential calculus, for which the maximum–operator did
    not apply to non-differentiable functions. Continuing von Neumann's work in cooperative game theory, game
    theorists Lloyd S. Shapley, Martin Shubik, Hervé Moulin, Nimrod Megiddo, Bezalel Peleg influenced economic
    research in politics and economics. For example, research on the fair prices in cooperative games and fair values for
    voting games led to changed rules for voting in legislatures and for accounting for the costs in public–works
    projects. For example, cooperative game theory was used in designing the water distribution system of Southern
    Sweden and for setting rates for dedicated telephone lines in the USA.
    Earlier neoclassical theory had bounded only the range of bargaining outcomes and in special cases, for example
    bilateral monopoly or along the contract curve of the Edgeworth box.[87] Von Neumann and Morgenstern's results
    were similarly weak. Following von Neumann's program, however, John Nash used fixed–point theory to prove
    conditions under which the bargaining problem and noncooperative games can generate a unique equilibrium
    solution.[88] Noncooperative game theory has been adopted as a fundamental aspect of experimental economics,[89]
    behavioral economics,[90] information economics,[91] industrial organization,[92] and political economy.[93] It has
    also given rise to the subject of mechanism design (sometimes called reverse game theory), which has private and
    public-policy applications as to ways of improving economic efficiency through incentives for information
    sharing.[94]
    In 1994, Nash, John Harsanyi, and Reinhard Selten received the Nobel Memorial Prize in Economic Sciences their
    work on non–cooperative games. Harsanyi and Selten were awarded for their work on repeated games. Later work
    extended their results to computational methods of modeling.[95]
Mathematical economics                                                                                                        17


    Agent-based computational economics
    Agent-based computational economics (ACE) as a named field is relatively recent, dating from about the 1990s as to
    published work. It studies economic processes, including whole economies, as dynamic systems of interacting agents
    over time. As such, it falls in the paradigm of complex adaptive systems.[96] In corresponding agent-based models,
    agents are not real people but "computational objects modeled as interacting according to rules" ... "whose
    micro-level interactions create emergent patterns" in space and time.[97] The rules are formulated to predict behavior
    and social interactions based on incentives and information. The theoretical assumption of mathematical
    optimization by agents markets is replaced by the less restrictive postulate of agents with bounded rationality
    adapting to market forces.[98]
    ACE models apply numerical methods of analysis to computer-based simulations of complex dynamic problems for
    which more conventional methods, such as theorem formulation, may not find ready use.[99] Starting from specified
    initial conditions, the computational economic system is modeled as evolving over time as its constituent agents
    repeatedly interact with each other. In these respects, ACE has been characterized as a bottom-up culture-dish
    approach to the study of the economy.[100] In contrast to other standard modeling methods, ACE events are driven
    solely by initial conditions, whether or not equilibria exist or are computationally tractable. ACE modeling, however,
    includes agent adaptation, autonomy, and learning.[101] It has a similarity to, and overlap with, game theory as an
    agent-based method for modeling social interactions.[95] Other dimensions of the approach include such standard
    economic subjects as competition and collaboration,[102] market structure and industrial organization,[103] transaction
    costs,[104] welfare economics[105] and mechanism design,[106] information and uncertainty,[107] and
    macroeconomics.[108][109]
    The method is said to benefit from continuing improvements in modeling techniques of computer science and
    increased computer capabilities. Issues include those common to experimental economics in general[110] and by
    comparison[111] and to development of a common framework for empirical validation and resolving open questions
    in agent-based modeling.[112] The ultimate scientific objective of the method has been described as "test[ing]
    theoretical findings against real-world data in ways that permit empirically supported theories to cumulate over time,
    with each researcher's work building appropriately on the work that has gone before."[113]
Mathematical economics                                                                                                                      18


    Mathematicization of economics
    Over the course of the 20th century, articles in
    "core journals"[115] in economics have been
    almost exclusively written by economists in
    academia. As a result, much of the material
    transmitted in those journals relates to economic
    theory, and "economic theory itself has been
    continuously          more      abstract      and
                    [116]
    mathematical."        A subjective assessment of
    mathematical techniques[117] employed in these
    core journals showed a decrease in articles that
    use neither geometric representations nor
    mathematical notation from 95% in 1892 to 5.3%
    in 1990.[118] A 2007 survey of ten of the top
    economic journals finds that only 5.8% of the
    articles published in 2003 and 2004 both lacked
    statistical analysis of data and lacked displayed
                                                            The surface of the Volatility smile is a 3-D surface whereby the current
    mathematical expressions that were indexed with
                                                           market implied volatility (Z-axis) for all options on the underlier is plotted
    numbers at the margin of the page.[119]                      against strike price and time to maturity (X & Y-axes).
                                                                                                                           [114]




    Econometrics
    Between the world wars, advances in mathematical statistics and a cadre of mathematically trained economists led to
    econometrics, which was the name proposed for the discipline of advancing economics by using mathematics and
    statistics. Within economics, "econometrics" has often been used for statistical methods in economics, rather than
    mathematical economics. Statistical econometrics features the application of linear regression and time series
    analysis to economic data.
    Ragnar Frisch coined the word "econometrics" and helped to found both the Econometric Society in 1930 and the
    journal Econometrica in 1933.[120][121] A student of Frisch's, Trygve Haavelmo published The Probability Approach
    in Econometrics in 1944, where he asserted that precise statistical analysis could be used as a tool to validate
    mathematical theories about economic actors with data from complex sources.[122] This linking of statistical analysis
    of systems to economic theory was also promulgated by the Cowles Commission (now the Cowles Foundation)
    throughout the 1930s and 1940s.[123]


    Earlier work in econometrics
    The roots of modern econometrics can be traced to the American economist Henry L. Moore. Moore studied
    agricultural productivity and attempted to fit changing values of productivity for plots of corn and other crops to a
    curve using different values of elasticity. Moore made several errors in his work, some from his choice of models
    and some from limitations in his use of mathematics. The accuracy of Moore's models also was limited by the poor
    data for national accounts in the United States at the time. While his first models of production were static, in 1925
    he published a dynamic "moving equilibrium" model designed to explain business cycles—this periodic variation
    from overcorrection in supply and demand curves is now known as the cobweb model. A more formal derivation of
    this model was made later by Nicholas Kaldor, who is largely credited for its exposition.[124]
Mathematical economics                                                                                                                 19


    Application
    Much of classical economics can be
    presented in simple geometric terms or
    elementary       mathematical       notation.
    Mathematical       economics,       however,
    conventionally makes use of calculus and
    matrix algebra in economic analysis in order
    to make powerful claims that would be more
    difficult without such mathematical tools.
    These tools are prerequisites for formal
    study, not only in mathematical economics
    but in contemporary economic theory in
    general. Economic problems often involve
    so many variables that mathematics is the
    only practical way of attacking and solving
    them. Alfred Marshall argued that every
    economic problem which can be quantified,
    analytically expressed and solved, should be
    treated by means of mathematical work.[126]

    Economics has become increasingly                  The IS/LM model is a Keynesian macroeconomic model designed to make
                                                      predictions about the intersection of "real" economic activity (e.g. spending,
    dependent upon mathematical methods and
                                                   income, savings rates) and decisions made in the financial markets (Money supply
    the mathematical tools it employs have           and Liquidity preference). The model is no longer widely taught at the graduate
                                                                                                                            [125]
    become more sophisticated. As a result,               level but is common in undergraduate macroeconomics courses.
    mathematics has become considerably more
    important to professionals in economics and finance. Graduate programs in both economics and finance require
    strong undergraduate preparation in mathematics for admission and, for this reason, attract an increasingly high
    number of mathematicians. Applied mathematicians apply mathematical principles to practical problems, such as
    economic analysis and other economics-related issues, and many economic problems are often defined as integrated
    into the scope of applied mathematics.[17]

    This integration results from the formulation of economic problems as stylized models with clear assumptions and
    falsifiable predictions. This modeling may be informal or prosaic, as it was in Adam Smith's The Wealth of Nations,
    or it may be formal, rigorous and mathematical.
    Broadly speaking, formal economic models may be classified as stochastic or deterministic and as discrete or
    continuous. At a practical level, quantitative modeling is applied to many areas of economics and several
    methodologies have evolved more or less independently of each other.[127]
    • Stochastic models are formulated using stochastic processes. They model economically observable values over
      time. Most of econometrics is based on statistics to formulate and test hypotheses about these processes or
      estimate parameters for them. Between the World Wars, Herman Wold developed a representation of stationary
      stochastic processes in terms of autoregressive models and a determinist trend. Wold and Jan Tinbergen applied
      time-series analysis to economic data. Contemporary research on time series statistics consider additional
      formulations of stationary processes, such as autoregressive moving average models. More general models
      include autoregressive conditional heteroskedasticity (ARCH) models and generalized ARCH (GARCH) models.
    • Non-stochastic mathematical models may be purely qualitative (for example, models involved in some aspect of
      social choice theory) or quantitative (involving rationalization of financial variables, for example with hyperbolic
      coordinates, and/or specific forms of functional relationships between variables). In some cases economic
Mathematical economics                                                                                                          20


       predictions of a model merely assert the direction of movement of economic variables, and so the functional
       relationships are used only in a qualitative sense: for example, if the price of an item increases, then the demand
       for that item will decrease. For such models, economists often use two-dimensional graphs instead of functions.
    • Qualitative models are occasionally used. One example is qualitative scenario planning in which possible future
      events are played out. Another example is non-numerical decision tree analysis. Qualitative models often suffer
      from lack of precision.


    Criticisms and defences

    Adequacy of mathematics for qualitative and complicated economics
    Friedrich Hayek contended that the use of formal techniques projects a scientific exactness that does not
    appropriately account for informational limitations faced by real economic agents. [128]
    In an interview, the economic historian Robert Heilbroner stated:[129]
          I guess the scientific approach began to penetrate and soon dominate the profession in the past twenty to thirty
          years. This came about in part because of the "invention" of mathematical analysis of various kinds and,
          indeed, considerable improvements in it. This is the age in which we have not only more data but more
          sophisticated use of data. So there is a strong feeling that this is a data-laden science and a data-laden
          undertaking, which, by virtue of the sheer numerics, the sheer equations, and the sheer look of a journal page,
          bears a certain resemblance to science . . . That one central activity looks scientific. I understand that. I think
          that is genuine. It approaches being a universal law. But resembling a science is different from being a science.
    Heilbroner stated that "some/much of economics is not naturally quantitative and therefore does not lend itself to
    mathematical exposition."[130]


    Testing predictions of mathematical economics
    Philosopher Karl Popper discussed the scientific standing of economics in the 1940s and 1950s. He argued that
    mathematical economics suffered from being tautological. In other words, insofar that economics became a
    mathematical theory, mathematical economics ceased to rely on empirical refutation but rather relied on
    mathematical proofs and disproof.[131] According to Popper, falsifiable assumptions can be tested by experiment and
    observation while unfalsifiable assumptions can be explored mathematically for their consequences and for their
    consistency with other assumptions.[132]
    Sharing Popper's concerns about assumptions in economics generally, and not just mathematical economics, Milton
    Friedman declared that "all assumptions are unrealistic". Friedman proposed judging economic models by their
    predictive performance rather than by the match between their assumptions and reality.[133]


    Mathematical economics as a form of pure mathematics
    Considering mathematical economics, J.M. Keynes wrote in The General Theory:[134]
          It is a great fault of symbolic pseudo-mathematical methods of formalising a system of economic analysis ...
          that they expressly assume strict independence between the factors involved and lose their cogency and
          authority if this hypothesis is disallowed; whereas, in ordinary discourse, where we are not blindly
          manipulating and know all the time what we are doing and what the words mean, we can keep ‘at the back of
          our heads’ the necessary reserves and qualifications and the adjustments which we shall have to make later on,
          in a way in which we cannot keep complicated partial differentials ‘at the back’ of several pages of algebra
          which assume they all vanish. Too large a proportion of recent ‘mathematical’ economics are merely
          concoctions, as imprecise as the initial assumptions they rest on, which allow the author to lose sight of the
          complexities and interdependencies of the real world in a maze of pretentious and unhelpful symbols.
Mathematical economics                                                                                                        21


       Defense of mathematical economics
       In response to these criticisms, Paul Samuelson argued that mathematics is a language, repeating a thesis of Josiah
       Willard Gibbs. In economics, the language of mathematics is sometimes necessary for representing substantive
       problems. Moreover, mathematical economics has led to conceptual advances in economics.[135] In particular,
       Samuelson gave the example of microeconomics, writing that "few people are ingenious enough to grasp [its] more
       complex parts... without resorting to the language of mathematics, while most ordinary individuals can do so fairly
       easily with the aid of mathematics."[136]
       Some economists state that mathematical economics deserves support just like other forms of mathematics,
       particularly its neighbors in mathematical optimization and mathematical statistics and increasingly in theoretical
       computer science. Mathematical economics and other mathematical sciences have a history in which theoretical
       advances have regularly contributed to the reform of the more applied branches of economics. In particular,
       following the program of John von Neumann, game theory now provides the foundations for describing much of
       applied economics, from statistical decision theory (as "games against nature") and econometrics to general
       equilibrium theory and industrial organization. In the last decade, with the rise of the internet, mathematical
       economicists and optimization experts and computer scientists have worked on problems of pricing for on-line
       services --- their contributions using mathematics from cooperative game theory, nondifferentiable optimization, and
       combinatorial games.
       Robert M. Solow concluded that mathematical economics was the core "infrastructure" of contemporary economics:
             Economics is no longer a fit conversation piece for ladies and gentlemen. It has become a technical
             subject. Like any technical subject it attracts some people who are more interested in the technique than
             the subject. That is too bad, but it may be inevitable. In any case, do not kid yourself: the technical core
             of economics is indispensable infrastructure for the political economy. That is why, if you consult [a
             reference in contemporary economics] looking for enlightenment about the world today, you will be led
             to technical economics, or history, or nothing at all.[137]


       Mathematical economists
       Prominent mathematical economists include, but are not limited to, the following (by century of birth).


       19th century

   •     Enrico Barone                   •   Francis Ysidro Edgeworth       •   Irving Fisher    •   William Stanley Jevons
   •     Antoine Augustin Cournot



       20th century
Mathematical economics                                                                                                                                       22


   •     Charalambos D. Aliprantis             •   Nicholas Georgescu-Roegen              •   Andreu Mas-Colell              •   Leonard J. Savage
   •     R. G. D. Allen                        •   Roger Guesnerie                        •   Eric Maskin                    •   Herbert Scarf
   •     Maurice Allais                        •   Frank Hahn                             •   Nimrod Megiddo                 •   Reinhard Selten
   •     Kenneth J. Arrow                      •   John C. Harsanyi                       •   James Mirrlees                 •   Amartya Sen
   •     Robert J. Aumann                      •   John R. Hicks                          •   Roger Myerson                  •   Lloyd S. Shapley
   •     Yves Balasko                          •   Werner Hildenbrand                     •   John Forbes Nash, Jr.          •   Stephen Smale
   •     David Blackwell                       •   Harold Hotelling                       •   John von Neumann               •   Robert Solow
   •     Lawrence E. Blume                     •   Leonid Hurwicz                         •   Edward C. Prescott             •   Hugo F. Sonnenschein
   •     Graciela Chichilnisky                 •   Leonid Kantorovich                     •   Roy Radner                     •   Albert W. Tucker
   •     George B. Dantzig                     •   Tjalling Koopmans                      •   Frank Ramsey                   •   Hirofumi Uzawa
   •     Gérard Debreu                         •   David M. Kreps                         •   Donald John Roberts            •   Robert B. Wilson
   •     Jacques H. Drèze                      •   Harold W. Kuhn                         •   Paul Samuelson                 •   Hermann Wold
   •     David Gale                            •   Edmond Malinvaud                       •   Thomas Sargent                 •   Nicholas C. Yannelis



       Notes
       [1] Debreu, Gérard ([1987] 2008). "mathematical economics", section II, The New Palgrave Dictionary of Economics, 2nd Edition. Abstract.
           (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000107& edition=current& q=Mathematical economics& topicid=&
           result_number=1) Republished with revisions from 1986, "Theoretic Models: Mathematical Form and Economic Content", Econometrica,
           54(6), pp. 1259 (http:/ / www. jstor. org/ pss/ 1914299)-1270.
       [2] Elaborated at the JEL classification codes, Mathematical and quantitative methods JEL: C Subcategories.
       [3] Chiang, Alpha C.; and Kevin Wainwright (2005). Fundamental Methods of Mathematical Economics. McGraw-Hill Irwin. pp. 3–4.
           ISBN 0-07-010910-9. TOC. (http:/ / www. mhprofessional. com/ product. php?isbn=0070109109)
       [4] Varian, Hal (1997). "What Use Is Economic Theory?" in A. D'Autume and J. Cartelier, ed., Is Economics Becoming a Hard Science?,
           Edward Elgar. Pre-publication PDF. (http:/ / www. sims. berkeley. edu/ ~hal/ Papers/ theory. pdf) Retrieved 2008-04-01.
       [5] • As in Handbook of Mathematical Economics, 1st-page chapter links:
                Arrow, Kenneth J., and Michael D. Intriligator, ed., (1981), v. 1 (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL&
           _tockey=#TOC#24615#1981#999989999#565707#FLP#& _cdi=24615& _pubType=HS& _auth=y& _acct=C000050221& _version=1&
           _urlVersion=0& _userid=10& md5=01881ea3fe7d7990fed1c5b78d9f7be6)
                _____ (1982). v. 2 (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL&
           _tockey=#TOC#24615#1982#999979999#565708#FLP#& _cdi=24615& _pubType=HS& _auth=y& _acct=C000050221& _version=1&
           _urlVersion=0& _userid=10& md5=80dbd8f22c229a3640dc02e59ff80fe4)
                _____ (1986). v. 3 (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL&
           _tockey=#TOC#24615#1986#999969999#565709#FLP#& _cdi=24615& _pubType=HS& _auth=y& _acct=C000050221& _version=1&
           _urlVersion=0& _userid=10& md5=56f86ec2f0a1d2881e15bd1d0a45accd)
           Hildenbrand, Werner, and Hugo Sonnenschein, ed. (1991). v. 4. (http:/ / www. sciencedirect. com/ science/ handbooks/ 15734382)
              • Debreu, Gérard (1983). Mathematical Economics: Twenty Papers of Gérard Debreu, Contents (http:/ / books. google. com/
           books?id=wKJp6DepYncC& pg=PR7& source=bl& ots=mdhM3H6nCb& sig=zu38qa9R-3AiWqypwplcMibgdPo& hl=en&
           ei=etaJTu_XG6Lb0QG1nvXTAQ& sa=X& oi=book_result& ct=result& resnum=7& ved=0CF4Q6AEwBg#v=onepage& q& f=false).
              • Glaister, Stephen (1984). Mathematical Methods for Economists, 3rd ed., Blackwell. Contents. (http:/ / books. google. com/
           books?id=Ct2nrJSHxsQC& printsec=find& pg=PR5=onepage& q& f=false#v=onepage& q& f=false)
              • Takayama, Akira (1985). Mathematical Economics, 2nd ed. Cambridge. Description (http:/ / books. google. com/ books/ about/
           Mathematical_economics. html?id=685iPEaLAEcC) and Contents (http:/ / books. google. com/ books?id=685iPEaLAEcC& printsec=find&
           pg=PR9=onepage& q& f=false#v=onepage& q& f=false).
              • Michael Carter (2001). Foundations of Mathematical Economics, MIT Press. Description (http:/ / mitpress. mit. edu/ catalog/ item/ default.
           asp?ttype=2& tid=8630) and Contents (http:/ / books. google. com/ books?id=KysvrGGfzq0C& printsec=find& pg=PR7=onepage& q&
           f=false).
       [6] Chiang, Alpha C. (1992). Elements of Dynamic Optimization, Waveland. TOC (http:/ / www. waveland. com/ Titles/ Chiang. htm) &
           Amazon.com link (http:/ / www. amazon. com/ Elements-Dynamic-Optimization-Alpha-Chiang/ dp/ 157766096X) to inside, first pp.
       [7] Samuelson, Paul ((1947) [1983]). Foundations of Economic Analysis. Harvard University Press. ISBN 0-674-31301-1.
       [8] • Debreu, Gérard ([1987] 2008). "mathematical economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ /
           www. dictionaryofeconomics. com/ article?id=pde2008_M000107& edition=current& q=Mathematical economics& topicid=&
           result_number=1) Republished with revisions from 1986, "Theoretic Models: Mathematical Form and Economic Content", Econometrica,
           54(6), pp. 1259 (http:/ / www. jstor. org/ pss/ 1914299)-1270.
              • von Neumann, John, and Oskar Morgenstern (1944). Theory of Games and Economic Behavior. Princeton University Press.
Mathematical economics                                                                                                                                      23

    [9] Schumpeter, J.A. (1954). Elizabeth B. Schumpeter. ed. History of Economic Analysis (http:/ / books. google. com/ ?id=xjWiAAAACAAJ).
        New York: Oxford University Press. pp. 209–212. ISBN 978-0-04-330086-2. OCLC 13498913. .
    [10] Schumpeter (1954) p. 212-215
    [11] Schnieder, Erich (1934). "Johann Heinrich von Thünen". Econometrica (The Econometric Society) 2 (1): 1–12. doi:10.2307/1907947.
        ISSN 0012-9682. JSTOR 1907947. OCLC 35705710.
    [12] Schumpeter (1954) p. 465-468
    [13] Philip Mirowski, 1991. "The When, the How and the Why of Mathematical Expression in the History of Economics Analysis", Journal of
        Economic Perspectives, 5(1) pp. 145-157. (http:/ / www. fcs. edu. uy/ multi/ phes/ mirowski_math_econ_JEP. pdf)
    [14] Weintraub, E. Roy (2008). "mathematics and economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_M000372& edition=current& q=& topicid=& result_number=1).
    [15] Jevons, W.S. (1866). "Brief Account of a General Mathematical Theory of Political Economy", Journal of the Royal Statistical Society,
        XXIX (June) pp. 282-87. Read in Section F of the British Association, 1862. PDF. (http:/ / www. adelinotorres. com/ economia/ STANLEY
        JEVONS_Teoria Matemática e Economia Politica. pdf)
    [16] Jevons, W. Stanley (1871). The Principles of Political Economy, pp. 4, 25. (http:/ / books. google. com/ books?id=Sw8ZAAAAYAAJ&
        printsec=frontcover& dq="The+ Theory+ of+ Political+ Economy,"+ jevons+ 1871#v=onepage& q="The Theory of Political Economy,"
        jevons 1871& f=false). .
    [17] Sheila C., Dow (1999-05-21). "The Use of Mathematics in Economics" (http:/ / www. ioe. ac. uk/ esrcmaths/ sheila1. html). ESRC Public
        Understanding of Mathematics Seminar. Birmingham: Economic and Social Research Council. . Retrieved 2008-07-06.
    [18] While the concept of cardinality has fallen out of favor in neoclassical economics, the differences between cardinal utility and ordinal utility
        are minor for most applications.
    [19] Nicola, PierCarlo (2000). Mainstream Mathermatical Economics in the 20th Century (http:/ / books. google. com/ ?id=KR0Rbi8o4QQC).
        Springer. pp. 4. ISBN 978-3-540-67084-1. . Retrieved 2008-08-21.
    [20] Augustin Cournot (1838, tr. 1897) Researches nto the Mathematical Principles of Wealth. Links to description (http:/ / books. google. com/
        books?id=AGoGpyJY_SAC) and chapters. (http:/ / books. google. com/ books?id=AGoGpyJY_SAC& printsec=titlepage#PPR9,M1)
    [21] Hotelling, Harold (1990). "Stability in Competition" (http:/ / books. google. com/ ?id=dYbbHQAACAAJ). In Darnell, Adrian C.. The
        Collected Economics Articles of Harold Hotelling. Springer. pp. 51, 52. ISBN 3-540-97011-8. OCLC 20217006. . Retrieved 2008-08-21.
    [22] "Antoine Augustin Cournot, 1801-1877" (http:/ / cepa. newschool. edu/ het/ profiles/ cournot. htm). The History of Economic Thought
        Website. The New School for Social Research. . Retrieved 2008-08-21.
    [23] Gibbons, Robert (1992). Game Theory for Applied Economists (http:/ / books. google. com/ ?id=_6qgHgAACAAJ). Princeton, New Jersey:
        Princeton University Press. pp. 14, 15. ISBN 0-691-00395-5. .
    [24] Nicola, p. 9-12
    [25] Edgeworth, Francis Ysidro (September 5, 1889). "The Mathematical Theory of Political Economy: Review of Léon Walras, Éléments
        d'économie politique pure" (http:/ / cepa. newschool. edu/ het/ texts/ edgeworth/ edgewalras89. pdf) (PDF). Nature 40 (1036): 434–436.
        doi:10.1038/040434a0. ISSN 0028-0836. . Retrieved 2008-08-21.
    [26] Nicholson, Walter; Snyder, Christopher, p. 350-353.
    [27] Dixon, Robert. "Walras Law and Macroeconomics" (http:/ / web. archive. org/ web/ 20080417102559/ http:/ / www. economics. unimelb.
        edu. au/ rdixon/ wlaw. html). Walras Law Guide. Department of Economics, University of Melbourne. Archived from the original (http:/ /
        www. economics. unimelb. edu. au/ rdixon/ wlaw. html) on April 17, 2008. . Retrieved 2008-09-28.
    [28] Dixon, Robert. "A Formal Proof of Walras Law" (http:/ / web. archive. org/ web/ 20080430033548/ http:/ / www. economics. unimelb. edu.
        au/ rdixon/ walproof. html). Walras Law Guide. Department of Economics, University of Melbourne. Archived from the original (http:/ /
        www. economics. unimelb. edu. au/ rdixon/ walproof. html) on April 30, 2008. . Retrieved 2008-09-28.
    [29] Rima, Ingrid H. (1977). "Neoclassicism and Dissent 1890-1930" (http:/ / books. google. com/ ?id=s7cJAAAAIAAJ& printsec=find&
        pg=PR5=onepage& q#v=onepage& q& f=false). In Weintraub, Sidney. Modern Economic Thought. University of Pennsylvania Press. pp. 10,
        11. ISBN 0-8122-7712-0. .
    [30] Heilbroner, Robert L. (1953 [1999]). The Worldly Philosophers (http:/ / books. google. com/ ?id=N_3cj4urgJcC) (Seventh ed.). New York:
        Simon and Schuster. pp. 172–175, 313. ISBN 978-0-684-86214-9. .
    [31] Edgeworth, Francis Ysidro (1881 [1961]). Mathematical Psychics (http:/ / books. google. com/ ?id=Q4WCGAAACAAJ). London: Kegan
        Paul [A. M. Kelley]. pp. 15–19. .
    [32] Nicola, p. 14, 15, 258-261
    [33] Bowley, Arthur Lyon (1924 [1960]). The Mathematical Groundwork of Economics: an Introductory Treatise (http:/ / books. google. com/
        ?id=_cgkAAAAMAAJ). Oxford: Clarendon Press [Kelly]. .
    [34] Gillies, D. B. (1969). "Solutions to general non-zero-sum games" (http:/ / books. google. com/ ?id=9lSVFzsTGWsC). In Tucker, A. W. &
        Luce, R. D.. Contributions to the Theory of Games. Annals of Mathematics. 40. Princeton, New Jersey: Princeton University Press. pp. 47–85.
        ISBN 978-0-691-07937-0. .
    [35] Moss, Lawrence S. (2003). "The Seligman-Edgeworth Debate about the Analysis of Tax Incidence: The Advent of Mathematical
        Economics, 1892–1910". History of Political Economy (Duke University Press) 35 (2): 207, 212, 219, 234–237.
        doi:10.1215/00182702-35-2-205. ISSN 0018-2702.
    [36] Hotelling, Harold (1990). "Note on Edgeworth's Taxation Phenomenon and Professor Garver's Additional Condition on Demand Functions"
        (http:/ / books. google. com/ ?id=dYbbHQAACAAJ). In Darnell, Adrian C.. The Collected Economics Articles of Harold Hotelling. Springer.
Mathematical economics                                                                                                                                     24

        pp. 94–122. ISBN 3-540-97011-8. OCLC 20217006. . Retrieved 2008-08-26.
    [37] Herstein, I.N. (October 1953). "Some Mathematical Methods and Techniques in Economics". Quarterly of Applied Mathematics (American
        Mathematical Society) 11 (3): 249, 252, 260. ISSN 1552-4485. [Pp. 249-62 (http:/ / cowles. econ. yale. edu/ P/ cp/ p00b/ p0078. pdf).
    [38] • Weintraub, E. Roy (2008). "mathematics and economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ /
        www. dictionaryofeconomics. com/ article?id=pde2008_M000372& edition=current& q=& topicid=& result_number=1).
           • _____ (2002). How Economics Became a Mathematical Science. Duke University Press. Description (http:/ / www. dukeupress. edu/
        Catalog/ ViewProduct. php?productid=6807& viewby=series& categoryid=47& sort=newest) and preview (http:/ / books. google. com/
        books?hl=en& lr=& id=ntc_Fs36AoQC& oi=fnd& pg=PR7& dq=& f=false#v=onepage& q& f=false).
    [39] Nicholson, Walter; Snyder, Christopher (2007). "General Equilibrium and Welfare". Intermediate Microeconomics and Its Applications
        (10th ed.). Thompson. pp. 364, 365. ISBN 0-324-31968-1.
    [40] Jolink, Albert (2006). "What Went Wrong with Walras?". In Backhaus, Juergen G.; Maks, J.A. Hans. From Walras to Pareto. The European
        Heritage in Economics and the Social Sciences. IV. Springer. doi:10.1007/978-0-387-33757-9_6. ISBN 978-0-387-33756-2.
           • Blaug, Mark (2007). "The Fundamental Theorems of Modern Welfare Economics, Historically Contemplated". History of Political
        Economy (Duke University Press) 39 (2): 186–188. doi:10.1215/00182702-2007-001. ISSN 0018-2702.
    [41] Blaug (2007), p. 185, 187
    [42] Metzler, Lloyd (1948). "Review of Foundations of Economic Analysis". American Economic Review (The American Economic Review,
        Vol. 38, No. 5) 38 (5): 905–910. ISSN 0002-8282. JSTOR 1811704.
    [43] Neumann, J. von (1937). "Über ein ökonomisches Gleichungssystem und ein Verallgemeinerung des Brouwerschen Fixpunktsatzes",
        Ergebnisse eines Mathematischen Kolloquiums, 8, pp. 73-83, translated and published in 1945-46, as "A Model of General Equilibrium",
        Review of Economic Studies, 13, pp. 1–9.
    [44] For this problem to have a unique solution, it suffices that the nonnegative matrices A and B satisfy an irreducibility condition, generalizing
        that of the Perron–Frobenius theorem of nonnegative matrices, which considers the (simplified) eigenvalue problem

           A - λ I q = 0,
        where the nonnegative matrix A must be square and where the diagonal matrix I is the identity matrix. Von Neumann's irreducibility condition
        was called the "whales and wranglers" hypothesis by David Champernowne, who provided a verbal and economic commentary on the English
        translation of von Neumann's article. Von Neumann's hypothesis implied that every economic process used a positive amount of every
        economic good. Weaker "irreducibility" conditions were given by David Gale and by John Kemeny, Oskar Morgenstern, and Gerald L.
        Thompson in the 1950s and then by Stephen M. Robinson in the 1970s.
    [45] David Gale. The theory of linear economic models. McGraw-Hill, New York, 1960.
    [46] Morgenstern, Oskar; Thompson, Gerald L. (1976). Mathematical theory of expanding and contracting economies. Lexington Books.
        Lexington, Massachusetts: D. C. Heath and Company. pp. xviii+277.
    [47] Alexander Schrijver, Theory of Linear and Integer Programming. John Wiley & sons, 1998, ISBN 0-471-98232-6.
    [48] •Rockafellar, R. Tyrrell (1967). Monotone processes of convex and concave type. Memoirs of the American Mathematical Society.
        Providence, R.I.: American Mathematical Society. pp. i+74.
           • Rockafellar, R. T. (1974). "Convex algebra and duality in dynamic models of production". In Josef Loz and Maria Loz. Mathematical
        models in economics (Proc. Sympos. and Conf. von Neumann Models, Warsaw, 1972). Amsterdam: North-Holland and Polish Adademy of
        Sciences (PAN). pp. 351–378.
           •Rockafellar, R. T. (1970 (Reprint 1997 as a Princeton classic in mathematics)). Convex analysis. Princeton, New Jersey: Princeton
        University Press.
    [49] Kenneth Arrow, Paul Samuelson, John Harsanyi, Sidney Afriat, Gerald L. Thompson, and Nicholas Kaldor. (1989). Mohammed Dore,
        Sukhamoy Chakravarty, Richard Goodwin. ed. John Von Neumann and modern economics. Oxford:Clarendon. pp. 261.
    [50] Chapter 9.1 "The von Neumann growth model" (pages 277–299): Yinyu Ye. Interior point algorithms: Theory and analysis. Wiley. 1997.
    [51] Screpanti, Ernesto; Zamagni, Stefano (1993). An Outline of the History of Economic Thought. New York: Oxford University Press.
        pp. 288–290. ISBN 0-19-828370-9. OCLC 57281275.
    [52] David Gale. The theory of linear economic models. McGraw-Hill, New York, 1960.
    [53] Morgenstern, Oskar; Thompson, Gerald L. (1976). Mathematical theory of expanding and contracting economies. Lexington Books.
        Lexington, Massachusetts: D. C. Heath and Company. pp. xviii+277.
    [54] " The Nature of Mathematical Programming (http:/ / glossary. computing. society. informs. org/ index. php?page=nature. html)",
        Mathematical Programming Glossary, INFORMS Computing Society.
    [55] Schmedders, Karl (2008). "numerical optimization methods in economics", The New Palgrave Dictionary of Economics, 2nd Edition, v. 6,
        pp. 138-57. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_N000148& edition=current& q=optimization&
        topicid=& result_number=1)
    [56] Robbins, Lionel (1935, 2nd ed.). An Essay on the Nature and Significance of Economic Science, Macmillan, p. 16.
    [57] Blume, Lawrence E. (2008). "duality", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_D000196)
    [58] Dixit, A. K. ([1976] 1990). Optimization in Economic Theory, 2nd ed., Oxford. Description (http:/ / books. google. com/
        books?id=dHrsHz0VocUC& pg=find& pg=PA194=false#v=onepage& q& f=false) and contents preview (http:/ / books. google. com/
        books?id=dHrsHz0VocUC& pg=PR7& lpg=PR6& dq=false& lr=#v=onepage& q=false& f=false).
Mathematical economics                                                                                                                                 25

    [59] • Samuelson, Paul A., 1998. "How Foundations Came to Be", Journal of Economic Literature, 36(3), pp. 1375 (http:/ / www. jstor. org/ pss/
        2564803)–1386.
           • _____ (1970). "Maximum Principles in Analytical Economics" (http:/ / www. nobelprize. org/ nobel_prizes/ economics/ laureates/ 1970/
        samuelson-lecture. pdf), Nobel Prize lecture.
    [60] • Allan M. Feldman (3008). "welfare economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_W000050& edition=current& q=fundamental theorems & topicid=& result_number=3).
           • Mas-Colell, Andreu, Michael D. Whinston, and Jerry R. Green (1995), Microeconomic Theory, Chapter 16. Oxford University Press,
        ISBN 0-19-510268-1. Description (http:/ / www. oup. com/ us/ catalog/ general/ subject/ Economics/ MicroeconomicTheory/ ?view=usa&
        ci=9780195073409) and contents (http:/ / www. oup. com/ us/ catalog/ general/ subject/ Economics/ MicroeconomicTheory/ ?view=usa&
        sf=toc& ci=9780195073409).
    [61] • Geanakoplos, John ([1987] 2008). "Arrow–Debreu model of general equilibrium", The New Palgrave Dictionary of Economics, 2nd
        Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_A000133& edition=current& q=optimization& topicid=&
        result_number=20).
          • Arrow, Kenneth J., and Gérard Debreu (1954). "Existence of an Equilibrium for a Competitive Economy", Econometrica 22(3), pp. 265
        (http:/ / www. jstor. org/ pss/ 1907353)-290.
    [62] • Scarf, Herbert E. (2008). "computation of general equilibria", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ /
        www. dictionaryofeconomics. com/ article?id=pde2008_C000573& q=computational economics& topicid=& result_number=3)
          • Kubler, Felix (2008). "computation of general equilibria (new developments)", The New Palgrave Dictionary of Economics, 2nd Edition.
        Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000564& q=computational economics& topicid=&
        result_number=2)
    [63] Nicola, p. 133
    [64] Dorfman, Robert, Paul A. Samuelson, and Robert M. Solow (1958). Linear Programming and Economic Analysis. McGraw–Hill.
        Chapter-preview links. (http:/ / books. google. com/ books?hl=en& lr=& id=k5_vzaCNQP4C& oi=fnd& pg=PP11& dq=& ots=ticEpVKqDz&
        sig=FEIiGNmox4KDifmeKEfir7pZZNw#v=onepage& q& f=false)
    [65] M. Padberg, Linear Optimization and Extensions, Second Edition, Springer-Verlag, 1999.
    [66] Dantzig, George B. ([1987] 2008). "linear programming", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_L000106& edition=current& q=Linear programming& topicid=& result_number=1).
    [67] • Intriligator, Michael D. (2008). "nonlinear programming", The New Palgrave Dictionary of Economics, 2nd Edition. TOC (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_N000083& edition=current& q=non-linear programming& topicid=& result_number=3).
        • Blume, Lawrence E. (2008). "convex programming", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_C000348& edition=current& q=optimization& topicid=& result_number=4).
           • Kuhn, H. W.; Tucker, A. W. (1951). "Nonlinear programming". Proceedings of 2nd Berkeley Symposium. Berkeley: University of
        California Press. pp. 481–492.
    [68] • Bertsekas, Dimitri P. (1999). Nonlinear Programming (Second ed.). Cambridge, Massachusetts.: Athena Scientific. ISBN 1-886529-00-0.
           • Vapnyarskii, I.B. (2001), "Lagrange multipliers" (http:/ / www. encyclopediaofmath. org/ index. php?title=L/ l057190), in Hazewinkel,
        Michiel, Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4, .
           • Lasdon, Leon S. (1970). Optimization theory for large systems. Macmillan series in operations research. New York: The Macmillan
        Company. pp. xi+523. MR337317.
           • Lasdon, Leon S. (2002). Optimization theory for large systems (reprint of the 1970 Macmillan ed.). Mineola, New York: Dover
        Publications, Inc.. pp. xiii+523. MR1888251.
           • Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). "XII Abstract duality for practitioners". Convex analysis and minimization
        algorithms, Volume II: Advanced theory and bundle methods. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of
        Mathematical Sciences]. 306. Berlin: Springer-Verlag. pp. 136–193 (and Bibliographical comments on pp. 334–335). ISBN 3-540-56852-2.
    [69] Lemaréchal, Claude (2001). "Lagrangian relaxation". In Michael Jünger and Denis Naddef. Computational combinatorial optimization:
        Papers from the Spring School held in Schloß Dagstuhl, May 15–19, 2000. Lecture Notes in Computer Science. 2241. Berlin:
        Springer-Verlag. pp. 112–156. doi:10.1007/3-540-45586-8_4. ISBN 3-540-42877-1. MR1900016.doi:10.1007/3-540-45586-8_4.
    [70] Pontryagin, L. S.; Boltyanski, V. G., Gamkrelidze, R. V., Mischenko, E. F. (1962). The Mathematical Theory of Optimal Processes (http:/ /
        books. google. com/ books?id=kwzq0F4cBVAC& printsec=frontcover#v=onepage& q& f=false). New York: Wiley. ISBN 68981.
    [71] • Zelikin, M. I. ([1987] 2008). "Pontryagin's principle of optimality", The New Palgrave Dictionary of Economics, 2nd Edition. Preview link
        (http:/ / www. dictionaryofeconomics. com/ search_results?q="Pontryagin's+ principle+ of+ optimality"& edition=current&
        button_search=GO).
           • Martos, Béla (1987). "control and coordination of economic activity", The New Palgrave: A Dictionary of Economics. Description link
        (http:/ / www. dictionaryofeconomics. com/ search_results?q=control+ coordination+ Martos+ & field=content& edition=all& topicid=).
           • Brock, W. A. (1987). "optimal control and economic dynamics", The New Palgrave: A Dictionary of Economics. Outline (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde1987_X001613).
           • Shell, K., ed. (1967). Essays on the Theory of Optimal Economic Growth. Cambridge, Massachusetts: The MIT Press.
        ISBN 0-262-19036-2.]
    [72] Stokey, Nancy L. and Robert E. Lucas with Edward Prescott (1989). Recursive Methods in Economic Dynamics, Harvard University Press,
        chapter 5. Desecription (http:/ / www. hup. harvard. edu/ catalog. php?isbn=9780674750968) and chapter-preview links (http:/ / books.
Mathematical economics                                                                                                                                  26

        google. com/ books?hl=en& lr=& id=tWYo0QolyLAC& oi=fnd& pg=PR11& dq=#v=onepage& q=f=false& f=false).
    [73] Malliaris, A.G. (2008). "stochastic optimal control", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_S000269& edition=& field=keyword& q=Taylor's th& topicid=& result_number=1).
    [74] • Arrow, K. J.; Kurz, M. (1970). Public Investment, the Rate of Return, and Optimal Fiscal Policy. Baltimore, Maryland: The Johns Hopkins
        Press. ISBN 0-8018-1124-4. Abstract. (http:/ / md1. csa. com/ partners/ viewrecord. php?requester=gs& collection=ENV& recid=7107596&
        q=& uid=788819967& setcookie=yes)
           • Sethi, S. P.; Thompson, G. L. (2000). Optimal Control Theory: Applications to Management Science and Economics, Second Edition. New
        York: Springer. ISBN 0-7923-8608-6. Scroll to chapter-preview links. (http:/ / books. google. com/ books?id=gLMmaLYRy4QC&
        printsec=frontcover& source=gbs_atb#v=onepage& q& f=false)
    [75] Andrew McLennan, 2008. "fixed point theorems", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_F000135& edition=current& q=).
    [76] Weintraub, E. Roy (1977). "General Equilibrium Theory" (http:/ / books. google. com/ ?id=JDqAAAAAIAAJ). In Weintraub, Sidney.
        Modern Economic Thought. University of Pennsylvania Press. pp. 107–109. ISBN 0-8122-7712-0. .
           • Arrow, Kenneth J.; Debreu, Gérard (1954). "Existence of an equilibrium for a competitive economy". Econometrica (The Econometric
        Society) 22 (3): 265–290. doi:10.2307/1907353. ISSN 0012-9682. JSTOR 1907353.
    [77] Kantorovich, Leonid, and Victor Polterovich (2008). "Functional analysis", in S. Durlauf and L. Blume, ed., The New Palgrave Dictionary
        of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_F000236), ed., Palgrave Macmillan.
    [78] Kantorovich, L. V (1990). ""My journey in science (supposed report to the Moscow Mathematical Society)" [expanding Russian Math.
        Surveys 42 (1987), no. 2, pp. 233–270]". In Lev J. Leifman. Functional analysis, optimization, and mathematical economics: A collection of
        papers dedicated to the memory of Leonid Vitalʹevich Kantorovich. New York: The Clarendon Press, Oxford University Press. pp. 8–45.
        ISBN 0-19-505729-5. MR898626.
    [79] Page 406: Polyak, B. T. (2002). "History of mathematical programming in the USSR: Analyzing the phenomenon (Chapter 3 The pioneer:
        L. V. Kantorovich, 1912–1986, pp. 405–407)". Mathematical Programming 91 (ISMP 2000, Part 1 (Atlanta, GA)): pp. 401–416.
        doi:10.1007/s101070100258. MR1888984.
    [80] "Leonid Vitaliyevich Kantorovich — Prize Lecture ("Mathematics in economics: Achievements, difficulties, perspectives")" (http:/ /
        nobelprize. org/ nobel_prizes/ economics/ laureates/ 1975/ kantorovich-lecture. html). Nobelprize.org. . Retrieved 12 Dec 2010.
    [81] Aliprantis, Charalambos D.; Brown, Donald J.; Burkinshaw, Owen (1990). Existence and optimality of competitive equilibria. Berlin:
        Springer–Verlag. pp. xii+284. ISBN 3-540-52866-0. MR1075992.
    [82] Rockafellar, R. Tyrrell. Conjugate duality and optimization. Lectures given at the Johns Hopkins University, Baltimore, Maryland, June,
        1973. Conference Board of the Mathematical Sciences Regional Conference Series in Applied Mathematics, No. 16. Society for Industrial and
        Applied Mathematics, Philadelphia, Pa., 1974. vi+74 pp.
    [83] Lester G. Telser and Robert L. Graves Functional Analysis in Mathematical Economics: Optimization Over Infinite Horizons 1972.
        University of Chicago Press, 1972, ISBN 978-0-226-79190-6.
    [84] Neumann, John von, and Oskar Morgenstern (1944) Theory of Games and Economic Behavior, Princeton.
    [85] Mas-Colell, Andreu (1985). The Theory of general economic equilibrium: A differentiable approach. Econometric Society monographs.
        Cambridge UP. ISBN 0-521-26514-2. MR1113262.
    [86] Yves Balasko. Foundations of the Theory of General Equilibrium, 1988, ISBN 0-12-076975-1.
    [87] Creedy, John (2008). "Francis Ysidro (1845–1926)", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_E000041& edition=current& q=edgeworth& topicid=& result_number=3).
    [88] • Nash, John F., Jr. (1950). "The Bargaining Problem", Econometrica, 18(2), pp. 155-162 (http:/ / www. stern. nyu. edu/ networks/
        Nash_The_Bargaining_Problem. pdf).
           • Serrano, Roberto (2008). "bargaining", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_B000073& edition=current& q=bargaining& topicid=& result_number=1).
    [89] • Smith,Vernon L. (1992). "Game Theory and Experimental Economics: Beginnings and Early Influences", in E. R. Weintraub, ed., Towards
        a History of Game Theory, pp. 241- (http:/ / books. google. com/ books?hl=en& lr=& id=9CHY2Gozh1MC& oi=fnd& pg=PA241&
        ots=onepage& q& f=false#v=onepage& q& f=false*) 282.
           • _____ (2001). "Experimental Economics", International Encyclopedia of the Social & Behavioral Sciences, pp. 5100-5108. Abstract
        (http:/ / www. sciencedirect. com/ science/ article/ pii/ B0080430767022324) per sect. 1.1 & 2.1.
           • Plott, Charles R., and Vernon L. Smith, ed. (2008). Handbook of Experimental Economics Results, v. 1, Elsevier, Part 4, Games, ch. 45-66
        preview links (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL& _hubEid=1-s2. 0-S1574072207X00015& _cid=277334&
        _pubType=HS& _auth=y& _acct=C000228598& _version=1& _urlVersion=0& _userid=10& md5=49f8b6d5e3024eac39ed5fad351fe568).
           • Shubik, Martin (2002). "Game Theory and Experimental Gaming", in R. Aumann and S. Hart, ed., Handbook of Game Theory with
        Economic Applications, Elsevier, v. 3, pp. 2327-2351. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574000502030254).
    [90] From The New Palgrave Dictionary of Economics (2008), 2nd Edition:
           • Gul, Faruk. "behavioural economics and game theory." Abstract. (http:/ / www. dictionaryofeconomics. com/
        article?id=pde2008_G000210& q=Behavioral economics & topicid=& result_number=2)
           • Camerer, Colin F. "behavioral game theory." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_B000302&
        q=Behavioral economics & topicid=& result_number=13)
Mathematical economics                                                                                                                                  27

    [91] • Rasmusen, Eric (2007). Games and Information, 4th ed. Description (http:/ / www. wiley. com/ WileyCDA/ WileyTitle/
        productCd-EHEP001009. html) and chapter-preview links. (http:/ / books. google. com/ books?id=5XEMuJwnBmUC& printsec=fnd&
        pg=PR5& dq=gbs_atb#v=onepage& q& f=false)
           • Aumann, R., and S. Hart, ed. (1992, 2002). Handbook of Game Theory with Economic Applications v. 1, links at ch. 3-6 (http:/ / www.
        sciencedirect. com/ science/ handbooks/ 15740005/ 1) and v. 3, ch. 43 (http:/ / www. sciencedirect. com/ science/ article/ pii/
        S1574000502030060).
    [92] • Tirole, Jean (1988). The Theory of Industrial Organization, MIT Press. Description (http:/ / mitpress. mit. edu/ catalog/ item/ default.
        asp?ttype=2& tid=8224) and chapter-preview links, pp. vii-ix (http:/ / books. google. com/ books?id=HIjsF0XONF8C& printsec=find&
        pg=PR7=onepage& q& f=false#v=onepage& q& f=false), "General Organization", pp. 5-6 (http:/ / books. google. com/
        books?id=HIjsF0XONF8C& dq=find& pg=PA5& source=bn& hl=en& ei=e2vES-H-O8T68Abxp_GyDw& sa=X& oi=book_result&
        ct=result& resnum=4& ved=0CB8Q6AEwAw#v=onepage& q& f=false), and "Non-Cooperative Game Theory: A User's Guide Manual,' " ch.
        11, pp. 423-59 (http:/ / books. google. com/ books?id=HIjsF0XONF8C& dq=find& pg=PA423& source=bn& hl=en&
        ei=e2vES-H-O8T68Abxp_GyDw& sa=X& oi=book_result& ct=result& resnum=4& ved=0CB8Q6AEwAw#v=onepage& q& f=false).
           • Bagwell, Kyle, and Asher Wolinsky (2002). "Game theory and Industrial Organization", ch. 49, Handbook of Game Theory with Economic
        Applications, v. 3, pp. 1851 (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574000502030126)-1895.
    [93] • Shubik, Martin (1981). "Game Theory Models and Methods in Political Economy", in Handbook of Mathematical Economics,, v. 1, pp.
        285 (http:/ / www. sciencedirect. com/ science?_ob=ArticleURL& _udi=B7P5Y-4FDF0FN-C& _user=10& _coverDate=01/ 01/ 1981&
        _rdoc=11& _fmt=high& _orig=browse& _origin=browse& _zone=rslt_list_item&
        _srch=doc-info(#toc#24615#1981#999989999#565707#FLP#display#Volume)& _cdi=24615& _sort=d& _docanchor=& _ct=14&
        _acct=C000050221& _version=1& _urlVersion=0& _userid=10& md5=cb34198ec88c9ab8fa59af6d5634e9cf& searchtype=a)-330.
    [94] • The New Palgrave Dictionary of Economics (2008), 2nd Edition:
        Myerson, Roger B. "mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000132&
        edition=current& q=mechanism design& topicid=& result_number=3)
             _____. "revelation principle." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_R000137& edition=current&
        q=moral& topicid=& result_number=1)
             Sandholm, Tuomas. "computing in mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/
        article?id=pde2008_C000563& edition=& field=keyword& q=algorithmic mechanism design& topicid=& result_number=1)
           • Nisan, Noam, and Amir Ronen (2001). "Algorithmic Mechanism Design", Games and Economic Behavior, 35(1-2), pp. 166–196 (http:/ /
        www. cs. cmu. edu/ ~sandholm/ cs15-892F09/ Algorithmic mechanism design. pdf).
           • Nisan, Noam, et al., ed. (2007). Algorithmic Game Theory, Cambridge University Press. Description (http:/ / www. cup. cam. ac. uk/ asia/
        catalogue/ catalogue. asp?isbn=9780521872829).
    [95] • Halpern, Joseph Y. (2008). "computer science and game theory", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/
        / www. dictionaryofeconomics. com/ article?id=pde2008_C000566& edition=current& q=& topicid=& result_number=1).
                 • Shoham, Yoav (2008). "Computer Science and Game Theory", Communications of the ACM, 51(8), pp. 75-79 (http:/ / www. robotics.
        stanford. edu/ ~shoham/ www papers/ CSGT-CACM-Shoham. pdf).
                 • Roth,Alvin E. (2002). "The Economist as Engineer: Game Theory, Experimentation, and Computation as Tools for Design
        Economics", Econometrica, 70(4), pp. 1341–1378 (http:/ / kuznets. fas. harvard. edu/ ~aroth/ papers/ engineer. pdf).
    [96] • Kirman, Alan (2008). "economy as a complex system", The New Palgrave Dictionary of Economics , 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_E000246& edition=current& q=Computational economics& topicid=& result_number=20).
           • Tesfatsion, Leigh (2003). "Agent-based Computational Economics: Modeling Economies as Complex Adaptive Systems", Information
        Sciences, 149(4), pp. 262-268 (http:/ / copper. math. buffalo. edu/ urgewiki/ uploads/ Literature/ Tesfatsion2002. pdf).
    [97] Scott E. Page (2008), "agent-based models", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_A000218& edition=current& q=agent-based computational modeling& topicid=&
        result_number=1).
    [98] • Holland, John H., and John H. Miller (1991). "Artificial Adaptive Agents in Economic Theory", American Economic Review, 81(2), [pp.
        365-370 (http:/ / www. santafe. edu/ media/ workingpapers/ 91-05-025. pdf) p. 366.
           • Arthur, W. Brian, 1994. "Inductive Reasoning and Bounded Rationality", American Economic Review, 84(2), pp. 406-411 (http:/ /
        www-personal. umich. edu/ ~samoore/ bit885f2011/ arthur-inductive. pdf).
           • Schelling, Thomas C. (1978 [2006]). Micromotives and Macrobehavior, Norton. Description (http:/ / books. wwnorton. com/ books/
        978-0-393-32946-9/ ), preview (http:/ / books. google. com/ books?id=DenWKRgqzWMC& printsec=find& pg=PA1=#v=onepage& q&
        f=false).
           • Sargent, Thomas J. (1994). Bounded Rationality in Macroeconomics, Oxford. Description (http:/ / www. oup. com/ us/ catalog/ general/
        subject/ Economics/ MacroeconomicTheory/ ?view=usa& ci=9780198288695) and chapter-preview 1st-page links (http:/ / www. questia.
        com/ library/ book/ bounded-rationality-in-macroeconomics-thomas-j-sargent-by-thomas-j-sargent. jsp).
    [99] • Judd, Kenneth L. (2006). "Computationally Intensive Analyses in Economics", Handbook of Computational Economics, v. 2, ch. 17,
        Introduction, p. 883. [Pp. 881- (http:/ / books. google. com/ books?hl=en& lr=& id=6ITfRkNmKQcC& oi=fnd& pg=PA881&
        ots=2j0cCBB5S6& sig=a1DlAKMWcxFQZwSkGVVp2zlHIb8#v=onepage& q& f=false) 893. Pre-pub PDF (http:/ / www2. econ. iastate.
        edu/ tesfatsi/ Judd. finalrev. pdf).
           • _____ (1998). Numerical Methods in Economics, MIT Press. Links to description (http:/ / mitpress. mit. edu/ catalog/ item/ default.
Mathematical economics                                                                                                                                         28

       asp?ttype=2& tid=3257) and chapter previews (http:/ / books. google. com/ books?id=9Wxk_z9HskAC& pg=PR7& source=gbs_toc_r&
       cad=3#v=onepage& q& f=false).
    [100] • Tesfatsion, Leigh (2002). "Agent-Based Computational Economics: Growing Economies from the Bottom Up", Artificial Life, 8(1),
       pp.55-82. Abstract (http:/ / www. mitpressjournals. org/ doi/ abs/ 10. 1162/ 106454602753694765) and pre-pub PDF (http:/ / www. econ.
       brown. edu/ fac/ Peter_Howitt/ SummerSchool/ Agent. pdf).
          • _____ (1997). "How Economists Can Get Alife", in W. B. Arthur, S. Durlauf, and D. Lane, eds., The Economy as an Evolving Complex
       System, II, pp. 533-564. Addison-Wesley. Pre-pub PDF (http:/ / ageconsearch. umn. edu/ bitstream/ 18196/ 1/ er37. pdf).
    [101] Tesfatsion, Leigh (2006), "Agent-Based Computational Economics: A Constructive Approach to Economic Theory", ch. 16, Handbook of
       Computational Economics, v. 2, part 2, ACE study of economic system. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/
       S1574002105020162) and pre-pub PDF (http:/ / econ2. econ. iastate. edu/ tesfatsi/ hbintlt. pdf).
    [102] Axelrod, Robert (1997). The Complexity of Cooperation: Agent-Based Models of Competition and Collaboration, Princeton. Description
       (http:/ / press. princeton. edu/ titles/ 6144. html), contents (http:/ / press. princeton. edu/ titles/ 6144. html#TOC), and preview (http:/ / books.
       google. com/ books?id=J0dgRGMdjmQC& printsec=find& pg=PR11#v=onepage& q& f=false).
    [103] • Leombruni, Roberto, and Matteo Richiardi, ed. (2004), Industry and Labor Dynamics: The Agent-Based Computational Economics
       Approach. World Scientific Publishing ISBN 981-256-100-5. Description (http:/ / www. worldscibooks. com/ economics/ 5706. html) and
       chapter-preview links (http:/ / books. google. com/ books?id=P5O7A5D55nQC& printsec=fond& pg=PR5#v=onepage& q& f=false).
          • Epstein, Joshua M. (2006). "Growing Adaptive Organizations: An Agent-Based Computational Approach", in Generative Social Science:
       Studies in Agent-Based Computational Modeling, pp. 309 - (http:/ / books. google. com/ books?hl=en& lr=& id=543OS3qdxBYC& oi=fnd&
       pg=PA326& dq=false#v=onepage& q=false& f=false) 344. Description (http:/ / press. princeton. edu/ titles/ 8277. html) and abstract (http:/ /
       www. santafe. edu/ research/ working-papers/ abstract/ 99895b6465e8b87656612f8e3570b34c/ ).
    [104] Klosa, Tomas B., and Bart Nooteboom, 2001. "Agent-based Computational Transaction Cost Economics", Journal of Economic Dynamics
       and Control 25(3–4), pp. 503–52. Abstract. (http:/ / www. sciencedirect. com/ science/ article/ pii/ S0165188900000348)
    [105] Axtell, Robert (2005). "The Complexity of Exchange", Economic Journal, 115(504, Features), pp. F193-F210 (http:/ / econfaculty. gmu.
       edu/ pboettke/ workshop/ archives/ f05/ Axtell. pdf).
    [106] • The New Palgrave Dictionary of Economics (2008), 2nd Edition:
       Myerson, Roger B. "mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000132&
       edition=current& q=mechanism design& topicid=& result_number=3)
            _____. "revelation principle." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_R000137& edition=current&
       q=moral& topicid=& result_number=1)
            Sandholm, Tuomas. "computing in mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/
       article?id=pde2008_C000563& edition=& field=keyword& q=algorithmic mechanism design& topicid=& result_number=1)
          • Nisan, Noam, and Amir Ronen (2001). "Algorithmic Mechanism Design", Games and Economic Behavior, 35(1-2), pp. 166–196 (http:/ /
       www. cs. cmu. edu/ ~sandholm/ cs15-892F09/ Algorithmic mechanism design. pdf).
          • Nisan, Noam, et al., ed. (2007). Algorithmic Game Theory, Cambridge University Press. Description (http:/ / www. cup. cam. ac. uk/ asia/
       catalogue/ catalogue. asp?isbn=9780521872829).
    [107] Sandholm, Tuomas W., and Victor R. Lesser (2001)."Leveled Commitment Contracts and Strategic Breach", Games and Economic
       Behavior, 35(1-2), pp. 212-270 (http:/ / www. cs. cmu. edu/ afs/ . cs. cmu. edu/ Web/ People/ sandholm/ leveled. geb. pdf).
    [108] • Colander, David, Peter Howitt, Alan Kirman, Axel Leijonhufvud, and Perry Mehrling (2008). "Beyond DSGE Models: Toward an
       Empirically Based Macroeconomics", American Economic Review, 98(2), pp. 236 (http:/ / www. jstor. org/ pss/ 29730026)-240. Pre-pub PDF
       (http:/ / www. econ. brown. edu/ fac/ peter_howitt/ publication/ complex macro6. pdf).
          • Sargent, Thomas J. (1994). Bounded Rationality in Macroeconomics, Oxford. Description (http:/ / www. oup. com/ us/ catalog/ general/
       subject/ Economics/ MacroeconomicTheory/ ?view=usa& ci=9780198288695) and chapter-preview 1st-page links (http:/ / www. questia.
       com/ library/ book/ bounded-rationality-in-macroeconomics-thomas-j-sargent-by-thomas-j-sargent. jsp).
    [109] Tesfatsion, Leigh (2006), "Agent-Based Computational Economics: A Constructive Approach to Economic Theory", ch. 16, Handbook of
       Computational Economics, v. 2, pp. 832-865. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020162) and
       pre-pub PDF (http:/ / econ2. econ. iastate. edu/ tesfatsi/ hbintlt. pdf).
    [110] Smith, Vernon L. (2008). "experimental economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
       dictionaryofeconomics. com/ article?id=pde2008_E000277& q=experimental & topicid=& result_number=2).
    [111] Duffy, John (2006). "Agent-Based Models and Human Subject Experiments", ch. 19, Handbook of Computational Economics, v.2, pp.
       949–101. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020198).
    [112] • Namatame, Akira, and Takao Terano (2002). "The Hare and the Tortoise: Cumulative Progress in Agent-based Simulation", in
       Agent-based Approaches in Economic and Social Complex Systems. pp. 3- (http:/ / books. google. com/ books?id=nTkwxVUYvfIC&
       printsec=find& pg=PA3=onepage& q& f=false#v=onepage& q& f=false) 14, IOS Press. Description (http:/ / www. iospress. nl/ book/
       agent-based-approaches-in-economic-and-social-complex-systems-2/ ).
          • Fagiolo, Giorgio, Alessio Moneta, and Paul Windrum (2007). "A Critical Guide to Empirical Validation of Agent-Based Models in
       Economics: Methodologies, Procedures, and Open Problems", Computational Economics, 30, pp. 195 (http:/ / www. springerlink. com/
       content/ t683473172528275/ )–226.
    [113] • Tesfatsion, Leigh (2006). "Agent-Based Computational Economics: A Constructive Approach to Economic Theory", ch. 16, Handbook of
       Computational Economics, v. 2, [pp. 831-880] sect. 5. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020162)
Mathematical economics                                                                                                                                   29

       and pre-pub PDF (http:/ / econ2. econ. iastate. edu/ tesfatsi/ hbintlt. pdf).
          • Judd, Kenneth L. (2006). "Computationally Intensive Analyses in Economics", Handbook of Computational Economics, v. 2, ch. 17, pp.
       881- (http:/ / books. google. com/ books?hl=en& lr=& id=6ITfRkNmKQcC& oi=fnd& pg=PA881& ots=2j0cCBB5S6&
       sig=a1DlAKMWcxFQZwSkGVVp2zlHIb8#v=onepage& q& f=false) 893. Pre-pub PDF (http:/ / www2. econ. iastate. edu/ tesfatsi/ Judd.
       finalrev. pdf).
          • Tesfatsion, Leigh, and Kenneth L. Judd, ed. (2006). Handbook of Computational Economics, v. 2. Description (http:/ / www. elsevier.
       com/ wps/ find/ bookdescription. cws_home/ 660847/ description#description) & and chapter-preview links. (http:/ / www. sciencedirect.
       com/ science?_ob=PublicationURL& _hubEid=1-s2. 0-S1574002105X02003& _cid=273377& _pubType=HS& _auth=y&
       _acct=C000228598& _version=1& _urlVersion=0& _userid=10& md5=e4757b4f65755ed6340a11fee9615200)
    [114] Brockhaus, Oliver; Farkas, Michael; Ferraris, Andrew; Long, Douglas; Overhaus, Marcus (2000). Equity Derivatives and Market Risk
       Models (http:/ / books. google. com/ ?id=JGZPAAAAMAAJ). Risk Books. pp. 13–17. ISBN 978-1-899332-87-8. . Retrieved 2008-08-17.
    [115] Liner, Gaines H. (2002). "Core Journals in Economics". Economic Inquiry (Oxford University Press) 40 (1): 140. doi:10.1093/ei/40.1.138.
    [116] Stigler, George J.; Stigler, Steven J.; Friedland, Claire (April 1995). "The Journals of Economics". The Journal of Political Economy (The
       University of Chicago Press) 103 (2): 339. doi:10.1086/261986. ISSN 0022-3808. JSTOR 2138643.
    [117] Stigler et al. reviewed journal articles in core economic journals (as defined by the authors but meaning generally non-specialist journals)
       throughout the 20th century. Journal articles which at any point used geometric representation or mathematical notation were noted as using
       that level of mathematics as its "highest level of mathematical technique". The authors refer to "verbal techniques" as those which conveyed
       the subject of the piece without notation from geometry, algebra or calculus.
    [118] Stigler et al., p. 342
    [119] Sutter, Daniel and Rex Pjesky. "Where Would Adam Smith Publish Today?: The Near Absence of Math-free Research in Top Journals"
       (May 2007). (http:/ / econjwatch. org/ issues/ volume-4-number-1-may-2007)
    [120] Arrow, Kenneth J. (April 1960). "The Work of Ragnar Frisch, Econometrician". Econometrica (Blackwell Publishing) 28 (2): 175–192.
       doi:10.2307/1907716. ISSN 0012-9682. JSTOR 1907716.
    [121] Bjerkholt, Olav (July 1995). "Ragnar Frisch, Editor of Econometrica 1933-1954". Econometrica (Blackwell Publishing) 63 (4): 755–765.
       doi:10.2307/2171799. ISSN 0012-9682. JSTOR 1906940.
    [122] Lange, Oskar (1945). "The Scope and Method of Economics". Review of Economic Studies (The Review of Economic Studies Ltd.) 13 (1):
       19–32. doi:10.2307/2296113. ISSN 0034-6527. JSTOR 2296113.
    [123] Aldrich, John (January 1989). "Autonomy". Oxford Economic Papers (Oxford University Press) 41 (1, History and Methodology of
       Econometrics): 15–34. ISSN 0030-7653. JSTOR 2663180.
    [124] Epstein, Roy J. (1987). A History of Econometrics. Contributions to Economic Analysis. North-Holland. pp. 13–19.
       ISBN 978-0-444-70267-8. OCLC 230844893.
    [125] Colander, David C. (2004). "The Strange Persistence of the IS-LM Model". History of Political Economy (Duke University Press) 36
       (Annual Supplement): 305–322. doi:10.1215/00182702-36-Suppl_1-305. ISSN 0018-2702.
    [126] Brems, Hans (Oct., 1975). "Marshall on Mathematics". Journal of Law and Economics (University of Chicago Press) 18 (2): 583–585.
       doi:10.1086/466825. ISSN 0022-2186. JSTOR 725308.
    [127] Frigg, R.; Hartman, S. (February 27, 2006). Edward N. Zalta. ed. Models in Science (http:/ / plato. stanford. edu/ entries/ models-science/
       #OntWhaMod). Stanford Encyclopedia of Philosophy. Stanford, California: The Metaphysics Research Lab. ISSN 1095-5054. . Retrieved
       2008-08-16.
    [128] Hayek, Friedrich (September 1945). "The Use of Knowledge in Society". American Economic Review 35 (4): 519–530. JSTOR 1809376.
    [129] Heilbroner, Robert (May–June 1999). "The end of the Dismal Science?" (http:/ / findarticles. com/ p/ articles/ mi_m1093/ is_3_42/
       ai_54682627/ print). Challenge Magazine. .
    [130] Beed & Owen, 584
    [131] Boland, L. A. (2007). "Seven Decades of Economic Methodology" (http:/ / books. google. com/ ?id=w-BEoTj0axoC). In I. C. Jarvie, K.
       Milford, D.W. Miller. Karl Popper:A Centenary Assessment. London: Ashgate Publishing. pp. 219. ISBN 978-0-7546-5375-2. . Retrieved
       2008-06-10.
    [132] Beed, Clive; Kane, Owen (1991). "What Is the Critique of the Mathematization of Economics?". Kyklos 44 (4): 581–612.
       doi:10.1111/j.1467-6435.1991.tb01798.x.
    [133] Friedman, Milton (1953). Essays in Positive Economics (http:/ / books. google. com/ ?id=rSGekjfpf4cC). Chicago: University of Chicago
       Press. pp. 30, 33, 41. ISBN 978-0-226-26403-5. .
    [134] Keynes, John Maynard (1936). The General Theory of Employment, Interest and Money (http:/ / www. marxists. org/ reference/ subject/
       economics/ keynes/ general-theory/ ch21. htm). Cambridge: Macmillan. pp. 297. ISBN 0-333-10729-2. .
    [135] Paul A. Samuelson (1952). "Economic Theory and Mathematics — An Appraisal", American Economic Review, 42(2), pp. 56, 64-65
       (http:/ / cowles. econ. yale. edu/ P/ cp/ p00b/ p0061. pdf) (press +).
    [136] D.W. Bushaw and R.W. Clower (1957). Introduction to Mathematical Economics, p. vii. (http:/ / babel. hathitrust. org/ cgi/ pt?id=uc1.
       b3513586;page=root;view=image;size=100;seq=11;num=vii)
    [137] Solow, Robert M. (20 March 1988). "The Wide, Wide World Of Wealth (The New Palgrave: A Dictionary of Economics'. Edited by John
       Eatwell, Murray Milgate and Peter Newman. Four volumes. 4,103 pp. New York: Stockton Press. $650)" (http:/ / www. nytimes. com/ 1988/
       03/ 20/ books/ the-wide-wide-world-of-wealth. html?scp=1). New York Times. .
Mathematical economics                                                                                                                       30


    External links
    • Journal of Mathematical Economics Aims & Scope (http://www.elsevier.com/wps/find/journaldescription.
      cws_home/505577/description#description)
    • Mathematical Economics and Financial Mathematics (http://www.dmoz.org/Science/Math/Applications/
      Mathematical_Economics_and_Financial_Mathematics//) at the Open Directory Project



    Statistics
    Statistics is the study of the collection, organization, analysis, interpretation, and presentation of data.[1][2] It deals
    with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments.[1]
    A statistician is someone who is particularly well-versed in the ways of thinking necessary to successful apply
    statistical analysis. Such people often gain experience through working in any of a wide number of fields. A
    discipline called mathematical statistics studies statistics mathematically.
    The word statistics, when referring to the scientific discipline, is singular, as in "Statistics is an art."[3] This should
    not be confused with the word statistic, referring to a quantity (such as mean or median) calculated from a set of
    data,[4] whose plural is statistics ("this statistic seems wrong" or "these statistics are misleading").


    Scope
    Some        consider    statistics     a
    mathematical body of science that
    pertains to the collection, analysis,
    interpretation or explanation, and
    presentation of data,[5] while others
    consider it a branch of mathematics[6]
    concerned      with   collecting    and
    interpreting data. Because of its
    empirical roots and its focus on
    applications, statistics is usually
    considered a distinct mathematical
    science rather than a branch of
    mathematics.[7][8] Much of statistics is
    non-mathematical: ensuring that data
                                                   More probability density is found the closer one gets to the expected (mean) value in a
    collection is undertaken in a way that          normal distribution. Statistics used in standardized testing assessment are shown. The
    produces valid conclusions; coding and           scales include standard deviations, cumulative percentages, percentile equivalents,
    archiving data so that information is                  Z-scores, T-scores, standard nines, and percentages in standard nines.

    retained and made useful for
    international comparisons of official statistics; reporting of results and summarised data (tables and graphs) in ways
    comprehensible to those must use them; implementing procedures that ensure the privacy of census information.

    Statisticians improve data quality by developing specific experiment designs and survey samples. Statistics itself also
    provides tools for prediction and forecasting the use of data and statistical models. Statistics is applicable to a wide
    variety of academic disciplines, including natural and social sciences, government, and business. Statistical
    consultants can help organizations and companies that don't have in-house expertise relevant to their particular
    questions.
Statistics                                                                                                                       31


     Statistical methods can summarize or describe a collection of data. This is called descriptive statistics. This is
     particularly useful in communicating the results of experiments and research. In addition, data patterns may be
     modeled in a way that accounts for randomness and uncertainty in the observations.
     These models can be used to draw inferences about the process or population under study—a practice called
     inferential statistics. Inference is a vital element of scientific advance, since it provides a way to draw conclusions
     from data that are subject to random variation. To prove the propositions being investigated further, the conclusions
     are tested as well, as part of the scientific method. Descriptive statistics and analysis of the new data tend to provide
     more information as to the truth of the proposition.
     "Applied statistics" comprises descriptive statistics and the application of inferential statistics.[9] Theoretical
     statistics concerns both the logical arguments underlying justification of approaches to statistical inference, as well
     encompassing mathematical statistics. Mathematical statistics includes not only the manipulation of probability
     distributions necessary for deriving results related to methods of estimation and inference, but also various aspects of
     computational statistics and the design of experiments.
     Statistics is closely related to probability theory, with which it is often grouped. The difference is, roughly, that
     probability theory starts from the given parameters of a total population to deduce probabilities that pertain to
     samples. Statistical inference, however, moves in the opposite direction—inductively inferring from samples to the
     parameters of a larger or total population.


     History
     Statistical methods date back at least to the 5th century BC. The earliest known writing on statistics appears in a 9th
     century book entitled Manuscript on Deciphering Cryptographic Messages, written by Al-Kindi. In this book,
     Al-Kindi provides a detailed description of how to use statistics and frequency analysis to decipher encrypted
     messages. This was the birth of both statistics and cryptanalysis, according to the Saudi engineer Ibrahim
     Al-Kadi.[10][11]
     The Nuova Cronica, a 14th century history of Florence by the Florentine banker and official Giovanni Villani,
     includes much statistical information on population, ordinances, commerce, education, and religious facilities, and
     has been described as the first introduction of statistics as a positive element in history.[12]
     Some scholars pinpoint the origin of statistics to 1663, with the publication of Natural and Political Observations
     upon the Bills of Mortality by John Graunt.[13] Early applications of statistical thinking revolved around the needs of
     states to base policy on demographic and economic data, hence its stat- etymology. The scope of the discipline of
     statistics broadened in the early 19th century to include the collection and analysis of data in general. Today,
     statistics is widely employed in government, business, and natural and social sciences.
     Its mathematical foundations were laid in the 17th century with the development of the probability theory by Blaise
     Pascal and Pierre de Fermat. Probability theory arose from the study of games of chance. The method of least
     squares was first described by Carl Friedrich Gauss around 1794. The use of modern computers has expedited
     large-scale statistical computation, and has also made possible new methods that are impractical to perform
     manually.


     Overview
     In applying statistics to a scientific, industrial, or societal problem, it is necessary to begin with a population or
     process to be studied. Populations can be diverse topics such as "all persons living in a country" or "every atom
     composing a crystal". A population can also be composed of observations of a process at various times, with the data
     from each observation serving as a different member of the overall group. Data collected about this kind of
     "population" constitutes what is called a time series.
Statistics                                                                                                                                       32


     For practical reasons, a chosen subset of the population called a sample is studied—as opposed to compiling data
     about the entire group (an operation called census). Once a sample that is representative of the population is
     determined, data is collected for the sample members in an observational or experimental setting. This data can then
     be subjected to statistical analysis, serving two related purposes: description and inference.
     • Descriptive statistics summarize the population data by describing what was observed in the sample numerically
       or graphically. Numerical descriptors include mean and standard deviation for continuous data types (like heights
       or weights), while frequency and percentage are more useful in terms of describing categorical data (like race).
     • Inferential statistics uses patterns in the sample data to draw inferences about the population represented,
       accounting for randomness. These inferences may take the form of: answering yes/no questions about the data
       (hypothesis testing), estimating numerical characteristics of the data (estimation), describing associations within
       the data (correlation) and modeling relationships within the data (for example, using regression analysis).
       Inference can extend to forecasting, prediction and estimation of unobserved values either in or associated with
       the population being studied; it can include extrapolation and interpolation of time series or spatial data, and can
       also include data mining.[14]
     "... it is only the manipulation of uncertainty that interests us. We are not concerned with the matter that is uncertain. Thus we do not
     study the mechanism of rain; only whether it will rain."
                            [15]
     Dennis Lindley, 2000

     The concept of correlation is particularly noteworthy for the potential confusion it can cause. Statistical analysis of a
     data set often reveals that two variables (properties) of the population under consideration tend to vary together, as if
     they were connected. For example, a study of annual income that also looks at age of death might find that poor
     people tend to have shorter lives than affluent people. The two variables are said to be correlated; however, they may
     or may not be the cause of one another. The correlation phenomena could be caused by a third, previously
     unconsidered phenomenon, called a lurking variable or confounding variable. For this reason, there is no way to
     immediately infer the existence of a causal relationship between the two variables. (See Correlation does not imply
     causation.)
     To use a sample as a guide to an entire population, it is important that it truly representat that overall population.
     Representative sampling assures that inferences and conclusions can safely extend from the sample to the population
     as a whole. A major problem lies in determining the extent that the sample chosen is actually representative.
     Statistics offers methods to estimate and correct for any random trending within the sample and data collection
     procedures. There are also methods of experimental design for experiments that can lessen these issues at the outset
     of a study, strengthening its capability to discern truths about the population.
     Randomness is studied using the mathematical discipline of probability theory. Probability is used in "mathematical
     statistics" (alternatively, "statistical theory") to study the sampling distributions of sample statistics and, more
     generally, the properties of statistical procedures. The use of any statistical method is valid when the system or
     population under consideration satisfies the assumptions of the method.
     Misuse of statistics can produce subtle, but serious errors in description and interpretation—subtle in the sense that
     even experienced professionals make such errors, and serious in the sense that they can lead to devastating decision
     errors. For instance, social policy, medical practice, and the reliability of structures like bridges all rely on the proper
     use of statistics. See below for further discussion.
     Even when statistical techniques are correctly applied, the results can be difficult to interpret for those lacking
     expertise. The statistical significance of a trend in the data—which measures the extent to which a trend could be
     caused by random variation in the sample—may or may not agree with an intuitive sense of its significance. The set
     of basic statistical skills (and skepticism) that people need to deal with information in their everyday lives properly is
     referred to as statistical literacy.
Statistics                                                                                                                     33


     Statistical methods

     Experimental and observational studies
     A common goal for a statistical research project is to investigate causality, and in particular to draw a conclusion on
     the effect of changes in the values of predictors or independent variables on dependent variables or response. There
     are two major types of causal statistical studies: experimental studies and observational studies. In both types of
     studies, the effect of differences of an independent variable (or variables) on the behavior of the dependent variable
     are observed. The difference between the two types lies in how the study is actually conducted. Each can be very
     effective. An experimental study involves taking measurements of the system under study, manipulating the system,
     and then taking additional measurements using the same procedure to determine if the manipulation has modified the
     values of the measurements. In contrast, an observational study does not involve experimental manipulation. Instead,
     data are gathered and correlations between predictors and response are investigated.

     Experiments
     The basic steps of a statistical experiment are:
     1. Planning the research, including finding the number of replicates of the study, using the following information:
        preliminary estimates regarding the size of treatment effects, alternative hypotheses, and the estimated
        experimental variability. Consideration of the selection of experimental subjects and the ethics of research is
        necessary. Statisticians recommend that experiments compare (at least) one new treatment with a standard
        treatment or control, to allow an unbiased estimate of the difference in treatment effects.
     2. Design of experiments, using blocking to reduce the influence of confounding variables, and randomized
        assignment of treatments to subjects to allow unbiased estimates of treatment effects and experimental error. At
        this stage, the experimenters and statisticians write the experimental protocol that shall guide the performance of
        the experiment and that specifies the primary analysis of the experimental data.
     3. Performing the experiment following the experimental protocol and analyzing the data following the
        experimental protocol.
     4. Further examining the data set in secondary analyses, to suggest new hypotheses for future study.
     5. Documenting and presenting the results of the study.
     Experiments on human behavior have special concerns. The famous Hawthorne study examined changes to the
     working environment at the Hawthorne plant of the Western Electric Company. The researchers were interested in
     determining whether increased illumination would increase the productivity of the assembly line workers. The
     researchers first measured the productivity in the plant, then modified the illumination in an area of the plant and
     checked if the changes in illumination affected productivity. It turned out that productivity indeed improved (under
     the experimental conditions). However, the study is heavily criticized today for errors in experimental procedures,
     specifically for the lack of a control group and blindness. The Hawthorne effect refers to finding that an outcome (in
     this case, worker productivity) changed due to observation itself. Those in the Hawthorne study became more
     productive not because the lighting was changed but because they were being observed.
Statistics                                                                                                                      34


     Observational study
     An example of an observational study is one that explores the correlation between smoking and lung cancer. This
     type of study typically uses a survey to collect observations about the area of interest and then performs statistical
     analysis. In this case, the researchers would collect observations of both smokers and non-smokers, perhaps through
     a case-control study, and then look for the number of cases of lung cancer in each group.


     Levels of measurement
     There are four main levels of measurement used in statistics: nominal, ordinal, interval, and ratio.[16] Each of these
     have different degrees of usefulness in statistical research. Ratio measurements have both a meaningful zero value
     and the distances between different measurements defined; they provide the greatest flexibility in statistical methods
     that can be used for analyzing the data. Interval measurements have meaningful distances between measurements
     defined, but the zero value is arbitrary (as in the case with longitude and temperature measurements in Celsius or
     Fahrenheit). Ordinal measurements have imprecise differences between consecutive values, but have a meaningful
     order to those values. Nominal measurements have no meaningful rank order among values.
     Because variables conforming only to nominal or ordinal measurements cannot be reasonably measured numerically,
     sometimes they are grouped together as categorical variables, whereas ratio and interval measurements are grouped
     together as quantitative variables, which can be either discrete or continuous, due to their numerical nature.


     Key terms used in statistics

     Null hypothesis
     Interpretation of statistical information can often involve the development of a null hypothesis in that the assumption
     is that whatever is proposed as a cause has no effect on the variable being measured.
     The best illustration for a novice is the predicament encountered by a jury trial. The null hypothesis, H0, asserts that
     the defendant is innocent, whereas the alternative hypothesis, H1, asserts that the defendant is guilty. The indictment
     comes because of suspicion of the guilt. The H0 (status quo) stands in opposition to H1 and is maintained unless H1 is
     supported by evidence"beyond a reasonable doubt". However,"failure to reject H0" in this case does not imply
     innocence, but merely that the evidence was insufficient to convict. So the jury does not necessarily accept H0 but
     fails to reject H0. While one can not "prove" a null hypothesis one can test how close it is to being true with a power
     test, which tests for type II errors.

     Error
     Working from a null hypothesis two basic forms of error are recognized:
     • Type I errors where the null hypothesis is falsely rejected giving a "false positive".
     • Type II errors where the null hypothesis fails to be rejected and an actual difference between populations is
       missed giving a false negative.
     Error also refers to the extent to which individual observations in a sample differ from a central value, such as the
     sample or population mean. Many statistical methods seek to minimize the mean-squared error, and these are called
     "methods of least squares."
     Measurement processes that generate statistical data are also subject to error. Many of these errors are classified as
     random (noise) or systematic (bias), but other important types of errors (e.g., blunder, such as when an analyst
     reports incorrect units) can also be important.
Statistics                                                                                                                            35


     Interval estimation
     Most studies only sample part of a population , so results don't fully represent the whole population. Any estimates
     obtained from the sample only approximate the population value. Confidence intervals allow statisticians to express
     how closely the sample estimate matches the true value in the whole population. Often they are expressed as 95%
     confidence intervals. Formally, a 95% confidence interval for a value is a range where, if the sampling and analysis
     were repeated under the same conditions (yielding a different dataset), the interval would include the true
     (population) value 95% of the time. This does not imply that the probability that the true value is in the confidence
     interval is 95%. From the frequentist perspective, such a claim does not even make sense, as the true value is not a
     random variable. Either the true value is or is not within the given interval. However, it is true that, before any data
     are sampled and given a plan for how to construct the confidence interval, the probability is 95% that the
     yet-to-be-calculated interval will cover the true value: at this point, the limits of the interval are yet-to-be-observed
     random variables. One approach that does yield an interval that can be interpreted as having a given probability of
     containing the true value is to use a credible interval from Bayesian statistics: this approach depends on a different
     way of interpreting what is meant by "probability", that is as a Bayesian probability.

     Significance
     Statistics rarely give a simple Yes/No type answer to the question asked of them. Interpretation often comes down to
     the level of statistical significance applied to the numbers and often refers to the probability of a value accurately
     rejecting the null hypothesis (sometimes referred to as the p-value).
     Referring to statistical significance does not necessarily mean that the overall result is significant in real world terms.
     For example, in a large study of a drug it may be shown that the drug has a statistically significant but very small
     beneficial effect, such that the drug is unlikely to help the patient noticeably.
     Criticisms arise because the hypothesis testing approach forces one hypothesis (the null hypothesis) to be "favored,"
     and can also seem to exaggerate the importance of minor differences in large studies. A difference that is highly
     statistically significant can still be of no practical significance, but it is possible to properly formulate tests in account
     for this. (See also criticism of hypothesis testing.)
     One response involves going beyond reporting only the significance level to include the p-value when reporting
     whether a hypothesis is rejected or accepted. The p-value, however, does not indicate the size of the effect. A better
     and increasingly common approach is to report confidence intervals. Although these are produced from the same
     calculations as those of hypothesis tests or p-values, they describe both the size of the effect and the uncertainty
     surrounding it.


     Examples
     Some well-known statistical tests and procedures are:
     •   Analysis of variance (ANOVA)
     •   Chi-squared test
     •   Correlation
     •   Factor analysis
     •   Mann–Whitney U
     •   Mean square weighted deviation (MSWD)
     •   Pearson product-moment correlation coefficient
     •   Regression analysis
     •   Spearman's rank correlation coefficient
     •   Student's t-test
     • Time series analysis
Statistics                                                                                                                    36


     Specialized disciplines
     Statistical techniques are used in a wide range of types of scientific and social research, including: biostatistics,
     computational biology, computational sociology, network biology, social science, sociology and social research.
     Some fields of inquiry use applied statistics so extensively that they have specialized terminology. These disciplines
     include:
     •   Actuarial science (assesses risk in the insurance and finance industries)
     •   Applied information economics
     •   Biostatistics
     •   Business statistics
     •   Chemometrics (for analysis of data from chemistry)
     •   Data mining (applying statistics and pattern recognition to discover knowledge from data)
     •   Demography
     •   Econometrics
     •   Energy statistics
     •   Engineering statistics
     •   Epidemiology
     •   Geography and Geographic Information Systems, specifically in Spatial analysis
     •   Image processing
     •   Psychological statistics
     •   Reliability engineering
     •   Social statistics
     In addition, there are particular types of statistical analysis that have also developed their own specialised
     terminology and methodology:
     •   Bootstrap & Jackknife Resampling
     •   Multivariate statistics
     •   Statistical classification
     •   Statistical surveys
     •   Structured data analysis (statistics)
     •   Structural equation modelling
     •   Survival analysis
     •   Statistics in various sports, particularly baseball and cricket
     Statistics form a key basis tool in business and manufacturing as well. It is used to understand measurement systems
     variability, control processes (as in statistical process control or SPC), for summarizing data, and to make
     data-driven decisions. In these roles, it is a key tool, and perhaps the only reliable tool.
Statistics                                                                                                                         37


     Statistical computing
     The rapid and sustained increases in
     computing power starting from the second
     half of the 20th century have had a
     substantial impact on the practice of
     statistical science. Early statistical models
     were almost always from the class of linear
     models, but powerful computers, coupled
     with suitable numerical algorithms, caused
     an increased interest in nonlinear models
     (such as neural networks) as well as the
     creation of new types, such as generalized
     linear models and multilevel models.

     Increased computing power has also led to
     the growing popularity of computationally
     intensive methods based on resampling,
     such as permutation tests and the bootstrap,               gretl, an example of an open source statistical package
     while techniques such as Gibbs sampling
     have made use of Bayesian models more feasible. The computer revolution has implications for the future of
     statistics with new emphasis on "experimental" and "empirical" statistics. A large number of both general and special
     purpose statistical software are now available.


     Misuse
     There is a general perception that statistical knowledge is all-too-frequently intentionally misused by finding ways to
     interpret only the data that are favorable to the presenter.[17] A mistrust and misunderstanding of statistics is
     associated with the quotation, "There are three kinds of lies: lies, damned lies, and statistics". Misuse of statistics can
     be both inadvertent and intentional, and the book How to Lie With Statistics[17] outlines a range of considerations. In
     an attempt to shed light on the use and misuse of statistics, reviews of statistical techniques used in particular fields
     are conducted (e.g. Warne, Lazo, Ramos, and Ritter (2012)).[18]
     Ways to avoid misuse of statistics include using proper diagrams and avoiding bias.[19] "The misuse occurs when
     such conclusions are held to be representative of the universe by those who either deliberately or unconsciously
     overlook the sampling bias.[20]" Bar graphs are arguably the easiest diagrams to use and understand, and they can be
     made either with simple computer programs or hand drawn.[19] Unfortunately, most people do not look for bias or
     errors, so they do not see them. Thus, we believe something true that is not well-represented.[20] To make data
     gathered from statistics believable and accurate, the sample taken must be representative of the whole.[21] As Huff's
     book states,"The dependability of a sample can be destroyed by [bias]… allow yourself some degree of
     skepticism."[22]
Statistics                                                                                                                                             38


     Statistics applied to mathematics or the arts
     Traditionally, statistics was concerned with drawing inferences using a semi-standardized methodology that was
     "required learning" in most sciences. This has changed with use of statistics in non-inferential contexts. What was
     once considered a dry subject, taken in many fields as a degree-requirement, is now viewed enthusiastically. Initially
     derided by some mathematical purists, it is now considered essential methodology in certain areas.
     • In number theory, scatter plots of data generated by a distribution function may be transformed with familiar tools
       used in statistics to reveal underlying patterns, which may then lead to hypotheses.
     • Methods of statistics including predictive methods in forecasting are combined with chaos theory and fractal
       geometry to create video works that are considered to have great beauty.
     • The process art of Jackson Pollock relied on artistic experiments whereby underlying distributions in nature were
       artistically revealed. With the advent of computers, methods of statistics were applied to formalize such
       distribution driven natural processes, in order to make and analyze moving video art.
     • Methods of statistics may be used predicatively in performance art, as in a card trick based on a Markov process
       that only works some of the time, the occasion of which can be predicted using statistical methodology.
     • Statistics can be used to predicatively create art, as in the statistical or stochastic music invented by Iannis
       Xenakis, where the music is performance-specific. Though this type of artistry does not always come out as
       expected, it does behave in ways that are predictable and tunable using statistics.


     References
     [1]  Dodge, Y. (2006) The Oxford Dictionary of Statistical Terms, OUP. ISBN 0-19-920613-9
     [2]  The Free Online Dictionary (http:/ / www. thefreedictionary. com/ dict. asp?Word=statistics)
     [3]  "Statistics" (http:/ / www. merriam-webster. com/ dictionary/ statistics). Merriam-Webster Online Dictionary. .
     [4]  "Statistic" (http:/ / www. merriam-webster. com/ dictionary/ statistic). Merriam-Webster Online Dictionary. .
     [5]  Moses, Lincoln E. (1986) Think and Explain with Statistics, Addison-Wesley, ISBN 978-0-201-15619-5 . pp. 1–3
     [6]  Hays, William Lee, (1973) Statistics for the Social Sciences, Holt, Rinehart and Winston, p.xii, ISBN 978-0-03-077945-9
     [7]  Moore, David (1992). "Teaching Statistics as a Respectable Subject". In F. Gordon and S. Gordon. Statistics for the Twenty-First Century.
         Washington, DC: The Mathematical Association of America. pp. 14–25. ISBN 978-0-88385-078-7.
     [8] Chance, Beth L.; Rossman, Allan J. (2005). "Preface" (http:/ / www. rossmanchance. com/ iscam/ preface. pdf). Investigating Statistical
         Concepts, Applications, and Methods. Duxbury Press. ISBN 978-0-495-05064-3. .
     [9] Anderson, D.R.; Sweeney, D.J.; Williams, T.A.. (1994) Introduction to Statistics: Concepts and Applications, pp. 5–9. West Group. ISBN
         978-0-314-03309-3
     [10] Al-Kadi, Ibrahim A. (1992) "The origins of cryptology: The Arab contributions”, Cryptologia, 16(2) 97–126.
         doi:10.1080/0161-119291866801
     [11] Singh, Simon (2000). The code book : the science of secrecy from ancient Egypt to quantum cryptography (1st Anchor Books ed.). New
         York: Anchor Books. ISBN 0-385-49532-3.
     [12] Villani, Giovanni. Encyclopædia Britannica. Encyclopædia Britannica 2006 Ultimate Reference Suite DVD. Retrieved on 2008-03-04.
     [13] Willcox, Walter (1938) "The Founder of Statistics". Review of the International Statistical Institute 5(4):321–328. JSTOR 1400906
     [14] Breiman, Leo (2001). "Statistical Modelling: the two cultures". Statistical Science 16 (3): 199–231. doi:10.1214/ss/1009213726.
         MR1874152. CiteSeerX: 10.1.1.156.4933 (http:/ / citeseerx. ist. psu. edu/ viewdoc/ summary?doi=10. 1. 1. 156. 4933).
     [15] Lindley, D. (2000). "The Philosophy of Statistics". Journal of the Royal Statistical Society, Series D (The Statistician) 49 (3): 293–337.
         doi:10.1111/1467-9884.00238. JSTOR 2681060.
     [16] Thompson, B. (2006). Foundations of behavioral statistics. New York, NY: Guilford Press.
     [17] Huff, Darrell (1954) How to Lie With Statistics, WW Norton & Company, Inc. New York, NY. ISBN 0-393-31072-8
     [18] Warne, R. Lazo, M., Ramos, T. and Ritter, N. (2012). Statistical Methods Used in Gifted Education Journals, 2006–2010. Gifted Child
         Quarterly, 56(3) 134–149. doi:10.1177/0016986212444122
     [19] Encyclopedia of Archaeology. Credo Reference: Oxford: Elsevier Science. 2008.
     [20] Cohen, Jerome B. (December 1938). "Misuse of Statistics". Journal of the American Statistical Association (JSTOR) 33 (204): 657–674.
         doi:10.1080/01621459.1938.10502344.
     [21] Freund, J. F. (1988). "Modern Elementary Statistics". Credo Reference.
     [22] Huff, Darrell; Irving Geis (1954). How to Lie with Statistics. New York: Norton. "The dependability of a sample can be destroyed by
         [bias]… allow yourself some degree of skepticism."
Bayesian inference                                                                                                                 39



    Bayesian inference
    In statistics, Bayesian inference is a method of inference in which Bayes' rule is used to update the probability
    estimate for a hypothesis as additional evidence is learned. Bayesian updating is an important technique throughout
    statistics, and especially in mathematical statistics: For some cases, exhibiting a Bayesian derivation for a statistical
    method automatically ensures that the method works as well as any competing method. Bayesian updating is
    especially important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a
    range of fields including science, engineering, medicine, and law.
    In the philosophy of decision theory, Bayesian inference is closely related to discussions of subjective probability,
    often called "Bayesian probability." Bayesian probability provides a rational method for updating beliefs;[1][2]
    however, non-Bayesian updating rules are compatible with rationality, according to philosophers Ian Hacking and
    Bas van Fraassen.


    Introduction to Bayes' rule

    Formal
    Bayesian inference derives the posterior probability as a consequence of two antecedents, a prior probability and a
    "likelihood function" derived from a probability model for the data to be observed. Bayesian inference computes the
    posterior probability according to Bayes' rule:




    where
    •     means given.
    •      stands for any hypothesis whose probability may be affected by data (called evidence below). Often there are
      competing hypotheses, from which one chooses the most probable.
    • the evidence  corresponds to data that were not used in computing the prior probability.
    •           , the prior probability, is the probability of    before        is observed. This indicates one's preconceived
        beliefs about how likely different hypotheses are, absent evidence regarding the instance under study.
    •              , the posterior probability, is the probability of  given    , i.e., after  is observed. This tells us
        what we want to know: the probability of a hypothesis given the observed evidence.
    •            , the probability of observing     given    , is also known as the likelihood. It indicates the
        compatibility of the evidence with the given hypothesis.
    •         is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible
        hypotheses being considered. (This can be seen by the fact that the hypothesis           does not appear anywhere in
        the symbol, unlike for all the other factors.) This means that this factor does not enter into determining the relative
        probabilities of different hypotheses.
    Note that what affects the value of               for different values of       is only the factors       and              ,
    which both appear in the numerator, and hence the posterior probability is proportional to both. In words:
    • (more exactly) The posterior probability of a hypothesis is determined by a combination of the inherent likeliness
      of a hypothesis (the prior) and the compatibility of the observed evidence with the hypothesis (the likelihood).
    • (more concisely) Posterior is proportional to prior times likelihood.
    Note that Bayes' rule can also be written as follows:
Bayesian inference                                                                                                               40


    where the factor          represents the impact of    on the probability of     .


    Informal
    Rationally, Bayes' rule makes a great deal of sense. If the evidence doesn't match up with a hypothesis, one should
    reject the hypothesis. But if a hypothesis is extremely unlikely a priori, one should also reject it, even if the evidence
    does appear to match up.
    For example, imagine that I have various hypotheses about the nature of a newborn baby of a friend, including:
    •       : the baby is a brown-haired boy.
    •       : the baby is a blond-haired girl.
    •       : the baby is a dog.
    Then consider two scenarios:
    1. I'm presented with evidence in the form of a picture of a blond-haired baby girl. I find this evidence supports
       and opposes     and     .
    2. I'm presented with evidence in the form of a picture of a baby dog. Although this evidence, treated in isolation,
       supports    , my prior belief in this hypothesis (that a human can give birth to a dog) is extremely small, so the
        posterior probability is nevertheless small.
    The critical point about Bayesian inference, then, is that it provides a principled way of combining new evidence
    with prior beliefs, through the application of Bayes' rule. (Contrast this with frequentist inference, which relies only
    on the evidence as a whole, with no reference to prior beliefs.) Furthermore, Bayes' rule can be applied iteratively:
    after observing some evidence, the resulting posterior probability can then be treated as a prior probability, and a
    new posterior probability computed from new evidence. This allows for Bayesian principles to be applied to various
    kinds of evidence, whether viewed all at once or over time. This procedure is termed Bayesian updating.


    Bayesian updating
    Bayesian updating is widely used and computationally convenient. However, it is not the only updating rule that
    might be considered "rational."
    Ian Hacking noted that traditional "Dutch book" arguments did not specify Bayesian updating: they left open the
    possibility that non-Bayesian updating rules could avoid Dutch books. Hacking wrote[3] "And neither the Dutch
    book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic
    assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is
    true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose
    its savour."
    Indeed, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on
    "probability kinematics" following the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the case
    where the evidence itself is assigned a probability [4]). The additional hypotheses needed to uniquely require
    Bayesian updating have been deemed to be substantial, complicated, and unsatisfactory.[5]
Bayesian inference                                                                                                               41


    Formal description of Bayesian inference

    Definitions
    •     , a data point in general. This may in fact be a vector of values.
    •     , the parameter of the data point's distribution, i.e.            . This may in fact be a vector of parameters.
    •     , the hyperparameter of the parameter, i.e.               . This may in fact be a vector of hyperparameters.
    •      , a set of observed data points, i.e.                 .
    •     , a new data point whose distribution is to be predicted.


    Bayesian inference
    • The prior distribution is the distribution of the parameter(s) before any data is observed, i.e.        .
    • The prior distribution might not be easily determined. In this case, we can use the Jeffreys prior to obtain the
      posterior distribution before updating them with newer observations.
    • The sampling distribution is the distribution of the observed data conditional on its parameters, i.e.            .
      This is also termed the likelihood, especially when viewed as a function of the parameter(s), sometimes written
                             .
    • The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data
        marginalized over the parameter(s), i.e.                                      .

    • The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is
      determined by Bayes' rule, which forms the heart of Bayesian inference:




    Note that this is expressed in words as "posterior is proportional to prior times likelihood", or sometimes as
    "posterior = prior times likelihood, over evidence".


    Bayesian prediction
    • The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior:




    • The prior predictive distribution is the distribution of a new data point, marginalized over the prior:




    Bayesian theory calls for the use of the posterior predictive distribution to do predictive inference, i.e. to predict the
    distribution of a new, unobserved data point. That is, instead of a fixed point as a prediction, a distribution over
    possible points is returned. Only this way is the entire posterior distribution of the parameter(s) used. By comparison,
    prediction in frequentist statistics often involves finding an optimum point estimate of the parameter(s) — e.g. by
    maximum likelihood or maximum a posteriori estimation (MAP) — and then plugging this estimate into the formula
    for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value
    of the parameter, and hence will underestimate the variance of the predictive distribution.
    (In some instances, frequentist statistics can work around this problem. For example, confidence intervals and
    prediction intervals in frequentist statistics when constructed from a normal distribution with unknown mean and
    variance are constructed using a Student's t-distribution. This correctly estimates the variance, due to the fact that (1)
    the average of normally distributed random variables is also normally distributed; (2) the predictive distribution of a
    normally distributed data point with unknown mean and variance, using conjugate or uninformative priors, has a
Bayesian inference                                                                                                                   42


    Student's t-distribution. In Bayesian statistics, however, the posterior predictive distribution can always be
    determined exactly — or at least, to an arbitrary level of precision, when numerical methods are used.)
    Note that both types of predictive distributions have the form of a compound probability distribution (as does the
    marginal likelihood). In fact, if the prior distribution is a conjugate prior, and hence the prior and posterior
    distributions come from the same family, it can easily be seen that both prior and posterior predictive distributions
    also come from the same family of compound distributions. The only difference is that the posterior predictive
    distribution uses the updated values of the hyperparameters (applying the Bayesian update rules given in the
    conjugate prior article), while the prior predictive distribution uses the values of the hyperparameters that appear in
    the prior distribution.


    Inference over exclusive and exhaustive possibilities
    If evidence is simultaneously used to update belief over a set of exclusive and exhaustive propositions, Bayesian
    inference may be thought of as acting on this belief distribution as a whole.


    General formulation
    Suppose a process is generating independent and identically distributed
    events    , but the probability distribution is unknown. Let the event
    space    represent the current state of belief for this process. Each
    model is represented by event              . The conditional probabilities
                     are specified to define the models.              is the
    degree of belief in       . Before the first inference step,
    is a set of initial prior probabilities. These must sum to 1, but are
    otherwise arbitrary.
    Suppose that the process is observed to generate               . For
    each                   , the prior           is updated to the posterior
                                         [6]
               . From Bayes' theorem:
                                                                                  Diagram illustrating event space    in general
                                                                                 formulation of Bayesian inference. Although this
                                                                                  diagram shows discrete models and events, the
    Upon observation of further evidence, this procedure may be repeated.        continuous case may be visualized similarly using
                                                                                               probability densities.

    Multiple observations
    For a set of independent and identically distributed observations                               , it may be shown that
    repeated application of the above is equivalent to



    Where



    This may be used to optimize practical calculations.
Bayesian inference                                                                                                               43


    Parametric formulation
    By parametrizing the space of models, the belief in all models may be updated in a single step. The distribution of
    belief over the model space may then be thought of as a distribution of belief over the parameter space. The
    distributions in this section are expressed as continuous, represented by probability densities, as this is the usual
    situation. The technique is however equally applicable to discrete distributions.
    Let the vector    span the parameter space. Let the initial prior distribution over    be         , where    is a set of
    parameters to the prior itself, or hyperparameters. Let                         be a set of independent and identically
    distributed event observations, where all       are distributed as        for some      . Bayes' theorem is applied to
    find the posterior distribution over   :




    Where




    Mathematical properties

    Interpretation of factor
                                               . That is, if the model were true, the evidence would be more likely than is
    predicted by the current state of belief. The reverse applies for a decrease in belief. If the belief does not change,
                                               . That is, the evidence is independent of the model. If the model were true,
    the evidence would be exactly as likely as predicted by the current state of belief.

    Cromwell's rule
    If               then                  . If              , then                 . This can be interpreted to mean that
    hard convictions are insensitive to counter-evidence.
    The former follows directly from Bayes' theorem. The latter can be derived by applying the first rule to the event
    "not    " in place of "     ," yielding "if                , then                     ," from which the result
    immediately follows.

    Asymptotic behaviour of posterior
    Consider the behaviour of a belief distribution as it is updated a large number of times with independent and
    identically distributed trials. For sufficiently nice prior probabilities, the Bernstein-von Mises theorem gives that in
    the limit of infinite trials and the posterior converges to a Gaussian distribution independent of the initial prior under
    some conditions firstly outlined and rigorously proven by Joseph Leo Doob in 1948, namely if the random variable
    in consideration has a finite probability space. The more general results were obtained later by the statistician David
    A. Freedman who published in two seminal research papers in 1963 and 1965 when and under what circumstances
    the asymptotic behaviour of posterior is guaranteed. His 1963 paper treats, like Doob (1949), the finite case and
    comes to a satisfactory conclusion. However, if the random variable has an infinite but countable probability space
    (i.e. corresponding to a die with infinite many faces) the 1965 paper demonstrates that for a dense subset of priors
    the Bernstein-von Mises theorem is not applicable. In this case there is almost surely no asymptotic convergence.
    Later in the eighties and nineties Freedman and Persi Diaconis continued to work on the case of infinite countable
Bayesian inference                                                                                                             44


    probability spaces.[7] To summarise, there may be insufficient trials to suppress the effects of the initial choice, and
    especially for large (but finite) systems the convergence might be very slow.


    Conjugate priors
    In paramaterized form, the prior distribution is often assumed to come from a family of distributions called conjugate
    priors. The usefulness of a conjugate prior is that the corresponding posterior distribution will be in the same family,
    and the calculation may be expressed in closed form.


    Estimates of parameters and predictions
    It is often desired to use a posterior distribution to estimate a parameter or variable. Several methods of Bayesian
    estimation select measurements of central tendency from the posterior distribution.
    First, when the parameter space has two dimensions or more, there exists a unique median of the posterior
    distribution. For one-dimensional problems, a unique median exists for practical continuous problems. The posterior
    median is attractive as a robust estimator.[8]
    Second, if there exists a finite mean for the posterior distribution, then the posterior mean is a method of estimation.




    Third, taking a value with the greatest probability defines any set of maximum a posteriori (MAP) estimates:


    There are examples where no maximum is attained, in which case the set of MAP estimates is empty.
    There are other methods of estimation that minimize the posterior risk (expected-posterior loss) with respect to a loss
    function, and these are of interest to statistical decision theory using the sampling distribution ("frequentist
    statistics").
    The posterior predictive distribution of a new observation       (that is exchangeable with previous observations) is
    determined by




    Examples

    Probability of a hypothesis
    Suppose there are two full bowls of cookies. Bowl #1 has 10 chocolate chip and 30 plain cookies, while bowl #2 has
    20 of each. Our friend Fred picks a bowl at random, and then picks a cookie at random. We may assume there is no
    reason to believe Fred treats one bowl differently from another, likewise for the cookies. The cookie turns out to be a
    plain one. How probable is it that Fred picked it out of bowl #1?
    Intuitively, it seems clear that the answer should be more than a half, since there are more plain cookies in bowl #1.
    The precise answer is given by Bayes' theorem. Let       correspond to bowl #1, and       to bowl #2. It is given that
    the bowls are identical from Fred's point of view, thus                        , and the two must add up to 1, so both
    are equal to 0.5. The event     is the observation of a plain cookie. From the contents of the bowls, we know that
                                     and                                 . Bayes' formula then yields
Bayesian inference                                                                                                                 45




    Before we observed the cookie, the probability we assigned for Fred having chosen bowl #1 was the prior
    probability,      , which was 0.5. After observing the cookie, we must revise the probability to     ,
    which is 0.6.

    Making a prediction
    An archaeologist is working at a site thought to be from the medieval
    period, between the 11th century to the 16th century. However, it is
    uncertain exactly when in this period the site was inhabited. Fragments
    of pottery are found, some of which are glazed and some of which are
    decorated. It is expected that if the site were inhabited during the early
    medieval period, then 1% of the pottery would be glazed and 50% of
    its area decorated, whereas if it had been inhabited in the late medieval
    period then 81% would be glazed and 5% of its area decorated. How              Example results for archaeology example. This
                                                                                      simulation was generated using c=15.2.
    confident can the archaeologist be in the date of inhabitation as
    fragments are unearthed?
    The degree of belief in the continuous variable        (century) is to be calculated, with the discrete set of events
                                 as evidence. Assuming linear variation of glaze and decoration with time, and that these
    variables are independent,




    Assume a uniform prior of                   , and that trials are independent and identically distributed. When a new
    fragment of type    is discovered, Bayes' theorem is applied to update the degree of belief for each       :



    A computer simulation of the changing belief as 50 fragments are unearthed is shown on the graph. In the
    simulation, the site was inhabited around 1520, or     . By calculating the area under the relevant portion of
    the graph for 50 trials, the archaeologist can say that there is practically no chance the site was inhabited in the 11th
    and 12th centuries, about 1% chance that it was inhabited during the 13th century, 63% chance during the 14th
    century and 36% during the 15th century. Note that the Bernstein-von Mises theorem asserts here the asymptotic
    convergence to the "true" distribution because the probability space corresponding to the discrete set of events
                                 is finite (see above section on asymptotic behaviour of the posterior).
Bayesian inference                                                                                                           46


    In frequentist statistics and decision theory
    A decision-theoretic justification of the use of Bayesian inference was given by Abraham Wald, who proved that
    every Bayesian procedure is admissible. Conversely, every admissible statistical procedure is either a Bayesian
    procedure or a limit of Bayesian procedures.[9]
    Wald characterized admissible procedures as Bayesian procedures (and limits of Bayesian procedures), making the
    Bayesian formalism a central technique in such areas of frequentist inference as parameter estimation, hypothesis
    testing, and computing confidence intervals.[10] For example:
    • "Under some conditions, all admissible procedures are either Bayes procedures or limits of Bayes procedures (in
      various senses). These remarkable results, at least in their original form, are due essentially to Wald. They are
      useful because the property of being Bayes is easier to analyze than admissibility."[9]
    • "In decision theory, a quite general method for proving admissibility consists in exhibiting a procedure as a
      unique Bayes solution."[11]
    • "In the first chapters of this work, prior distributions with finite support and the corresponding Bayes procedures
      were used to establish some of the main theorems relating to the comparison of experiments. Bayes procedures
      with respect to more general prior distributions have played a very important role in the development of statistics,
      including its asymptotic theory." "There are many problems where a glance at posterior distributions, for suitable
      priors, yields immediately interesting information. Also, this technique can hardly be avoided in sequential
      analysis."[12]
    • "A useful fact is that any Bayes decision rule obtained by taking a proper prior over the whole parameter space
      must be admissible"[13]
    • "An important area of investigation in the development of admissibility ideas has been that of conventional
      sampling-theory procedures, and many interesting results have been obtained."[14]


    Applications

    Computer applications
    Bayesian inference has applications in artificial intelligence and expert systems. Bayesian inference techniques have
    been a fundamental part of computerized pattern recognition techniques since the late 1950s. There is also an ever
    growing connection between Bayesian methods and simulation-based Monte Carlo techniques since complex models
    cannot be processed in closed form by a Bayesian analysis, while a graphical model structure may allow for efficient
    simulation algorithms like the Gibbs sampling and other Metropolis–Hastings algorithm schemes.[15] Recently
    Bayesian inference has gained popularity amongst the phylogenetics community for these reasons; a number of
    applications allow many demographic and evolutionary parameters to be estimated simultaneously. In the areas of
    population genetics and dynamical systems theory, approximate Bayesian computation (ABC) is also becoming
    increasingly popular.
    As applied to statistical classification, Bayesian inference has been used in recent years to develop algorithms for
    identifying e-mail spam. Applications which make use of Bayesian inference for spam filtering include CRM114,
    DSPAM, Bogofilter, SpamAssassin, SpamBayes, and Mozilla. Spam classification is treated in more detail in the
    article on the naive Bayes classifier.
    Solomonoff's Inductive inference is the theory of prediction based on observations; for example, predicting the next
    symbol based upon a given series of symbols. The only assumption is that the environment follows some unknown
    but computable probability distribution. It is a formal inductive framework that combines two well-studied principles
    of inductive inference: Bayesian statistics and Occam’s Razor.[16] Solomonoff's universal prior probability of any
    prefix p of a computable sequence x is the sum of the probabilities of all programs (for a universal computer) that
    compute something starting with p. Given some p and any computable but unknown probability distribution from
Bayesian inference                                                                                                               47


    which x is sampled, the universal prior and Bayes' theorem can be used to predict the yet unseen parts of x in optimal
    fashion. [17][18]


    In the courtroom
    Bayesian inference can be used by jurors to coherently accumulate the evidence for and against a defendant, and to
    see whether, in totality, it meets their personal threshold for 'beyond a reasonable doubt'.[19][20][21] Bayes' theorem is
    applied successively to all evidence presented, with the posterior from one stage becoming the prior for the next. The
    benefit of a Bayesian approach is that it gives the juror an unbiased, rational mechanism for combining evidence. It
    may be appropriate to explain Bayes' theorem to jurors in odds form, as betting odds are more widely understood
    than probabilities. Alternatively, a logarithmic approach, replacing multiplication with addition, might be easier for a
    jury to handle.
    If the existence of the crime is not in doubt, only the
    identity of the culprit, it has been suggested that the
    prior should be uniform over the qualifying
    population.[22] For example, if 1000 people could
    have committed the crime, the prior probability of
    guilt would be 1/1000.

    The use of Bayes' theorem by jurors is controversial.
    In the United Kingdom, a defence expert witness
    explained Bayes' theorem to the jury in R v Adams.
    The jury convicted, but the case went to appeal on the
    basis that no means of accumulating evidence had
    been provided for jurors who did not wish to use
    Bayes' theorem. The Court of Appeal upheld the
    conviction, but it also gave the opinion that "To
    introduce Bayes' Theorem, or any similar method,
    into a criminal trial plunges the jury into
    inappropriate and unnecessary realms of theory and                                Adding up evidence.
    complexity, deflecting them from their proper task."

    Gardner-Medwin[23] argues that the criterion on which a verdict in a criminal trial should be based is not the
    probability of guilt, but rather the probability of the evidence, given that the defendant is innocent (akin to a
    frequentist p-value). He argues that if the posterior probability of guilt is to be computed by Bayes' theorem, the prior
    probability of guilt must be known. This will depend on the incidence of the crime, which is an unusual piece of
    evidence to consider in a criminal trial. Consider the following three propositions:
          A The known facts and testimony could have arisen if the defendant is guilty
          B The known facts and testimony could have arisen if the defendant is innocent
          C The defendant is guilty.
    Gardner-Medwin argues that the jury should believe both A and not-B in order to convict. A and not-B implies the
    truth of C, but the reverse is not true. It is possible that B and C are both true, but in this case he argues that a jury
    should acquit, even though they know that they will be letting some guilty people go free. See also Lindley's
    paradox.
Bayesian inference                                                                                                                  48


    Bayesian Epistemology
    Bayesian Epistemology is an epistemological movement that uses techniques of Bayesian inference as a means of
    justifying the rules of inductive logic.
    Karl Popper and David Miller have rejected the alleged rationality of Bayesianism, ie., using Bayes rule to make
    epistemological inferences:[24] It is prone to the same vicious circle as any other justificationist epistemology,
    because it presupposes what it attempts to justify. According to this view, a rational interpretation of Bayesian
    inference would see it merely as a probabilistic version of falsification, rejecting the belief, commonly held by
    Bayesianists, that high likelyhood achieved by a series of Bayesian updates would prove the hypothesis beyond any
    reasonable doubt, or even with likelyhood greater than 0.


    Other
    • The scientific method is sometimes interpreted as an application of Bayesian inference. In this view, Bayes' rule
      guides (or should guide) the updating of probabilities about hypotheses conditional on new observations or
      experiments.[25]
    • Bayesian search theory is used to search for lost objects.
    • Bayesian inference in phylogeny
    • Bayesian tool for methylation analysis


    Bayes and Bayesian inference
    The problem considered by Bayes in Proposition 9 of his essay, "An Essay towards solving a Problem in the
    Doctrine of Chances", is the posterior distribution for the parameter a (the success rate) of the binomial distribution.
    What is "Bayesian" about Proposition 9 is that Bayes presented it as a probability for the parameter . That is, not
    only can one compute probabilities for experimental outcomes, but also for the parameter which governs them, and
    the same algebra is used to make inferences of either kind. Interestingly, Bayes actually states his question in a way
    that might make the idea of assigning a probability distribution to a parameter palatable to a frequentist. He supposes
    that a billiard ball is thrown at random onto a billiard table, and that the probabilities p and q are the probabilities that
    subsequent billiard balls will fall above or below the first ball. By making the binomial parameter depend on a
    random event, he cleverly escapes a philosophical quagmire that was an issue he most likely was not even aware of.


    History
    The term Bayesian refers to Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes'
    theorem. However, it was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and
    used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence.[26] Early
    Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called
    "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes[27]).
    After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called
    frequentist statistics.[27]
    In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective
    and subjective currents in Bayesian practice. In the objective or "non-informative" current, the statistical analysis
    depends on only the model assumed, the data analyzed,[28] and the method assigning the prior, which differs from
    one objective Bayesian to another objective Bayesian. In the subjective or "informative" current, the specification of
    the prior depends on the belief (that is, propositions on which the analysis is prepared to act), which can summarize
    information from experts, previous studies, etc.
    In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the
    discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an
Bayesian inference                                                                                                                                      49


    increasing interest in nonstandard, complex applications.[29] Despite growth of Bayesian research, most
    undergraduate teaching is still based on frequentist statistics.[30] Nonetheless, Bayesian methods are widely accepted
    and used, such as for example in the field of machine learning.[31]


    Notes
    [1]  Stanford encyclopedia of philosophy; Bayesian Epistemology; http:/ / plato. stanford. edu/ entries/ epistemology-bayesian
    [2]  Gillies, Donald (2000); "Philosophical Theories of Probability"; Routledge; Chapter 4 "The subjective theory"
    [3]  Hacking (1967, Section 3, page 316), Hacking (1988, page 124)
    [4]  http:/ / plato. stanford. edu/ entries/ bayes-theorem/
    [5]  van Fraassen, B. (1989) Laws and Symmetry, Oxford University Press. ISBN 0-19-824860-1
    [6]  Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Rubin, Donald B. (2003). Bayesian Data Analysis, Second Edition. Boca Raton, FL:
        Chapman and Hall/CRC. ISBN 1-58488-388-X.
    [7] Larry Wasserman et alia, JASA 2000.
    [8] Sen, Pranab K.; Keating, J. P.; Mason, R. L. (1993). Pitman's measure of closeness: A comparison of statistical estimators. Philadelphia:
        SIAM.
    [9] Bickel & Doksum (2001, page 32)
    [10] * Kiefer, J. and Schwartz, R. (1965). "Admissible Bayes character of T2-, R2-, and other fully invariant tests for multivariate normal
        problems". Annals of Mathematical Statistics 36: 747–770. doi:10.1214/aoms/1177700051.
          •   Schwartz, R. (1969). "Invariant proper Bayes tests for exponential families". Annals of Mathematical Statistics 40: 270–283.
              doi:10.1214/aoms/1177697822.
          • Hwang, J. T. and Casella, George (1982). "Minimax confidence sets for the mean of a multivariate normal distribution". Annals of
            Statistics 10: 868–881. doi:10.1214/aos/1176345877.
    [11] Lehmann, Erich (1986). Testing Statistical Hypotheses (Second ed.). (see page 309 of Chapter 6.7 "Admissibilty", and pages 17–18 of
        Chapter 1.8 "Complete Classes"
    [12] Le Cam, Lucien (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag. ISBN 0-387-96307-3. (From "Chapter 12
        Posterior Distributions and Bayes Solutions", page 324)
    [13] Cox, D. R. and Hinkley, D. V (1974). Theoretical Statistics. Chapman and Hall. ISBN 0-04-121537-0. page 432
    [14] Cox, D. R. and Hinkley, D. V (1974). Theoretical Statistics. Chapman and Hall. ISBN 0-04-121537-0. page 433)
    [15] Jim Albert (2009). Bayesian Computation with R, Second edition. New York, Dordrecht, etc.: Springer. ISBN 978-0-387-92297-3.
    [16] Samuel Rathmanner and Marcus Hutter. A philosophical treatise of universal induction. Entropy, 13(6):1076–1136, 2011.
    [17] "The problem of old evidence" ,in §5 of "On universal prediction and Bayesian confirmation", M Hutter - Theoretical Computer Science,
        2007 - Elsevier http:/ / arxiv. org/ pdf/ 0709. 1516
    [18] "Raymond J. Solomonoff", Peter Gacs, Paul M. B. Vitanyi, 2011 cs.bu.edu, http:/ / citeseerx. ist. psu. edu/ viewdoc/ download?doi=10. 1. 1.
        186. 8268& rep=rep1& type=pdf
    [19] Dawid, A.P. and Mortera, J. (1996) "Coherent analysis of forensic identification evidence". Journal of the Royal Statistical Society, Series
        B, 58,425–443.
    [20] Foreman, L.A; Smith, A.F.M. and Evett, I.W. (1997). "Bayesian analysis of deoxyribonucleic acid profiling data in forensic identification
        applications (with discussion)". Journal of the Royal Statistical Society, Series A, 160, 429–469.
    [21] Robertson, B. and Vignaux, G.A. (1995) Interpreting Evidence: Evaluating Forensic Science in the Courtroom. John Wiley and Sons.
        Chichester. ISBN 978-0-471-96026-3
    [22] Dawid, A.P. (2001) "Bayes' Theorem and Weighing Evidence by Juries"; http:/ / 128. 40. 111. 250/ evidence/ content/ dawid-paper. pdf
    [23] Gardner-Medwin, A. (2005) "What probability should the jury address?". Significance, 2 (1), March 2005
    [24] David Miller: Critical Rationalism
    [25] Howson & Urbach (2005), Jaynes (2003)
    [26] Stephen M. Stigler (1986) The history of statistics. Harvard University press. Chapter 3.
    [27] Stephen. E. Fienberg, (2006) "When did Bayesian Inference become "Bayesian"? (http:/ / ba. stat. cmu. edu/ journal/ 2006/ vol01/ issue01/
        fienberg. pdf) Bayesian Analysis, 1 (1), 1–40. See page 5.
    [28] JM. Bernardo (2005), "Reference analysis", Handbook of statistics, 25, 17–90
    [29] Wolpert, RL. (2004) A conversation with James O. Berger, Statistical science, 9, 205–218
    [30] José M. Bernardo (2006) A Bayesian mathematical statistics primer (http:/ / www. ime. usp. br/ ~abe/ ICOTS7/ Proceedings/ PDFs/
        InvitedPapers/ 3I2_BERN. pdf). ICOTS-7
    [31] Bishop, C.M. (2007) Pattern Recognition and Machine Learning. Springer, 2007
Bayesian inference                                                                                                       50


    References
    • Aster, Richard; Borchers, Brian, and Thurber, Clifford (2012). Parameter Estimation and Inverse Problems,
      Second Edition, Elsevier. ISBN 0123850487, ISBN 978-0123850485
    • Bickel, Peter J. and Doksum, Kjell A. (2001). Mathematical Statistics, Volume 1: Basic and Selected Topics
      (Second (updated printing 2007) ed.). Pearson Prentice–Hall. ISBN 0-13-850363-X.
    • Box, G.E.P. and Tiao, G.C. (1973) Bayesian Inference in Statistical Analysis, Wiley, ISBN 0-471-57428-7
    • Edwards, Ward (1968). "Conservatism in Human Information Processing". In Kleinmuntz, B. Formal
      Representation of Human Judgment. Wiley.
    • Edwards, Ward (1982). "Conservatism in Human Information Processing (excerpted)". In Daniel Kahneman, Paul
      Slovic and Amos Tversky. Judgment under uncertainty: Heuristics and biases. Cambridge University Press.
    • Jaynes E.T. (2003) Probability Theory: The Logic of Science, CUP. ISBN 978-0-521-59271-0 ( Link to
      Fragmentary Edition of March 1996 (http://www-biba.inrialpes.fr/Jaynes/prob.html)).
    • Howson, C. and Urbach, P. (2005). Scientific Reasoning: the Bayesian Approach (3rd ed.). Open Court
      Publishing Company. ISBN 978-0-8126-9578-6.
    • Phillips, L.D.; Edwards, W. (October 2008). "Chapter 6: Conservatism in a simple probability inference task
      (Journal of Experimental Psychology (1966) 72: 346-354)". In Jie W. Weiss and David J. Weiss. A Science of
      Decision Making:The Legacy of Ward Edwards. Oxford University Press. pp. 536. ISBN 978-0-19-532298-9.


    Further reading

    Elementary
    The following books are listed in ascending order of probabilistic sophistication:
    • Colin Howson and Peter Urbach (2005). Scientific Reasoning: the Bayesian Approach (3rd ed.). Open Court
      Publishing Company. ISBN 978-0-8126-9578-6.
    • Berry, Donald A. (1996). Statistics: A Bayesian Perspective. Duxbury. ISBN 0-534-23476-3.
    • Morris H. DeGroot and Mark J. Schervish (2002). Probability and Statistics (third ed.). Addison-Wesley.
      ISBN 978-0-201-52488-8.
    • Bolstad, William M. (2007) Introduction to Bayesian Statistics: Second Edition, John Wiley ISBN 0-471-27020-2
    • Winkler, Robert L, Introduction to Bayesian Inference and Decision, 2nd Edition (2003) ISBN 0-9647938-4-9
    • Lee, Peter M. Bayesian Statistics: An Introduction. Fourth Edition (2012), John Wiley ISBN 978-1-1183-3257-3
    • Carlin, Bradley P. and Louis, Thomas A. (2008). Bayesian Methods for Data Analysis, Third Edition. Boca
      Raton, FL: Chapman and Hall/CRC. ISBN 1-58488-697-8.
    • Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Rubin, Donald B. (2003). Bayesian Data Analysis, Second
      Edition. Boca Raton, FL: Chapman and Hall/CRC. ISBN 1-58488-388-X.


    Intermediate or advanced
    • Berger, James O (1985). Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics (Second
      ed.). Springer-Verlag. ISBN 0-387-96098-8.
    • Bernardo, José M.; Smith, Adrian F. M. (1994). Bayesian Theory. Wiley.
    • DeGroot, Morris H., Optimal Statistical Decisions. Wiley Classics Library. 2004. (Originally published (1970) by
      McGraw-Hill.) ISBN 0-471-68029-X.
    • Schervish, Mark J. (1995). Theory of statistics. Springer-Verlag. ISBN 0-387-94546-6.
    • Jaynes, E.T. (1998) Probability Theory: The Logic of Science (http://www-biba.inrialpes.fr/Jaynes/prob.
      html).
    • O'Hagan, A. and Forster, J. (2003) Kendall's Advanced Theory of Statistics, Volume 2B: Bayesian Inference.
      Arnold, New York. ISBN 0-340-52922-9.
Bayesian inference                                                                                                              51


    • Robert, Christian P (2001). The Bayesian Choice – A Decision-Theoretic Motivation (second ed.). Springer.
      ISBN 0-387-94296-3.
    • Glenn Shafer and Pearl, Judea, eds. (1988) Probabilistic Reasoning in Intelligent Systems, San Mateo, CA:
      Morgan Kaufmann.


    External links
    • Hazewinkel, Michiel, ed. (2001), "Bayesian approach to statistical problems" (http://www.encyclopediaofmath.
      org/index.php?title=p/b015390), Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4
    • Bayesian Statistics (http://www.scholarpedia.org/article/Bayesian_statistics) from Scholarpedia.
    • Introduction to Bayesian probability (http://www.dcs.qmw.ac.uk/~norman/BBNs/BBNs.htm) from Queen
      Mary University of London
    • Mathematical notes on Bayesian statistics and Markov chain Monte Carlo (http://webuser.bus.umich.edu/
      plenk/downloads.htm)
    • Bayesian reading list (http://cocosci.berkeley.edu/tom/bayes.html), categorized and annotated by Tom
      Griffiths (http://psychology.berkeley.edu/faculty/profiles/tgriffiths.html).
    • Stanford Encyclopedia of Philosophy: "Inductive Logic" (http://plato.stanford.edu/entries/logic-inductive/)
    • Bayesian Confirmation Theory (http://faculty-staff.ou.edu/H/James.A.Hawthorne-1/
      Hawthorne--Bayesian_Confirmation_Theory.pdf)
    • What is Bayesian Learning? (http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-7.html)



    Statistical hypothesis testing
    A statistical hypothesis test is a method of making decisions using data from a scientific study. In statistics, a result
    is called statistically significant if it has been predicted as unlikely to have occurred by chance alone, according to a
    pre-determined threshold probability, the significance level. The phrase "test of significance" was coined by
    statistician Ronald Fisher.[1] These tests are used in determining what outcomes of a study would lead to a rejection
    of the null hypothesis for a pre-specified level of significance; this can help to decide whether results contain enough
    information to cast doubt on conventional wisdom, given that conventional wisdom has been used to establish the
    null hypothesis. The critical region of a hypothesis test is the set of all outcomes which cause the null hypothesis to
    be rejected in favor of the alternative hypothesis. Statistical hypothesis testing is sometimes called confirmatory
    data analysis, in contrast to exploratory data analysis, which may not have pre-specified hypotheses. Statistical
    hypothesis testing is a key technique of frequentist statistical inference.
    Statistical hypothesis tests define a procedure which controls (fixes) the probability of incorrectly deciding that a
    default position (null hypothesis) is incorrect based on how likely it would be for a set of observations to occur if the
    null hypothesis were true. Note that this probability of making an incorrect decision is not the probability that the
    null hypothesis is true, nor whether any specific alternative hypothesis is true. This contrasts with other possible
    techniques of decision theory in which the null and alternative hypothesis are treated on a more equal basis. One
    naive Bayesian approach to hypothesis testing is to base decisions on the posterior probability,[2][3] but this fails
    when comparing point and continuous hypotheses. Other approaches to decision making, such as Bayesian decision
    theory, attempt to balance the consequences of incorrect decisions across all possibilities, rather than concentrating
    on a single null hypothesis. A number of other approaches to reaching a decision based on data are available via
    decision theory and optimal decisions, some of which have desirable properties, yet hypothesis testing is a dominant
    approach to data analysis in many fields of science. Extensions to the theory of hypothesis testing include the study
    of the power of tests, which refers to the probability of correctly rejecting the null hypothesis when a given state of
    nature exists. Such considerations can be used for the purpose of sample size determination prior to the collection of
    data.
Statistical hypothesis testing                                                                                                      52


     In a famous example of hypothesis testing, known as the Lady tasting tea example,[4] a female colleague of Fisher
     claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups,
     four of each variety, in random order. One could then ask what the probability was for her getting the number she got
     correct, but just by chance. The null hypothesis was that the Lady had no such ability. The test statistic was a simple
     count of the number of successes in selecting the 4 cups. The critical region was the single case of 4 successes of 4
     possible based on a conventional probability criterion (< 5%; 1 of 70 ≈ 1.4%). Fisher asserted that no alternative
     hypothesis was (ever) required. The lady correctly identified every cup,[5] which would be considered a statistically
     significant result.


     The testing process
     In the statistical literature, statistical hypothesis testing plays a fundamental role.[6] The usual line of reasoning is as
     follows:
     1. There is an initial research hypothesis of which the truth is unknown.
     2. The first step is to state the relevant null and alternative hypotheses. This is important as mis-stating the
        hypotheses will muddy the rest of the process. Specifically, the null hypothesis allows to attach an attribute: it
        should be chosen in such a way that it allows us to conclude whether the alternative hypothesis can either be
        accepted or stays undecided as it was before the test.[7]
     3. The second step is to consider the statistical assumptions being made about the sample in doing the test; for
        example, assumptions about the statistical independence or about the form of the distributions of the observations.
        This is equally important as invalid assumptions will mean that the results of the test are invalid.
     4. Decide which test is appropriate, and state the relevant test statistic T.
     5. Derive the distribution of the test statistic under the null hypothesis from the assumptions. In standard cases this
        will be a well-known result. For example the test statistic may follow a Student's t distribution or a normal
        distribution.
     6. Select a significance level (α), a probability threshold below which the null hypothesis will be rejected. Common
        values are 5% and 1%.
     7. The distribution of the test statistic under the null hypothesis partitions the possible values of T into those for
        which the null-hypothesis is rejected, the so called critical region, and those for which it is not. The probability of
        the critical region is α.
     8. Compute from the observations the observed value tobs of the test statistic T.
     9. Decide to either reject the null hypothesis in favor of the alternative or not reject it. The decision rule is to reject
        the null hypothesis H0 if the observed value tobs is in the critical region, and to accept or "fail to reject" the
        hypothesis otherwise.
     An alternative process is commonly used:

        • Compute from the observations the observed value tobs of the test statistic T.
     2. From the statistic calculate a probability of the observation under the null hypothesis (the p-value).
     3. Reject the null hypothesis in favor of the alternative or not reject it. The decision rule is to reject the null
        hypothesis if and only if the p-value is less than the significance level (the selected probability) threshold.
     The two processes are equivalent.[8] The former process was advantageous in the past when only tables of test
     statistics at common probability thresholds were available. It allowed a decision to be made without the calculation
     of a probability. It was adequate for classwork and for operational use, but it was deficient for reporting results.
     The latter process relied on extensive tables or on computational support not always available. The explicit
     calculation of a probability is useful for reporting. The calculations are now trivially performed with appropriate
     software.
     The difference in the two processes applied to the Radioactive suitcase example:
Statistical hypothesis testing                                                                                                      53


     • "The Geiger-counter reading is 10. The limit is 9. Check the suitcase."
     • "The Geiger-counter reading is high; 97% of safe suitcases have lower readings. The limit is 95%. Check the
       suitcase."
     The former report is adequate, the latter gives a more detailed explanation of the data and the reason why the suitcase
     is being checked.
     It is important to note the philosophical difference between accepting the null hypothesis and simply failing to reject
     it. The "fail to reject" terminology highlights the fact that the null hypothesis is assumed to be true from the start of
     the test; if there is a lack of evidence against it, it simply continues to be assumed true. The phrase "accept the null
     hypothesis" may suggest it has been proved simply because it has not been disproved, a logical fallacy known as the
     argument from ignorance. Unless a test with particularly high power is used, the idea of "accepting" the null
     hypothesis may be dangerous. Nonetheless the terminology is prevalent throughout statistics, where its meaning is
     well understood.
     Alternatively, if the testing procedure forces us to reject the null hypothesis (H0), we can accept the alternative
     hypothesis (H1) and we conclude that the research hypothesis is supported by the data. This fact expresses that our
     procedure is based on probabilistic considerations in the sense we accept that using another set of data could lead us
     to a different conclusion.
     The processes described here are perfectly adequate for computation. They seriously neglect the design of
     experiments considerations.[9][10]
     It is particularly critical that appropriate sample sizes be estimated before conducting the experiment.


     Interpretation
     If the p-value is less than the required significance level (equivalently, if the observed test statistic is in the critical
     region), then we say the null hypothesis is rejected at the given level of significance. Rejection of the null hypothesis
     is a conclusion. This is like a "guilty" verdict in a criminal trial - the evidence is sufficient to reject innocence, thus
     proving guilt. We might accept the alternative hypothesis (and the research hypothesis).
     If the p-value is not less than the required significance level (equivalently, if the observed test statistic is outside the
     critical region), then the test has no result. The evidence is insufficient to support a conclusion. (This is like a jury
     that fails to reach a verdict.) The researcher typically gives extra consideration to those cases where the p-value is
     close to the significance level.
     In the Lady tasting tea example (above), Fisher required the Lady to properly categorize all of the cups of tea to
     justify the conclusion that the result was unlikely to result from chance. He defined the critical region as that case
     alone. The region was defined by a probability (that the null hypothesis was correct) of less than 5%.
     Whether rejection of the null hypothesis truly justifies acceptance of the research hypothesis depends on the structure
     of the hypotheses. Rejecting the hypothesis that a large paw print originated from a bear does not immediately prove
     the existence of Bigfoot. Hypothesis testing emphasizes the rejection which is based on a probability rather than the
     acceptance which requires extra steps of logic.
Statistical hypothesis testing                                                                                                    54


     Use and Importance
     Statistics are helpful in analyzing most collections of data. This is equally true of hypothesis testing which can justify
     conclusions even when no scientific theory exists. In the Lady tasting tea example, it was "obvious" that no
     difference existed between (milk poured into tea) and (tea poured into milk). The data contradicted the "obvious".
     Real world applications of hypothesis testing include:[11]
     •   Testing whether more men than women suffer from nightmares
     •   Establishing authorship of documents
     •   Evaluating the effect of the full moon on behavior
     •   Determining the range at which a bat can detect an insect by echo
     •   Deciding whether hospital carpeting results in more infections
     •   Selecting the best means to stop smoking
     •   Checking whether bumper stickers reflect car owner behavior
     •   Testing the claims of handwriting analysts
     Statistical hypothesis testing plays an important role in the whole of statistics and in statistical inference. For
     example, Lehmann (1992) in a review of the fundamental paper by Neyman and Pearson (1933) says: "Nevertheless,
     despite their shortcomings, the new paradigm formulated in the 1933 paper, and the many developments carried out
     within its framework continue to play a central role in both the theory and practice of statistics and can be expected
     to do so in the foreseeable future".
     Significance testing has been the favored statistical tool in some experimental social sciences (over 90% of articles in
     the Journal of Applied Psychology during the early 1990s).[12] Other fields have favored the estimation of
     parameters (e.g., effect size).


     Cautions
     "If the government required statistical procedures to carry warning labels like those on drugs, most inference
     methods would have long labels indeed."[13] This caution applies to hypothesis tests and alternatives to them.
     The successful hypothesis test is associated with a probability and a type-I error rate. The conclusion might be
     wrong.
     The conclusion of the test is only as solid as the sample upon which it is based. The design of the experiment is
     critical. A number of unexpected effects have been observed including:
     • The Clever Hans effect. A horse appeared to be capable of doing simple arithmetic.
     • The Hawthorne effect. Industrial workers were more productive in better illumination, and most productive in
       worse.
     • The Placebo effect. Pills with no medically active ingredients were remarkably effective.
     A statistical analysis of misleading data produces misleading conclusions. The issue of data quality can be more
     subtle. In forecasting for example, there is no agreement on a measure of forecast accuracy. In the absence of a
     consensus measurement, no decision based on measurements will be without controversy.
     The book How to Lie with Statistics[14][15] is the most popular book on statistics ever published.[16] It does not much
     consider hypothesis testing, but its cautions are applicable, including: Many claims are made on the basis of samples
     too small to convince. If a report does not mention sample size, be doubtful.
     Hypothesis testing acts as a filter of statistical conclusions; Only those results meeting a probability threshold are
     publishable. Economics also acts as a publication filter; Only those results favorable to the author and funding source
     may be submitted for publication. The impact of filtering on publication is termed publication bias. A related
     problem is that of multiple testing (sometimes linked to data mining), in which a variety of tests for a variety of
     possible effects are applied to a single data set and only those yielding a significant result are reported. These are
     often dealt with by using multiplicity correction procedures that control the family wise error rate (FWER) or the
Statistical hypothesis testing                                                                                                    55


     false discovery rate (FDR).
     Those making critical decisions based on the results of a hypothesis test are prudent to look at the details rather than
     the conclusion alone. In the physical sciences most results are fully accepted only when independently confirmed.
     The general advice concerning statistics is, "Figures never lie, but liars figure" (anonymous).


     Examples
     The following examples should solidify these ideas.


     Analogy – Courtroom trial
     A statistical test procedure is comparable to a criminal trial; a defendant is considered not guilty as long as his or her
     guilt is not proven. The prosecutor tries to prove the guilt of the defendant. Only when there is enough charging
     evidence the defendant is convicted.
     In the start of the procedure, there are two hypotheses       : "the defendant is not guilty", and   : "the defendant is
     guilty". The first one is called null hypothesis, and is for the time being accepted. The second one is called
     alternative (hypothesis). It is the hypothesis one hopes to support.
     The hypothesis of innocence is only rejected when an error is very unlikely, because one doesn't want to convict an
     innocent defendant. Such an error is called error of the first kind (i.e. the conviction of an innocent person), and the
     occurrence of this error is controlled to be rare. As a consequence of this asymmetric behaviour, the error of the
     second kind (acquitting a person who committed the crime), is often rather large.

                                                                 H0 is true        H1 is true
                                                               Truly not guilty   Truly guilty

                                              Accept Null       Right decision       Wrong
                                              Hypothesis                            decision
                                               Acquittal                          Type II Error

                                         Reject Null Hypothesis Wrong decision    Right decision
                                               Conviction        Type I Error


     A criminal trial can be regarded as either or both of two decision processes: guilty vs not guilty or evidence vs a
     threshold ("beyond a reasonable doubt"). In one view, the defendant is judged; in the other view the performance of
     the prosecution (which bears the burden of proof) is judged. A hypothesis test can be regarded as either a judgment
     of a hypothesis or as a judgment of evidence.


     Example 1 – Philosopher's beans
     The following example was produced by a philosopher describing scientific methods generations before hypothesis
     testing was formalized and popularized.[17]
           Few beans of this handful are white.
           Most beans in this bag are white.
           Therefore: Probably, these beans were taken from another bag.
           This is an hypothetical [sic] inference.
     The beans in the bag are the population. The handful are the sample. The null hypothesis is that the sample
     originated from the population. The criterion for rejecting the null-hypothesis is the "obvious" difference in
     appearance (an informal difference in the mean). The interesting result is that consideration of a real population and
     a real sample produced an imaginary bag. The philosopher was considering logic rather than probability. To be a real
     statistical hypothesis test, this example requires the formalities of a probability calculation and a comparison of that
     probability to a standard.
Statistical hypothesis testing                                                                                                       56


     A simple generalization of the example considers a mixed bag of beans and a handful that contain either very few or
     very many white beans. The generalization considers both extremes. It requires more calculations and more
     comparisons to arrive at a formal answer, but the core philosophy is unchanged; If the composition of the handful is
     greatly different that of the bag, then the sample probably originated from another bag. The original example is
     termed a one-sided or a one-tailed test while the generalization is termed a two-sided or two-tailed test.


     Example 2 – Clairvoyant card game
     A person (the subject) is tested for clairvoyance. He is shown the reverse of a randomly chosen playing card 25 times
     and asked which of the four suits it belongs to. The number of hits, or correct answers, is called X.
     As we try to find evidence of his clairvoyance, for the time being the null hypothesis is that the person is not
     clairvoyant. The alternative is, of course: the person is (more or less) clairvoyant.
     If the null hypothesis is valid, the only thing the test person can do is guess. For every card, the probability (relative
     frequency) of any single suit appearing is 1/4. If the alternative is valid, the test subject will predict the suit correctly
     with probability greater than 1/4. We will call the probability of guessing correctly p. The hypotheses, then, are:

     • null hypothesis                             (just guessing)
     and

     • alternative hypothesis                     (true clairvoyant).
     When the test subject correctly predicts all 25 cards, we will consider him clairvoyant, and reject the null hypothesis.
     Thus also with 24 or 23 hits. With only 5 or 6 hits, on the other hand, there is no cause to consider him so. But what
     about 12 hits, or 17 hits? What is the critical number, c, of hits, at which point we consider the subject to be
     clairvoyant? How do we determine the critical value c? It is obvious that with the choice c=25 (i.e. we only accept
     clairvoyance when all cards are predicted correctly) we're more critical than with c=10. In the first case almost no
     test subjects will be recognized to be clairvoyant, in the second case, a certain number will pass the test. In practice,
     one decides how critical one will be. That is, one decides how often one accepts an error of the first kind – a false
     positive, or Type I error. With c = 25 the probability of such an error is:


     and hence, very small. The probability of a false positive is the probability of randomly guessing correctly all 25
     times.
     Being less critical, with c=10, gives:




     Thus, c = 10 yields a much greater probability of false positive.
     Before the test is actually performed, the maximum acceptable probability of a Type I error (α) is determined.
     Typically, values in the range of 1% to 5% are selected. (If the maximum acceptable error rate is zero, an infinite
     number of correct guesses is required.) Depending on this Type 1 error rate, the critical value c is calculated. For
     example, if we select an error rate of 1%, c is calculated thus:


     From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of a Type II
     error, a false negative. For the above example, we select:      .
Statistical hypothesis testing                                                                                                    57


     Example 3 – Radioactive suitcase
     As an example, consider determining whether a suitcase contains some radioactive material. Placed under a Geiger
     counter, it produces 10 counts per minute. The null hypothesis is that no radioactive material is in the suitcase and
     that all measured counts are due to ambient radioactivity typical of the surrounding air and harmless objects. We can
     then calculate how likely it is that we would observe 10 counts per minute if the null hypothesis were true. If the null
     hypothesis predicts (say) on average 9 counts per minute and a standard deviation of 1 count per minute, then we say
     that the suitcase is compatible with the null hypothesis (this does not guarantee that there is no radioactive material,
     just that we don't have enough evidence to suggest there is). On the other hand, if the null hypothesis predicts 3
     counts per minute and a standard deviation of 1 count per minute, then the suitcase is not compatible with the null
     hypothesis, and there are likely other factors responsible to produce the measurements.
     The test does not directly assert the presence of radioactive material. A successful test asserts that the claim of no
     radioactive material present is unlikely given the reading (and therefore ...). The double negative (disproving the null
     hypothesis) of the method is confusing, but using a counter-example to disprove is standard mathematical practice.
     The attraction of the method is its practicality. We know (from experience) the expected range of counts with only
     ambient radioactivity present, so we can say that a measurement is unusually large. Statistics just formalizes the
     intuitive by using numbers instead of adjectives. We probably do not know the characteristics of the radioactive
     suitcases; We just assume that they produce larger readings.
     To slightly formalize intuition: Radioactivity is suspected if the Geiger-count with the suitcase is among or exceeds
     the greatest (5% or 1%) of the Geiger-counts made with ambient radiation alone. This makes no assumptions about
     the distribution of counts. Many ambient radiation observations are required to obtain good probability estimates for
     rare events.
     The test described here is more fully the null-hypothesis statistical significance test. The null hypothesis represents
     what we would believe by default, before seeing any evidence. Statistical significance is a possible finding of the
     test, declared when the observed sample is unlikely to have occurred by chance if the null hypothesis were true. The
     name of the test describes its formulation and its possible outcome. One characteristic of the test is its crisp decision:
     to reject or not reject the null hypothesis. A calculated value is compared to a threshold, which is determined from
     the tolerable risk of error.


     Definition of terms
     The following definitions are mainly based on the exposition in the book by Lehmann and Romano:[6]
     Statistical hypothesis
            A statement about the parameters describing a population (not a sample).
     Statistic
            A value calculated from a sample, often to summarize the sample for comparison purposes.
     Simple hypothesis
            Any hypothesis which specifies the population distribution completely.
     Composite hypothesis
            Any hypothesis which does not specify the population distribution completely.
     Null hypothesis (H0)
            A simple hypothesis associated with a contradiction to a theory one would like to prove.
     Alternative hypothesis (H1)
            A hypothesis (often composite) associated with a theory one would like to prove.
     Statistical test
Statistical hypothesis testing                                                                                                       58


           A procedure whose inputs are samples and whose result is a hypothesis.
     Region of acceptance
           The set of values of the test statistic for which we fail to reject the null hypothesis.
     Region of rejection / Critical region
           The set of values of the test statistic for which the null hypothesis is rejected.
     Critical value
           The threshold value delimiting the regions of acceptance and rejection for the test statistic.
     Power of a test (1 − β)
           The test's probability of correctly rejecting the null hypothesis. The complement of the false negative rate, β.
           Power is termed sensitivity in biostatistics. ("This is a sensitive test. Because the result is negative, we can
           confidently say that the patient does not have the condition.") See sensitivity and specificity and Type I and
           type II errors for exhaustive definitions.
     Size / Significance level of a test (α)
           For simple hypotheses, this is the test's probability of incorrectly rejecting the null hypothesis. The false
           positive rate. For composite hypotheses this is the upper bound of the probability of rejecting the null
           hypothesis over all cases covered by the null hypothesis. The complement of the false positive rate, (1 − α), is
           termed specificity in biostatistics. ("This is a specific test. Because the result is positive, we can confidently
           say that the patient has the condition.") See sensitivity and specificity and Type I and type II errors for
           exhaustive definitions.
     p-value
           The probability, assuming the null hypothesis is true, of observing a result at least as extreme as the test
           statistic.
     Statistical significance test
           A predecessor to the statistical hypothesis test (see the Origins section). An experimental result was said to be
           statistically significant if a sample was sufficiently inconsistent with the (null) hypothesis. This was variously
           considered common sense, a pragmatic heuristic for identifying meaningful experimental results, a convention
           establishing a threshold of statistical evidence or a method for drawing conclusions from data. The statistical
           hypothesis test added mathematical rigor and philosophical consistency to the concept by making the
           alternative hypothesis explicit. The term is loosely used to describe the modern version which is now part of
           statistical hypothesis testing.
     Conservative test
           A test is conservative if, when constructed for a given nominal significance level, the true probability of
           incorrectly rejecting the null hypothesis is never greater than the nominal level.
     Exact test
           A test in which the significance level or critical value can be computed exactly, i.e., without any
           approximation. In some contexts this term is restricted to tests applied to categorical data and to permutation
           tests, in which computations are carried out by complete enumeration of all possible outcomes and their
           probabilities.
     A statistical hypothesis test compares a test statistic (z or t for examples) to a threshold. The test statistic (the formula
     found in the table below) is based on optimality. For a fixed level of Type I error rate, use of these statistics
     minimizes Type II error rates (equivalent to maximizing power). The following terms describe tests in terms of such
     optimality:
     Most powerful test
Statistical hypothesis testing                                                                                                   59


           For a given size or significance level, the test with the greatest power (probability of rejection) for a given
           value of the parameter(s) being tested, contained in the alternative hypothesis.
     Uniformly most powerful test (UMP)
           A test with the greatest power for all values of the parameter(s) being tested, contained in the alternative
           hypothesis.


     Common test statistics
     One-sample tests are appropriate when a sample is being compared to the population from a hypothesis. The
     population characteristics are known from theory or are calculated from the population.
     Two-sample tests are appropriate for comparing two samples, typically experimental and control samples from a
     scientifically controlled experiment.
     Paired tests are appropriate for comparing two samples where it is impossible to control important variables. Rather
     than comparing two sets, members are paired between samples so the difference between the members becomes the
     sample. Typically the mean of the differences is then compared to zero.
     Z-tests are appropriate for comparing means under stringent conditions regarding normality and a known standard
     deviation.
     T-tests are appropriate for comparing means under relaxed conditions (less is assumed).
     Tests of proportions are analogous to tests of means (the 50% proportion).
     Chi-squared tests use the same calculations and the same probability distribution for different applications:
     • Chi-squared tests for variance are used to determine whether a normal population has a specified variance. The
       null hypothesis is that it does.
     • Chi-squared tests of independence are used for deciding whether two variables are associated or are independent.
       The variables are categorical rather than numeric. It can be used to decide whether left-handedness is correlated
       with libertarian politics (or not). The null hypothesis is that the variables are independent. The numbers used in
       the calculation are the observed and expected frequencies of occurrence (from contingency tables).
     • Chi-squared goodness of fit tests are used to determine the adequacy of curves fit to data. The null hypothesis is
       that the curve fit is adequate. It is common to determine curve shapes to minimize the mean square error, so it is
       appropriate that the goodness-of-fit calculation sums the squared errors.
     F-tests (analysis of variance, ANOVA) are commonly used when deciding whether groupings of data by category are
     meaningful. If the variance of test scores of the left-handed in a class is much smaller than the variance of the whole
     class, then it may be useful to study lefties as a group. The null hypothesis is that two variances are the same - so the
     proposed grouping is not meaningful.
     In the table below, the symbols used are defined at the bottom of the table. Many other tests can be found in other
     articles.
Statistical hypothesis testing                                                                                                                                60


            Name                               Formula                                                   Assumptions or notes

    One-sample z-test                                                         (Normal population or n > 30) and σ known.
                                                                              (z is the distance from the mean in relation to the standard deviation of the
                                                                              mean). For non-normal distributions it is possible to calculate a minimum
                                                                              proportion of a population that falls within k standard deviations for any k
                                                                              (see: Chebyshev's inequality).

    Two-sample z-test                                                         Normal population and independent observations and σ1 and σ2 are known



    One-sample t-test                                                         (Normal population or n > 30) and      unknown




    Paired t-test                                                             (Normal population of differences or n > 30) and      unknown or small
                                                                              sample size n < 30


    Two-sample pooled                                                         (Normal populations or n1 + n2 > 40) and independent observations and σ1
    t-test, equal variances                                                   = σ2 unknown




                                                             [18]

    Two-sample                                                                (Normal populations or n1 + n2 > 40) and independent observations and σ1
    unpooled t-test,                                                          ≠ σ2 both unknown
    unequal variances


                                                               [18]



    One-proportion z-test                                                     n .p0 > 10 and n (1 − p0) > 10 and it is a SRS (Simple Random Sample),
                                                                              see notes.

    Two-proportion                                                            n1 p1 > 5 and n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and
    z-test, pooled for                                                        independent observations, see notes.


    Two-proportion                                                            n1 p1 > 5 and n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and
    z-test, unpooled for                                                      independent observations, see notes.


    Chi-squared test for                                                      Normal population
    variance

    Chi-squared test for                                                      df = k - 1 - # parameters estimated, and one of these must hold. • All
    goodness of fit                                                                                           [19]
                                                                              expected counts are at least 5.
                                                                              • All expected counts are > 1 and no more than 20% of expected counts are
                                                                                          [20]
                                                                              less than 5

    Two-sample F test                                                         Normal populations
    for equality of                                                           Arrange so         and reject H0 for
    variances                                                                 [21]

    In general, the subscript 0 indicates a value taken from the null hypothesis, H0, which should be used as much as possible in constructing its test
    statistic. ... Definitions of other symbols:
Statistical hypothesis testing                                                                                                                         61



     •     , the probability of Type I error (rejecting a null   •      = sample variance               •     = x/n = sample proportion, unless
         hypothesis when it is in fact true)                                                                specified otherwise
     •     = sample size                                         •      = sample 1 standard deviation   •      = hypothesized population
                                                                                                            proportion
     •      = sample 1 size                                      •      = sample 2 standard deviation   •      = proportion 1
     •      = sample 2 size                                      •     = t statistic                    •      = proportion 2
     •     = sample mean                                         •      = degrees of freedom            •      = hypothesized difference in
                                                                                                            proportion
     •      = hypothesized population mean                       •     = sample mean of differences     •                     = minimum of n1 and n2
     •      = population 1 mean                                  •      = hypothesized population       •
                                                                     mean difference
     •      = population 2 mean                                  •       = standard deviation of        •
                                                                     differences
     •     = population standard deviation                       •      = Chi-squared statistic         •     = F statistic
     •      = population variance
     •     = sample standard deviation

     •       = sum (of k numbers)
Statistical hypothesis testing                                                                                                        62


     Origins and Early Controversy
     Hypothesis testing is largely the product of Ronald Fisher, and was further
     developed by Jerzy Neyman, Karl Pearson and (son) Egon Pearson. Ronald Fisher,
     genius mathematician and biologist described by Richard Dawkins as "the greatest
     biologist since Darwin", began his life in statistics as a Bayesian (Zabell 1992), but
     Fisher soon grew disenchanted with the subjectivity involved, and sought to
     provide a more "objective" approach to inductive inference.[22]

     According to Stephen M. Stigler:
           "Fisher was an agricultural statistician who emphasized rigorous
           experimental design and methods to extract a result from few samples
           assuming Gaussian distributions. Neyman (who teamed with the younger
           Pearson) emphasized mathematical rigor and methods to obtain more results
           from many samples and a wider range of distributions. Modern hypothesis
           testing is an inconsistent hybrid of the Fisher vs Neyman/Pearson
           formulation, methods and terminology developed in the early 20th century.
           While hypothesis testing was popularized early in the 20th century, evidence
           of its use can be found much earlier. In the 1770s Laplace considered the
           statistics of almost half a million births. The statistics showed an excess of
           boys compared to girls. He concluded by calculation of a p-value that the
           excess was a real, but unexplained, effect."[23]

     Fisher popularized the "significance test". He required a null-hypothesis
     (corresponding to a population frequency distribution) and a sample. His (now
     familiar) calculations determined whether to reject the null-hypothesis or not.
                                                                                                   The likely originator of the
     Significance testing did not utilize an alternative hypothesis so there was no             confused yet widespread "hybrid"
     concept of a Type II error.                                                                 method of hypothesis testing, as
                                                                                                   well as the use of "nil" null
     The p-value was devised as an informal, but objective, index meant to help a
                                                                                                 hypotheses, is E.F. Lindquist in
     researcher determine (based on other knowledge) whether to modify future                   his statistics textbook: Lindquist,
     experiments or strengthen one's faith in the null hypothesis.[24] Hypothesis testing       E.F. (1940) Statistical Analysis In
     (and Type I/II errors) were devised by Neyman and Pearson as a more objective               Educational Research. Boston:
                                                                                                        Houghton Mifflin.
     alternative to Fisher's p-value, also meant to determine researcher behaviour, but
     without requiring any inductive inference by the researcher.[25][26] Neither strategy
     is meant to provide any way of drawing conclusions from a single experiment.[25][27] Both strategies were meant to
     assess the results of experiments that were replicated multiple times.[28]

     Neyman & Pearson considered a different problem (which they called "hypothesis testing"). They initially
     considered two simple hypotheses (both with frequency distributions). They calculated two probabilities and
     typically selected the hypothesis associated with the higher probability (the hypothesis more likely to have generated
     the sample). Their method always selected a hypothesis. It also allowed the calculation of both types of error
     probabilities.
     Fisher and Neyman/Pearson clashed bitterly. Neyman/Pearson considered their formulation to be an improved
     generalization of significance testing.(The defining paper was abstract. Mathematicians have generalized and refined
     the theory for three generations.) Fisher thought that it was not applicable to scientific research because often, during
     the course of the experiment, it is discovered that the initial assumptions about the null hypothesis are questionable
     due to unexpected sources of error. He believed that the use of rigid reject/accept decisions based on models
     formulated before data is collected was incompatible with this common scenario faced by scientists and attempts to
     apply this method to scientific research would lead to mass confusion.
Statistical hypothesis testing                                                                                                        63


     Fisher wrote in 1955 that "the conclusions drawn by a scientific worker from a test of significance are provisional,
     and involve an intelligent attempt to understand the experimental situation," [24] and later in 1958, "We are quite in
     danger of sending highly-trained and highly intelligent young men out into the world with tables of erroneous
     numbers under their arms, and with a dense fog in the place where their brains ought to be. In this century, of course,
     they will be working on guided missiles and advising the medical profession on the control of disease, and there is
     no limit to the extent to which they could impede every sort of national effort."[29]
     Further, Fisher taught that his method was only applicable if little was known about the problem at hand and "no
     relevant sub-set [of the population] should be recognizable". He wrote that if "there were knowledge a priori of the
     distribution of [the population mean]... the method of Bayes would give a probability statement...[that] would
     supersede the fiducial value [the result of a significance test]... therefore the first condition [of significance testing] is
     that there should be no knowledge a priori." [29]
     In the same paper regarding the use of statistics in science Fisher complained:
           "it would be a mistake to think that mathematicians as such are particularly good at the inductive logical
           processes which are needed in improving or knowledge of the natural world"
     To clarify Fisher's view on on the use of Bayesian probability for scientific inference: He often praised Thomas
     Bayes for his insight into the special case in which one has prior information about the problem at hand, but he
     believed that "the essential information on which Bayes' theorem is based" [a prior probability distribution], was
     often "in reality absent",[30] and thus the theorem was inapplicable. Early writers on probability (especially Pierre
     Laplace) advocated use of the "principle of indifference" under these circumstances, which (in the opinion of Fisher,
     arbitrarily) claims that in the absence of contrary information one should assume all possible results are equally
     probable (i.e. use a uniform prior distribution). He saw such an assumption as completely unjustified, leading to
     statements such as:
           "Perhaps we were lucky in England in having the whole mass of fallacious rubbish put out of sight until we
           had time to think about probability in concrete terms and in relation, above all, to the purposes for which we
           wanted the idea in the natural sciences."[29]
     The modern version of hypothesis testing is a hybrid of the two approaches that resulted from confusion by writers
     of statistical textbooks (as predicted by Fisher) beginning in the 1940s.[31] (But signal detection, for example, still
     uses the Neyman/Pearson formulation.) Great conceptual differences and many caveats in addition to those
     mentioned above were ignored. Neyman and Pearson provided the stronger terminology, the more rigorous
     mathematics and the more consistent philosophy, but the subject taught today in introductory statistics has more
     similarities with Fisher's method than theirs.[32] This history explains the inconsistent terminology (example: the null
     hypothesis is never accepted, but there is a region of acceptance).
     Sometime around 1940,[31] in an apparent effort to provide researchers with a "non-controversial"[33] way to have
     their cake and eat it too, the authors of statistical text books began anonymously combining these two strategies by
     using the p-value in place of the test statistic (or data) to test against the Neyman-Pearson "significance level".[31]
     Thus, researchers were encouraged to infer the strength of their data against some null hypothesis using p-values,
     while also thinking they are retaining the post-data collection objectivity provided by hypothesis testing. It then
     became customary for the null hypothesis, which was originally some realistic research hypothesis, to be used almost
     solely as a strawman "nil" hypothesis (one where a treatment has no effect, regardless of the context).[34]
     A comparison between Fisherian, frequentist (Neyman-Pearson).
Statistical hypothesis testing                                                                                                                                64


                   Fisher’s null hypothesis testing                                            Neyman–Pearson decision theory

    1. Set up a statistical null hypothesis. The null need not be a   1. Set up two statistical hypotheses, H1 and H2, and decide about α, β, and sample
    nil hypothesis (i.e., zero difference).                           size before the experiment, based on subjective cost-benefit considerations. These
                                                                      define a rejection region for each hypothesis.

    2. Report the exact level of significance (e.g., p = 0.051 or p = 2. If the data falls into the rejection region of H1, accept H2; otherwise accept H1.
    0.049). Do not use a conventional 5% level, and do not talk       Note that accepting a hypothesis does not mean that you believe in it, but only that
    about accepting or rejecting hypotheses.                          you act as if it were true.

    3. Use this procedure only if little is known about the problem 3. The usefulness of the procedure is limited among others to situations where you
    at hand, and only to draw provisional conclusions in the        have a disjunction of hypotheses (e.g., either μ1 =8 or μ2 = 10 is true) and where
    context of an attempt to understand the experimental            you can make meaningful cost-benefit trade-offs for choosing alpha and beta.
    situation.




     Ongoing Controversy
     Given the prevalence of significance testing in published research literature, as well as the emergence of alternative
     approaches (namely, Bayesian statistics), hypothesis testing has been the subject of ongoing debate since its initial
     formulations by Fisher and Neyman/Pearson. The volume of criticism and rebuttal has filled books with language
     seldom used in the scholarly debate of a dry subject.[35][36][37][38]
     Nickerson surveyed the issues in the year 2000.[39] He included 300 references and reported 20 criticisms and almost
     as many recommendations, alternatives and supplements. Authors of statistical textbooks have largely ignored this
     controversy,.[40] According to psychologist Gerd Gigerenzer there is evidence that the majority of psychologists
     teaching statistics might not understand the nature of a p-value[41] and may be unaware of the controversy.
     According to Hubbard and Bayarri, researchers frequently misinterpret the p-value, viewing it "simultaneously in
     Neyman–Pearson terms as a deductive assessment of error in long-run repeated sampling situations, and in a
     Fisherian sense as a measure of inductive evidence in a single study. In fact, a p value from a significance test has no
     place in the Neyman–Pearson hypothesis testing framework. Contrary to popular misconception, p’s and α’s are not
     the same thing; they measure different concepts.[42]
     One set of criticisms focuses on the logic underlying hypothesis testing. Null hypothesis significance testing, as
     commonly performed, is neither based on a Fisherian strategy of inductive reasoning nor the Neyman-Pearson
     strategy of inductive behaviour. Although both strategies have merit on their own, the current practice has arisen due
     an erroneous combination of the two strategies that has no logical or mathematical basis.[43]
     Several criticisms of hypothesis testing relate to how the technique is commonly used in practice (rather than the
     fundamental statistical or philosophical implications of hypothesis testing per se). Some of these criticisms therefore
     aim to influence statistics education as much as critique the method. According to Cohen, "we, as teachers,
     consultants, authors, and otherwise perpetrators of quantitative methods, are responsible for the ritualization of null
     hypothesis significance testing (NHST, I resisted the temptation to call it statistical hypothesis inference testing) to
     the point of meaninglessness and beyond. I argue herein that NHST has not only failed to support the advance of
     psychology as a science but also has seriously impeded it [...] What's wrong with NHST? Well, among many other
     things, it does not tell us what we want to know," (the probability a hypothesis is true) "and we so much want to
     know what we want to know that, out of desperation, we nevertheless believe that it does!"[44] Importantly, the
     choice of one-tailed or two-tailed hypothesis tests, as well as the choice of significance level, can be seen as a
     subjective choice by the researcher, potentially to lower (or raise) the threshold for finding significant test
     statistics.[45] Often the arbitrary significance level of .05 is chosen because it is commonly used in published
     research, and it is commonly used because it is often chosen. Although this is not a problem with significance testing
     per se. One related view is that proper use of hypothesis testing requires penalizing researchers for making multiple
     comparisons (i.e., conducting multiple, distinct hypothesis tests on the same data).[46][47] As with any type of data
     analysis, data mining can introduce problems.
Statistical hypothesis testing                                                                                                    65


     Additionally, many commonly used tests (e.g., ANOVA) require potentially implausible assumptions about
     characteristics of the sample and/or population (e.g. normality, equal Ns, sphericity, equal variances), or require
     complicated adjustments to account for violations of these assumptions that have only been validated under a small
     subset of possible types of violations.[48] In practice, it is sometimes difficult to verify that these assumptions hold
     (e.g., in the case of small sample size) and multiple violations (e.g., of normality and homoscedasticity) may interact
     unpredictably. By contrast, nonparametric statistics aims to minimize the necessary assumptions about the
     distributions of test statistics, while Bayesian statistics simply makes distributional assumptions more explicit rather
     than eliminating them. As with Bayesian methods, the result of a significance test depends in part on the sample size
     used and the precision of measurement.[49] Accordingly, very large samples are more likely to produce significant
     results just as they would dramatically influence the shape of the posterior distribution in a Bayesian analysis.
     Another set of concerns relates to the interpretation of statistical hypothesis tests. As one example, concerns have
     been expressed about the prevalence of "no effect" null hypotheses (e.g., those where any difference between two
     group means is interpreted as meaningful) rather than hypothesis tests with larger or smaller null values.[50] There is
     no statistical requirement that a null hypothesis be "no effect," but this is the default setting in most statistical
     software (and consequently, most common in practice). Using "no effect" as a default could be considered a
     strawman.[51] According to Savage (1957), "Null hypotheses of no difference are usually known to be false before
     the data are collected; when they are, their rejection or acceptance simply reflects the size of the sample and the
     power of the test, and is not a contribution to science."[49]
     Because hypothesis testing looks only at the sample data and typically incorporates no other information, statistically
     significant results may be substantively unimportant (e.g., detecting an effect known to exist due to considerable
     previous research). However, hypothesis tests can be constructed such that a null hypothesis reflects the results of
     prior findings, thereby testing whether the new data suggest an effect different from that found before. And,
     meta-analysis can be used to aggregate the results of multiple studies to draw inferences about the distribution of
     effects. In general, statistical significance does not imply practical significance, in that a statistically significant
     difference may have a trivial effect size, especially when using large samples. For example, in one NAEP analysis of
     100,000 students, male science test scores were found to be significantly higher than those of females,[52] suggesting
     that there is a small difference between males and females, but the substantive size of that difference is trivial.
     Indeed, in this example, the mean difference was only 4 out of 300 points,[53] implying heavily overlapped
     distributions of scores, illustrating that differences in expected values of distributions may overshadow the shape of
     those distributions relative to one another. In this example, a hypothesis test could have been constructed to look for
     an effect size of substantive (rather than just statistical) significance.
     Part of the concern with hypothesis testing relates to how much emphasis is placed on p-values. P-values are widely
     interpreted as error rates, evidence against the null hypothesis, or even evidence for the research hypothesis. These
     misconceptions continue to persist despite decades of effort to eradicate them.[54] When interpreted in the common,
     psuedo-Fisherian sense, P-values can overstate the evidence against the null by an order of magnitude[55][56]
     Researchers' focus on p-values can be particularly problematic as it relates to the publication of research findings,
     particularly when statistical significance is used as a criterion for publication, which can yield publication bias.
     When publication is contingent on p-values, several problems emerge. Published Type I errors are difficult to
     correct. Published effect sizes are also biased upward (because achieving statistical significance requires effect sizes
     that are sufficiently large).[57] Type II errors (false negatives) might also be common due to insufficient power to
     detect small effects; but both Bayesian methods and meta-analysis can address this problem when the results of
     several underpowered studies are available.
     A focus on p-values also emerges in other areas. For example, some countries publish unusually high proportions of
     positive results.[58] and ‘‘positive’’ results increase down the hierarchy of the sciences.[59] Some research suggests
     that negative results are disappearing from most disciplines and countries[60] due to significance being tied to the
     publishability of research. Flexibility in data collection, analysis, and reporting can increase false-positive rates.[61]
Statistical hypothesis testing                                                                                                    66


     P-values in the psychology literature are much more common immediately below .05 than expected.[62]
     Controversy over significance testing, and its effects on publication bias in particular, has produced several
     results.[41] The American Psychological Association has strengthened its statistical reporting requirements after
     review,[63] medical journal publishers have recognized the obligation to publish some results that are not statistically
     significant to combat publication bias[64] and a journal (Journal of Articles in Support of the Null Hypothesis) has
     been created to publish such results exclusively.[65] Textbooks have added some cautions[66] and increased coverage
     of the tools necessary to estimate the size of the sample required to produce significant results. Major organizations
     have not abandoned use of significance tests although some have discussed doing so.[63]


     Alternative to significance testing
     However, the numerous criticisms of significance testing do not lead to a single alternative or even to a unified set of
     alternatives. A unifying position of critics is that statistics should not lead to a conclusion or a decision but to a
     probability or to an estimated value with a confidence interval rather than to an accept-reject decision regarding a
     particular hypothesis. There is some consensus that the hybrid testing procedure[67] that is commonly used is
     fundamentally flawed. It is unlikely that the controversy surrounding significance testing will be resolved in the near
     future. Its supposed flaws and unpopularity do not eliminate the need for an objective and transparent means of
     reaching conclusions regarding studies that produce statistical results. Critics have not unified around an alternative.
     Some of them have, however, suggested reforms for statistical and marketing research education to include a more
     thorough analysis of the meaning of statistical significance.[68] Other forms of reporting confidence or uncertainty
     could probably grow in popularity. Some recent work includes reconstruction and defense of Neyman–Pearson
     testing.[67] One strong critic of significance testing suggested a list of reporting alternatives:[69]
     effect sizes for importance, prediction intervals for confidence, replications and extensions for replicability,
     meta-analyses for generality. None of these suggested alternatives produces a conclusion/decision. Lehmann said
     that hypothesis testing theory can be presented in terms of conclusions/decisions, probabilities, or confidence
     intervals. "The distinction between the ... approaches is largely one of reporting and interpretation."[70]
     On one "alternative" there is no disagreement: Fisher himself said,[4] "In relation to the test of significance, we may
     say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will
     rarely fail to give us a statistically significant result." Cohen, an influential critic of significance testing,
     concurred,[71]
     "...don't look for a magic alternative to NHST [null hypothesis significance testing] ... It doesn't exist." "...given the
     problems of statistical induction, we must finally rely, as have the older sciences, on replication." The "alternative"
     to significance testing is repeated testing. The easiest way to decrease statistical uncertainty is by obtaining more
     data, whether by increased sample size or by repeated tests. Nickerson claimed to have never seen the publication of
     a literally replicated experiment in psychology.[39] However, an indirect approach to replication is meta-analysis.
     Bayesian inference is one alternative to significance testing. For example, Bayesian parameter estimation can
     provide rich information about the data from which researchers can draw inferences, while using uncertain priors
     that exert only minimal influence on the results when enough data is available. Psychologist Kruschke, John K. has
     suggested Bayesian estimation as an alternative for the t-test.[72] Alternatively two competing models/hypothesis can
     be compared using Bayes factors.[73] Bayesian methods could be criticized for requiring information that is seldom
     available in the
     cases where significance testing is most heavily used.
     Advocates of a Bayesian approach sometimes claim that the goal of a researcher is most often to objectively assess
     the probability that a hypothesis is true based on the data they have collected. Neither Fisher's significance testing,
     nor Neyman-Pearson hypothesis testing can provide this information, and do not claim to. The probability a
     hypothesis is true can only be derived from use of Bayes' Theorem, which was unsatisfactory to both the Fisher and
     Neyman-Pearson camps due to the explicit use of subjectivity in the form of the prior probability.[25][74] Fisher's
Statistical hypothesis testing                                                                                                                              67


     strategy is to sidestep this with the p-value (an objective index based on the data alone) followed by inductive
     inference, while Neyman-Pearson devised their approach of inductive behaviour.


     Education
     Statistics is increasingly being taught in schools with hypothesis testing being one of the elements taught.[75][76]
     Many conclusions reported in the popular press (political opinion polls to medical studies) are based on statistics. An
     informed public should understand the limitations of statistical conclusions[77][78] and many college fields of study
     require a course in statistics for the same reason.[77][78] An introductory college statistics class places much emphasis
     on hypothesis testing - perhaps half of the course. Such fields as literature and divinity now include findings based
     on statistical analysis (see the Bible Analyzer). An introductory statistics class teaches hypothesis testing as a
     cookbook process. Hypothesis testing is also taught at the postgraduate level. Statisticians learn how to create good
     statistical test procedures (like z, Student's t, F and chi-squared). Statistical hypothesis testing is considered a mature
     area within statistics,[79] but a limited amount of development continues.


     References
     [1] R. A. Fisher (1925).Statistical Methods for Research Workers, Edinburgh: Oliver and Boyd, 1925, p.43.
     [2] Schervish, M (1996) Theory of Statistics, p. 218. Springer ISBN 0-387-94546-6
     [3] Kaye, David H.; Freedman, David A. (2011). "Reference Guide on Statistics" (http:/ / www. nap. edu/ openbook. php?record_id=13163&
         page=211). Reference manual on scientific evidence (3rd ed.). Eagan, MN Washington, D.C: West National Academies Press. p. 259.
         ISBN 978-0-309-21421-6. .
     [4] Fisher, Sir Ronald A. (1956) [1935]. "Mathematics of a Lady Tasting Tea" (http:/ / books. google. com/ ?id=oKZwtLQTmNAC&
         pg=PA1512& dq="mathematics+ of+ a+ lady+ tasting+ tea"). In James Roy Newman. The World of Mathematics, volume 3 [Design of
         Experiments]. Courier Dover Publications. ISBN 978-0-486-41151-4. . Originally from Fisher's book Design of Experiments.
     [5] Box, Joan Fisher (1978). R.A. Fisher, The Life of a Scientist. New York: Wiley. p. 134. ISBN 0-471-09300-9.
     [6] Lehmann, E.L.; Romano, Joseph P. (2005). Testing Statistical Hypotheses (3E ed.). New York: Springer. ISBN 0-387-98864-5.
     [7] Adèr,J.H. (2008). Chapter 12: Modelling. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research
         Methods: A consultant's companion (pp. 183–209). Huizen, The Netherlands: Johannes van Kessel Publishing
     [8] Triola, Mario (2001). Elementary statistics (8 ed.). Boston: Addison-Wesley. p. 388. ISBN 0-201-61477-4.
     [9] Hinkelmann, Klaus and Kempthorne, Oscar (2008). Design and Analysis of Experiments. I and II (Second ed.). Wiley.
         ISBN 978-0-470-38551-7.
     [10] Montgomery, Douglas (2009). Design and analysis of experiments. Hoboken, NJ: Wiley. ISBN 978-0-470-12866-4.
     [11] Richard J. Larsen, Donna Fox Stroup (1976). Statistics in the Real World.
     [12] Hubbard, R.; Parsa, A. R.; Luthy, M. R. (1997). "The spread of statistical significance testing in psychology: The case of the Journal of
         Applied Psychology". Theory and Psychology 7 (4): 545–554. doi:10.1177/0959354397074006.
     [13] Moore, David (2003). Introduction to the practice of statistics. New York: W.H. Freeman and Co. p. 426. ISBN 9780716796572.
     [14] Huff, Darrell (1993). How to lie with statistics. New York: Norton. ISBN 0-393-31072-8.
     [15] Huff, Darrell (1991). How to lie with statistics. London: Penguin Books. ISBN 0-14-013629-0.
     [16] "Over the last fifty years, How to Lie with Statistics has sold more copies than any other statistical text." J. M. Steele. " Darrell Huff and
         Fifty Years of How to Lie with Statistics (http:/ / www-stat. wharton. upenn. edu/ ~steele/ Publications/ PDF/ TN148. pdf). Statistical Science,
         20 (3), 2005, 205–209.
     [17] C. S. Peirce (August 1878). "Illustrations of the Logic of Science VI: Deduction, Induction, and Hypothesis" (http:/ / en. wikisource. org/ w/
         index. php?oldid=3592335). Popular Science Monthly 13. . Retrieved 30 March 2012.
     [18] NIST handbook: Two-Sample t-Test for Equal Means (http:/ / www. itl. nist. gov/ div898/ handbook/ eda/ section3/ eda353. htm)
     [19] Steel, R.G.D, and Torrie, J. H., Principles and Procedures of Statistics with Special Reference to the Biological Sciences., McGraw Hill,
         1960, page 350.
     [20] Weiss, Neil A. (1999). Introductory Statistics (5th ed.). pp. 802. ISBN 0-201-59877-9.
     [21] NIST handbook: F-Test for Equality of Two Standard Deviations (http:/ / www. itl. nist. gov/ div898/ handbook/ eda/ section3/ eda359. htm)
         (Testing standard deviations the same as testing variances)
     [22] Raymond Hubbard, M.J. Bayarri, P Values are not Error Probabilities (http:/ / ftp. isds. duke. edu/ WorkingPapers/ 03-26. pdf). A working
         paper that explains the difference between Fisher's evidential p-value and the Neyman–Pearson Type I error rate           .
     [23] Stigler, Stephen M. (1986). The history of statistics : the measurement of uncertainty before 1900. Cambridge, Mass: Belknap Press of
         Harvard University Press. p. 134. ISBN 0-674-40340-1.
     [24] Fisher, R (1955). "Statistical Methods and Scientific Induction" (http:/ / www. phil. vt. edu/ dmayo/ PhilStatistics/ Triad/ Fisher 1955. pdf).
         Journal of the Royal Statistical Society. Series B (Methodological) 17 (1): 69–78. .
Statistical hypothesis testing                                                                                                                             68

     [25] Neyman, J (January 1, 1933). "On the Problem of the most Efficient Tests of Statistical Hypotheses.". Phil. Trans. R. Soc. Lond. A 231
         (694-706): 289–337. doi:10.1098/rsta.1933.0009.
     [26] Goodman, S N (June 15, 1999). "Toward evidence-based medical statistics. 1: The P Value Fallacy." (http:/ / annals. org/ article.
         aspx?articleid=712762). Ann Intern Med 130 (12): 995–1004. .
     [27] Fisher, R A (1926). "The Arrangement of Field Experiments." (http:/ / digital. library. adelaide. edu. au/ dspace/ bitstream/ 2440/ 15191/ 1/
         48. pdf). Journal of the Ministry of Agriculture of Great Britain 33 (N/A): 503–513. .
     [28] Sterne, J A C (August 2001). "Sifting the evidence--what's wrong with significance tests?" (http:/ / www. physther. org/ content/ 81/ 8/
         1464. full. pdf+ html). Physical Therapy 81 (8): 1464–1469. .
     [29] Fisher, R (1958). "The Nature of Probability" (http:/ / www. york. ac. uk/ depts/ maths/ histstat/ fisher272. pdf). Centennial Review 2 (N/A):
         261–274. .
     [30] Fisher, R (1930). "Inverse Probability" (http:/ / www. cse. buffalo. edu/ ~peter/ refs/ Historical/ Fisher_1930. pdf). Proceedings of the
         Cambridge Philosophical Society 26 (N/A): 528–535. .
     [31] Halpin, P F (Winter 2006). "Inductive Inference or Inductive Behavior: Fisher and Neyman: Pearson Approaches to Statistical Testing in
         Psychological Research (1940-1960)" (http:/ / www. jstor. org/ stable/ 20445367). The American Journal of Psychology 119 (4): 625–653. .
     [32] Gigerenzer, Gerd; Zeno Swijtink, Theodore Porter, Lorraine Daston, John Beatty, Lorenz Kruger (1990). "Part 3: The inference experts".
         The Empire of Chance: How Probability Changed Science and Everyday Life. Cambridge University Press. pp. 70–122.
         ISBN 978-0-521-39838-1.
     [33] Gigerenzer, G (November 2004). "Mindless statistics". The Journal of Socio-Economics 33 (5): 587–606. doi:10.1016/j.socec.2004.09.033.
     [34] Loftus, G R (1991). "On the Tyranny of Hypothesis Testing in the Social Sciences" (https:/ / www. ics. uci. edu/ ~sternh/ courses/ 210/
         loftus91_tyranny. pdf). Contemporary Psychology 36 (2): 102–105. .
     [35] Harlow, Lisa Lavoie; Stanley A. Mulaik; James H. Steiger, ed. (1997). What If There Were No Significance Tests?. Lawrence Erlbaum
         Associates. ISBN 978-0-8058-2634-0.
     [36] Morrison, Denton; Henkel, Ramon, ed. (2006) [1970]. The Significance Test Controversy. AldineTransaction. ISBN 0-202-30879-0.
     [37] McCloskey, Deirdre N.; Stephen T. Ziliak (2008). The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and
         Lives. University of Michigan Press. ISBN 0-472-05007-9.
     [38] Chow, Siu L. (1997). Statistical Significance: Rationale, Validity and Utility. ISBN 0-7619-5205-5.
     [39] Nickerson, Raymond S. (2000). "Null Hypothesis Significance Tests: A Review of an Old and Continuing Controversy". Psychological
         Methods 5 (2): 241–301. doi:10.1037/1082-989X.5.2.241. PMID 10937333.
     [40] Gliner, J A (2002). "Problems With Null Hypothesis Significance Testing (NHST): What Do the Textbooks Say?". The Journal of
         Experimental Education 71 (1): 83–92. doi:10.1080/00220970209602058.
     [41] Gigerenzer, G (2004). "The Null Ritual What You Always Wanted to Know About Significant Testing but Were Afraid to Ask" (http:/ /
         library. mpib-berlin. mpg. de/ ft/ gg/ GG_Null_2004. pdf). The SAGE Handbook of Quantitative Methodology for the Social Sciences N/A
         (N/A): 391–408. doi:10.4135/9781412986311. .
     [42] Raymond Hubbard, M.J. Bayarri, P Values are not Error Probabilities (http:/ / ftp. isds. duke. edu/ WorkingPapers/ 03-26. pdf). A working
         paper that explains the difference between Fisher's evidential p-value and the Neyman–Pearson Type I error rate       .
     [43] Raymond Hubbard, M.J. Bayarri, P Values are not Error Probabilities (http:/ / ftp. isds. duke. edu/ WorkingPapers/ 03-26. pdf). A working
         paper that explains the difference between Fisher's evidential p-value and the Neyman–Pearson Type I error rate       .
     [44] Cohen, J (December 1994). "The Earth Is Round (p < .05)" (http:/ / ist-socrates. berkeley. edu/ ~maccoun/ PP279_Cohen1. pdf). American
         Psychologist 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997. .
     [45] Bakan, D (December 1966). "The Test of Significance in Psychological Research". Psychological Bulletin 66 (6): 423–437.
         doi:10.1037/h0020412.
     [46] Kruschke, J K (September/October 2010). "Bayesian Data Analysis". Wiley Interdisciplinary Reviews: Cognitive Science 1 (5): 658–676.
         doi:10.1002/wcs.72.
     [47] Rothman, K J (January 1990). "No Adjustments Are Needed for Multiple Comparisons" (http:/ / www. jstor. org/ stable/ 20065622).
         Epidemiology 1 (1): 43–46. .
     [48] Nester, M R (1996). "An Applied Statistician's Creed". Journal of the Royal Statistical Society. Series C (Applied Statistics) 45 (4):
         401–410. JSTOR 2986064.
     [49] Savage, R (September 1957). "Nonparametric Statistics" (http:/ / www. jstor. org/ stable/ 2280903). Journal of the American Statistical
         Association 52 (279): 331–344. .
     [50] Cohen, J (December 1994). "The Earth Is Round (p < .05)" (http:/ / ist-socrates. berkeley. edu/ ~maccoun/ PP279_Cohen1. pdf). American
         Psychologist 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997. .
     [51] Meehl, P (June 1967). "Theory-Testing in Psychology and Physics: A Methodological Paradox" (http:/ / www. psych. ucsb. edu/ ~janusonis/
         meehl1967. pdf). Philosophy of Science 34 (2): 103–115. .
     [52] "Male and Female Students Make Gains Since 2000 at Grade 4; Males Outperform Females at all Three Grades in 2005" (http:/ /
         nationsreportcard. gov/ science_2005/ s0110. asp). The Nation's Report Card. U.S. Department of Education. . Retrieved 16 September 2012.
     [53] Hyde, Janet Shibley; Marcia C. Linn (27 October 2006). "Gender similarities in mathematics and science" (http:/ / www. sciencemag. org/
         content/ 314/ 5799/ 599. summary). Science 314 (5799): 599–600. doi:10.1126/science.1132154. PMID 17068246. . Retrieved 16 September
         2012.
Statistical hypothesis testing                                                                                                                                   69

     [54] Goodman, Steven (July 2008). "A Dirty Dozen: Twelve P-Value Misconceptions". Seminars in Hematology 45 (3): 135–140.
         doi:10.1053/j.seminhematol.2008.04.003. PMID 18582619.
     [55] Berger, J O (March 1987). "Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence". Journal of the American
         Statistical Association 82 (397): 112–122. JSTOR 2289131.
     [56] Goodman, S N (June 15, 1999). "Toward evidence-based medical statistics. 2: The Bayes factor." (http:/ / annals. org/ article.
         aspx?articleid=712763). Ann Intern Med 130 (12): 1005–1013. doi:10.7326/0003-4819-130-12-199906150-00009. PMID 10383350. .
     [57] Sutton, A J; Duval, SJ; Tweedie, RL; Abrams, KR; Jones, DR (10 June 2000). "Empirical assessment of effect of publication bias on
         meta-analyses". BMJ 320 (7249): 1574–1577. doi:10.1136/bmj.320.7249.1574. PMC 27401. PMID 10845965.
     [58] Vickers, A (April 1998). "Do Certain Countries Produce Only Positive Results? A Systematic Review of Controlled Trials" (http:/ / dx. doi.
         org/ 10. 1016/ S0197-2456(97)00150-5). Controlled Clinical Trials 19 (2): 159–166. .
     [59] Fanelli, D (April 7, 2010). "‘‘Positive’’ Results Increase Down the Hierarchy of the Sciences". PLOS ONE 5 (4:e10068): N/A.
         doi:10.1371/journal.pone.0010068.
     [60] Fanelli, D (January 2012volume=90). "Negative results are disappearing from most disciplines and countries". Scientometrics (3): 891–904.
         doi:10.1007/s11192-011-0494-7.
     [61] Simmons, J P (2011). "Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". Psychological Science 22
         (11): 1359–1366. doi:10.1177/0956797611417632.
     [62] Masicampo, E J (August 2, 2012). "A peculiar prevalence of p values just below .05" (http:/ / dx. doi. org/ 10. 1080/ 17470218. 2012.
         711335). The Quarterly Journal of Experimental Psychology 65 (11): 2271–2279. .
     [63] Wilkinson, Leland (1999). "Statistical Methods in Psychology Journals; Guidelines and Explanations". American Psychologist 54 (8):
         594–604. doi:10.1037/0003-066X.54.8.594.
     [64] "ICMJE: Obligation to Publish Negative Studies" (http:/ / www. icmje. org/ publishing_1negative. html). . Retrieved 3 September 2012.
         "Editors should seriously consider for publication any carefully done study of an important question, relevant to their readers, whether the
         results for the primary or any additional outcome are statistically significant. Failure to submit or publish findings because of lack of statistical
         significance is an important cause of publication bias."
     [65] Journal of Articles in Support of the Null Hypothesis website: JASNH homepage (http:/ / www. jasnh. com/ ). Volume 1 number 1 was
         published in 2002, and all articles are on psychology-related subjects.
     [66] Howell, David (2002). Statistical Methods for Psychology (5 ed.). Duxbury. p. 94. ISBN 0-534-37770-X.
     [67] Mayo, D. G.; Spanos, A. (2006). "Severe Testing as a Basic Concept in a Neyman-Pearson Philosophy of Induction". The British Journal
         for the Philosophy of Science 57 (2): 323. doi:10.1093/bjps/axl003.
     [68] Hubbard, R.; Armstrong, J. S. (2006). "Why We Don't Really Know What Statistical Significance Means: Implications for Educators".
         Journal of Marketing Education 28 (2): 114. doi:10.1177/0273475306288399. Preprint (http:/ / marketing. wharton. upenn. edu/ ideas/ pdf/
         Armstrong/ StatisticalSignificance. pdf)
     [69] Armstrong, J. Scott (2007). "Significance tests harm progress in forecasting" (http:/ / repository. upenn. edu/ cgi/ viewcontent.
         cgi?article=1104& context=marketing_papers& sei-redir=1& referer=http:/ / scholar. google. com/ scholar?q=Significance+ tests+ harm+
         progress+ in+ forecasting& hl=en& btnG=Search& as_sdt=1%2C5& as_sdtp=on#search="Significance tests harm progress forecasting").
         International Journal of Forecasting 23 (2): 321–327. doi:10.1016/j.ijforecast.2007.03.004. .
     [70] E. L. Lehmann (1997). "Testing Statistical Hypotheses: The Story of a Book". Statistical Science 12 (1): 48–52.
         doi:10.1214/ss/1029963261.
     [71] Jacob Cohen (December 1994). "The Earth Is Round (p < .05)". American Psychologist 49 (12): 997–1003.
         doi:10.1037/0003-066X.49.12.997. This paper lead to the review of statistical practices

     by the APA. Cohen was a member of the Task Force that did the review.
     [72] Kruschke, J K (July 9, 2012). "Bayesian Estimation Supersedes the t test". Journal of Experimental Psychology: General. N/A (N/A): N/A.
         doi:10.1037/a0029146.
     [73] Kass, R E (1993). Bayes factors and model uncertainty (http:/ / www. stat. washington. edu/ research/ reports/ 1993/ tr254. pdf). N/A.
         pp. N/A. .
     [74] Aldrich, J (2008). "R. A. Fisher on Bayes and Bayes' theorem" (http:/ / ba. stat. cmu. edu/ journal/ 2008/ vol03/ issue01/ aldrich. pdf).
         Bayesian Analysis 3 (1): 161–170. .
     [75] Mathematics > High School: Statistics & Probability > Introduction (http:/ / www. corestandards. org/ the-standards/ mathematics/
         hs-statistics-and-probability/ introduction/ ) Common Core State Standards Initiative (relates to USA students)
     [76] College Board Tests > AP: Subjects > Statistics (http:/ / www. collegeboard. com/ student/ testing/ ap/ sub_stats. html) The College Board
         (relates to USA students)
     [77] Huff, Darrell (1993). How to lie with statistics. New York: Norton. p. 8. ISBN 0-393-31072-8. 'Statistical methods and statistical terms are
         necessary in reporting the mass data of social and economic trends, business conditions, "opinion" polls, the census. But without writers who
         use the words with honesty and readers who know what they mean, the result can only be semantic nonsense.'
     [78] Snedecor, George W.; Cochran, William G. (1967). Statistical Methods (6 ed.). Ames, Iowa: Iowa State University Press. p. 3. "...the basic
         ideas in statistics assist us in thinking clearly about the problem, provide some guidance about the conditions that must be satisfied if sound
         inferences are to be made, and enable us to detect many inferences that have no good logical foundation."
     [79] Lehmann, E. L. (1997). "Testing statistical hypotheses: The story of a book". Statistical Science 12: 48. doi:10.1214/ss/1029963261.
Statistical hypothesis testing                                                                                           70


     Further reading
     • Lehmann E.L. (1992) "Introduction to Neyman and Pearson (1933) On the Problem of the Most Efficient Tests of
       Statistical Hypotheses". In: Breakthroughs in Statistics, Volume 1, (Eds Kotz, S., Johnson, N.L.),
       Springer-Verlag. ISBN 0-387-94037-5 (followed by reprinting of the paper)
     • Neyman, J.; Pearson, E.S. (1933). "On the Problem of the Most Efficient Tests of Statistical Hypotheses". Phil.
       Trans. R. Soc., Series A 231 (694–706): 289–337. doi:10.1098/rsta.1933.0009.


     External links
     • Hazewinkel, Michiel, ed. (2001), "Statistical hypotheses, verification of" (http://www.encyclopediaofmath.org/
       index.php?title=p/s087400), Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4
     • Wilson González, Georgina; Karpagam Sankaran (September 10, 1997). "Hypothesis Testing" (http://www.cee.
       vt.edu/ewr/environmental/teach/smprimer/hypotest/ht.html). Environmental Sampling & Monitoring Primer.
       Virginia Tech.
     • Bayesian critique of classical hypothesis testing (http://www.cs.ucsd.edu/users/goguen/courses/275f00/stat.
       html)
     • Critique of classical hypothesis testing highlighting long-standing qualms of statisticians (http://www.npwrc.
       usgs.gov/resource/methods/statsig/stathyp.htm)
     • Dallal GE (2007) The Little Handbook of Statistical Practice (http://www.tufts.edu/~gdallal/LHSP.HTM) (A
       good tutorial)
     • References for arguments for and against hypothesis testing (http://core.ecu.edu/psyc/wuenschk/StatHelp/
       NHST-SHIT.htm)
     • Statistical Tests Overview: (http://www.wiwi.uni-muenster.de/ioeb/en/organisation/pfaff/
       stat_overview_table.html) How to choose the correct statistical test
     • An Interactive Online Tool to Encourage Understanding Hypothesis Testing (http://wasser.heliohost.org/
       ?l=en)
     • A non mathematical way to understand Hypothesis Testing (http://simplifyingstats.com/data/
       HypothesisTesting.pdf)
Estimation theory                                                                                                              71



    Estimation theory
    Estimation theory is a branch of statistics and signal processing that deals with estimating the values of parameters
    based on measured/empirical data that has a random component. The parameters describe an underlying physical
    setting in such a way that their value affects the distribution of the measured data. An estimator attempts to
    approximate the unknown parameters using the measurements.
    For example, it is desired to estimate the proportion of a population of voters who will vote for a particular
    candidate. That proportion is the parameter sought; the estimate is based on a small random sample of voters.
    Or, for example, in radar the goal is to estimate the range of objects (airplanes, boats, etc.) by analyzing the two-way
    transit timing of received echoes of transmitted pulses. Since the reflected pulses are unavoidably embedded in
    electrical noise, their measured values are randomly distributed, so that the transit time must be estimated.
    In estimation theory, it is assumed the measured data is random with probability distribution dependent on the
    parameters of interest. For example, in electrical communication theory, the measurements which contain
    information regarding the parameters of interest are often associated with a noisy signal. Without randomness, or
    noise, the problem would be deterministic and estimation would not be needed.


    Estimation process
    The entire purpose of estimation theory is to arrive at an estimator — preferably an easily implementable one. The
    estimator takes the measured data as input and produces an estimate of the parameters.
    It is also preferable to derive an estimator that exhibits optimality. Estimator optimality usually refers to achieving
    minimum average error over some class of estimators, for example, a minimum variance unbiased estimator. In this
    case, the class is the set of unbiased estimators, and the average error measure is variance (average squared error
    between the value of the estimate and the parameter). However, optimal estimators do not always exist.
    These are the general steps to arrive at an estimator:
    • In order to arrive at a desired estimator, it is first necessary to determine a probability distribution for the
      measured data, and the distribution's dependence on the unknown parameters of interest. Often, the probability
      distribution may be derived from physical models that explicitly show how the measured data depends on the
      parameters to be estimated, and how the data is corrupted by random errors or noise. In other cases, the
      probability distribution for the measured data is simply "assumed", for example, based on familiarity with the
      measured data and/or for analytical convenience.
    • After deciding upon a probabilistic model, it is helpful to find the theoretically achievable (optimal) precision
      available to any estimator based on this model. The Cramér–Rao bound is useful for this.
    • Next, an estimator needs to be developed, or applied (if an already known estimator is valid for the model). There
      are a variety of methods for developing estimators; maximum likelihood estimators are often the default although
      they may be hard to compute or even fail to exist. If possible, the theoretical performance of the estimator should
      be derived and compared with the optimal performance found in the last step.
    • Finally, experiments or simulations can be run using the estimator to test its performance.
    After arriving at an estimator, real data might show that the model used to derive the estimator is incorrect, which
    may require repeating these steps to find a new estimator. A non-implementable or infeasible estimator may need to
    be scrapped and the process started anew.
    In summary, the estimator estimates the parameters of a physical model based on measured data.
Estimation theory                                                                                                                  72


    Basics
    To build a model, several statistical "ingredients" need to be known. These are needed to ensure the estimator has
    some mathematical tractability instead of being based on "good feel".
    The first is a set of statistical samples taken from a random vector (RV) of size N. Put into a vector,




    Secondly, there are the corresponding M parameters




    which need to be established with their continuous probability density function (pdf) or its discrete counterpart, the
    probability mass function (pmf)


    It is also possible for the parameters themselves to have a probability distribution (e.g., Bayesian statistics). It is then
    necessary to define the Bayesian probability


    After the model is formed, the goal is to estimate the parameters, commonly denoted           , where the "hat" indicates
    the estimate.
    One common estimator is the minimum mean squared error estimator, which utilizes the error between the estimated
    parameters and the actual value of the parameters


    as the basis for optimality. This error term is then squared and minimized for the MMSE estimator.


    Estimators
    Commonly used estimators and estimation methods, and topics related to them:
    •   Maximum likelihood estimators
    •   Bayes estimators
    •   Method of moments estimators
    •   Cramér–Rao bound
    •   Minimum mean squared error (MMSE), also known as Bayes least squared error (BLSE)
    •   Maximum a posteriori (MAP)
    •   Minimum variance unbiased estimator (MVUE)
    •   Best linear unbiased estimator (BLUE)
    •   Unbiased estimators — see estimator bias.
    •   Particle filter
    •   Markov chain Monte Carlo (MCMC)
    •   Kalman filter, and its various derivatives
    • Wiener filter
Estimation theory                                                                                                           73


    Examples

    Unknown constant in additive white Gaussian noise
    Consider a received discrete signal,      , of       independent samples that consists of an unknown constant    with
    additive white Gaussian noise (AWGN)                 with known variance     (i.e.,          ). Since the variance is
    known then the only unknown parameter is         .
    The model for the signal is then


    Two possible (of many) estimators are:

    •

    •                        which is the sample mean

    Both of these estimators have a mean of           , which can be shown through taking the expected value of each
    estimator



    and




    At this point, these two estimators would appear to perform the same. However, the difference between them
    becomes apparent when comparing the variances.



    and




    It would seem that the sample mean is a better estimator since its variance is lower for every N>1.

    Maximum likelihood
    Continuing the example using the maximum likelihood estimator, the probability density function (pdf) of the noise
    for one sample     is




    and the probability of      becomes (       can be thought of a               )



    By independence, the probability of     becomes




    Taking the natural logarithm of the pdf
Estimation theory                                                                                                                 74


    and the maximum likelihood estimator is


    Taking the first derivative of the log-likelihood function




    and setting it to zero




    This results in the maximum likelihood estimator




    which is simply the sample mean. From this example, it was found that the sample mean is the maximum likelihood
    estimator for   samples of a fixed, unknown parameter corrupted by AWGN.

    Cramér–Rao lower bound
    To find the Cramér–Rao lower bound (CRLB) of the sample mean estimator, it is first necessary to find the Fisher
    information number




    and copying from above




    Taking the second derivative



    and   finding    the     negative   expected   value   is    trivial   since   it   is   now   a   deterministic   constant



    Finally, putting the Fisher information into



    results in



    Comparing this to the variance of the sample mean (determined previously) shows that the sample mean is equal to
    the Cramér–Rao lower bound for all values of      and    . In other words, the sample mean is the (necessarily
    unique) efficient estimator, and thus also the minimum variance unbiased estimator (MVUE), in addition to being
    the maximum likelihood estimator.
Estimation theory                                                                                                               75


    Maximum of a uniform distribution
    One of the simplest non-trivial examples of estimation is the estimation of the maximum of a uniform distribution. It
    is used as a hands-on classroom exercise and to illustrate basic principles of estimation theory. Further, in the case of
    estimation based on a single sample, it demonstrates philosophical issues and possible misunderstandings in the use
    of maximum likelihood estimators and likelihood functions.
    Given a discrete uniform distribution                    with unknown maximum, the UMVU estimator for the
    maximum is given by



    where m is the sample maximum and k is the sample size, sampling without replacement.[1][2] This problem is
    commonly known as the German tank problem, due to application of maximum estimation to estimates of German
    tank production during World War II.
    The formula may be understood intuitively as:
           "The sample maximum plus the average gap between observations in the sample",
    the gap being added to compensate for the negative bias of the sample maximum as an estimator for the population
    maximum.[3]
    This has a variance of[1]




    so a standard deviation of approximately          , the (population) average size of a gap between samples; compare
        above. This can be seen as a very simple case of maximum spacing estimation.

    The sample maximum is the maximum likelihood estimator for the population maximum, but, as discussed above, it
    is biased.


    Applications
    Numerous fields require the use of estimation theory. Some of these fields include (but are by no means limited to):
    •   Interpretation of scientific experiments
    •   Signal processing
    •   Clinical trials
    •   Opinion polls
    •   Quality control
    •   Telecommunications
    •   Project management
    •   Software engineering
    •   Control theory (in particular Adaptive control)
    •   Network intrusion detection system
    •   Orbit determination
    Measured data are likely to be subject to noise or uncertainty and it is through statistical probability that optimal
    solutions are sought to extract as much information from the data as possible.
Estimation theory                                                                                                                                    76


    Notes
    [1] Johnson, Roger (1994), "Estimating the Size of a Population", Teaching Statistics (http:/ / www. rsscse. org. uk/ ts/ index. htm) 16 (2
        (Summer)): 50, doi:10.1111/j.1467-9639.1994.tb00688.x
    [2] Johnson, Roger (2006), "Estimating the Size of a Population" (http:/ / www. rsscse. org. uk/ ts/ gtb/ johnson. pdf), Getting the Best from
        Teaching Statistics (http:/ / www. rsscse. org. uk/ ts/ gtb/ contents. html),
    [3] The sample maximum is never more than the population maximum, but can be less, hence it is a biased estimator: it will tend to
        underestimate the population maximum.



    References

    References
    •   Theory of Point Estimation by E.L. Lehmann and G. Casella. (ISBN 0387985026)
    •   Systems Cost Engineering by Dale Shermon. (ISBN 978-0-566-08861-2)
    •   Mathematical Statistics and Data Analysis by John Rice. (ISBN 0-534-209343)
    •   Fundamentals of Statistical Signal Processing: Estimation Theory by Steven M. Kay (ISBN 0-13-345711-7)
    •   An Introduction to Signal Detection and Estimation by H. Vincent Poor (ISBN 0-387-94173-8)
    • Detection, Estimation, and Modulation Theory, Part 1 by Harry L. Van Trees (ISBN 0-471-09517-6; website
      (http://gunston.gmu.edu/demt/demtp1/))
    • Optimal State Estimation: Kalman, H-infinity, and Nonlinear Approaches by Dan Simon website (http://
      academic.csuohio.edu/simond/estimation/)
    • Ali H. Sayed, Adaptive Filters, Wiley, NJ, 2008, ISBN 978-0-470-25388-5.
    • Ali H. Sayed, Fundamentals of Adaptive Filtering, Wiley, NJ, 2003, ISBN 0-471-46126-1.
    • Thomas Kailath, Ali H. Sayed, and Babak Hassibi, Linear Estimation, Prentice-Hall, NJ, 2000, ISBN
      978-0-13-022464-4.
    • Babak Hassibi, Ali H. Sayed, and Thomas Kailath, Indefinite Quadratic Estimation and Control: A Unified
      Approach to H2 and Hoo Theories, Society for Industrial & Applied Mathematics (SIAM), PA, 1999, ISBN
      978-0-89871-411-1.
    • V.G.Voinov, M.S.Nikulin, "Unbiased estimators and their applications. Vol.1: Univariate case", Kluwer
      Academic Publishers, 1993, ISBN 0-7923-2382-3.
    • V.G.Voinov, M.S.Nikulin, "Unbiased estimators and their applications. Vol.2: Multivariate case", Kluwer
      Academic Publishers, 1996, ISBN 0-7923-3939-8.
Semiparametric model                                                                                                                  77



    Semiparametric model
    In statistics a semiparametric model is a model that has parametric and nonparametric components.
    A model is a collection of distributions:                      indexed by a parameter         .
    • A parametric model is one in which the indexing parameter is a finite-dimensional vector (in                   -dimensional
       Euclidean space for some integer         ); i.e. the set of possible values for     is a subset of     , or            . In
      this case we say that is finite-dimensional.
    • In nonparametric models, the set of possible values of the parameter               is a subset of some space, not necessarily
       finite dimensional. For example, we might consider the set of all distributions with mean 0. Such spaces are
       vector spaces with topological structure, but may not be finite dimensional as vector spaces. Thus,          for
      some possibly infinite dimensional space .
    • In semiparametric models, the parameter has both a finite dimensional component and an infinite dimensional
      component (often a real-valued function defined on the real line). Thus the parameter space in a
       semiparametric model satisfies                       , where     is an infinite dimensional space.
    It may appear at first that semiparametric models include nonparametric models, since they have an infinite
    dimensional as well as a finite dimensional component. However, a semiparametric model is considered to be
    "smaller" than a completely nonparametric model because we are often interested only in the finite-dimensional
    component of . That is, we are not interested in estimating the infinite-dimensional component. In nonparametric
    models, by contrast, the primary interest is in estimating the infinite dimensional parameter. Thus the estimation task
    is statistically harder in nonparametric models.
    These models often use smoothing or kernels.


    Example
    A well-known example of a semiparametric model is the Cox proportional hazards model. If we are interested in
    studying the time to an event such as death due to cancer or failure of a light bulb, the Cox model specifies the
    following distribution function for     :



    where         is a known function of time (the covariate vector at time                  ), and     and            are unknown
    parameters.                    . Here       is finite dimensional and is of interest;             is an unknown non-negative
    function of time (known as the baseline hazard function) and is often a nuisance parameter. The collection of
    possible candidates for         is infinite dimensional.
Non-parametric statistics                                                                                                       78



     Non-parametric statistics
     In statistics, the term non-parametric statistics has at least two different meanings:
     1. The first meaning of non-parametric covers techniques that do not rely on data belonging to any particular
        distribution. These include, among others:
           • distribution free methods, which do not rely on assumptions that the data are drawn from a given
              probability distribution. As such it is the opposite of parametric statistics. It includes non-parametric
              statistical models, inference and statistical tests.
           • non-parametric statistics (in the sense of a statistic over data, which is defined to be a function on a sample
              that has no dependency on a parameter), whose interpretation does not depend on the population fitting any
              parametrized distributions. Statistics based on the ranks of observations are one example of such statistics
              and these play a central role in many non-parametric approaches.
     2. The second meaning of non-parametric covers techniques that do not assume that the structure of a model is
        fixed. Typically, the model grows in size to accommodate the complexity of the data. In these techniques,
        individual variables are typically assumed to belong to parametric distributions, and assumptions about the types
        of connections among variables are also made. These techniques include, among others:
           • non-parametric regression, which refers to modeling where the structure of the relationship between
             variables is treated non-parametrically, but where nevertheless there may be parametric assumptions about
             the distribution of model residuals.
           • non-parametric hierarchical Bayesian models, such as models based on the Dirichlet process, which allow
             the number of latent variables to grow as necessary to fit the data, but where individual variables still follow
             parametric distributions and even the process controlling the rate of growth of latent variables follows a
             parametric distribution.


     Applications and purpose
     Non-parametric methods are widely used for studying populations that take on a ranked order (such as movie
     reviews receiving one to four stars). The use of non-parametric methods may be necessary when data have a ranking
     but no clear numerical interpretation, such as when assessing preferences. In terms of levels of measurement,
     non-parametric methods result in "ordinal" data.
     As non-parametric methods make fewer assumptions, their applicability is much wider than the corresponding
     parametric methods. In particular, they may be applied in situations where less is known about the application in
     question. Also, due to the reliance on fewer assumptions, non-parametric methods are more robust.
     Another justification for the use of non-parametric methods is simplicity. In certain cases, even when the use of
     parametric methods is justified, non-parametric methods may be easier to use. Due both to this simplicity and to their
     greater robustness, non-parametric methods are seen by some statisticians as leaving less room for improper use and
     misunderstanding.
     The wider applicability and increased robustness of non-parametric tests comes at a cost: in cases where a parametric
     test would be appropriate, non-parametric tests have less power. In other words, a larger sample size can be required
     to draw conclusions with the same degree of confidence.
Non-parametric statistics                                                                                                     79


     Non-parametric models
     Non-parametric models differ from parametric models in that the model structure is not specified a priori but is
     instead determined from data. The term non-parametric is not meant to imply that such models completely lack
     parameters but that the number and nature of the parameters are flexible and not fixed in advance.
     • A histogram is a simple nonparametric estimate of a probability distribution
     • Kernel density estimation provides better estimates of the density than histograms.
     • Nonparametric regression and semiparametric regression methods have been developed based on kernels, splines,
       and wavelets.
     • Data envelopment analysis provides efficiency coefficients similar to those obtained by multivariate analysis
       without any distributional assumption.


     Methods
     Non-parametric (or distribution-free) inferential statistical methods are mathematical procedures for statistical
     hypothesis testing which, unlike parametric statistics, make no assumptions about the probability distributions of the
     variables being assessed. The most frequently used tests include
     • Anderson–Darling test
     •   Statistical Bootstrap Methods
     •   Cochran's Q
     •   Cohen's kappa
     •   Friedman two-way analysis of variance by ranks
     •   Kaplan–Meier
     •   Kendall's tau
     •   Kendall's W
     •   Kolmogorov–Smirnov test
     •   Kruskal-Wallis one-way analysis of variance by ranks
     •   Kuiper's test
     •   Logrank Test
     •   Mann–Whitney U or Wilcoxon rank sum test
     •   McNemar's test
     •   median test
     •   Pitman's permutation test
     •   Rank products
     •   Siegel–Tukey test
     •   sign test
     •   Spearman's rank correlation coefficient
     •   Wald–Wolfowitz runs test
     •   Wilcoxon signed-rank test.
Non-parametric statistics                                                                                                   80


     General references
     • Corder, G.W. & Foreman, D.I. (2009) Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach,
       Wiley ISBN 978-0-470-45461-9
     • Gibbons, Jean Dickinson and Chakraborti, Subhabrata (2003) Nonparametric Statistical Inference, 4th Ed. CRC
       ISBN 0-8247-4052-1
     • Hettmansperger, T. P.; McKean, J. W. (1998). Robust nonparametric statistical methods. Kendall's Library of
       Statistics. 5 (First ed.). London: Edward Arnold. pp. xiv+467 pp.. ISBN 0-340-54937-8, 0-471-19479-4.
       MR1604954.
     • Wasserman, Larry (2007) All of nonparametric statistics, Springer. ISBN 0-387-25145-6
     • Bagdonavicius, V., Kruopis, J., Nikulin, M.S. (2011). "Non-parametric tests for complete data", ISTE&WILEY:
       London&Hoboken. ISBN 978-1-84821-269-5



     Simulation
     Simulation is the imitation of the operation of a
     real-world process or system over time.[1] The act of
     simulating something first requires that a model be
     developed; this model represents the key characteristics
     or behaviors of the selected physical or abstract system or
     process. The model represents the system itself, whereas
     the simulation represents the operation of the system over
     time.

     Simulation is used in many contexts, such as simulation
     of technology for performance optimization, safety
     engineering, testing, training, education, and video
                                                                         Wooden mechanical horse simulator during WWI.
     games. Training simulators include flight simulators for
     training aircraft pilots to provide them with a lifelike
     experience. Simulation is also used with scientific modelling of natural systems or human systems to gain insight
     into their functioning.[2] Simulation can be used to show the eventual real effects of alternative conditions and
     courses of action. Simulation is also used when the real system cannot be engaged, because it may not be accessible,
     or it may be dangerous or unacceptable to engage, or it is being designed but not yet built, or it may simply not
     exist.[3]

     Key issues in simulation include acquisition of valid source information about the relevant selection of key
     characteristics and behaviours, the use of simplifying approximations and assumptions within the simulation, and
     fidelity and validity of the simulation outcomes.
Simulation                                                                                                                        81


    Classification and terminology
    Historically, simulations used in different fields developed largely
    independently, but 20th century studies of Systems theory and
    Cybernetics combined with spreading use of computers across all those
    fields have led to some unification and a more systematic view of the
    concept.
    Physical simulation refers to simulation in which physical objects are
    substituted for the real thing (some circles[4] use the term for computer
    simulations modelling selected laws of physics, but this article doesn't).
    These physical objects are often chosen because they are smaller or
    cheaper than the actual object or system.
    Interactive simulation is a special kind of physical simulation, often
    referred to as a human in the loop simulation, in which physical
    simulations include human operators, such as in a flight simulator or a
    driving simulator.
    Human in the loop simulations can include a computer simulation as a
    so-called synthetic environment.[5]
                                                                                   Human-in-the-loop simulation of outer space.

    Computer simulation
    A computer simulation (or "sim") is an attempt to model a real-life or
    hypothetical situation on a computer so that it can be studied to see
    how the system works. By changing variables in the simulation,
    predictions may be made about the behaviour of the system. It is a tool
    to virtually investigate the behaviour of the system under study.[1]
    Computer simulation has become a useful part of modeling many
    natural systems in physics, chemistry and biology,[6] and human           Visualization of a direct numerical simulation
    systems in economics and social science (the computational sociology)                         model.

    as well as in engineering to gain insight into the operation of those
    systems. A good example of the usefulness of using computers to simulate can be found in the field of network
    traffic simulation. In such simulations, the model behaviour will change each simulation according to the set of
    initial parameters assumed for the environment.

    Traditionally, the formal modeling of systems has been via a mathematical model, which attempts to find analytical
    solutions enabling the prediction of the behaviour of the system from a set of parameters and initial conditions.
    Computer simulation is often used as an adjunct to, or substitution for, modeling systems for which simple closed
    form analytic solutions are not possible. There are many different types of computer simulation, the common feature
    they all share is the attempt to generate a sample of representative scenarios for a model in which a complete
    enumeration of all possible states would be prohibitive or impossible.
    Several software packages exist for running computer-based simulation modeling (e.g. Monte Carlo simulation,
    stochastic modeling, multimethod modeling) that makes all the modeling almost effortless.
    Modern usage of the term "computer simulation" may encompass virtually any computer-based representation.
Simulation                                                                                                                       82


    Computer science
    In computer science, simulation has some specialized meanings: Alan Turing used the term "simulation" to refer to
    what happens when a universal machine executes a state transition table (in modern terminology, a computer runs a
    program) that describes the state transitions, inputs and outputs of a subject discrete-state machine. The computer
    simulates the subject machine. Accordingly, in theoretical computer science the term simulation is a relation between
    state transition systems, useful in the study of operational semantics.
    Less theoretically, an interesting application of computer simulation is to simulate computers using computers. In
    computer architecture, a type of simulator, typically called an emulator, is often used to execute a program that has
    to run on some inconvenient type of computer (for example, a newly designed computer that has not yet been built or
    an obsolete computer that is no longer available), or in a tightly controlled testing environment (see Computer
    architecture simulator and Platform virtualization). For example, simulators have been used to debug a
    microprogram or sometimes commercial application programs, before the program is downloaded to the target
    machine. Since the operation of the computer is simulated, all of the information about the computer's operation is
    directly available to the programmer, and the speed and execution of the simulation can be varied at will.
    Simulators may also be used to interpret fault trees, or test VLSI logic designs before they are constructed. Symbolic
    simulation uses variables to stand for unknown values.
    In the field of optimization, simulations of physical processes are often used in conjunction with evolutionary
    computation to optimize control strategies...


    Simulation in education and training
    Simulation is extensively used for educational purposes. It is frequently used by way of adaptive hypermedia.
    Simulation is often used in the training of civilian and military personnel.[7] This usually occurs when it is
    prohibitively expensive or simply too dangerous to allow trainees to use the real equipment in the real world. In such
    situations they will spend time learning valuable lessons in a "safe" virtual environment yet living a lifelike
    experience (or at least it is the goal). Often the convenience is to permit mistakes during training for a safety-critical
    system. For example, in simSchool [8] teachers practice classroom management and teaching techniques on
    simulated students, which avoids "learning on the job" that can damage real students. There is a distinction, though,
    between simulations used for training and Instructional simulation.
    Training simulations typically come in one of three categories:[9]
    • "live" simulation (where actual players use genuine systems in a real environment);
    • "virtual" simulation (where actual players use simulated systems in a synthetic environment [5]), or
    • "constructive" simulation (where simulated players use simulated systems in a synthetic environment).
      Constructive simulation is often referred to as "wargaming" since it bears some resemblance to table-top war
      games in which players command armies of soldiers and equipment that move around a board.
    In standardized tests, "live" simulations are sometimes called "high-fidelity", producing "samples of likely
    performance", as opposed to "low-fidelity", "pencil-and-paper" simulations producing only "signs of possible
    performance",[10] but the distinction between high, moderate and low fidelity remains relative, depending on the
    context of a particular comparison.
    Simulations in education are somewhat like training simulations. They focus on specific tasks. The term 'microworld'
    is used to refer to educational simulations which model some abstract concept rather than simulating a realistic
    object or environment, or in some cases model a real world environment in a simplistic way so as to help a learner
    develop an understanding of the key concepts. Normally, a user can create some sort of construction within the
    microworld that will behave in a way consistent with the concepts being modeled. Seymour Papert was one of the
    first to advocate the value of microworlds, and the Logo programming environment developed by Papert is one of
    the most famous microworlds. As another example, the Global Challenge Award online STEM learning web site
Simulation                                                                                                                            83


    uses microworld simulations to teach science concepts related to global warming and the future of energy. Other
    projects for simulations in educations are Open Source Physics, NetSim etc.
    Management games (or business simulations) have been finding favour in business education in recent years.[11]
    Business simulations that incorporate a dynamic model enable experimentation with business strategies in a risk free
    environment and provide a useful extension to case study discussions.
    Social simulations may be used in social science classrooms to illustrate social and political processes in
    anthropology, economics, history, political science, or sociology courses, typically at the high school or university
    level. These may, for example, take the form of civics simulations, in which participants assume roles in a simulated
    society, or international relations simulations in which participants engage in negotiations, alliance formation, trade,
    diplomacy, and the use of force. Such simulations might be based on fictitious political systems, or be based on
    current or historical events. An example of the latter would be Barnard College's Reacting to the Past series of
    historical educational games.[12] The National Science Foundation has also supported the creation of reacting games
    that address science and math education.[13]
    In recent years, there has been increasing use of social simulations for staff training in aid and development agencies.
    The Carana simulation, for example, was first developed by the United Nations Development Programme, and is
    now used in a very revised form by the World Bank for training staff to deal with fragile and conflict-affected
    countries.[14]


    Common user interaction systems for virtual simulations
    Virtual simulations represent a specific category of simulation that utilizes simulation equipment to create a
    simulated world for the user. Virtual simulations allow users to interact with a virtual world. Virtual worlds operate
    on platforms of integrated software and hardware components. In this manner, the system can accept input from the
    user (e.g., body tracking, voice/sound recognition, physical controllers) and produce output to the user (e.g., visual
    display, aural display, haptic display) .[15] Virtual Simulations use the aforementioned modes of interaction to
    produce a sense of immersion for the user.


    Virtual simulation input hardware
    There is a wide variety of input hardware
    available to accept user input for virtual
    simulations. The following list briefly
    describes several of them:
    Body tracking The motion capture method
    is often used to record the user’s movements
    and translate the captured data into inputs
    for the virtual simulation. For example, if a
    user physically turns their head, the motion
    would be captured by the simulation
    hardware in some way and translated to a
    corresponding shift in view within the
    simulation.

    • Capture suits and/or gloves may be used
                                                         Motorcycle simulator of Bienal do Automóvel exhibition, in Belo Horizonte,
      to capture movements of users body                                                  Brazil.
      parts. The systems may have sensors
Simulation                                                                                                                     84


      incorporated inside them to sense movements of different body parts (e.g., fingers). Alternatively, these systems
      may have exterior tracking devices or marks that can be detected by external ultrasound, optical receivers or
      electromagnetic sensors. Internal inertial sensors are also available on some systems. The units may transmit data
      either wirelessly or through cables.
    • Eye trackers can also be used to detect eye movements so that the system can determine precisely where a user is
      looking at any given instant.
    Physical controllers Physical controllers provide input to the simulation only through direct manipulation by the
    user. In virtual simulations, tactile feedback from physical controllers is highly desirable in a number of simulation
    environments.
    • Omni directional treadmills can be used to capture the users locomotion as they walk or run.
    • High fidelity instrumentation such as instrument panels in virtual aircraft cockpits provides users with actual
      controls to raise the level of immersion. For example, pilots can use the actual global positioning system controls
      from the real device in a simulated cockpit to help them practice procedures with the actual device in the context
      of the integrated cockpit system.
    Voice/sound recognition This form of interaction may be used either to interact with agents within the simulation
    (e.g., virtual people) or to manipulate objects in the simulation (e.g., information). Voice interaction presumably
    increases the level of immersion for the user.
    • Users may use headsets with boom microphones, lapel microphones or the room may be equipped with
      strategically located microphones.
    Current research into user input systems Research in future input systems hold a great deal of promise for virtual
    simulations. Systems such as brain-computer interfaces (BCIs)Brain-computer interface offer the ability to further
    increase the level of immersion for virtual simulation users. Lee, Keinrath, Scherer, Bischof, Pfurtscheller [16] proved
    that naïve subjects could be trained to use a BCI to navigate a virtual apartment with relative ease. Using the BCI,
    the authors found that subjects were able to freely navigate the virtual environment with relatively minimal effort. It
    is possible that these types of systems will become standard input modalities in future virtual simulation systems.
    Simulation is a one of the part of an engineering students and also imp for main electrical students its come in form
    of education purpose.


    Virtual simulation output hardware
    There is a wide variety of output hardware available to deliver stimulus to users in virtual simulations. The following
    list briefly describes several of them:
    Visual display Visual displays provide the visual stimulus to the user.
    • Stationary displays can vary from a conventional desktop display to 360-degree wrap around screens to stereo
      three-dimensional screens. Conventional desktop displays can vary in size from 15 to 60+ inches. Wrap around
      screens are typically utilized in what is known as a Cave Automatic Virtual Environment (CAVE) Cave
      Automatic Virtual Environment. Stereo three-dimensional screens produce three-dimensional images either with
      or without special glasses—depending on the design.
    • Head mounted displays (HMDs) have small displays that are mounted on headgear worn by the user. These
      systems are connected directly into the virtual simulation to provide the user with a more immersive experience.
      Weight, update rates and field of view are some of the key variables that differentiate HMDs. Naturally, heavier
      HMDs are undesirable as they cause fatigue over time. If the update rate is too slow, the system is unable to
      update the displays fast enough to correspond with a quick head turn by the user. Slower update rates tend to
      cause simulation sickness and disrupt the sense of immersion. Field of view or the angular extent of the world that
      is seen at a given moment Field of view can vary from system to system and has been found to affect the users
      sense of immersion.
Simulation                                                                                                                   85


    Aural display Several different types of audio systems exist to help the user hear and localize sounds spatially.
    Special software can be used to produce 3D audio effects 3D audio to create the illusion that sound sources are
    placed within a defined three-dimensional space around the user.
    • Stationary conventional speaker systems may be used provide dual or multi-channel surround sound. However,
      external speakers are not as effective as headphones in producing 3D audio effects.[15]
    • Conventional headphones offer a portable alternative to stationary speakers. They also have the added advantages
      of masking real world noise and facilitate more effective 3D audio sound effects.[15]
    Haptic display These displays provide sense of touch to the user Haptic technology. This type of output is
    sometimes referred to as force feedback.
    • Tactile tile displays use different types of actuators such as inflatable bladders, vibrators, low frequency
      sub-woofers, pin actuators and/or thermo-actuators to produce sensations for the user.
    • End effector displays can respond to users inputs with resistance and force.[15] These systems are often used in
      medical applications for remote surgeries that employ robotic instruments.[17]
    Vestibular display These displays provide a sense of motion to the user Motion simulator. They often manifest as
    motion bases for virtual vehicle simulation such as driving simulators or flight simulators. Motion bases are fixed in
    place but use actuators to move the simulator in ways that can produce the sensations pitching, yawing or rolling.
    The simulators can also move in such a way as to produce a sense of acceleration on all axes (e.g., the motion base
    can produce the sensation of falling).


    Clinical healthcare simulators
    Medical simulators are increasingly being developed and deployed to teach therapeutic and diagnostic procedures as
    well as medical concepts and decision making to personnel in the health professions. Simulators have been
    developed for training procedures ranging from the basics such as blood draw, to laparoscopic surgery [18] and
    trauma care. They are also important to help on prototyping new devices[19] for biomedical engineering problems.
    Currently, simulators are applied to research and develop tools for new therapies,[20] treatments[21] and early
    diagnosis[22] in medicine.
    Many medical simulators involve a computer connected to a plastic simulation of the relevant anatomy.
    Sophisticated simulators of this type employ a life size mannequin that responds to injected drugs and can be
    programmed to create simulations of life-threatening emergencies. In other simulations, visual components of the
    procedure are reproduced by computer graphics techniques, while touch-based components are reproduced by haptic
    feedback devices combined with physical simulation routines computed in response to the user's actions. Medical
    simulations of this sort will often use 3D CT or MRI scans of patient data to enhance realism. Some medical
    simulations are developed to be widely distributed (such as web-enabled simulations [23] and procedural simulations
    [24]
         that can be viewed via standard web browsers) and can be interacted with using standard computer interfaces,
    such as the keyboard and mouse.
    Another important medical application of a simulator — although, perhaps, denoting a slightly different meaning of
    simulator — is the use of a placebo drug, a formulation that simulates the active drug in trials of drug efficacy (see
    Placebo (origins of technical term)).


    Improving patient safety
    Patient safety is a concern in the medical industry. Patients have been known to suffer injuries and even death due to
    management error, and lack of using best standards of care and training. According to Building a National Agenda
    for Simulation-Based Medical Education (Eder-Van Hook, Jackie, 2004), “A health care provider’s ability to react
    prudently in an unexpected situation is one of the most critical factors in creating a positive outcome in medical
    emergency, regardless of whether it occurs on the battlefield, freeway, or hospital emergency room.” simulation.
Simulation                                                                                                                   86


    Eder-Van Hook (2004) also noted that medical errors kill up to 98,000 with an estimated cost between $37 and $50
    million and $17 to $29 billion for preventable adverse events dollars per year. “Deaths due to preventable adverse
    events exceed deaths attributable to motor vehicle accidents, breast cancer, or AIDS” Eder-Van Hook (2004). With
    these types of statistics it is no wonder that improving patient safety is a prevalent concern in the industry.
    Innovative simulation training solutions are now being used to train medical professionals in an attempt to reduce the
    number of safety concerns that have adverse effects on the patients. However, according to the article Does
    Simulation Improve Patient Safety? Self-efficacy, Competence, Operational Performance, and Patient Safety
    (Nishisaki A., Keren R., and Nadkarni, V., 2007), the jury is still out. Nishisaki states that “There is good evidence
    that simulation training improves provider and team self-efficacy and competence on manikins. There is also good
    evidence that procedural simulation improves actual operational performance in clinical settings.[25] However, no
    evidence yet shows that crew resource management training through simulation, despite its promise, improves team
    operational performance at the bedside. Also, no evidence to date proves that simulation training actually improves
    patient outcome. Even so, confidence is growing in the validity of medical simulation as the training tool of the
    future.” This could be because there are not enough research studies yet conducted to effectively determine the
    success of simulation initiatives to improve patient safety. Examples of [recently implemented] research simulations
    used to improve patient care [and its funding] can be found at Improving Patient Safety through Simulation Research
    (US Department of Human Health Services) http://www.ahrq.gov/qual/simulproj.htm.
    One such attempt to improve patient safety through the use of simulations training is pediatric care to deliver
    just-in-time service or/and just-in-place. This training consists of 20 minutes of simulated training just before
    workers report to shift. It is hoped that the recentness of the training will increase the positive and reduce the
    negative results that have generally been associated with the procedure. The purpose of this study is to determine if
    just-in-time training improves patient safety and operational performance of orotracheal intubation and decrease
    occurrences of undesired associated events and “to test the hypothesis that high fidelity simulation may enhance the
    training efficacy and patient safety in simulation settings.” The conclusion as reported in Abstract P38: Just-In-Time
    Simulation Training Improves ICU Physician Trainee Airway Resuscitation Participation without Compromising
    Procedural Success or Safety (Nishisaki A., 2008), were that simulation training improved resident participation in
    real cases; but did not sacrifice the quality of service. It could be therefore hypothesized that by increasing the
    number of highly trained residents through the use of simulation training, that the simulation training does in fact
    increase patient safety. This hypothesis would have to be researched for validation and the results may or may not
    generalize to other situations.


    History of simulation in healthcare
    The first medical simulators were simple models of human patients.[26]
    Since antiquity, these representations in clay and stone were used to demonstrate clinical features of disease states
    and their effects on humans. Models have been found from many cultures and continents. These models have been
    used in some cultures (e.g., Chinese culture) as a "diagnostic" instrument, allowing women to consult male
    physicians while maintaining social laws of modesty. Models are used today to help students learn the anatomy of
    the musculoskeletal system and organ systems.[26]
Simulation                                                                                                                                87


    Type of models
    Active models
             Active models that attempt to reproduce living anatomy or physiology are recent developments. The famous
             “Harvey” mannequin was developed at the University of Miami and is able to recreate many of the physical
             findings of the cardiology examination, including palpation, auscultation, and electrocardiography.[27]
    Interactive models
             More recently, interactive models have been developed that respond to actions taken by a student or
             physician.[27] Until recently, these simulations were two dimensional computer programs that acted more like
             a textbook than a patient. Computer simulations have the advantage of allowing a student to make judgments,
             and also to make errors. The process of iterative learning through assessment, evaluation, decision making,
             and error correction creates a much stronger learning environment than passive instruction.
    Computer simulators
             Simulators have been proposed as an ideal tool for assessment of
             students for clinical skills.[28] For patients, "cybertherapy" can be
             used for sessions simulating traumatic expericences, from fear of
             heights to social anxiety.[29]
             Programmed patients and simulated clinical situations, including
             mock disaster drills, have been used extensively for education
             and evaluation. These “lifelike” simulations are expensive, and
             lack reproducibility. A fully functional "3Di" simulator would be
             the most specific tool available for teaching and measurement of
             clinical skills. Gaming platforms have been applied to create           3DiTeams learner is percussing the patient's chest
                                                                                                 in virtual field hospital
             these virtual medical environments to create an interactive
             method for learning and application of information in a clinical
             context.[30][31]

             Immersive disease state simulations allow a doctor or HCP to experience what a disease actually feels like.
             Using sensors and transducers symptomatic effects can be delivered to a participant allowing them to
             experience the patients disease state.
             Such a simulator meets the goals of an objective and standardized examination for clinical competence.[32]
             This system is superior to examinations that use "standard patients" because it permits the quantitative
             measurement of competence, as well as reproducing the same objective findings.[33]


    Simulation in entertainment
    Simulation in entertainment encompasses many large and popular industries such as film, television, video games
    (including serious games) and rides in theme parks. Although modern simulation is thought to have its roots in
    training and the military, in the 20th century it also became a conduit for enterprises which were more hedonistic in
    nature. Advances in technology in the 1980s and 1990s caused simulation to become more widely used and it began
    to appear in movies such as Jurassic Park (1993) and in computer-based games such as Atari’s Battlezone (1980).
Simulation                                                                                                                    88


    History

    Early history (1940’s and 50’s)
    The first simulation game may have been created as early as 1947 by Thomas T. Goldsmith Jr. and Estle Ray Mann.
    This was a straightforward game that simulated a missile being fired at a target. The curve of the missile and its
    speed could be adjusted using several knobs. In 1958 a computer game called “Tennis for Two” was created by Willy
    Higginbotham which simulated a tennis game between two players who could both play at the same time using hand
    controls and was displayed on an oscilloscope.[34] This was one of the first electronic video games to use a graphical
    display.

    Modern simulation (1980’s-present)
    Advances in technology in the 1980s made the computer more affordable and more capable than they were in
    previous decades [35] which facilitated the rise of computer such as the Xbox gaming. The first video game consoles
    released in the 1970s and early 1980s fell prey to the industry crash in 1983, but in 1985, Nintendo released the
    Nintendo Entertainment System (NES) which became the best selling console in video game history.[36] In the
    1990s, computer games became widely popular with the release of such game as The Sims and Command & Conquer
    and the still increasing power of desktop computers. Today, computer simulation games such as World of Warcraft
    are played by millions of people around the world.
    Computer-generated imagery was used in film to simulate objects as early as 1976, though in 1982, the film Tron
    was the first film to use computer-generated imagery for more than a couple of minutes. However, the commercial
    failure of the movie may have caused the industry to step away from the technology.[37] In 1993, the film Jurassic
    Park became the first popular film to use computer-generated graphics extensively, integrating the simulated
    dinosaurs almost seamlessly into live action scenes. This event transformed the film industry; in 1995, the film Toy
    Story was the first film to use only computer-generated images and by the new millennium computer generated
    graphics were the leading choice for special effects in films.[38]
    Simulators have been used for entertainment since the Link Trainer in the 1930s.[39] The first modern simulator ride
    to open at a theme park was Disney’s Star Tours in 1987 soon followed by Universal’s The Funtastic World of
    Hanna-Barbera in 1990 which was the first ride to be done entirely with computer graphics.[40]


    Examples of entertainment simulation

    Computer and video games
    Simulation games, as opposed to other genres of video and computer games, represent or simulate an environment
    accurately. Moreover, they represent the interactions between the playable characters and the environment
    realistically. These kinds of games are usually more complex in terms of game play.[41] Simulation games have
    become incredibly popular among people of all ages.[42] Popular simulation games include SimCity, Tiger Woods
    PGA Tour and Virtonomics. There are also Flight Simulation and Driving Simulation games.

    Film
    Computer-generated imagery is “the application of the field of 3D computer graphics to special effects”. This
    technology is used for visual effects because they are high in quality, controllable, and can create effects that would
    not be feasible using any other technology either because of cost, resources or safety.[43] Computer-generated
    graphics can be seen in many live action movies today, especially those of the action genre. Further, computer
    generated imagery has almost completely supplanted hand-drawn animation in children's movies which are
    increasingly computer-generated only. Examples of movies that use computer-generated imagery include Finding
    Nemo, 300 and Iron Man.
Simulation                                                                                                                       89


    Theme park rides
    Simulator rides are the progeny of military training simulators and commercial simulators, but they are different in a
    fundamental way. While military training simulators react realistically to the input of the trainee in real time, ride
    simulators only feel like they move realistically and move according to prerecorded motion scripts.[40] One of the
    first simulator rides, Star Tours, which cost $32 million, used a hydraulic motion based cabin. The movement was
    programmed by a joystick. Today’s simulator rides, such as The Amazing Adventures of Spider-Man include
    elements to increase the amount of immersion experienced by the riders such as: 3D imagery, physical effects
    (spraying water or producing scents), and movement through an environment.[44] Examples of simulation rides
    include Mission Space and The Simpsons Ride. There are many simulation rides at themeparks like Disney,
    Universal etc., Examples are Flint Stones, Earth Quake, Time Machine, King Kong.


    Simulation and manufacturing
    Manufacturing represents one of the most important applications of Simulation. This technique represents a valuable
    tool used by engineers when evaluating the effect of capital investment in equipments and physical facilities like
    factory plants, warehouses, and distribution centers. Simulation can be used to predict the performance of an existing
    or planned system and to compare alternative solutions for a particular design problem.[45]
    Another important goal of manufacturing-simulations is to quantify system performance. Common measures of
    system performance include the following:[46]
    •   Throughput under average and peak loads;
    •   System cycle time (how long it take to produce one part);
    •   Utilization of resource, labor, and machines;
    •   Bottlenecks and choke points;
    •   Queuing at work locations;
    •   Queuing and delays caused by material-handling devices and systems;
    •   WIP storages needs;
    •   Staffing requirements;
    •   Effectiveness of scheduling systems;
    •   Effectiveness of control systems.


    More examples in various areas

    Automobile simulator
    An automobile simulator provides an opportunity to reproduce the
    characteristics of real vehicles in a virtual environment. It replicates the
    external factors and conditions with which a vehicle interacts enabling
    a driver to feel as if they are sitting in the cab of their own vehicle.
    Scenarios and events are replicated with sufficient reality to ensure that
    drivers become fully immersed in the experience rather than simply
    viewing it as an educational experience.
    The simulator provides a constructive experience for the novice driver
    and enables more complex exercises to be undertaken by the more                A soldier tests out a heavy-wheeled-vehicle
    mature driver. For novice drivers, truck simulators provide an                               driver simulator.

    opportunity to begin their career by applying best practice. For mature

    drivers, simulation provides the ability to enhance good driving or to detect poor practice and to suggest the
    necessary steps for remedial action. For companies, it provides an opportunity to educate staff in the driving skills
Simulation                                                                                                                        90


    that achieve reduced maintenance costs, improved productivity and, most importantly, to ensure the safety of their
    actions in all possible situations.


    Biomechanics simulators
    An open-source simulation platform for creating dynamic mechanical models built from combinations of rigid and
    deformable bodies, joints, constraints, and various force actuators. It is specialized for creating biomechanical
    models of human anatomical structures, with the intention to study their function and eventually assist in the design
    and planning of medical treatment.
    A biomechanics simulator is used to analyze walking dynamics, study sports performance, simulate surgical
    procedures, analyze joint loads, design medical devices, and animate human and animal movement.
    A neuromechanical simulator that combines biomechanical and biologically realistic neural network simulation. It
    allows the user to test hypotheses on the neural basis of behavior in a physically accurate 3-D virtual environment.


    City and urban simulation
    A city simulator can be a city-building game but can also be a tool used by urban planners to understand how cities
    are likely to evolve in response to various policy decisions. AnyLogic is an example of modern, large-scale urban
    simulators designed for use by urban planners. City simulators are generally agent-based simulations with explicit
    representations for land use and transportation. UrbanSim and LEAM are examples of large-scale urban simulation
    models that are used by metropolitan planning agencies and military bases for land use and transportation planning.


    Classroom of the future
    The "classroom of the future" will probably contain several kinds of simulators, in addition to textual and visual
    learning tools. This will allow students to enter the clinical years better prepared, and with a higher skill level. The
    advanced student or postgraduate will have a more concise and comprehensive method of retraining — or of
    incorporating new clinical procedures into their skill set — and regulatory bodies and medical institutions will find it
    easier to assess the proficiency and competency of individuals.
    The classroom of the future will also form the basis of a clinical skills unit for continuing education of medical
    personnel; and in the same way that the use of periodic flight training assists airline pilots, this technology will assist
    practitioners throughout their career.
    The simulator will be more than a "living" textbook, it will become an integral a part of the practice of medicine. The
    simulator environment will also provide a standard platform for curriculum development in institutions of medical
    education.


    Communication satellite simulation
    Modern satellite communications systems (SatCom) are often large and complex with many interacting parts and
    elements. In addition, the need for broadband connectivity on a moving vehicle has increased dramatically in the past
    few years for both commercial and military applications. To accurately predict and deliver high quality of service,
    satcom system designers have to factor in terrain as well as atmospheric and meteorological conditions in their
    planning. To deal with such complexity, system designers and operators increasingly turn towards computer models
    of their systems to simulate real world operational conditions and gain insights in to usability and requirements prior
    to final product sign-off. Modeling improves the understanding of the system by enabling the SatCom system
    designer or planner to simulate real world performance by injecting the models with multiple hypothetical
    atmospheric and environmental conditions. Simulation is often used in the training of civilian and military
    personnel.[7] This usually occurs when it is prohibitively expensive or simply too dangerous to allow trainees to use
    the real equipment in the real world. In such situations they will spend time learning valuable lessons in a "safe"
Simulation                                                                                                                        91


    virtual environment yet living a lifelike experience (or at least it is the goal). Often the convenience is to permit
    mistakes during training for a safety-critical system. For example, in simSchool teachers practice classroom
    management and teaching techniques on simulated students, which avoids "learning on the job" that can damage real
    students.


    Digital Lifecycle Simulation
    Simulation solutions are being increasingly integrated with CAx
    (CAD, CAM, CAE....) solutions and processes. The use of simulation
    throughout the product lifecycle, especially at the earlier concept and
    design stages, has the potential of providing substantial benefits. These
    benefits range from direct cost issues such as reduced prototyping and
    shorter time-to-market, to better performing products and higher
    margins. However, for some companies, simulation has not provided
    the expected benefits.
    The research firm Aberdeen Group has found that nearly all                          Simulation of airflow over an engine

    best-in-class manufacturers use simulation early in the design process
    as compared to 3 or 4 laggards who do not.
    The successful use of simulation, early in the lifecycle, has been largely driven by increased integration of simulation
    tools with the entire CAD, CAM and PLM solution-set. Simulation solutions can now function across the extended
    enterprise in a multi-CAD environment, and include solutions for managing simulation data and processes and
    ensuring that simulation results are made part of the product lifecycle history. The ability to use simulation across the
    entire lifecycle has been enhanced through improved user interfaces such as tailorable user interfaces and "wizards"
    which allow all appropriate PLM participants to take part in the simulation process.


    Disaster preparedness and simulation training
    Simulation training has become a method for preparing people for disasters. Simulations can replicate emergency
    situations and track how learners respond thanks to a lifelike experience. Disaster preparedness simulations can
    involve training on how to handle terrorism attacks, natural disasters, pandemic outbreaks, or other life-threatening
    emergencies.
    One organization that has used simulation training for disaster preparedness is CADE (Center for Advancement of
    Distance Education). CADE[47] has used a video game to prepare emergency workers for multiple types of attacks.
    As reported by News-Medical.Net, ”The video game is the first in a series of simulations to address bioterrorism,
    pandemic flu, smallpox and other disasters that emergency personnel must prepare for.[48]” Developed by a team
    from the University of Illinois at Chicago (UIC), the game allows learners to practice their emergency skills in a
    safe, controlled environment.
    The Emergency Simulation Program (ESP) at the British Columbia Institute of Technology (BCIT), Vancouver,
    British Columbia, Canada is another example of an organization that uses simulation to train for emergency
    situations. ESP uses simulation to train on the following situations: forest fire fighting, oil or chemical spill response,
    earthquake response, law enforcement, municipal fire fighting, hazardous material handling, military training, and
    response to terrorist attack [49] One feature of the simulation system is the implementation of “Dynamic Run-Time
    Clock,” which allows simulations to run a 'simulated' time frame, 'speeding up' or 'slowing down' time as desired”[49]
    Additionally, the system allows session recordings, picture-icon based navigation, file storage of individual
    simulations, multimedia components, and launch external applications.
    At the University of Québec in Chicoutimi, a research team at the outdoor research and expertise laboratory
    (Laboratoire d'Expertise et de Recherche en Plein Air - LERPA) specializes in using wilderness backcountry
    accident simulations to verify emergency response coordination.
Simulation                                                                                                                     92


    Instructionally, the benefits of emergency training through simulations are that learner performance can be tracked
    through the system. This allows the developer to make adjustments as necessary or alert the educator on topics that
    may require additional attention. Other advantages are that the learner can be guided or trained on how to respond
    appropriately before continuing to the next emergency segment—this is an aspect that may not be available in the
    live-environment. Some emergency training simulators also allows for immediate feedback, while other simulations
    may provide a summary and instruct the learner to engage in the learning topic again.
    In a live-emergency situation, emergency responders do not have time to waste. Simulation-training in this
    environment provides an opportunity for learners to gather as much information as they can and practice their
    knowledge in a safe environment. They can make mistakes without risk of endangering lives and be given the
    opportunity to correct their errors to prepare for the real-life emergency.


    Economics simulation
    In economics and especially macroeconomics, the effects of proposed policy actions, such as fiscal policy changes or
    monetary policy changes, are simulated to judge their desirability. A mathematical model of the economy, having
    been fitted to historical economic data, is used as a proxy for the actual economy; proposed values of government
    spending, taxation, open market operations, etc. are used as inputs to the simulation of the model, and various
    variables of interest such as the inflation rate, the unemployment rate, the balance of trade deficit, the government
    budget deficit, etc. are the outputs of the simulation. The simulated values of these variables of interest are compared
    for different proposed policy inputs to determine which set of outcomes is most desirable.


    Engineering, technology or process simulation
    Simulation is an important feature in engineering systems or any system that involves many processes. For example
    in electrical engineering, delay lines may be used to simulate propagation delay and phase shift caused by an actual
    transmission line. Similarly, dummy loads may be used to simulate impedance without simulating propagation, and
    is used in situations where propagation is unwanted. A simulator may imitate only a few of the operations and
    functions of the unit it simulates. Contrast with: emulate.[50]
    Most engineering simulations entail mathematical modeling and computer assisted investigation. There are many
    cases, however, where mathematical modeling is not reliable. Simulation of fluid dynamics problems often require
    both mathematical and physical simulations. In these cases the physical models require dynamic similitude. Physical
    and chemical simulations have also direct realistic uses, rather than research uses; in chemical engineering, for
    example, process simulations are used to give the process parameters immediately used for operating chemical
    plants, such as oil refineries.


    Equipment simulation
    Due to the dangerous and expensive nature of training on heavy equipment, simulation has become a common
    solution across many industries. Types of simulated equipment include cranes, mining reclaimers and construction
    equipment, among many others. Often the simulation units will include pre-built scenarios by which to teach
    trainees, as well as the ability to customize new scenarios. Such equipment simulators are intended to create a safe
    and cost effective alternative to training on live equipment.[51]


    Ergonomics simulation
    Ergonomic simulation involves the analysis of virtual products or manual tasks within a virtual environment. In the
    engineering process, the aim of ergonomics is to develop and to improve the design of products and work
    environments.[52] Ergonomic simulation utilizes an anthropometric virtual representation of the human, commonly
    referenced as a mannequin or Digital Human Models (DHMs), to mimic the postures, mechanical loads, and
    performance of a human operator in a simulated environment such as an airplane, automobile, or manufacturing
Simulation                                                                                                                     93


    facility. DHMs are recognized as evolving and valuable tool for performing proactive ergonomics analysis and
    design.[53] The simulations employ 3D-graphics and physics-based models to animate the virtual humans.
    Ergonomics software uses inverse kinematics (IK) capability for posing the DHMs.[52] Several ergonomic simulation
    tools have been developed including Jack, SAFEWORK, RAMSIS, and SAMMIE.
    The software tools typically calculate biomechanical properties including individual muscle forces, joint forces and
    moments. Most of these tools employ standard ergonomic evaluation methods such as the NIOSH lifting equation
    and Rapid Upper Limb Assessment (RULA). Some simulations also analyze physiological measures including
    metabolism, energy expenditure, and fatigue limits Cycle time studies, design and process validation, user comfort,
    reachability, and line of sight are other human-factors that may be examined in ergonomic simulation packages.[54]
    Modeling and simulation of a task can be performed by manually manipulating the virtual human in the simulated
    environment. Some ergonomics simulation software permits interactive, real-time simulation and evaluation through
    actual human input via motion capture technologies. However, motion capture for ergonomics requires expensive
    equipment and the creation of props to represent the environment or product.
    Some applications of ergonomic simulation in include analysis of solid waste collection, disaster management tasks,
    interactive gaming,[55] automotive assembly line,[56] virtual prototyping of rehabilitation aids,[57] and aerospace
    product design.[58] Ford engineers use ergonomics simulation software to perform virtual product design reviews.
    Using engineering data, the simulations assist evaluation of assembly ergonomics. The company uses Siemen’s Jack
    and Jill ergonomics simulation software in improving worker safety and efficiency, without the need to build
    expensive prototypes.[59]


    Finance simulation
    In finance, computer simulations are often used for scenario planning. Risk-adjusted net present value, for example,
    is computed from well-defined but not always known (or fixed) inputs. By imitating the performance of the project
    under evaluation, simulation can provide a distribution of NPV over a range of discount rates and other variables.
    Simulations are frequently used in financial training to engage participants in experiencing various historical as well
    as fictional situations. There are stock market simulations, portfolio simulations, risk management simulations or
    models and forex simulations. Such simulations are typically based on stochastic asset models. Using these
    simulations in a training program allows for the application of theory into a something akin to real life. As with other
    industries, the use of simulations can be technology or case-study driven.


    Flight simulation
    Flight Simulation Training Devices (FSTD) are used to train pilots on the ground. In comparison to training in an
    actual aircraft, simulation based training allows for the training of maneuvers or situations that may be impractical
    (or even dangerous) to perform in the aircraft, while keeping the pilot and instructor in a relatively low-risk
    environment on the ground. For example, electrical system failures, instrument failures, hydraulic system failures,
    and even flight control failures can be simulated without risk to the pilots or an aircraft.
    Instructors can also provide students with a higher concentration of training tasks in a given period of time than is
    usually possible in the aircraft. For example, conducting multiple instrument approaches in the actual aircraft may
    require significant time spent repositioning the aircraft, while in a simulation, as soon as one approach has been
    completed, the instructor can immediately preposition the simulated aircraft to an ideal (or less than ideal) location
    from which to begin the next approach.
    Flight simulation also provides an economic advantage over training in an actual aircraft. Once fuel, maintenance,
    and insurance costs are taken into account, the operating costs of an FSTD are usually substantially lower than the
    operating costs of the simulated aircraft. For some large transport category airplanes, the operating costs may be
    several times lower for the FSTD than the actual aircraft.
Simulation                                                                                                                      94


    Some people who use simulator software, especially flight simulator software, build their own simulator at home.
    Some people — to further the realism of their homemade simulator — buy used cards and racks that run the same
    software used by the original machine. While this involves solving the problem of matching hardware and software
    — and the problem that hundreds of cards plug into many different racks — many still find that solving these
    problems is well worthwhile. Some are so serious about realistic simulation that they will buy real aircraft parts, like
    complete nose sections of written-off aircraft, at aircraft boneyards. This permits people to simulate a hobby that
    they are unable to pursue in real life.


    Marine simulators
    Bearing resemblance to flight simulators, marine simulators train ships' personnel. The most common marine
    simulators include:
    •   Ship's bridge simulators
    •   Engine room simulators
    •   Cargo handling simulators
    •   Communication / GMDSS simulators
    •   ROV simulators
    Simulators like these are mostly used within maritime colleges, training institutions and navies. They often consist of
    a replication of a ships' bridge, with operating console(s), and a number of screens on which the virtual surroundings
    are projected.


    Military simulations
    Military simulations, also known informally as war games, are models in which theories of warfare can be tested and
    refined without the need for actual hostilities. They exist in many different forms, with varying degrees of realism. In
    recent times, their scope has widened to include not only military but also political and social factors (for example,
    the NationLab series of strategic exercises in Latin America).[60] While many governments make use of simulation,
    both individually and collaboratively, little is known about the model's specifics outside professional circles.


    Payment and securities settlement system simulations
    Simulation techniques have also been applied to payment and securities settlement systems. Among the main users
    are central banks who are generally responsible for the oversight of market infrastructure and entitled to contribute to
    the smooth functioning of the payment systems.
    Central banks have been using payment system simulations to evaluate things such as the adequacy or sufficiency of
    liquidity available ( in the form of account balances and intraday credit limits) to participants (mainly banks) to
    allow efficient settlement of payments.[61][62] The need for liquidity is also dependent on the availability and the type
    of netting procedures in the systems, thus some of the studies have a focus on system comparisons.[63]
    Another application is to evaluate risks related to events such as communication network breakdowns or the inability
    of participants to send payments (e.g. in case of possible bank failure).[64] This kind of analysis falls under the
    concepts of Stress testing or scenario analysis.
    A common way to conduct these simulations is to replicate the settlement logics of the real payment or securities
    settlement systems under analysis and then use real observed payment data. In case of system comparison or system
    development, naturally also the other settlement logics need to be implemented.
    To perform stress testing and scenario analysis, the observed data needs to be altered, e.g. some payments delayed or
    removed. To analyze the levels of liquidity, initial liquidity levels are varried. System comparisons
    (benchmarking)or evaluations of new netting algorithms or rules are performed by running simulations with a fixed
    set of data and varying only the system setups.
Simulation                                                                                                                   95


    Inference is usually done by comparing the benchmark simulation results to the results of altered simulation setups
    by comparing indicators such as unsettled transactions or settlement delays.


    Robotics simulators
    A robotics simulator is used to create embedded applications for a specific (or not) robot without being dependent on
    the 'real' robot. In some cases, these applications can be transferred to the real robot (or rebuilt) without
    modifications. Robotics simulators allow reproducing situations that cannot be 'created' in the real world because of
    cost, time, or the 'uniqueness' of a resource. A simulator also allows fast robot prototyping. Many robot simulators
    feature physics engines to simulate a robot's dynamics.


    Production simulation
    Simulations of production systems is used mainly to examine the effect of improvements or investments in a
    production system. Most often this is done using a static spreadsheet with process times and transportation times. For
    more sophisticated simulations Discrete Event Simulation (DES) is used with the advantages to simulate dynamics in
    the production system. A production system is very much dynamic depending on variations in manufacturing
    processes, assembly times, machine set-ups, breaks, breakdowns and small stoppages.[65] There are lots of programs
    commonly used for discrete event simulation. They differ in usability and markets but do often share the same
    foundation. There is an academic project investigating the possibilities to use production simulation software for
    ecology labeling, named EcoProIT.


    Sales process simulators
    Simulations are useful in modeling the flow of transactions through business processes, such as in the field of sales
    process engineering, to study and improve the flow of customer orders through various stages of completion (say,
    from an initial proposal for providing goods/services through order acceptance and installation). Such simulations
    can help predict the impact of how improvements in methods might impact variability, cost, labor time, and the
    quantity of transactions at various stages in the process. A full-featured computerized process simulator can be used
    to depict such models, as can simpler educational demonstrations using spreadsheet software, pennies being
    transferred between cups based on the roll of a die, or dipping into a tub of colored beads with a scoop.[66]


    Sports Simulation
    In sports, computer simulations are often done to predict the outcome of games and the performance of individual
    players. They attempt to recreate the game through models built from statistics. The increase in technology has
    allowed anyone with knowledge of programming the ability to run simulations of their models. The simulations are
    built from a series of mathematical algorithms, or models, and can vary with accuracy. Accuscore, which is licensed
    by companies such as ESPN, is a well known simulation program for all major sports. It offers detailed analysis of
    games through simulated betting lines, projected point totals and overall probabilities.
    With the increased interest in fantasy sports simulation models that predict individual player performance have
    gained popularity. Companies like What If Sports and StatFox specialize in not only using their simulations for
    predicting game results, but how well individual players will do as well. Many people use models to determine who
    to start in their fantasy leagues.
    Another way simulations are helping the sports field is in the use of biomechanics. Models are derived and
    simulations are run from data received from sensors attached to athletes and video equipment. Sports biomechanics
    aided by simulation models answer questions regarding training techniques such as: the effect of fatigue on throwing
    performance (height of throw) and biomechanical factors of the upper limbs (reactive strength index; hand contact
    time).[67]
Simulation                                                                                                               96


    Computer simulations allow us to take models which before were too complex to run, and give us answers.
    Simulations have proven to be some of our best insights into both play performance and team predictability.


    Space shuttle countdown simulation
    Simulation is used at Kennedy Space Center
    (KSC) to train and certify Space Shuttle
    engineers     during     simulated     launch
    countdown operations. The Space Shuttle
    engineering community participates in a
    launch countdown integrated simulation
    before each shuttle flight. This simulation is
    a virtual simulation where real people
    interact with simulated Space Shuttle
    vehicle and Ground Support Equipment
    (GSE) hardware. The Shuttle Final
    Countdown Phase Simulation, also known
    as S0044, involves countdown processes                         Firing Room 1 configured for space shuttle launches
    that integrate many of the Space Shuttle
    vehicle and GSE systems. Some of the Shuttle systems integrated in the simulation are the main propulsion system,
    main engines, solid rocket boosters, ground liquid hydrogen and liquid oxygen, external tank, flight controls,
    navigation, and avionics.[68] The high-level objectives of the Shuttle Final Countdown Phase Simulation are:

    • To demonstrate Firing Room final countdown phase operations.
    • To provide training for system engineers in recognizing, reporting and evaluating system problems in a time
      critical environment.
    • To exercise the launch teams ability to evaluate, prioritize and respond to problems in an integrated manner
      within a time critical environment.
    • To provide procedures to be used in performing failure/recovery testing of the operations performed in the final
      countdown phase.[69]
    The Shuttle Final Countdown Phase Simulation takes place at the Kennedy Space Center Launch Control Center
    Firing Rooms. The firing room used during the simulation is the same control room where real launch countdown
    operations are executed. As a result, equipment used for real launch countdown operations is engaged. Command
    and control computers, application software, engineering plotting and trending tools, launch countdown procedure
    documents, launch commit criteria documents, hardware requirement documents, and any other items used by the
    engineering launch countdown teams during real launch countdown operations are used during the simulation. The
    Space Shuttle vehicle hardware and related GSE hardware is simulated by mathematical models (written in Shuttle
    Ground Operations Simulator (SGOS) modeling language [70]) that behave and react like real hardware. During the
    Shuttle Final Countdown Phase Simulation, engineers command and control hardware via real application software
    executing in the control consoles – just as if they were commanding real vehicle hardware. However, these real
    software applications do not interface with real Shuttle hardware during simulations. Instead, the applications
    interface with mathematical model representations of the vehicle and GSE hardware. Consequently, the simulations
    bypass sensitive and even dangerous mechanisms while providing engineering measurements detailing how the
    hardware would have reacted. Since these math models interact with the command and control application software,
    models and simulations are also used to debug and verify the functionality of application software.[71]
Simulation                                                                                                                                               97


    Satellite navigation simulators
    The only true way to test GNSS receivers (commonly known as Sat-Nav's in the commercial world)is by using an
    RF Constellation Simulator. A receiver that may for example be used on an aircraft, can be tested under dynamic
    conditions without the need to take it on a real flight. The test conditions can be repeated exactly, and there is full
    control over all the test parameters. this is not possible in the 'real-world' using the actual signals. For testing
    receivers that will use the new Galileo (satellite navigation) there is no alternative, as the real signals do not yet exist.


    Weather simulation
    Predicting weather conditions by extrapolating/interpolating previous data is one of the real use of simulation. Most
    of the weather forecats use this information published by Weather buereaus. This kind of simulations help in
    predicting and forewarning about extreme weather conditions like the path of an active hurricane/cyclone. Numerical
    weather prediction for forecasting involves complicated numeric computer models to predict weather accurately by
    taking many parameters in to account.


    Simulation and games
    Strategy games — both traditional and modern — may be viewed as simulations of abstracted decision-making for
    the purpose of training military and political leaders (see History of Go for an example of such a tradition, or
    Kriegsspiel for a more recent example).
    Many other video games are simulators of some kind. Such games can simulate various aspects of reality, from
    business, to government, to construction, to piloting vehicles (see above).


    Historical usage
    Historically, the word had negative connotations:
             …for Distinction Sake, a Deceiving by Words, is commonly called a Lye, and a Deceiving by Actions,
             Gestures, or Behavior, is called Simulation…
             —Robert South, South, 1697, p.525
    However, the connection between simulation and dissembling later faded out and is now only of linguistic
    interest.[72]


    References
    [1] J. Banks, J. Carson, B. Nelson, D. Nicol (2001). Discrete-Event System Simulation. Prentice Hall. p. 3. ISBN 0-13-088702-1.
    [2] In the words of the Simulation article (http:/ / www. modelbenders. com/ encyclopedia/ encyclopedia. html) in Encyclopedia of Computer
        Science, "designing a model of a real or imagined system and conducting experiments with that model".
    [3] Sokolowski, J.A., Banks, C.M. (2009). Principles of Modeling and Simulation. Hoboken, NJ: Wiley. p. 6. ISBN 978-0-470-28943-3.
    [4] For example in computer graphics SIGGRAPH 2007 | For Attendees | Papers (http:/ / www. siggraph. org/ s2007/ attendees/ papers/ 12. html)
        Doc:Tutorials/Physics/BSoD - BlenderWiki (http:/ / wiki. blender. org/ index. php/ BSoD/ Physical_Simulation).
    [5] Thales defines synthetic environment as "the counterpart to simulated models of sensors, platforms and other active objects" for "the
        simulation of the external factors that affect them" (http:/ / www. thalesresearch. com/ Default. aspx?tabid=181) while other vendors use the
        term for more visual, virtual reality-style simulators (http:/ / www. cae. com/ www2004/ Products_and_Services/
        Civil_Simulation_and_Training/ Simulation_Equipment/ Visual_Solutions/ Synthetic_Environments/ index. shtml).
    [6] For a popular research project in the field of biochemistry where "computer simulation is particularly well suited to address these questions"
        Folding@home - Main (http:/ / folding. stanford. edu/ Pande/ Main), see Folding@Home.
    [7] For an academic take on a training simulator, see e.g. Towards Building an Interactive, Scenario-based Training Simulator (http:/ / gel. msu.
        edu/ magerko/ papers/ 11TH-CGF-058. pdf), for medical application Medical Simulation Training Benefits (http:/ / www. immersion. com/
        medical/ benefits1. php) as presented by a simulator vendor and for military practice A civilian's guide to US defense and security assistance
        to Latin America and the Caribbean (http:/ / ciponline. org/ facts/ exe. htm) published by Center for International Policy.
    [8] http:/ / www. simschool. org
    [9] Classification used by the Defense Modeling and Simulation Office.
Simulation                                                                                                                                              98

    [10] "High Versus Low Fidelity Simulations: Does the Type of Format Affect Candidates' Performance or Perceptions?" (http:/ / www. ipmaac.
        org/ conf/ 03/ havighurst. pdf)
    [11] For example All India management association (http:/ / www. aima-ind. org/ ) maintains that playing to win, participants "imbibe new forms
        of competitive behavior that are ideal for today's highly chaotic business conditions" (http:/ / www. aima-ind. org/ management_games. asp)
        and IBM claims that "the skills honed playing massive multiplayer dragon-slaying games like World of Warcraft can be useful when
        managing modern multinationals".
    [12] "Reacting to the Past Home Page" (http:/ / www. barnard. columbia. edu/ reacting/ )
    [13] https:/ / sites. google. com/ site/ reactingscience/ Reacting to the Past:STEM Games
    [14] "Carana," at 'PaxSims' blog, 27 January 2009 (http:/ / paxsims. wordpress. com/ 2009/ 01/ 27/ carana/ )
    [15] Sherman, W.R., Craig, A.B. (2003). Understanding Virtual Reality. San Francisco, CA: Morgan Kaufmann. ISBN 978-1-55860-353-0.
    [16] Leeb, R., Lee, F., Keinrath, C., Schere, R., Bischof, H., Pfurtscheller, G. (2007). "Brain-Computer Communication: Motivation, Aim, and
        Impact of Exploring a Virtual Apartment". IEEE Transactions on Neural Systems and Rehabilitation Engineering 15 (4): 473–481.
        doi:10.1109/TNSRE.2007.906956.
    [17] Zahraee, A.H., Szewczyk, J., Paik, J.K., Guillaume, M. (2010). Robotic hand-held surgical device: evaluation of end-effector’s kinematics
        and development of proof-of-concept prototypes. Proceedings of the 13th International Conference on Medical Image Computing and
        Computer Assisted Intervention, Beijing, China.
    [18] Ahmed K, Keeling AN, Fakhry M, Ashrafian H, Aggarwal R, Naughton PA, Darzi A, Cheshire N, et al. (January 2010). "Role of Virtual
        Reality Simulation in Teaching and Assessing Technical Skills in Endovascular Intervention". J Vasc Interv Radiol 21.
    [19] Narayan, Roger; Kumta, Prashant; Sfeir, Charles; Lee, Dong-Hyun; Choi, Daiwon; Olton, Dana (October 2004). "Nanostructured ceramics
        in medical devices: Applications and prospects" (http:/ / www. ingentaconnect. com/ content/ tms/ jom/ 2004/ 00000056/ 00000010/
        art00011). JOM 56 (10): 38–43. Bibcode 2004JOM....56j..38N. doi:10.1007/s11837-004-0289-x. PMID 11196953. .
    [20] Couvreur P, Vauthier C (July 2006). "Nanotechnology: intelligent design to treat complex disease". Pharm. Res. 23 (7): 1417–50.
        doi:10.1007/s11095-006-0284-8. PMID 16779701.
    [21] Hede S, Huilgol N (2006). ""Nano": the new nemesis of cancer" (http:/ / www. cancerjournal. net/ article.
        asp?issn=0973-1482;year=2006;volume=2;issue=4;spage=186;epage=195;aulast=Hede). J Cancer Res Ther 2 (4): 186–95.
        doi:10.4103/0973-1482.29829. PMID 17998702. .
    [22] Leary SP, Liu CY, Apuzzo ML (June 2006). "Toward the emergence of nanoneurosurgery: part III—nanomedicine: targeted nanotherapy,
        nanosurgery, and progress toward the realization of nanoneurosurgery" (http:/ / meta. wkhealth. com/ pt/ pt-core/ template-journal/
        lwwgateway/ media/ landingpage. htm?issn=0148-396X& volume=58& issue=6& spage=1009). Neurosurgery 58 (6): 1009–26; discussion
        1009–26. doi:10.1227/01.NEU.0000217016.79256.16. PMID 16723880. .
    [23] http:/ / vam. anest. ufl. edu/ wip. html
    [24] http:/ / onlinelibrary. wiley. com/ doi/ 10. 1111/ j. 1445-2197. 2010. 05349. x/ abstract
    [25] Nishisaki A, Keren R, Nadkarni V (June 2007). "Does simulation improve patient safety? Self-efficacy, competence, operational
        performance, and patient safety" (http:/ / linkinghub. elsevier. com/ retrieve/ pii/ S1932-2275(07)00025-0). Anesthesiol Clin 25 (2): 225–36.
        doi:10.1016/j.anclin.2007.03.009. PMID 17574187. .
    [26] Meller, G. (1997). "A Typology of Simulators for Medical Education" (http:/ / www. medsim. com/ profile/ article1. html). Journal of
        Digital Imaging. .
    [27] Cooper Jeffery B, Taqueti VR (2008-12). "A brief history of the development of mannequin simulators for clinical education and training"
        (http:/ / pmj. bmj. com/ content/ 84/ 997/ 563. long). Postgrad Med J. 84 (997): 563–570. doi:10.1136/qshc.2004.009886. PMID 19103813. .
        Retrieved 2011-05-24.
    [28] Murphy D, Challacombe B, Nedas T, Elhage O, Althoefer K, Seneviratne L, Dasgupta P. (May 2007). "[Equipment and technology in
        robotics]" (in Spanish; Castilian). Arch. Esp. Urol. 60 (4): 349–55. PMID 17626526.
    [29] "In Cybertherapy, Avatars Assist With Healing" (http:/ / www. nytimes. com/ 2010/ 11/ 23/ science/ 23avatar. html?_r=1& ref=science).
        New York Times. 2010-11-22. . Retrieved 2010-11-23.
    [30] Dagger, Jacob (May–June 2008). Update: "The New Game Theory" (http:/ / www. dukemagazine. duke. edu/ dukemag/ issues/ 050608/
        depupd. html). 94. Duke Magazine. . Retrieved 2011-02-08.
    [31] Steinberg, Scott (2011-01-31). "How video games can make you smarter" (http:/ / articles. cnn. com/ 2011-01-31/ tech/ video. games.
        smarter. steinberg_1_video-games-interactive-simulations-digital-world?_s=PM:TECH). Cable News Network (CNN Tech). . Retrieved
        2011-02-08.
    [32] Vlaovic PD, Sargent ER, Boker JR, et al. (2008). "Immediate impact of an intensive one-week laparoscopy training program on
        laparoscopic skills among postgraduate urologists" (http:/ / openurl. ingenta. com/ content/ nlm?genre=article& issn=1086-8089&
        volume=12& issue=1& spage=1& aulast=Vlaovic). JSLS 12 (1): 1–8. PMC 3016039. PMID 18402731. .
    [33] Leung J, Foster E (April 2008). "How do we ensure that trainees learn to perform biliary sphincterotomy safely, appropriately, and
        effectively?" (http:/ / www. current-reports. com/ article_frame. cfm?PubID=GR10-2-2-03& Type=Abstract). Curr Gastroenterol Rep 10 (2):
        163–8. doi:10.1007/s11894-008-0038-3. PMID 18462603. .
    [34] http:/ / www. pong-story. com/ intro. htm Archived (http:/ / www. webcitation. org/ 5seElSgBC) 10 September 2010 at WebCite
    [35] History of Computers 1980 (http:/ / homepages. vvm. com/ ~jhunt/ compupedia/ History of Computers/ history_of_computers_1980. htm)
    [36] "Video Game Console Timeline - Video Game History - Xbox 360 - TIME Magazine" (http:/ / www. time. com/ time/ covers/ 1101050523/
        console_timeline/ ). Time. 2005-05-23. . Retrieved 2010-05-23.
Simulation                                                                                                                                               99

    [37] TRON - The 1982 Movie (http:/ / design. osu. edu/ carlson/ history/ tron. html)
    [38] http:/ / www. beanblossom. in. us/ larryy/ cgi. html
    [39] Link Trainer Restoration (http:/ / www. starksravings. com/ linktrainer/ linktrainer. htm)
    [40] http:/ / www. trudang. com/ simulatr/ simulatr. html
    [41] Simulation - General Information | Open-Site.org (http:/ / open-site. org/ Games/ Video_Games/ Simulation)
    [42] Video Games in the US Market Research | IBISWorld (http:/ / www. ibisworld. com/ industry/ retail. aspx?indid=2003& chid=1)
    [43] Computer-generated imagery (http:/ / www. sciencedaily. com/ articles/ c/ computer-generated_imagery. htm)
    [44] Bringing Spidey to Life: Kleiser-Walczak Construction Company (http:/ / www. awn. com/ mag/ issue4. 02/ 4. 02pages/ kenyonspiderman.
        php3)
    [45] Benedettini, O., Tjahjono, B. (2008). "Towards an improved tool to facilitate simulation modeling of complex manufacturing systems".
        International Journal of Advanced Manufacturing Technology 43 (1/2): 191–9. doi:10.1007/s00170-008-1686-z.
    [46] Banks, J., Carson J., Nelson B.L., Nicol, D. (2005). Discrete-event system simulation (4th ed.). Upper Saddle River, NJ: Pearson Prentice
        Hall. ISBN 978-0-13-088702-3.
    [47] CADE- http:/ / www. uic. edu/ sph/ cade/
    [48] News-Medical.Net article- http:/ / www. news-medical. net/ news/ 2005/ 10/ 27/ 14106. aspx
    [49] Emergency Response Training (http:/ / www. straylightmm. com/ )
    [50] Federal Standard 1037C
    [51] GlobalSim | World Class Training Simulators | Crane Simulator (http:/ / www. globalsim. com)
    [52] Reed, M. P., Faraway, J., Chaffin, D. B., & Martin, B. J. (2006). The HUMOSIM Ergonomics Framework: A new approach to digital human
        simulation for ergonomic analysis. SAE Technical Paper, 01-2365
    [53] Chaffin, D. B. (2007). Human motion simulation for vehicle and workplace design. Human Factors and Ergonomics in Manufacturing &
        Service Industries,17(5), 475-484
    [54] Jack and Process Simulate Human: Siemens PLM Software (http:/ / www. plm. automation. siemens. com/ en_us/ products/ tecnomatix/
        assembly_planning/ jack/ index. shtml)
    [55] Bush, P. M., Gaines, S., Gammoh, F., & Wooden, S. A Comparison of Software Tools for Occupational Biomechanics and Ergonomic
        Research.
    [56] Niu, J. W., Zhang, X. W., Zhang, X., & Ran, L. H. (2010, December). Investigation of ergonomics in automotive assembly line using Jack.
        InIndustrial Engineering and Engineering Management (IEEM), 2010 IEEE International Conference on (pp. 1381-1385). IEEE.
    [57] Beitler, Matthew T., Harwin, William S., & Mahoney, Richard M. (1996) In Proceedings of the virtual prototyping of rehabilitation aids,
        RESNA 96, pp. 360–363.
    [58] G.R. Bennett. The application of virtual prototyping in development of complex aerospace products. Virtual Prototyping Journal, 1 (1)
        (1996), pp. 13–20
    [59] From the floor of the 2012 Chicago Auto Show: Automation World shows how Ford uses the power of simulation « Siemens PLM Software
        Blog (http:/ / blog. industrysoftware. automation. siemens. com/ blog/ 2012/ 03/ 21/
        floor-2012-chicago-auto-show-automation-world-shows-ford-power-simulation/ )
    [60] The Economist provides a current (as of 2012) survey of public projects attempting to simulate some theories in "The science of civil war:
        What makes heroic strife" (http:/ / www. economist. com/ node/ 21553006).
    [61] Leinonen (ed.): Simulation studies of liquidity needs, risks and efficiency in payment networks (Bank of Finland Studies E:39/2007)
        Simulation publications (http:/ / pss. bof. fi/ Pages/ Publications. aspx)
    [62] Neville Arjani: Examining the Trade-Off between Settlement Delay and Intraday Liquidity in Canada's LVTS: A Simulation Approach
        (Working Paper 2006-20, Bank of Canada) Simulation publications (http:/ / pss. bof. fi/ Pages/ Publications. aspx)
    [63] Johnson, K. - McAndrews, J. - Soramäki, K. 'Economizing on Liquidity with Deferred Settlement Mechanisms' (Reserve Bank of New York
        Economic Policy Review, December 2004)
    [64] H. Leinonen (ed.): Simulation analyses and stress testing of payment networks (Bank of Finland Studies E:42/2009) Simulation publications
        (http:/ / pss. bof. fi/ Pages/ Publications. aspx)
    [65] Ulf, Eriksson (2005). Diffusion of Discrete Event Simulation in Swedish Industry. Gothenburg: Doktorsavhandlingar vid Chalmers tekniska
        högskola. ISBN 91-7291-577-3.
    [66] Paul H. Selden (1997). Sales Process Engineering: A Personal Workshop. Milwaukee, WI: ASQ Quality Press. ISBN 978-0-87389-418-0.
    [67] Harrison, Andrew J (2011). "Throwing and catching movements exhibit post-activation potentiation effects following fatigue". Sports
        Biomechanics 10 (3): 185–196. doi:10.1080/TNSRE10.1080/14763141.2011.592544.
    [68] Sikora, E.A. (2010, July 27). Space Shuttle Main Propulsion System expert, John F. Kennedy Space Center. Interview.
    [69] Shuttle Final Countdown Phase Simulation. National Aeronautics and Space Administration KSC Document # RTOMI S0044, Revision
        AF05, 2009.
    [70] Shuttle Ground Operations Simulator (SGOS) Summary Description Manual. National Aeronautics and Space Administration KSC
        Document # KSC-LPS-SGOS-1000, Revision 3 CHG-A, 1995.
    [71] Math Model Main Propulsion System (MPS) Requirements Document, National Aeronautics and Space Administration KSC Document #
        KSCL-1100-0522, Revision 9, June 2009.
    [72] South, in the passage quoted, was speaking of the differences between a falsehood and an honestly mistaken statement; the difference being
        that in order for the statement to be a lie the truth must be known, and the opposite of the truth must have been knowingly uttered. And, from
Simulation                                                                                                                                                100

       this, to the extent to which a lie involves deceptive words, a simulation involves deceptive actions, deceptive gestures, or deceptive behavior.
       Thus, it would seem, if a simulation is false, then the truth must be known (in order for something other than the truth to be presented in its
       stead); and, for the simulation to simulate. Because, otherwise, one would not know what to offer up in simulation. Bacon’s essay Of
       Simulation and Dissimulation (http:/ / www. authorama. com/ essays-of-francis-bacon-7. html) expresses somewhat similar views; it is also
       significant that Samuel Johnson thought so highly of South's definition, that he used it in the entry for simulation in his Dictionary of the
       English Language.



    Further reading
    • C. Aldrich (2003). Learning by Doing : A Comprehensive Guide to Simulations, Computer Games, and Pedagogy
      in e-Learning and Other Educational Experiences. San Francisco: Pfeifer — John Wiley & Sons.
      ISBN 978-0-7879-7735-1.
    • C. Aldrich (2004). Simulations and the future of learning: an innovative (and perhaps revolutionary) approach to
      e-learning. San Francisco: Pfeifer — John Wiley & Sons. ISBN 978-0-7879-6962-2.
    • Steve Cohen (2006). Virtual Decisions. Mahwah, NJ: Lawrence Erlbaum Associates. ISBN 978-0-8058-4994-3.
    • R. Frigg, S. Hartmann (2007). "Models in Science" (http://plato.stanford.edu/entries/models-science/).
      Stanford Encyclopedia of Philosophy.
    • S. Hartmann (1996). "The World as a Process: Simulations in the Natural and Social Sciences" (http://
      philsci-archive.pitt.edu/archive/00002412/). In R. Hegselmann, et al.. Modelling and Simulation in the Social
      Sciences from the Philosophy of Science Point of View. Theory and Decision Library. Dordrecht: Kluwer.
      pp. 77–100.
    • J.P. Hertel (2002). Using Simulations to Promote Learning in Higher Education. Sterling, Virginia: Stylus.
      ISBN 978-1-57922-052-5.
    • P. Humphreys (2004). Extending Ourselves: Computational Science, Empiricism, and Scientific Method. Oxford:
      Oxford University Press. ISBN 978-0-19-515870-0.
    • F. Percival, S. Lodge, D. Saunders (1993). The Simulation and Gaming Yearbook: Developing Transferable Skills
      in Education and Training. London: Kogan Page.
    • James J. Nutaro, Building Software for Simulation: Theory and Algorithms, with Applications in C++. Wiley,
      2010.
    • D. Saunders, ed. (2000). The International Simulation and Gaming Research Yearbook. London: Kogan Page.
    • Roger D. Smith: Simulation Article (http://www.modelbenders.com/encyclopedia/encyclopedia.html),
      Encyclopedia of Computer Science, Nature Publishing Group, ISBN 978-0-333-77879-1.
    • Roger D. Smith: "Simulation: The Engine Behind the Virtual World" (http://www.modelbenders.com/
      Bookshop/techpapers.html), eMatter, December, 1999.
    • R. South (1688). "A Sermon Delivered at Christ-Church, Oxon., Before the University, Octob. 14. 1688: Prov.
      XII.22 Lying Lips are abomination to the Lord", pp. 519–657 in South, R., Twelve Sermons Preached Upon
      Several Occasions (Second Edition), Volume I, Printed by S.D. for Thomas Bennet, (London), 1697.
    • Gabriel A. Wainer (2009) Discrete-Event Modeling and Simulation: a practitioner's approach (http://cell-devs.
      sce.carleton.ca/mediawiki/index.php/Main_Page). CRC Press, 2009.
    • Eric Winsberg (1999) Sanctioning Models: The epistemology of simulation (http://www.cas.usf.edu/~ewinsb/
      SiC_Eric_Winsberg.pdf), in Sismondo, Sergio and Snait Gissis (eds.) (1999), Modeling and Simulation. Special
      Issue of Science in Context 12.
    • Eric Winsberg (2001). "Simulations, Models and Theories: Complex Physical Systems and their
      Representations". Philosophy of Science 68: 442–454.
    • Eric Winsberg (2003). "Simulated Experiments: Methodology for a Virtual World" (http://www.cas.usf.edu/
      ~ewinsb/methodology.pdf) (PDF). Philosophy of Science 70: 105–125. doi:10.1086/367872.
    • Joseph Wolfe, David Crookall (1998). "Developing a scientific knowledge of simulation/gaming" (http://sag.
      sagepub.com/cgi/reprint/29/1/7). Simulation & Gaming: an International Journal of Theory, Design and
      Research 29 (1): 7–19.
Simulation                                                                                                                        101


    • Ellen K. Levy (2004). "Synthetic Lighting: Complex Simulations of Nature". Photography Quarterly (88): 5–9.
    • Paul Humphreys and Cyrille Imbert, ed. (2012). Models, Simulations and Representations. London and New
      York: Routledge. ISBN 978-0-203-80841-2 (ebk).
    • World Simulations


    External links
    • Bibliographies containing more references (http://www.unice.fr/sg/resources/bibliographies.htm) to be
      found on the website of the journal Simulation & Gaming (http://www.unice.fr/sg/).



    Probability distribution
    In probability and statistics, a probability distribution assigns a probability to each of the possible outcomes of a
    random experiment. Examples are found in experiments whose sample space is non-numerical, where the
    distribution would be a categorical distribution; experiments whose sample space is encoded by discrete random
    variables, where the distribution is a probability mass function; and experiments with sample spaces encoded by
    continuous random variables, where the distribution is a probability density function. More complex experiments,
    such as those involving stochastic processes defined in continuous-time, may demand the use of more general
    probability measures.
    In applied probability, a probability distribution can be specified in a number of different ways, often chosen for
    mathematical convenience:
    •   by supplying a valid probability mass function or probability density function
    •   by supplying a valid cumulative distribution function or survival function
    •   by supplying a valid hazard function
    •   by supplying a valid characteristic function
    •   by supplying a rule for constructing a new random variable from other random variables whose joint probability
        distribution is known.
    Important and commonly encountered probability distributions include the binomial distribution, the hypergeometric
    distribution, and the normal distribution.


    Introduction
    To define probability distributions for the simplest
    cases, one needs to distinguish between discrete and
    continuous random variables. In the discrete case, one
    can easily assign a probability to each possible value:
    for example, when throwing a die, each of the six
    values 1 to 6 has the probability 1/6. In contrast, when
    a random variable takes values from a continuum,
    probabilities are nonzero only if they refer to finite
    intervals: in quality control one might demand that the
    probability of a "500 g" package containing between
    490 g and 510 g should be no less than 98%.

                                                                     Discrete probability distribution for the sum of two dice.
Probability distribution                                                                                                                     102


     If the random variable is real-valued (or more
     generally, if a total order is defined for its possible
     values), the cumulative distribution function gives
     the probability that the random variable is no larger
     than a given value; in the real-valued case it is the
     integral of the density.


     Terminology
                                                                     Normal distribution, also called Gaussian or "bell curve", the most
     As probability theory is used in quite diverse                       important continuous random distribution.

     applications, terminology is not uniform and sometimes
     confusing. The following terms are used for non-cumulative probability distribution functions:
     • Probability mass, Probability mass function, p.m.f.: for discrete random variables.
     • Categorical distribution: for discrete random variables with a finite set of values.
     • Probability density, Probability density function, p.d.f: Most often reserved for continuous random variables.
     The following terms are somewhat ambiguous as they can refer to non-cumulative or cumulative distributions,
     depending on authors' preferences:
     • Probability distribution function: Continuous or discrete, non-cumulative or cumulative.
     • Probability function: Even more ambiguous, can mean any of the above, or anything else.
     Finally,
     • Probability distribution: Either the same as probability distribution function. Or understood as something more
       fundamental underlying an actual mass or density function.


     Basic terms
     • Mode: most frequently occurring value in a distribution
     • Tail: region of least frequently occurring values in a distribution
     • Support: the smallest closed set whose complement has probability zero. It may be understood as the points or
       elements that are actual members of the distribution.


     Discrete probability distribution
     A discrete probability distribution shall be understood as a
     probability distribution characterized by a probability mass function.
     Thus, the distribution of a random variable X is discrete, and X is then
     called a discrete random variable, if


                                                                                          The probability mass function of a discrete
     as u runs through the set of all possible values of X. It follows that such        probability distribution. The probabilities of the
     a random variable can assume only a finite or countably infinite                  singletons {1}, {3}, and {7} are respectively 0.2,
     number of values.                                                                 0.5, 0.3. A set not containing any of these points
                                                                                                      has probability zero.
     In cases more frequently considered, this set of possible values is a
     topologically discrete set in the sense that all its points are isolated
     points. But there are discrete random variables for which this countable set is dense on the real line (for example, a
     distribution over rational numbers).
Probability distribution                                                                                                                    103


     Among the most well-known discrete probability distributions that are
     used for statistical modeling are the Poisson distribution, the Bernoulli
     distribution, the binomial distribution, the geometric distribution, and
     the negative binomial distribution. In addition, the discrete uniform
     distribution is commonly used in computer programs that make
                                                                                      The cdf of a discrete probability distribution, ...
     equal-probability random selections between a number of choices.


     Cumulative density
     Equivalently to the above, a discrete random variable can be defined as
     a random variable whose cumulative distribution function (cdf)
     increases only by jump discontinuities—that is, its cdf increases only
                                                                                       ... of a continuous probability distribution, ...
     where it "jumps" to a higher value, and is constant between those
     jumps. The points where jumps occur are precisely the values which
     the random variable may take. The number of such jumps may be finite
     or countably infinite. The set of locations of such jumps need not be
     topologically discrete; for example, the cdf might jump at each rational
     number.

                                                                                      ... of a distribution which has both a continuous
                                                                                                     part and a discrete part.




     Delta-function representation
     Consequently, a discrete probability distribution is often represented as a generalized probability density function
     involving Dirac delta functions, which substantially unifies the treatment of continuous and discrete distributions.
     This is especially useful when dealing with probability distributions involving both a continuous and a discrete part.


     Indicator-function representation
     For a discrete random variable X, let u0, u1, ... be the values it can take with non-zero probability. Denote


     These are disjoint sets, and by formula (1)




     It follows that the probability that X takes any value except for u0, u1, ... is zero, and thus one can write X as



     except on a set of probability zero, where          is the indicator function of A. This may serve as an alternative
     definition of discrete random variables.


     Continuous probability distribution
     A continuous probability distribution is a probability distribution that has a probability density function.
     Mathematicians also call such a distribution absolutely continuous, since its cumulative distribution function is
     absolutely continuous with respect to the Lebesgue measure λ. If the distribution of X is continuous, then X is called
     a continuous random variable. There are many examples of continuous probability distributions: normal, uniform,
     chi-squared, and others.
Probability distribution                                                                                                          104


     Intuitively, a continuous random variable is the one which can take a continuous range of values — as opposed to a
     discrete distribution, where the set of possible values for the random variable is at most countable. While for a
     discrete distribution an event with probability zero is impossible (e.g. rolling 3½ on a standard die is impossible, and
     has probability zero), this is not so in the case of a continuous random variable. For example, if one measures the
     width of an oak leaf, the result of 3½ cm is possible, however it has probability zero because there are uncountably
     many other potential values even between 3 cm and 4 cm. Each of these individual outcomes has probability zero,
     yet the probability that the outcome will fall into the interval (3 cm, 4 cm) is nonzero. This apparent paradox is
     resolved by the fact that the probability that X attains some value within an infinite set, such as an interval, cannot be
     found by naively adding the probabilities for individual values. Formally, each value has an infinitesimally small
     probability, which statistically is equivalent to zero.
     Formally, if X is a continuous random variable, then it has a probability density function ƒ(x), and therefore its
     probability of falling into a given interval, say [a, b] is given by the integral



     In particular, the probability for X to take any single value a (that is a ≤ X ≤ a) is zero, because an integral with
     coinciding upper and lower limits is always equal to zero.
     The definition states that a continuous probability distribution must possess a density, or equivalently, its cumulative
     distribution function be absolutely continuous. This requirement is stronger than simple continuity of the cdf, and
     there is a special class of distributions, singular distributions, which are neither continuous nor discrete nor their
     mixture. An example is given by the Cantor distribution. Such singular distributions however are never encountered
     in practice.
     Note on terminology: some authors use the term "continuous distribution" to denote the distribution with continuous
     cdf. Thus, their definition includes both the (absolutely) continuous and singular distributions.
     By one convention, a probability distribution          is called continuous if its cumulative distribution function
                            is continuous and, therefore, the probability measure of singletons                for all   .
     Another convention reserves the term continuous probability distribution for absolutely continuous distributions.
     These distributions can be characterized by a probability density function: a non-negative Lebesgue integrable
     function defined on the real numbers such that



     Discrete distributions and some continuous distributions (like the Cantor distribution) do not admit such a density.


     Probability distributions of scalar random variables
     The following applies to all types of scalar random variables. Because a probability distribution Pr on the real line is
     determined by the probability of a scalar random variable X being in a half-open interval (-∞, x], the probability
     distribution is completely characterized by its cumulative distribution function:




     Some properties
     • The probability distribution of the sum of two independent random variables is the convolution of each of their
       distributions.
     • Probability distributions are not a vector space – they are not closed under linear combinations, as these do not
       preserve non-negativity or total integral 1 – but they are closed under convex combination, thus forming a convex
       subset of the space of functions (or measures).
Probability distribution                                                                                                          105


     Kolmogorov definition
     In the measure-theoretic formalization of probability theory, a random variable is defined as a measurable function X
     from a probability space           to measurable space            . A probability distribution is the pushforward
     measure, P, satisfying X*P = PX −1 on         .


     Random number generation
     A frequent problem in statistical simulations (the Monte Carlo method) is the generation of pseudo-random numbers
     that are distributed in a given way. Most algorithms are based on a pseudorandom number generator that produces
     numbers X that are uniformly distributed in the interval [0,1). These random variates X are then transformed via
     some algorithm to create a new random variate having the required probability distribution.


     Applications
     The concept of the probability distribution and the random variables which they describe underlies the mathematical
     discipline of probability theory, and the science of statistics. There is spread or variability in almost any value that
     can be measured in a population (e.g. height of people, durability of a metal, sales growth, traffic flow, etc.); almost
     all measurements are made with some intrinsic error; in physics many processes are described probabilistically, from
     the kinetic properties of gases to the quantum mechanical description of fundamental particles. For these and many
     other reasons, simple numbers are often inadequate for describing a quantity, while probability distributions are often
     more appropriate.
     As a more specific example of an application, the cache language models and other statistical language models used
     in natural language processing to assign probabilities to the occurrence of particular words and word sequences do so
     by means of probability distributions.


     Common probability distributions
     The following is a list of some of the most common probability distributions, grouped by the type of process that
     they are related to. For a more complete list, see list of probability distributions, which groups by the nature of the
     outcome being considered (discrete, continuous, multivariate, etc.)
     Note also that all of the univariate distributions below are singly peaked; that is, it is assumed that the values cluster
     around a single point. In practice, actually observed quantities may cluster around multiple values. Such quantities
     can be modeled using a mixture distribution.


     Related to real-valued quantities that grow linearly (e.g. errors, offsets)
     • Normal distribution (Gaussian distribution), for a single such quantity; the most common continuous distribution


     Related to positive real-valued quantities that grow exponentially (e.g. prices, incomes,
     populations)
     • Log-normal distribution, for a single such quantity whose log is normally distributed
     • Pareto distribution, for a single such quantity whose log is exponentially distributed; the prototypical power law
       distribution


     Related to real-valued quantities that are assumed to be uniformly distributed over a
     (possibly unknown) region
     • Discrete uniform distribution, for a finite set of values (e.g. the outcome of a fair die)
     • Continuous uniform distribution, for continuously distributed values
Probability distribution                                                                                                       106


     Related to Bernoulli trials (yes/no events, with a given probability)
     • Basic distributions:
       • Bernoulli distribution, for the outcome of a single Bernoulli trial (e.g. success/failure, yes/no)
       • Binomial distribution, for the number of "positive occurrences" (e.g. successes, yes votes, etc.) given a fixed
         total number of independent occurrences
       • Negative binomial distribution, for binomial-type observations but where the quantity of interest is the number
         of failures before a given number of successes occurs
       • Geometric distribution, for binomial-type observations but where the quantity of interest is the number of
         failures before the first success; a special case of the negative binomial distribution
     • Related to sampling schemes over a finite population:
        • Hypergeometric distribution, for the number of "positive occurrences" (e.g. successes, yes votes, etc.) given a
          fixed number of total occurrences, using sampling without replacement
        • Beta-binomial distribution, for the number of "positive occurrences" (e.g. successes, yes votes, etc.) given a
          fixed number of total occurrences, sampling using a Polya urn scheme (in some sense, the "opposite" of
          sampling without replacement)


     Related to categorical outcomes (events with K possible outcomes, with a given probability
     for each outcome)
     • Categorical distribution, for a single categorical outcome (e.g. yes/no/maybe in a survey); a generalization of the
       Bernoulli distribution
     • Multinomial distribution, for the number of each type of categorical outcome, given a fixed number of total
       outcomes; a generalization of the binomial distribution
     • Multivariate hypergeometric distribution, similar to the multinomial distribution, but using sampling without
       replacement; a generalization of the hypergeometric distribution


     Related to events in a Poisson process (events that occur independently with a given rate)
     • Poisson distribution, for the number of occurrences of a Poisson-type event in a given period of time
     • Exponential distribution, for the time before the next Poisson-type event occurs


     Useful for hypothesis testing related to normally distributed outcomes
     • Chi-squared distribution, the distribution of a sum of squared standard normal variables; useful e.g. for inference
       regarding the sample variance of normally distributed samples (see chi-squared test)
     • Student's t distribution, the distribution of the ratio of a standard normal variable and the square root of a scaled
       chi squared variable; useful for inference regarding the mean of normally distributed samples with unknown
       variance (see Student's t-test)
     • F-distribution, the distribution of the ratio of two scaled chi squared variables; useful e.g. for inferences that
       involve comparing variances or involving R-squared (the squared correlation coefficient)
Probability distribution                                                                                                         107


     Useful as conjugate prior distributions in Bayesian inference
     • Beta distribution, for a single probability (real number between 0 and 1); conjugate to the Bernoulli distribution
       and binomial distribution
     • Gamma distribution, for a non-negative scaling parameter; conjugate to the rate parameter of a Poisson
       distribution or exponential distribution, the precision (inverse variance) of a normal distribution, etc.
     • Dirichlet distribution, for a vector of probabilities that must sum to 1; conjugate to the categorical distribution and
       multinomial distribution; generalization of the beta distribution
     • Wishart distribution, for a symmetric non-negative definite matrix; conjugate to the inverse of the covariance
       matrix of a multivariate normal distribution; generalization of the gamma distribution


     References
     • B. S. Everitt: The Cambridge Dictionary of Statistics, Cambridge University Press, Cambridge (3rd edition,
       2006). ISBN 0-521-69027-7
     • Bishop: Pattern Recognition and Machine Learning, Springer, ISBN 0-387-31073-8


     External links
     • Hazewinkel, Michiel, ed. (2001), "Probability distribution" (http://www.encyclopediaofmath.org/index.
       php?title=p/p074900), Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4



     Single-equation methods (econometrics)
     A variety of methods are used in econometrics to estimate models consisting of a single equation. The oldest and
     still the most commonly used is the ordinary least squares method used to estimate linear regressions.
     A variety of methods are available to estimate non-linear models. A particularly important class of non-linear models
     are those used to estimate relationships where the dependent variable is discrete, truncated or censored. These
     include logit, probit and Tobit models.
     Single equation methods may be applied to time-series, cross section or panel data.
Cross-sectional study                                                                                                            108



    Cross-sectional study
    Cross-sectional studies (also known as cross-sectional analyses, transversal studies, prevalence study) form a
    class of research methods that involve observation of all of a population, or a representative subset, at one specific
    point in time. They differ from case-control studies in that they aim to provide data on the entire population under
    study, whereas case-control studies typically include only individuals with a specific characteristic, with a sample,
    often a tiny minority, of the rest of the population. Cross-sectional studies are descriptive studies (neither
    longitudinal nor experimental). Unlike case-control studies, they can be used to describe, not only the Odds ratio, but
    also absolute risks and relative risks from prevalences (sometimes called prevalence risk ratio, or PRR).[1][2] They
    may be used to describe some feature of the population, such as prevalence of an illness, or they may support
    inferences of cause and effect. Longitudinal studies differ from both in making a series of observations more than
    once on members of the study population over a period of time.


    Cross-sectional studies in medicine
    Cross-sectional studies involve data collected at a defined time. They are often used to assess the prevalence of acute
    or chronic conditions, or to answer questions about the causes of disease or the results of medical intervention. They
    may also be described as censuses. Cross-sectional studies may involve special data collection, including questions
    about the past, but they often rely on data originally collected for other purposes. They are moderately expensive,
    and are not suitable for the study of rare diseases. Difficulty in recalling past events may also contribute bias.


    Use of routine data: large scale, low cost to the researcher
    The use of routinely collected data allows large cross-sectional studies to be made at little or no expense. This is a
    major advantage over other forms of epidemiological study. A natural progression has been suggested from cheap
    cross-sectional studies of routinely collected data which suggest hypotheses, to case-control studies testing them
    more specifically, then to cohort studies and trials which cost much more and take much longer, but may give
    stronger evidence. In a cross-sectional survey, a specific group is looked at to see if an activity, say alcohol
    consumption, is related to the health effect being investigated, say cirrhosis of the liver. If alcohol use is correlated
    with cirrhosis of the liver, this would support the hypothesis that alcohol use may cause cirrhosis.


    Routine data not designed to answer the specific question
    However, routinely collected data does not normally describe which variable is the cause and which the effect.
    Cross-sectional studies using data originally collected for other purposes are often unable to include data on
    confounding factors, other variables that affect the relationship between the putative cause and effect. For example,
    data only on present alcohol consumption and cirrhosis would not allow the role of past alcohol consumption, or of
    other causes, to be explored.
    Most case-control studies collect specifically designed data on all participants, including data fields designed to
    allow the hypothesis of interest to be tested. However, in issues where strong personal feelings may be involved,
    specific questions may be a source of bias. For example, past alcohol consumption may be incorrectly reported by an
    individual wishing to reduce their personal feelings of guilt. Such bias may be less in routinely collected statistics, or
    effectively eliminated if the observations are made by third parties, for example taxation records of alcohol by area.
Cross-sectional study                                                                                                                            109


    Aggregated data and the "ecological fallacy"
    Cross-sectional studies can contain individual-level data (one record per individual, for example, in national health
    surveys). However, in modern epidemiology it may be impossible to survey the entire population of interest, so
    cross-sectional studies often involve secondary analysis of data collected for another purpose. In many such cases,
    no individual records are available to the researcher, and group-level information must be used. Major sources of
    such data are often large institutions like the Census Bureau or the Centers for Disease Control in the United States.
    Recent census data is not provided on individuals - in the UK individual census data is released only after a century.
    Instead data are aggregated, usually by administrative area. Inferences about individuals based on aggregate data are
    weakened by the ecological fallacy. Also consider the potential for committing the "atomistic fallacy" where
    assumptions about aggregated counts are made based on the aggregation of individual level data (such as averaging
    census tracts to calculate a county average). For example, it might be true that there is no correlation between infant
    mortality and family income at the city level, while still being true that there is a strong relationship between infant
    mortality and family income at the individual level. All aggregate statistics are subject to compositional effects, so
    that what matters is not only the individual-level relationship between income and infant mortality, but also the
    proportions of low, middle, and high income individuals in each city. Because case-control studies are usually based
    on individual-level data, they do not have this problem.


    References
    [1] "When to use the odds ratio or the relative risk?", by Carsten Oliver Schmidt, Thomas Kohlmann, Int J Public Health 53 (2008) 165–167
        1661-8556/08/030165-3 DOI 10.1007/s000 -00 -7068-3 © Birkhäuser Verlag, Basel, 2008 link (http:/ / www. ncbi. nlm. nih. gov/ pubmed/
        19127890)
    [2] Letters to the Editor, "Odds Ratio or Relative Risk for Cross-Sectional Data?", From JAMES LEE, International Journal of Epidemiology,
        International Epktemiotogical Association 1994, Vol. 23, No. 1. link (http:/ / ije. oxfordjournals. org/ content/ 23/ 1/ 201. long)

    • Epidemiology for the Uninitiated by Coggon, Rose, and Barker, Chapter 8, "Case-control and cross-sectional
      studies", BMJ (British Medical Journal) Publishing, 1997 (http://resources.bmj.com/bmj/readers/readers/
      epidemiology-for-the-uninitiated/8-case-control-and-cross-sectional-studies)
    • Research Methods Knowledge Base by William M. K. Trochim, Web Center for Social Research Methods,
      copyright 2006 (http://www.socialresearchmethods.net/kb/timedim.php)
    • Cross-Sectional Design by Michelle A. Saint-Germain (http://www.csulb.edu/~msaintg/ppa696/696preex.
      htm#Cross-Sectional Design)


    External links
    • Study Design Tutorial (http://www.vet.cornell.edu/imaging/tutorial/4studydesigns/crosssectional.html)
      Cornell University College of Veterinary Medicine
Spatial analysis                                                                                                                            110



     Spatial analysis
     Spatial analysis or spatial statistics
     includes any of the formal techniques which
     study entities using their topological,
     geometric, or geographic properties. The
     phrase properly refers to a variety of
     techniques, many still in their early
     development, using different analytic
     approaches and applied in fields as diverse
     as astronomy, with its studies of the
     placement of galaxies in the cosmos, to chip
     fabrication engineering, with its use of
     'place and route' algorithms to build
     complex wiring structures. The phrase is
     often used in a more restricted sense to
     describe techniques applied to structures at
     the human scale, most notably in the
     analysis of geographic data. The phrase is
     even sometimes used to refer to a specific
     technique in a single area of research, for       Map by Dr. John Snow of London, showing clusters of cholera cases in the 1854
     example, to describe geostatistics.               Broad Street cholera outbreak. This was one of the first uses of map-based spatial
                                                                                           analysis.
     Complex issues arise in spatial analysis,
     many of which are neither clearly defined nor completely resolved, but form the basis for current research. The most
     fundamental of these is the problem of defining the spatial location of the entities being studied. For example, a
     study on human health could describe the spatial position of humans with a point placed where they live, or with a
     point located where they work, or by using a line to describe their weekly trips; each choice has dramatic effects on
     the techniques which can be used for the analysis and on the conclusions which can be obtained. Other issues in
     spatial analysis include the limitations of mathematical knowledge, the assumptions required by existing statistical
     techniques, and problems in computer based calculations.
     Classification of the techniques of spatial analysis is difficult because of the large number of different fields of
     research involved, the different fundamental approaches which can be chosen, and the many forms the data can take.


     The history of the Spatial Analysis
     Spatial analysis can perhaps be considered to have arisen with the early attempts at cartography and surveying but
     many fields have contributed to its rise in modern form. Biology contributed through botanical studies of global
     plant distributions and local plant locations, ethological studies of animal movement, landscape ecological studies of
     vegetation blocks, ecological studies of spatial population dynamics, and the study of biogeography. Epidemiology
     contributed with early work on disease mapping, notably [Dr. Snow's work] mapping an outbreak of cholera, with
     research on mapping the spread of disease and with locational studies for health care delivery. Statistics has
     contributed greatly through work in spatial statistics. Economics has contributed notably through spatial
     econometrics. Geographic information system is currently a major contributor due to the importance of geographic
     software in the modern analytic toolbox. Remote sensing has contributed extensively in morphometric and clustering
     analysis. Computer science has contributed extensively through the study of algorithms, notably in computational
     geometry. Mathematics continues to provide the fundamental tools for analysis and to reveal the complexity of the
Spatial analysis                                                                                                                            111


     spatial realm, for example, with recent work on fractals and scale invariance. Scientific modelling provides a useful
     framework for new approaches.


     Fundamental issues in spatial analysis
     Spatial analysis confronts many fundamental issues in the definition of its objects of study, in the construction of the
     analytic operations to be used, in the use of computers for analysis, in the limitations and particularities of the
     analyses which are known, and in the presentation of analytic results. Many of these issues are active subjects of
     modern research.
     Common errors often arise in spatial analysis, some due to the mathematics of space, some due to the particular ways
     data are presented spatially, some due to the tools which are available. Census data, because it protects individual
     privacy by aggregating data into local units, raises a number of statistical issues. The fractal nature of coastline
     makes precise measurements of its length difficult if not impossible. A computer software fitting straight lines to the
     curve of a coastline, can easily calculate the lengths of the lines which it defines. However these straight lines may
     have no inherent meaning in the real world, as was shown for the coastline of Britain.
     These problems represent a challenge in spatial analysis because of the power of maps as media of presentation.
     When results are presented as maps, the presentation combines spatial data which are generally accurate with
     analytic results which may be inaccurate, leading to an impression that analytic results are more accurate than the
     data would indicate.[1]


     Spatial characterization
     The definition of the spatial presence of an entity
     constrains the possible analysis which can be applied to
     that entity and influences the final conclusions that can be
     reached. While this property is fundamentally true of all
     analysis, it is particularly important in spatial analysis
     because the tools to define and study entities favor
     specific characterizations of the entities being studied.
     Statistical techniques favor the spatial definition of
     objects as points because there are very few statistical
     techniques which operate directly on line, area, or volume
     elements. Computer tools favor the spatial definition of
     objects as homogeneous and separate elements because of
     the limited number of database elements and
     computational structures available, and the ease with
     which these primitive structures can be created.


     Spatial dependency or auto-correlation                          Spread of bubonic plague in medieval Europe. The colors indicate
     Spatial dependency is the co-variation of properties            the spatial distribution of plague outbreaks over time. Possibly due
                                                                       to the limitations of printing or for a host of other reasons, the
     within geographic space: characteristics at proximal
                                                                      cartographer selected a discrete number of colors to characterize
     locations appear to be correlated, either positively or                                 (and simplify) reality.
     negatively. Spatial dependency leads to the spatial
     autocorrelation problem in statistics since, like temporal autocorrelation, this violates standard statistical techniques
     that assume independence among observations. For example, regression analyses that do not compensate for spatial

     dependency can have unstable parameter estimates and yield unreliable significance tests. Spatial regression models
     (see below) capture these relationships and do not suffer from these weaknesses. It is also appropriate to view spatial
Spatial analysis                                                                                                                  112


     dependency as a source of information rather than something to be corrected.[2]
     Locational effects also manifest as spatial heterogeneity, or the apparent variation in a process with respect to
     location in geographic space. Unless a space is uniform and boundless, every location will have some degree of
     uniqueness relative to the other locations. This affects the spatial dependency relations and therefore the spatial
     process. Spatial heterogeneity means that overall parameters estimated for the entire system may not adequately
     describe the process at any given location.


     Scaling
     Spatial measurement scale is a persistent issue in spatial analysis; more detail is available at the modifiable areal unit
     problem (MAUP) topic entry. Landscape ecologists developed a series of scale invariant metrics for aspects of
     ecology that are fractal in nature. In more general terms, no scale independent method of analysis is widely agreed
     upon for spatial statistics.


     Sampling
     Spatial sampling involves determining a limited number of locations in geographic space for faithfully measuring
     phenomena that are subject to dependency and heterogeneity. Dependency suggests that since one location can
     predict the value of another location, we do not need observations in both places. But heterogeneity suggests that this
     relation can change across space, and therefore we cannot trust an observed degree of dependency beyond a region
     that may be small. Basic spatial sampling schemes include random, clustered and systematic. These basic schemes
     can be applied at multiple levels in a designated spatial hierarchy (e.g., urban area, city, neighborhood). It is also
     possible to exploit ancillary data, for example, using property values as a guide in a spatial sampling scheme to
     measure educational attainment and income. Spatial models such as autocorrelation statistics, regression and
     interpolation (see below) can also dictate sample design.


     Common errors in spatial analysis
     The fundamental issues in spatial analysis lead to numerous problems in analysis including bias, distortion and
     outright errors in the conclusions reached. These issues are often interlinked but various attempts have been made to
     separate out particular issues from each other.

     Length
     In a paper by Benoit Mandelbrot on the coastline of Britain it was shown that it is inherently nonsensical to discuss
     certain spatial concepts despite an inherent presumption of the validity of the concept. Lengths in ecology depend
     directly on the scale at which they are measured and experienced. So while surveyors commonly measure the length
     of a river, this length only has meaning in the context of the relevance of the measuring technique to the question
     under study.
Spatial analysis                                                                                                                  113




           Britain measured using a long            Britain measured using a medium            Britain measured using a short
                     yardstick                                  yardstick                                yardstick



     Locational fallacy
     The locational fallacy refers to error due to the particular spatial characterization chosen for the elements of study, in
     particular choice of placement for the spatial presence of the element.
     Spatial characterizations may be simplistic or even wrong. Studies of humans often reduce the spatial existence of
     humans to a single point, for instance their home address. This can easily lead to poor analysis, for example, when
     considering disease transmission which can happen at work or at school and therefore far from the home.
     The spatial characterization may implicitly limit the subject of study. For example, the spatial analysis of crime data
     has recently become popular but these studies can only describe the particular kinds of crime which can be described
     spatially. This leads to many maps of assault but not to any maps of embezzlement with political consequences in the
     conceptualization of crime and the design of policies to address the issue.

     Atomic fallacy
     This describes errors due to treating elements as separate 'atoms' outside of their spatial context.

     Ecological fallacy
     The ecological fallacy describes errors due to performing analyses on aggregate data when trying to reach
     conclusions on the individual units. Errors occur in part from spatial aggregation. For example a pixel represents the
     average surface temperatures within an area. Ecological fallacy would be to assume that all points within the area
     have the same temperature. This topic is closely related to the modifiable areal unit problem.
Spatial analysis                                                                                                                           114


     Solutions to the fundamental issues

     Geographic space

     A mathematical space exists whenever we have a set of
     observations and quantitative measures of their attributes. For
     example, we can represent individuals' income or years of
     education within a coordinate system where the location of each
     individual can be specified with respect to both dimensions. The
     distances between individuals within this space is a quantitative
     measure of their differences with respect to income and education.
     However, in spatial analysis we are concerned with specific types
     of mathematical spaces, namely, geographic space. In geographic
     space, the observations correspond to locations in a spatial
     measurement framework that captures their proximity in the real
     world. The locations in a spatial measurement framework often
     represent locations on the surface of the Earth, but this is not
                                                                              Manhattan distance versus Euclidean distance: The red,
     strictly necessary. A spatial measurement framework can also                blue, and yellow lines have the same length (12) in
     capture proximity with respect to, say, interstellar space or within        both Euclidean and taxicab geometry. In Euclidean
     a biological entity such as a liver. The fundamental tenet is            geometry, the green line has length 6×√2 ≈ 8.48, and is
                                                                              the unique shortest path. In taxicab geometry, the green
     Tobler's First Law of Geography: if the interrelation between
                                                                                line's length is still 12, making it no shorter than any
     entities increases with proximity in the real world, then                                      other path shown.
     representation in geographic space and assessment using spatial
     analysis techniques are appropriate.
     The Euclidean distance between locations often represents their proximity, although this is only one possibility.
     There are an infinite number of distances in addition to Euclidean that can support quantitative analysis. For
     example, "Manhattan" (or "Taxicab") distances where movement is restricted to paths parallel to the axes can be
     more meaningful than Euclidean distances in urban settings. In addition to distances, other geographic relationships
     such as connectivity (e.g., the existence or degree of shared borders) and direction can also influence the
     relationships among entities. It is also possible to compute minimal cost paths across a cost surface; for example, this
     can represent proximity among locations when travel must occur across rugged terrain.


     Types of spatial analysis
     Spatial data comes in many varieties and it is not easy to
     arrive at a system of classification that is simultaneously
     exclusive, exhaustive, imaginative, and satisfying.
                                 -- G. Upton & B. Fingelton[3]


     Spatial data analysis
     Urban and Regional Studies deal with large tables of spatial data obtained from censuses and surveys. It is necessary
     to simplify the huge amount of detailed information in order to extract the main trends. Multivariate analysis (or
     Factor_analysis Factor analysis, FA) allows a change of variables, transforming the many variables of the census,
     usually correlated between themselves, into fewer independent "Factors" or "Principal Components" which are,
     actually, the eigenvectors of the data correlation matrix weighted by the inverse of their eigenvalues. This change of
     variables has two main advantages :
     1. Since information is concentrated on the first new factors, it is possible to keep only a few of them while losing
        only a small amount of information ; mapping them produces fewer and more significant maps
Spatial analysis                                                                                                                    115


     2. The factors, actually the eigenvectors, are orthogonal by construction, i.e. not correlated. In most cases, the
        dominant factor (with the largest eigenvalue) is the Social Component, separating rich and poor in the city. Since
        factors are not-correlated, other smaller processes than social status, which would have remained hidden
        otherwise, appear on the second, third, … factors.
     Factor analysis depends on measuring distances between observations : the choice of a significant metric is crucial.
     The Euclidean metric (Principal Component Analysis), the Chi-Square distance (Correspondence Analysis) or the
     Generalized Mahalanobis distance (Discriminant Analysis ) are among the more widely used.[4] More complicated
     models, using communalities or rotations have been proposed.[5]
     Using multivariate methods in spatial analysis began really in the 1950s (although some examples go back to the
     beginning of the century) and culminated in the 1970s, with the increasing power and accessibility of computers.
     Already in 1948, in a seminal publication, two sociologists, Bell and Shevky,[6] had shown that most city populations
     in the USA and in the world could be represented with three independent factors : 1- the « socio-economic status »
     opposing rich and poor districts and distributed in sectors running along highways from the city center, 2- the « life
     cycle », i.e. the age structure of households, distributed in concentric circles, and 3- « race and ethnicity », identifying
     patches of migrants located within the city. In 1961, in a groundbreaking study, British geographers used FA to
     classify British towns.[7] Brian J Berry, at the University of Chicago, and his students made a wide use of the
     method,[8] applying it to most important cities in the world and exhibiting common social structures.[9] The use of
     Factor Analysis in Geography, made so easy by modern computers, has been very wide but not always very wise.[10]
     Since the vectors extracted are determined by the data matrix, it is not possible to compare factors obtained from
     different censuses. A solution consists in fusing together several census matrices in a unique table which, then, may
     be analyzed. This, however, assumes that the definition of the variables has not changed over time and produces very
     large tables, difficult to manage. A better solution, proposed by psychometricians,[11] groups the data in a « cubic
     matrix », with three entries (for instance, locations, variables, time periods). A Three-Way Factor Analysis produces
     then three groups of factors related by a small cubic « core matrix ».[12] This method, which exhibits data evolution
     over time, has not been widely used in geography.[13] In Los Angeles,[14] however, it has exhibited the role,
     traditionally ignored, of Downtown as an organizing center for the whole city during several decades.


     Spatial autocorrelation
     Spatial autocorrelation statistics measure and analyze the degree of dependency among observations in a geographic
     space. Classic spatial autocorrelation statistics include Moran's , Geary's       , Getis's    and the standard
     deviational ellipse. These statistics require measuring a spatial weights matrix that reflects the intensity of the
     geographic relationship between observations in a neighborhood, e.g., the distances between neighbors, the lengths
     of shared border, or whether they fall into a specified directional class such as "west". Classic spatial autocorrelation
     statistics compare the spatial weights to the covariance relationship at pairs of locations. Spatial autocorrelation that
     is more positive than expected from random indicate the clustering of similar values across geographic space, while
     significant negative spatial autocorrelation indicates that neighboring values are more dissimilar than expected by
     chance, suggesting a spatial pattern similar to a chess board.
     Spatial autocorrelation statistics such as Moran's      and Geary's      are global in the sense that they estimate the
     overall degree of spatial autocorrelation for a dataset. The possibility of spatial heterogeneity suggests that the
     estimated degree of autocorrelation may vary significantly across geographic space. Local spatial autocorrelation
     statistics provide estimates disaggregated to the level of the spatial analysis units, allowing assessment of the
     dependency relationships across space.     statistics compare neighborhoods to a global average and identify local
     regions of strong autocorrelation. Local versions of the     and    statistics are also available.
Spatial analysis                                                                                                                 116


     Spatial interpolation
     Spatial interpolation methods estimate the variables at unobserved locations in geographic space based on the values
     at observed locations. Basic methods include inverse distance weighting: this attenuates the variable with decreasing
     proximity from the observed location. Kriging is a more sophisticated method that interpolates across space
     according to a spatial lag relationship that has both systematic and random components. This can accommodate a
     wide range of spatial relationships for the hidden values between observed locations. Kriging provides optimal
     estimates given the hypothesized lag relationship, and error estimates can be mapped to determine if spatial patterns
     exist.


     Spatial regression
     Spatial regression methods capture spatial dependency in regression analysis, avoiding statistical problems such as
     unstable parameters and unreliable significance tests, as well as providing information on spatial relationships among
     the variables involved. Depending on the specific technique, spatial dependency can enter the regression model as
     relationships between the independent variables and the dependent, between the dependent variables and a spatial lag
     of itself, or in the error terms. Geographically weighted regression (GWR) is a local version of spatial regression
     that generates parameters disaggregated by the spatial units of analysis.[15] This allows assessment of the spatial
     heterogeneity in the estimated relationships between the independent and dependent variables. The use of Markov
     Chain Monte Carlo (MCMC) methods can allow the estimation of complex functions, such as
     Poisson-Gamma-CAR, Poisson-lognormal-SAR, or Overdispersed logit models.


     Spatial interaction
     Spatial interaction or "gravity models" estimate the flow of people, material or information between locations in
     geographic space. Factors can include origin propulsive variables such as the number of commuters in residential
     areas, destination attractiveness variables such as the amount of office space in employment areas, and proximity
     relationships between the locations measured in terms such as driving distance or travel time. In addition, the
     topological, or connective, relationships between areas must be identified, particularly considering the often
     conflicting relationship between distance and topology; for example, two spatially close neighborhoods may not
     display any significant interaction if they are separated by a highway. After specifying the functional forms of these
     relationships, the analyst can estimate model parameters using observed flow data and standard estimation
     techniques such as ordinary least squares or maximum likelihood. Competing destinations versions of spatial
     interaction models include the proximity among the destinations (or origins) in addition to the origin-destination
     proximity; this captures the effects of destination (origin) clustering on flows. Computational methods such as
     artificial neural networks can also estimate spatial interaction relationships among locations and can handle noisy
     and qualitative data.


     Simulation and modeling
     Spatial interaction models are aggregate and top-down: they specify an overall governing relationship for flow
     between locations. This characteristic is also shared by urban models such as those based on mathematical
     programming, flows among economic sectors, or bid-rent theory. An alternative modeling perspective is to represent
     the system at the highest possible level of disaggregation and study the bottom-up emergence of complex patterns
     and relationships from behavior and interactions at the individual level.
     Complex adaptive systems theory as applied to spatial analysis suggests that simple interactions among proximal
     entities can lead to intricate, persistent and functional spatial entities at aggregate levels. Two fundamentally spatial
     simulation methods are cellular automata and agent-based modeling. Cellular automata modeling imposes a fixed
     spatial framework such as grid cells and specifies rules that dictate the state of a cell based on the states of its
     neighboring cells. As time progresses, spatial patterns emerge as cells change states based on their neighbors; this
Spatial analysis                                                                                                               117


     alters the conditions for future time periods. For example, cells can represent locations in an urban area and their
     states can be different types of land use. Patterns that can emerge from the simple interactions of local land uses
     include office districts and urban sprawl. Agent-based modeling uses software entities (agents) that have purposeful
     behavior (goals) and can react, interact and modify their environment while seeking their objectives. Unlike the cells
     in cellular automata, agents can be mobile with respect to space. For example, one could model traffic flow and
     dynamics using agents representing individual vehicles that try to minimize travel time between specified origins and
     destinations. While pursuing minimal travel times, the agents must avoid collisions with other vehicles also seeking
     to minimize their travel times. Cellular automata and agent-based modeling are complementary modeling strategies.
     They can be integrated into a common geographic automata system where some agents are fixed while others are
     mobile.


     Multiple-Point Geostatistics (MPS)
     Spatial analysis of a conceptual geological model is the main purpose of any MPS algorithm. The method analyzes
     the spatial statistics of the geological model, called the training image, and generates realizations of the phenomena
     that honor those input multiple-point statistics.
     One of the recent technique to accomplish this task is the pattern-based method of Honarkhah.[16] In this method, a
     distance-based approach is employed to analyze the patterns in the training image. This allows the reproduction of
     the multiple-point statistics, and the complex geometrical features of the given image. The final generated
     realizations of this, so called random field, can be used to quantify spatial uncertainty.
     In the recent method, Tahmasebi et al.[17] used a cross-correlation function to better spatial pattern reproduction and
     they presented the CCSIM algorithm. This method is able to quantify the spatial connectivity, variability and
     uncertainty. Furthermore, the method is not sensitive to any type of data and is able to simulate both categorical and
     continuous scenarios.


     Geographic information science and spatial analysis
     Geographic information systems (GIS) and the underlying geographic information science that advances these
     technologies have a strong influence on spatial analysis. The increasing ability to capture and handle geographic data
     means that spatial analysis is occurring within increasingly data-rich environments. Geographic data capture systems
     include remotely sensed imagery, environmental monitoring systems such as intelligent transportation systems, and
     location-aware technologies such as mobile devices that can report location in near-real time. GIS provide platforms
     for managing these data, computing spatial relationships such as distance, connectivity and directional relationships
     between spatial units, and visualizing both the raw data and spatial analytic results within a cartographic context.
Spatial analysis                                                                                                                                    118



     Content
     • Spatial location: Transfer positioning
       information of space objects with the
       help of space coordinate system.
       Projection transformation theory is the
       foundation of spatial object
       representation.
     • Spatial distribution: the similar spatial
       object groups positioning information,
                                                      This flow map of Napoleon's ill-fated march on Moscow is an early and celebrated
       including distribution, trends, contrast
                                                       example of geovisualization. It shows the army's direction as it traveled, the places
       etc..                                             the troops passed through, the size of the army as troops died from hunger and
     • Spatial form: the geometric shape of the                     wounds, and the freezing temperatures they experienced.
       spatial objects
     • Spatial space: the space objects' approaching degree
     • Spatial relationship: relationship between spatial objects, including topological, orientation, similarity, etc..
     Geovisualization (GVis) combines scientific visualization with digital cartography to support the exploration and
     analysis of geographic data and information, including the results of spatial analysis or simulation. GVis leverages
     the human orientation towards visual information processing in the exploration, analysis and communication of
     geographic data and information. In contrast with traditional cartography, GVis is typically three or four-dimensional
     (the latter including time) and user-interactive.
     Geographic knowledge discovery (GKD) is the human-centered process of applying efficient computational tools
     for exploring massive spatial databases. GKD includes geographic data mining, but also encompasses related
     activities such as data selection, data cleaning and pre-processing, and interpretation of results. GVis can also serve a
     central role in the GKD process. GKD is based on the premise that massive databases contain interesting (valid,
     novel, useful and understandable) patterns that standard analytical techniques cannot find. GKD can serve as a
     hypothesis-generating process for spatial analysis, producing tentative patterns and relationships that should be
     confirmed using spatial analytical techniques.
     Spatial Decision Support Systems (SDSS) take existing spatial data and use a variety of mathematical models to
     make projections into the future. This allows urban and regional planners to test intervention decisions prior to
     implementation.


     References
     [1] Mark Monmonier How to Lie with Maps University of Chicago Press, 1996.
     [2] De Knegt; H.J.; F. van Langevelde; M.B. Coughenour; A.K. Skidmore; W.F. de Boer; I.M.A. Heitkönig; N.M. Knox; R. Slotow; C. van der
         Waal and H.H.T. Prins (2010). Spatial autocorrelation and the scaling of species–environment relationships. Ecology 91: 2455–2465.
         doi:10.1890/09-1359.1
     [3] Graham J. Upton & Bernard Fingelton: Spatial Data Analysis by Example Volume 1: Point Pattern and Quantitative Data John Wiley &
         Sons, New York. 1985.
     [4] Harman H H (1960) Modern Factor Analysis, University of Chicago Press
     [5] Rummel R J (1970) Applied Factor Analysis. Evanston, ILL: Northwestern University Press.
     [6] Bell W & E Shevky (1955) Social Area Analysis, Stanford University Press
     [7] Moser C A & W Scott (1961) British Towns ; A Statistical Study of their Social and Economic Differences, Oliver & Boyd, London.
     [8] Berry B J & F Horton (1971) Geographic Perspectives on Urban Systems, John Wiley, N-Y.
     [9] Berry B J & K B Smith eds (1972) City Classification Handbook : Methods and Applications, John Wiley, N-Y.
     [10] Ciceri M-F (1974) Méthodes d’analyse multivariée dans la géographie anglo-saxonne, Université de Paris-1 ; free download on http:/ /
         www-ohp. univ-paris1. fr
     [11] Tucker L R (1964) « The extension of Factor Analysis to three-dimensional matrices », in Frederiksen N & H Gulliksen eds, Contributions
         to Mathematical Psychology, Holt, Reinhart and Winston, N-Y.
Spatial analysis                                                                                                                                       119

     [12] R. Coppi & S. Bolasco, eds. (1989), Multiway data analysis, Elsevier, Amsterdam.
     [13] Cant, R.G. (1971) « Changes in the location of manufacturing in New Zealand 1957-1968: An application of three-mode factor analysis. »,
         New Zealand Geographer, 27, 38-55.
     [14] Marchand B (1986) The Emergence of Los Angeles, 1940-1970, Pion Ltd, London
     [15] Fotheringham, A. S., Charlton, M. E., & Brunsdon, C. (1998). Geographically weighted regression: a natural evolution of the expansion
         method for spatial data analysis. Environment and Planning A, 30(11), 1905-1927. doi:10.1068/a301905
     [16] Honarkhah, M and Caers, J, 2010, Stochastic Simulation of Patterns Using Distance-Based Pattern Modeling (http:/ / dx. doi. org/ 10. 1007/
         s11004-010-9276-7), Mathematical Geosciences, 42: 487 - 517
     [17] Tahmasebi, P., Hezarkhani, A., Sahimi, M., 2012, Multiple-point geostatistical modeling based on the cross-correlation functions (http:/ /
         www. springerlink. com/ content/ 0150455247264825/ fulltext. pdf), Computational Geosciences, 16(3):779-79742



     Further reading
     • Abler, R., J. Adams, and P. Gould (1971) Spatial Organization–The Geographer's View of the World, Englewood
       Cliffs, NJ: Prentice-Hall.
     • Anselin, L. (1995) "Local indicators of spatial association – LISA". Geographical Analysis, 27, 93–115 (http://
       www.drs.wisc.edu/documents/articles/curtis/cesoc977/Anselin1995.pdf).
     • Banerjee, S., B.P. Carlin and A.E. Gelfand (2004). Hierarchical Modeling and Analysis for Spatial Data. Taylor
       and Francis: Chapman and Hall/CRC Press.
     • Benenson, I. and P. M. Torrens. (2004). Geosimulation: Automata-Based Modeling of Urban Phenomena. Wiley.
     • Fotheringham, A. S., C. Brunsdon and M. Charlton (2000) Quantitative Geography: Perspectives on Spatial Data
       Analysis, Sage.
     • Fotheringham, A. S. and M. E. O'Kelly (1989) Spatial Interaction Models: Formulations and Applications,
       Kluwer Academic
     • Fotheringham, A. S. and P. A. Rogerson (1993) "GIS and spatial analytical problems". International Journal of
       Geographical Information Systems, 7, 3–19.
     • Goodchild, M. F. (1987) "A spatial analytical perspective on geographical information systems". International
       Journal of Geographical Information Systems, 1, 327–44 (http://www.geog.ucsb.edu/~good/papers/95.pdf).
     • MacEachren, A. M. and D. R. F. Taylor (eds.) (1994) Visualization in Modern Cartography, Pergamon.
     • Levine, N. (2010). CrimeStat: A Spatial Statistics Program for the Analysis of Crime Incident Locations. Version
       3.3. Ned Levine & Associates, Houston, TX and the National Institute of Justice, Washington, DC. Ch. 1-17 + 2
       update chapters (http://www.icpsr.umich.edu/CrimeStat)
     • Miller, H. J. (2004) "Tobler's First Law and spatial analysis". Annals of the Association of American
       Geographers, 94, 284–289.
     • Miller, H. J. and J. Han (eds.) (2001) Geographic Data Mining and Knowledge Discovery, Taylor and Francis.
     • O'Sullivan, D. and D. Unwin (2002) Geographic Information Analysis, Wiley.
     • Parker, D. C., S. M. Manson, M.A. Janssen, M. J. Hoffmann and P. Deadman (2003) "Multi-agent systems for the
       simulation of land-use and land-cover change: A review". Annals of the Association of American Geographers,
       93, 314–337 (http://uwf.edu/zhu/evr6930/17.pdf).
     • White, R. and G. Engelen (1997) "Cellular automata as the basis of integrated dynamic regional modelling".
       Environment and Planning B: Planning and Design, 24, 235–246.
     • Scheldeman, X. & van Zonneveld, M. (2010). Training Manual on Spatial Analysis of Plant Diversity and
       Distribution (http://www.bioversityinternational.org/training/training_materials/gis_manual.html).
       Bioversity International.
     • Fisher MM, Leung Y (2001) Geocomputational Modelling: techniques and applications. Springer Verlag, Berlin
     • Fotheringham S, Clarke G, Abrahart B (1997) Geocomputation and GIS. Transactions in GIS 2:199-200
     • Openshaw S and Abrahart RJ (2000) GeoComputation. CRC Press
     • Diappi L (2004) Evolving Cities: Geocomputation in Territorial Planning. Ashgate, England
     • Longley PA, Brooks SM, McDonnell R, Macmillan B (1998), Geocomputation, a primer. John Wiley and Sons,
       Chichester
Spatial analysis                                                                                                                    120


     • Ehlen J, Caldwell DR and Harding S (2002) GeoComputation: what is it?,Comput Environ and Urban Syst
       26:257-265
     • Gahegan M (1999) What is Geocomputation? Transaction in GIS 3:203-206
     • Murgante B., Borruso G., Lapucci A. (2009) "Geocomputation and Urban Planning" Studies in Computational
       Intelligence, Vol. 176. Springer-Verlag, Berlin.
     • Fischer M., Leung Y.(2010) "GeoComputational Modelling: Techniques and Applications" Advances in Spatial
       Science. Springer-Verlag, Berlin.
     • Murgante B., Borruso G., Lapucci A. (2011) "Geocomputation, Sustainability and Environmental Planning"
       Studies in Computational Intelligence, Vol. 348. Springer-Verlag, Berlin.
     • Tahmasebi, P., Hezarkhani, A., Sahimi, M., 2012, Multiple-point geostatistical modeling based on the
       cross-correlation functions (http://www.springerlink.com/content/0150455247264825/fulltext.pdf),
       Computational Geosciences, 16(3):779-79742.


     External links
     • ICA Commission on Geospatial Analysis and Modeling (https://sites.google.com/site/commissionofica/)
     • An educational resource about spatial statistics and geostatistics (http://www.ai-geostats.org/)
     • A comprehensive guide to principles, techniques & software tools (http://www.spatialanalysisonline.com/)
     • Social and Spatial Inequalities (http://sasi.group.shef.ac.uk/)
     • National Center for Geographic Information and Analysis (NCGIA) (http://www.ncgia.ucsb.edu/)
     • International Cartographic Association (ICA) (http://www.icaci.org), the world body for mapping and
       GIScience professionals



     Effect size
     In statistics, an effect size is a measure of the strength of a phenomenon[1] (for example, the relationship between
     two variables in a statistical population) or a sample-based estimate of that quantity. An effect size calculated from
     data is a descriptive statistic that conveys the estimated magnitude of a relationship without making any statement
     about whether the apparent relationship in the data reflects a true relationship in the population. In that way, effect
     sizes complement inferential statistics such as p-values. Among other uses, effect size measures play an important
     role in meta-analysis studies that summarize findings from a specific area of research, and in statistical power
     analyses.
     The concept of effect size appears already in everyday language. For example, a weight loss program may boast that
     it leads to an average weight loss of 30 pounds. In this case, 30 pounds is an indicator of the claimed effect size.
     Another example is that a tutoring program may claim that it raises school performance by one letter grade. This
     grade increase is the claimed effect size of the program. These are both examples of "absolute effect sizes", meaning
     that they convey the average difference between two groups without any discussion of the variability within the
     groups. For example, if the weight loss program results in an average loss of 30 pounds, it is possible that every
     participant loses exactly 30 pounds, or half the participants lose 60 pounds and half lose no weight at all.
     Reporting effect sizes is considered good practice when presenting empirical research findings in many fields.[2][3]
     The reporting of effect sizes facilitates the interpretation of the substantive, as opposed to the statistical, significance
     of a research result.[4] Effect sizes are particularly prominent in social and medical research. Relative and absolute
     measures of effect size convey different information, and can be used complementarily. A prominent task force in
     the psychology research community expressed the following recommendation:
           Always present effect sizes for primary outcomes...If the units of measurement are meaningful on a practical
           level (e.g., number of cigarettes smoked per day), then we usually prefer an unstandardized measure
Effect size                                                                                                                         121


              (regression coefficient or mean difference) to a standardized measure (r or d).
                                                 — L. Wilkinson and APA Task Force on Statistical Inference (1999, p. 599)


     Overview

     Population and sample effect sizes
     The term effect size can refer to a statistic calculated from a sample of data, or to a parameter of a hypothetical
     statistical population. Conventions for distinguishing sample from population effect sizes follow standard statistical
     practices — one common approach is to use Greek letters like ρ to denote population parameters and Latin letters
     like r to denote the corresponding statistic; alternatively, a "hat" can be placed over the population parameter to
     denote the statistic, e.g. with being the estimate of the parameter .
     As in any statistical setting, effect sizes are estimated with error, and may be biased unless the effect size estimator
     that is used is appropriate for the manner in which the data were sampled and the manner in which the measurements
     were made. An example of this is publication bias, which occurs when scientists only report results when the
     estimated effect sizes are large or are statistically significant. As a result, if many researchers are carrying out studies
     under low statistical power, the reported results are biased to be stronger than true effects, if any.[5] Another example
     where effect sizes may be distorted is in a multiple trial experiment, where the effect size calculation is based on the
     averaged or aggregated response across the trials.[6]


     Relationship to test statistics
     Sample-based effect sizes are distinguished from test statistics used in hypothesis testing, in that they estimate the
     strength of an apparent relationship, rather than assigning a significance level reflecting whether the relationship
     could be due to chance. The effect size does not determine the significance level, or vice-versa. Given a sufficiently
     large sample size, a statistical comparison will always show a significant difference unless the population effect size
     is exactly zero. For example, a sample Pearson correlation coefficient of 0.1 is strongly statistically significant if the
     sample size is 1000. Reporting only the significant p-value from this analysis could be misleading if a correlation of
     0.1 is too small to be of interest in a particular application.


     Standardized and unstandardized effect sizes
     The term effect size can refer to a standardized measures of effect (such as r, Cohen's d, and odds ratio), or to an
     unstandardized measure (e.g., the raw difference between group means and unstandardized regression coefficients).
     Standardized effect size measures are typically used when the metrics of variables being studied do not have intrinsic
     meaning (e.g., a score on a personality test on an arbitrary scale), when results from multiple studies are being
     combined, when some or all of the studies use different scales, or when it is desired to convey the size of an effect
     relative to the variability in the population. In meta-analysis, standardized effect sizes are used as a common measure
     that can be calculated for different studies and then combined into an overall summary.
Effect size                                                                                                                         122


     Types

     Effect Sizes Based on "Variance Explained"
     These effect sizes estimate the amount of the variance within an experiment that is "explained" or "accounted for" by
     the experiment's model.

     Pearson r (correlation)
     Pearson's correlation, often denoted r and introduced by Karl Pearson, is widely used as an effect size when paired
     quantitative data are available; for instance if one were studying the relationship between birth weight and longevity.
     The correlation coefficient can also be used when the data are binary. Pearson's r can vary in magnitude from −1 to
     1, with −1 indicating a perfect negative linear relation, 1 indicating a perfect positive linear relation, and 0 indicating
     no linear relation between two variables. Cohen gives the following guidelines for the social sciences:[7][8]

                                                              Effect size    r

                                                              Small         0.10

                                                              Medium        0.30

                                                              Large         0.50



     Coefficient of determination
     A related effect size is r², the coefficient of determination (also referred to as "r-squared"), calculated as the square of
     the Pearson correlation r. In the case of paired data, this is a measure of the proportion of variance shared by the two
     variables, and varies from 0 to 1. For example, with an r of 0.21 the coefficient of determination is 0.0441, meaning
     that 4.4% of the variance of either variable is shared with the other variable. The r² is always positive, so does not
     convey the direction of the correlation between the two variables.

     Cohen's ƒ2
     Cohen's ƒ2 is one of several effect size measures to use in the context of an F-test for ANOVA or multiple regression.
     Note that it estimates for the sample rather than the population and is biased (overestimates effect size for the
     ANOVA).
     The ƒ2 effect size measure for multiple regression is defined as:




              where R2 is the squared multiple correlation.
     The       effect size measure for hierarchical multiple regression is defined as:



              where R2A is the variance accounted for by a set of one or more independent variables A, and R2AB is the
              combined variance accounted for by A and another set of one or more independent variables B.
     By convention, ƒ2A effect sizes of 0.02, 0.15, and 0.35 are termed small, medium, and large, respectively.[7]

     Cohen's       can also be found for factorial analysis of variance (ANOVA, aka the F-test) working backwards using :


     In a balanced design (equivalent sample sizes across groups) of ANOVA, the corresponding population parameter of
         is
Effect size                                                                                                                            123




     wherein μj denotes the population mean within the jth group of the total K groups, and σ the equivalent population
     standard deviations within each groups. SS is the sum of squares manipulation in ANOVA. An unbiased estimator
     for ANOVA would be based on Omega squared, which estimates for the population.

     Omega-squared, ω2
     A more unbiased estimator of the variance explained in the population is omega-squared[9][10][11]



     This form of the formula is limited to between-subjects analysis with equal sample sizes in all cells,[11]. Since it is
     less biased (although still biased), ω2 is preferable to Cohen's ƒ2; however, it can be more inconvenient to calculate
     for complex analyses. A generalized form of the estimator has been published for between-subjects and
     within-subjects analysis, repeated measure, mixed design, and randomized block design experiments.[12] In addition,
     methods to calculate partial Omega2 for individual factors and combined factors in designs with up to three
     independent variables have been published.[12]


     Effect sizes based on means or distances between/among means
     A (population) effect size θ based on means usually considers the
     standardized mean difference between two populations[13]:78



     where μ1 is the mean for one population, μ2 is the mean for the other
     population, and σ is a standard deviation based on either or both
     populations.
     In the practical setting the population values are typically not known
     and must be estimated from sample statistics. The several versions of
     effect sizes based on means differ with respect to which statistics are
     used.
                                                                                    Plots of Gaussian densities illustrating various
     This form for the effect size resembles the computation for a t-test
                                                                                                 values of Cohen's d.
     statistic, with the critical difference that the t-test statistic includes a
     factor of          . This means that for a given effect size, the
     significance level increases with the sample size. Unlike the t-test statistic, the effect size aims to estimate a
     population parameter, so is not affected by the sample size.

     Cohen's d
     Cohen's d is defined as the difference between two means divided by a standard deviation for the data



     Cohen's d is frequently used in estimating sample sizes. A lower Cohen's d indicates a necessity of larger sample
     sizes, and vice versa, as can subsequently be determined together with the additional parameters of desired
     significance level and statistical power.[14]
     What precisely the standard deviation s is was not originally made explicit by Jacob Cohen because he defined it
     (using the symbol "σ") as "the standard deviation of either population (since they are assumed equal)".[7]:20 Other
     authors make the computation of the standard deviation more explicit with the following definition for a pooled
Effect size                                                                                                                     124


     standard deviation[15]:14 with two independent samples.




     This definition of "Cohen's d" is termed the maximum likelihood estimator by Hedges and Olkin,[13] and it is related
     to Hedges' g (see below) by a scaling[13]:82




     Glass's Δ
     In 1976 Gene V. Glass proposed an estimator of the effect size that uses only the standard deviation of the second
     group[13]:78



     The second group may be regarded as a control group, and Glass argued that if several treatments were compared to
     the control group it would be better to use just the standard deviation computed from the control group, so that effect
     sizes would not differ under equal means and different variances.
     Under an assumption of equal population variances a pooled estimate for σ is more precise.

     Hedges' g
     Hedges' g, suggested by Larry Hedges in 1981,[16] is like the other measures based on a standardized difference[13]:79



     but its pooled standard deviation    is computed slightly differently from Cohen's d




     As an estimator for the population effect size θ it is biased. However, this bias can be corrected for by multiplication
     with a factor




     Hedges and Olkin refer to this unbiased estimator      as d,[13] but it is not the same as Cohen's d. The exact form for
     the correction factor J() involves the gamma function[13]:104
Effect size                                                                                                                       125


     Distribution of effect sizes based on means

     Provided that the data is Gaussian distributed a scaled Hedges' g,                                  , follows a noncentral

     t-distribution with the noncentrality parameter                               and (n1 + n2 − 2) degrees of freedom.

     Likewise, the scaled Glass' Δ is distributed with n2 − 1 degrees of freedom.
     From the distribution it is possible to compute the expectation and variance of the effect sizes.
     In some cases large sample approximations for the variance are used. One suggestion for the variance of Hedges'
     unbiased estimator is[13]:86




     Effect sizes for associations among categorical variables




                                                    Phi (φ)    Cramér's Phi (φc)


     Commonly-used measures of association for the chi-squared test are the Phi coefficient and Cramér's V. Phi is
     related to the point-biserial correlation coefficient and Cohen's d and estimates the extent of the relationship between
     two variables (2 x 2).[17] Cramér's Phi may be used with variables having more than two levels.
     Phi can be computed by finding the square root of the chi-squared statistic divided by the sample size.
     Similarly, Cramér's phi is computed by taking the square root of the chi-squared statistic divided by the sample size
     and the length of the minimum dimension (k is the smaller of the number of rows r or columns c).
     φc is the intercorrelation of the two discrete variables[18] and may be computed for any value of r or c. However, as
     chi-squared values tend to increase with the number of cells, the greater the difference between r and c, the more
     likely φc will tend to 1 without strong evidence of a meaningful correlation.
     Cramér's phi may also be applied to 'goodness of fit' chi-squared models (i.e. those where c=1). In this case it
     functions as a measure of tendency towards a single outcome (i.e. out of k outcomes).


     Odds ratio
     The odds ratio (OR) is another useful effect size. It is appropriate when the research question focuses on the degree
     of association between two binary variables. For example, consider a study of spelling ability. In a control group,
     two students pass the class for every one who fails, so the odds of passing are two to one (or 2/1 = 2). In the
     treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The
     effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in
     the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. Odds ratio statistics are on a different
     scale than Cohen's d, so this '3' is not comparable to a Cohen's d of 3.


     Relative risk
     The relative risk (RR), also called risk ratio, is simply the risk (probability) of an event relative to some independent
     variable. This measure of effect size differs from the odds ratio in that it compares probabilities instead of odds, but
     asymptotically approaches the latter for small probabilities. Using the example above, the probabilities for those in
     the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively. The effect size can be
     computed the same as above, but using the probabilities instead. Therefore, the relative risk is 1.28. Since rather
     large probabilities of passing were used, there is a large difference between relative risk and odds ratio. Had failure
Effect size                                                                                                                      126


     (a smaller probability) been used as the event (rather than passing), the difference between the two measures of
     effect size would not be so great.
     While both measures are useful, they have different statistical uses. In medical research, the odds ratio is commonly
     used for case-control studies, as odds, but not probabilities, are usually estimated.[19] Relative risk is commonly used
     in randomized controlled trials and cohort studies.[20] When the incidence of outcomes are rare in the study
     population (generally interpreted to mean less than 10%), the odds ratio is considered a good estimate of the risk
     ratio. However, as outcomes become more common, the odds ratio and risk ratio diverge, with the odds ratio
     overestimating or underestimating the risk ratio when the estimates are greater than or less than 1, respectively.
     When estimates of the incidence of outcomes are available, methods exist to convert odds ratios to risk ratios.[21][22]


     Confidence intervals by means of noncentrality parameters
     Confidence intervals of standardized effect sizes, especially Cohen's          and      , rely on the calculation of
     confidence intervals of noncentrality parameters (ncp). A common approach to construct the confidence interval of
     ncp is to find the critical ncp values to fit the observed statistic to tail quantiles α/2 and (1 − α/2). The SAS and
     R-package MBESS provides functions to find critical values of ncp.

     t-test for mean difference of single group or two related groups
     For a single group, M denotes the sample mean, μ the population mean, SD the sample's standard deviation, σ the
     population's standard deviation, and n is the sample size of the group. The t value is used to test the hypothesis on the
     difference between the mean and a baseline μbaseline. Usually, μbaseline is zero. In the case of two related groups, the
     single group is constructed by the differences in pair of samples, while SD and σ denote the sample's and
     population's standard deviations of differences rather than within original two groups.




     and Cohen's



     is the point estimate of



     So,
Effect size                                                                                                                     127


     t-test for mean difference between two independent groups
     n1 or n2 are the respective sample sizes.




     wherein




     and Cohen's

                                is the point estimate of

     So,




     One-way ANOVA test for mean difference across multiple independent groups
     One-way ANOVA test applies noncentral F distribution. While with a given population standard deviation             , the
     same test question applies noncentral chi-squared distribution.




     For each j-th sample within i-th group Xi,j, denote




     While,




     So, both ncp(s) of F and      equate


     In case of                                   for K independent groups of same size, the total sample size is N := n·K.



     The t-test for a pair of independent groups is a special case of one-way ANOVA. Note that the noncentrality
     parameter        of F is not comparable to the noncentrality parameter     of the corresponding t. Actually,

                     , and             .
Effect size                                                                                                                                                128


     "Small", "medium", "large"
     Some fields using effect sizes apply words such as "small", "medium" and "large" to the size of the effect. Whether
     an effect size should be interpreted small, medium, or large depends on its substantial context and its operational
     definition. Cohen's conventional criteria small, medium, or big[7] are near ubiquitous across many fields. Power
     analysis or sample size planning requires an assumed population parameter of effect sizes. Many researchers adopt
     Cohen's standards as default alternative hypotheses. Russell Lenth criticized them as T-shirt effect sizes[23]
              This is an elaborate way to arrive at the same sample size that has been used in past social science
              studies of large, medium, and small size (respectively). The method uses a standardized effect size as the
              goal. Think about it: for a "medium" effect size, you'll choose the same n regardless of the accuracy or
              reliability of your instrument, or the narrowness or diversity of your subjects. Clearly, important
              considerations are being ignored here. "Medium" is definitely not the message!
     For Cohen's d an effect size of 0.2 to 0.3 might be a "small" effect, around 0.5 a "medium" effect and 0.8 to infinity,
     a "large" effect.[7]:25 (But note that the d might be larger than one)
     Cohen's text[7] anticipates Lenth's concerns:
              "The terms 'small,' 'medium,' and 'large' are relative, not only to each other, but to the area of behavioral
              science or even more particularly to the specific content and research method being employed in any
              given investigation....In the face of this relativity, there is a certain risk inherent in offering conventional
              operational definitions for these terms for use in power analysis in as diverse a field of inquiry as
              behavioral science. This risk is nevertheless accepted in the belief that more is to be gained than lost by
              supplying a common conventional frame of reference which is recommended for use only when no
              better basis for estimating the ES index is available." (p. 25)
     In an ideal world, researchers would interpret the substantive significance of their results by grounding them in a
     meaningful context or by quantifying their contribution to knowledge. Where this is problematic, Cohen's effect size
     criteria may serve as a last resort.[4]


     References
     [1] Kelley, Ken; Preacher, Kristopher J. (2012). "On Effect Size". Psychological Methods 17 (2): 137–152. doi:10.1037/a0028086.
     [2] Wilkinson, Leland; APA Task Force on Statistical Inference (1999). "Statistical methods in psychology journals: Guidelines and
         explanations". American Psychologist 54 (8): 594–604. doi:10.1037/0003-066X.54.8.594.
     [3] Nakagawa, Shinichi; Cuthill, Innes C (2007). "Effect size, confidence interval and statistical significance: a practical guide for biologists".
         Biological Reviews Cambridge Philosophical Society 82 (4): 591–605. doi:10.1111/j.1469-185X.2007.00027.x. PMID 17944619.
     [4] Ellis, Paul D. (2010). The Essential Guide to Effect Sizes: An Introduction to Statistical Power, Meta-Analysis and the Interpretation of
         Research Results. United Kingdom: Cambridge University Press.
     [5] Brand A, Bradley MT, Best LA, Stoica G (2008). "Accuracy of effect size estimates from published psychological research" (http:/ /
         mtbradley. com/ brandbradelybeststoicapdf. pdf). Perceptual and Motor Skills 106 (2): 645–649. doi:10.2466/PMS.106.2.645-649.
         PMID 18556917. .
     [6] Brand A, Bradley MT, Best LA, Stoica G (2011). "Multiple trials may yield exaggerated effect size estimates" (http:/ / www. ipsychexpts.
         com/ brand_et_al_(2011). pdf). The Journal of General Psychology 138 (1): 1–11. doi:10.1080/00221309.2010.520360. .
     [7] Jacob Cohen (1988). Statistical Power Analysis for the Behavioral Sciences (second ed.). Lawrence Erlbaum Associates.
     [8] Cohen, J (1992). "A power primer". Psychological Bulletin 112 (1): 155–159. doi:10.1037/0033-2909.112.1.155. PMID 19565683.
     [9] Bortz, 1999, p. 269f.;
     [10] Bühner & Ziegler (2009, p. 413f)
     [11] Tabachnick & Fidell (2007, p. 55)
     [12] Olejnik, S. & Algina, J. 2003. Generalized Eta and Omega Squared Statistics: Measures of Effect Size for Some Common Research Designs
         Psychological Methods. 8:(4)434-447. http:/ / cps. nova. edu/ marker/ olejnik2003. pdf
     [13] Larry V. Hedges & Ingram Olkin (1985). Statistical Methods for Meta-Analysis. Orlando: Academic Press. ISBN 0-12-336380-2.
     [14] Chapter 13 (http:/ / davidakenny. net/ statbook/ chapter_13. pdf), page 215, in: Kenny, David A. (1987). Statistics for the social and
         behavioral sciences. Boston: Little, Brown. ISBN 0-316-48915-8.
     [15] Joachim Hartung, Guido Knapp & Bimal K. Sinha (2008). Statistical Meta-Analysis with Application. Hoboken, New Jersey: Wiley.
Effect size                                                                                                                                                129

     [16] Larry V. Hedges (1981). "Distribution theory for Glass's estimator of effect size and related estimators". Journal of Educational Statistics 6
         (2): 107–128. doi:10.3102/10769986006002107.
     [17] Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect-size indices: Problems with a commonly
         recommended formula. (http:/ / www. eric. ed. gov/ ERICWebPortal/ custom/ portlets/ recordDetails/ detailmini. jsp?_nfpb=true& _&
         ERICExtSearch_SearchValue_0=ED433353& ERICExtSearch_SearchType_0=no& accno=ED433353) Paper presented at the annual meeting
         of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353)
     [18] Sheskin, David J. (1997). Handbook of Parametric and Nonparametric Statistical Procedures. Boca Raton, Fl: CRC Press.
     [19] Deeks J (1998). "When can odds ratios mislead? : Odds ratios should be used only in case-control studies and logistic regression analyses".
         BMJ 317 (7166): 1155–6. PMC 1114127. PMID 9784470.
     [20] Medical University of South Carolina. Odds ratio versus relative risk (http:/ / www. musc. edu/ dc/ icrebm/ oddsratio. html). Accessed on:
         September 8, 2005.
     [21] Zhang, J.; Yu, K. (1998). "What's the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes". JAMA:
         the Journal of the American Medical Association 280 (19): 1690–1691. doi:10.1001/jama.280.19.1690. PMID 9832001.
     [22] Greenland, S. (2004). "Model-based Estimation of Relative Risks and Other Epidemiologic Measures in Studies of Common Outcomes and
         in Case-Control Studies". American Journal of Epidemiology 160 (4): 301–305. doi:10.1093/aje/kwh221. PMID 15286014.
     [23] Russell V. Lenth. "Java applets for power and sample size" (http:/ / www. stat. uiowa. edu/ ~rlenth/ Power/ ). Division of Mathematical
         Sciences, the College of Liberal Arts or The University of Iowa. . Retrieved 2008-10-08.


     Further reading
     • Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect-size indices:
       Problems with a commonly recommended formula. Paper presented at the annual meeting of the Florida
       Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353) (http://
       www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED433353)
     • Bonett, D.G. (2008). Confidence intervals for standardized linear contrasts of means, Psychological Methods, 13,
       99-109.
     • Bonett, D.G. (2009). Estimating standardized linear contrasts of means with desired precision, Psychological
       Methods"", 14, 1-5.
     • Cumming, G. and Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals
       that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 530–572.
     • Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation.
       Journal of Statistical Software, 20(8), 1-24. (http://www.jstatsoft.org/v20/i08/paper)
     • Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Sage: Thousand Oaks, CA.


     External links
     Online Applications
     • Free Effect Size Calculator for Multiple Regression (http://www.danielsoper.com/statcalc/calc05.aspx)
     • Free Effect Size Calculator for Hierarchical Multiple Regression (http://www.danielsoper.com/statcalc/calc13.
       aspx)
     • Copylefted Effect Size Confidence Interval R Code with RWeb service for t-test, ANOVA, regression, and
       RMSEA (http://xiaoxu.lxxm.com/ncp)
     Software
     • compute.es: Compute Effect Sizes (http://cran.r-project.org/web/packages/compute.es/index.html) (R
       package)
     • MBESS (http://cran.r-project.org/web/packages/MBESS/index.html) - One of R's packages providing
       confidence intervals of effect sizes based non-central parameters
     • MIX 2.0 (http://www.meta-analysis-made-easy.com) Software for professional meta-analysis in Excel. Many
       effect sizes available.
     • Effect Size Calculators (http://myweb.polyu.edu.hk/~mspaul/calculator/calculator.html) Calculate d and r
       from a variety of statistics.
Effect size                                                                                                                                   130


     • Free Effect Size Generator (http://www.clintools.com/victims/resources/software/effectsize/
       effect_size_generator.html) - PC & Mac Software
     • Free GPower Software (http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/) - PC & Mac Software
     • ES-Calc: a free add-on for Effect Size Calculation in ViSta 'The Visual Statistics System' (http://www.mdp.
       edu.ar/psicologia/vista/vista.htm). Computes Cohen's d, Glass's Delta, Hedges' g, CLES, Non-Parametric
       Cliff's Delta, d-to-r Conversion, etc.
     Further Explanations
     • Effect Size (ES) (http://www.uccs.edu/~faculty/lbecker/es.htm)
     • EffectSizeFAQ.com (http://effectsizefaq.com/)
     • Measuring Effect Size (http://davidmlane.com/hyperstat/effect_size.html)
     • Effect size for two independent groups (http://web.uccs.edu/lbecker/Psy590/es.htm#II.independent)
     • Effect size for two dependent groups (http://web.uccs.edu/lbecker/Psy590/es.htm#III.Effect size measures
       for two dependent)
     • Computing and Interpreting Effect size Measures with ViSta (http://www.tqmp.org/doc/vol5-1/p25-34.pdf)



     Time series
     In statistics, signal processing, pattern recognition,
     econometrics,     mathematical       finance,    weather
     forecasting,           earthquake             prediction,
     electroencephalography, control engineering and
     communications engineering a time series is a
     sequence of data points, measured typically at
     successive points in time spaced at uniform time
     intervals. Examples of time series are the daily closing
     value of the Dow Jones index and the annual flow
     volume of the Nile River at Aswan. Time series are
     very frequently plotted via line charts.

     Time series analysis comprises methods for analyzing             Time series: random data plus trend, with best-fit line and different
     time series data in order to extract meaningful statistics                                  smoothings
     and other characteristics of the data. Time series
     forecasting is the use of a model to predict future values based on previously observed values. While regression
     analysis is often employed in such a way as to test theories that the current value of one time series affects the
     current value of another time series, this type of analysis of time series is not called "time series analysis".

     Time series data have a natural temporal ordering. This makes time series analysis distinct from other common data
     analysis problems, in which there is no natural ordering of the observations (e.g. explaining people's wages by
     reference to their respective education levels, where the individuals' data could be entered in any order). Time series
     analysis is also distinct from spatial data analysis where the observations typically relate to geographical locations
     (e.g. accounting for house prices by the location as well as the intrinsic characteristics of the houses). A stochastic
     model for a time series will generally reflect the fact that observations close together in time will be more closely
     related than observations further apart. In addition, time series models will often make use of the natural one-way
     ordering of time so that values for a given period will be expressed as deriving in some way from past values, rather
     than from future values (see time reversibility.)
     Methods for time series analyses may be divided into two classes: frequency-domain methods and time-domain
     methods. The former include spectral analysis and recently wavelet analysis; the latter include auto-correlation and
Time series                                                                                                                  131


    cross-correlation analysis.
    Additionally time series analysis techniques may be divided into parametric and non-parametric methods. The
    parametric approaches assume that the underlying stationary stochastic process has a certain structure which can be
    described using a small number of parameters (for example, using an autoregressive or moving average model). In
    these approaches, the task is to estimate the parameters of the model that describes the stochastic process. By
    contrast, non-parametric approaches explicitly estimate the covariance or the spectrum of the process without
    assuming that the process has any particular structure.
    Additionally methods of time series analysis may be divided into linear and non-linear, univariate and multivariate.
    Time series analysis can be applied to:
    • real-valued, continuous data
    • discrete numeric data
    • discrete symbolic data (i.e. sequences of characters, such as letters and words in English language[1]).


    Analysis
    There are several types of motivation and data analysis available for time series which are appropriate for different
    purposes.


    Motivation
    In the context of statistics, econometrics, quantitative finance, seismology, meteorology, and geophysics the primary
    goal of time series analysis is forecasting. In the context of signal processing, control engineering and
    communication engineering it is used for signal detection and estimation, while in the context of data mining, pattern
    recognition and machine learning time series analysis can be used for clustering, classification, query by content,
    anomaly detection as well as forecasting.


    Exploratory analysis
    The clearest way to examine a regular time series manually is with a
    line chart such as the one shown for tuberculosis in the United States,
    made with a spreadsheet program. The number of cases was
    standardized to a rate per 100,000 and the percent change per year in
    this rate was calculated. The nearly steadily dropping line shows that
    the TB incidence was decreasing in most years, but the percent change
    in this rate varied by as much as +/- 10%, with 'surges' in 1975 and
    around the early 1990s. The use of both vertical axes allows the
    comparison of two time series in one graphic. Other techniques                   Tuberculosis incidence US 1953-2009

    include:

    • Autocorrelation analysis to examine serial dependence
    • Spectral analysis to examine cyclic behaviour which need not be related to seasonality. For example, sun spot
      activity varies over 11 year cycles.[2][3] Other common examples include celestial phenomena, weather patterns,
      neural activity, commodity prices, and economic activity.
    • Separation into components representing trend, seasonality, slow and fast variation, and cyclical irregularity: see
      trend estimation and decomposition of time series
Time series                                                                                                                    132


    Prediction and forecasting
    • Fully formed statistical models for stochastic simulation purposes, so as to generate alternative versions of the
      time series, representing what might happen over non-specific time-periods in the future
    • Simple or fully formed statistical models to describe the likely outcome of the time series in the immediate future,
      given knowledge of the most recent outcomes (forecasting).
    • Forecasting on time series is usually done using automated statistical software packages and programming
      languages, such as R (programming language), S (programming language), SAS (software), SPSS, Minitab and
      many others.


    Classification
    • Assigning time series pattern to a specific category, for example identify a word based on series of hand
      movements in sign language
    See main article: Statistical classification


    Regression analysis
    • Estimating future value of a signal based on its previous behavior, e.g. predict the price of AAPL stock based on
      its previous price movements for that hour, day or month, or predict position of Apollo 11 spacecraft at a certain
      future moment based on its current trajectory (i.e. time series of its previous locations).[4]
    • Regression analysis is usually based on statistical interpretation of time series properties in time domain,
      pioneered by statisticians George Box and Gwilym Jenkins in the 50s: see Box–Jenkins


    Signal Estimation
    • This approach is based on harmonic analysis and filtering of signals in the frequency domain using the Fourier
      transform, and spectral density estimation, the development of which was significantly accelerated during World
      War II by mathematician Norbert Wiener, electrical engineers Rudolf E. Kálmán, Dennis Gabor and others for
      filtering signals from noise and predicting signal values at a certain point in time. See Kalman Filter, Estimation
      theory and Digital Signal Processing


    Models
    Models for time series data can have many forms and represent different stochastic processes. When modeling
    variations in the level of a process, three broad classes of practical importance are the autoregressive (AR) models,
    the integrated (I) models, and the moving average (MA) models. These three classes depend linearly[5] on previous
    data points. Combinations of these ideas produce autoregressive moving average (ARMA) and autoregressive
    integrated moving average (ARIMA) models. The autoregressive fractionally integrated moving average (ARFIMA)
    model generalizes the former three. Extensions of these classes to deal with vector-valued data are available under
    the heading of multivariate time-series models and sometimes the preceding acronyms are extended by including an
    initial "V" for "vector", as in VAR for vector autoregression. An additional set of extensions of these models is
    available for use where the observed time-series is driven by some "forcing" time-series (which may not have a
    causal effect on the observed series): the distinction from the multivariate case is that the forcing series may be
    deterministic or under the experimenter's control. For these models, the acronyms are extended with a final "X" for
    "exogenous".
    Non-linear dependence of the level of a series on previous data points is of interest, partly because of the possibility
    of producing a chaotic time series. However, more importantly, empirical investigations can indicate the advantage
    of using predictions derived from non-linear models, over those from linear models, as for example in nonlinear
    autoregressive exogenous models.
Time series                                                                                                                  133


    Among other types of non-linear time series models, there are models to represent the changes of variance over time
    (heteroskedasticity). These models represent autoregressive conditional heteroskedasticity (ARCH) and the
    collection comprises a wide variety of representation (GARCH, TARCH, EGARCH, FIGARCH, CGARCH, etc.).
    Here changes in variability are related to, or predicted by, recent past values of the observed series. This is in
    contrast to other possible representations of locally varying variability, where the variability might be modelled as
    being driven by a separate time-varying process, as in a doubly stochastic model.
    In recent work on model-free analyses, wavelet transform based methods (for example locally stationary wavelets
    and wavelet decomposed neural networks) have gained favor. Multiscale (often referred to as multiresolution)
    techniques decompose a given time series, attempting to illustrate time dependence at multiple scales. See also
    Markov switching multifractal (MSMF) techniques for modeling volatility evolution.


    Notation
    A number of different notations are in use for time-series analysis. A common notation specifying a time series X
    that is indexed by the natural numbers is written
           X = {X1, X2, ...}.
    Another common notation is
           Y = {Yt: t ∈ T},
    where T is the index set.


    Conditions
    There are two sets of conditions under which much of the theory is built:
    • Stationary process
    • Ergodic process
    However, ideas of stationarity must be expanded to consider two important ideas: strict stationarity and second-order
    stationarity. Both models and applications can be developed under each of these conditions, although the models in
    the latter case might be considered as only partly specified.
    In addition, time-series analysis can be applied where the series are seasonally stationary or non-stationary.
    Situations where the amplitudes of frequency components change with time can be dealt with in time-frequency
    analysis which makes use of a time–frequency representation of a time-series or signal.[6]


    Models
    The general representation of an autoregressive model, well known as AR(p), is


    where the term εt is the source of randomness and is called white noise. It is assumed to have the following
    characteristics:
       •
       •
       •
    With these assumptions, the process is specified up to second-order moments and, subject to conditions on the
    coefficients, may be second-order stationary.
    If the noise also has a normal distribution, it is called normal or Gaussian white noise. In this case, the AR process
    may be strictly stationary, again subject to conditions on the coefficients.
    Tools for investigating time-series data include:
Time series                                                                                                               134


    • Consideration of the autocorrelation function and the spectral density function (also cross-correlation functions
      and cross-spectral density functions)
    • Scaled cross- and auto-correlation functions[7]
    • Performing a Fourier transform to investigate the series in the frequency domain
    • Use of a filter to remove unwanted noise
    • Principal components analysis (or empirical orthogonal function analysis)
    • Singular spectrum analysis
    • "Structural" models:
       • General State Space Models
       • Unobserved Components Models
    • Machine Learning
       • Artificial neural networks
       • Support Vector Machine
       • Fuzzy Logic
    • Hidden Markov model
    • Control chart
       • Shewhart individuals control chart
       • CUSUM chart
       • EWMA chart
    • Detrended fluctuation analysis
    • Dynamic time warping
    • Dynamic Bayesian network
    • Time-frequency analysis techniques:
       •   Fast Fourier Transform
       •   Continuous wavelet transform
       •   Short-time Fourier transform
       •   Chirplet transform
       •   Fractional Fourier transform
    • Chaotic analysis
       •   Correlation dimension
       •   Recurrence plots
       •   Recurrence quantification analysis
       •   Lyapunov exponents
       •   Entropy encoding
Time series                                                                                                      135


    Measures
    Time series metrics or features that can be used for time series classification or regression analysis[8]:
    • Univariate linear measures
       •   Moment (mathematics)
       •   Spectral band power
       •   Spectral edge frequency
       •   Accumulated Energy (signal processing)
       •   Characteristics of the autocorrelation function
       •   Hjorth parameters
       •   FFT parameters
       •   Autoregressive model parameters
       •   Mann–Kendall test
    • Univariate non-linear measures
       • Measures based on the correlation sum
       • Correlation dimension
       • Correlation integral
       •   Correlation density
       •   Correlation entropy
       •   Approximate Entropy[9]
       •   Sample entropy
       •   Fourier entropy
       •   Wavelet entropy
       •   Rényi entropy
       •   Higher-order methods
       •   Marginal predictability
       •   Dynamical similarity index
       •   State space dissimilarity measures
       •   Lyapunov exponent
       •   Permutation methods
       •   Local flow
    • Other univariate measures
       •   Algorithmic complexity
       •   Kolmogorov complexity estimates
       •   Hidden Markov Model states
       •   Surrogate time series and surrogate correction
       •   Loss of recurrence (degree of non-stationarity)
    • Bivariate linear measures
       • Maximum linear cross-correlation
       • Linear Coherence (signal processing)
    • Bivariate non-linear measures
       • Non-linear interdependence
       • Dynamical Entrainment (physics)
       • Measures for Phase synchronization
    • Similarity measures[10]:
       • Dynamic Time Warping
Time series                                                                                                                                         136


       •   Hidden Markov Models
       •   Edit distance
       •   Total correlation
       •   Newey–West estimator
       •   Prais-Winsten transformation
       •   Data as Vectors in a Metrizable Space
         • Minkowski distance
         • Mahalanobis distance
       • Data as Time Series with Envelopes
         • Global Standard Deviation
         • Local Standard Deviation
         • Windowed Standard Deviation
       • Data Interpreted as Stochastic Series
         • Pearson product-moment correlation coefficient
         • Spearman's rank correlation coefficient
       • Data Interpreted as a Probability Distribution Function
           • Kolmogorov-Smirnov test
           • Cramér-von Mises criterion


    References
    [1] Lin, Jessica and Keogh, Eamonn and Lonardi, Stefano and Chiu, Bill. A symbolic representation of time series, with implications for
        streaming algorithms. Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, 2003.
        url: http:/ / doi. acm. org/ 10. 1145/ 882082. 882086
    [2] Bloomfield, P. (1976). Fourier analysis of time series: An introduction. New York: Wiley.
    [3] Shumway, R. H. (1988). Applied statistical time series analysis. Englewood Cliffs, NJ: Prentice Hall.
    [4] Lawson, Charles L., Hanson, Richard, J. (1987). Solving Least Squares Problems. Society for Industrial and Applied Mathematics, 1987.
    [5] Gershenfeld, N. (1999). The nature of mathematical modeling. p.205-08
    [6] Boashash, B. (ed.), (2003) Time-Frequency Signal Analysis and Processing: A Comprehensive Reference, Elsevier Science, Oxford, 2003
        ISBN ISBN 0-08-044335-4
    [7] Nikolić D, Muresan RC, Feng W, Singer W (2012) Scaled correlation analysis: a better way to compute a cross-correlogram. European
        Journal of Neuroscience, pp. 1–21, doi:10.1111/j.1460-9568.2011.07987.x http:/ / www. danko-nikolic. com/ wp-content/ uploads/ 2012/ 03/
        Scaled-correlation-analysis. pdf
    [8] Mormann, Florian and Andrzejak, Ralph G. and Elger, Christian E. and Lehnertz, Klaus. 'Seizure prediction: the long and winding road.
        Brain, 2007,130 (2): 314-33.url : http:/ / brain. oxfordjournals. org/ content/ 130/ 2/ 314. abstract
    [9] Land, Bruce and Elias, Damian. Measuring the "Complexity" of a time series. URL: http:/ / www. nbb. cornell. edu/ neurobio/ land/
        PROJECTS/ Complexity/
    [10] Ropella, G.E.P.; Nag, D.A.; Hunt, C.A.; , "Similarity measures for automated comparison of in silico and in vitro experimental results,"
        Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE , vol.3, no., pp.
        2933- 2936 Vol.3, 17-21 Sept. 2003 doi: 10.1109/IEMBS.2003.1280532 URL: http:/ / ieeexplore. ieee. org/ stamp/ stamp. jsp?tp=&
        arnumber=1280532& isnumber=28615
Time series                                                                                                          137


    Further reading
    • Bloomfield, P. (1976). Fourier analysis of time series: An introduction. New York: Wiley.
    • Box, George; Jenkins, Gwilym (1976), Time series analysis: forecasting and control, rev. ed., Oakland,
      California: Holden-Day
    • Brillinger, D. R. (1975). Time series: Data analysis and theory. New York: Holt, Rinehart. & Winston.
    • Brigham, E. O. (1974). The fast Fourier transform. Englewood Cliffs, NJ: Prentice-Hall.
    • Elliott, D. F., & Rao, K. R. (1982). Fast transforms: Algorithms, analyses, applications. New York: Academic
      Press.
    • Gershenfeld, Neil (2000), The nature of mathematical modeling, Cambridge: Cambridge Univ. Press,
      ISBN 978-0-521-57095-4, OCLC 174825352
    • Hamilton, James (1994), Time Series Analysis, Princeton: Princeton Univ. Press, ISBN 0-691-04289-6
    • Jenkins, G. M., & Watts, D. G. (1968). Spectral analysis and its applications. San Francisco: Holden-Day.
    • Priestley, M. B. (1981). Spectral Analysis and Time Series. London: Academic Press. ISBN 978-0-12-564901-8
    • Shasha, D. (2004), High Performance Discovery in Time Series, Berlin: Springer, ISBN 0-387-00857-8
    • Shumway, R. H. (1988). Applied statistical time series analysis. Englewood Cliffs, NJ: Prentice Hall.
    • Wiener, N.(1964). Extrapolation, Interpolation, and Smoothing of Stationary Time Series.The MIT Press.
    • Wei, W. W. (1989). Time series analysis: Univariate and multivariate methods. New York: Addison-Wesley.
    • Weigend, A. S., and N. A. Gershenfeld (Eds.) (1994) Time Series Prediction: Forecasting the Future and
      Understanding the Past. Proceedings of the NATO Advanced Research Workshop on Comparative Time Series
      Analysis (Santa Fe, May 1992) MA: Addison-Wesley.
    • Durbin J., and Koopman S.J. (2001) Time Series Analysis by State Space Methods. Oxford University Press.


    External links
    • A First Course on Time Series Analysis (http://statistik.mathematik.uni-wuerzburg.de/timeseries/) - an open
      source book on time series analysis with SAS
    • Introduction to Time series Analysis (Engineering Statistics Handbook) (http://www.itl.nist.gov/div898/
      handbook/pmc/section4/pmc4.htm) - a practical guide to Time series analysis
    • MATLAB Toolkit for Computation of Multiple Measures on Time Series Data Bases (http://www.jstatsoft.org/
      v33/i05/paper)
    • A Matlab tutorial on power spectra, wavelet analysis, and coherence (http://www.nbtwiki.net/doku.
      php?id=tutorial:power_spectra_wavelet_analysis_and_coherence) on website with many other tutorials.
Panel data                                                                                                                     138



    Panel data
    In statistics and econometrics, the term panel data refers to multi-dimensional data frequently involving
    measurements over time. Panel data contain observations on multiple phenomena observed over multiple time
    periods for the same firms or individuals. In biostatistics, the term longitudinal data is often used instead,[1][2]
    wherein a subject or cluster constitutes a panel member or individual in a longitudinal study. (Somewhat
    confusingly, the Stata manual entry on the "xtset" command refers to a panel member simply as a "panel".)
    Time series and cross-sectional data are special cases of panel data that are in one dimension only (one panel
    member or individual for the former, one time point for the latter).


    Example
                                       balanced panel:                    unbalanced panel:




    In the example above, two data sets with a two-dimensional panel structure are shown, although the second data set
    might be a three-dimensional structure since it has three people. Individual characteristics (income, age, sex. educ)
    are collected for different persons and different years. In the left data set two persons (1, 2) are observed over three
    years (2003, 2004, 2005). Because each person is observed every year, the left-hand data set is called a balanced
    panel, whereas the data set on the right hand is called an unbalanced panel, since Person 1 is not observed in year
    2005 and Person 3 is not observed in 2003 or 2005.


    Analysis of panel data
    A panel has the form


    where     is the individual dimension and    is the time dimension. A general panel data regression model is written
    as                              Different assumptions can be made on the precise structure of this general model.
    Two important models are the fixed effects model and the random effects model. The fixed effects model is denoted
    as



         are individual-specific, time-invariant effects (for example in a panel of countries this could include geography,
    climate etc.) and because we assume they are fixed over time, this is called the fixed-effects model. The random
    effects model assumes in addition that


    and


    that is, the two error components are independent from each other.
Panel data                                                                                                                                       139


    Data sets which have a panel design
    •   German Socio-Economic Panel (SOEP)
    •   Household, Income and Labour Dynamics in Australia Survey (HILDA)
    •   British Household Panel Survey (BHPS)
    •   Survey of Family Income and Employment (SoFIE)
    •   Survey of Income and Program Participation (SIPP)
    •   Lifelong Labour Market Database (LLMDB)
    •   Panel Study of Income Dynamics (PSID)
    •   Korean Labor and Income Panel Study (KLIPS)
    •   Chinese Family Panel Studies (CFPS)
    •   German Family Panel (pairfam)
    •   National Longitudinal Surveys (NLSY)


    Notes
    [1] Peter J. Diggle, Patrick Heagerty, Kung-Yee Liang, and Scott L. Zeger, 2002. Analysis of Longitudinal Data. 2nd ed., Oxford University
        Press, p. 2.
    [2] Garrett M. Fitzmaurice, Nan M. Laird, and James H. Ware, 2004. Applied Longitudinal Analysis. John Wiley & Sons, Inc., p. 2.



    References
    Baltagi, Badi H., 2008. Econometric Analysis of Panel Data, John Wiley & Sons, Ltd.
    Hsiao, Cheng, 2003. Analysis of Panel Data, Cambridge University Press.
    Davies, A. and Lahiri, K., 2000. "Re-examining the Rational Expectations Hypothesis Using Panel Data on
    Multi-Period Forecasts," Analysis of Panels and Limited Dependent Variable Models, Cambridge University Press.
    Davies, A. and Lahiri, K., 1995. "A New Framework for Testing Rationality and Measuring Aggregate Shocks
    Using Panel Data," Journal of Econometrics 68: 205-227.
    Frees, E., 2004. Longitudinal and Panel Data, Cambridge University Press.


    External links
    • PSID (http://psidonline.isr.umich.edu/)
    • KLIPS (http://www.kli.re.kr/klips)
    • pairfam (http://www.pairfam.uni-bremen.de/en/study.html)
Truncated regression model                                                                                                    140



    Truncated regression model
    Truncated regression models arise in many applications of statistics, for example in econometrics, in cases where
    observations with values in the outcome variable below or above certain thresholds systematically excluded from the
    sample. Therefore, whole observations are missing, so that neither the dependent nor the independent variable is
    known.
    Truncated regression models are often confused with censored regression models where only the value of the
    dependent variable is clustered at a lower threshold, an upper threshold, or both, while the value for independent
    variables is available.


    Example
    One example of truncated samples come from historical military height records. Many armies imposed a minimum
    height requirement (MHR) on soldiers. This implies that men shorter than the MHR are not included in the sample.
    This implies that samples drawn from such records are perforce deficient i.e., incomplete, inasmuch as a substantial
    portion of the underlying population's height distribution is unavailable for analysis. Consequently, without proper
    statistical correction, any results obtained from such deficient samples, such as means, correlations, or regression
    coefficients are wrong (biased). In such a case truncated regression has the considerable advantage of immediately
    providing consistent and unbiased estimates of the coefficients of the independent variables, as well as their standard
    errors, thereby allowing for further statistical inference, such as the calculation of the t-values of the estimates.


    References
    • A'Hearn, Brian (2004) "A Restricted Maximum Likelihood Estimator for Truncated Height Samples." Economics
      and Human Biology 2 (1), 5–20.
    • Komlos, John (2004) "How to (and How Not to) Analyze Deficient Height Samples: an Introduction." Historical
      Methods, 37, No. 4, Fall, 160–173.
    • Wolynetz, M.S. (1979) "Maximum Likelihood estimation in a Linear model from Confined and Censored Normal
      Data". Journal of the Royal Statistical Society (Series C), 28(2), 195–206
Censored regression model                                                                                                  141



    Censored regression model
    Censored regression models commonly arise in econometrics in cases where the variable of interest is only
    observable under certain conditions. A common example is labor supply. Data are frequently available on the hours
    worked by employees, and a labor supply model estimates the relationship between hours worked and characteristics
    of employees such as age, education and family status. However, such estimates undertaken using linear regression
    will be biased by the fact that for people who are unemployed it is not possible to observe the number of hours they
    would have worked had they had employment. Still we know age, education and family status for those
    observations.
    A model commonly used to deal with censored data is the Tobit model, including variations such as the Tobit Type
    II, Type III, and Type IV models.
    These and other censored regression models are often confused with truncated regression models. Truncated
    regression models are used for data where whole observations are missing so that the values for the dependent and
    the independent variables are unknown. Censored regression models are used for data where only the value for the
    dependent variable (hours of work in the example above) is unknown while the values of the independent variables
    (age, education, family status) are still available.
    Censored regression models are usually estimated using maximum likelihood estimation. The general validity of this
    approach has been shown by Schnedler (2005) who also provides a method to find the likelihood for a broad class of
    applications.


    References
    • Schnedler, Wendelin (2005). "Likelihood estimation for censored random vectors". Econometric Reviews 24
      (2),195–217.
Poisson regression                                                                                                              142



    Poisson regression
    In statistics, Poisson regression is a form of regression analysis used to model count data and contingency tables.
    Poisson regression assumes the response variable Y has a Poisson distribution, and assumes the logarithm of its
    expected value can be modeled by a linear combination of unknown parameters. A Poisson regression model is
    sometimes known as a log-linear model, especially when used to model contingency tables.
    Poisson regression models are generalized linear models with the logarithm as the (canonical) link function, and the
    Poisson distribution function.


    Regression models
    If          is a vector of independent variables, then the model takes the form


    where              and        . Sometimes this is written more compactly as


    where x is now an (n + 1)-dimensional vector consisting of n independent variables concatenated to some constant,
    usually 1. Here θ is simply a concatenated to b.
    Thus, when given a Poisson regression model θ and an input vector        , the predicted mean of the associated Poisson
    distribution is given by


    If Yi are independent observations with corresponding values xi of the predictor variable, then θ can be estimated by
    maximum likelihood. The maximum-likelihood estimates lack a closed-form expression and must be found by
    numerical methods. The probability surface for maximum-likelihood Poisson regression is always convex, making
    Newton–Raphson or other gradient-based methods appropriate estimation techniques.


    Maximum likelihood-based parameter estimation
    Given a set of parameters θ and an input vector x, the mean of the predicted Poisson distribution, as stated above, is
    given by

                             ,
    and thus, the Poisson distribution's probability mass function is given by




    Now suppose we are given a data set consisting of m vectors                                     , along with a set of m
    values                       . Then, for a given set of parameters θ, the probability of attaining this particular set of
    data is given by



    By the method of maximum likelihood, we wish to find the set of parameters θ that makes this probability as large as
    possible. To do this, the equation is first rewritten as a likelihood function in terms of θ:

                                                 .
Poisson regression                                                                                                                 143


    Note that the expression on the right hand side has not actually changed. A formula in this form is typically difficult
    to work with; instead, one uses the log-likelihood:

                                                                                       .

    Notice that the parameters θ only appear in the first two terms of each term in the summation. Therefore, given that
    we are only interested in finding the best value for θ we may drop the yi! and simply write

                                                    .


    To find a maximum, we need to solve an equation                           which has no closed-form solution. However,

    the negative log-likelhood,                  , is a convex function, and so standard convex optimization techniques
    such as gradient descent can be applied to find the optimal value of θ.

    Poisson regression in practice
    Poisson regression may be appropriate when the dependent variable is a count, for instance of events such as the
    arrival of a telephone call at a call centre. The events must be independent in the sense that the arrival of one call will
    not make another more or less likely, but the probability per unit time of events is understood to be related to
    covariates such as time of day.


    "Exposure" and offset
    Poisson regression may also be appropriate for rate data, where the rate is a count of events occurring to a particular
    unit of observation, divided by some measure of that unit's exposure. For example, biologists may count the number
    of tree species in a forest, and the rate would be the number of species per square kilometre. Demographers may
    model death rates in geographic areas as the count of deaths divided by person−years. More generally, event rates
    can be calculated as events per unit time, which allows the observation window to vary for each unit. In these
    examples, exposure is respectively unit area, person−years and unit time. In Poisson regression this is handled as an
    offset, where the exposure variable enters on the right-hand side of the equation, but with a parameter estimate (for
    log(exposure)) constrained to 1.


    which implies




    Offset in the case of a GLM in R can be achieved using the offset() function:

    glm.fit <- glm(y ~ offset(log(exposure)) + x, family=poisson(link=log) )


    Overdispersion
    A characteristic of the Poisson distribution is that its mean is equal to its variance. In certain circumstances, it will be
    found that the observed variance is greater than the mean; this is known as overdispersion and indicates that the
    model is not appropriate. A common reason is the omission of relevant explanatory variables, or dependent
    observations. Under some circumstances, the problem of overdispersion can be solved by using a negative binomial
    distribution instead.[1][2]
    Another common problem with Poisson regression is excess zeros: if there are two processes at work, one
    determining whether there are zero events or any events, and a Poisson process determining how many events there
    are, there will be more zeros than a Poisson regression would predict. An example would be the distribution of
Poisson regression                                                                                                            144


    cigarettes smoked in an hour by members of a group where some individuals are non-smokers.
    Other generalized linear models such as the negative binomial model may function better in these cases.


    Use in survival analysis
    Poisson regression creates proportional hazards models, one class of survival analysis: see proportional hazards
    models for descriptions of Cox models.


    Tests of over-dispersion
    One method for testing for over dispersion in the data is to regress a variable zi against the predicted values of t
    estimated from the Poisson regression.[3] This test has three steps.
    1. Estimate a Poisson regression of yi on xi and generate the predicted values ( ti )
    2. Calculate the zi variable




    3. Regress zi against ti with ordinary least squares. In symbols


    where a is a constant and ei is a random variable with an expectation of zero.
    The null hypothesis being tested here is that the data are Poisson-distributed: in this case a = 0.


    Extensions

    Regularized Poisson regression
    When estimating the parameters for Poisson regression, one typically tries to find values for θ that maximize the
    likelihood of an expression of the form




    where m is the number of examples in the data set, and                  is the probability mass function of the Poisson
    distribution with the mean set to            . Regularization can be added to this optimization problem by instead
    maximizing

                                             ,

    for some positive constant     . This technique, similar to ridge regression, can reduce overfitting.


    Implementations
    Some statistics packages include implementations of Poisson regression.
    • MATLAB Statistics Toolbox: Poisson regression can be performed using the "glmfit" and "glmval" functions.[4]
    • Microsoft Excel: Excel is not capable of doing Poisson regression by default. One of the Excel Add-ins for
      Poisson regression is XPost [5]
    • R: The function for fitting a generalized linear model in R is glm(), and can be used for Poisson Regression
    • SAS: Poisson regression in SAS is done by using GENMOD
    • SPSS: In SPSS, Poisson regression is done by using the GENLIN command
    • Stata: Stata has a procedure for Poisson regression named "poisson"
    • mPlus: mPlus allows for Poisson regression using the command COUNT IS when specifying the data
Poisson regression                                                                                                                                145


    References
    • Cameron, A.C. and P.K. Trivedi (1998). Regression analysis of count data, Cambridge University Press. ISBN
      0-521-63201-3
    • Christensen, Ronald (1997). Log-linear models and logistic regression. Springer Texts in Statistics (Second ed.).
      New York: Springer-Verlag. pp. xvi+483. ISBN 0-387-98247-7. MR1633357.
    • Hilbe, J. M. (2007). Negative Binomial Regression, Cambridge University Press. ISBN 978-0-521-85772-7
    [1] Paternoster R, Brame R (1997). "Multiple routes to delinquency? A test of developmental and general theories of crime". Criminology 35:
        45–84.
    [2] Berk R, MacDonald J (2008). "Overdispersion and Poisson regression" (http:/ / www. crim. upenn. edu/ faculty/ papers/ berk/ regression.
        pdf). Journal of Quantitative Criminology 24: 269–284. .
    [3] https:/ / files. nyu. edu/ mrg217/ public/ count. pdf
    [4] http:/ / www. mathworks. com/ help/ toolbox/ stats/ glmfit. html
    [5] http:/ / www. indiana. edu/ ~jslsoc/ files_research/ xpost/ xpost. pdf




    Discrete choice
    In economics, discrete choice problems involve choices between two or more discrete alternatives, such as entering
    or not entering the labor market, or choosing between modes of transport. Such choices contrast with standard
    consumption models in which the quantity of each good consumed is assumed to be a continuous variable. In the
    continuous case, calculus methods (e.g. first-order conditions) can be used to determine the optimum, and demand
    can be modeled using regression analysis. On the other hand, discrete choice analysis examines situations in which
    the potential outcomes are discrete, such that the optimum is not characterized by standard first-order conditions.
    Loosely, regression analysis examines “how much” while discrete choice analysis examines “which.” However,
    discrete choice analysis can be and has been used to examine the chosen quantity in particular situations, such as the
    number of vehicles a household chooses to own [1] and the number of minutes of telecommunications service a
    customer decides to use.[2]
    Discrete choice models are statistical procedures that model choices made by people among a finite set of
    alternatives. The models have been used to examine, e.g., the choice of which car to buy,[1][3] where to go to
    college,[4] , which mode of transport (car, bus, rail) to take to work[5] among numerous other applications. Discrete
    choice models are also used to examine choices by organizations, such as firms or government agencies. In the
    discussion below, the decision-making unit is assumed to be a person, though the concepts are applicable more
    generally. Daniel McFadden won the Nobel prize in 2000 for his pioneering work in developing the theoretical basis
    for discrete choice.
    Discrete choice models statistically relate the choice made by each person to the attributes of the person and the
    attributes of the alternatives available to the person. For example, the choice of which car a person buys is
    statistically related to the person’s income and age as well as to price, fuel efficiency, size, and other attributes of
    each available car. The models estimate the probability that a person chooses a particular alternative. The models are
    often used to forecast how people’s choices will change under changes in demographics and/or attributes of the
    alternatives.
Discrete choice                                                                                                                  146


    Applications
    • Marketing researchers use discrete choice models to study consumer demand and to predict competitive business
      responses, enabling choice modelers to solve a range of business problems, such as pricing, product development,
      and demand estimation problems.[1]
    • Transportation planners use discrete choice models to predict demand for planned transportation systems, such as
      which route a driver will take and whether someone will take rapid transit systems.[5][6] The first applications of
      discrete choice models were in transportation planning, and much of the most advanced research in discrete
      choice models is conducted by transportation researchers.
    • Energy forecasters and policymakers use discrete choice models for households’ and firms’ choice of heating
      system, appliance efficiency levels, and fuel efficiency level of vehicles.[7][8]
    • Environmental studies utilize discrete choice models to examine the recreators’ choice of, e.g., fishing or skiing
      site and to infer the value of amenities, such as campgrounds, fish stock, and warming huts, and to estimate the
      value of water quality improvements.[9]
    • Labor economists use discrete choice models to examine participation in the work force, occupation choice, and
      choice of college and training programs.[4]


    Common Features of Discrete Choice Models
    Discrete choice models take many forms, including: Binary Logit, Binary Probit, Multinomial Logit, Conditional
    Logit, Multinomial Probit, Nested Logit, Generalized Extreme Value Models, Mixed Logit, and Exploded Logit. All
    of these models have the features described below in common.


    Choice Set
    The choice set is the set of alternatives that are available to the person. For a discrete choice model, the choice set
    must meet three requirements:
    1. The set of alternatives must be exhaustive, meaning that the set includes all possible alternatives. This
       requirement implies that the person necessarily does choose an alternative from the set.
    2. The alternatives must be mutually exclusive, meaning that choosing one alternative means not choosing any other
       alternatives. This requirement implies that the person chooses only one alternative from the set.
    3. The set must contain a finite number of alternatives. This third requirement distinguishes discrete choice analysis
       from regression analysis in which the dependent variable can (theoretically) take an infinite number of values.
    • Example: The choice set for a person deciding which mode of transport to take to work includes driving alone,
      carpooling, taking bus, etc. The choice set is complicated by the fact that a person can use multiple modes for a
      given trip, such as driving a car to a train station and then taking train to work. In this case, the choice set can
      include each possible combination of modes. Alternatively, the choice can be defined as the choice of “primary”
      mode, with the set consisting of car, bus, rail, and other (e.g. walking, bicycles, etc.). Note that “other” alternative
      is used to make the choice set exhaustive.
    Different people may have different choice sets, depending on their circumstances. For instance, Toyota-owned
    Scion is not sold in Canada as of 2009, so new car buyers in Canada face different choice sets from those of
    American consumers.
Discrete choice                                                                                                                147


    Defining Choice Probabilities
    A discrete choice model specifies the probability that a person chooses a particular alternative, with the probability
    expressed as a function of observed variables that relate to the alternatives and the person. In its general form, the
    probability that person n chooses alternative i is expressed as:


    where
               is a vector of attributes of alternative i faced by person n,
                       is a vector of attributes of the other alternatives (other than i) faced by person n,
               is a vector of characteristics of person n, and
              is a set of parameters that relate variables to probabilities, which are estimated statistically.
    In the mode of transport example above, the attributes of modes (xni), such as travel time and cost, and the
    characteristics of consumer (sn), such as annual income, age, and gender, can be used to calculate choice
    probabilities. The attributes of the alternatives can differ over people; e.g., cost and time for travel to work by car,
    bus, and rail are different for each person depending on the location of home and work of that person.
    Properties:
       • Pni is between 0 and 1
       •                 where J is the total number of alternatives.
       • Expected share choosing i                    where N is the number of people making the choice.
    Different models (i.e. different function G) have different properties. Prominent models are introduced below.


    Consumer Utility
    Discrete choice models can be derived from utility theory. This derivation is useful for three reasons:
    1. It gives a precise meaning to the probabilities Pni
    2. It motivates and distinguishes alternative model specifications, e.g., G’s.
    3. It provides the theoretical basis for calculation of changes in consumer surplus (compensating variation) from
       changes in the attributes of the alternatives.
    Uni is the utility (or net benefit or well-being) that person n obtains from choosing alternative i. The behavior of the
    person is utility-maximizing: person n chooses the alternative that provides the highest utility. The choice of the
    person is designated by dummy variables, yni, for each alternative:




    Consider now the researcher who is examining the choice. The person’s choice depends on many factors, some of
    which the researcher observes and some of which the researcher does not. The utility that the person obtains from
    choosing an alternative is decomposed into a part that depends on variables that the researcher observes and a part
    that depends on variables that the researcher does not observe. In a linear form, this decomposition is expressed as


    where
                 is a vector of observed variables relating to alternative i for person n that depends on attributes of the
            alternative, xni, interacted perhaps with attributes of the person, sn, such that it can be expressed as
                                               for some numerical function z,
              is a corresponding vector of coefficients of the observed variables, and
                  captures the impact of all unobserved factors that affect the person’s choice.
Discrete choice                                                                                                                     148


    The choice probability is then




    Given β, the choice probability is the probability that the random terms, εnj − εni (which are random from the
    researcher’s perspective, since the researcher does not observe them) are below the respective quantities
                             . Different choice models (i.e. different specifications of G) arise from different
    distributions of εni for all i and different treatments of β.

    Properties of Discrete Choice Models Implied by Utility Theory

    Only differences matter
    The probability that a person chooses a particular alternative is determined by comparing the utility of choosing that
    alternative to the utility of choosing other alternatives:




    As the last term indicates, the choice probability depends only on the difference in utilities between alternatives, not
    on the absolute level of utilities. Equivalently, adding a constant to the utilities of all the alternatives does not change
    the choice probabilities.

    Scale must be normalized
    Since utility has no units, it is necessary to normalize the scale of utilities. The scale of utility is often defined by the
    variance of the error term in discrete choice models. This variance may differ depending on the characteristics of the
    dataset, such as when or where the data are collected. Normalization of the variance therefore affects the
    interpretation of parameters estimated across diverse datasets.


    Prominent Types of Discrete Choice Models
    Discrete choice models can first be classified according to the number of available alternatives.
          * Binomial choice models (dichotomous): 2 available alternatives
          * Multinomial choice models (polytomous): 3 or more available alternatives
    Multinomial choice models can further be classified according to the model specification:
          * Models, such as standard logit, that assume no correlation in unobserved factors over alternatives
          * Models that allow correlation in unobserved factors among alternatives
    In addition, specific forms of the models are available for examining rankings of alternatives (i.e., first choice,
    second choice, third choice, etc.) and for ratings data.
    Details for each model are provided in the following sections.
Discrete choice                                                                                                                  149


    Binary Choice

    A. Logit with attributes of the person but no attributes of the alternatives
    Un is the utility (or net benefit) that person n obtains from taking an action (as opposed to not taking the action). The
    utility the person obtains from taking the action depends on the characteristics of the person, some of which are
    observed by the researcher and some are not:


    The person takes the action, yn = 1, if Un > 0. The unobserved term, εn, is assumed to have a logistic distribution.
    The specification is written succinctly as:
       • Un = βsn + εn

       •

       • ε ∼ Logistic,
    Then the probability of taking the action is




    B. Probit with attributes of the person but no attributes of the alternatives
    The description of the model is the same as model A, except the unobserved terms are distributed standard normal
    instead of logistic.
       • Un = βsn + εn

       •

       • ε ∼ Standard normal,
    Then the probability of taking the action is
                                                   ,
                  where Φ() is cumulative distribution function of standard normal.

    C. Logit with variables that vary over alternatives
    Uni is the utility person n obtains from choosing alternative i. The utility of each alternative depends on the attributes
    of the alternatives interacted perhaps with the attributes of the person. The unobserved terms are assumed to have an
    extreme value distribution.[10]
       • Un1 = βzn1 + εn1,
       • Un2 = βzn2 + εn2,
       •               iid extreme value,
    which gives this expression for the probability




    We can relate this specification to model A above, which is also binary logit. In particular, Pn1 can also be expressed
    as
Discrete choice                                                                                                                    150


    Note that if two error terms are iid extreme value,[10] their difference is distributed logistic, which is the basis for the
    equivalence of the two specifications.

    D. Probit with variables that vary over alternatives
    The description of the model is the same as model C, except the difference of the two unobserved terms are
    distributed standard normal instead of logistic.
    Then the probability of taking the action is


                  where Φ is the cumulative distribution function of standard normal.


    Multinomial Choice without Correlation Among Alternatives

    E. Logit with attributes of the person but no attributes of the alternatives
    The utility for all alternatives depends on the same variables, sn, but the coefficients are different for different
    alternatives:
       • Uni = βi'sn + εni,
       • Since only differences in utility matter, it is necessary to normalize        for one alternative. Assuming         ,
       • εni ∼ iid extreme value [10]
    The choice probability takes the form




                  where J is the total number of alternatives.

    F. Logit with variables that vary over alternatives (also called conditional logit)
    The utility for each alternative depends on attributes of that alternative, interacted perhaps with attributes of the
    person:
       • Uni = βzni + εni,
       • εni ∼ iid extreme value,[10]
    The choice probability takes the form




                  where J is the total number of alternatives.
    Note that model E can be expressed in the same form as model F by appropriate respecification of variables.
       • Let      be a dummy variable that identifies alternative k:




       • Multiply sn from model E with each of these dummies:                   .
       • Then, model F is obtained by using                            and                   , where J is the number of
          alternatives.
Discrete choice                                                                                                                 151


    Multinomial Choice with Correlation Among Alternatives
    A standard logit model is not always suitable, since it assumes that there is no correlation in unobserved factors over
    alternatives. This lack of correlation translates into a particular pattern of substitution among alternatives that might
    not always be realistic in a given situation. This pattern of substitution is often called the Independence of Irrelevant
    Alternatives (IIA) property of standard logit models. See the Red Bus/Blue Bus example [11] or path choice
    example.[11] A number of models have been proposed to allow correlation over alternatives and more general
    substitution patterns:
    • Nested Logit Model - Captures correlations between alternatives by partitioning the choice set into 'nests'
       • Cross-nested Logit model[12] (CNL) - Alternatives may belong to more than one nest
       • C-logit Model[13] - Captures correlations between alternatives using 'commonality factor'
       • Paired Combinatorial Logit Model[14] - Suitable for route choice problems.
    • Generalised Extreme Value Model[15] - General class of model, derived from the random utility model[11] to
      which multinomial logit and nested logit belong
    • Conditional probit [16][17]- Allows full covariance among alternatives using a joint normal distribution.
    • Mixed logit [8][9][17]- Allows any form of correlation and substitution patterns.[18] When a mixed logit is with
      jointly normal random terms, the models is sometimes called "multinomial probit model with logit kernel"<ref
      name=benakiva-bierlaire-1999/.[19] Can be applied to route choice [20]
    The following sections describe Nested Logit, GEV, Probit, and Mixed Logit models in detail.

    G. Nested Logit and Generalized Extreme Value (GEV) models
    The model is the same as model F except that the unobserved component of utility is correlated over alternatives
    rather than being independent over alternatives.
       • Uni = βzni + εni,
       • The marginal distribution of each εni is extreme value,[10] but their joint distribution allows correlation among
         them.
       • The probability takes many forms depending on the pattern of correlation that is specified. See Generalized
         Extreme Value.

    H. Multinomial Probit
    The model is the same as model G except that the unobserved terms are distributed jointly normal, which allows any
    pattern of correlation and heteroscedasticity:
       • Uni = βzni + εni,
       •
    The choice probability is




                       where          is the joint normal density with mean zero and covariance     .
       • The integral for this choice probability does not have a closed form, and so the probability is approximated by
         quadrature or simulation [21].
       • When is the identity matrix (such that there is no correlation or heteroscedasticity), the model is called
         independent probit.
Discrete choice                                                                                                                152


    I. Mixed Logit
    Mixed Logit models have become increasingly popular in recent years for several reasons. First, the model allows β
    to be random in addition to ε. The randomness in β accommodates random taste variation over people and
    correlation across alternatives that generates flexible substitution patterns. Second, the advent in simulation has made
    approximation of the model fairly easy. In addition, McFadden and Train[] have shown that any true choice model
    can be approximated, to any degree of accuracy by a mixed logit with appropriate specification of explanatory
    variables and distribution of coefficients.
       • Uni = βzni + εni,
       •             for any distribution   , where is the set of distribution parameters (e.g. mean and variance) to be
         estimated,
       • εni ∼ iid extreme value,[10]
    The choice probability is



                  where

                                                   is logit probability evaluated at

                    is the total number of alternatives.
    The integral for this choice probability does not have a closed form, so the probability is approximated by simulation
    [22]
         . Also see Mixed logit for further details.


    Model Applications
    The models described above are adapted to accommodate rankings and ratings data.

    Ranking of Alternatives
    In many situations, a person's ranking of alternatives is observed, rather than just their chosen alternative. For
    example, a person who has bought a new car might be asked what he/she would have bought if that car was not
    offered, which provides information on the person's second choice in addition to their first choice. Or, in a survey, a
    respondent might be asked:
                  Example: Rank the following cell phone calling plans from your most preferred to your least preferred.
                  * $60 per month for unlimited anytime minutes, two-year contract with $100 early termination fee
                  * $30 per month for 400 anytime minutes, 3 cents per minute after 400 minutes, one-year contract with
                  $125 early termination fee
                  * $35 per month for 500 anytime minutes, 3 cents per minute after 500 minutes, no contract or early
                  termination fee
                  * $50 per month for 1000 anytime minutes, 5 cents per minute after 1000 minutes, two-year contract
                  with $75 early termination fee
    The models described above can be adapted to account for rankings beyond the first choice. The most prominent
    model for rankings data is the exploded logit and its mixed version.
Discrete choice                                                                                                                153


    J. Exploded Logit
    Under the same assumptions as for a standard logit (model F), the probability for a ranking of the alternatives is a
    product of standard logits. The model is called "exploded logit" because the choice situation that is usually
    represented as one logit formula for the chosen alternative is expanded ("exploded") to have a separate logit formula
    for each ranked alternative. The exploded logit model is the product of standard logit models with the choice set
    decreasing as each alternative is ranked and leaves the set of available choices in the subsequent choice.
    Without loss of generality, the alternatives can be relabeled to represent the person's ranking, such that alternative 1
    is the first choice, 2 the second choice, etc. The choice probability of ranking J alternatives as 1, 2, …, J is then




    As with standard logit, the exploded logit model assumes no correlation in unobserved factors over alternatives. The
    exploded logit can be generalized, in the same way as the standard logit is generalized, to accommodate correlations
    among alternatives and random taste variation. The "mixed exploded logit" model is obtained by probability of the
    ranking, given above, for Lni in the mixed logit model (model I).
    This model is also known in econometrics as the rank ordered logit model and it was introduced in that field by
    Beggs, Cardell and Hausman in 1981[23] · .[24] One application is the Combes et alii paper explaining the ranking of
    candidates to become professor.[24] Is is also known as Plackett–Luce model in biomedical literature.[24]

    Ratings Data
    In survey, respondents are often asked to give ratings, such as:
                  Example: Please give your rating of how well the President is doing.
                  1: Very badly
                  2: Badly
                  3: Fine
                  4: Good
                  5: Very good
    Or,
                  Example: On a 1-5 scale where 1 means disagree completely and 5 means agree completely, how much
                  do you agree with the following statement. "The Federal government should do more to help people
                  facing foreclosure on their homes."
    A multinomial discrete-choice model can examine the responses to these questions (model G, model H, model I).
    However, these models are derived under the concept that the respondent obtains some utility for each possible
    answer and gives the answer that provides the greatest utility. It might be more natural to think that the respondent
    has some latent measure or index associated with the question and answers in response to how high this measure is.
    Ordered logit and ordered probit models are derived under this concept.
Discrete choice                                                                                                            154


    K. Ordered Logit
    Let Un represent the strength of survey respondent n’s feelings or opinion on the survey subject. Assume that there
    are cutoffs of the level of the opinion in choosing particular response. For instance, in the example of the helping
    people facing foreclosure, the person chooses
    1.   1, if Un < a
    2.   2, if a < Un < b
    3.   3, if b < Un < c
    4.   4, if c < Un < d
    5.   5, if Un > d,
    for some real numbers a, b, c, d.
    Defining                            Logistic, then the probability of each possible response is:




    and so on up to




    The parameters of the model are the coefficients β and the cut-off points a − d, one of which must be normalized for
    identification. When there are only two possible responses, the ordered logit is the same a binary logit (model A),
    with one cut-off point normalized to zero.

    L. Ordered Probit
    The description of the model is the same as model K, except the unobserved terms are distributed standard normal
    instead of logistic.
    Then the choice probabilities are
         • Prob(choosing 1) = Φ(a − βzn),
         • Prob(choosing 2) = Φ(b − βzn) − Φ(a − βzn),
    and so on. where Φ(.) is the cumulative distribution function of standard normal.
Discrete choice                                                                                                                                                155


    Textbooks for further reading
    • McFadden, Daniel L. (1984). Econometric analysis of qualitative response models. Handbook of Econometrics,
      Volume II. Chapter 24. Elsevier Science Publishers BV.
    • Ben-Akiva, M [25] and S. Lerman [26] (1985). Discrete Choice Analysis: Theory and Application to Travel
      Demand [27], MIT Press.
    • Hensher, D. [28], J. Rose [29], and W. Greene [30] (2005). Applied Choice Analysis: A Primer [31], Cambridge
      University Press.
    • Maddala, G. (1983). Limited-dependent and Qualitative Variables in Econometrics [32], Cambridge University
      Press.
    • Train, K. (2003). Discrete Choice Methods with Simulation [33], Cambridge University Press.


    Notes
    [1] Train, K. (1986). Qualitative Choice Analysis: Theory, Econometrics, and an Application to Automobile Demand, MIT Press, Chapter 8
        (http:/ / emlab. berkeley. edu/ books/ choice. html).
    [2] Train, K. (1987). McFadden, D., and Ben-Akiva, M., “ The Demand for Local Telephone Service: A Fully Discrete Model of Residential Call
        Patterns and Service Choice (http:/ / elsa. berkeley. edu/ reprints/ misc/ demand. pdf), Rand Journal of Economics, Vol. 18, No. 1, pp109-123
    [3] Train, K. and Winston, C. (http:/ / www. brookings. edu/ experts/ w/ winstonc. aspx/ ) (2007). “ Vehicle Choice Behavior and the Declining
        Market Share of US Automakers (http:/ / elsa. berkeley. edu/ ~train/ tw104. pdf),” International Economic Review, Vol. 48, No. 4, pp.
        1469-1496.
    [4] Fuller, WC, Manski, C., and Wise, D. (1982). " New Evidence on the Economic Determinants of Post-secondary Schooling Choices (http:/ /
        www. jstor. org/ pss/ 145612)." Journal of Human Resources 17(4): 477-498.
    [5] Train, K. (1978). “ A Validation Test of a Disaggregate Mode Choice Model (http:/ / elsa. berkeley. edu/ ~train/ valtrb. pdf)”, Transportation
        Research, Vol. 12, pp. 167-174.
    [6] Ramming, M.S. (2001). “ Network Knowledge and Route Choice (http:/ / library. mit. edu/ item/ 001107149)”. Unpublished Ph.D. Thesis,
        Massachusetts Institute of Technology. MIT catalogue
    [7] Andrew Goett, Kathleen Hudson, and Train, K (2002). "Customer Choice Among Retail Energy Suppliers," Energy Journal, Vol. 21, No. 4,
        pp. 1-28.
    [8] David Revelt and Train, K (1998). " Mixed Logit with Repeated Choices: Households' Choices of Appliance Efficiency Level (http:/ / www.
        jstor. org/ stable/ pdfplus/ 2646846. pdf)," Review of Economics and Statistics, Vol. 80, No. 4, pp. 647-657
    [9] Train, K (1998)." Recreation Demand Models with Taste Variation (http:/ / citeseerx. ist. psu. edu/ viewdoc/ summary?doi=10. 1. 1. 27.
        4879)," Land Economics, Vol. 74, No. 2, pp. 230-239.
    [10] The density of the extreme value distribution is ƒ(εnj) = exp( − εnj)exp( − exp( − εnj)), and the cumulative distribution function is F(εnj) = exp(
        − exp( − εnj)).

    This distribution is also called the Gumbel or type I extreme value distribution, a special type of generalized extreme
    value distribution.
    [11] Ben-Akiva, M (http:/ / cee. mit. edu/ ben-akiva/ ) and Lerman, S (http:/ / cee. mit. edu/ lerman/ ) (1985). " Discrete Choice Analysis: Theory
        and Application to Travel Demand (Transportation Studies) (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?tid=8271& ttype=2)",
        Massachusetts: MIT Press.
    [12] Vovsha, P. (1997). " Application of Cross-Nested Logit Model to Mode Choice in Tel Aviv, Israel, Metropolitan Area (http:/ / trb.
        metapress. com/ content/ l341607q38j850j7/ )," Transportation Research Record, 1607.
    [13] Cascetta, E., A. Nuzzolo, F. Russo, and A.Vitetta (1996). “ A Modified Logit Route Choice Model Overcoming Path Overlapping Problems:
        Specification and Some Calibration Results for Interurban Networks (http:/ / www2. informatik. hu-berlin. de/ alkox/ lehre/ lvws0809/
        verkehr/ logit. pdf).” In J.B. Lesort (ed.), Transportation and Traffic Theory. Proceedings from the Thirteenth International Symposium on
        Transportation and Traffic Theory, Lyon, France, Pergamon pp. 697–711. .
    [14] Chu, C. (1989). “A Paired Combinatorial Logit Model for Travel Demand Analysis.” In Proceedings of the 5th World Conference on
        Transportation Research, 4, Ventura, CA, pp. 295–309.
    [15] McFadden, D. (1978). “ Modeling the Choice of Residential Location (http:/ / cowles. econ. yale. edu/ P/ cd/ d04b/ d0477. pdf).” In A.
        Karlqvist et al. (eds.), Spatial Interaction Theory and Residential Location, North Holland, Amsterdam pp. 75–96
    [16] J. Hausman and D. Wise (1978). "A Conditional Probit Model for Qualitative Choice: Discrete Decisions Recognizing Interdependence and
        Heterogenous Preferences," Econometrica, Vol. 48, No. 2, pp. 403-426
    [17] Train, K(2003). " Discrete Choice Methods with Simulation (http:/ / elsa. berkeley. edu/ books/ choice2. html)", Massachusetts: Cambridge
        University Press.
    [18] McFadden, D. and Train, K. (2000). “ Mixed MNL Models for Discrete Response (http:/ / elsa. berkeley. edu/ wp/ mcfadden1198/
        mcfadden1198. pdf),” Journal of Applied Econometrics, Vol. 15, No. 5, pp. 447-470,
Discrete choice                                                                                                                                           156

    [19] M. Ben-Akiva (http:/ / cee. mit. edu/ ben-akiva/ ) and D. Bolduc (http:/ / www. ecn. ulaval. ca/ no_cache/ professeurs/ fiche_de_professeurs/
        ?tx_fsgprofs_pi1[prof]=7& tx_fsgprofs_pi1[backPid]=60) (1996). “ Multinomial Probit with a Logit Kernel and a General Parametric
        Specification of the Covariance Structure (http:/ / elsa. berkeley. edu/ reprints/ misc/ multinomial. pdf).” Working Paper.
    [20] Bekhor, S. (http:/ / www. technion. ac. il/ ~civil/ bekhor/ ), Ben-Akiva, M. (http:/ / cee. mit. edu/ ben-akiva/ ), and M.S. Ramming (2002). “
        Adaptation of Logit Kernel to Route Choice Situation (http:/ / trb. metapress. com/ content/ 126847136p81w0p3/ ).” Transportation Research
        Record, 1805, 78–85.
    [21] http:/ / elsa. berkeley. edu/ choice2/ ch5. pdf
    [22] http:/ / elsa. berkeley. edu/ choice2/ ch6. pdf
    [23] Beggs, S., Cardell, S., Hausman, J., 1981. Assessing the potential demand for electric cars. Journal of Econometrics 17 (1), 1–19
        (September).
    [24] Pierre-Philippe Combes, Laurent Linnemer, Michael Visser, Publish or peer-rich? The role of skills and networks in hiring economics
        professors, Labour Economics, Volume 15, Issue 3, June 2008, Pages 423-441, ISSN 0927-5371, 10.1016/j.labeco.2007.04.003. (http:/ /
        www. sciencedirect. com/ science/ article/ pii/ S0927537107000413)
    [25] http:/ / cee. mit. edu/ ben-akiva/
    [26] http:/ / cee. mit. edu/ lerman/
    [27] http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?tid=8271& ttype=2
    [28] http:/ / www. econ. usyd. edu. au/ staff/ davidh/
    [29] http:/ / www. econ. usyd. edu. au/ staff/ johnr/
    [30] http:/ / pages. stern. nyu. edu/ ~wgreene/
    [31] http:/ / books. google. com/ books?hl=en& lr=& id=8yZrtCCABAgC& oi=fnd& pg=PR17& dq=Applied+ Choice+ Analysis:+ A+ Primer&
        ots=RCKM2_nbA4& sig=tKOOWUvIF3QcF-z8vN0wyxR7_4w
    [32] http:/ / books. google. com/ books?hl=en& lr=& id=-Ji1ZaUg7gcC& oi=fnd& pg=PR11& dq=G. S. + Maddala,+ Limited-dependent+ and+
        Qualitative+ Variables+ in+ Econometrics,+ New+ York+ :+ Cambridge+ University+ Press,+ 1983. + & ots=7d1s4GmQHK&
        sig=knQEH5Ew6d_T-OQTzYYetvoIaJo
    [33] http:/ / elsa. berkeley. edu/ books/ choice2. html



    References


    Simultaneous equations model
    Simultaneous equation models are a form of statistical model in the form of a set of linear simultaneous equations.
    They are often used in econometrics.


    Structural and reduced form
    Suppose there are m regression equations of the form


    where i is the equation number, and t = 1, …, T is the observation index. In these equations xit is the ki×1 vector of
    exogenous variables, yit is the dependent variable, y−i,t is the ni×1 vector of all other endogenous variables which
    enter the ith equation on the right-hand side, and uit are the error terms. The “−i” notation indicates that the vector y−i,t
    may contain any of the y’s except for yit (since it is already present on the left-hand side). The regression coefficients
    βi and γi are of dimensions ki×1 and ni×1 correspondingly. Vertically stacking the T observations corresponding to
    the ith equation, we can write each equation in vector form as


    where yi and ui are T×1 vectors, Xi is a T×ki matrix of exogenous regressors, and Y−i is a T×ni matrix of endogenous
    regressors on the right-hand side of the ith equation. Finally, we can move all endogenous variables to the left-hand
    side and write the m equations jointly in vector form as


    This representation is known as the structural form. In this equation Y = [y1 y2 … ym] is the T×m matrix of
    dependent variables. Each of the matrices Y−i is in fact an ni-columned submatrix of this Y. The m×m matrix Γ,
Simultaneous equations model                                                                                                   157


    which describes the relation between the dependent variables, has a complicated structure. It has ones on the
    diagonal, and all other elements of each column i are either the components of the vector −γi or zeros, depending on
    which columns of Y were included in the matrix Y−i. The T×k matrix X contains all exogenous regressors from all
    equations, but without repetitions (that is, matrix X should be of full rank). Thus, each Xi is a ki-columned submatrix
    of X. Matrix Β has size k×m, and each of its columns consists of the components of vectors βi and zeros, depending
    on which of the regressors from X were included or excluded from Xi. Finally, U = [u1 u2 … um] is a T×m matrix of
    the error terms.
    Postmultiplying the structural equation by Γ −1, the system can be written in the reduced form as


    This is already a simple general linear model, and it can be estimated for example by ordinary least squares.
    Unfortunately, the task of decomposing the estimated matrix into the individual factors Β and Γ −1 is quite
    complicated, and therefore the reduced form is more suitable for prediction but not inference.


    Assumptions
    Firstly, the rank of the matrix X of exogenous regressors must be equal to k, both in finite samples and in the limit as
    T → ∞ (this later requirement means that in the limit the expression        should converge to a nondegenerate k×k
    matrix). Matrix Γ is also assumed to be non-degenerate.
    Secondly, error terms are assumed to be serially independent and identically distributed. That is, if the tth row of
    matrix U is denoted by u(t), then the sequence of vectors {u(t)} should be iid, with zero mean and some covariance
    matrix Σ (which is unknown). In particular, this implies that E[U] = 0, and E[U′U] = T Σ.
    Lastly, the identification conditions requires that the number of unknowns in this system of equations should not
    exceed the number of equations. More specifically, the order condition requires that for each equation ki + ni ≤ k,
    which can be phrased as “the number of excluded exogenous variables is greater or equal to the number of included
    endogenous variables”. The rank condition of identifiability is that rank(Πi0) = ni, where Πi0 is a (k − ki)×ni matrix
    which is obtained from Π by crossing out those columns which correspond to the excluded endogenous variables,
    and those rows which correspond to the included exogenous variables.


    Estimation

    Two-stages least squares (2SLS)
    The simplest and the most common[1] estimation method for the simultaneous equations model is the so-called
    two-stage least squares method, developed independently by Theil (1953) and Basmann (1957). It is an
    equation-by-equation technique, where the endogenous regressors on the right-hand side of each equation are being
    instrumented with the regressors X from all other equations. The method is called “two-stage” because it conducts
    estimation in two steps:[2]
          Step 1: Regress Y−i on X and obtain the predicted values       ;
          Step 2: Estimate γi, βi by the ordinary least squares regression of yi on   and Xi.
          th
    If the i equation in the model is written as




    where Zi is a T×(ni + ki) matrix of both endogenous and exogenous regressors in the ith equation, and δi is an (ni +
    ki)-dimensional vector of regression coefficients, then the 2SLS estimator of δi will be given by[3]


    where P = X (X ′X)−1X ′ is the projection matrix onto the linear space spanned by the exogenous regressors X.
Simultaneous equations model                                                                                                                         158


    Indirect least squares
    Indirect least squares is an approach in econometrics where the coefficients in a simultaneous equations model are
    estimated from the reduced form model using ordinary least squares.[4][5] For this, the structural system of equations
    is transformed into the reduced form first. Once the coefficients are estimated the model is put back into the
    structural form.


    Limited information maximum likelihood (LIML)
    The “limited information” maximum likelihood method was suggested by Anderson & Rubin (1949). The explicit
    formula for this estimator is:[6]



    where M = I − X (X ′X)−1X ′, Mi = I − Xi (Xi′Xi)−1Xi′, and λ is the smallest characteristic root of the matrix



    Note that when λ = 1 the LIML estimator coincides with the 2SLS estimator.


    Three-stage least squares (3SLS)
    The three-stage least squares estimator was introduced by Zellner & Theil (1962). It combines two-stage least
    squares (2SLS) with seemingly unrelated regressions (SUR).


    Notes
    [1]  Greene (2003, p. 398)
    [2]  Greene (2003, p. 399)
    [3]  Greene (2003, p. 399)
    [4]  Park, S-B. (1974) "On Indirect Least Squares Estimation of a Simultaneous Equation System", The Canadian Journal of Statistics / La Revue
        Canadienne de Statistique, 2 (1), 75–82 JSTOR 3314964
    [5] Vajda, S., Valko, P. Godfrey, K.R. (1987) "Direct and indirect least squares methods in continuous-time parameter estimation", Automatica,
        23 (6), 707–718 doi:10.1016/0005-1098(87)90027-6
    [6] Amemiya (1985, p. 235)



    References
    • Amemiya, Takeshi (1985). Advanced econometrics. Cambridge, Massachusetts: Harvard University Press.
      ISBN 0-674-00560-0.
    • Anderson, T.W.; Rubin, H. (1949). "Estimator of the parameters of a single equation in a complete system of
      stochastic equations". Annals of Mathematical Statistics 20 (1): 46–63. JSTOR 2236803.
    • Basmann, R.L. (1957). "A generalized classical method of linear estimation of coefficients in a structural
      equation". Econometrica 25 (1): 77–83. JSTOR 1907743.
    • Davidson, Russell; MacKinnon, James G. (1993). Estimation and inference in econometrics. Oxford University
      Press. ISBN 978-0-19-506011-9.
    • Greene, William H. (2002). Econometric analysis (5th ed.). Prentice Hall. ISBN 0-13-066198-9.
    • Zellner, A.; Theil, H. (1962). "Three-stage least squares: simultaneous estimation of simultaneous equations".
      Econometrica 30 (1): 54–78. JSTOR 1911287.
Simultaneous equations model                                                                                                    159


    External links
    • About.com:economics (http://economics.about.com/library/glossary/bldef-ils.htm) Online dictionary of
      economics, entry for ILS



    Survival analysis
    Survival analysis is a branch of statistics which deals with death in biological organisms and failure in mechanical
    systems. This topic is called reliability theory or reliability analysis in engineering, and duration analysis or
    duration modeling in economics or event history analysis in sociology. Survival analysis attempts to answer
    questions such as: what is the fraction of a population which will survive past a certain time? Of those that survive,
    at what rate will they die or fail? Can multiple causes of death or failure be taken into account? How do particular
    circumstances or characteristics increase or decrease the odds of survival?
    To answer such questions, it is necessary to define "lifetime". In the case of biological survival, death is
    unambiguous, but for mechanical reliability, failure may not be well-defined, for there may well be mechanical
    systems in which failure is partial, a matter of degree, or not otherwise localized in time. Even in biological
    problems, some events (for example, heart attack or other organ failure) may have the same ambiguity. The theory
    outlined below assumes well-defined events at specific times; other cases may be better treated by models which
    explicitly account for ambiguous events.
    More generally, survival analysis involves the modeling of time to event data; in this context, death or failure is
    considered an "event" in the survival analysis literature – traditionally only a single event occurs for each subject,
    after which the organism or mechanism is dead or broken. Recurring event or repeated event models relax that
    assumption. The study of recurring events is relevant in systems reliability, and in many areas of social sciences and
    medical research.


    General formulation

    Survival function
    The object of primary interest is the survival function, conventionally denoted S, which is defined as


    where t is some time, T is a random variable denoting the time of death, and "Pr" stands for probability. That is, the
    survival function is the probability that the time of death is later than some specified time t. The survival function is
    also called the survivor function or survivorship function in problems of biological survival, and the reliability
    function in mechanical survival problems. In the latter case, the reliability function is denoted R(t).
    Usually one assumes S(0) = 1, although it could be less than 1 if there is the possibility of immediate death or failure.
    The survival function must be non-increasing: S(u) ≤ S(t) if u ≥ t. This property follows directly because T>u implies
    T>t. This reflects the notion that survival to a later age is only possible if all younger ages are attained. Given this
    property, the lifetime distribution function and event density (F and f below) are well-defined.
    The survival function is usually assumed to approach zero as age increases without bound, i.e., S(t) → 0 as t → ∞,
    although the limit could be greater than zero if eternal life is possible. For instance, we could apply survival analysis
    to a mixture of stable and unstable carbon isotopes; unstable isotopes would decay sooner or later, but the stable
    isotopes would last indefinitely.
Survival analysis                                                                                                                      160


    Lifetime distribution function and event density
    Related quantities are defined in terms of the survival function.
    The lifetime distribution function, conventionally denoted F, is defined as the complement of the survival function,


    If F is differentiable then the derivative , which is the density function of the lifetime distribution, is conventionally
    denoted f,



    The function f is sometimes called the event density; it is the rate of death or failure events per unit time.
    The survival function can be expressed in terms of distribution and density functions



    Similarly, a survival event density function can be defined as




    Hazard function and cumulative hazard function
    The hazard function, conventionally denoted               , is defined as the event rate at time t conditional on survival until
    time t or later (that is, T ≥ t),




    Force of mortality is a synonym of hazard function which is used particularly in demography and actuarial science,
    where it is denoted by . The term hazard rate is another synonym.
    The hazard function must be non-negative, λ(t) ≥ 0, and its integral over                        must be infinite, but is not
    otherwise constrained; it may be increasing or decreasing, non-monotonic, or discontinuous. An example is the
    bathtub curve hazard function, which is large for small values of t, decreasing to some minimum, and thereafter
    increasing again; this can model the property of some mechanical systems to either fail soon after operation, or much
    later, as the system ages.
    The hazard function can alternatively be represented in terms of the cumulative hazard function, conventionally
    denoted :


    so transposing signs and exponentiating


    or differentiating (with the chain rule)




    The name "cumulative hazard function" is derived from the fact that



    which is the "accumulation" of the hazard over time.
    From the definition of              , we see that it increases without bound as t tends to infinity (assuming that S(t) tends to
    zero). This implies that              must not decrease too quickly, since, by definition, the cumulative hazard has to
    diverge. For example,                     is not the hazard function of any survival distribution, because its integral
Survival analysis                                                                                                               161


    converges to 1.


    Quantities derived from the survival distribution
    Future lifetime at a given time          is the time remaining until death, given survival to age   . Thus, it is    in
    the present notation. The expected future lifetime is the expected value of future lifetime. The probability of death
    at or before age        , given survival until age , is just



    Therefore the probability density of future lifetime is




    and the expected future lifetime is




    where the second expression is obtained using integration by parts.
    For             , that is, at birth, this reduces to the expected lifetime.
    In reliability problems, the expected lifetime is called the mean time to failure, and the expected future lifetime is
    called the mean residual lifetime.
    As the probability of an individual surviving until age t or later is S(t), by definition, the expected number of
    survivors at age t out of an initial population of n newborns is n × S(t), assuming the same survival function for all
    individuals. Thus the expected proportion of survivors is S(t). If the survival of different individuals is independent,
    the number of survivors at age t has a binomial distribution with parameters n and S(t), and the variance of the
    proportion of survivors is S(t) × (1-S(t))/n.
    The age at which a specified proportion of survivors remain can be found by solving the equation S(t) = q for t,
    where q is the quantile in question. Typically one is interested in the median lifetime, for which q = 1/2, or other
    quantiles such as q = 0.90 or q = 0.99.
    One can also make more complex inferences from the survival distribution. In mechanical reliability problems, one
    can bring cost (or, more generally, utility) into consideration, and thus solve problems concerning repair or
    replacement. This leads to the study of renewal theory and reliability theory of aging and longevity.


    Censoring
    Censoring is a form of missing data problem which is common in survival analysis. Ideally, both the birth and death
    dates of a subject are known, in which case the lifetime is known.
    If it is known only that the date of death is after some date, this is called right censoring. Right censoring will occur
    for those subjects whose birth date is known but who are still alive when they are lost to follow-up or when the study
    ends.
    If a subject's lifetime is known to be less than a certain duration, the lifetime is said to be left-censored.
    It may also happen that subjects with a lifetime less than some threshold may not be observed at all: this is called
    truncation. Note that truncation is different from left censoring, since for a left censored datum, we know the subject
    exists, but for a truncated datum, we may be completely unaware of the subject. Truncation is also common. In a
    so-called delayed entry study, subjects are not observed at all until they have reached a certain age. For example,
    people may not be observed until they have reached the age to enter school. Any deceased subjects in the pre-school
    age group would be unknown. Left-truncated data is common in actuarial work for life insurance and pensions
    (Richards, 2010).
Survival analysis                                                                                                                                    162


    We generally encounter right-censored data. Left-censored data can occur when a person's survival time becomes
    incomplete on the left side of the follow-up period for the person. As an example, we may follow up a patient for any
    infectious disorder from the time of his or her being tested positive for the infection. We may never know the exact
    time of exposure to the infectious agent.[1]


    Fitting parameters to data
    Survival models can be usefully viewed as ordinary regression models in which the response variable is time.
    However, computing the likelihood function (needed for fitting parameters or making other kinds of inferences) is
    complicated by the censoring. The likelihood function for a survival model, in the presence of censored data, is
    formulated as follows. By definition the likelihood function is the conditional probability of the data given the
    parameters of the model. It is customary to assume that the data are independent given the parameters. Then the
    likelihood function is the product of the likelihood of each datum. It is convenient to partition the data into four
    categories: uncensored, left censored, right censored, and interval censored. These are denoted "unc.", "l.c.", "r.c.",
    and "i.c." in the equation below.



    For an uncensored datum, with              equal to the age at death, we have


    For a left censored datum, such that the age at death is known to be less than                     , we have


    For a right censored datum, such that the age at death is known to be greater than                       , we have


    For an interval censored datum, such that the age at death is known to be less than                            and greater than           , we
    have


    An important application where interval censored data arises is current status data, where the actual occurrence of an
    event   is only known to the extent that it known not to occurred before observation time and to have occurred
    before the next.


    Non-parametric estimation
    The Nelson–Aalen estimator can be used to provide a non-parametric estimate of the cumulative hazard rate
    function.


    Distributions used in survival analysis
    • Exponential distribution
    • Weibull distribution
    • Exponential-logarithmic distribution


    References
    [1] Singh R, Mukhopadhyay K. Survival analysis in clinical trials: Basics and must know areas. Perspect Clin Res [serial online] 2011 [cited
        2011 Nov 1];2:145-8. Available from: http:/ / www. picronline. org/ text. asp?2011/ 2/ 4/ 145/ 86872
Survival analysis                                                                                                           163


    Sources
    • David Collett. Modelling Survival Data in Medical Research, Second Edition. Boca Raton: Chapman &
      Hall/CRC. 2003. ISBN 978-1-58488-325-8
    • Regina Elandt-Johnson and Norman Johnson. Survival Models and Data Analysis. New York: John Wiley &
      Sons. 1980/1999.
    • J. D. Kalbfleisch and Ross L. Prentice. The statistical analysis of failure time data. New York: John Wiley &
      Sons. 1980 (1st ed.), 2002 (2nd ed.) ISBN 978-0-471-36357-6
    • Jerald F. Lawless. Statistical Models and Methods for Lifetime Data, 2nd edition. John Wiley and Sons, Hoboken.
      2003.
    • Terry Therneau. "A Package for Survival Analysis in S". http://www.mayo.edu/hsr/people/therneau/survival.
      ps, at: http://mayoresearch.mayo.edu/mayo/research/biostat/therneau.cfm
    • "Engineering Statistics Handbook", NIST/SEMATEK, (http://www.itl.nist.gov/div898/handbook/)
    • Rausand, M. and Hoyland, A. System Reliability Theory: Models, Statistical Methods, and Applications, John
      Wiley & Sons, Hoboken, 2004. See web site (http://www.ntnu.no/ross/books/srt).
    • Richards, S. J. A handbook of parametric survival models for actuarial use. Scandinavian Actuarial Journal (http:/
      /www.informaworld.com/smpp/content~db=all~content=a926035258~frm=titlelink)
    Singh R, Mukhopadhyay K. Survival analysis in clinical trials: Basics and must know areas. Perspect Clin Res
    [serial online] 2011 [cited 2011 Nov 1];2:145-8. Available from: http:/ / www. picronline. org/ text. asp?2011/ 2/ 4/
    145/86872


    External links
    • SOCR, Survival analysis applet (http://www.socr.ucla.edu/htmls/ana/Survival_Analysis.html) and
      interactive learning activity (http://wiki.stat.ucla.edu/socr/index.php/
      SOCR_EduMaterials_AnalysisActivities_Survival).
    • Survival/Failure Time Analysis (http://www.statsoft.com/textbook/stsurvan.html) @ Statistics' Textbook
      Page (http://www.statsoft.com/textbook/)
    • Survival Analysis in R (http://www.netstorm.be/home/survival)
Survey methodology                                                                                                           164



    Survey methodology
    In statistics, survey methodology is the field that studies the sampling of individuals from a population with a view
    towards making statistical inferences about the population using the sample. Polls about public opinion, such as
    political beliefs, are reported in the news media in democracies. Other types of survey are used for scientific
    purposes. Surveys provide important information for all kinds of research fields, e.g., marketing research,
    psychology, health professionals and sociology.[1] A survey may focus on different topics such as preferences (e.g.,
    for a presidential candidate), behavior (smoking and drinking behavior), or factual information (e.g., income),
    depending on its purpose. Since survey research is always based on a sample of the population, the success of the
    research is dependent on the representativeness of the population of concern (see also sampling (statistics) and
    survey sampling).
    Survey methodology seeks to identify principles about the design, collection, processing, and analysis of surveys in
    connection to the cost and quality of survey estimates. It focuses on improving quality within cost constraints, or
    alternatively, reducing costs for a fixed level of quality. Survey methodology is both a scientific field and a
    profession. Part of the task of a survey methodologist is making a large set of decisions about thousands of
    individual features of a survey in order to improve it.[2]
    The most important methodological challenges of a survey methodologist include making decisions on how to:[2]
    •   Identify and select potential sample members.
    •   Contact sampled individuals and collect data from those who are hard to reach (or reluctant to respond).
    •   Evaluate and test questions.
    •   Select the mode for posing questions and collecting responses.
    •   Train and supervise interviewers (if they are involved).
    •   Check data files for accuracy and internal consistency.
    •   Adjust survey estimates to correct for identified errors.


    Selecting samples
    Survey samples can be broadly divided into two types: probability samples and non-probability samples. Stratified
    sampling is a method of probability sampling such that sub-populations within an overall population are identified
    and included in the sample selected in a balanced way.


    Modes of data collection
    There are several ways of administering a survey. The choice between administration modes is influenced by several
    factors, including 1) costs, 2) coverage of the target population, 3) flexibility of asking questions, 4) respondents'
    willingness to participate and 5) response accuracy. Different methods create mode effects that change how
    respondents answer, and different methods have different advantages. The most common modes of administration
    can be summarized as:[3]
    •   Telephone
    •   Mail (post)
    •   Online surveys
    •   Personal in-home surveys
    •   Personal mall or street intercept survey
    •   Hybrids of the above.
Survey methodology                                                                                                            165


    Cross-sectional and longitudinal surveys
    There is a distinction between one-time (cross-sectional) surveys, which involve a single questionnaire or interview
    administered to each sample member, and surveys which repeatedly collect information from the sample people over
    time. The latter are known as longitudinal surveys. Longitudinal surveys have considerable analytical advantages but
    they are also challenging to implement successfully. Consequently, specialist methods have been developed to select
    longitudinal samples, to collect data repeatedly, to keep track of sample members over time, to keep respondents
    motivated to participate, and to process and analyse longitudinal survey data [4]


    Response formats
    Usually, a survey consists of a number of questions that the respondent has to answer in a set format. A distinction is
    made between open-ended and closed-ended questions. An open-ended question asks the respondent to formulate his
    own answer, whereas a closed-ended question has the respondent pick an answer from a given number of options.
    The response options for a closed-ended question should be exhaustive and mutually exclusive. Four types of
    response scales for closed-ended questions are distinguished:
    • Dichotomous, where the respondent has two options
    • Nominal-polytomous, where the respondent has more than two unordered options
    • Ordinal-polytomous, where the respondent has more than two ordered options
    • (bounded)Continuous, where the respondent is presented with a continuous scale
    A respondent's answer to an open-ended question can be coded into a response scale afterwards,[3] or analysed using
    more qualitative methods.


    Advantages and disadvantages

    Advantages
    •   They are relatively easy to administer.
    •   Can be developed in less time compared with other data-collection methods.
    •   Can be cost-effective.
    •   Few 'experts' are required to develop a survey, which may increase the reliability of the survey data.
    •   If conducted remotely, can reduce or obviate geographical dependence.
    •   Useful in describing the characteristics of a large population assuming the sampling is valid.
    •   Can be administered remotely via the Web, mobile devices, mail, e-mail, telephone, etc.
    •   Efficient at collecting information from a large number of respondents.
    •   Statistical techniques can be applied to the survey data to determine validity, reliability, and statistical
        significance even when analyzing multiple variables.
    •   Many questions can be asked about a given topic giving considerable flexibility to the analysis.
    •   Support both between and within-subjects study designs.
    •   A wide range of information can be collected (e.g., attitudes, values, beliefs, and behaviour).
    •   Because they are standardized, they are relatively free from several types of errors.
    •   Surveys are ideal for scientific research studies because they provide all the participants with a standardized
        stimulus. With such high reliability obtained, the researcher’s own biases are eliminated
Survey methodology                                                                                                             166


    Disadvantages
    The reliability of survey data may depend on the following:
    • Respondents' motivation, honesty, memory, and ability to respond:
      • Respondents may not be motivated to give accurate answers.
      • Respondents may be motivated to give answers that present themselves in a favorable light.
      • Respondents may not be fully aware of their reasons for any given action.
    • Structured surveys, particularly those with closed ended questions, may have low validity when researching
      affective variables.
    • Self-selection bias. Although the individuals chosen to participate in surveys are often randomly sampled, errors
      due to non-response may exist (see also chapter 13 of Adér et al. (2008) for more information on how to deal with
      non-responders and biased data). That is, people who choose to respond on the survey may be different from
      those who do not respond, thus biasing the estimates. For example, polls or surveys that are conducted by calling
      a random sample of publicly available telephone numbers will not include the responses of people with unlisted
      telephone numbers, mobile (cell) phone numbers, people who are unable to answer the phone (e.g., because they
      normally sleep during the time of day the survey is conducted, because they are at work, etc.), people who do not
      answer calls from unknown or unfamiliar telephone numbers. Likewise, such a survey will include a
      disproportionate number of respondents who have traditional, land-line telephone service with listed phone
      numbers, and people who stay home much of the day and are much more likely to be available to participate in
      the survey (e.g., people who are unemployed, disabled, elderly, etc.).
    • Question design. Survey question answer-choices could lead to vague data sets because at times they are relative
      only to a personal abstract notion concerning "strength of choice". For instance the choice "moderately agree"
      may mean different things to different subjects, and to anyone interpreting the data for correlation. Even 'yes' or
      'no' answers are problematic because subjects may for instance put "no" if the choice "only once" is not available.


    Nonresponse reduction
    The following ways have been recommended for reducing nonresponse[5] in telephone and face-to-face surveys:[6]
    • Advance letter. A short letter is sent in advance to inform the sampled respondents about the upcoming survey.
      The style of the letter should be personalized but not overdone. First, it announces that a phone call will be made/
      or an interviewer wants to make an appointment to do the survey face-to-face. Second, the research topic will be
      described. Last, it allows both an expression of the surveyor's appreciation of cooperation and an opening to ask
      questions on the survey.
    • Training. The interviewers are thoroughly trained in how to ask respondents questions, how to work with
      computers and making schedules for callbacks to respondents who were not reached.
    • Short introduction. The interviewer should always start with a short instruction about him or herself. She/he
      should give her name, the institute she is working for, the length of the interview and goal of the interview. Also it
      can be useful to make clear that you are not selling anything: this has been shown to lead led to a slightly higher
      responding rate.[7]
    • Respondent-friendly survey questionnaire. The questions asked must be clear, non-offensive and easy to respond
      to for the subjects under study.
Survey methodology                                                                                                                                        167


    Other methods to increase response rates
    • brevity – single page if possible
    • financial incentives
      • paid in advance
      • paid at completion
    • non-monetary incentives
        • commodity giveaways (pens, notepads)
        • entry into a lottery, draw or contest
        • discount coupons
        • promise of contribution to charity
    •   preliminary notification
    •   foot-in-the-door techniques – start with a small inconsequential request
    •   personalization of the request – address specific individuals
    •   follow-up requests – multiple requests
    •   emotional appeals
    •   bids for sympathy
    •   convince respondent that they can make a difference
    • guarantee anonymity
    • legal compulsion (certain government-run surveys)


    Interviewer effects
    Survey methodologists have devoted much effort to determine the extent to which interviewee responses are affected
    by physical characteristics of the interviewer. Main interviewer traits that have been demonstrated to influence
    survey responses are race [8] , gender [9] and relative body weight (BMI) .[10] These interviewer effects are
    particularly operant when questions are related to the interviewer trait. Hence, race of interviewer has been shown to
    affect responses to measures regarding racial attitudes ,[11] interviewer sex responses to questions involving gender
    issues ,[12] and interviewer BMI answers to eating and dieting-related questions .[13] While interviewer effects have
    been investigated mainly for face-to-face surveys, they have also been shown to exist for interview modes with no
    visual contact, such as telephone surveys and in video-enhanced web surveys. The explanation typically provided for
    interviewer effects is that of social desirability. Survey participants may attempt to project a positive self-image in an
    effort to conform to the norms they attribute to the interviewer asking questions.


    Notes
    [1] http:/ / whatisasurvey. info/
    [2] Groves, R.M.; Fowler, F. J.; Couper, M.P.; Lepkowski, J.M.; Singer, E.; Tourangeau, R. (2009). Survey Methodology. New Jersey: John
        Wiley & Sons. ISBN 978-1-118-21134-2.
    [3] Mellenbergh, G.J. (2008). Chapter 9: Surveys. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on
        Research Methods: A consultant's companion (pp. 183–209). Huizen, The Netherlands: Johannes van Kessel Publishing.
    [4] Lynn, P. (2009) (Ed.) Methodology of Longitudinal Surveys. Wiley. ISBN 0-470-01871-2
    [5] Lynn, P. (2008) "The problem of non-response", chapter 3, 35-55, in International Handbook of Survey Methodology (ed.s E.de Leeuw,
        J.Hox & D.Dillman). Erlbaum. ISBN 0-8058-5753-2
    [6] Dillman, D.A. (1978) Mail and telephone surveys: The total design method. Wiley. ISBN 0-471-21555-4
    [7] De Leeuw, E.D. (2001). "I am not selling anything: Experiments in telephone introductions". Kwantitatieve Methoden, 22, 41–48.
    [8] Hill, M.E (2002). "Race of the interviewer and perception of skin color: Evidence from the multi-city study of urban inequality" (http:/ /
        www. jstor. org/ stable/ 3088935). American Sociological Review 67 (1): 99–108. .
    [9] Flores-Macias, F.; Lawson, C. (2008). "Effects of interviewer gender on survey responses: Findings from a household survey in Mexico".
        International Journal of Public Opinion Research 20 (1): 100–110. doi:10.1093/ijpor/edn007.
    [10] Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B.; Van Strien, T. (2011). "BMI of interviewer effects". International Journal of Public
        Opinion Research 23 (4): 530–543. doi:10.1093/ijpor/edr026.
Survey methodology                                                                                                                                  168

    [11] Anderson, B.A.; Abramson, B.D. (1988). "The effects of the race of the interviewer on race-related attitudes of black respondents in
        SRC/CPS national election studies". Public Opinion Quarterly 52 (3): 1–28. doi:10.1086/269108.
    [12] Kane, E.W.; Macaulay, L.J. (1993). "Interviewer gender and gender attitudes". Public Opinion Quarterly 57 (1): 1–28. doi:10.1086/269352.
    [13] Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B.. "Interviewer BMI effects on under- and over-reporting of restrained eating.
        Evidence from a national Dutch face-to-face survey and a postal follow-up". International Journal of Public Health 57 (3): 643-647.
        doi:10.1007/s00038-011-0323-z.



    References
    • Abramson, J.J. and Abramson, Z.H. (1999).Survey Methods in Community Medicine: Epidemiological Research,
      Programme Evaluation, Clinical Trials (5th edition). London: Churchill Livingstone/Elsevier Health Sciences
      ISBN 0-443-06163-7
    • Groves, R.M. (1989). Survey Errors and Survey Costs Wiley. ISBN 0-471-61171-9
    • Ornstein, M.D. (1998). "Survey Research." Current Sociology 46(4): iii-136.
    • Shaughnessy, J. J., Zechmeister, E. B., & Zechmeister, J. S. (2006). Research Methods in Psychology (Seventh
      Edition ed.). McGraw–Hill Higher Education. ISBN 0-07-111655-9 (pp. 143–192)
    • Adèr, H. J., Mellenbergh, G. J., & Hand, D. J. (2008). Advising on research methods: A consultant's companion.
      Huizen, The Netherlands: Johannes van Kessel Publishing.
    • Dillman, D.A. (1978) Mail and telephone surveys: The total design method. New York: Wiley. ISBN
      0-471-21555-4


    Further reading
    • Andres, Lesley (2012). "Designing and Doing Survey Research" (http://www.uk.sagepub.com/books/
      Book234957?siteId=sage-uk&prodTypes=any&q=andres). London: Sage.
    • Leung, Wai-Ching (2001) "Conducting a Survey" (http://archive.student.bmj.com/back_issues/0601/
      education/187.html), in Student BMJ, (British Medical Journal, Student Edition), May 2001


    External links
    • Surveys (http://www.dmoz.org/Science/Social_Sciences/Methodology/Survey/) at the Open Directory
      Project
    • OmniPHP(tm) SurveyEngine (http://surveyengine.sourceforge.net/) – An open source advanced survey
      development application that allows creating any type of web-based survey.
    • Nonprofit Research Collection on the Use of Surveys in Nonprofit Research (http://www.issuelab.org/closeup/
      Jan_2009/) Published on IssueLab
    • Survey Question Bank (http://www.surveynet.ac.uk/sqb)
    • Designing surveys – a basic guide (http://www.snapsurveys.com/surveys/)
Index (economics)                                                                                                               169



    Index (economics)
    In economics and finance, an index is a statistical measure of changes in a representative group of individual data
    points. These data may be derived from any number of sources, including company performance, prices,
    productivity, and employment. Economic indices (index, plural) track economic health from different perspectives.
    Influential global financial indices such as the Global Dow, and the NASDAQ Composite track the performance of
    selected large and powerful companies in order to evaluate and predict economic trends. The Dow Jones Industrial
    Average and the S&P 500 primarily track U.S. markets, though some legacy international companies are included.[1]
    The Consumer Price Index tracks the variation in prices for different consumer goods and services over time in a
    constant geographical location, and is integral to calculations used to adjust salaries, bond interest rates, and tax
    thresholds for inflation. The GDP Deflator Index, or real GDP, measures the level of prices of all new, domestically
    produced, final goods and services in an economy.[2] Market performance indices include the labour market
    index/job index and proprietary stock market index investment instruments offered by brokerage houses.
    Some indices display market variations that cannot be captured in other ways. For example, the Economist provides
    a Big Mac Index that expresses the adjusted cost of a globally ubiquitous Big Mac as a percentage over or under the
    cost of a Big Mac in the U.S. in USD (estimated: $3.57).[3] The least relatively expensive Big Mac price occurs in
    Hong Kong, at a 52% reduction from U.S. prices, or $1.71 U.S. Such indices can be used to help forecast currency
    values. From this example, it would be assumed that Hong Kong currency is undervalued, and provides a currency
    investment opportunity.


    Index numbers
    An index number is an economic data figure reflecting price or quantity compared with a standard or base
    value.[4][5] The base usually equals 100 and the index number is usually expressed as 100 times the ratio to the base
    value. For example, if a commodity costs twice as much in 1970 as it did in 1960, its index number would be 200
    relative to 1960. Index numbers are used especially to compare business activity, the cost of living, and employment.
    They enable economists to reduce unwieldy business data into easily understood terms.
    In economics, index numbers generally are time series summarising movements in a group of related variables. In
    some cases, however, index numbers may compare geographic areas at a point in time. An example is a country's
    purchasing power parity. The best-known index number is the consumer price index, which measures changes in
    retail prices paid by consumers. In addition, a cost-of-living index (COLI) is a price index number that measures
    relative cost of living over time.[6] In contrast to a COLI based on the true but unknown utility function, a superlative
    index number is an index number that can be calculated.[6] Thus, superlative index numbers are used to provide a
    fairly close approximation to the underlying cost-of-living index number in a wide range of circumstances.[6]
    There is a substantial body of economic analysis concerning the construction of index numbers, desirable properties
    of index numbers and the relationship between index numbers and economic theory.


    Index number problem
    The "index number problem" refers to the difficulty of constructing a valid index when both price and quantity
    change over time. For instance, in the construction of price indices for inflation, the nature of goods in the economy
    changes over time as well as their prices. A price index constructed in 1950 using a standard basket of goods based
    on 1950 consumption would not well represent the prices faced by consumers in 2000, as goods in some categories
    are no longer traded in 2000, new categories of goods have been introduced, and the relative spending on different
    categories of goods will change drastically. Furthermore, the goods in the basket may have changed in quality.
    There is no theoretically ideal solution to this problem. In practice for retail price indices, the "basket of goods" is
    updated incrementally every few years to reflect changes. Nevertheless, the fact remains that many economic indices
Index (economics)                                                                                                  170


    taken over the long term are not really like-for-like comparisons and this is an issue taken into account by
    researchers in economic history.


    Indices
    Provider: Dow Jones
    • Dow Jones Industrial Average
    Provider: Standard & Poor's
    •   S&P 500
    •   S&P 400
    •   S&P 600
    •   S&P 1500
    •   S&P/ASX 200
    •   S&P/TSX Composite Index
    •   S&P Global 1200
    •   S&P Custom Group of indices
    •   S&P Leveraged Loan Index
    • Case–Shiller index
    Provider: Russell Investments
    •   Russell 1000 Index
    •   Russell 2000 Index
    •   Russell 3000 Index
    •   Russell Midcap Index
    •   Russell Microcap Index
    •   Russell Global Index
    •   Russell Developed Index
    •   Russell Europe Index
    •   Russell Asia Pacific Index
    •   Russell Emerging Markets Index
    Provider: Morgan Stanley Capital International
    • MSCI World Index
    • MSCI EAFE (Europe, Australasia, and Far East) Index
    Provider: Bombay Stock Exchange
    • BSE SENSEX
    Provider: Reuters
    • Reuters-CRB Commodities Index
    Provider: Markit
    • ABX
    • CDX / iTraxx
    • CMBX
    Provider: Historic Automobile Group
    • HAGI Top Index
Index (economics)                                                                                                                                         171


       References
       [1]  http:/ / www. investopedia. com/ university/ indexes/ index1. asp
       [2]  http:/ / www. politonomist. com/ gdp-deflator-and-measuring-inflation-00491/
       [3]  http:/ / www. oanda. com/ currency/ big-mac-index
       [4]  Diewert, W. E., "Index Numbers", in Eatwell, John; Milgate, Murray; Newman, Peter, The New Palgrave: A Dictionary of Economics, 2,
           pp. 767–780
       [5] Moulton, Brent R.; Smith, Jeffrey W., "Price Indices", in Newman, Peter; Milgate, Murray; Eatwell, John, The New Palgrave Dictionary of
           Money and Finance, 3, pp. 179–181
       [6] Turvey, Ralph. (2004) Consumer Price Index Manual: Theory And Practice. (http:/ / books. google. com/ books?id=HOqcFW9b5VoC&
           pg=PA11& dq=Superlative+ index+ number& as_brr=3& sig=0pz8BjGjaNoB-HEPK8o09xOH57Q#PPA11,M1) Page 11. Publisher:
           International Labour Organization. ISBN 92-2-113699-X.



       Further reading
       • Robin Marris, Economic Arithmetic, (1958).


       External links

   •       Humboldt Economic Index (http:/ / www. humboldt. edu/ ~indexhum/ )                                      •   SG Index (http:/ / www. sgindex.
   •       S&P Indices (http:/ / www2. standardandpoors. com/ portal/ site/ sp/ en/ us/ page. category/ indices/       com/ )
           2,3,1,0,0,0,0,0,0,0,0,0,0,0,0,0. html?lid=us_topnav_indicies)                                           •   Dow Jones Indexes (http:/ /
                                                                                                                       www. djindexes. com/ )




       Aggregate demand
       In macroeconomics, aggregate demand (AD) is the total demand for final goods and services in the economy (Y) at
       a given time and price level.[1] It is the amount of goods and services in the economy that will be purchased at all
       possible price levels.[2] This is the demand for the gross domestic product of a country when inventory levels are
       static. It is often called effective demand, though at other times this term is distinguished.
       It is often cited that the aggregate demand curve is downward sloping because at lower price levels a greater quantity
       is demanded. While this is correct at the microeconomic, single good level, at the aggregate level this is incorrect.
       The aggregate demand curve is in fact downward sloping as a result of three distinct effects: Pigou's wealth effect,
       the Keynes' interest rate effect and the Mundell-Fleming exchange-rate effect.


       Components
       An aggregate demand curve is the sum of individual demand curves for different sectors of the economy. The
       aggregate demand is usually described as a linear sum of four separable demand sources.[3]


       where
       •          is consumption (may also be known as consumer spending) =                                        ,
       •         is Investment,
       •          is Government spending,
       •                          is Net export,
             •      is total exports, and
             •       is total imports =                                  .
       These four major parts, which can be stated in either 'nominal' or 'real' terms, are:
Aggregate demand                                                                                                           172


    • personal consumption expenditures (C) or "consumption," demand by households and unattached individuals; its
      determination is described by the consumption function. The consumption function is C= a + (mpc)(Y-T)
      • a is autonomous consumption, mpc is the marginal propensity to consume, (Y-T) is the disposable income.
    • gross private domestic investment (I), such as spending by business firms on factory construction. This includes
      all private sector spending aimed at the production of some future consumable.
      • In Keynesian economics, not all of gross private domestic investment counts as part of aggregate demand.
        Much or most of the investment in inventories can be due to a short-fall in demand (unplanned inventory
        accumulation or "general over-production"). The Keynesian model forecasts a decrease in national output and
        income when there is unplanned investment. (Inventory accumulation would correspond to an excess supply of
        products; in the National Income and Product Accounts, it is treated as a purchase by its producer.) Thus, only
        the planned or intended or desired part of investment (Ip) is counted as part of aggregate demand. (So, I does
        not include the 'investment' in running up or depleting inventory levels.)
      • Investment is affected by the output and the interest rate (i). Consequently, we can write it as I(Y,i).
        Investment has positive relationship with the output and negative relationship with the interest rate. For
        example, an increase in the interest rate will cause aggregate demand to decline. Interest costs are part of the
        cost of borrowing and as they rise, both firms and households will cut back on spending. This shifts the
        aggregate demand curve to the left. This lowers equilibrium GDP below potential GDP. As production falls for
        many firms, they begin to lay off workers, and unemployment rises. The declining demand also lowers the
        price level. The economy is in recession.
    • gross government investment and consumption expenditures (G).
    • net exports (NX and sometimes (X-M)), i.e., net demand by the rest of the world for the country's output.
    In sum, for a single country at a given time, aggregate demand (D or AD) = C + Ip + G + (X-M).
    These macrovariables are constructed from varying types of microvariables from the price of each, so these variables
    are denominated in (real or nominal) currency terms.
Aggregate demand                                                                                                                173


    Aggregate demand curves
    Understanding of the aggregate demand curve depends on whether it is examined based on changes in demand as
    income changes, or as price change.


    Aggregate demand-aggregate supply model
    Sometimes, especially in textbooks, "aggregate demand" refers to an entire demand curve that looks like that in a
    typical Marshallian supply and demand diagram.
    Thus, that we could refer to an "aggregate
    quantity demanded" (Yd = C + Ip + G + NX
    in real or inflation-corrected terms) at any
    given aggregate average price level (such as
    the GDP deflator), P.
    In these diagrams, typically the Yd rises as
    the average price level (P) falls, as with the
    AD line in the diagram. The main
    theoretical reason for this is that if the
    nominal money supply (Ms) is constant, a falling
    P implies that the real money supply (M
                                            s/P)rises,
    encouraging lower interest rates and higher
    spending. This is often called the "Keynes
    effect."

    Carefully using ideas from the theory of                                 Aggregate supply/demand graph

    supply and demand, aggregate supply can
    help determine the extent to which increases in aggregate demand lead to increases in real output or instead to
    increases in prices (inflation). In the diagram, an increase in any of the components of AD (at any given P) shifts the
    AD curve to the right. This increases both the level of real production (Y) and the average price level (P).
    But different levels of economic activity imply different mixtures of output and price increases. As shown, with very
    low levels of real gross domestic product and thus large amounts of unemployed resources, most economists of the
    Keynesian school suggest that most of the change would be in the form of output and employment increases. As the
    economy gets close to potential output (Y*), we would see more and more price increases rather than output
    increases as AD increases.
    Beyond Y*, this gets more intense, so that price increases dominate. Worse, output levels greater than Y* cannot be
    sustained for long. The AS is a short-term relationship here. If the economy persists in operating above potential, the
    AS curve will shift to the left, making the increases in real output transitory.
    At low levels of Y, the world is more complicated. First, most modern industrial economies experience few if any
    falls in prices. So the AS curve is unlikely to shift down or to the right. Second, when they do suffer price cuts (as in
    Japan), it can lead to disastrous deflation.
Aggregate demand                                                                                                                      174


    If they do not result in greater aggregate demand, increasing business
    profits do not necessarily lead to increased economic growth. When
    businesses and banks have disincentive to spend accumulated capital,
    such as cash repatriation taxes from profits in overseas tax havens and
    interest on excess reserves paid to banks, increased profits can lead to
    decreasing growth.[4][5]


    Debt                                                                            Red: corporate profits after tax and inventory
    A Post-Keynesian theory of aggregate demand emphasizes the role of             valuation adjustment. Blue: nonresidential fixed
                                                                                       investment (roughly speaking, business
    debt, which it considers a fundamental component of aggregate
                                                                                     investment), both as fractions of U.S. GDP,
    demand;[6] the contribution of change in debt to aggregate demand is                             1989-2012.
    referred to by some as the credit impulse.[7] Aggregate demand is
    spending, be it on consumption, investment, or other categories. Spending is related to income via:
          Income – Spending = Net Savings
    Rearranging this yields:
          Spending = Income – Net Savings = Income + Net Increase in Debt
    In words: what you spend is what you earn, plus what you borrow: if you spend $110 and earned $100, then you
    must have net borrowed $10; conversely if you spend $90 and earn $100, then you have net savings of $10, or have
    reduced debt by $10, for net change in debt of –$10.
    If debt grows or shrinks slowly as a percentage of GDP, its impact on aggregate demand is small; conversely, if debt
    is significant, then changes in the dynamics of debt growth can have significant impact on aggregate demand.
    Change in debt is tied to the level of debt:[6] if the overall debt level is 10% of GDP and 1% of loans are not repaid,
    this impacts GDP by 1% of 10% = 0.1% of GDP, which is statistical noise. Conversely, if the debt level is 300% of
    GDP and 1% of loans are not repaid, this impacts GDP by 1% of 300% = 3% of GDP, which is significant: a change
    of this magnitude will generally cause a recession. Similarly, changes in the repayment rate (debtors paying down
    their debts) impact aggregate demand in proportion to the level of debt. Thus, as the level of debt in an economy
    grows, the economy becomes more sensitive to debt dynamics, and credit bubbles are of macroeconomic concern.
    Since write-offs and savings rates both spike in recessions, both of which result in shrinkage of credit, the resulting
    drop in aggregate demand can worsen and perpetuate the recession in a vicious cycle.
    This perspective originates in, and is intimately tied to, the debt-deflation theory of Irving Fisher, and the notion of a
    credit bubble (credit being the flip side of debt), and has been elaborated in the Post-Keynesian school.[6] If the
    overall level of debt is rising each year, then aggregate demand exceeds Income by that amount. However, if the
    level of debt stops rising and instead starts falling (if "the bubble bursts"), then aggregate demand falls short of
    income, by the amount of net savings (largely in the form of debt repayment or debt writing off, such as in
    bankruptcy). This causes a sudden and sustained drop in aggregate demand, and this shock is argued to be the
    proximate cause of a class of economic crises, properly financial crises. Indeed, a fall in the level of debt is not
    necessary – even a slowing in the rate of debt growth causes a drop in aggregate demand (relative to the higher
    borrowing year).[8] These crises then end when credit starts growing again, either because most or all debts have
    been repaid or written off, or for other reasons as below.
    From the perspective of debt, the Keynesian prescription of government deficit spending in the face of an economic
    crisis consists of the government net dis-saving (increasing its debt) to compensate for the shortfall in private debt: it
    replaces private debt with public debt. Other alternatives include seeking to restart the growth of private debt
    ("reflate the bubble"), or slow or stop its fall; and debt relief, which by lowering or eliminating debt stops credit from
    contracting (as it cannot fall below zero) and allows debt to either stabilize or grow – this has the further effect of
    redistributing wealth from creditors (who write off debts) to debtors (whose debts are relieved).
Aggregate demand                                                                                                                                        175


    Criticisms
    Austrian theorist Henry Hazlitt argued that aggregate demand is a meaningless concept in economic analysis.[9]
    Friedrich Hayek, another Austrian theorist, argued that Keynes' study of the aggregate relations in an economy is
    fallacious, as recessions are caused by micro-economic factors.[10]


    References
    [1] Sexton, Robert; Fortura, Peter (2005). Exploring Economics. ISBN 0-17-641482-7. "This is the sum of the demand for all final goods and
        services in the economy. It can also be seen as the quantity of real GDP demanded at different price levels."
    [2] O'Sullivan, Arthur; Steven M. Sheffrin (2003). oi = Economics: Principles in action (http:/ / www. pearsonschool. com/ index.
        cfm?locator=PSZ3R9& PMDbSiteId=2781& PMDbSolutionId=6724& PMDbCategoryId=& PMDbProgramId=12881& level=4). Upper
        Saddle River, New Jersey 07458: Pearson Prentice Hall. pp. 307. ISBN 0-13-063085-3. oi =.
    [3] "aggregate demand (AD)" (http:/ / www. tutor2u. net/ economics/ content/ topics/ ad_as/ ad-as_notes. htm). Archived (http:/ / web. archive.
        org/ web/ 20071109072803/ http:/ / www. tutor2u. net/ economics/ content/ topics/ ad_as/ ad-as_notes. htm) from the original on 9 November
        2007. . Retrieved 2007-11-04.
    [4] "Profits and Business Investment" (http:/ / krugman. blogs. nytimes. com/ 2013/ 02/ 09/ profits-and-business-investment/ ) Paul Krugman,
        New York Times", February 9, 2013
    [5] "Still Say’s Law After All These Years" (http:/ / krugman. blogs. nytimes. com/ 2013/ 02/ 10/ still-says-law-after-all-these-years/ ) Paul
        Krugman, New York Times, February 10, 2013
    [6] Debtwatch No 41, December 2009: 4 Years of Calling the GFC (http:/ / www. debtdeflation. com/ blogs/ 2009/ 12/ 01/
        debtwatch-no-41-december-2009-4-years-of-calling-the-gfc/ ), Steve Keen, December 1, 2009
    [7] Credit and Economic Recovery: Demystifying Phoenix Miracles (http:/ / ssrn. com/ paper=1595980), Michael Biggs, Thomas Mayer,
        Andreas Pick, March 15, 2010
    [8] "However much you borrow and spend this year, if it is less than last year, it means your spending will go into recession." Dhaval Joshi, RAB
        Capital, quoted in Noughty boys on trading floor led us into debt-laden fantasy (http:/ / www. smh. com. au/ business/
        noughty-boys-on-trading-floor-led-us-into-debtladen-fantasy-20091222-lbs4. html)
    [9] Hazlitt, Henry (1959). The Failure of the 'New Economics': An Analysis of the Keynesian Fallacies (http:/ / www. mises. org/ books/
        failureofneweconomics. pdf). D. Van Nostrand. .
    [10] Hayek, Friedrich (1989). The Collected Works of F.A. Hayek. University of Chicago Press. p. 202. ISBN 978-0-226-32097-7.



    External links
    • Elmer G. Wiens: Classical & Keynesian AD-AS Model (http://www.egwald.ca/macroeconomics/keynesian.
      php) - An on-line, interactive model of the Canadian Economy.
Operations research                                                                                                           176



    Operations research
    Operations research, or operational research in British usage, is a discipline that deals with the application of
    advanced analytical methods to help make better decisions.[1] It is often considered to be a sub-field of
    mathematics.[2] The terms management science and decision science are sometimes used as more modern-sounding
    synonyms.[3]
    Employing techniques from other mathematical sciences, such as mathematical modeling, statistical analysis, and
    mathematical optimization, operations research arrives at optimal or near-optimal solutions to complex
    decision-making problems. Because of its emphasis on human-technology interaction and because of its focus on
    practical applications, operations research has overlap with other disciplines, notably industrial engineering and
    operations management, and draws on psychology and organization science. Operations Research is often concerned
    with determining the maximum (of profit, performance, or yield) or minimum (of loss, risk, or cost) of some
    real-world objective. Originating in military efforts before World War II, its techniques have grown to concern
    problems in a variety of industries.[4]


    Overview
    Operational research (OR) encompasses a wide range of problem-solving techniques and methods applied in the
    pursuit of improved decision-making and efficiency, such as simulation, mathematical optimization, queueing theory
    and other stochastic-process models, Markov decision processes, econometric methods, data envelopment analysis,
    neural networks, expert systems, decision analysis, and the analytic hierarchy process.[5] Nearly all of these
    techniques involve the construction of mathematical models that attempt to describe the system. Because of the
    computational and statistical nature of most of these fields, OR also has strong ties to computer science and
    analytics. Operational researchers faced with a new problem must determine which of these techniques are most
    appropriate given the nature of the system, the goals for improvement, and constraints on time and computing power.
    The major subdisciplines in modern operational research, as identified by the journal Operations Research,[6] are:
    •   Computing and information technologies
    •   Environment, energy, and natural resources
    •   Financial engineering
    •   Manufacturing, service sciences, and supply chain management
    •   Marketing Engineering[7]
    •   Policy modeling and public sector work
    •   Revenue management
    •   Simulation
    •   Stochastic models
    •   Transportation.


    History
    As a formal discipline, operational research originated in the efforts of military planners during World War II. In the
    decades after the war, the techniques began to be applied more widely to problems in business, industry and society.
    Since that time, operational research has expanded into a field widely used in industries ranging from petrochemicals
    to airlines, finance, logistics, and government, moving to a focus on the development of mathematical models that
    can be used to analyse and optimize complex systems, and has become an area of active academic and industrial
    research.[4]
Operations research                                                                                                             177


    Historical origins
    In the World War II era, operational research was defined as "a scientific method of providing executive departments
    with a quantitative basis for decisions regarding the operations under their control."[8] Other names for it included
    operational analysis (UK Ministry of Defence from 1962)[9] and quantitative management.[10]
    Prior to the formal start of the field, early work in operational research was carried out by individuals such as Charles
    Babbage. His research into the cost of transportation and sorting of mail led to England's universal "Penny Post" in
    1840, and studies into the dynamical behaviour of railway vehicles in defence of the GWR's broad gauge.[11] Percy
    Bridgman brought operational research to bear on problems in physics in the 1920s and would later attempt to
    extend these to the social sciences.[12] The modern field of operational research arose during World War II.
    Modern operational research originated at the Bawdsey Research Station in the UK in 1937 and was the result of an
    initiative of the station's superintendent, A. P. Rowe. Rowe conceived the idea as a means to analyse and improve the
    working of the UK's early warning radar system, Chain Home (CH). Initially, he analysed the operating of the radar
    equipment and its communication networks, expanding later to include the operating personnel's behaviour. This
    revealed unappreciated limitations of the CH network and allowed remedial action to be taken.[13]
    Scientists in the United Kingdom including Patrick Blackett (later Lord Blackett OM PRS), Cecil Gordon, C. H.
    Waddington, Owen Wansbrough-Jones, Frank Yates, Jacob Bronowski and Freeman Dyson, and in the United States
    with George Dantzig looked for ways to make better decisions in such areas as logistics and training schedules. After
    the war it began to be applied to similar problems in industry.


    Second World War
    During the Second World War close to 1,000 men and women in Britain were engaged in operational research.
    About 200 operational research scientists worked for the British Army.[14]
    Patrick Blackett worked for several different organizations during the war. Early in the war while working for the
    Royal Aircraft Establishment (RAE) he set up a team known as the "Circus" which helped to reduce the number of
    anti-aircraft artillery rounds needed to shoot down an enemy aircraft from an average of over 20,000 at the start of
    the Battle of Britain to 4,000 in 1941.[15]
    In 1941 Blackett moved from the RAE to the Navy, after first working with RAF Coastal Command, in 1941 and
    then early in 1942 to the Admiralty.[16] Blackett's team at Coastal Command's Operational Research Section
    (CC-ORS) included two future Nobel prize winners and many other people who went on to be pre-eminent in their
    fields.[17] They undertook a number of crucial analyses that aided the war effort. Britain introduced the convoy
    system to reduce shipping losses, but while the principle of using warships to accompany merchant ships was
    generally accepted, it was unclear whether it was better for convoys to be small or large. Convoys travel at the speed
    of the slowest member, so small convoys can travel faster. It was also argued that small convoys would be harder for
    German U-boats to detect. On the other hand, large convoys could deploy more warships against an attacker.
    Blackett's staff showed that the losses suffered by convoys depended largely on the number of escort vessels present,
    rather than on the overall size of the convoy. Their conclusion, therefore, was that a few large convoys are more
    defensible than many small ones.[18]
    While performing an analysis of the methods used by RAF Coastal Command to hunt and destroy submarines, one
    of the analysts asked what colour the aircraft were. As most of them were from Bomber Command they were painted
    black for nighttime operations. At the suggestion of CC-ORS a test was run to see if that was the best colour to
    camouflage the aircraft for daytime operations in the grey North Atlantic skies. Tests showed that aircraft painted
    white were on average not spotted until they were 20% closer than those painted black. This change indicated that
    30% more submarines would be attacked and sunk for the same number of sightings.[19]
    Other work by the CC-ORS indicated that on average if the trigger depth of aerial-delivered depth charges (DCs)
    were changed from 100 feet to 25 feet, the kill ratios would go up. The reason was that if a U-boat saw an aircraft
Operations research                                                                                                               178


    only shortly before it arrived over the target then at 100 feet the charges would do no damage (because the U-boat
    wouldn't have had time to descend as far as 100 feet), and if it saw the aircraft a long way from the target it had time
    to alter course under water so the chances of it being within the 20-foot kill zone of the charges was small. It was
    more efficient to attack those submarines close to the surface when the targets' locations were better known than to
    attempt their destruction at greater depths when their positions could only be guessed. Before the change of settings
    from 100 feet to 25 feet, 1% of submerged U-boats were sunk and 14% damaged. After the change, 7% were sunk
    and 11% damaged. (If submarines were caught on the surface, even if attacked shortly after submerging, the numbers
    rose to 11% sunk and 15% damaged). Blackett observed "there can be few cases where such a great operational gain
    had been obtained by such a small and simple change of tactics".[20]
    Bomber Command's Operational Research Section (BC-ORS), analysed a report of a survey carried out by RAF
    Bomber Command. For the survey, Bomber Command inspected all bombers returning from bombing raids over
    Germany over a particular period. All damage inflicted by German air defences was noted and the recommendation
    was given that armour be added in the most heavily damaged areas. This recommendation was not adopted because
    the fact that the aircraft returned with these areas damaged indicated these areas were NOT vital, and adding armor
    weight to non-vital areas where damage is acceptable negatively affects aircraft performance. Their suggestion to
    remove some of the crew so that an aircraft loss would result in fewer personnel loss was also rejected by RAF
    command. Blackett's team made the logical recommendation that the armour be placed in the areas which were
    completely untouched by damage in the bombers which returned. They reasoned that the survey was biased, since it
    only included aircraft that returned to Britain. The untouched areas of returning aircraft were probably vital areas,
    which, if hit, would result in the loss of the aircraft.[21]
                                           When Germany organised its air defences into the Kammhuber Line, it was
                                           realised by the British that if the RAF bombers were to fly in a bomber stream
                                           they could overwhelm the night fighters who flew in individual cells directed
                                           to their targets by ground controllers. It was then a matter of calculating the
                                           statistical loss from collisions against the statistical loss from night fighters to
                                           calculate how close the bombers should fly to minimise RAF losses.[22]

                                           The "exchange rate" ratio of output to input was a characteristic feature of
                                           operational research. By comparing the number of flying hours put in by
                                           Allied aircraft to the number of U-boat sightings in a given area, it was
                                           possible to redistribute aircraft to more productive patrol areas. Comparison
                                           of exchange rates established "effectiveness ratios" useful in planning. The
                                           ratio of 60 mines laid per ship sunk was common to several campaigns:
                                           German mines in British ports, British mines on German routes, and United
            Map of Kammhuber Line          States mines in Japanese routes.[23]

                                            Operational research doubled the on-target bomb rate of B-29s bombing
    Japan from the Marianas Islands by increasing the training ratio from 4 to 10 percent of flying hours; revealed that
    wolf-packs of three United States submarines were the most effective number to enable all members of the pack to
    engage targets discovered on their individual patrol stations; revealed that glossy enamel paint was more effective
    camouflage for night fighters than traditional dull camouflage paint finish, and the smooth paint finish increased
    airspeed by reducing skin friction.[23]

    On land, the operational research sections of the Army Operational Research Group (AORG) of the Ministry of
    Supply (MoS) were landed in Normandy in 1944, and they followed British forces in the advance across Europe.
    They analysed, among other topics, the effectiveness of artillery, aerial bombing, and anti-tank shooting.
Operations research                                                                                                       179


    After World War II
    With expanded techniques and growing awareness of the field at the close of the war, operational research was no
    longer limited to only operational, but was extended to encompass equipment procurement, training, logistics and
    infrastructure.


    Problems addressed with operational research
    • Critical path analysis or project planning: identifying those processes in a complex project which affect the
      overall duration of the project
    • Floorplanning: designing the layout of equipment in a factory or components on a computer chip to reduce
      manufacturing time (therefore reducing cost)
    • Network optimization: for instance, setup of telecommunications networks to maintain quality of service during
      outages
    • Allocation problems
    • Facility location
    • Assignment Problems:
        • Assignment problem
        • Generalized assignment problem
        • Quadratic assignment problem
        • Weapon target assignment problem
    •   Bayesian search theory : looking for a target
    •   Optimal search
    •   Routing, such as determining the routes of buses so that as few buses are needed as possible
    •   Supply chain management: managing the flow of raw materials and products based on uncertain demand for the
        finished products
    •   Efficient messaging and customer response tactics
    •   Automation: automating or integrating robotic systems in human-driven operations processes
    •   Globalization: globalizing operations processes in order to take advantage of cheaper materials, labor, land or
        other productivity inputs
    •   Transportation: managing freight transportation and delivery systems (Examples: LTL Shipping, intermodal
        freight transport)
    •   Scheduling:
      • Personnel staffing
      • Manufacturing steps
      • Project tasks
      • Network data traffic: these are known as queueing models or queueing systems.
      • Sports events and their television coverage
    • Blending of raw materials in oil refineries
    • Determining optimal prices, in many retail and B2B settings, within the disciplines of pricing science
    Operational research is also used extensively in government where evidence-based policy is used.
Operations research                                                                                                               180


        Management science
        In 1967 Stafford Beer characterized the field of management science as "the business use of operations research".[24]
        However, in modern times the term management science may also be used to refer to the separate fields of
        organizational studies or corporate strategy. Like operational research itself, management science (MS) is an
        interdisciplinary branch of applied mathematics devoted to optimal decision planning, with strong links with
        economics, business, engineering, and other sciences. It uses various scientific research-based principles, strategies,
        and analytical methods including mathematical modeling, statistics and numerical algorithms to improve an
        organization's ability to enact rational and meaningful management decisions by arriving at optimal or near optimal
        solutions to complex decision problems. In short, management sciences help businesses to achieve their goals using
        the scientific methods of operational research.
        The management scientist's mandate is to use rational, systematic, science-based techniques to inform and improve
        decisions of all kinds. Of course, the techniques of management science are not restricted to business applications
        but may be applied to military, medical, public administration, charitable groups, political groups or community
        groups.
        Management science is concerned with developing and applying models and concepts that may prove useful in
        helping to illuminate management issues and solve managerial problems, as well as designing and developing new
        and better models of organizational excellence.[25]
        The application of these models within the corporate sector became known as management science.[26]


        Related fields
        Some of the fields that have considerable overlap with Operations Research and Management Science include:

    •       Business Analytics                 •   Logistics
    •       Data mining                        •   Mathematical modeling
    •       Decision analysis                  •   Mathematical optimization
    •       Engineering                        •   Probability and statistics
    •       Financial engineering              •   Project management
    •       Forecasting                        •   Policy analysis
    •       Game theory                        •   Simulation
    •       Graph theory                       •   Social network/Transportation forecasting models
    •       Industrial engineering             •   Stochastic processes
                                               •   Supply chain management



        Applications of management science
        Applications of management science are abundant in industry as airlines, manufacturing companies, service
        organizations, military branches, and in government. The range of problems and issues to which management
        science has contributed insights and solutions is vast. It includes:[25]
        •   scheduling airlines, including both planes and crew,
        •   deciding the appropriate place to site new facilities such as a warehouse, factory or fire station,
        •   managing the flow of water from reservoirs,
        •   identifying possible future development paths for parts of the telecommunications industry,
        •   establishing the information needs and appropriate systems to supply them within the health service, and
        •   identifying and understanding the strategies adopted by companies for their information systems
        Management science is also concerned with so-called ”soft-operational analysis”, which concerns methods for
        strategic planning, strategic decision support, and Problem Structuring Methods (PSM). In dealing with these sorts of
        challenges mathematical modeling and simulation are not appropriate or will not suffice. Therefore, during the past
Operations research                                                                                                          181


    30 years, a number of non-quantified modeling methods have been developed. These include:
    •   stakeholder based approaches including metagame analysis and drama theory
    •   morphological analysis and various forms of influence diagrams.
    •   approaches using cognitive mapping
    •   the Strategic Choice Approach
    •   robustness analysis


    Societies and journals
    Societies
    The International Federation of Operational Research Societies (IFORS)[27] is an umbrella organization for
    operational research societies worldwide, representing approximately 50 national societies including those in the
    US,[28] UK,[29] France,[30] Germany, Canada,[31] Australia,[32] New Zealand,[33] Philippines,[34] India,[35] Japan and
    South Africa (ORSSA).[36] The constituent members of IFORS form regional groups, such as that in Europe.[37]
    Other important operational research organizations are Simulation Interoperability Standards Organization
    (SISO)[38] and Interservice/Industry Training, Simulation and Education Conference (I/ITSEC)[39]
    In 2004 the US-based organization INFORMS began an initiative to market the OR profession better, including a
    website entitled The Science of Better[40] which provides an introduction to OR and examples of successful
    applications of OR to industrial problems. This initiative has been adopted by the Operational Research Society in
    the UK, including a website entitled Learn about OR.[41]
    Journals
    INFORMS publishes thirteen scholarly journals about operations research, including the top two journals in their
    class, according to 2005 Journal Citation Reports.[42] They are:
    •   Decision Analysis[43]
    •   Information Systems Research
    •   INFORMS Journal on Computing
    •   INFORMS Transactions on Education[44] (an open access journal)
    •   Interfaces: An International Journal of the Institute for Operations Research and the Management Sciences
    •   Management Science: A Journal of the Institute for Operations Research and the Management Sciences
    •   Manufacturing & Service Operations Management
    •   Marketing Science
    •   Mathematics of Operations Research
    •   Operations Research: A Journal of the Institute for Operations Research and the Management Sciences
    •   Organization Science
    •   Service Science [45]
    •   Transportation Science.
    Other journals
    • 4OR-A Quarterly Journal of Operations Research: jointly published the Belgian, French and Italian Operations
      Research Societies (Springer);
    • Decision Sciences published by Wiley-Blackwell on behalf of the Decision Sciences Institute
    • European Journal of Operational Research (EJOR): Founded in 1975 and is presently by far the largest
      operational research journal in the world, with its around 9,000 pages of published papers per year. In 2004, its
      total number of citations was the second largest amongst Operational Research and Management Science
      journals;
    • INFOR Journal: published and sponsored by the Canadian Operational Research Society;
Operations research                                                                                                                                     182


    • International Journal of Operations Research and Information Systems (IJORIS)": an official publication of the
      Information Resources Management Association, published quarterly by IGI Global;[46]
    • Journal of Defense Modeling and Simulation (JDMS): Applications, Methodology, Technology: a quarterly
      journal devoted to advancing the science of modeling and simulation as it relates to the military and defense.[47]
    • Journal of the Operational Research Society (JORS): an official journal of The OR Society; this is the oldest
      continuously published journal of OR in the world, published by Palgrave;[48]
    • Journal of Simulation (JOS): an official journal of The OR Society, published by Palgrave;[48]
    • Mathematical Methods of Operations Research (MMOR): the journal of the German and Dutch OR Societies,
      published by Springer;[49]
    • Military Operations Research (MOR): published by the Military Operations Research Society;
    • Opsearch: official journal of the Operational Research Society of India;
    • OR Insight: a quarterly journal of The OR Society, published by Palgrave;[48]
    • Production and Operations Management, the official journal of the Production and Operations Management
      Society
    • TOP: the official journal of the Spanish Society of Statistics and Operations Research.[50]


    Notes
    [1] "About Operations Research" (http:/ / www. informs. org/ About-INFORMS/ About-Operations-Research). INFORMS.org. . Retrieved 7
        January 2012.
    [2] "Mathematics Subject Classification" (http:/ / www. mathontheweb. org/ mathweb/ mi-mathbyclass. html). American Mathematical Society.
        23 May 2011. . Retrieved 7 January 2012.
    [3] "Definition of Operations Research" (http:/ / www. answers. com/ topic/ operations-research). Answers.com. . Retrieved 7 January 2012.
    [4] "What is OR" (http:/ / www. hsor. org/ what_is_or. cfm). HSOR.org. . Retrieved 13 November 2011.
    [5] "Operations Research Analysts" (http:/ / www. bls. gov/ oco/ ocos044. htm). Bls.gov. . Retrieved 27 January 2012.
    [6] "OR / Pubs / IOL Home" (http:/ / www3. informs. org/ site/ OperationsResearch/ index. php?c=10& kat=Forthcoming+ Papers).
        INFORMS.org. 2 January 2009. . Retrieved 13 November 2011.
    [7] "DecisionPro, Inc. – Makers of Marketing Engineering Software – Home Page" (http:/ / www. decisionpro. biz). Decisionpro.biz. . Retrieved
        13 November 2011.
    [8] "Operational Research in the British Army 1939–1945, October 1947, Report C67/3/4/48, UK National Archives file WO291/1301
        Quoted on the dust-jacket of: Morse, Philip M, and Kimball, George E, Methods of Operations Research, 1st Edition Revised, pub MIT Press
        & J Wiley, 5th printing, 1954.
    [9] UK National Archives Catalogue for WO291 (http:/ / www. nationalarchives. gov. uk/ catalogue/ displaycataloguedetails. asp?CATID=109&
        CATLN=2& Highlight=& FullDetails=True) lists a War Office organisation called Army Operational Research Group (AORG) that existed
        from 1946 to 1962. "In January 1962 the name was changed to Army Operational Research Establishment (AORE). Following the creation of
        a unified Ministry of Defence, a tri-service operational research organisation was established: the Defence Operational Research
        Establishment (DOAE) which was formed in 1965, and it absorbed the Army Operational Research Establishment based at West Byfleet."
    [10] http:/ / brochure. unisa. ac. za/ myunisa/ data/ subjects/ Quantitative%20Management. pdf
    [11] M.S. Sodhi, "What about the 'O' in O.R.?" OR/MS Today, December, 2007, p. 12, http:/ / www. lionhrtpub. com/ orms/ orms-12-07/ frqed.
        html
    [12] P. W. Bridgman, The Logic of Modern Physics, The MacMillan Company, New York, 1927
    [13] "operations research (industrial engineering) :: History – Britannica Online Encyclopedia" (http:/ / www. britannica. com/ EBchecked/ topic/
        682073/ operations-research/ 68171/ History#ref22348). Britannica.com. . Retrieved 13 November 2011.
    [14] Kirby, p. 117 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA117)
    [15] Kirby, pp. 91–94 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA94)
    [16] Kirby, p. 96,109 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA109)
    [17] Kirby, p. 96 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA96)
    [18] ""Numbers are Essential": Victory in the North Atlantic Reconsidered, March–May 1943" (http:/ / www. familyheritage. ca/ Articles/
        victory1943. html). Familyheritage.ca. 24 May 1943. . Retrieved 13 November 2011.
    [19] Kirby, p. 101 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA101)
    [20] (Kirby, pp. 102,103 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA103))
    [21] James F. Dunnigan (1999). Dirty Little Secrets of the Twentieth Century. Harper Paperbacks. pp. 215–217.
    [22] "RAF History – Bomber Command 60th Anniversary" (http:/ / www. raf. mod. uk/ bombercommand/ thousands. html). Raf.mod.uk. .
        Retrieved 13 November 2011.
    [23] Milkman, Raymond H. (May 1968). Operations Research in World War II. United States Naval Institute Proceedings.
Operations research                                                                                                                                  183

    [24] Stafford Beer (1967) Management Science: The Business Use of Operations Research
    [25] What is Management Science? (http:/ / www. lums. lancs. ac. uk/ departments/ ManSci/ DeptProfile/ WhatisManSci/ ) Lancaster University,
        2008. Retrieved 5 June 2008.
    [26] What is Management Science? (http:/ / bus. utk. edu/ soms/ information/ whatis_msci. html) The University of Tennessee, 2006. Retrieved 5
        June 2008.
    [27] "IFORS" (http:/ / www. ifors. org/ ). IFORS. . Retrieved 13 November 2011.
    [28] Leszczynski, Mary (8 November 2011). "Informs" (http:/ / www. informs. org/ ). Informs. . Retrieved 13 November 2011.
    [29] "The OR Society" (http:/ / www. orsoc. org. uk). Orsoc.org.uk. . Retrieved 13 November 2011.
    [30] "Société française de Recherche Opérationnelle et d'Aide à la Décision" (http:/ / www. roadef. org/ content/ index. htm). ROADEF. .
        Retrieved 13 November 2011.
    [31] www.cors.ca. "CORS" (http:/ / www. cors. ca). Cors.ca. . Retrieved 13 November 2011.
    [32] "ASOR" (http:/ / www. asor. org. au). ASOR. 1 January 1972. . Retrieved 13 November 2011.
    [33] "ORSNZ" (http:/ / www. orsnz. org. nz/ ). ORSNZ. . Retrieved 13 November 2011.
    [34] "ORSP" (http:/ / www. orsp. org. ph/ ). ORSP. . Retrieved 13 November 2011.
    [35] "ORSI" (http:/ / www. orsi. in/ ). Orsi.in. . Retrieved 13 November 2011.
    [36] "ORSSA" (http:/ / www. orssa. org. za/ ). ORSSA. 23 September 2011. . Retrieved 13 November 2011.
    [37] "EURO" (http:/ / www. euro-online. org/ ). Euro-online.org. . Retrieved 13 November 2011.
    [38] "SISO" (http:/ / www. sisostds. org/ ). Sisostds.org. . Retrieved 13 November 2011.
    [39] "I/Itsec" (http:/ / www. iitsec. org/ ). I/Itsec. . Retrieved 13 November 2011.
    [40] "The Science of Better" (http:/ / www. scienceofbetter. org/ ). The Science of Better. . Retrieved 13 November 2011.
    [41] "Learn about OR" (http:/ / www. learnaboutor. co. uk/ ). Learn about OR. . Retrieved 13 November 2011.
    [42] "INFORMS Journals" (http:/ / www. informs. org/ index. php?c=31& kat=-+ INFORMS+ Journals). Informs.org. . Retrieved 13 November
        2011.
    [43] "''Decision Analysis''" (http:/ / www. informs. org/ site/ DA/ ). Informs.org. . Retrieved 13 November 2011.
    [44] "INFORMS Transactions on Education" (http:/ / www. informs. org/ site/ ITE/ ). Informs.org. . Retrieved 13 November 2011.
    [45] http:/ / servsci. journal. informs. org/
    [46] "International Journal of Operations Research and Information Systems (IJORIS) (1947–9328)(1947–9336): John Wang: Journals" (http:/ /
        www. igi-global. com/ Bookstore/ TitleDetails. aspx?TitleId=1141). IGI Global. . Retrieved 13 November 2011.
    [47] The Society for Modeling & Simulation International. "JDMS" (http:/ / www. scs. org/ pubs/ jdms/ jdms. html). Scs.org. . Retrieved 13
        November 2011.
    [48] The OR Society (http:/ / www. orsoc. org. uk);
    [49] "Mathematical Methods of Operations Research website" (http:/ / www. springer. com/ mathematics/ journal/ 186). Springer.com. .
        Retrieved 13 November 2011.
    [50] "TOP" (http:/ / www. springer. com/ east/ home/ business/ operations+
        research?SGWID=5-40521-70-173677307-detailsPage=journal|description). Springer.com. . Retrieved 13 November 2011.



    References
    • Kirby, M. W. (Operational Research Society (Great Britain)). Operational Research in War and Peace: The
      British Experience from the 1930s to 1970, Imperial College Press, 2003. ISBN 1-86094-366-7, ISBN
      978-1-86094-366-9


    Further reading
    • C. West Churchman, Russell L. Ackoff & E. L. Arnoff, Introduction to Operations Research, New York: J.
      Wiley and Sons, 1957
    • Joseph G. Ecker & Michael Kupferschmid, Introduction to Operations Research, Krieger Publishing Co.
    • Frederick S. Hillier & Gerald J. Lieberman, Introduction to Operations Research, McGraw-Hill: Boston MA; 8th.
      (International) Edition, 2005
    • Michael Pidd, Tools for Thinking: Modelling in Management Science, J. Wiley & Sons Ltd., Chichester; 2nd.
      Edition, 2003
    • Hamdy A. Taha, Operations Research: An Introduction, Prentice Hall; 9th. Edition, 2011
    • Wayne Winston, Operations Research: Applications and Algorithms, Duxbury Press; 4th. Edition, 2003
    • Kenneth R. Baker, Dean H. Kropp (1985). Management Science: An Introduction to the Use of Decision Models
    • David Charles Heinze (1982). Management Science: Introductory Concepts and Applications
Operations research                                                                                                184


    • Lee J. Krajewski, Howard E. Thompson (1981). "Management Science: Quantitative Methods in Context"
    • Thomas W. Knowles (1989). Management science: Building and Using Models
    • Kamlesh Mathur, Daniel Solow (1994). Management Science: The Art of Decision Making
    • Laurence J. Moore, Sang M. Lee, Bernard W. Taylor (1993). Management Science
    • William Thomas Morris (1968). Management Science: A Bayesian Introduction.
    • William E. Pinney, Donald B. McWilliams (1987). Management Science: An Introduction to Quantitative
      Analysis for Management
    • Shrader, Charles R. (2006). History of Operations Research in the United States Army, Volume 1:1942–1962
      (http://www.history.army.mil/html/books/hist_op_research/index.html). Washington, D.C.: United States
      Army Center of Military History. CMH Pub 70-102-1.
    • Gerald E. Thompson (1982). Management Science: An Introduction to Modern Quantitative Analysis and
      Decision Making. New York : McGraw-Hill Publishing Co.
    • Saul I. Gass & Arjang A. Assad (2005). An Annotated Timeline of Operations Research: An Informal History.
      New York : Kluwer Academic Publishers.


    External links
    • INFORMS OR/MS Resource Collection (http://www.informs.org/Resources/): a comprehensive set of OR
      links.
    • International Federation of Operational Research Societies (http://www.ifors.org/)
    • Occupational Outlook Handbook, U.S. Department of Labor Bureau of Labor Statistics (http://stats.bls.gov/
      oco/ocos044.htm/)
    • "Operation Everything: It Stocks Your Grocery Store, Schedules Your Favorite Team's Games, and Helps Plan
      Your Vacation. The Most Influential Academic Discipline You've Never Heard Of." Boston Globe, 27 June 2004
      (http://www.boston.com/news/globe/reprints/062704_postrel/)
    • "Optimal Results: IT-powered advances in operations research can enhance business processes and boost the
      corporate bottom line." Computerworld, 20 November 2000 (http://www.computerworld.com/s/article/
      54157/Optimal_Results?taxonomyId=063/)
Decision theory                                                                                                                    185



    Decision theory
    Decision theory in economics, psychology, philosophy, mathematics, and statistics is concerned with identifying the
    values, uncertainties and other issues relevant in a given decision, its rationality, and the resulting optimal decision.
    It is closely related to the field of game theory as to interactions of agents with at least partially conflicting interests
    whose decisions affect each other.


    Normative and descriptive decision theory
    Most of decision theory is normative or prescriptive, i.e., it is concerned with identifying the best decision to take (in
    practice, there are situations in which "best" is not necessarily the maximal, optimum may also include values in
    addition to maximum, but within a specific or approximative range), assuming an ideal decision maker who is fully
    informed, able to compute with perfect accuracy, and fully rational. The practical application of this prescriptive
    approach (how people ought to make decisions) is called decision analysis, and aimed at finding tools,
    methodologies and software to help people make better decisions. The most systematic and comprehensive software
    tools developed in this way are called decision support systems.
    Since people usually do not behave in ways consistent with axiomatic rules, often their own, leading to violations of
    optimality, there is a related area of study, called a positive or descriptive discipline, attempting to describe what
    people will actually do. Since the normative, optimal decision often creates hypotheses for testing against actual
    behaviour, the two fields are closely linked. Furthermore it is possible to relax the assumptions of perfect
    information, rationality and so forth in various ways, and produce a series of different prescriptions or predictions
    about behaviour, allowing for further tests of the kind of decision-making that occurs in practice.
    In recent decades, there has been increasing interest in what is sometimes called 'behavioral decision theory' and this
    has contributed to a re-evaluation of what rational decision-making requires (see for instance Anand, 1993).


    What kinds of decisions need a theory?

    Choice under uncertainty
    This area represents the heart of decision theory. The procedure now referred to as expected value was known from
    the 17th century. Blaise Pascal invoked it in his famous wager (see below), which is contained in his Pensées,
    published in 1670. The idea of expected value is that, when faced with a number of actions, each of which could give
    rise to more than one possible outcome with different probabilities, the rational procedure is to identify all possible
    outcomes, determine their values (positive or negative) and the probabilities that will result from each course of
    action, and multiply the two to give an expected value. The action to be chosen should be the one that gives rise to
    the highest total expected value. In 1738, Daniel Bernoulli published an influential paper entitled Exposition of a
    New Theory on the Measurement of Risk, in which he uses the St. Petersburg paradox to show that expected value
    theory must be normatively wrong. He also gives an example in which a Dutch merchant is trying to decide whether
    to insure a cargo being sent from Amsterdam to St Petersburg in winter, when it is known that there is a 5% chance
    that the ship and cargo will be lost. In his solution, he defines a utility function and computes expected utility rather
    than expected financial value (see[1] for a review).
    In the 20th century, interest was reignited by Abraham Wald's 1939 paper[2] pointing out that the two central
    procedures of sampling–distribution based statistical-theory, namely hypothesis testing and parameter estimation, are
    special cases of the general decision problem. Wald's paper renewed and synthesized many concepts of statistical
    theory, including loss functions, risk functions, admissible decision rules, antecedent distributions, Bayesian
    procedures, and minimax procedures. The phrase "decision theory" itself was used in 1950 by E. L. Lehmann.[3]
Decision theory                                                                                                                186


    The revival of subjective probability theory, from the work of Frank Ramsey, Bruno de Finetti, Leonard Savage and
    others, extended the scope of expected utility theory to situations where subjective probabilities can be used. At this
    time, von Neumann's theory of expected utility proved that expected utility maximization followed from basic
    postulates about rational behavior.
    The work of Maurice Allais and Daniel Ellsberg showed that human
    behavior has systematic and sometimes important departures from
    expected-utility maximization. The prospect theory of Daniel Kahneman
    and Amos Tversky renewed the empirical study of economic behavior with
    less emphasis on rationality presuppositions. Kahneman and Tversky found
    three regularities — in actual human decision-making, "losses loom larger
    than gains"; persons focus more on changes in their utility–states than they
    focus on absolute utilities; and the estimation of subjective probabilities is
    severely biased by anchoring.

    Castagnoli and LiCalzi (1996), Bordley and LiCalzi (2000) recently
    showed that maximizing expected utility is mathematically equivalent to
    maximizing the probability that the uncertain consequences of a decision
                                                                                               Daniel Kahneman
    are preferable to an uncertain benchmark (e.g., the probability that a mutual
    fund strategy outperforms the S&P 500 or that a firm outperforms the
    uncertain future performance of a major competitor.). This reinterpretation relates to psychological work suggesting
    that individuals have fuzzy aspiration levels (Lopes & Oden), which may vary from choice context to choice context.
    Hence it shifts the focus from utility to the individual's uncertain reference point.

    Pascal's Wager is a classic example of a choice under uncertainty. It is possible that the reward for belief is infinite
    (i.e. if God exists and is the sort of God worshiped by evangelical Christians). However, it is also possible that the
    reward for non-belief is infinite (i.e. if a capricious God exists that rewards us for not believing in God). Therefore,
    either believing in God or not believing in God, when you include these results, lead to infinite rewards and so we
    have no decision-theoretic reason to prefer one to the other. (There are several criticisms of the argument.)


    Intertemporal choice
    This area is concerned with the kind of choice where different actions lead to outcomes that are realised at different
    points in time. If someone received a windfall of several thousand dollars, they could spend it on an expensive
    holiday, giving them immediate pleasure, or they could invest it in a pension scheme, giving them an income at some
    time in the future. What is the optimal thing to do? The answer depends partly on factors such as the expected rates
    of interest and inflation, the person's life expectancy, and their confidence in the pensions industry. However even
    with all those factors taken into account, human behavior again deviates greatly from the predictions of prescriptive
    decision theory, leading to alternative models in which, for example, objective interest rates are replaced by
    subjective discount rates.


    Competing decision makers
    Some decisions are difficult because of the need to take into account how other people in the situation will respond
    to the decision that is taken. The analysis of such social decisions is more often treated under the label of game
    theory, rather than decision theory, though it involves the same mathematical methods. From the standpoint of game
    theory most of the problems treated in decision theory are one-player games (or the one player is viewed as playing
    against an impersonal background situation). In the emerging socio-cognitive engineering, the research is especially
    focused on the different types of distributed decision-making in human organizations, in normal and
    abnormal/emergency/crisis situations.
    Signal detection theory is based on decision theory.
Decision theory                                                                                                                187


    Complex decisions
    Other areas of decision theory are concerned with decisions that are difficult simply because of their complexity, or
    the complexity of the organization that has to make them. In such cases the issue is not the deviation between real
    and optimal behaviour, but the difficulty of determining the optimal behaviour in the first place. The Club of Rome,
    for example, developed a model of economic growth and resource usage that helps politicians make real-life
    decisions in complex situations.


    Alternatives to decision theory
    A highly controversial issue is whether one can replace the use of probability in decision theory by other alternatives.


    Probability theory
    The Advocates of probability theory point to:
    • the work of Richard Threlkeld Cox for justification of the probability axioms,
    • the Dutch book paradoxes of Bruno de Finetti as illustrative of the theoretical difficulties that can arise from
      departures from the probability axioms, and
    • the complete class theorems, which show that all admissible decision rules are equivalent to the Bayesian decision
      rule for some utility function and some prior distribution (or for the limit of a sequence of prior distributions).
      Thus, for every decision rule, either the rule may be reformulated as a Bayesian procedure, or there is a (perhaps
      limiting) Bayesian rule that is sometimes better and never worse.


    Alternatives to probability theory
    The proponents of fuzzy logic, possibility theory, Dempster–Shafer theory, and info-gap decision theory maintain
    that probability is only one of many alternatives and point to many examples where non-standard alternatives have
    been implemented with apparent success; notably, probabilistic decision theory is sensitive to assumptions about the
    probabilities of various events, while non-probabilistic rules such as minimax are robust, in that they do not make
    such assumptions.


    General criticism
    A general criticism of decision theory based on a fixed universe of possibilities is that it considers the "known
    unknowns", not the "unknown unknowns": it focuses on expected variations, not on unforeseen events, which some
    argue (as in black swan theory) have outsized impact and must be considered – significant events may be "outside
    model". This line of argument, called the ludic fallacy, is that there are inevitable imperfections in modeling the real
    world by particular models, and that unquestioning reliance on models blinds one to their limits.
Decision theory                                                                                                                                      188


    References
    [1] Schoemaker, P. J. H. (1982). "The Expected Utility Model: Its Variants, Purposes, Evidence and Limitations". Journal of Economic
        Literature 20: 529–563.
    [2] Wald, Abraham (1939). "Contributions to the Theory of Statistical Estimation and Testing Hypotheses". Annals of Mathematical Statistics 10
        (4): 299–326. doi:10.1214/aoms/1177732144. MR932.
    [3] Lehmann, E. L. (1950). "Some Principles of the Theory of Testing Hypotheses". Annals of Mathematical Statistics 21 (1): 1–26.
        doi:10.1214/aoms/1177729884. JSTOR 2236552.



    Further reading
    • Akerlof, George A., Yellen, Janet L. (May 1987). Rational Models of Irrational Behavior. 77. pp. 137–142.
    • Anand, Paul (1993). Foundations of Rational Choice Under Risk. Oxford: Oxford University Press.
      ISBN 0-19-823303-5. (an overview of the philosophical foundations of key mathematical axioms in subjective
      expected utility theory – mainly normative)
    • Arthur, W. Brian (May 1991). "Designing Economic Agents that Act like Human Agents: A Behavioral
      Approach to Bounded Rationality". The American Economic Review 81 (2): 353–9.
    • Berger, James O. (1985). Statistical decision theory and Bayesian Analysis (2nd ed.). New York:
      Springer-Verlag. ISBN 0-387-96098-8. MR0804611.
    • Bernardo, José M.; Smith, Adrian F. M. (1994). Bayesian Theory. Wiley. ISBN 0-471-92416-4. MR1274699.
    • Clemen, Robert (1996). Making Hard Decisions: An Introduction to Decision Analysis (2nd ed.). Belmont CA:
      Duxbury Press. ISBN 0-534-26035-7. (covers normative decision theory)
    • De Groot, Morris, Optimal Statistical Decisions. Wiley Classics Library. 2004. (Originally published 1970.)
      ISBN 0-471-68029-X.
    • Goodwin, Paul and Wright, George (2004). Decision Analysis for Management Judgment (3rd ed.). Chichester:
      Wiley. ISBN 0-470-86108-8. (covers both normative and descriptive theory)
    • Hansson, Sven Ove. "Decision Theory: A Brief Introduction" (http://www.infra.kth.se/~soh/decisiontheory.
      pdf) (PDF).
    • Khemani, Karan, Ignorance is Bliss: A study on how and why humans depend on recognition heuristics in social
      relationships, the equity markets and the brand market-place, thereby making successful decisions, 2005.
    • Miller L (1985). "Cognitive risk-taking after frontal or temporal lobectomy—I. The synthesis of fragmented
      visual information". Neuropsychologia 23 (3): 359–69. doi:10.1016/0028-3932(85)90022-3. PMID 4022303.
    • Miller L, Milner B (1985). "Cognitive risk-taking after frontal or temporal lobectomy—II. The synthesis of
      phonemic and semantic information". Neuropsychologia 23 (3): 371–9. doi:10.1016/0028-3932(85)90023-5.
      PMID 4022304.
    • North, D.W. (1968). "A tutorial introduction to decision theory". IEEE Transactions on Systems Science and
      Cybernetics 4 (3): 200–210. doi:10.1109/TSSC.1968.300114. Reprinted in Shafer & Pearl. (also about normative
      decision theory)
    • Peterson, Martin (2009). An Introduction to Decision Theory. Cambridge University Press.
      ISBN 978-0-521-71654-3.
    • Raiffa, Howard (1997). Decision Analysis: Introductory Readings on Choices Under Uncertainty. McGraw Hill.
      ISBN 0-07-052579-X.
    • Robert, Christian (2007). The Bayesian Choice (2nd ed.). New York: Springer. doi:10.1007/0-387-71599-1.
      ISBN 0-387-95231-4. MR1835885.
    • Shafer, Glenn and Pearl, Judea, ed. (1990). Readings in uncertain reasoning. San Mateo, CA: Morgan Kaufmann.
    • Smith, J.Q. (1988). Decision Analysis: A Bayesian Approach. Chapman and Hall. ISBN 0-412-27520-1.
    • Charles Sanders Peirce and Joseph Jastrow (1885). "On Small Differences in Sensation" (http://psychclassics.
      yorku.ca/Peirce/small-diffs.htm). Memoirs of the National Academy of Sciences 3: 73–83. http://psychclassics.
      yorku.ca/Peirce/small-diffs.htm
Decision theory                                                                                                                         189


    • Ramsey, Frank Plumpton; “Truth and Probability” ( PDF (http://cepa.newschool.edu/het//texts/ramsey/
      ramsess.pdf)), Chapter VII in The Foundations of Mathematics and other Logical Essays (1931).
    • de Finetti, Bruno (September 1989). "Probabilism: A Critical Essay on the Theory of Probability and on the Value
      of Science". Erkenntnis 31. (translation of 1931 article)
    • de Finetti, Bruno (1937). "La Prévision: ses lois logiques, ses sources subjectives". Annales de l'Institut Henri
      Poincaré.
          de Finetti, Bruno. "Foresight: its Logical Laws, Its Subjective Sources," (translation of the 1937 article (http:/ /
          www. numdam. org/ item?id=AIHP_1937__7_1_1_0) in French) in H. E. Kyburg and H. E. Smokler (eds),
          Studies in Subjective Probability, New York: Wiley, 1964.
    • de Finetti, Bruno. Theory of Probability, (translation by AFM Smith of 1970 book) 2 volumes, New York: Wiley,
      1974-5.
    • Donald Davidson, Patrick Suppes and Sidney Siegel (1957). Decision-Making: An Experimental Approach.
      Stanford University Press.
    • Pfanzagl, J (1967). "Subjective Probability Derived from the Morgenstern-von Neumann Utility Theory". In
      Martin Shubik. Essays in Mathematical Economics In Honor of Oskar Morgenstern. Princeton University Press.
      pp. 237–251.
    • Pfanzagl, J. in cooperation with V. Baumann and H. Huber (1968). "Events, Utility and Subjective Probability".
      Theory of Measurement. Wiley. pp. 195–220.
    • Morgenstern, Oskar (1976). "Some Reflections on Utility". In Andrew Schotter. Selected Economic Writings of
      Oskar Morgenstern. New York University Press. pp. 65–70. ISBN 0-8147-7771-6.



    Neural network
    The term neural network was traditionally used to refer to a network
    or circuit of biological neurons.[1] The modern usage of the term often
    refers to artificial neural networks, which are composed of artificial
    neurons or nodes. Thus the term has two distinct usages:
    1. Biological neural networks are made up of real biological neurons
       that are connected or functionally related in a nervous system. In the
       field of neuroscience, they are often identified as groups of neurons
       that perform a specific physiological function in laboratory analysis.
    2. Artificial neural networks are composed of interconnecting artificial
       neurons (programming constructs that mimic the properties of
       biological neurons) for solving artificial intelligence problems
       without creating a model of a real system. Neural network
       algorithms abstract away the biological complexity by focusing on
       the most important information. The goal of artificial neural
       networks is good, or human-like, predictive ability.
                                                                                   Simplified view of a feedforward artificial neural
                                                                                                       network
Neural network                                                                                                                  190


    Overview
    A biological neural network is composed of a group or groups of chemically connected or functionally associated
    neurons. A single neuron may be connected to many other neurons and the total number of neurons and connections
    in a network may be extensive. Connections, called synapses, are usually formed from axons to dendrites, though
    dendrodendritic microcircuits[2] and other connections are possible. Apart from the electrical signaling, there are
    other forms of signaling that arise from neurotransmitter diffusion.
    Artificial intelligence and cognitive modeling try to simulate some properties of biological neural networks. While
    similar in their techniques, the former has the aim of solving particular tasks, while the latter aims to build
    mathematical models of biological neural systems.
    In the artificial intelligence field, artificial neural networks have been applied successfully to speech recognition,
    image analysis and adaptive control, in order to construct software agents (in computer and video games) or
    autonomous robots. Most of the currently employed artificial neural networks for artificial intelligence are based on
    statistical estimations, classification optimization and control theory.
    The cognitive modelling field involves the physical or mathematical modeling of the behavior of neural systems;
    ranging from the individual neural level (e.g. modeling the spike response curves of neurons to a stimulus), through
    the neural cluster level (e.g. modelling the release and effects of dopamine in the basal ganglia) to the complete
    organism (e.g. behavioral modelling of the organism's response to stimuli). Artificial intelligence, cognitive
    modelling, and neural networks are information processing paradigms inspired by the way biological neural systems
    process data.


    History of the neural network analogy
    In the brain, spontaneous order appears to arise out of decentralized networks of simple units (neurons).
    Neural network theory has served both to better identify how the neurons in the brain function and to provide the
    basis for efforts to create artificial intelligence. The preliminary theoretical base for contemporary neural networks
    was independently proposed by Alexander Bain[3] (1873) and William James[4] (1890). In their work, both thoughts
    and body activity resulted from interactions among neurons within the brain.
    For Bain,[3] every activity led to the firing of a certain set of neurons. When activities were repeated, the connections
    between those neurons strengthened. According to his theory, this repetition was what led to the formation of
    memory. The general scientific community at the time was skeptical of Bain’s[3] theory because it required what
    appeared to be an inordinate number of neural connections within the brain. It is now apparent that the brain is
    exceedingly complex and that the same brain “wiring” can handle multiple problems and inputs.
    James’s[4] theory was similar to Bain’s,[3] however, he suggested that memories and actions resulted from electrical
    currents flowing among the neurons in the brain. His model, by focusing on the flow of electrical currents, did not
    require individual neural connections for each memory or action.
    C. S. Sherrington[5] (1898) conducted experiments to test James’s theory. He ran electrical currents down the spinal
    cords of rats. However, instead of demonstrating an increase in electrical current as projected by James, Sherrington
    found that the electrical current strength decreased as the testing continued over time. Importantly, this work led to
    the discovery of the concept of habituation.
    McCulloch and Pitts[6] (1943) created a computational model for neural networks based on mathematics and
    algorithms. They called this model threshold logic. The model paved the way for neural network research to split
    into two distinct approaches. One approach focused on biological processes in the brain and the other focused on the
    application of neural networks to artificial intelligence.
    In the late 1940s psychologist Donald Hebb[7] created a hypothesis of learning based on the mechanism of neural
    plasticity that is now known as Hebbian learning. Hebbian learning is considered to be a 'typical' unsupervised
Neural network                                                                                                                  191


    learning rule and its later variants were early models for long term potentiation. These ideas started being applied to
    computational models in 1948 with Turing's B-type machines.
    Farley and Clark[8] (1954) first used computational machines, then called calculators, to simulate a Hebbian network
    at MIT. Other neural network computational machines were created by Rochester, Holland, Habit, and Duda[9]
    (1956).
    Rosenblatt[10] (1958) created the perceptron, an algorithm for pattern recognition based on a two-layer learning
    computer network using simple addition and subtraction. With mathematical notation, Rosenblatt also described
    circuitry not in the basic perceptron, such as the exclusive-or circuit, a circuit whose mathematical computation
    could not be processed until after the backpropagation algorithm was created by Werbos[11] (1975).
    The perceptron is essentially a linear classifier for classifying data                      specified by parameters
                       and an output function                   . Its parameters are adapted with an ad-hoc rule similar to
    stochastic steepest gradient descent. Because the inner product is a linear operator in the input space, the perceptron
    can only perfectly classify a set of data for which different classes are linearly separable in the input space, while it
    often fails completely for non-separable data. While the development of the algorithm initially generated some
    enthusiasm, partly because of its apparent relation to biological mechanisms, the later discovery of this inadequacy
    caused such models to be abandoned until the introduction of non-linear models into the field.
    Neural network research stagnated after the publication of machine learning research by Minsky and Papert[12]
    (1969). They discovered two key issues with the computational machines that processed neural networks. The first
    issue was that single-layer neural networks were incapable of processing the exclusive-or circuit. The second
    significant issue was that computers were not sophisticated enough to effectively handle the long run time required
    by large neural networks. Neural network research slowed until computers achieved greater processing power. Also
    key in later advances was the backpropogation algorithm which effectively solved the exclusive-or problem (Werbos
    1975).[11]
    The cognitron (1975) designed by Kunihiko Fukushima[13] was an early multilayered neural network with a training
    algorithm. The actual structure of the network and the methods used to set the interconnection weights change from
    one neural strategy to another, each with its advantages and disadvantages. Networks can propagate information in
    one direction only, or they can bounce back and forth until self-activation at a node occurs and the network settles on
    a final state. The ability for bi-directional flow of inputs between neurons/nodes was produced with adaptive
    resonance theory, the neocognitron and the Hopfield net, and specialization of these node layers for specific
    purposes was introduced through the first hybrid network.
    The parallel distributed processing of the mid-1980s became popular under the name connectionism. The text by
    Rumelhart and McClelland[14] (1986) provided a full exposition on the use of connectionism in computers to
    simulate neural processes.
    The rediscovery of the backpropagation algorithm was probably the main reason behind the repopularisation of
    neural networks after the publication of "Learning Internal Representations by Error Propagation" in 1986 (Though
    backpropagation itself dates from 1969). The original network utilized multiple layers of weight-sum units of the
    type                    , where was a sigmoid function or logistic function such as used in logistic regression.
    Training was done by a form of stochastic gradient descent. The employment of the chain rule of differentiation in
    deriving the appropriate parameter updates results in an algorithm that seems to 'backpropagate errors', hence the
    nomenclature. However, it is essentially a form of gradient descent. Determining the optimal parameters in a model
    of this type is not trivial, and local numerical optimization methods such as gradient descent can be sensitive to
    initialization because of the presence of local minima of the training criterion. In recent times, networks with the
    same architecture as the backpropagation network are referred to as multilayer perceptrons. This name does not
    impose any limitations on the type of algorithm used for learning.
    The backpropagation network generated much enthusiasm at the time and there was much controversy about whether
    such learning could be implemented in the brain or not, partly because a mechanism for reverse signaling was not
Neural network                                                                                                                    192


    obvious at the time, but most importantly because there was no plausible source for the 'teaching' or 'target' signal.
    However, since 2006, several unsupervised learning procedures have been proposed for neural networks with one or
    more layers, using so-called deep learning algorithms. These algorithms can be used to learn intermediate
    representations, with or without a target signal, that capture the salient features of the distribution of sensory signals
    arriving at each layer of the neural network.


    The brain, neural networks and computers
    Neural networks, as used in artificial intelligence, have traditionally
    been viewed as simplified models of neural processing in the brain,
    even though the relation between this model and brain biological
    architecture is debated, as it is not clear to what degree artificial neural
    networks mirror brain function.[16]
    A subject of current research in computational neuroscience is the
    question surrounding the degree of complexity and the properties that
    individual neural elements should have to reproduce something
    resembling animal cognition.
    Historically, computers evolved from the von Neumann model, which
    is based on sequential processing and execution of explicit instructions.
                                                                                       Computer simulation of the branching
    On the other hand, the origins of neural networks are based on efforts
                                                                                     architecture of the dendrites of pyramidal
    to model information processing in biological systems, which may rely                           neurons.
                                                                                                             [15]
    largely on parallel processing as well as implicit instructions based on
    recognition of patterns of 'sensory' input from external sources. In other words, at its very heart a neural network is a
    complex statistical processor (as opposed to being tasked to sequentially process and execute).

    Neural coding is concerned with how sensory and other information is represented in the brain by neurons. The main
    goal of studying neural coding is to characterize the relationship between the stimulus and the individual or ensemble
    neuronal responses and the relationship among electrical activity of the neurons in the ensemble.[17] It is thought that
    neurons can encode both digital and analog information.[18]


    Neural networks and artificial intelligence
    A neural network (NN), in the case of artificial neurons called artificial neural network (ANN) or simulated neural
    network (SNN), is an interconnected group of natural or artificial neurons that uses a mathematical or computational
    model for information processing based on a connectionistic approach to computation. In most cases an ANN is an
    adaptive system that changes its structure based on external or internal information that flows through the network.
    In more practical terms neural networks are non-linear statistical data modeling or decision making tools. They can
    be used to model complex relationships between inputs and outputs or to find patterns in data.
    However, the paradigm of neural networks - i.e., implicit, not explicit , learning is stressed - seems more to
    correspond to some kind of natural intelligence than to the traditional symbol-based Artificial Intelligence, which
    would stress, instead, rule-based learning.
Neural network                                                                                                              193


    Background
    An artificial neural network involves a network of simple processing elements (artificial neurons) which can exhibit
    complex global behavior, determined by the connections between the processing elements and element parameters.
    Artificial neurons were first proposed in 1943 by Warren McCulloch, a neurophysiologist, and Walter Pitts, a
    logician, who first collaborated at the University of Chicago.[19]
    One classical type of artificial neural network is the recurrent Hopfield net.
    In a neural network model simple nodes (which can be called by a number of names, including "neurons",
    "neurodes", "Processing Elements" (PE) and "units"), are connected together to form a network of nodes — hence
    the term "neural network". While a neural network does not have to be adaptive per se, its practical use comes with
    algorithms designed to alter the strength (weights) of the connections in the network to produce a desired signal
    flow.
    In modern software implementations of artificial neural networks the approach inspired by biology has more or less
    been abandoned for a more practical approach based on statistics and signal processing. In some of these systems,
    neural networks, or parts of neural networks (such as artificial neurons), are used as components in larger systems
    that combine both adaptive and non-adaptive elements.
    The concept of a neural network appears to have first been proposed by Alan Turing in his 1948 paper "Intelligent
    Machinery".

    Applications of natural and of artificial neural networks
    The utility of artificial neural network models lies in the fact that they can be used to infer a function from
    observations and also to use it. Unsupervised neural networks can also be used to learn representations of the input
    that capture the salient characteristics of the input distribution, e.g., see the Boltzmann machine (1983), and more
    recently, deep learning algorithms, which can implicitly learn the distribution function of the observed data.
    Learning in neural networks is particularly useful in applications where the complexity of the data or task makes the
    design of such functions by hand impractical.
    The tasks to which artificial neural networks are applied tend to fall within the following broad categories:
    • Function approximation, or regression analysis, including time series prediction and modeling.
    • Classification, including pattern and sequence recognition, novelty detection and sequential decision making.
    • Data processing, including filtering, clustering, blind signal separation and compression.
    Application areas of ANNs include system identification and control (vehicle control, process control), game-playing
    and decision making (backgammon, chess, racing), pattern recognition (radar systems, face identification, object
    recognition), sequence recognition (gesture, speech, handwritten text recognition), medical diagnosis, financial
    applications, data mining (or knowledge discovery in databases, "KDD"), visualization and e-mail spam filtering.


    Neural networks and neuroscience
    Theoretical and computational neuroscience is the field concerned with the theoretical analysis and computational
    modeling of biological neural systems. Since neural systems are intimately related to cognitive processes and
    behaviour, the field is closely related to cognitive and behavioural modeling.
    The aim of the field is to create models of biological neural systems in order to understand how biological systems
    work. To gain this understanding, neuroscientists strive to make a link between observed biological processes (data),
    biologically plausible mechanisms for neural processing and learning (biological neural network models) and theory
    (statistical learning theory and information theory).
Neural network                                                                                                                    194


    Types of models
    Many models are used; defined at a different levels of abstraction, and modeling different aspects of neural systems.
    They range from models of the short-term behaviour of individual neurons, through models of the dynamics of
    neural circuitry arising from interactions between individual neurons, to models of behaviour arising from abstract
    neural modules that represent complete subsystems. These include models of the long-term and short-term plasticity
    of neural systems and its relation to learning and memory, from the individual neuron to the system level.


    Current research
    While initially research had been concerned mostly with the electrical characteristics of neurons, a particularly
    important part of the investigation in recent years has been the exploration of the role of neuromodulators such as
    dopamine, acetylcholine, and serotonin on behaviour and learning.
    Biophysical models, such as BCM theory, have been important in understanding mechanisms for synaptic plasticity,
    and have had applications in both computer science and neuroscience. Research is ongoing in understanding the
    computational algorithms used in the brain, with some recent biological evidence for radial basis networks and
    neural backpropagation as mechanisms for processing data.
    Computational devices have been created in CMOS for both biophysical simulation and neuromorphic computing.
    More recent efforts show promise for creating nanodevices[20] for very large scale principal components analyses
    and convolution. If successful, these efforts could usher in a new era of neural computing[21] that is a step beyond
    digital computing, because it depends on learning rather than programming and because it is fundamentally analog
    rather than digital even though the first instantiations may in fact be with CMOS digital devices.


    Architecture
    The basic architecture consists of three types of neuron layers: input, hidden, and output. In feed-forward networks,
    the signal flow is from input to output units, strictly in a feed-forward direction. The data processing can extend over
    multiple layers of units, but no feedback connections are present. Recurrent networks contain feedback connections.
    Contrary to feed-forward networks, the dynamical properties of the network are important. In some cases, the
    activation values of the units undergo a relaxation process such that the network will evolve to a stable state in which
    these activations do not change anymore.
    In other applications, the changes of the activation values of the output neurons are significant, such that the
    dynamical behavior constitutes the output of the network. Other neural network architectures include adaptive
    resonance theory maps and competitive networks.


    Criticism
    A common criticism of neural networks, particularly in robotics, is that they require a large diversity of training for
    real-world operation. This is not surprising, since any learning machine needs sufficient representative examples in
    order to capture the underlying structure that allows it to generalize to new cases. Dean Pomerleau, in his research
    presented in the paper "Knowledge-based Training of Artificial Neural Networks for Autonomous Robot Driving,"
    uses a neural network to train a robotic vehicle to drive on multiple types of roads (single lane, multi-lane, dirt, etc.).
    A large amount of his research is devoted to (1) extrapolating multiple training scenarios from a single training
    experience, and (2) preserving past training diversity so that the system does not become overtrained (if, for
    example, it is presented with a series of right turns – it should not learn to always turn right). These issues are
    common in neural networks that must decide from amongst a wide variety of responses, but can be dealt with in
    several ways, for example by randomly shuffling the training examples, by using a numerical optimization algorithm
    that does not take too large steps when changing the network connections following an example, or by grouping
    examples in so-called mini-batches.
Neural network                                                                                                                195


    A. K. Dewdney, a former Scientific American columnist, wrote in 1997, "Although neural nets do solve a few toy
    problems, their powers of computation are so limited that I am surprised anyone takes them seriously as a general
    problem-solving tool." (Dewdney, p. 82)
    Arguments for Dewdney's position are that to implement large and effective software neural networks, much
    processing and storage resources need to be committed. While the brain has hardware tailored to the task of
    processing signals through a graph of neurons, simulating even a most simplified form on Von Neumann technology
    may compel a NN designer to fill many millions of database rows for its connections - which can consume vast
    amounts of computer memory and hard disk space. Furthermore, the designer of NN systems will often need to
    simulate the transmission of signals through many of these connections and their associated neurons - which must
    often be matched with incredible amounts of CPU processing power and time. While neural networks often yield
    effective programs, they too often do so at the cost of efficiency (they tend to consume considerable amounts of time
    and money).
    Arguments against Dewdney's position are that neural nets have been successfully used to solve many complex and
    diverse tasks, ranging from autonomously flying aircraft [22] to detecting credit card fraud .
    Technology writer Roger Bridgman commented on Dewdney's statements about neural nets:
          Neural networks, for instance, are in the dock not only because they have been hyped to high heaven, (what
          hasn't?) but also because you could create a successful net without understanding how it worked: the bunch of
          numbers that captures its behaviour would in all probability be "an opaque, unreadable table...valueless as a
          scientific resource".
          In spite of his emphatic declaration that science is not technology, Dewdney seems here to pillory neural nets
          as bad science when most of those devising them are just trying to be good engineers. An unreadable table that
          a useful machine could read would still be well worth having.[23]
    In response to this kind of criticism, one should note that although it is true that analyzing what has been learned by
    an artificial neural network is difficult, it is much easier to do so than to analyze what has been learned by a
    biological neural network. Furthermore, researchers involved in exploring learning algorithms for neural networks
    are gradually uncovering generic principles which allow a learning machine to be successful. For example, Bengio
    and LeCun (2007) wrote an article regarding local vs non-local learning, as well as shallow vs deep architecture [24].
    Some other criticisms came from believers of hybrid models (combining neural networks and symbolic approaches).
    They advocate the intermix of these two approaches and believe that hybrid models can better capture the
    mechanisms of the human mind (Sun and Bookman, 1990).


    Recent improvements
    Between 2009 and 2012, the recurrent neural networks and deep feedforward neural networks developed in the
    research group of Jürgen Schmidhuber at the Swiss AI Lab IDSIA have won eight international competitions in
    pattern recognition and machine learning.[25] For example, multi-dimensional long short term memory
    (LSTM)[26][27] won three competitions in connected handwriting recognition at the 2009 International Conference
    on Document Analysis and Recognition (ICDAR), without any prior knowledge about the three different languages
    to be learned.
    Variants of the back-propagation algorithm as well as unsupervised methods by Geoff Hinton and colleagues at the
    University of Toronto[28][29] can be used to train deep, highly nonlinear neural architectures similar to the 1980
    Neocognitron by Kunihiko Fukushima,[30] and the "standard architecture of vision",[31] inspired by the simple and
    complex cells identified by David H. Hubel and Torsten Wiesel in the primary visual cortex.
    Deep learning feedforward networks alternate convolutional layers and max-pooling layers, topped by several pure
    classification layers. Fast GPU-based implementations of this approach have won several pattern recognition
    contests, including the IJCNN 2011 Traffic Sign Recognition Competition[32] and the ISBI 2012 Segmentation of
Neural network                                                                                                                                            196


    Neuronal Structures in Electron Microscopy Stacks challenge.[33] Such neural networks also were the first artificial
    pattern recognizers to achieve human-competitive or even superhuman performance[34] on benchmarks such as
    traffic sign recognition (IJCNN 2012), or the MNIST handwritten digits problem of Yann LeCun and colleagues at
    NYU.


    References
    [1] J. J. HOPFIELD Neural networks and physical systems with emergent collective computational abilities. Proc. NatL Acad. Sci. USA Vol. 79,
        pp. 2554-2558, April 1982 Biophysics (http:/ / www. pnas. org/ content/ 79/ 8/ 2554. full. pdf)
    [2] Arbib, p.666
    [3] Bain (1873). Mind and Body: The Theories of Their Relation. New York: D. Appleton and Company.
    [4] James (1890). The Principles of Psychology. New York: H. Holt and Company.
    [5] Sherrington, C.S.. "Experiments in Examination of the Peripheral Distribution of the Fibers of the Posterior Roots of Some Spinal Nerves".
        Proceedings of the Royal Society of London 190: 45–186.
    [6] McCulloch, Warren; Walter Pitts (1943). "A Logical Calculus of Ideas Immanent in Nervous Activity". Bulletin of Mathematical Biophysics
        5 (4): 115–133. doi:10.1007/BF02478259.
    [7] Hebb, Donald (1949). The Organization of Behavior. New York: Wiley.
    [8] Farley, B; W.A. Clark (1954). "Simulation of Self-Organizing Systems by Digital Computer". IRE Transactions on Information Theory 4 (4):
        76–84. doi:10.1109/TIT.1954.1057468.
    [9] Rochester, N.; J.H. Holland, L.H. Habit, and W.L. Duda (1956). "Tests on a cell assembly theory of the action of the brain, using a large
        digital computer". IRE Transactions on Information Theory 2 (3): 80–93. doi:10.1109/TIT.1956.1056810.
    [10] Rosenblatt, F. (1958). "The Perceptron: A Probalistic Model For Information Storage And Organization In The Brain". Psychological
        Review 65 (6): 386–408. doi:10.1037/h0042519. PMID 13602029.
    [11] Werbos, P.J. (1975). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences.
    [12] Minsky, M.; S. Papert (1969). An Introduction to Computational Geometry. MIT Press. ISBN 0-262-63022-2.
    [13] Fukushima, Kunihiko (1975). "Cognitron: A self-organizing multilayered neural network". Biological Cybernetics 20 (3–4): 121–136.
        doi:10.1007/BF00342633. PMID 1203338.
    [14] Rumelhart, D.E; James McClelland (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Cambridge:
        MIT Press.
    [15] "PLoS Computational Biology Issue Image". PLoS Computational Biology 6 (8): ev06.ei08. 2010. doi:10.1371/image.pcbi.v06.i08.
    [16] Russell, Ingrid. "Neural Networks Module" (http:/ / uhaweb. hartford. edu/ compsci/ neural-networks-definition. html). . Retrieved 2012.
    [17] Brown EN, Kass RE, Mitra PP. (2004). "Multiple neural spike train data analysis: state-of-the-art and future challenges". Nature
        Neuroscience 7 (5): 456–61. doi:10.1038/nn1228. PMID 15114358.
    [18] Spike arrival times: A highly efficient coding scheme for neural networks (http:/ / pop. cerco. ups-tlse. fr/ fr_vers/ documents/
        thorpe_sj_90_91. pdf), SJ Thorpe - Parallel processing in neural systems, 1990
    [19] McCulloch, Warren; Pitts, Walter, "A Logical Calculus of Ideas Immanent in Nervous Activity", 1943, Bulletin of Mathematical Biophysics
        5:115-133.
    [20] Yang, J. J.; Pickett, M. D.; Li, X. M.; Ohlberg, D. A. A.; Stewart, D. R.; Williams, R. S. Nat. Nanotechnol. 2008, 3, 429–433.
    [21] Strukov, D. B.; Snider, G. S.; Stewart, D. R.; Williams, R. S. Nature 2008, 453, 80–83.
    [22] http:/ / www. nasa. gov/ centers/ dryden/ news/ NewsReleases/ 2003/ 03-49. html
    [23] Roger Bridgman's defence of neural networks (http:/ / members. fortunecity. com/ templarseries/ popper. html)
    [24] http:/ / www. iro. umontreal. ca/ ~lisa/ publications2/ index. php/ publications/ show/ 4
    [25] http:/ / www. kurzweilai. net/ how-bio-inspired-deep-learning-keeps-winning-competitions 2012 Kurzweil AI Interview with Jürgen
        Schmidhuber on the eight competitions won by his Deep Learning team 2009-2012
    [26] Graves, Alex; and Schmidhuber, Jürgen; Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks, in Bengio,
        Yoshua; Schuurmans, Dale; Lafferty, John; Williams, Chris K. I.; and Culotta, Aron (eds.), Advances in Neural Information Processing
        Systems 22 (NIPS'22), December 7th–10th, 2009, Vancouver, BC, Neural Information Processing Systems (NIPS) Foundation, 2009, pp.
        545–552
    [27] A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J. Schmidhuber. A Novel Connectionist System for Improved Unconstrained
        Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009.
    [28] http:/ / www. scholarpedia. org/ article/ Deep_belief_networks /
    [29] Hinton, G. E.; Osindero, S.; Teh, Y. (2006). "A fast learning algorithm for deep belief nets" (http:/ / www. cs. toronto. edu/ ~hinton/ absps/
        fastnc. pdf). Neural Computation 18 (7): 1527–1554. doi:10.1162/neco.2006.18.7.1527. PMID 16764513. .
    [30] K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in
        position. Biological Cybernetics, 36(4): 93-202, 1980.
    [31] M Riesenhuber, T Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 1999.
    [32] D. C. Ciresan, U. Meier, J. Masci, J. Schmidhuber. Multi-Column Deep Neural Network for Traffic Sign Classification. Neural Networks,
        2012.
Neural network                                                                                                                                197

    [33] D. Ciresan, A. Giusti, L. Gambardella, J. Schmidhuber. Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy
        Images. In Advances in Neural Information Processing Systems (NIPS 2012), Lake Tahoe, 2012.
    [34] D. C. Ciresan, U. Meier, J. Schmidhuber. Multi-column Deep Neural Networks for Image Classification. IEEE Conf. on Computer Vision
        and Pattern Recognition CVPR 2012.



    External links
    • A Brief Introduction to Neural Networks (D. Kriesel) (http://www.dkriesel.com/en/science/neural_networks)
      - Illustrated, bilingual manuscript about artificial neural networks; Topics so far: Perceptrons, Backpropagation,
      Radial Basis Functions, Recurrent Neural Networks, Self Organizing Maps, Hopfield Networks.
    • Review of Neural Networks in Materials Science (http://www.msm.cam.ac.uk/phase-trans/abstracts/neural.
      review.html)
    • Artificial Neural Networks Tutorial in three languages (Univ. Politécnica de Madrid) (http://www.gc.ssr.upm.
      es/inves/neural/ann1/anntutorial.html)
    • Another introduction to ANN (http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html)
    • Next Generation of Neural Networks (http://youtube.com/watch?v=AyzOUbkUf3M) - Google Tech Talks
    • Performance of Neural Networks (http://www.msm.cam.ac.uk/phase-trans/2009/performance.html)
    • Neural Networks and Information (http://www.msm.cam.ac.uk/phase-trans/2009/
      review_Bhadeshia_SADM.pdf)



    Econometric model
    Econometric models are statistical models used in econometrics. An econometric model specifies the statistical
    relationship that is believed to hold between the various economic quantities pertaining to a particular economic
    phenomenon under study. An econometric model can be derived from a deterministic economic model by allowing
    for uncertainty, or from an economic model which itself is stochastic. However, it is also possible to use econometric
    models that are not tied to any specific economic theory.[1]
    A simple example of an econometric model is one that assumes that monthly spending by consumers is linearly
    dependent on consumers' income in the previous month. Then the model will consist of the equation


    where Ct is consumer spending in month t, Yt-1 is income during the previous month, and et is an error term
    measuring the extent to which the model cannot fully explain consumption. Then one objective of the
    econometrician is to obtain estimates of the parameters a and b; these estimated parameter values, when used in the
    model's equation, enable predictions for future values of consumption to be made contingent on the prior month's
    income.


    Formal definition
    In econometrics, as in statistics in general, it is presupposed that the quantities being analyzed can be treated as
    random variables. An econometric model then is a set of joint probability distributions to which the true joint
    probability distribution of the variables under study is supposed to belong. In the case in which the elements of this
    set can be indexed by a finite number of real-valued parameters, the model is called a parametric model; otherwise it
    is a nonparametric or semiparametric model. A large part of econometrics is the study of methods for selecting
    models, estimating them, and carrying out inference on them.
    The most common econometric models are structural, in that they convey causal and counterfactual information,[2]
    and are used for policy evaluation. For example, an equation modeling consumption spending based on income
    could be used to see what consumption would be contingent on any of various hypothetical levels of income, only
Econometric model                                                                                                                         198


    one of which (depending on the choice of a fiscal policy) will end up actually occurring.


    Basic models
    Some of the common econometric models are:
    •   Linear regression
    •   Generalized linear models
    •   Tobit
    •   ARIMA
    •   Vector Autoregression
    •   Cointegration


    Use in policy-making
    Comprehensive models of macroeconomic relationships are used by central banks and governments to evaluate and
    guide economic policy. One famous econometric models of this nature are the Federal Reserve Bank econometric
    model.


    References
    [1] Sims, Christopher A. (Jan., 1980). "Macroeconomics and Reality". Econometrica 48 (1): 1–48. doi:10.2307/1912017. JSTOR 1912017.
    [2] Pearl, J. Causality: Models, Reasoning, and Inference, Cambridge University Press, 2000.

    • Granger, Clive (1991). Modelling Economic Series: Readings in Econometric Methodology. Oxford University
      Press. ISBN 978-0-19-828736-0.
    • Davidson, Russell; James G. MacKinnon (1993). Estimation and Inference in Econometrics. Oxford University
      Press. ISBN 978-0-19-506011-9.
    • Adrian, Pagan; Aman Ullah (1999). Nonparametric Econometrics. Cambridge University Press.
      ISBN 978-0-521-58611-5.


    External links
    • Manuscript of Bruce Hansen's book on Econometrics (http://www.ssc.wisc.edu/~bhansen/econometrics/)
Statistical model                                                                                                                 199



     Statistical model
     A statistical model is a formalization of relationships between variables in the form of mathematical equations. A
     statistical model describes how one or more random variables are related to one or more other variables. The model
     is statistical as the variables are not deterministically but stochastically related. In mathematical terms, a statistical
     model is frequently thought of as a pair              where      is the set of possible observations and    the set of
     possible probability distributions on     . It is assumed that there is a distinct element of      which generates the
     observed data. Statistical inference enables us to make statements about which element(s) of this set are likely to be
     the true one.
     Most statistical tests can be described in the form of a statistical model. For example, the Student's t-test for
     comparing the means of two groups can be formulated as seeing if an estimated parameter in the model is different
     from 0. Another similarity between tests and models is that there are assumptions involved. Error is assumed to be
     normally distributed in most models.[1]


     Formal definition
     A statistical model is a collection of probability distribution functions or probability density functions (collectively
     referred to as distributions for brevity). A parametric model is a collection of distributions, each of which is indexed
     by a unique finite-dimensional parameter:                             , where is a parameter and                 is the
     feasible region of parameters, which is a subset of d-dimensional Euclidean space. A statistical model may be used
     to describe the set of distributions from which one assumes that a particular data set is sampled. For example, if one
     assumes that data arise from a univariate Gaussian distribution, then one has assumed a Gaussian model:
                                                                                        .

     A non-parametric model is a set of probability distributions with infinite dimensional parameters, and might be
     written as                            . A semi-parametric model also has infinite dimensional parameters, but is
     not dense in the space of distributions. For example, a mixture of Gaussians with one Gaussian at each data point is
     dense in the space of distributions. Formally, if d is the dimension of the parameter, and n is the number of samples,
     if             as         and             as           , then the model is semi-parametric.


     Model comparison
     Models can be compared to each other. This can either be done when you have done an exploratory data analysis or a
     confirmatory data analysis. In an exploratory analysis, you formulate all models you can think of, and see which
     describes your data best. In a confirmatory analysis you test which of your models you have described before the
     data was collected fits the data best, or test if your only model fits the data. In linear regression analysis you can
     compare the amount of variance explained by the independent variables, R2, across the different models. In general,
     you can compare models that are nested by using a Likelihood-ratio test. Nested models are models that can be
     obtained by restricting a parameter in a more complex model to be zero.


     An example
     Length and age are probabilistically distributed over humans. They are stochastically related, when you know that a
     person is of age 7, this influences the chance of this person being 6 feet tall. You could formalize this relationship in
     a linear regression model of the following form: lengthi = b0 + b1agei + εi, where b0 is the intercept, b1 is a parameter
     that age is multiplied by to get a prediction of length, ε is the error term, and i is the subject. This means that length
     starts at some value, there is a minimum length when someone is born, and it is predicted by age to some amount.
     This prediction is not perfect as error is included in the model. This error contains variance that stems from sex and
Statistical model                                                                                                                                   200


     other variables. When sex is included in the model, the error term will become smaller, as you will have a better idea
     of the chance that a particular 16-year-old is 6 feet tall when you know this 16-year-old is a girl. The model would
     become lengthi = b0 + b1agei + b2sexi + εi, where the variable sex is dichotomous. This model would presumably
     have a higher R2. The first model is nested in the second model: the first model is obtained from the second when b2
     is restricted to zero.


     Classification
     According to the number of the endogenous variables and the number of equations, models can be classified as
     complete models (the number of equations equals to the number of endogenous variables) and incomplete models.
     Some other statistical models are the general linear model (restricted to continuous dependent variables), the
     generalized linear model (for example, logistic regression), the multilevel model, and the structural equation
     model.[2]


     References
     [1] Field, A. (2005). Discovering statistics using SPSS. Sage, London.
     [2] Adèr, H.J. (2008). Chapter 12: Modelling. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research
         Methods: A consultant's companion (pp. 271-304). Huizen, The Netherlands: Johannes van Kessel Publishing.




     Model selection
     Model selection is the task of selecting a statistical model from a set of candidate models, given data. In the simplest
     cases, a pre-existing set of data is considered. However, the task can also involve the design of experiments such that
     the data collected is well-suited to the problem of model selection. Given candidate models of similar predictive or
     explanatory power, the simplest model is most likely to be correct.[1]


     Introduction
     In its most basic forms, model selection is one of the fundamental tasks
     of scientific inquiry. Determining the principle that explains a series of
     observations is often linked directly to a mathematical model
     predicting those observations. For example, when Galileo performed
     his inclined plane experiments, he demonstrated that the motion of the
     balls fitted the parabola predicted by his model.

     Of the countless number of possible mechanisms and processes that
     could have produced the data, how can one even begin to choose the
     best model? The mathematical approach commonly taken decides
     among a set of candidate models; this set must be chosen by the
     researcher. Often simple models such as polynomials are used, at least
     initially. Burnham and Anderson (2002) emphasize the importance of                                  The scientific observation cycle.

     choosing models based on sound scientific principles, modeling the
     underlying data throughout their book.

     Once the set of candidate models has been chosen, the mathematical analysis allows us to select the best of these
     models. What is meant by best is controversial. A good model selection technique will balance goodness of fit with
     simplicity. More complex models will be better able to adapt their shape to fit the data (for example, a fifth-order
     polynomial can exactly fit six points), but the additional parameters may not represent anything useful. (Perhaps
Model selection                                                                                                                 201


    those six points are really just randomly distributed about a straight line.) Goodness of fit is generally determined
    using a likelihood ratio approach, or an approximation of this, leading to a chi-squared test. The complexity is
    generally measured by counting the number of parameters in the model.
    Model selection techniques can be considered as estimators of some physical quantity, such as the probability of the
    model producing the given data. The bias and variance are both important measures of the quality of this estimator.
    Asymptotic efficiency is also often considered.
    A standard example of model selection is that of curve fitting, where, given a set of points and other background
    knowledge (e.g. points are a result of i.i.d. samples), we must select a curve that describes the function that generated
    the points.


    Methods for choosing the set of candidate models
    • Scientific method
    • Statistical hypothesis testing


    Experiments for choosing the set of candidate models
    • Design of experiments
        • Optimal design
        • Fractional factorial design (Screening experiments)


    Criteria for model selection
    •   Akaike information criterion
    •   Bayes factor
    •   Bayesian information criterion
    •   Deviance information criterion
    •   False discovery rate
    •   Focused information criterion
    •   Mallows' Cp
    •   Minimum description length (Algorithmic information theory)
    •   Minimum message length (Algorithmic information theory)
    •   Structural Risk Minimization
    •   Stepwise regression
    •   Cross-validation


    References
    [1] This follows directly from formal expressions of Occam's Razor such as Minimum Message Length and others.

    • Anderson, D.R. (2008), Model Based Inference in the Life Sciences, Springer.
    • Aznar Grasa, A. (1989). Econometric Model Selection: A New Approach, Springer. ISBN 978-0-7923-0321-3
    • Burnham, K.P., and Anderson, D.R. (2002). Model Selection and Multimodel Inference: A Practical
      Information-Theoretic Approach, 2nd ed. Springer-Verlag. ISBN 0-387-95364-7 [This has over 16000 citations
      on Google Scholar.]
    • Chamberlin, T.C. (1890) "The method of multiple working hypotheses", Science 15: 93. (Reprinted 1965, Science
      148: 754–759 (http://www.d.umn.edu/~mille066/Teaching/3000/Chamberlin-MWH.pdf).)
    • Claeskens, G., and Hjort, N.L. (2008). Model Selection and Model Averaging, Cambridge University Press.
Model selection                                                                                                             202


    • Phil C. Gregory (http://www.physics.ubc.ca/~gregory/gregory.html) (2005) Bayesian logical data analysis
      for the physical sciences: A comparative approach with Mathematica support (Cambridge U. Press, Cambridge
      UK) preview (http://books.google.com/books?id=yJ_5VFo0zGMC).
    • Lahiri, P. (2001). Model Selection, Institute of Mathematical Statistics.
    • Massart, P. (2007). Concentration Inequalities and Model Selection, Springer.



    Economic forecasting
    Economic forecasting is the process of making predictions about the economy. Forecasts can be carried out at a
    high level of aggregation - for example for GDP, inflation, unemployment or the fiscal deficit - or at a more
    disaggregated level, for specific sectors of the economy or even specific firms.
    Many institutions engage in forecasting, including international organisations such as the IMF [1], World Bank [2]
    and the OECD [3], national governments and central banks, and private sector entities, be they think tanks, banks or
    others. Some forecasts are produced annually, but many are updated more frequently.
    The World Bank model is available for individuals and organizations to run their own simulations and forecasts
    using its iSimulate platform [4].
    Consensus Economics Inc [5]., among others, compiles the macroeconomic forecasts prepared by a variety of
    forecasters, and publishes them every month. The Economist magazine [6] regularly provides such a snapshot as well,
    for a broad range of countries (a recent example is provided here [7]).
    The financial and economic crisis that erupted in 2007 - arguably the worst since the Great Depression of the 1930s -
    was not foreseen by most of the forecasters, even if a few lone analysts had been crying wolf for some time (for
    example, Nouriel Roubini and Robert Shiller). The failure to forecast the "Great Recession" has caused a lot of soul
    searching in the profession. The Queen of England herself asked why had nobody noticed that the credit crunch was
    on its way, and a group of economists - experts from business, the City, its regulators, academia, and government -
    tried to explain in a letter.[8]
    Methods of forecasting include Econometric models, Economic base analysis, Shift-share analysis, Input-output
    model and the Grinold and Kroner Model.
    See also Land use forecasting, Reference class forecasting, Transportation planning, Calculating Demand Forecast
    Accuracy and Consensus forecasts


    References
    • IMF forecasts can be found here: IMF World Economic Outlook [9], World Bank forecasts are here World Bank
      Global Economic Prospects [10], and OECD forecasts here: Economic Outlook [11]
    • Two of the leading journals in the field of economic forecasting are the Journal of Forecasting [12] and the
      International Journal of Forecasting [13]
    • For a comprehensive but quite technical compendium, see Handbook of Economic Forecasting [14],
      North-Holland: Elsevier, 2006
    • A more compact and more accessible, but pre-crisis overview is provided in Elements of Forecasting [15], 4th edn,
      Cincinnati, OH: South-Western College Publishing, 2007
    • A recent, comprehensive and accessible guide to forecasting is Economic Forecasting and Policy [16], 2d edn,
      Palgrave, 2011
Economic forecasting                                                                                                                                  203


    External links
    • International Institute of Forecasters [17]
    • How to predict the economy? [18]


    Note
    [1] http:/ / www. imf. org/ external/ index. htm
    [2] http:/ / www. worldbank. org
    [3] http:/ / www. oecd. org
    [4] http:/ / isimulate. worldbank. org
    [5] http:/ / www. consensuseconomics. com/
    [6] http:/ / www. economist. com
    [7] http:/ / www. economist. com/ node/ 18959289?story_id=18959289
    [8] http:/ / www. ft. com/ intl/ cms/ 3e3b6ca8-7a08-11de-b86f-00144feabdc0. pdf
    [9] http:/ / www. imf. org/ external/ ns/ cs. aspx?id=29
    [10] http:/ / www. worldbank. org/ globaloutlook
    [11] http:/ / www. oecd. org/ eco/ economicoutlook. htm''OECD
    [12] http:/ / onlinelibrary. wiley. com/ journal/ 10. 1002/ (ISSN)1099-131X
    [13] http:/ / www. elsevier. com/ wps/ find/ journaldescription. cws_home/ 505555/ description
    [14] http:/ / www. elsevier. com/ wps/ find/ bookdescription. cws_home/ 672846/ description#description
    [15]   http:/ / www. amazon. com/ Elements-Forecasting-Francis-X-Diebold/ dp/ 0538862440
    [16]   http:/ / us. macmillan. com/ economicforecastingandpolicy
    [17]   http:/ / www. forecasters. org/
    [18]   http:/ / www. simonspicks. com/ economics/ howtopredicttheeconomy




    Mathematical optimization
    In mathematics, statistics, empirical sciences, computer science, or
    management science, mathematical optimization (alternatively,
    optimization or mathematical programming) is the selection of a
    best element (with regard to some criteria) from some set of available
    alternatives.[1]
    In the simplest case, an optimization problem consists of maximizing
    or minimizing a real function by systematically choosing input values
    from within an allowed set and computing the value of the function.
    The generalization of optimization theory and techniques to other                                     Graph of a paraboloid given by f(x,y) =
    formulations comprises a large area of applied mathematics. More                                   -(x²+y²)+4. The global maximum at (0,0,4) is
    generally, optimization includes finding "best available" values of                                            indicated by a red dot.

    some objective function given a defined domain, including a variety of
    different types of objective functions and different types of domains.
Mathematical optimization                                                                                                         204


    Optimization problems
    An optimization problem can be represented in the following way
          Given: a function f : A      R from some set A to the real numbers
          Sought: an element x0 in A such that f(x0) ≤ f(x) for all x in A ("minimization") or such that f(x0) ≥ f(x) for all x
          in A ("maximization").
    Such a formulation is called an optimization problem or a mathematical programming problem (a term not
    directly related to computer programming, but still in use for example in linear programming - see History below).
    Many real-world and theoretical problems may be modeled in this general framework. Problems formulated using
    this technique in the fields of physics and computer vision may refer to the technique as energy minimization,
    speaking of the value of the function f as representing the energy of the system being modeled.
    Typically, A is some subset of the Euclidean space Rn, often specified by a set of constraints, equalities or
    inequalities that the members of A have to satisfy. The domain A of f is called the search space or the choice set,
    while the elements of A are called candidate solutions or feasible solutions.
    The function f is called, variously, an objective function, cost function (minimization),[2] indirect utility function
    (minimization),[3] utility function (maximization), or, in certain fields, energy function, or energy functional. A
    feasible solution that minimizes (or maximizes, if that is the goal) the objective function is called an optimal
    solution.
    By convention, the standard form of an optimization problem is stated in terms of minimization. Generally, unless
    both the objective function and the feasible region are convex in a minimization problem, there may be several local
    minima, where a local minimum x* is defined as a point for which there exists some δ > 0 so that for all x such that


    the expression


    holds; that is to say, on some region around x* all of the function values are greater than or equal to the value at that
    point. Local maxima are defined similarly.
    A large number of algorithms proposed for solving non-convex problems – including the majority of commercially
    available solvers – are not capable of making a distinction between local optimal solutions and rigorous optimal
    solutions, and will treat the former as actual solutions to the original problem. The branch of applied mathematics
    and numerical analysis that is concerned with the development of deterministic algorithms that are capable of
    guaranteeing convergence in finite time to the actual optimal solution of a non-convex problem is called global
    optimization.


    Notation
    Optimization problems are often expressed with special notation. Here are some examples.


    Minimum and maximum value of a function
    Consider the following notation:



    This denotes the minimum value of the objective function               , when choosing x from the set of real numbers
       . The minimum value in this case is     , occurring at        .
    Similarly, the notation
Mathematical optimization                                                                                                   205


    asks for the maximum value of the objective function 2x, where x may be any real number. In this case, there is no
    such maximum as the objective function is unbounded, so the answer is "infinity" or "undefined".


    Optimal input arguments
    Consider the following notation:



    or equivalently



    This represents the value (or values) of the argument x in the interval            that minimizes (or minimize) the
    objective function x2 + 1 (the actual minimum value of that function is not what the problem asks for). In this case,
    the answer is x = -1, since x = 0 is infeasible, i.e. does not belong to the feasible set.
    Similarly,



    or equivalently



    represents the          pair (or pairs) that maximizes (or maximize) the value of the objective function           ,
    with the added constraint that x lie in the interval        (again, the actual maximum value of the expression does
    not matter). In this case, the solutions are the pairs of the form (5, 2kπ) and (−5,(2k+1)π), where k ranges over all
    integers.
    Arg min and arg max are sometimes also written argmin and argmax, and stand for argument of the minimum
    and argument of the maximum.


    History
    Fermat and Lagrange found calculus-based formulas for identifying optima, while Newton and Gauss proposed
    iterative methods for moving towards an optimum. Historically, the first term for optimization was "linear
    programming", which was due to George B. Dantzig, although much of the theory had been introduced by Leonid
    Kantorovich in 1939. Dantzig published the Simplex algorithm in 1947, and John von Neumann developed the
    theory of duality in the same year.
    The term programming in this context does not refer to computer programming. Rather, the term comes from the use
    of program by the United States military to refer to proposed training and logistics schedules, which were the
    problems Dantzig studied at that time.
    Later important researchers in mathematical optimization include the following:
Mathematical optimization                                                                                                         206


                                           •   Richard Bellman       •   Arkadi Nemirovski
                                           •   Ronald A. Howard      •   Yurii Nesterov
                                           •   Narendra Karmarkar    •   Boris Polyak
                                           •   William Karush        •   Lev Pontryagin
                                           •   Leonid Khachiyan      •   James Renegar
                                           •   Bernard Koopman       •   R. Tyrrell Rockafellar
                                           •   Harold Kuhn           •   Cornelis Roos
                                           •   Joseph Louis Lagrange •   Naum Z. Shor
                                           •   László Lovász         •   Michael J. Todd
                                                                     •   Albert Tucker



    Major subfields
    • Convex programming studies the case when the objective function is convex (minimization) or concave
      (maximization) and the constraint set is convex. This can be viewed as a particular case of nonlinear
      programming or as generalization of linear or convex quadratic programming.
        • Linear programming (LP), a type of convex programming, studies the case in which the objective function f is
           linear and the set of constraints is specified using only linear equalities and inequalities. Such a set is called a
           polyhedron or a polytope if it is bounded.
        • Second order cone programming (SOCP) is a convex program, and includes certain types of quadratic
           programs.
        • Semidefinite programming (SDP) is a subfield of convex optimization where the underlying variables are
           semidefinite matrices. It is generalization of linear and convex quadratic programming.
        • Conic programming is a general form of convex programming. LP, SOCP and SDP can all be viewed as conic
           programs with the appropriate type of cone.
        • Geometric programming is a technique whereby objective and inequality constraints expressed as posynomials
           and equality constraints as monomials can be transformed into a convex program.
    •   Integer programming studies linear programs in which some or all variables are constrained to take on integer
        values. This is not convex, and in general much more difficult than regular linear programming.
    •   Quadratic programming allows the objective function to have quadratic terms, while the feasible set must be
        specified with linear equalities and inequalities. For specific forms of the quadratic term, this is a type of convex
        programming.
    •   Fractional programming studies optimization of ratios of two nonlinear functions. The special class of concave
        fractional programs can be transformed to a convex optimization problem.
    •   Nonlinear programming studies the general case in which the objective function or the constraints or both contain
        nonlinear parts. This may or may not be a convex program. In general, whether the program is convex affects the
        difficulty of solving it.
    •   Stochastic programming studies the case in which some of the constraints or parameters depend on random
        variables.
    •   Robust programming is, like stochastic programming, an attempt to capture uncertainty in the data underlying the
        optimization problem. This is not done through the use of random variables, but instead, the problem is solved
        taking into account inaccuracies in the input data.
    •   Combinatorial optimization is concerned with problems where the set of feasible solutions is discrete or can be
        reduced to a discrete one.
    •   Stochastic optimization for use with random (noisy) function measurements or random inputs in the search
        process.
Mathematical optimization                                                                                                      207


    • Infinite-dimensional optimization studies the case when the set of feasible solutions is a subset of an
      infinite-dimensional space, such as a space of functions.
    • Heuristics and metaheuristics make few or no assumptions about the problem being optimized. Usually, heuristics
      do not guarantee that any optimal solution need be found. On the other hand, heuristics are used to find
      approximate solutions for many complicated optimization problems.
    • Constraint satisfaction studies the case in which the objective function f is constant (this is used in artificial
      intelligence, particularly in automated reasoning).
      • Constraint programming.
    • Disjunctive programming is used where at least one constraint must be satisfied but not all. It is of particular use
      in scheduling.
    In a number of subfields, the techniques are designed primarily for optimization in dynamic contexts (that is,
    decision making over time):
    • Calculus of variations seeks to optimize an objective defined over many points in time, by considering how the
      objective function changes if there is a small change in the choice path.
    • Optimal control theory is a generalization of the calculus of variations.
    • Dynamic programming studies the case in which the optimization strategy is based on splitting the problem into
      smaller subproblems. The equation that describes the relationship between these subproblems is called the
      Bellman equation.
    • Mathematical programming with equilibrium constraints is where the constraints include variational inequalities
      or complementarities.


    Multi-objective optimization
    Adding more than one objective to an optimization problem adds complexity. For example, to optimize a structural
    design, one would want a design that is both light and rigid. Because these two objectives conflict, a trade-off exists.
    There will be one lightest design, one stiffest design, and an infinite number of designs that are some compromise of
    weight and stiffness. The set of trade-off designs that cannot be improved upon according to one criterion without
    hurting another criterion is known as the Pareto set. The curve created plotting weight against stiffness of the best
    designs is known as the Pareto frontier.
    A design is judged to be "Pareto optimal" (equivalently, "Pareto efficient" or in the Pareto set) if it is not dominated
    by any other design: If it is worse than another design in some respects and no better in any respect, then it is
    dominated and is not Pareto optimal.
    The choice among "Pareto optimal" solutions to determine the "favorite solution" is delegated to the decision maker.
    In other words, defining the problem as multiobjective optimization signals that some information is missing:
    desirable objectives are given but not their detailed combination. In some cases, the missing information can be
    derived by interactive sessions with the decision maker.
    Multi-objective optimization problems have been generalized further to vector optimization problems where the
    (partial) ordering is no longer given by the Pareto ordering.
Mathematical optimization                                                                                                          208


    Multi-modal optimization
    Optimization problems are often multi-modal; that is, they possess multiple good solutions. They could all be
    globally good (same cost function value) or there could be a mix of globally good and locally good solutions.
    Obtaining all (or at least some of) the multiple solutions is the goal of a multi-modal optimizer.
    Classical optimization techniques due to their iterative approach do not perform satisfactorily when they are used to
    obtain multiple solutions, since it is not guaranteed that different solutions will be obtained even with different
    starting points in multiple runs of the algorithm. Evolutionary Algorithms are however a very popular approach to
    obtain multiple solutions in a multi-modal optimization task. See Evolutionary multi-modal optimization.


    Classification of critical points and extrema

    Feasibility problem
    The satisfiability problem, also called the feasibility problem, is just the problem of finding any feasible solution
    at all without regard to objective value. This can be regarded as the special case of mathematical optimization where
    the objective value is the same for every solution, and thus any solution is optimal.
    Many optimization algorithms need to start from a feasible point. One way to obtain such a point is to relax the
    feasibility conditions using a slack variable; with enough slack, any starting point is feasible. Then, minimize that
    slack variable until slack is null or negative.


    Existence
    The extreme value theorem of Karl Weierstrass states that a continuous real-valued function on a compact set attains
    its maximum and minimum value. More generally, a lower semi-continuous function on a compact set attains its
    minimum; an upper semi-continuous function on a compact set attains its maximum.


    Necessary conditions for optimality
    One of Fermat's theorems states that optima of unconstrained problems are found at stationary points, where the first
    derivative or the gradient of the objective function is zero (see first derivative test). More generally, they may be
    found at critical points, where the first derivative or gradient of the objective function is zero or is undefined, or on
    the boundary of the choice set. An equation (or set of equations) stating that the first derivative(s) equal(s) zero at an
    interior optimum is called a 'first-order condition' or a set of first-order conditions.
    Optima of inequality-constrained problems are instead found by the Lagrange multiplier method. This method
    calculates a system of inequalities called the 'Karush–Kuhn–Tucker conditions' or 'complementary slackness
    conditions', which may then be used to calculate the optimum.


    Sufficient conditions for optimality
    While the first derivative test identifies points that might be extrema, this test does not distinguish a point that is a
    minimum from one that is a maximum or one that is neither. When the objective function is twice differentiable,
    these cases can be distinguished by checking the second derivative or the matrix of second derivatives (called the
    Hessian matrix) in unconstrained problems, or the matrix of second derivatives of the objective function and the
    constraints called the bordered Hessian in constrained problems. The conditions that distinguish maxima, or minima,
    from other stationary points are called 'second-order conditions' (see 'Second derivative test'). If a candidate solution
    satisfies the first-order conditions, then satisfaction of the second-order conditions as well is sufficient to establish at
    least local optimality.
Mathematical optimization                                                                                                         209


    Sensitivity and continuity of optima
    The envelope theorem describes how the value of an optimal solution changes when an underlying parameter
    changes. The process of computing this change is called comparative statics.
    The maximum theorem of Claude Berge (1963) describes the continuity of an optimal solution as a function of
    underlying parameters.


    Calculus of optimization
    For unconstrained problems with twice-differentiable functions, some critical points can be found by finding the
    points where the gradient of the objective function is zero (that is, the stationary points). More generally, a zero
    subgradient certifies that a local minimum has been found for minimization problems with convex functions and
    other locally Lipschitz functions.
    Further, critical points can be classified using the definiteness of the Hessian matrix: If the Hessian is positive
    definite at a critical point, then the point is a local minimum; if the Hessian matrix is negative definite, then the point
    is a local maximum; finally, if indefinite, then the point is some kind of saddle point.
    Constrained problems can often be transformed into unconstrained problems with the help of Lagrange multipliers.
    Lagrangian relaxation can also provide approximate solutions to difficult constrained problems.
    When the objective function is convex, then any local minimum will also be a global minimum. There exist efficient
    numerical techniques for minimizing convex functions, such as interior-point methods.


    Computational optimization techniques
    To solve problems, researchers may use algorithms that terminate in a finite number of steps, or iterative methods
    that converge to a solution (on some specified class of problems), or heuristics that may provide approximate
    solutions to some problems (although their iterates need not converge).


    Optimization algorithms
    •   Simplex algorithm of George Dantzig, designed for linear programming.
    •   Extensions of the simplex algorithm, designed for quadratic programming and for linear-fractional programming.
    •   Variants of the simplex algorithm that are especially suited for network optimization.
    •   Combinatorial algorithms


    Iterative methods
    The iterative methods used to solve problems of nonlinear programming differ according to whether they evaluate
    Hessians, gradients, or only function values. While evaluating Hessians (H) and gradients (G) improves the rate of
    convergence, for functions for which these quantities exist and vary sufficiently smoothly, such evaluations increase
    the computational complexity (or computational cost) of each iteration. In some cases, the computational complexity
    may be excessively high.
    One major criterion for optimizers is just the number of required function evaluations as this often is already a large
    computational effort, usually much more effort than within the optimizer itself, which mainly has to operate over the
    N variables. The derivatives provide detailed information for such optimizers, but are even harder to calculate, e.g.
    approximating the gradient takes at least N+1 function evaluations. For approximations of the 2nd derivatives
    (collected in the Hessian matrix) the number of function evaluations is in the order of N². Newton's method requires
    the 2nd order derivates, so for each iteration the number of function calls is in the order of N², but for a simpler pure
    gradient optimizer it is only N. However, gradient optimizers need usually more iterations than Newton's algorithm.
    Which one is best with respect to the number of function calls depends on the problem itself.
Mathematical optimization                                                                                                     210


    • Methods that evaluate Hessians (or approximate Hessians, using finite differences):
       • Newton's method
          • Sequential quadratic programming: A Newton-based method for small-medium scale constrained problems.
            Some versions can handle large-dimensional problems.
    • Methods that evaluate gradients or approximate gradients using finite differences (or even subgradients):
       • Quasi-Newton methods: Iterative methods for medium-large problems (e.g. N<1000).
       • Conjugate gradient methods: Iterative methods for large problems. (In theory, these methods terminate in a
         finite number of steps with quadratic objective functions, but this finite termination is not observed in practice
         on finite–precision computers.)
       • Interior point methods: This is a large class of methods for constrained optimization. Some interior-point
         methods use only (sub)gradient information, and others of which require the evaluation of Hessians.
       • Gradient descent (alternatively, "steepest descent" or "steepest ascent"): A (slow) method of historical and
         theoretical interest, which has had renewed interest for finding approximate solutions of enormous problems.
       • Subgradient methods - An iterative method for large locally Lipschitz functions using generalized gradients.
         Following Boris T. Polyak, subgradient–projection methods are similar to conjugate–gradient methods.
       • Bundle method of descent: An iterative method for small–medium sized problems with locally Lipschitz
         functions, particularly for convex minimization problems. (Similar to conjugate gradient methods)
       • Ellipsoid method: An iterative method for small problems with quasiconvex objective functions and of great
         theoretical interest, particularly in establishing the polynomial time complexity of some combinatorial
         optimization problems. It has similarities with Quasi-Newton methods.
       • Reduced gradient method (Frank–Wolfe) for approximate minimization of specially structured problems with
         linear constraints, especially with traffic networks. For general unconstrained problems, this method reduces to
         the gradient method, which is regarded as obsolete (for almost all problems).
       • Simultaneous perturbation stochastic approximation (SPSA) method for stochastic optimization; uses random
         (efficient) gradient approximation.
    • Methods that evaluate only function values: If a problem is continuously differentiable, then gradients can be
      approximated using finite differences, in which case a gradient-based method can be used.
       • Interpolation methods
       • Pattern search methods, which have better convergence properties than the Nelder–Mead heuristic (with
         simplices), which is listed below.

    Global convergence
    More generally, if the objective function is not a quadratic function, then many optimization methods use other
    methods to ensure that some subsequence of iterations converges to an optimal solution. The first and still popular
    method for ensuring convergence relies on line searches, which optimize a function along one dimension. A second
    and increasingly popular method for ensuring convergence uses trust regions. Both line searches and trust regions are
    used in modern methods of non-differentiable optimization. Usually a global optimizer is much slower than
    advanced local optimizers (such as BFGS), so often an efficient global optimizer can be constructed by starting the
    local optimizer from different starting points.
Mathematical optimization                                                                                                     211


    Heuristics
    Besides (finitely terminating) algorithms and (convergent) iterative methods, there are heuristics that can provide
    approximate solutions to some optimization problems:
    •   Memetic algorithm
    •   Differential evolution
    •   Differential Search algorithm [4] Matlab code-link has been provided in Çivicioglu, P.,(2012).
    •   Dynamic relaxation
    •   Genetic algorithms
    •   Hill climbing
    •   Nelder-Mead simplicial heuristic: A popular heuristic for approximate minimization (without calling gradients)
    •   Particle swarm optimization
    •   Artificial bee colony optimization
    •   Simulated annealing
    •   Tabu search
    •   Reactive Search Optimization (RSO)[5] implemented in LIONsolver


    Applications

    Mechanics and engineering
    Problems in rigid body dynamics (in particular articulated rigid body dynamics) often require mathematical
    programming techniques, since you can view rigid body dynamics as attempting to solve an ordinary differential
    equation on a constraint manifold; the constraints are various nonlinear geometric constraints such as "these two
    points must always coincide", "this surface must not penetrate any other", or "this point must always lie somewhere
    on this curve". Also, the problem of computing contact forces can be done by solving a linear complementarity
    problem, which can also be viewed as a QP (quadratic programming) problem.
    Many design problems can also be expressed as optimization programs. This application is called design
    optimization. One subset is the engineering optimization, and another recent and growing subset of this field is
    multidisciplinary design optimization, which, while useful in many problems, has in particular been applied to
    aerospace engineering problems.


    Economics
    Economics is closely enough linked to optimization of agents that an influential definition relatedly describes
    economics qua science as the "study of human behavior as a relationship between ends and scarce means" with
    alternative uses.[6] Modern optimization theory includes traditional optimization theory but also overlaps with game
    theory and the study of economic equilibria. The Journal of Economic Literature codes classify mathematical
    programming, optimization techniques, and related topics under JEL:C61-C63.
    In microeconomics, the utility maximization problem and its dual problem, the expenditure minimization problem,
    are economic optimization problems. Insofar as they behave consistently, consumers are assumed to maximize their
    utility, while firms are usually assumed to maximize their profit. Also, agents are often modeled as being risk-averse,
    thereby preferring to avoid risk. Asset prices are also modeled using optimization theory, though the underlying
    mathematics relies on optimizing stochastic processes rather than on static optimization. Trade theory also uses
    optimization to explain trade patterns between nations. The optimization of market portfolios is an example of
    multi-objective optimization in economics.
    Since the 1970s, economists have modeled dynamic decisions over time using control theory. For example,
    microeconomists use dynamic search models to study labor-market behavior.[7] A crucial distinction is between
Mathematical optimization                                                                                                                               212


    deterministic and stochastic models.[8] Macroeconomists build dynamic stochastic general equilibrium (DSGE)
    models that describe the dynamics of the whole economy as the result of the interdependent optimizing decisions of
    workers, consumers, investors, and governments.[9][10]


    Operations research
    Another field that uses optimization techniques extensively is operations research. Operations research also uses
    stochastic modeling and simulation to support improved decision-making. Increasingly, operations research uses
    stochastic programming to model dynamic decisions that adapt to events; such problems can be solved with
    large-scale optimization and stochastic optimization methods.


    Control engineering
    Mathematical optimization is used in much modern controller design. High-level controllers such as Model
    predictive control (MPC) or Real-Time Optimization (RTO) employ mathematical optimization. These algorithms
    run online and repeatedly determine values for decision variables, such as choke openings in a process plant, by
    iteratively solving a mathematical optimization problem including constraints and a model of the system to be
    controlled.


    Notes
    [1] " The Nature of Mathematical Programming (http:/ / glossary. computing. society. informs. org/ index. php?page=nature. html),"
        Mathematical Programming Glossary, INFORMS Computing Society.
    [2] W. Erwin Diewert (2008). "cost functions," The New Palgrave Dictionary of Economics, 2nd Edition Contents (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_C000390& edition=current& q=).
    [3] Peter Newman (2008). "indirect utility function," The New Palgrave Dictionary of Economics, 2nd Edition. Contents. (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_I000066& edition=)
    [4] Civicioglu, P. (2012). "Transforming geocentric cartesian coordinates to geodetic coordinates by using differential search algorithm".
        Computers & Geosciences 46: 229–247. doi:10.1016/j.cageo.2011.12.011.
    [5] Battiti, Roberto; Mauro Brunato; Franco Mascia (2008). Reactive Search and Intelligent Optimization (http:/ / reactive-search. org/ thebook).
        Springer Verlag. ISBN 978-0-387-09623-0. .
    [6] Lionel Robbins (1935, 2nd ed.) An Essay on the Nature and Significance of Economic Science, Macmillan, p. 16.
    [7] A. K. Dixit ([1976] 1990). Optimization in Economic Theory, 2nd ed., Oxford. Description (http:/ / books. google. com/
        books?id=dHrsHz0VocUC& pg=find& pg=PA194=false#v=onepage& q& f=false) and contents preview (http:/ / books. google. com/
        books?id=dHrsHz0VocUC& pg=PR7& lpg=PR6& dq=false& lr=#v=onepage& q=false& f=false).
    [8] A.G. Malliaris (2008). "stochastic optimal control," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www.
        dictionaryofeconomics. com/ article?id=pde2008_S000269& edition=& field=keyword& q=Taylor's th& topicid=& result_number=1).
    [9] Julio Rotemberg and Michael Woodford (1997), "An Optimization-based Econometric Framework for the Evaluation of Monetary
        Policy.NBER Macroeconomics Annual, 12, pp. 297-346. (http:/ / people. hbs. edu/ jrotemberg/ PublishedArticles/
        OptimizBasedEconometric_97. pdf)
    [10] From The New Palgrave Dictionary of Economics (2008), 2nd Edition with Abstract links:
           • " numerical optimization methods in economics (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_N000148&
        edition=current& q=optimization& topicid=& result_number=1)" by Karl Schmedders
           • " convex programming (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000348& edition=current& q=optimization&
        topicid=& result_number=4)" by Lawrence E. Blume
           • " Arrow–Debreu model of general equilibrium (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_A000133&
        edition=current& q=optimization& topicid=& result_number=20)" by John Geanakoplos.
Mathematical optimization                                                                                             213


    Further reading

    Comprehensive

    Undergraduate level
    • Bradley, S.; Hax, A.; Magnanti, T. (1977). Applied mathematical programming. Addison Wesley.
    • Rardin, Ronald L. (1997). Optimization in operations research. Prentice Hall. pp. 919. ISBN 0-02-398415-5.
    • Strang, Gilbert (1986). Introduction to applied mathematics (http://www.wellesleycambridge.com/tocs/
      toc-appl). Wellesley, MA: Wellesley-Cambridge Press (Strang's publishing company). pp. xii+758.
      ISBN 0-9614088-0-4. MR870634.

    Graduate level
    • Magnanti, Thomas L. (1989). "Twenty years of mathematical programming". In Cornet, Bernard; Tulkens, Henry.
      Contributions to Operations Research and Economics: The twentieth anniversary of CORE (Papers from the
      symposium held in Louvain-la-Neuve, January 1987). Cambridge, MA: MIT Press. pp. 163–227.
      ISBN 0-262-03149-3. MR1104662.
    • Minoux, M. (1986). Mathematical programming: Theory and algorithms (Translated by Steven Vajda from the
      (1983 Paris: Dunod) French ed.). Chichester: A Wiley-Interscience Publication. John Wiley & Sons, Ltd..
      pp. xxviii+489. ISBN 0-471-90170-9. MR2571910. (2008 Second ed., in French: Programmation mathématique:
      Théorie et algorithmes. Editions Tec & Doc, Paris, 2008. xxx+711 pp. ISBN 978-2-7430-1000-3..
    • Nemhauser, G. L.; Rinnooy Kan, A. H. G.; Todd, M. J., eds. (1989). Optimization. Handbooks in Operations
      Research and Management Science. 1. Amsterdam: North-Holland Publishing Co.. pp. xiv+709.
      ISBN 0-444-87284-1. MR1105099.
       • J. E. Dennis, Jr. and Robert B. Schnabel, A view of unconstrained optimization (pp. 1–72);
       • Donald Goldfarb and Michael J. Todd, Linear programming (pp. 73–170);
       • Philip E. Gill, Walter Murray, Michael A. Saunders, and Margaret H. Wright, Constrained nonlinear
         programming (pp. 171–210);
       • Ravindra K. Ahuja, Thomas L. Magnanti, and James B. Orlin, Network flows (pp. 211–369);
       • W. R. Pulleyblank, Polyhedral combinatorics (pp. 371–446);
       • George L. Nemhauser and Laurence A. Wolsey, Integer programming (pp. 447–527);
       • Claude Lemaréchal, Nondifferentiable optimization (pp. 529–572);
       • Roger J-B Wets, Stochastic programming (pp. 573–629);
       • A. H. G. Rinnooy Kan and G. T. Timmer, Global optimization (pp. 631–662);
       • P. L. Yu, Multiple criteria decision making: five basic concepts (pp. 663–699).
    • Shapiro, Jeremy F. (1979). Mathematical programming: Structures and algorithms. New York:
      Wiley-Interscience [John Wiley & Sons]. pp. xvi+388. ISBN 0-471-77886-9. MR544669.
    • Spall, J. C. (2003), Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control,
      Wiley, Hoboken, NJ.
Mathematical optimization                                                                                                 214


    Continuous optimization
    • Mordecai Avriel (2003). Nonlinear Programming: Analysis and Methods. Dover Publishing.
      ISBN 0-486-43227-0.
    • Bonnans, J. Frédéric; Gilbert, J. Charles; Lemaréchal, Claude; Sagastizábal, Claudia A. (2006). Numerical
      optimization: Theoretical and practical aspects (http://www.springer.com/mathematics/applications/book/
      978-3-540-35445-1). Universitext (Second revised ed. of translation of 1997 French ed.). Berlin: Springer-Verlag.
      pp. xiv+490. doi:10.1007/978-3-540-35447-5. ISBN 3-540-35445-X. MR2265882.
    • Bonnans, J. Frédéric; Shapiro, Alexander (2000). Perturbation analysis of optimization problems. Springer Series
      in Operations Research. New York: Springer-Verlag. pp. xviii+601. ISBN 0-387-98705-3. MR1756264.
    • Boyd, Stephen P.; Vandenberghe, Lieven (2004) (pdf). Convex Optimization (http://www.stanford.edu/~boyd/
      cvxbook/bv_cvxbook.pdf). Cambridge University Press. ISBN 978-0-521-83378-3. Retrieved October 15, 2011.
    • Jorge Nocedal and Stephen J. Wright (2006). Numerical Optimization (http://www.ece.northwestern.edu/
      ~nocedal/book/num-opt.html). Springer. ISBN 0-387-30303-0.
    • Ruszczyński, Andrzej (2006). Nonlinear Optimization. Princeton, NJ: Princeton University Press. pp. xii+454.
      ISBN 978-0691119151. MR2199043.


    Combinatorial optimization
    • R. K. Ahuja, Thomas L. Magnanti, and James B. Orlin (1993). Network Flows: Theory, Algorithms, and
      Applications. Prentice-Hall, Inc. ISBN 0-13-617549-X.
    • William J. Cook, William H. Cunningham, William R. Pulleyblank, Alexander Schrijver; Combinatorial
      Optimization; John Wiley & Sons; 1 edition (November 12, 1997); ISBN 0-471-55894-X.
    • Gondran, Michel; Minoux, Michel (1984). Graphs and algorithms. Wiley-Interscience Series in Discrete
      Mathematics (Translated by Steven Vajda from the second (Collection de la Direction des Études et Recherches
      d'Électricité de France [Collection of the Department of Studies and Research of Électricité de France], v. 37.
      Paris: Éditions Eyrolles 1985. xxviii+545 pp. MR868083) French ed.). Chichester: John Wiley & Sons, Ltd..
      pp. xix+650. ISBN 0-471-10374-8. MR2552933. (Fourth ed. Collection EDF R&D. Paris: Editions Tec & Doc
      2009. xxxii+784 pp..
    • Eugene Lawler (2001). Combinatorial Optimization: Networks and Matroids. Dover. ISBN 0-486-41453-1.
    • Lawler, E. L.; Lenstra, J. K.; Rinnooy Kan, A. H. G.; Shmoys, D. B. (1985), The traveling salesman problem: A
      guided tour of combinatorial optimization, John Wiley & Sons, ISBN 0-471-90413-9.
    • Jon Lee; A First Course in Combinatorial Optimization (http://books.google.com/
      books?id=3pL1B7WVYnAC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q&
      f=false); Cambridge University Press; 2004; ISBN 0-521-01012-8.
    • Christos H. Papadimitriou and Kenneth Steiglitz Combinatorial Optimization : Algorithms and Complexity;
      Dover Pubns; (paperback, Unabridged edition, July 1998) ISBN 0-486-40258-4.


    Journals
    • Computational Optimization and Applications (http://www.springer.com/mathematics/journal/10589)
    • Journal of Computational Optimization in Economics and Finance (https://www.novapublishers.com/catalog/
      product_info.php?products_id=6353)
    • Journal of Economic Dynamics and Control (http://www.journals.elsevier.com/
      journal-of-economic-dynamics-and-control/)
    • SIAM Journal on Optimization (SIOPT) (http://www.siam.org/journals/siopt.php) and Editorial Policy (http:/
      /www.siam.org/journals/siopt/policy.php)
    • SIAM Journal on Control and Optimization (SICON) (http://www.siam.org/journals/sicon.php) and Editorial
      Policy (http://www.siam.org/journals/sicon/policy.php)
Mathematical optimization                                                                                                   215


    External links
    •   COIN-OR (http://www.coin-or.org/)—Computational Infrastructure for Operations Research
    •   Decision Tree for Optimization Software (http://plato.asu.edu/guide.html) Links to optimization source codes
    •   Global optimization (http://www.mat.univie.ac.at/~neum/glopt.html)
    •   Mathematical Programming Glossary (http://glossary.computing.society.informs.org/)
    •   Mathematical Programming Society (http://www.mathprog.org/)
    •   NEOS Guide (http://www-fp.mcs.anl.gov/otc/Guide/index.html) currently being replaced by the NEOS
        Wiki (http://wiki.mcs.anl.gov/neos)
    •   Optimization Online (http://www.optimization-online.org) A repository for optimization e-prints
    •   Optimization Related Links (http://www2.arnes.si/~ljc3m2/igor/links.html)
    •   Convex Optimization I (http://see.stanford.edu/see/courseinfo.
        aspx?coll=2db7ced4-39d1-4fdb-90e8-364129597c87) EE364a: Course from Stanford University
    •   Convex Optimization – Boyd and Vandenberghe (http://www.stanford.edu/~boyd/cvxbook) Book on Convex
        Optimization
    •   Book and Course (http://apmonitor.com/me575/index.php/Main/BookChapters) on Optimization Methods
        for Engineering Design
    •   Optimization Online (http://www.optimization-online.org) A repository for optimization e-prints



    Programming paradigm
    A programming paradigm is a fundamental style of computer programming. There are four main paradigms:
    object-oriented, imperative, functional and declarative.[1] Their foundations are distinct models of computation:
    Turing machine for object-oriented and imperative programming, lambda calculus for functional programming, and
    first order logic for logic programming.


    Overview
    A programming model is an abstraction of a computer system. For example, the "von Neumann model" is a model
    used in traditional sequential computers. For parallel computing, there are many possible models typically reflecting
    different ways processors can be interconnected. The most common are based on shared memory, distributed
    memory with message passing, or a hybrid of the two.
    A programming language can support multiple paradigms. For example, programs written in C++ or Object Pascal
    can be purely procedural, or purely object-oriented, or contain elements of both paradigms. Software designers and
    programmers decide how to use those paradigm elements.
    In object-oriented programming, programmers can think of a program as a collection of interacting objects, while in
    functional programming a program can be thought of as a sequence of stateless function evaluations. When
    programming computers or systems with many processors, process-oriented programming allows programmers to
    think about applications as sets of concurrent processes acting upon logically shared data structures.
    Just as different groups in software engineering advocate different methodologies, different programming languages
    advocate different programming paradigms. Some languages are designed to support one particular paradigm
    (Smalltalk supports object-oriented programming, Haskell supports functional programming), while other
    programming languages support multiple paradigms (such as Object Pascal, C++, Java, C#, Scala, Visual Basic,
    Common Lisp, Scheme, Perl, Python, Ruby, Oz and F#).
    Many programming paradigms are as well known for what techniques they forbid as for what they enable. For
    instance, pure functional programming disallows the use of side-effects, while structured programming disallows the
    use of the goto statement. Partly for this reason, new paradigms are often regarded as doctrinaire or overly rigid by
Programming paradigm                                                                                                       216


    those accustomed to earlier styles.[2] Avoiding certain techniques can make it easier to prove theorems about a
    program's correctness—or simply to understand its behavior.


    Multi-paradigm programming language
    A multi-paradigm programming language is a programming language that supports more than one programming
    paradigm. As Leda designer Timothy Budd puts it: "The idea of a multiparadigm language is to provide a framework
    in which programmers can work in a variety of styles, freely intermixing constructs from different paradigms." The
    design goal of such languages is to allow programmers to use the best tool for a job, admitting that no one paradigm
    solves all problems in the easiest or most efficient way.
    One example is C#, which includes imperative and object-oriented paradigms as well as some support for functional
    programming through type inference, anonymous functions and Language Integrated Query. Some other ones are F#
    and Scala, which provides similar functionality to C# but also includes full support for functional programming
    (including currying, pattern matching, algebraic data types, lazy evaluation, tail recursion, immutability, etc.).
    Perhaps the most extreme example is Oz, which has subsets that are logic (Oz descends from logic programming), a
    functional, an object-oriented, a dataflow concurrent, and other language paradigms. Oz was designed over a
    ten-year period to combine in a harmonious way concepts that are traditionally associated with different
    programming paradigms. Lisp, while often taught as a functional language, is known for its malleability and thus its
    ability to engulf many paradigms.


    History
    The lowest level programming paradigms are machine code, which directly represents the instructions (the contents
    of program memory) as a sequence of numbers, and assembly language where the machine instructions are
    represented by mnemonics and memory addresses can be given symbolic labels. These are sometimes called first-
    and second-generation languages. In the 1960s assembly languages were developed to support library COPY and
    quite sophisticated conditional macro generation and pre-processing capabilities, CALL to (subroutines), external
    variables and common sections (globals), enabling significant code re-use and isolation from hardware specifics via
    use of logical operators such as READ/WRITE/GET/PUT. Assembly was, and still is, used for time critical systems
    and frequently in embedded systems as it gives the most direct control of what the machine actually does.
    The next advance was the development of procedural languages. These third-generation languages (the first
    described as high-level languages) use vocabulary related to the problem being solved. For example,
    • C - developed c. 1970 at Bell Labs
    • COBOL (Common Business Oriented Language) - uses terms like file, move and copy.
    • FORTRAN (FORmula TRANslation) - using mathematical language terminology, it was developed mainly for
      scientific and engineering problems.
    • ALGOL (ALGOrithmic Language) - focused on being an appropriate language to define algorithms, while using
      mathematical language terminology and targeting scientific and engineering problems just like FORTRAN.
    • PL/I (Programming Language One) - a hybrid commercial/scientific general purpose language supporting
      pointers.
    • BASIC (Beginners All purpose Symbolic Instruction Code) - was developed to enable more people to write
      programs.
    All these languages follow the procedural paradigm. That is, they describe, step by step, exactly the procedure that
    should, according to the particular programmer at least, be followed to solve a specific problem. The efficacy and
    efficiency of any such solution are both therefore entirely subjective and highly dependent on that programmer's
    experience, inventiveness and ability.
Programming paradigm                                                                                                                                217


    Later, object-oriented languages (like Simula, Smalltalk, C++, Eiffel and Java) were created. In these languages,
    data, and methods of manipulating the data, are kept as a single unit called an object. The only way that a user can
    access the data is via the object's 'methods' (subroutines). Because of this, the internal workings of an object may be
    changed without affecting any code that uses the object. There is still some controversy by notable programmers
    such as Alexander Stepanov, Richard Stallman[3] and others, concerning the efficacy of the OOP paradigm versus
    the procedural paradigm. The necessity of every object to have associative methods leads some skeptics to associate
    OOP with software bloat. Polymorphism was developed as one attempt to resolve this dilemma.
    Since object-oriented programming is considered a paradigm, not a language, it is possible to create even an
    object-oriented assembler language. High Level Assembly (HLA) is an example of this that fully supports advanced
    data types and object-oriented assembly language programming - despite its early origins. Thus, differing
    programming paradigms can be thought of as more like 'motivational memes' of their advocates - rather than
    necessarily representing progress from one level to the next. Precise comparisons of the efficacy of competing
    paradigms are frequently made more difficult because of new and differing terminology applied to similar (but not
    identical) entities and processes together with numerous implementation distinctions across languages.
    Within imperative programming, which is based on procedural languages, an alternative to the computer-centered
    hierarchy of structured programming is literate programming, which structures programs instead as a
    human-centered web, as in a hypertext essay – documentation is integral to the program, and the program is
    structured following the logic of prose exposition, rather than compiler convenience.
    Independent of the imperative branch, declarative programming paradigms were developed. In these languages the
    computer is told what the problem is, not how to solve the problem - the program is structured as a collection of
    properties to find in the expected result, not as a procedure to follow. Given a database or a set of rules, the computer
    tries to find a solution matching all the desired properties. The archetypical example of a declarative language is the
    fourth generation language SQL, as well as the family of functional languages and logic programming.
    Functional programming is a subset of declarative programming. Programs written using this paradigm use
    functions, blocks of code intended to behave like mathematical functions. Functional languages discourage changes
    in the value of variables through assignment, making a great deal of use of recursion instead.
    The logic programming paradigm views computation as automated reasoning over a corpus of knowledge. Facts
    about the problem domain are expressed as logic formulae, and programs are executed by applying inference rules
    over them until an answer to the problem is found, or the collection of formulae is proved inconsistent.


    References
    [1] Nørmark, Kurt. Overview of the four main programming paradigms (http:/ / people. cs. aau. dk/ ~normark/ prog3-03/ html/ notes/
        paradigms_themes-paradigm-overview-section. html). Aalborg University, 9 May 2011. Retrieved 22 September 2012.
    [2] Frank Rubin published a criticism of Dijkstra's letter in the March 1987 CACM where it appeared under the title 'GOTO Considered Harmful'
        Considered Harmful. Frank Rubin (March 1987). "'GOTO Considered Harmful' Considered Harmful" (http:/ / www. ecn. purdue. edu/
        ParaMount/ papers/ rubin87goto. pdf) (PDF). Communications of the ACM 30 (3): 195–196. doi:10.1145/214748.315722. .
    [3] "Mode inheritance, cloning, hooks & OOP (Google Groups Discussion)" (http:/ / groups. google. com/ group/ comp. emacs. xemacs/
        browse_thread/ thread/ d0af257a2837640c/ 37f251537fafbb03?lnk=st& q="Richard+ Stallman"+ oop& rnum=5& hl=en#37f251537fafbb03).
        .



    External links
    • Classification of the principal programming paradigms (http://www.info.ucl.ac.be/~pvr/paradigms.html)
Mathematical model                                                                                                             218



    Mathematical model
    A mathematical model is a description of a system using mathematical concepts and language. The process of
    developing a mathematical model is termed mathematical modelling. Mathematical models are used not only in the
    natural sciences (such as physics, biology, earth science, meteorology) and engineering disciplines (e.g. computer
    science, artificial intelligence), but also in the social sciences (such as economics, psychology, sociology and
    political science); physicists, engineers, statisticians, operations research analysts and economists use mathematical
    models most extensively. A model may help to explain a system and to study the effects of different components,
    and to make predictions about behaviour.
    Mathematical models can take many forms, including but not limited to dynamical systems, statistical models,
    differential equations, or game theoretic models. These and other types of models can overlap, with a given model
    involving a variety of abstract structures. In general, mathematical models may include logical models, as far as
    logic is taken as a part of mathematics. In many cases, the quality of a scientific field depends on how well the
    mathematical models developed on the theoretical side agree with results of repeatable experiments. Lack of
    agreement between theoretical mathematical models and experimental measurements often leads to important
    advances as better theories are developed.


    Examples of mathematical models
    • Many everyday activities carried out without a thought are uses of mathematical models. A geographical map
      projection of a region of the earth onto a small, plane surface is a model[1] which can be used for many purposes
      such as planning travel.
    • Another simple activity is predicting the position of a vehicle from its initial position, direction and speed of
      travel, using the equation that distance travelled is the product of time and speed. This is known as dead
      reckoning when used more formally. Mathematical modelling in this way does not necessarily require formal
      mathematics; animals have been shown to use dead reckoning.[2][3]
    • Population Growth. A simple (though approximate) model of population growth is the Malthusian growth model.
      A slightly more realistic and largely used population growth model is the logistic function, and its extensions.
    • Model of a particle in a potential-field. In this model we consider a particle as being a point of mass which
      describes a trajectory in space which is modeled by a function giving its coordinates in space as a function of
      time. The potential field is given by a function V : R3 → R and the trajectory is a solution of the differential
      equation



          Note this model assumes the particle is a point mass, which is certainly known to be false in many cases in
          which we use this model; for example, as a model of planetary motion.
    • Model of rational behavior for a consumer. In this model we assume a consumer faces a choice of n commodities
      labeled 1,2,...,n each with a market price p1, p2,..., pn. The consumer is assumed to have a cardinal utility function
      U (cardinal in the sense that it assigns numerical values to utilities), depending on the amounts of commodities x1,
      x2,..., xn consumed. The model further assumes that the consumer has a budget M which is used to purchase a
      vector x1, x2,..., xn in such a way as to maximize U(x1, x2,..., xn). The problem of rational behavior in this model
      then becomes an optimization problem, that is:


                subject to:
Mathematical model                                                                                                             219



          This model has been used in general equilibrium theory, particularly to show existence and Pareto efficiency
          of economic equilibria. However, the fact that this particular formulation assigns numerical values to levels of
          satisfaction is the source of criticism (and even ridicule). However, it is not an essential ingredient of the
          theory and again this is an idealization.
    • Neighbour-sensing model explains the mushroom formation from the initially chaotic fungal network.
    • Computer Science: models in Computer Networks, data models, surface model,...
    • Mechanics: movement of rocket model,...
    Modeling requires selecting and identifying relevant aspects of a situation in the real world.


    Significance in the natural sciences
    Mathematical models are of great importance in the natural sciences, particularly in physics. Physical theories are
    almost invariably expressed using mathematical models.
    Throughout history, more and more accurate mathematical models have been developed. Newton's laws accurately
    describe many everyday phenomena, but at certain limits relativity theory and quantum mechanics must be used,
    even these do not apply to all situations and need further refinement. It is possible to obtain the less accurate models
    in appropriate limits, for example relativistic mechanics reduces to Newtonian mechanics at speeds much less than
    the speed of light. Quantum mechanics reduces to classical physics when the quantum numbers are high. For
    example the de Broglie wavelength of a tennis ball is insignificantly small, so classical physics is a good
    approximation to use in this case.
    It is common to use idealized models in physics to simplify things. Massless ropes, point particles, ideal gases and
    the particle in a box are among the many simplified models used in physics. The laws of physics are represented with
    simple equations such as Newton's laws, Maxwell's equations and the Schrödinger equation. These laws are such as
    a basis for making mathematical models of real situations. Many real situations are very complex and thus modeled
    approximate on a computer, a model that is computationally feasible to compute is made from the basic laws or from
    approximate models made from the basic laws. For example, molecules can be modeled by molecular orbital models
    that are approximate solutions to the Schrödinger equation. In engineering, physics models are often made by
    mathematical methods such as finite element analysis.
    Different mathematical models use different geometries that are not necessarily accurate descriptions of the
    geometry of the universe. Euclidean geometry is much used in classical physics, while special relativity and general
    relativity are examples of theories that use geometries which are not Euclidean.


    Some applications
    Since prehistorical times simple models such as maps and pre-designed diagrams have been used.
    Often when engineers analyze a system to be controlled or optimized, they use a mathematical model. In analysis,
    engineers can build a descriptive model of the system as a hypothesis of how the system could work, or try to
    estimate how an unforeseeable event could affect the system. Similarly, in control of a system, engineers can try out
    different control approaches in simulations.
    A mathematical model usually describes a system by a set of variables and a set of equations that establish
    relationships between the variables. Variables may be of many types; real or integer numbers, boolean values or
    strings, for example. The variables represent some properties of the system, for example, measured system outputs
    often in the form of signals, timing data, counters, and event occurrence (yes/no). The actual model is the set of
    functions that describe the relations between the different variables.
Mathematical model                                                                                                               220


    Building blocks
    In business and engineering, mathematical models may be used to maximize a certain output. The system under
    consideration will require certain inputs. The system relating inputs to outputs depends on other variables too:
    decision variables, state variables, exogenous variables, and random variables.
    Decision variables are sometimes known as independent variables. Exogenous variables are sometimes known as
    parameters or constants. The variables are not independent of each other as the state variables are dependent on the
    decision, input, random, and exogenous variables. Furthermore, the output variables are dependent on the state of the
    system (represented by the state variables).
    Objectives and constraints of the system and its users can be represented as functions of the output variables or state
    variables. The objective functions will depend on the perspective of the model's user. Depending on the context, an
    objective function is also known as an index of performance, as it is some measure of interest to the user. Although
    there is no limit to the number of objective functions and constraints a model can have, using or optimizing the
    model becomes more involved (computationally) as the number increases.
    For example, in economics students often apply linear algebra when using input-output models. Complicated
    mathematical models that have many variables may be consolidated by use of vectors where one symbol represents
    several variables.


    Classifying mathematical models
    Many mathematical models can be classified in some of the following ways:
    1. Linear vs. nonlinear: Mathematical models are usually composed by variables, which are abstractions of
       quantities of interest in the described systems, and operators that act on these variables, which can be algebraic
       operators, functions, differential operators, etc. If all the operators in a mathematical model exhibit linearity, the
       resulting mathematical model is defined as linear. A model is considered to be nonlinear otherwise.
       The question of linearity and nonlinearity is dependent on context, and linear models may have nonlinear
       expressions in them. For example, in a statistical linear model, it is assumed that a relationship is linear in the
       parameters, but it may be nonlinear in the predictor variables. Similarly, a differential equation is said to be linear
       if it can be written with linear differential operators, but it can still have nonlinear expressions in it. In a
       mathematical programming model, if the objective functions and constraints are represented entirely by linear
       equations, then the model is regarded as a linear model. If one or more of the objective functions or constraints
       are represented with a nonlinear equation, then the model is known as a nonlinear model.
       Nonlinearity, even in fairly simple systems, is often associated with phenomena such as chaos and irreversibility.
       Although there are exceptions, nonlinear systems and models tend to be more difficult to study than linear ones. A
       common approach to nonlinear problems is linearization, but this can be problematic if one is trying to study
       aspects such as irreversibility, which are strongly tied to nonlinearity.
    2. Deterministic vs. probabilistic (stochastic): A deterministic model is one in which every set of variable states is
       uniquely determined by parameters in the model and by sets of previous states of these variables. Therefore,
       deterministic models perform the same way for a given set of initial conditions. Conversely, in a stochastic model,
       randomness is present, and variable states are not described by unique values, but rather by probability
       distributions.
    3. Static vs. dynamic: A static model does not account for the element of time, while a dynamic model does.
       Dynamic models typically are represented with difference equations or differential equations.
    4. Discrete vs. Continuous: A discrete model does not take into account the function of time and usually uses
       time-advance methods, while a Continuous model does. Continuous models typically are represented with f(t) and
       the changes are reflected over continuous time intervals.
Mathematical model                                                                                                            221


    5. Deductive, inductive, or floating: A deductive model is a logical structure based on a theory. An inductive
       model arises from empirical findings and generalization from them. The floating model rests on neither theory nor
       observation, but is merely the invocation of expected structure. Application of mathematics in social sciences
       outside of economics has been criticized for unfounded models.[4] Application of catastrophe theory in science
       has been characterized as a floating model.[5]


    A priori information
    Mathematical modeling problems are used often classified into black box or white box models, according to how
    much a priori information is available of the system. A black-box model is a system of which there is no a priori
    information available. A white-box model (also called glass box or clear box) is a system where all necessary
    information is available. Practically all systems are somewhere between the black-box and white-box models, so this
    concept is useful only as an intuitive guide for deciding which approach to take.
    Usually it is preferable to use as much a priori information as possible to make the model more accurate. Therefore
    the white-box models are usually considered easier, because if you have used the information correctly, then the
    model will behave correctly. Often the a priori information comes in forms of knowing the type of functions relating
    different variables. For example, if we make a model of how a medicine works in a human system, we know that
    usually the amount of medicine in the blood is an exponentially decaying function. But we are still left with several
    unknown parameters; how rapidly does the medicine amount decay, and what is the initial amount of medicine in
    blood? This example is therefore not a completely white-box model. These parameters have to be estimated through
    some means before one can use the model.
    In black-box models one tries to estimate both the functional form of relations between variables and the numerical
    parameters in those functions. Using a priori information we could end up, for example, with a set of functions that
    probably could describe the system adequately. If there is no a priori information we would try to use functions as
    general as possible to cover all different models. An often used approach for black-box models are neural networks
    which usually do not make assumptions about incoming data. The problem with using a large set of functions to
    describe a system is that estimating the parameters becomes increasingly difficult when the amount of parameters
    (and different types of functions) increases.


    Subjective information
    Sometimes it is useful to incorporate subjective information into a mathematical model. This can be done based on
    intuition, experience, or expert opinion, or based on convenience of mathematical form. Bayesian statistics provides
    a theoretical framework for incorporating such subjectivity into a rigorous analysis: one specifies a prior probability
    distribution (which can be subjective) and then updates this distribution based on empirical data. An example of
    when such approach would be necessary is a situation in which an experimenter bends a coin slightly and tosses it
    once, recording whether it comes up heads, and is then given the task of predicting the probability that the next flip
    comes up heads. After bending the coin, the true probability that the coin will come up heads is unknown, so the
    experimenter would need to make an arbitrary decision (perhaps by looking at the shape of the coin) about what
    prior distribution to use. Incorporation of the subjective information is necessary in this case to get an accurate
    prediction of the probability, since otherwise one would guess 1 or 0 as the probability of the next flip being heads,
    which would be almost certainly wrong.[6]
Mathematical model                                                                                                                222


    Complexity
    In general, model complexity involves a trade-off between simplicity and accuracy of the model. Occam's razor is a
    principle particularly relevant to modeling; the essential idea being that among models with roughly equal predictive
    power, the simplest one is the most desirable. While added complexity usually improves the realism of a model, it
    can make the model difficult to understand and analyze, and can also pose computational problems, including
    numerical instability. Thomas Kuhn argues that as science progresses, explanations tend to become more complex
    before a Paradigm shift offers radical simplification.
    For example, when modeling the flight of an aircraft, we could embed each mechanical part of the aircraft into our
    model and would thus acquire an almost white-box model of the system. However, the computational cost of adding
    such a huge amount of detail would effectively inhibit the usage of such a model. Additionally, the uncertainty
    would increase due to an overly complex system, because each separate part induces some amount of variance into
    the model. It is therefore usually appropriate to make some approximations to reduce the model to a sensible size.
    Engineers often can accept some approximations in order to get a more robust and simple model. For example
    Newton's classical mechanics is an approximated model of the real world. Still, Newton's model is quite sufficient
    for most ordinary-life situations, that is, as long as particle speeds are well below the speed of light, and we study
    macro-particles only.


    Training
    Any model which is not pure white-box contains some parameters that can be used to fit the model to the system it is
    intended to describe. If the modeling is done by a neural network, the optimization of parameters is called training.
    In more conventional modeling through explicitly given mathematical functions, parameters are determined by curve
    fitting.


    Model evaluation
    A crucial part of the modeling process is the evaluation of whether or not a given mathematical model describes a
    system accurately. This question can be difficult to answer as it involves several different types of evaluation.


    Fit to empirical data
    Usually the easiest part of model evaluation is checking whether a model fits experimental measurements or other
    empirical data. In models with parameters, a common approach to test this fit is to split the data into two disjoint
    subsets: training data and verification data. The training data are used to estimate the model parameters. An accurate
    model will closely match the verification data even though these data were not used to set the model's parameters.
    This practice is referred to as cross-validation in statistics.
    Defining a metric to measure distances between observed and predicted data is a useful tool of assessing model fit.
    In statistics, decision theory, and some economic models, a loss function plays a similar role.
    While it is rather straightforward to test the appropriateness of parameters, it can be more difficult to test the validity
    of the general mathematical form of a model. In general, more mathematical tools have been developed to test the fit
    of statistical models than models involving differential equations. Tools from non-parametric statistics can
    sometimes be used to evaluate how well the data fit a known distribution or to come up with a general model that
    makes only minimal assumptions about the model's mathematical form.
Mathematical model                                                                                                                                  223


    Scope of the model
    Assessing the scope of a model, that is, determining what situations the model is applicable to, can be less
    straightforward. If the model was constructed based on a set of data, one must determine for which systems or
    situations the known data is a "typical" set of data.
    The question of whether the model describes well the properties of the system between data points is called
    interpolation, and the same question for events or data points outside the observed data is called extrapolation.
    As an example of the typical limitations of the scope of a model, in evaluating Newtonian classical mechanics, we
    can note that Newton made his measurements without advanced equipment, so he could not measure properties of
    particles travelling at speeds close to the speed of light. Likewise, he did not measure the movements of molecules
    and other small particles, but macro particles only. It is then not surprising that his model does not extrapolate well
    into these domains, even though his model is quite sufficient for ordinary life physics.


    Philosophical considerations
    Many types of modeling implicitly involve claims about causality. This is usually (but not always) true of models
    involving differential equations. As the purpose of modeling is to increase our understanding of the world, the
    validity of a model rests not only on its fit to empirical observations, but also on its ability to extrapolate to situations
    or data beyond those originally described in the model. One can argue that a model is worthless unless it provides
    some insight which goes beyond what is already known from direct investigation of the phenomenon being studied.
    An example of such criticism is the argument that the mathematical models of Optimal foraging theory do not offer
    insight that goes beyond the common-sense conclusions of evolution and other basic principles of ecology.[7]


    References
    [1] landinfo.com, definition of map projection (http:/ / www. landinfo. com/ resources_dictionaryMP. htm)
    [2] Gallistel. The Organization of Learning. 1990.
    [3] Dead reckoning (path integration) requires the hippocampal formation: evidence from spontaneous exploration and spatial learning tasks in
        light (allothetic) and dark (idiothetic) tests, IQ Whishaw, DJ Hines, DG Wallace, Behavioural Brain Research 127 (2001) 49 – 69 (http:/ /
        hsinnamon. web. wesleyan. edu/ wescourses/ NSB-Psyc255/ Readings/ 17. Spatial Limbic System/ Whishaw. pdf)
    [4] Stanislav Andreski (1972) Social Sciences as Sorcery, St. Martin’s Press
    [5] Clifford Truesdell (1984) An Idiot’s Fugitive Essays on Science, 121–7, Springer ISBN 3-540-90703-3
    [6] MacKay, D.J. Information Theory, Inference, and Learning Algorithms, Cambridge, (2003-2004). ISBN 0-521-64298-1
    [7] "Optimal Foraging Theory: A Critical Review - Annual Review of Ecology and Systematics, 15(1):523 - First Page Image" (http:/ /
        arjournals. annualreviews. org/ doi/ abs/ 10. 1146/ annurev. es. 15. 110184. 002515?journalCode=ecolsys). Arjournals.annualreviews.org.
        2003-11-28. . Retrieved 2011-03-27.



    Further reading
    Books
    • Aris, Rutherford [ 1978 ] ( 1994 ). Mathematical Modelling Techniques, New York : Dover. ISBN 0-486-68131-9
    • Bender, E.A. [ 1978 ] ( 2000 ). An Introduction to Mathematical Modeling, New York : Dover. ISBN
      0-486-41180-X
    • Lin, C.C. & Segel, L.A. ( 1988 ). Mathematics Applied to Deterministic Problems in the Natural Sciences,
      Philadelphia : SIAM. ISBN 0-89871-229-7
    • Gershenfeld, N. (1998) The Nature of Mathematical Modeling, Cambridge University Press ISBN 0-521-57095-6
      .
    • Yang, X.-S. (2008) Mathematical Modelling for Earth Sciences, Dunedin Academic, ISBN 1-903765-92-7 .
    Specific applications
    • Peierls, Rudolf (January 1980) Model-making in physics (http://www.informaworld.com/smpp/
      content~content=a752582770~db=all~order=page), Contemporary Physics Volume 21(1): 3–17.
Mathematical model                                                                                                        224


    • Korotayev A., Malkov A., Khaltourina D. (2006). Introduction to Social Macrodynamics: Compact Macromodels
      of the World System Growth (http://cliodynamics.ru/index.php?option=com_content&task=view&id=124&
      Itemid=70). Moscow : Editorial URSS (http://urss.ru/cgi-bin/db.pl?cp=&lang=en&blang=en&list=14&
      page=Book&id=34250) ISBN 5-484-00414-4 .


    External links
    General reference material
    • McLaughlin, Michael P. ( 1999 ) 'A Tutorial on Mathematical Modeling' (http://www.causascientia.org/
      math_stat/Tutorial.pdf)  PDF (264 KiB)
    • Patrone, F. Introduction to modeling via differential equations (http://www.fioravante.patrone.name/mat/u-u/
      en/differential_equations_intro.htm), with critical remarks.
    • Plus teacher and student package: Mathematical Modelling. (http://plus.maths.org/issue44/package/index.
      html) Brings together all articles on mathematical modeling from Plus, the online mathematics magazine
      produced by the Millennium Mathematics Project at the University of Cambridge.
    Philosophical background
    • Frigg, R. and S. Hartmann, Models in Science (http://plato.stanford.edu/entries/models-science/), in: The
      Stanford Encyclopedia of Philosophy, (Spring 2006 Edition)
    • Griffiths, E. C. (2010) What is a model? (http://www.emily-griffiths.postgrad.shef.ac.uk/models.pdf)



    Simulation modeling
    Simulation modeling is the process of creating and analyzing a digital prototype of a physical model to predict its
    performance in the real world. Simulation modeling is used to help designers and engineers understand whether,
    under what conditions, and in which ways a part could fail and what loads it can withstand. Simulation modeling can
    also help predict fluid flow and heat transfer patterns.


    Uses of Simulation Modeling
    Simulation modeling allows designers and engineers to avoid repeated building of multiple physical prototypes to
    analyze designs for new or existing parts. Before creating the physical prototype, users can virtually investigate
    many digital prototypes. Using the technique, they can:
    • Optimize geometry for weight and strength
    • Select materials that meet weight, strength, and budget requirements
    • Simulate part failure and identify the loading conditions that cause them
    • Assess extreme environmental conditions or loads not easily tested on physical prototypes, such as earthquake
      shock load
    • Verify hand calculations
    • Validate the likely safety and survival of a physical prototype before testing
Simulation modeling                                                                                                             225


    Typical Simulation Modeling Workflow
    Simulation modeling follows a process much like this:
    1. Use a 2D or 3D CAD tool to develop a virtual model, also known as a digital prototype, to represent a design.
    2. Generate a 2D or 3D mesh for analysis calculations. Automatic algorithms can create finite element meshes, or
       users can create structured meshes to maintain control over element quality.
    3. Define finite element analysis data (loads, constraints, or materials) based on analysis type (thermal, structural, or
       fluid). Apply boundary conditions to the model to represent how the part will be restrained during use.
    4. Perform finite element analysis, review results, and make engineering judgments based on results.


    Simulation Modeling Software Programs
    •     Abaqus
    •     ANSYS
    •     Autodesk Algor Simulation
    •     Autodesk Inventor Professional
    •     Nastran
    •     Solidworks Simulation


    References
    •     The CBS Interactive Business Network [1]
    •     University of Central Florida, Institute for Simulation and Training [2]
    •     Winsberg, Eric (2003), Simulated Experiments: Methodology for a Virtual World [3]
    •     Roger D. Smith: "Simulation: The Engine Behind the Virtual World" [4], eMatter, December, 1999


    External links
    •     Institute for Simulation and Training, University of Central Florida [5]
    •     National Center for Simulation [6]
    •     Simulation Interoperability Standards Organization [7]
    •     The Society for Modeling and Simulation International (Formerly the Society of Computer Simulation) [8]


    References
    [1]   http:/ / www. bnet. com/ topics/ simulation+ model
    [2]   http:/ / www. ist. ucf. edu/ background. htm
    [3]   http:/ / www. cas. usf. edu/ ~ewinsb/ methodology. pdf
    [4]   http:/ / www. modelbenders. com/ Bookshop/ techpapers. html
    [5]   http:/ / www. ist. ucf. edu
    [6]   http:/ / www. simulationinformation. com
    [7]   http:/ / www. sisostds. org
    [8]   http:/ / www. scs. org
Dynamical system                                                                                                                        226



    Dynamical system
    A dynamical system is a concept in mathematics where a fixed
    rule describes the time dependence of a point in a geometrical
    space. Examples include the mathematical models that describe
    the swinging of a clock pendulum, the flow of water in a pipe, and
    the number of fish each springtime in a lake.
    At any given time a dynamical system has a state given by a set of
    real numbers (a vector) that can be represented by a point in an
    appropriate state space (a geometrical manifold). Small changes in
    the state of the system create small changes in the numbers. The
    evolution rule of the dynamical system is a fixed rule that
    describes what future states follow from the current state. The rule
    is deterministic; in other words, for a given time interval only one
    future state follows from the current state.                               The Lorenz attractor arises in the study of the Lorenz
                                                                                        Oscillator, a dynamical system.

    Overview
    The concept of a dynamical system has its origins in Newtonian mechanics. There, as in other natural sciences and
    engineering disciplines, the evolution rule of dynamical systems is given implicitly by a relation that gives the state
    of the system only a short time into the future. (The relation is either a differential equation, difference equation or
    other time scale.) To determine the state for all future times requires iterating the relation many times—each
    advancing time a small step. The iteration procedure i