VIEWS: 48 PAGES: 391 CATEGORY: Economics POSTED ON: 2/25/2013 Attribution-ShareAlike
Econometrics Mathematical Economics PDF generated using the open source mwlib toolkit. See http://code.pediapress.com/ for more information. PDF generated at: Sat, 23 Feb 2013 22:26:53 UTC Contents Articles Econometrics 1 Mathematical economics 9 Statistics 30 Bayesian inference 39 Statistical hypothesis testing 51 Estimation theory 71 Semiparametric model 77 Non-parametric statistics 78 Simulation 80 Probability distribution 101 Single-equation methods (econometrics) 107 Cross-sectional study 108 Spatial analysis 110 Effect size 120 Time series 130 Panel data 138 Truncated regression model 140 Censored regression model 141 Poisson regression 142 Discrete choice 145 Simultaneous equations model 156 Survival analysis 159 Survey methodology 164 Index (economics) 169 Aggregate demand 171 Operations research 176 Decision theory 185 Neural network 189 Econometric model 197 Statistical model 199 Model selection 200 Economic forecasting 202 Mathematical optimization 203 Programming paradigm 215 Mathematical model 218 Simulation modeling 224 Dynamical system 226 Existence theorem 235 Stability theory 236 Economic equilibrium 239 Computational economics 245 Input-output model 247 Computable general equilibrium 252 Game theory 255 Bargaining 274 Cooperative game 278 Non-cooperative game 286 Stochastic game 286 Sequential game 288 Evolutionary game theory 289 Matching 310 Economic data 310 Computer program 314 Survey sampling 318 Official statistics 321 Comparison of statistical packages 330 Software 342 Experimental economics 348 Experimental psychology 355 Social psychology 364 Field experiment 376 References Article Sources and Contributors 379 Image Sources, Licenses and Contributors 386 Article Licenses License 388 Econometrics 1 Econometrics Econometrics is the application of mathematics , statistical methods and , more recently, computer science , to economic data and described as the branch of economics that aims to give empirical content to economic relations. [1] More precisely, it is "the quantitative analysis of actual economic phenomena based on the concurrent development of theory and observation, related by appropriate methods of inference."[2] An influential introductory economics textbook describes econometrics as allowing economists "to sift through mountains of data to extract simple relationships."[3] The first known use of the term "econometrics" (in cognate form) was by Paweł Ciompa in 1910. Ragnar Frisch is credited with coining the term in the sense that it is used today.[4] Econometrics is the unification of economics, mathematics, and statistics. This unification produces more than the sum of its parts.[5] Econometrics adds empirical content to economic theory allowing theories to be tested and used for forecasting and policy evaluation.[6] Basic econometric models: linear regression The basic tool for econometrics is the linear regression model. In modern econometrics, other statistical tools are frequently used, but linear regression is still the most frequently used starting point for an analysis.[7] Estimating a linear regression on two variables can be visualized as fitting a line through data points representing paired values of the independent and dependent variables. For example, consider Okun's law, which relates GDP growth to the unemployment rate. This relationship is represented in a linear regression where the change in unemployment rate ( ) is a function of an intercept ( ), a given value of GNP growth multiplied by a slope coefficient and an error term, : The unknown parameters and can be estimated. Here is Okun's law representing the relationship between estimated to be -1.77 and is estimated to be 0.83. This means that if GDP growth and the unemployment rate. The GNP grew one point faster, the unemployment rate would be predicted fitted line is found using regression analysis. to drop by .94 points (-1.77*1+0.83). The model could then be tested for statistical significance as to whether an increase in growth is associated with a decrease in the unemployment, as hypothesized. If the estimate of were not significantly different from 0, we would fail to find evidence that changes in the growth rate and unemployment rate were related. Theory Econometric theory uses statistical theory to evaluate and develop econometric methods. Econometricians try to find estimators that have desirable statistical properties including unbiasedness, efficiency, and consistency. An estimator is unbiased if its expected value is the true value of the parameter; It is consistent if it converges to the true value as sample size gets larger, and it is efficient if the estimator has lower standard error than other unbiased estimators for a given sample size. Ordinary least squares (OLS) is often used for estimation since it provides the BLUE or "best linear unbiased estimator" (where "best" means most efficient, unbiased estimator) given the Gauss-Markov assumptions. When these assumptions are violated or other statistical properties are desired, other estimation techniques such as maximum likelihood estimation, generalized method of moments, or generalized least squares are used. Estimators that incorporate prior beliefs are advocated by those who favor Bayesian statistics over traditional, classical or "frequentist" approaches. Econometrics 2 Gauss-Markov theorem The Gauss-Markov theorem shows that the OLS estimator is the best (minimum variance), unbiased estimator assuming the model is linear, the expected value of the error term is zero, errors are homoskedastic and not autocorrelated, and there is no perfect multicollinearity. Linearity The dependent variable is assumed to be a linear function of the variables specified in the model. The specification must be linear in its parameters. This does not mean that there must be a linear relationship between the independent and dependent variables. The independent variables can take non-linear forms as long as the parameters are linear. The equation qualifies as linear while , does not. Data transformations can be used to convert an equation into a linear form. For example, the Cobb-Douglas equation—often used in economics—is nonlinear: But it can be expressed in linear form by taking the natural logarithm of both sides:[8] This assumption also covers specification issues: assuming that the proper functional form has been selected and there are no omitted variables. Expected error is zero The expected value of the error term is assumed to be zero. This assumption can be violated if the measurement of the dependent variable is consistently positive or negative. The miss-measurement will bias the estimation of the intercept parameter, but the slope parameters will remain unbiased.[9] The intercept may also be biased if there is a logarithmic transformation. See the Cobb-Douglas equation above. The multiplicative error term will not have a mean of 0, so this assumption will be violated.[10] This assumption can also be violated in limited dependent variable models. In such cases, both the intercept and slope parameters may be biased.[11] Spherical errors Error terms are assumed to be spherical otherwise the OLS estimator is inefficient. The OLS estimator remains unbiased, however. Spherical errors occur when errors have both uniform variance (homoscedasticity) and are uncorrelated with each other.[12] The term "spherical errors" will describe the multivariate normal distribution: if in the multivariate normal density, then the equation f(x)=c is the formula for a “ball” centered at μ with radius σ in n-dimensional space.[13] Heteroskedacity occurs when the amount of error is correlated with an independent variable. For example, in a regression on food expenditure and income, the error is correlated with income. Low income people generally spend a similar amount on food, while high income people may spend a very large amount or as little as low income people spend. Heteroskedacity can also be caused by changes in measurement practices. For example, as statistical offices improve their data, measurement error decreases, so the error term declines over time. This assumption is violated when there is autocorrelation. Autocorrelation can be visualized on a data plot when a given observation is more likely to lie above a fitted line if adjacent observations also lie above the fitted regression line. Autocorrelation is common in time series data where a data series may experience "inertia."[14] If a dependent variable takes a while to fully absorb a shock. Spatial autocorrelation can also occur geographic areas are likely to have similar errors. Autocorrelation may be the result of misspecification such as choosing the wrong functional Econometrics 3 form. In these cases, correcting the specification is the preferred way to deal with autocorrelation. In the presence of non-spherical errors, the generalized least squares estimator can be shown to be BLUE.[15] Exogeneity of independent variables This assumption is violated if the variables are endogenous. Endogeneity can be the result of simultaneity, where causality flows back and forth between both the dependent and independent variable. Instrumental variable techniques are commonly used to address this problem. Full rank The sample data matrix must have full rank or OLS cannot be estimated. There must be at least one observation for every parameter being estimated and the data cannot have perfect multicollinearity.[16] Perfect multicollinearity will occur in a "dummy variable trap" when a base dummy variable is not omitted resulting in perfect correlation between the dummy variables and the constant term. Multicollinearity (as long as it is not "perfect") can be present resulting in a less efficient, but still unbiased estimate. Methods Applied econometrics uses theoretical econometrics and real-world data for assessing economic theories, developing econometric models, analyzing economic history, and forecasting.[17] Econometrics may use standard statistical models to study economic questions, but most often they are with observational data, rather than in controlled experiments. In this, the design of observational studies in econometrics is similar to the design of studies in other observational disciplines, such as astronomy, epidemiology, sociology and political science. Analysis of data from an observational study is guided by the study protocol, although exploratory data analysis may by useful for generating new hypotheses.[18] Economics often analyzes systems of equations and inequalities, such as supply and demand hypothesized to be in equilibrium. Consequently, the field of econometrics has developed methods for identification and estimation of simultaneous-equation models. These methods are analogous to methods used in other areas of science, such as the field of system identification in systems analysis and control theory. Such methods may allow researchers to estimate models and investigate their empirical consequences, without directly manipulating the system. One of the fundamental statistical methods used by econometricians is regression analysis. For an overview of a linear implementation of this framework, see linear regression. Regression methods are important in econometrics because economists typically cannot use controlled experiments. Econometricians often seek illuminating natural experiments in the absence of evidence from controlled experiments. Observational data may be subject to omitted-variable bias and a list of other problems that must be addressed using causal analysis of simultaneous-equation models.[19] Econometrics 4 Experimental economics In recent decades, econometricians have increasingly turned to use of experiments to evaluate the often-contradictory conclusions of observational studies. Here, controlled and randomized experiments provide statistical inferences that may yield better empirical performance than do purely observational studies.[20] Data Data sets to which econometric analyses are applied can be classified as time-series data, cross-sectional data, panel data, and multidimensional panel data. Time-series data sets contain observations over time; for example, inflation over the course of several years. Cross-sectional data sets contain observations at a single point in time; for example, many individuals' incomes in a given year. Panel data sets contain both time-series and cross-sectional observations. Multi-dimensional panel data sets contain observations across time, cross-sectionally, and across some third dimension. For example, the Survey of Professional Forecasters contains forecasts for many forecasters (cross-sectional observations), at many points in time (time series observations), and at multiple forecast horizons (a third dimension). Instrumental variables In many econometric contexts, the commonly-used ordinary least squares method may not recover the theoretical relation desired or may produce estimates with poor statistical properties, because the assumptions for valid use of the method are violated. One widely-used remedy is the method of instrumental variables (IV). For an economic model described by more than one equation, simultaneous-equation methods may be used to remedy similar problems, including two IV variants, Two-Stage Least Squares (2SLS), and Three-Stage Least Squares (3SLS).[21] Computational methods Computational concerns are important for evaluating econometric methods and for use in decision making.[22] Such concerns include mathematical well-posedness: the existence, uniqueness, and stability of any solutions to econometric equations. Another concern is the numerical efficiency and accuracy of software.[23] A third concern is also the usability of econometric software.[24] Example A simple example of a relationship in econometrics from the field of labor economics is: This example assumes that the natural logarithm of a person's wage is a linear function of (among other things) the number of years of education that person has acquired. The parameter measures the increase in the natural log of the wage attributable to one more year of education. The term is a random variable representing all other factors that may have direct influence on wage. The econometric goal is to estimate the parameters, under specific assumptions about the random variable . For example, if is uncorrelated with years of education, then the equation can be estimated with ordinary least squares. If the researcher could randomly assign people to different levels of education, the data set thus generated would allow estimation of the effect of changes in years of education on wages. In reality, those experiments cannot be conducted. Instead, the econometrician observes the years of education of and the wages paid to people who differ along many dimensions. Given this kind of data, the estimated coefficient on Years of Education in the equation above reflects both the effect of education on wages and the effect of other variables on wages, if those other variables were correlated with education. For example, people born in certain places may have higher wages and higher levels of education. Unless the econometrician controls for place of birth in the above equation, the effect of birthplace on wages may be falsely attributed to the effect of education on wages. Econometrics 5 The most obvious way to control for birthplace is to include a measure of the effect of birthplace in the equation above. Exclusion of birthplace, together with the assumption that is uncorrelated with education produces a misspecified model. Another technique is to include in the equation additional set of measured covariates which are not instrumental variables, yet render identifiable.[25] An overview of econometric methods used to study this problem can be found in Card (1999).[26] Journals The main journals which publish work in econometrics are Econometrica, the Journal of Econometrics, the Review of Economics and Statistics, Econometric Theory, the Journal of Applied Econometrics, Econometric Reviews, the Econometrics Journal,[27] Applied Econometrics and International Development, the Journal of Business & Economic Statistics, and the Journal of Economic and Social Measurement [28]. Limitations and criticisms See also Criticisms of econometrics Like other forms of statistical analysis, badly specified econometric models may show a spurious correlation where two variables are correlated but causally unrelated. In a study of the use of econometrics in major economics journals, McCloskey concluded that economists report p values (following the Fisherian tradition of tests of significance of point null-hypotheses), neglecting concerns of type II errors; economists fail to report estimates of the size of effects (apart from statistical significance) and to discuss their economic importance. Economists also fail to use economic reasoning for model selection, especially for deciding which variables to include in a regression.[29][30] In some cases, economic variables cannot be experimentally manipulated as treatments randomly assigned to subjects.[31] In such cases, economists rely on observational studies, often using data sets with many strongly associated covariates, resulting in enormous numbers of models with similar explanatory ability but different covariates and regression estimates. Regarding the plurality of models compatible with observational data-sets, Edward Leamer urged that "professionals ... properly withhold belief until an inference can be shown to be adequately insensitive to the choice of assumptions".[32] Economists from the Austrian School argue that aggregate economic models are not well suited to describe economic reality because they waste a large part of specific knowledge. Friedrich Hayek in his The Use of Knowledge in Society argued that "knowledge of the particular circumstances of time and place" is not easily aggregated and is often ignored by professional economists.[33][34] Notes [1] M. Hashem Pesaran (1987). "Econometrics," The New Palgrave: A Dictionary of Economics, v. 2, p. 8 [pp. 8-22]. Reprinted in J. Eatwell et al., eds. (1990). Econometrics: The New Palgrave, p. 1 (http:/ / books. google. com/ books?id=gBsgr7BPJsoC& dq=econometrics& printsec=find& pg=PA1=false#v=onepage& q& f=false) [pp. 1-34]. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_E000007& edition=current& q=Econometrics& topicid=& result_number=2) (2008 revision by J. Geweke, J. Horowitz, and H. P. Pesaran). [2] P. A. Samuelson, T. C. Koopmans, and J. R. N. Stone (1954). "Report of the Evaluative Committee for Econometrica," Econometrica 22(2), p. 142. [p p. 141 (http:/ / www. jstor. org/ pss/ 1907538)-146], as described and cited in Pesaran (1987) above. [3] Paul A. Samuelson and William D. Nordhaus, 2004. Economics. 18th ed., McGraw-Hill, p. 5. [4] • H. P. Pesaran (1990), "Econometrics," Econometrics: The New Palgrave, p. 2 (http:/ / books. google. com/ books?id=gBsgr7BPJsoC& dq=econometrics& printsec=find& pg=PA2=false#v=onepage& q& f=false), citing Ragnar Frisch (1936), "A Note on the Term 'Econometrics'," Econometrica, 4(1), p. 95. • Aris Spanos (2008), "statistics and economics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_S000502& edition=current& q=statistics& topicid=& result_number=1) [5] Greene, 1. [6] Geweke, Horowitz & Pesaran 2008 [7] Greene (2012), 12. Econometrics 6 [8] Kennedy 2003, p. 110 [9] Kennedy 2003, p. 129 [10] Kennedy 2003, p. 131 [11] Kennedy 2003, p. 130 [12] Kennedy 2003, p. 133 [13] Greene 2012, p. 23-note [14] Greene 2010, p. 22 [15] Kennedy 2003, p. 135 [16] Kennedy 2003, p. 205 [17] Clive Granger (2008). "forecasting," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_F000161& edition=current& q=forecast& topicid=& result_number=7) [18] Herman O. Wold (1969). "Econometrics as Pioneering in Nonexperimental Model Building," Econometrica, 37(3), pp. 369 (http:/ / www. jstor. org/ pss/ 1912787)-381. [19] Edward E. Leamer (2008). "specification problems in econometrics," The New Palgrave Dictionary of Economics. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_S000200& edition=current& q=Specification problems in econometrics& topicid=& result_number=1) [20] • H. Wold 1954. "Causality and Econometrics," Econometrica, 22(2), p p. 162 (http:/ / www. jstor. org/ pss/ 1907540)-177. • Kevin D. Hoover (2008). "causality in economics and econometrics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000569& q=experimental methods in economics& topicid=& result_number=11) and galley proof. (http:/ / econ. duke. edu/ ~kdh9/ Source Materials/ Research/ Palgrave_Causality_Final. pdf) [21] Peter Kennedy (economist) (2003). A Guide to Econometrics, 5th ed. Description (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?ttype=2& tid=9577), preview (http:/ / books. google. com/ books?id=B8I5SP69e4kC& printsec=find& pg=PR5=gbs_atb#v=onepage& q& f=false), and TOC (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?ttype=2& tid=9577& mode=toc), ch. 9, 10, 13, and 18. [22] • Keisuke Hirano (2008). "decision theory in econometrics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_D000244& edition=current& q=Computational economics& topicid=& result_number=19). • James O. Berger (2008). "statistical decision theory," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_S000251& edition=& field=keyword& q=statistical decision theory& topicid=& result_number=1) [23] B. D. McCullough and H. D. Vinod (1999). "The Numerical Reliability of Econometric Software," Journal of Economic Literature, 37(2), pp. 633-665 (http:/ / www. pages. drexel. edu/ ~bdm25/ jel. pdf). [24] • Vassilis A. Hajivassiliou (2008). "computational methods in econometrics," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000559& edition=current& q=& result_number=1) • Richard E. Quandt (1983). "Computational Problems and Methods," ch. 12, in Handbook of Econometrics, v. 1, pp. 699 (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1573441283010168)-764. • Ray C. Fair (1996). "Computational Methods for Macroeconometric Models," Handbook of Computational Economics, v. 1, pp. (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002196010052143)-169. [25] Judea Pearl (2000). Causality: Model, Reasoning, and Inference, Cambridge University Press. [26] David Card (1999) "The Causal Effect of Education on Earning," in Ashenfelter, O. and Card, D., (eds.) Handbook of Labor Economics, pp 1801-63. [27] http:/ / www. wiley. com/ bw/ journal. asp?ref=1368-4221 [28] http:/ / www. iospress. nl/ html/ 07479662. php [29] McCloskey (May 1985). "The Loss Function has been mislaid: the Rhetoric of Significance Tests". American Economic Review 75 (2). [30] Stephen T. Ziliak and Deirdre N. McCloskey (2004). "Size Matters: The Standard Error of Regressions in the American Economic Review," Journal of Socio-economics, 33(5), pp. 527-46 (http:/ / faculty. roosevelt. edu/ Ziliak/ doc/ Size Matters Journal of Socio-Economics Ziliak and McCloskey. pdf) (press +). [31] Leamer, Edward (March 1983). "Let's Take the Con out of Econometrics" (http:/ / www. jstor. org/ pss/ 1803924). American Economic Review 73 (1): 34. . [32] Leamer, Edward (March 1983). "Let's Take the Con out of Econometrics" (http:/ / www. jstor. org/ pss/ 1803924). American Economic Review 73 (1): 43. . [33] Robert F. Garnett. What Do Economists Know? New Economics of Knowledge. Routledge, 1999. ISBN 978-0-415-15260-0. p. 170 [34] G. M. P. Swann. Putting Econometrics in Its Place: A New Direction in Applied Economics. Edward Elgar Publishing, 2008. ISBN 978-1-84720-776-0. p. 62-64 Econometrics 7 References • Handbook of Econometrics Elsevier. Links to volume chapter-preview links: Zvi Griliches and Michael D. Intriligator, ed. (1983). v. 1 (http://www.sciencedirect.com/science/handbooks/ 15734412/1); (1984), v. 2 (http://www.sciencedirect.com/science/handbooks/15734412/2); (1986), description (http://www.elsevier.com/wps/find/bookdescription.cws_home/601080/ description#description), v. 3 (http://www.sciencedirect.com/science/handbooks/15734412/3); (1994), description (http://www.elsevier.com/wps/find/bookdescription.cws_home/601081/ description#description), v. 4 (http://www.sciencedirect.com/science/handbooks/15734412/4) Robert F. Engle and Daniel L. McFadden, ed. (2001). Description (http://www.elsevier.com/wps/find/ bookdescription.cws_home/601082/description#description), v. 5 (http://www.sciencedirect.com/science/ handbooks/15734412/5) James J. Heckman and Edward E. Leamer, ed. (2007). Description (http://www.elsevier.com/wps/find/ bookdescription.cws_home/712946/description#description), v. 6A (http://www.sciencedirect.com/science/ handbooks/15734412/6/part/PA) & v. 6B (http://www.sciencedirect.com/science/handbooks/15734412/6/ part/PB) • Handbook of Statistics, v. 11, Econometrics (1993), Elsevier. Links to first-page chapter previews. (http://www. sciencedirect.com/science/handbooks/01697161/11) • International Encyclopedia of the Social & Behavioral Sciences (2001), Statistics, "Econometrics and Time Series," links (http://www.sciencedirect.com/science?_ob=RefWorkIndexURL&_idxType=SC& _cdi=23486&_refWorkId=21&_explode=151000377,151000380&_alpha=&_acct=C000050221& _version=1&_userid=10&md5=10d43da5ed3104bf3d8bb99f72c80e11&refID=151000380#151000380) to first-page previews of 21 articles. • Angrist, Joshua & Pischke, Jörn‐Steffen (2010). "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics], 24(2), , pp. 3–30. Abstract. (http://www. ingentaconnect.com/content/aea/jep/2010/00000024/00000002/art00001) • Eatwell, John, et al., eds. (1990). Econometrics: The New Palgrave. Article-preview links (http://books.google. com/books?id=gBsgr7BPJsoC&dq=econometrics&printsec=find&pg=PR5=false#v=onepage&q&f=false) (from The New Palgrave: A Dictionary of Economics, 1987). • Geweke, John; Horowitz, Joel; Pesaran, Hashem (2008). Durlauf, Steven N.; Blume, Lawrence E.. eds. "Econometrics" (http://www.dictionaryofeconomics.com.proxyau.wrlc.org/article?id=pde2008_E000007). The New Palgrave Dictionary of Economics (Palgrave Macmillan). doi:10.1057/9780230226203.0425. • Greene, William H. (1999, 4th ed.) Econometric Analysis, Prentice Hall. • Hayashi, Fumio. (2000) Econometrics, Princeton University Press. ISBN 0-691-01018-8 Description and contents links. (http://press.princeton.edu/titles/6946.html) • Hamilton, James D. (1994) Time Series Analysis, Princeton University Press. Description (http://press. princeton.edu/titles/5386.html) and preview. (http://books.google.com/books/p/ princeton?id=B8_1UBmqVUoC&printsec=frontcover&cd=1&source=gbs_ViewAPI&hl=en#v=onepage&q& f=false) • Hughes Hallett, Andrew J. "Econometrics and the Theory of Economic Policy: The Tinbergen-Theil Contributions 40 Years On," Oxford Economic Papers (1989) 41#1 pp 189-214 • Kelejian, Harry H., and Wallace E. Oates (1989, 3rd ed.) Introduction to Econometrics. • Kennedy, Peter (2003). A guide to econometrics. Cambridge, Mass: MIT Press. ISBN 978-0-262-61183-1. • Russell Davidson and James G. MacKinnon (2004). Econometric Theory and Methods. New York: Oxford University Press. Description. (http://www.oup.com/us/catalog/general/subject/Economics/Econometrics/ ~~/dmlldz11c2EmY2k9OTc4MDE5NTEyMzcyMg==?view=usa&ci=9780195123722#Description) • Mills, Terence C., and Kerry Patterson, ed. Palgrave Handbook of Econometrics: Econometrics 8 (2007) v. 1: Econometric Theoryv. 1. Links (http:/ / www. palgrave. com/ products/ title. aspx?pid=269866) to description and contents. (2009) v. 2, Applied Econometrics. Palgrave Macmillan. ISBN 978-1-4039-1799-7 Links (http:/ / www. palgrave.com/products/title.aspx?PID=267962) to description and contents. • Pearl, Judea (2009, 2nd ed.). Causality: Models, Reasoning and Inference, Cambridge University Press, Description (http://books.google.com/books?id=wnGU_TsW3BQC&source=gbs_navlinks_s), TOC (http:// bayes.cs.ucla.edu/BOOK-09/book-toc-final.pdf), and preview, ch. 1-10 (http://books.google.com/ books?id=wnGU_TsW3BQC&printsec=find&pg=PR7=gbs_atb#v=onepage&q&f=false) and ch. 11 (http:// bayes.cs.ucla.edu/BOOK-09/ch11-toc-plus-p331-final.pdf). 5 economics-journal reviews (http://bayes.cs. ucla.edu/BOOK-2K/), including Kevin D. Hoover, Economics Journal. • Pindyck, Robert S., and Daniel L. Rubinfeld (1998, 4th ed.). Econometric Methods and Economic Forecasts, McGraw-Hill. • Studenmund, A.H. (2011, 6th ed.). Using Econometrics: A Practical Guide. Contents (chapter-preview) links. (http://www.coursesmart.com/9780131367760/chap01) • Wooldridge, Jeffrey (2003). Introductory Econometrics: A Modern Approach. Mason: Thomson South-Western. ISBN 0-324-11364-1 Chapter-preview links in brief (http://books.google.com/books?id=64vt5TDBNLwC& printsec=find&pg=PR3=gbs_atb#v=onepage&q&f=false) and detail. (http://books.google.com/ books?id=64vt5TDBNLwC&printsec=find&pg=PR4=gbs_atb#v=onepage&q&f=false) Further reading • Econometric Theory book on Wikibooks • Giovanini, Enrico Understanding Economic Statistics (http://www.oecd.org/statistics/ understandingeconomicstatistics), OECD Publishing, 2008, ISBN 978-92-64-03312-2 External links • Econometric Society (http://www.econometricsociety.org) • The Econometrics Journal (http://www.ectj.org) • Econometric Links (http://www.econometriclinks.com) • Teaching Econometrics (http://www.economicsnetwork.ac.uk/subjects/econometrics.htm) (Index by the Economics Network (UK)) • Applied Econometric Association (http://www.aea-eu.com) • The Society for Financial Econometrics (http://sofie.stern.nyu.edu/) Mathematical economics 9 Mathematical economics Mathematical economics is the application of mathematical methods to represent theories and analyze problems in economics. An advantage claimed for the approach is its allowing formulation of theoretical relationships with rigor, generality, and simplicity.[1] By convention, the applied methods refer to those beyond simple geometry, such as differential and integral calculus, difference and differential equations, matrix algebra, mathematical programming, and other computational methods.[2][3] It is argued that mathematics allows economists to form meaningful, testable propositions about wide-ranging and complex subjects which could less easily be expressed informally. Further, the language of mathematics allows economists to make specific, positive claims about controversial or contentious subjects that would be impossible without mathematics.[4] Much of economic theory is currently presented in terms of mathematical economic models, a set of stylized and simplified mathematical relationships asserted to clarify assumptions and implications.[5] Broad applications include: • optimization problems as to goal equilibrium, whether of a household, business firm, or policy maker • static (or equilibrium) analysis in which the economic unit (such as a household) or economic system (such as a market or the economy) is modeled as not changing • comparative statics as to a change from one equilibrium to another induced by a change in one or more factors • dynamic analysis, tracing changes in an economic system over time, for example from economic growth.[3][6][7] Formal economic modeling began in the 19th century with the use of differential calculus to represent and explain economic behavior, such as utility maximization, an early economic application of mathematical optimization. Economics became more mathematical as a discipline throughout the first half of the 20th century, but introduction of new and generalized techniques in the period around the Second World War, as in game theory, would greatly broaden the use of mathematical formulations in economics.[8][7] This rapid systematizing of economics alarmed critics of the discipline as well as some noted economists. John Maynard Keynes, Robert Heilbroner, Friedrich Hayek and others have criticized the broad use of mathematical models for human behavior, arguing that some human choices are irreducible to mathematics. History The use of mathematics in the service of social and economic analysis dates back to the 17th century. Then, mainly in German universities, a style of instruction emerged which dealt specifically with detailed presentation of data as it related to public administration. Gottfried Achenwall lectured in this fashion, coining the term statistics. At the same time, a small group of professors in England established a method of "reasoning by figures upon things relating to government" and referred to this practice as Political Arithmetick.[9] Sir William Petty wrote at length on issues that would later concern economists, such as taxation, Velocity of money and national income, but while his analysis was numerical, he rejected abstract mathematical methodology. Petty's use of detailed numerical data (along with John Graunt) would influence statisticians and economists for some time, even though Petty's works were largely ignored by English scholars.[10] The mathematization of economics began in earnest in the 19th century. Most of the economic analysis of the time was what would later be called classical economics. Subjects were discussed and dispensed with through algebraic means, but calculus was not used. More importantly, until Johann Heinrich von Thünen's The Isolated State in 1826, economists did not develop explicit and abstract models for behavior in order to apply the tools of mathematics. Thünen's model of farmland use represents the first example of marginal analysis.[11] Thünen's work was largely theoretical, but he also mined empirical data in order to attempt to support his generalizations. In comparison to his contemporaries, Thünen built economic models and tools, rather than applying previous tools to new problems.[12] Mathematical economics 10 Meanwhile a new cohort of scholars trained in the mathematical methods of the physical sciences gravitated to economics, advocating and applying those methods to their subject,[13] and described today as moving from geometry to mechanics.[14] These included W.S. Jevons who presented paper on a "general mathematical theory of political economy" in 1862, providing an outline for use of the theory of marginal utility in political economy.[15] In 1871, he published The Principles of Political Economy, declaring that the subject as science "must be mathematical simply because it deals with quantities." Jevons expected the only collection of statistics for price and quantities would permit the subject as presented to become an exact science.[16] Others preceded and followed in expanding mathematical representations of economic problems. Marginalists and the roots of neoclassical economics Augustin Cournot and Léon Walras built the tools of the discipline axiomatically around utility, arguing that individuals sought to maximize their utility across choices in a way that could be described mathematically.[17] At the time, it was thought that utility was quantifiable, in units known as utils.[18] Cournot, Walras and Francis Ysidro Edgeworth are considered the precursors to modern mathematical economics.[19] Augustin Cournot Cournot, a professor of Mathematics, developed a mathematical treatment in 1838 for duopoly—a market condition defined by competition between two sellers.[19] This treatment of competition, first published in Researches into the Mathematical Principles of Wealth,[20] is referred to as Cournot Equilibrium quantities as a solution to two reaction functions in Cournot duopoly. It is assumed that both sellers had equal duopoly. Each reaction function is expressed as a linear equation dependent access to the market and could produce their upon quantity demanded. goods without cost. Further, it assumed that both goods were homogeneous. Each seller would vary her output based on the output of the other and the market price would be determined by the total quantity supplied. The profit for each firm would be determined by multiplying their output and the per unit Market price. Differentiating the profit function with respect to quantity supplied for each firm left a system of linear equations, the simultaneous solution of which gave the equilibrium quantity, price and profits.[21] Cournot's contributions to the mathematization of economics would be neglected for decades, but eventually influenced many of the marginalists.[21][22] Cournot's models of duopoly and Oligopoly also represent one of the first formulations of non-cooperative games. Today the solution can be given as a Nash equilibrium but Cournot's work preceded modern Game theory by over 100 years.[23] Léon Walras While Cournot provided a solution for what would later be called partial equilibrium, Léon Walras attempted to formalize discussion of the economy as a whole through a theory of general competitive equilibrium. The behavior of every economic actor would be considered on both the production and consumption side. Walras originally presented four separate models of exchange, each recursively included in the next. The solution of the resulting system of equations (both linear and non-linear) is the general equilibrium.[24] At the time, no general solution could be expressed for a system of arbitrarily many equations, but Walras's attempts produced two famous results in Mathematical economics 11 economics. The first is Walras' law and the second is the principle of tâtonnement. Walras' method was considered highly mathematical for the time and Edgeworth commented at length about this fact in his review of Éléments d'économie politique pure (Elements of Pure Economics).[25] Walras' law was introduced as a theoretical answer to the problem of determining the solutions in general equilibrium. His notation is different from modern notation but can be constructed using more modern summation notation. Walras assumed that in equilibrium, all money would be spent on all goods: every good would be sold at the market price for that good and every buyer would expend their last dollar on a basket of goods. Starting from this assumption, Walras could then show that if there were n markets and n-1 markets cleared (reached equilibrium conditions) that the nth market would clear as well. This is easiest to visualize with two markets (considered in most texts as a market for goods and a market for money). If one of two markets has reached an equilibrium state, no additional goods (or conversely, money) can enter or exit the second market, so it must be in a state of equilibrium as well. Walras used this statement to move toward a proof of existence of solutions to general equilibrium but it is commonly used today to illustrate market clearing in money markets at the undergraduate level.[26] Tâtonnement (roughly, French for groping toward) was meant to serve as the practical expression of Walrasian general equilibrium. Walras abstracted the marketplace as an auction of goods where the auctioneer would call out prices and market participants would wait until they could each satisfy their personal reservation prices for the quantity desired (remembering here that this is an auction on all goods, so everyone has a reservation price for their desired basket of goods).[27] Only when all buyers are satisfied with the given market price would transactions occur. The market would "clear" at that price—no surplus or shortage would exist. The word tâtonnement is used to describe the directions the market takes in groping toward equilibrium, settling high or low prices on different goods until a price is agreed upon for all goods. While the process appears dynamic, Walras only presented a static model, as no transactions would occur until all markets were in equilibrium. In practice very few markets operate in this manner.[28] Francis Ysidro Edgeworth Edgeworth introduced mathematical elements to Economics explicitly in Mathematical Psychics: An Essay on the Application of Mathematics to the Moral Sciences, published in 1881.[29] He adopted Jeremy Bentham's felicific calculus to economic behavior, allowing the outcome of each decision to be converted into a change in utility.[30] Using this assumption, Edgeworth built a model of exchange on three assumptions: individuals are self-interested, individuals act to maximize utility, and individuals are "free to recontract with another independently of...any third party."[31] Mathematical economics 12 Given two individuals, the set of solutions where the both individuals can maximize utility is described by the contract curve on what is now known as an Edgeworth Box. Technically, the construction of the two-person solution to Edgeworth's problem was not developed graphically until 1924 by Arthur Lyon Bowley.[33] The contract curve of the Edgeworth box (or more generally on any set of solutions to Edgeworth's problem for more actors) is referred to as the core of an economy.[34] Edgeworth devoted considerable effort An Edgeworth box displaying the contract curve an economy with two participants. to insisting that mathematical proofs Referred to as the "core" of the economy in modern parlance, there are infinitely many were appropriate for all schools of [32] solutions along the curve for economies with two participants thought in economics. While at the helm of The Economic Journal, he published several articles criticizing the mathematical rigor of rival researchers, including Edwin Robert Anderson Seligman, a noted skeptic of mathematical economics.[35] The articles focused on a back and forth over tax incidence and responses by producers. Edgeworth noticed that a monopoly producing a good that had jointness of supply but not jointness of demand (such as first class and economy on an airplane, if the plane flies, both sets of seats fly with it) might actually lower the price seen by the consumer for one of the two commodities if a tax were applied. Common sense and more traditional, numerical analysis seemed to indicate that this was preposterous. Seligman insisted that the results Edgeworth achieved were a quirk of his mathematical formulation. He suggested that the assumption of a continuous demand function and an infinitesimal change in the tax resulted in the paradoxical predictions. Harold Hotelling later showed that Edgeworth was correct and that the same result (a "diminution of price as a result of the tax") could occur with a discontinuous demand function and large changes in the tax rate).[36] Modern mathematical economics From the later-1930s, an array of new mathematical tools from the differential calculus and differential equations, convex sets, and graph theory were deployed to advance economic theory in a way similar to new mathematical methods earlier applied to physics.[8][37] The process was later described as moving from mechanics to axiomatics.[38] Differential calculus Vilfredo Pareto analyzed microeconomics by treating decisions by economic actors as attempts to change a given allotment of goods to another, more preferred allotment. Sets of allocations could then be treated as Pareto efficient (Pareto optimal is an equivalent term) when no exchanges could occur between actors that could make at least one individual better off without making any other individual worse off.[39] Pareto's proof is commonly conflated with Walrassian equilibrium or informally ascribed to Adam Smith's Invisible hand hypothesis.[40] Rather, Pareto's statement was the first formal assertion of what would be known as the first fundamental theorem of welfare economics.[41] These models lacked the inequalities of the next generation of mathematical economics. Mathematical economics 13 In the landmark treatise Foundations of Economic Analysis (1947), Paul Samuelson identified a common paradigm and mathematical structure across multiple fields in the subject, building on previous work by Alfred Marshall. Foundations took mathematical concepts from physics and applied them to economic problems. This broad view (for example, comparing Le Chatelier's principle to tâtonnement) drives the fundamental premise of mathematical economics: systems of economic actors may be modeled and their behavior described much like any other system. This extension followed on the work of the marginalists in the previous century and extended it significantly. Samuelson approached the problems of applying individual utility maximization over aggregate groups with comparative statics, which compares two different equilibrium states after an exogenous change in a variable. This and other methods in the book provided the foundation for mathematical economics in the 20th century.[7][42] Linear models Restricted models of general equilibrium were formulated by John von Neumann in 1937.[43] Unlike earlier versions, the models of von Neumann had inequality constraints. For his model of an expanding economy, von Neumann proved the existence and uniqueness of an equilibrium using his generalization of Brouwer's fixed point theorem. Von Neumann's model of an expanding economy considered the matrix pencil A - λ B with nonnegative matrices A and B; von Neumann sought probability vectors p and q and a positive number λ that would solve the complementarity equation pT (A - λ B) q = 0, along with two inequality systems expressing economic efficiency. In this model, the (transposed) probability vector p represents the prices of the goods while the probability vector q represents the "intensity" at which the production process would run. The unique solution λ represents the rate of growth of the economy, which equals the interest rate. Proving the existence of a positive growth rate and proving that the growth rate equals the interest rate were remarkable achievements, even for von Neumann.[44][45][46] Von Neumann's results have been viewed as a special case of linear programming, where von Neumann's model uses only nonnegative matrices.[47] The study of von Neumann's model of an expanding economy continues to interest mathematical economists with interests in computational economics.[48][49][50] Input-output economics In 1936, the Russian–born economist Wassily Leontief built his model of input-output analysis from the 'material balance' tables constructed by Soviet economists, which themselves followed earlier work by the physiocrats. With his model, which described a system of production and demand processes, Leontief described how changes in demand in one economic sector would influence production in another.[51] In practice, Leontief estimated the coefficients of his simple models, to address economically interesting questions. In production economics, "Leontief technologies" produce outputs using constant proportions of inputs, regardless of the price of inputs, reducing the value of Leontief models for understanding economies but allowing their parameters to be estimated relatively easily. In contrast, the von Neumann model of an expanding economy allows for choice of techniques, but the coefficients must be estimated for each technology.[52][53] Mathematical economics 14 Mathematical optimization In mathematics, mathematical optimization (or optimization or mathematical programming) refers to the selection of a best element from some set of available alternatives.[54] In the simplest case, an optimization problem involves maximizing or minimizing a real function by selecting input values of the function and computing the corresponding values of the function. The solution process includes satisfying general necessary and sufficient conditions for optimality. For optimization problems, specialized notation may be used as to the function and its input(s). More generally, optimization includes finding Red dot in z direction as maximum for paraboloid the best available element of some function given a defined domain function of (x, y) inputs and may use a variety of different computational optimization techniques.[55] Economics is closely enough linked to optimization by agents in an economy that an influential definition relatedly describes economics qua science as the "study of human behavior as a relationship between ends and scarce means" with alternative uses.[56] Optimization problems run through modern economics, many with explicit economic or technical constraints. In microeconomics, the utility maximization problem and its dual problem, the expenditure minimization problem for a given level of utility, are economic optimization problems.[57] Theory posits that consumers maximize their utility, subject to their budget constraints and that firms maximize their profits, subject to their production functions, input costs, and market demand.[58] Economic equilibrium is studied in optimization theory as a key ingredient of economic theorems that in principle could be tested against empirical data.[7][59] Newer developments have occurred in dynamic programming and modeling optimization with risk and uncertainty, including applications to portfolio theory, the economics of information, and search theory.[58] Optimality properties for an entire market system may be stated in mathematical terms, as in formulation of the two fundamental theorems of welfare economics[60] and in the Arrow–Debreu model of general equilibrium (also discussed below).[61] More concretely, many problems are amenable to analytical (formulaic) solution. Many others may be sufficiently complex to require numerical methods of solution, aided by software.[55] Still others are complex but tractable enough to allow computable methods of solution, in particular computable general equilibrium models for the entire economy.[62] Linear and nonlinear programming have profoundly affected microeconomics, which had earlier considered only equality constraints.[63] Many of the mathematical economists who received Nobel Prizes in Economics had conducted notable research using linear programming: Leonid Kantorovich, Leonid Hurwicz, Tjalling Koopmans, Kenneth J. Arrow, and Robert Dorfman, Paul Samuelson, and Robert Solow.[64] Both Kantorovich and Koopmans acknowledged that George B. Dantzig deserved to share their Nobel Prize for linear programming. Economists who conducted research in nonlinear programming also have won the Nobel prize, notably Ragnar Frisch in addition to Kantorovich, Hurwicz, Koopmans, Arrow, and Samuelson. Mathematical economics 15 Linear optimization Linear programming was developed to aid the allocation of resources in firms and in industries during the 1930s in Russia and during the 1940s in the United States. During the Berlin airlift (1948), linear programming was used to plan the shipment of supplies to prevent Berlin from starving after the Soviet blockade.[65][66] Nonlinear programming Extensions to nonlinear optimization with inequality constraints were achieved in 1951 by Albert W. Tucker and Harold Kuhn, who considered the nonlinear optimization problem: Minimize ( ) subject to ( ) ≤ 0 and ( ) = 0 where i j . ( ) is the function to be minimized . ()( = 1, ..., ) are the functions of the inequality constraints i . j ()( = 1, ..., ) are the functions of the equality constraints. In allowing inequality constraints, the Kuhn–Tucker approach generalized the classic method of Lagrange multipliers, which (until then) had allowed only equality constraints.[67] The Kuhn–Tucker approach inspired further research on Lagrangian duality, including the treatment of inequality constraints.[68][69] The duality theory of nonlinear programming is particularly satisfactory when applied to convex minimization problems, which enjoy the convex-analytic duality theory of Fenchel and Rockafellar; this convex duality is particularly strong for polyhedral convex functions, such as those arising in linear programming. Lagrangian duality and convex analysis are used daily in operations research, in the scheduling of power plants, the planning of production schedules for factories, and the routing of airlines (routes, flights, planes, crews).[69] Variational calculus and optimal control Economic dynamics allows for changes in economic variables over time, including in dynamic systems. The problem of finding optimal functions for such changes is studied in variational calculus and in optimal control theory. Before the Second World War, Frank Ramsey and Harold Hotelling used the calculus of variations to that end. Following Richard Bellman's work on dynamic programming and the 1962 English translation of L. Pontryagin et al.'s earlier work,[70] optimal control theory was used more extensively in economics in addressing dynamic problems, especially as to economic growth equilibrium and stability of economic systems,[71] of which a textbook example is optimal consumption and saving.[72] A crucial distinction is between deterministic and stochastic control models.[73] Other applications of optimal control theory include those in finance, inventories, and production for example.[74] Functional analysis It was in the course of proving of the existence of an optimal equilibrium in his 1937 model of economic growth that John von Neumann introduced functional analytic methods to include topology in economic theory, in particular, fixed-point theory through his generalization of Brouwer's fixed-point theorem.[8][43][75] Following von Neumann's program, Kenneth Arrow and Gérard Debreu formulated abstract models of economic equilibria using convex sets and fixed–point theory. In introducing the Arrow–Debreu model in 1954, they proved the existence (but not the uniqueness) of an equilibrium and also proved that every Walras equilibrium is Pareto efficient; in general, equilibria need not be unique.[76] In their models, the ("primal") vector space represented quantitites while the "dual" vector space represented prices.[77] In Russia, the mathematician Leonid Kantorovich developed economic models in partially ordered vector spaces, that emphasized the duality between quantities and prices.[78] Oppressed by communism, Kantorovich renamed prices as "objectively determined valuations" which were abbreviated in Russian as "o. o. o.", alluding to the difficulty of discussing prices in the Soviet Union.[77][79][80] Mathematical economics 16 Even in finite dimensions, the concepts of functional analysis have illuminated economic theory, particularly in clarifying the role of prices as normal vectors to a hyperplane supporting a convex set, representing production or consumption possibilities. However, problems of describing optimization over time or under uncertainty require the use of infinite–dimensional function spaces, because agents are choosing among functions or stochastic processes.[77][81][82][83] Differential decline and rise John von Neumann's work on functional analysis and topology in broke new ground in mathematics and economic theory.[43][84] It also left advanced mathematical economics with fewer applications of differential calculus. In particular, general equilibrium theorists used general topology, convex geometry, and optimization theory more than differential calculus, because the approach of differential calculus had failed to establish the existence of an equilibrium. However, the decline of differential calculus should not be exaggerated, because differential calculus has always been used in graduate training and in applications. Moreover, differential calculus has returned to the highest levels of mathematical economics, general equilibrium theory (GET), as practiced by the "GET-set" (the humorous designation due to Jacques H. Drèze). In the 1960s and 1970s, however, Gérard Debreu and Stephen Smale led a revival of the use of differential calculus in mathematical economics. In particular, they were able to prove the existence of a general equilibrium, where earlier writers had failed, because of their novel mathematics: Baire category from general topology and Sard's lemma from differential topology. Other economists asssociated with the use of differential analysis include Egbert Dierker, Andreu Mas-Colell, and Yves Balasko.[85][86] These advances have changed the traditional narrative of the history of mathematical economics, following von Neumann, which celebrated the abandonment of differential calculus. Game theory John von Neumann, working with Oskar Morgenstern on the theory of games, broke new mathematical ground in 1944 by extending functional analytic methods related to convex sets and topological fixed-point theory to economic analysis.[8][84] Their work thereby avoided the traditional differential calculus, for which the maximum–operator did not apply to non-differentiable functions. Continuing von Neumann's work in cooperative game theory, game theorists Lloyd S. Shapley, Martin Shubik, Hervé Moulin, Nimrod Megiddo, Bezalel Peleg influenced economic research in politics and economics. For example, research on the fair prices in cooperative games and fair values for voting games led to changed rules for voting in legislatures and for accounting for the costs in public–works projects. For example, cooperative game theory was used in designing the water distribution system of Southern Sweden and for setting rates for dedicated telephone lines in the USA. Earlier neoclassical theory had bounded only the range of bargaining outcomes and in special cases, for example bilateral monopoly or along the contract curve of the Edgeworth box.[87] Von Neumann and Morgenstern's results were similarly weak. Following von Neumann's program, however, John Nash used fixed–point theory to prove conditions under which the bargaining problem and noncooperative games can generate a unique equilibrium solution.[88] Noncooperative game theory has been adopted as a fundamental aspect of experimental economics,[89] behavioral economics,[90] information economics,[91] industrial organization,[92] and political economy.[93] It has also given rise to the subject of mechanism design (sometimes called reverse game theory), which has private and public-policy applications as to ways of improving economic efficiency through incentives for information sharing.[94] In 1994, Nash, John Harsanyi, and Reinhard Selten received the Nobel Memorial Prize in Economic Sciences their work on non–cooperative games. Harsanyi and Selten were awarded for their work on repeated games. Later work extended their results to computational methods of modeling.[95] Mathematical economics 17 Agent-based computational economics Agent-based computational economics (ACE) as a named field is relatively recent, dating from about the 1990s as to published work. It studies economic processes, including whole economies, as dynamic systems of interacting agents over time. As such, it falls in the paradigm of complex adaptive systems.[96] In corresponding agent-based models, agents are not real people but "computational objects modeled as interacting according to rules" ... "whose micro-level interactions create emergent patterns" in space and time.[97] The rules are formulated to predict behavior and social interactions based on incentives and information. The theoretical assumption of mathematical optimization by agents markets is replaced by the less restrictive postulate of agents with bounded rationality adapting to market forces.[98] ACE models apply numerical methods of analysis to computer-based simulations of complex dynamic problems for which more conventional methods, such as theorem formulation, may not find ready use.[99] Starting from specified initial conditions, the computational economic system is modeled as evolving over time as its constituent agents repeatedly interact with each other. In these respects, ACE has been characterized as a bottom-up culture-dish approach to the study of the economy.[100] In contrast to other standard modeling methods, ACE events are driven solely by initial conditions, whether or not equilibria exist or are computationally tractable. ACE modeling, however, includes agent adaptation, autonomy, and learning.[101] It has a similarity to, and overlap with, game theory as an agent-based method for modeling social interactions.[95] Other dimensions of the approach include such standard economic subjects as competition and collaboration,[102] market structure and industrial organization,[103] transaction costs,[104] welfare economics[105] and mechanism design,[106] information and uncertainty,[107] and macroeconomics.[108][109] The method is said to benefit from continuing improvements in modeling techniques of computer science and increased computer capabilities. Issues include those common to experimental economics in general[110] and by comparison[111] and to development of a common framework for empirical validation and resolving open questions in agent-based modeling.[112] The ultimate scientific objective of the method has been described as "test[ing] theoretical findings against real-world data in ways that permit empirically supported theories to cumulate over time, with each researcher's work building appropriately on the work that has gone before."[113] Mathematical economics 18 Mathematicization of economics Over the course of the 20th century, articles in "core journals"[115] in economics have been almost exclusively written by economists in academia. As a result, much of the material transmitted in those journals relates to economic theory, and "economic theory itself has been continuously more abstract and [116] mathematical." A subjective assessment of mathematical techniques[117] employed in these core journals showed a decrease in articles that use neither geometric representations nor mathematical notation from 95% in 1892 to 5.3% in 1990.[118] A 2007 survey of ten of the top economic journals finds that only 5.8% of the articles published in 2003 and 2004 both lacked statistical analysis of data and lacked displayed The surface of the Volatility smile is a 3-D surface whereby the current mathematical expressions that were indexed with market implied volatility (Z-axis) for all options on the underlier is plotted numbers at the margin of the page.[119] against strike price and time to maturity (X & Y-axes). [114] Econometrics Between the world wars, advances in mathematical statistics and a cadre of mathematically trained economists led to econometrics, which was the name proposed for the discipline of advancing economics by using mathematics and statistics. Within economics, "econometrics" has often been used for statistical methods in economics, rather than mathematical economics. Statistical econometrics features the application of linear regression and time series analysis to economic data. Ragnar Frisch coined the word "econometrics" and helped to found both the Econometric Society in 1930 and the journal Econometrica in 1933.[120][121] A student of Frisch's, Trygve Haavelmo published The Probability Approach in Econometrics in 1944, where he asserted that precise statistical analysis could be used as a tool to validate mathematical theories about economic actors with data from complex sources.[122] This linking of statistical analysis of systems to economic theory was also promulgated by the Cowles Commission (now the Cowles Foundation) throughout the 1930s and 1940s.[123] Earlier work in econometrics The roots of modern econometrics can be traced to the American economist Henry L. Moore. Moore studied agricultural productivity and attempted to fit changing values of productivity for plots of corn and other crops to a curve using different values of elasticity. Moore made several errors in his work, some from his choice of models and some from limitations in his use of mathematics. The accuracy of Moore's models also was limited by the poor data for national accounts in the United States at the time. While his first models of production were static, in 1925 he published a dynamic "moving equilibrium" model designed to explain business cycles—this periodic variation from overcorrection in supply and demand curves is now known as the cobweb model. A more formal derivation of this model was made later by Nicholas Kaldor, who is largely credited for its exposition.[124] Mathematical economics 19 Application Much of classical economics can be presented in simple geometric terms or elementary mathematical notation. Mathematical economics, however, conventionally makes use of calculus and matrix algebra in economic analysis in order to make powerful claims that would be more difficult without such mathematical tools. These tools are prerequisites for formal study, not only in mathematical economics but in contemporary economic theory in general. Economic problems often involve so many variables that mathematics is the only practical way of attacking and solving them. Alfred Marshall argued that every economic problem which can be quantified, analytically expressed and solved, should be treated by means of mathematical work.[126] Economics has become increasingly The IS/LM model is a Keynesian macroeconomic model designed to make predictions about the intersection of "real" economic activity (e.g. spending, dependent upon mathematical methods and income, savings rates) and decisions made in the financial markets (Money supply the mathematical tools it employs have and Liquidity preference). The model is no longer widely taught at the graduate [125] become more sophisticated. As a result, level but is common in undergraduate macroeconomics courses. mathematics has become considerably more important to professionals in economics and finance. Graduate programs in both economics and finance require strong undergraduate preparation in mathematics for admission and, for this reason, attract an increasingly high number of mathematicians. Applied mathematicians apply mathematical principles to practical problems, such as economic analysis and other economics-related issues, and many economic problems are often defined as integrated into the scope of applied mathematics.[17] This integration results from the formulation of economic problems as stylized models with clear assumptions and falsifiable predictions. This modeling may be informal or prosaic, as it was in Adam Smith's The Wealth of Nations, or it may be formal, rigorous and mathematical. Broadly speaking, formal economic models may be classified as stochastic or deterministic and as discrete or continuous. At a practical level, quantitative modeling is applied to many areas of economics and several methodologies have evolved more or less independently of each other.[127] • Stochastic models are formulated using stochastic processes. They model economically observable values over time. Most of econometrics is based on statistics to formulate and test hypotheses about these processes or estimate parameters for them. Between the World Wars, Herman Wold developed a representation of stationary stochastic processes in terms of autoregressive models and a determinist trend. Wold and Jan Tinbergen applied time-series analysis to economic data. Contemporary research on time series statistics consider additional formulations of stationary processes, such as autoregressive moving average models. More general models include autoregressive conditional heteroskedasticity (ARCH) models and generalized ARCH (GARCH) models. • Non-stochastic mathematical models may be purely qualitative (for example, models involved in some aspect of social choice theory) or quantitative (involving rationalization of financial variables, for example with hyperbolic coordinates, and/or specific forms of functional relationships between variables). In some cases economic Mathematical economics 20 predictions of a model merely assert the direction of movement of economic variables, and so the functional relationships are used only in a qualitative sense: for example, if the price of an item increases, then the demand for that item will decrease. For such models, economists often use two-dimensional graphs instead of functions. • Qualitative models are occasionally used. One example is qualitative scenario planning in which possible future events are played out. Another example is non-numerical decision tree analysis. Qualitative models often suffer from lack of precision. Criticisms and defences Adequacy of mathematics for qualitative and complicated economics Friedrich Hayek contended that the use of formal techniques projects a scientific exactness that does not appropriately account for informational limitations faced by real economic agents. [128] In an interview, the economic historian Robert Heilbroner stated:[129] I guess the scientific approach began to penetrate and soon dominate the profession in the past twenty to thirty years. This came about in part because of the "invention" of mathematical analysis of various kinds and, indeed, considerable improvements in it. This is the age in which we have not only more data but more sophisticated use of data. So there is a strong feeling that this is a data-laden science and a data-laden undertaking, which, by virtue of the sheer numerics, the sheer equations, and the sheer look of a journal page, bears a certain resemblance to science . . . That one central activity looks scientific. I understand that. I think that is genuine. It approaches being a universal law. But resembling a science is different from being a science. Heilbroner stated that "some/much of economics is not naturally quantitative and therefore does not lend itself to mathematical exposition."[130] Testing predictions of mathematical economics Philosopher Karl Popper discussed the scientific standing of economics in the 1940s and 1950s. He argued that mathematical economics suffered from being tautological. In other words, insofar that economics became a mathematical theory, mathematical economics ceased to rely on empirical refutation but rather relied on mathematical proofs and disproof.[131] According to Popper, falsifiable assumptions can be tested by experiment and observation while unfalsifiable assumptions can be explored mathematically for their consequences and for their consistency with other assumptions.[132] Sharing Popper's concerns about assumptions in economics generally, and not just mathematical economics, Milton Friedman declared that "all assumptions are unrealistic". Friedman proposed judging economic models by their predictive performance rather than by the match between their assumptions and reality.[133] Mathematical economics as a form of pure mathematics Considering mathematical economics, J.M. Keynes wrote in The General Theory:[134] It is a great fault of symbolic pseudo-mathematical methods of formalising a system of economic analysis ... that they expressly assume strict independence between the factors involved and lose their cogency and authority if this hypothesis is disallowed; whereas, in ordinary discourse, where we are not blindly manipulating and know all the time what we are doing and what the words mean, we can keep ‘at the back of our heads’ the necessary reserves and qualifications and the adjustments which we shall have to make later on, in a way in which we cannot keep complicated partial differentials ‘at the back’ of several pages of algebra which assume they all vanish. Too large a proportion of recent ‘mathematical’ economics are merely concoctions, as imprecise as the initial assumptions they rest on, which allow the author to lose sight of the complexities and interdependencies of the real world in a maze of pretentious and unhelpful symbols. Mathematical economics 21 Defense of mathematical economics In response to these criticisms, Paul Samuelson argued that mathematics is a language, repeating a thesis of Josiah Willard Gibbs. In economics, the language of mathematics is sometimes necessary for representing substantive problems. Moreover, mathematical economics has led to conceptual advances in economics.[135] In particular, Samuelson gave the example of microeconomics, writing that "few people are ingenious enough to grasp [its] more complex parts... without resorting to the language of mathematics, while most ordinary individuals can do so fairly easily with the aid of mathematics."[136] Some economists state that mathematical economics deserves support just like other forms of mathematics, particularly its neighbors in mathematical optimization and mathematical statistics and increasingly in theoretical computer science. Mathematical economics and other mathematical sciences have a history in which theoretical advances have regularly contributed to the reform of the more applied branches of economics. In particular, following the program of John von Neumann, game theory now provides the foundations for describing much of applied economics, from statistical decision theory (as "games against nature") and econometrics to general equilibrium theory and industrial organization. In the last decade, with the rise of the internet, mathematical economicists and optimization experts and computer scientists have worked on problems of pricing for on-line services --- their contributions using mathematics from cooperative game theory, nondifferentiable optimization, and combinatorial games. Robert M. Solow concluded that mathematical economics was the core "infrastructure" of contemporary economics: Economics is no longer a fit conversation piece for ladies and gentlemen. It has become a technical subject. Like any technical subject it attracts some people who are more interested in the technique than the subject. That is too bad, but it may be inevitable. In any case, do not kid yourself: the technical core of economics is indispensable infrastructure for the political economy. That is why, if you consult [a reference in contemporary economics] looking for enlightenment about the world today, you will be led to technical economics, or history, or nothing at all.[137] Mathematical economists Prominent mathematical economists include, but are not limited to, the following (by century of birth). 19th century • Enrico Barone • Francis Ysidro Edgeworth • Irving Fisher • William Stanley Jevons • Antoine Augustin Cournot 20th century Mathematical economics 22 • Charalambos D. Aliprantis • Nicholas Georgescu-Roegen • Andreu Mas-Colell • Leonard J. Savage • R. G. D. Allen • Roger Guesnerie • Eric Maskin • Herbert Scarf • Maurice Allais • Frank Hahn • Nimrod Megiddo • Reinhard Selten • Kenneth J. Arrow • John C. Harsanyi • James Mirrlees • Amartya Sen • Robert J. Aumann • John R. Hicks • Roger Myerson • Lloyd S. Shapley • Yves Balasko • Werner Hildenbrand • John Forbes Nash, Jr. • Stephen Smale • David Blackwell • Harold Hotelling • John von Neumann • Robert Solow • Lawrence E. Blume • Leonid Hurwicz • Edward C. Prescott • Hugo F. Sonnenschein • Graciela Chichilnisky • Leonid Kantorovich • Roy Radner • Albert W. Tucker • George B. Dantzig • Tjalling Koopmans • Frank Ramsey • Hirofumi Uzawa • Gérard Debreu • David M. Kreps • Donald John Roberts • Robert B. Wilson • Jacques H. Drèze • Harold W. Kuhn • Paul Samuelson • Hermann Wold • David Gale • Edmond Malinvaud • Thomas Sargent • Nicholas C. Yannelis Notes [1] Debreu, Gérard ([1987] 2008). "mathematical economics", section II, The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000107& edition=current& q=Mathematical economics& topicid=& result_number=1) Republished with revisions from 1986, "Theoretic Models: Mathematical Form and Economic Content", Econometrica, 54(6), pp. 1259 (http:/ / www. jstor. org/ pss/ 1914299)-1270. [2] Elaborated at the JEL classification codes, Mathematical and quantitative methods JEL: C Subcategories. [3] Chiang, Alpha C.; and Kevin Wainwright (2005). Fundamental Methods of Mathematical Economics. McGraw-Hill Irwin. pp. 3–4. ISBN 0-07-010910-9. TOC. (http:/ / www. mhprofessional. com/ product. php?isbn=0070109109) [4] Varian, Hal (1997). "What Use Is Economic Theory?" in A. D'Autume and J. Cartelier, ed., Is Economics Becoming a Hard Science?, Edward Elgar. Pre-publication PDF. (http:/ / www. sims. berkeley. edu/ ~hal/ Papers/ theory. pdf) Retrieved 2008-04-01. [5] • As in Handbook of Mathematical Economics, 1st-page chapter links: Arrow, Kenneth J., and Michael D. Intriligator, ed., (1981), v. 1 (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL& _tockey=#TOC#24615#1981#999989999#565707#FLP#& _cdi=24615& _pubType=HS& _auth=y& _acct=C000050221& _version=1& _urlVersion=0& _userid=10& md5=01881ea3fe7d7990fed1c5b78d9f7be6) _____ (1982). v. 2 (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL& _tockey=#TOC#24615#1982#999979999#565708#FLP#& _cdi=24615& _pubType=HS& _auth=y& _acct=C000050221& _version=1& _urlVersion=0& _userid=10& md5=80dbd8f22c229a3640dc02e59ff80fe4) _____ (1986). v. 3 (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL& _tockey=#TOC#24615#1986#999969999#565709#FLP#& _cdi=24615& _pubType=HS& _auth=y& _acct=C000050221& _version=1& _urlVersion=0& _userid=10& md5=56f86ec2f0a1d2881e15bd1d0a45accd) Hildenbrand, Werner, and Hugo Sonnenschein, ed. (1991). v. 4. (http:/ / www. sciencedirect. com/ science/ handbooks/ 15734382) • Debreu, Gérard (1983). Mathematical Economics: Twenty Papers of Gérard Debreu, Contents (http:/ / books. google. com/ books?id=wKJp6DepYncC& pg=PR7& source=bl& ots=mdhM3H6nCb& sig=zu38qa9R-3AiWqypwplcMibgdPo& hl=en& ei=etaJTu_XG6Lb0QG1nvXTAQ& sa=X& oi=book_result& ct=result& resnum=7& ved=0CF4Q6AEwBg#v=onepage& q& f=false). • Glaister, Stephen (1984). Mathematical Methods for Economists, 3rd ed., Blackwell. Contents. (http:/ / books. google. com/ books?id=Ct2nrJSHxsQC& printsec=find& pg=PR5=onepage& q& f=false#v=onepage& q& f=false) • Takayama, Akira (1985). Mathematical Economics, 2nd ed. Cambridge. Description (http:/ / books. google. com/ books/ about/ Mathematical_economics. html?id=685iPEaLAEcC) and Contents (http:/ / books. google. com/ books?id=685iPEaLAEcC& printsec=find& pg=PR9=onepage& q& f=false#v=onepage& q& f=false). • Michael Carter (2001). Foundations of Mathematical Economics, MIT Press. Description (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?ttype=2& tid=8630) and Contents (http:/ / books. google. com/ books?id=KysvrGGfzq0C& printsec=find& pg=PR7=onepage& q& f=false). [6] Chiang, Alpha C. (1992). Elements of Dynamic Optimization, Waveland. TOC (http:/ / www. waveland. com/ Titles/ Chiang. htm) & Amazon.com link (http:/ / www. amazon. com/ Elements-Dynamic-Optimization-Alpha-Chiang/ dp/ 157766096X) to inside, first pp. [7] Samuelson, Paul ((1947) [1983]). Foundations of Economic Analysis. Harvard University Press. ISBN 0-674-31301-1. [8] • Debreu, Gérard ([1987] 2008). "mathematical economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000107& edition=current& q=Mathematical economics& topicid=& result_number=1) Republished with revisions from 1986, "Theoretic Models: Mathematical Form and Economic Content", Econometrica, 54(6), pp. 1259 (http:/ / www. jstor. org/ pss/ 1914299)-1270. • von Neumann, John, and Oskar Morgenstern (1944). Theory of Games and Economic Behavior. Princeton University Press. Mathematical economics 23 [9] Schumpeter, J.A. (1954). Elizabeth B. Schumpeter. ed. History of Economic Analysis (http:/ / books. google. com/ ?id=xjWiAAAACAAJ). New York: Oxford University Press. pp. 209–212. ISBN 978-0-04-330086-2. OCLC 13498913. . [10] Schumpeter (1954) p. 212-215 [11] Schnieder, Erich (1934). "Johann Heinrich von Thünen". Econometrica (The Econometric Society) 2 (1): 1–12. doi:10.2307/1907947. ISSN 0012-9682. JSTOR 1907947. OCLC 35705710. [12] Schumpeter (1954) p. 465-468 [13] Philip Mirowski, 1991. "The When, the How and the Why of Mathematical Expression in the History of Economics Analysis", Journal of Economic Perspectives, 5(1) pp. 145-157. (http:/ / www. fcs. edu. uy/ multi/ phes/ mirowski_math_econ_JEP. pdf) [14] Weintraub, E. Roy (2008). "mathematics and economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000372& edition=current& q=& topicid=& result_number=1). [15] Jevons, W.S. (1866). "Brief Account of a General Mathematical Theory of Political Economy", Journal of the Royal Statistical Society, XXIX (June) pp. 282-87. Read in Section F of the British Association, 1862. PDF. (http:/ / www. adelinotorres. com/ economia/ STANLEY JEVONS_Teoria Matemática e Economia Politica. pdf) [16] Jevons, W. Stanley (1871). The Principles of Political Economy, pp. 4, 25. (http:/ / books. google. com/ books?id=Sw8ZAAAAYAAJ& printsec=frontcover& dq="The+ Theory+ of+ Political+ Economy,"+ jevons+ 1871#v=onepage& q="The Theory of Political Economy," jevons 1871& f=false). . [17] Sheila C., Dow (1999-05-21). "The Use of Mathematics in Economics" (http:/ / www. ioe. ac. uk/ esrcmaths/ sheila1. html). ESRC Public Understanding of Mathematics Seminar. Birmingham: Economic and Social Research Council. . Retrieved 2008-07-06. [18] While the concept of cardinality has fallen out of favor in neoclassical economics, the differences between cardinal utility and ordinal utility are minor for most applications. [19] Nicola, PierCarlo (2000). Mainstream Mathermatical Economics in the 20th Century (http:/ / books. google. com/ ?id=KR0Rbi8o4QQC). Springer. pp. 4. ISBN 978-3-540-67084-1. . Retrieved 2008-08-21. [20] Augustin Cournot (1838, tr. 1897) Researches nto the Mathematical Principles of Wealth. Links to description (http:/ / books. google. com/ books?id=AGoGpyJY_SAC) and chapters. (http:/ / books. google. com/ books?id=AGoGpyJY_SAC& printsec=titlepage#PPR9,M1) [21] Hotelling, Harold (1990). "Stability in Competition" (http:/ / books. google. com/ ?id=dYbbHQAACAAJ). In Darnell, Adrian C.. The Collected Economics Articles of Harold Hotelling. Springer. pp. 51, 52. ISBN 3-540-97011-8. OCLC 20217006. . Retrieved 2008-08-21. [22] "Antoine Augustin Cournot, 1801-1877" (http:/ / cepa. newschool. edu/ het/ profiles/ cournot. htm). The History of Economic Thought Website. The New School for Social Research. . Retrieved 2008-08-21. [23] Gibbons, Robert (1992). Game Theory for Applied Economists (http:/ / books. google. com/ ?id=_6qgHgAACAAJ). Princeton, New Jersey: Princeton University Press. pp. 14, 15. ISBN 0-691-00395-5. . [24] Nicola, p. 9-12 [25] Edgeworth, Francis Ysidro (September 5, 1889). "The Mathematical Theory of Political Economy: Review of Léon Walras, Éléments d'économie politique pure" (http:/ / cepa. newschool. edu/ het/ texts/ edgeworth/ edgewalras89. pdf) (PDF). Nature 40 (1036): 434–436. doi:10.1038/040434a0. ISSN 0028-0836. . Retrieved 2008-08-21. [26] Nicholson, Walter; Snyder, Christopher, p. 350-353. [27] Dixon, Robert. "Walras Law and Macroeconomics" (http:/ / web. archive. org/ web/ 20080417102559/ http:/ / www. economics. unimelb. edu. au/ rdixon/ wlaw. html). Walras Law Guide. Department of Economics, University of Melbourne. Archived from the original (http:/ / www. economics. unimelb. edu. au/ rdixon/ wlaw. html) on April 17, 2008. . Retrieved 2008-09-28. [28] Dixon, Robert. "A Formal Proof of Walras Law" (http:/ / web. archive. org/ web/ 20080430033548/ http:/ / www. economics. unimelb. edu. au/ rdixon/ walproof. html). Walras Law Guide. Department of Economics, University of Melbourne. Archived from the original (http:/ / www. economics. unimelb. edu. au/ rdixon/ walproof. html) on April 30, 2008. . Retrieved 2008-09-28. [29] Rima, Ingrid H. (1977). "Neoclassicism and Dissent 1890-1930" (http:/ / books. google. com/ ?id=s7cJAAAAIAAJ& printsec=find& pg=PR5=onepage& q#v=onepage& q& f=false). In Weintraub, Sidney. Modern Economic Thought. University of Pennsylvania Press. pp. 10, 11. ISBN 0-8122-7712-0. . [30] Heilbroner, Robert L. (1953 [1999]). The Worldly Philosophers (http:/ / books. google. com/ ?id=N_3cj4urgJcC) (Seventh ed.). New York: Simon and Schuster. pp. 172–175, 313. ISBN 978-0-684-86214-9. . [31] Edgeworth, Francis Ysidro (1881 [1961]). Mathematical Psychics (http:/ / books. google. com/ ?id=Q4WCGAAACAAJ). London: Kegan Paul [A. M. Kelley]. pp. 15–19. . [32] Nicola, p. 14, 15, 258-261 [33] Bowley, Arthur Lyon (1924 [1960]). The Mathematical Groundwork of Economics: an Introductory Treatise (http:/ / books. google. com/ ?id=_cgkAAAAMAAJ). Oxford: Clarendon Press [Kelly]. . [34] Gillies, D. B. (1969). "Solutions to general non-zero-sum games" (http:/ / books. google. com/ ?id=9lSVFzsTGWsC). In Tucker, A. W. & Luce, R. D.. Contributions to the Theory of Games. Annals of Mathematics. 40. Princeton, New Jersey: Princeton University Press. pp. 47–85. ISBN 978-0-691-07937-0. . [35] Moss, Lawrence S. (2003). "The Seligman-Edgeworth Debate about the Analysis of Tax Incidence: The Advent of Mathematical Economics, 1892–1910". History of Political Economy (Duke University Press) 35 (2): 207, 212, 219, 234–237. doi:10.1215/00182702-35-2-205. ISSN 0018-2702. [36] Hotelling, Harold (1990). "Note on Edgeworth's Taxation Phenomenon and Professor Garver's Additional Condition on Demand Functions" (http:/ / books. google. com/ ?id=dYbbHQAACAAJ). In Darnell, Adrian C.. The Collected Economics Articles of Harold Hotelling. Springer. Mathematical economics 24 pp. 94–122. ISBN 3-540-97011-8. OCLC 20217006. . Retrieved 2008-08-26. [37] Herstein, I.N. (October 1953). "Some Mathematical Methods and Techniques in Economics". Quarterly of Applied Mathematics (American Mathematical Society) 11 (3): 249, 252, 260. ISSN 1552-4485. [Pp. 249-62 (http:/ / cowles. econ. yale. edu/ P/ cp/ p00b/ p0078. pdf). [38] • Weintraub, E. Roy (2008). "mathematics and economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000372& edition=current& q=& topicid=& result_number=1). • _____ (2002). How Economics Became a Mathematical Science. Duke University Press. Description (http:/ / www. dukeupress. edu/ Catalog/ ViewProduct. php?productid=6807& viewby=series& categoryid=47& sort=newest) and preview (http:/ / books. google. com/ books?hl=en& lr=& id=ntc_Fs36AoQC& oi=fnd& pg=PR7& dq=& f=false#v=onepage& q& f=false). [39] Nicholson, Walter; Snyder, Christopher (2007). "General Equilibrium and Welfare". Intermediate Microeconomics and Its Applications (10th ed.). Thompson. pp. 364, 365. ISBN 0-324-31968-1. [40] Jolink, Albert (2006). "What Went Wrong with Walras?". In Backhaus, Juergen G.; Maks, J.A. Hans. From Walras to Pareto. The European Heritage in Economics and the Social Sciences. IV. Springer. doi:10.1007/978-0-387-33757-9_6. ISBN 978-0-387-33756-2. • Blaug, Mark (2007). "The Fundamental Theorems of Modern Welfare Economics, Historically Contemplated". History of Political Economy (Duke University Press) 39 (2): 186–188. doi:10.1215/00182702-2007-001. ISSN 0018-2702. [41] Blaug (2007), p. 185, 187 [42] Metzler, Lloyd (1948). "Review of Foundations of Economic Analysis". American Economic Review (The American Economic Review, Vol. 38, No. 5) 38 (5): 905–910. ISSN 0002-8282. JSTOR 1811704. [43] Neumann, J. von (1937). "Über ein ökonomisches Gleichungssystem und ein Verallgemeinerung des Brouwerschen Fixpunktsatzes", Ergebnisse eines Mathematischen Kolloquiums, 8, pp. 73-83, translated and published in 1945-46, as "A Model of General Equilibrium", Review of Economic Studies, 13, pp. 1–9. [44] For this problem to have a unique solution, it suffices that the nonnegative matrices A and B satisfy an irreducibility condition, generalizing that of the Perron–Frobenius theorem of nonnegative matrices, which considers the (simplified) eigenvalue problem A - λ I q = 0, where the nonnegative matrix A must be square and where the diagonal matrix I is the identity matrix. Von Neumann's irreducibility condition was called the "whales and wranglers" hypothesis by David Champernowne, who provided a verbal and economic commentary on the English translation of von Neumann's article. Von Neumann's hypothesis implied that every economic process used a positive amount of every economic good. Weaker "irreducibility" conditions were given by David Gale and by John Kemeny, Oskar Morgenstern, and Gerald L. Thompson in the 1950s and then by Stephen M. Robinson in the 1970s. [45] David Gale. The theory of linear economic models. McGraw-Hill, New York, 1960. [46] Morgenstern, Oskar; Thompson, Gerald L. (1976). Mathematical theory of expanding and contracting economies. Lexington Books. Lexington, Massachusetts: D. C. Heath and Company. pp. xviii+277. [47] Alexander Schrijver, Theory of Linear and Integer Programming. John Wiley & sons, 1998, ISBN 0-471-98232-6. [48] •Rockafellar, R. Tyrrell (1967). Monotone processes of convex and concave type. Memoirs of the American Mathematical Society. Providence, R.I.: American Mathematical Society. pp. i+74. • Rockafellar, R. T. (1974). "Convex algebra and duality in dynamic models of production". In Josef Loz and Maria Loz. Mathematical models in economics (Proc. Sympos. and Conf. von Neumann Models, Warsaw, 1972). Amsterdam: North-Holland and Polish Adademy of Sciences (PAN). pp. 351–378. •Rockafellar, R. T. (1970 (Reprint 1997 as a Princeton classic in mathematics)). Convex analysis. Princeton, New Jersey: Princeton University Press. [49] Kenneth Arrow, Paul Samuelson, John Harsanyi, Sidney Afriat, Gerald L. Thompson, and Nicholas Kaldor. (1989). Mohammed Dore, Sukhamoy Chakravarty, Richard Goodwin. ed. John Von Neumann and modern economics. Oxford:Clarendon. pp. 261. [50] Chapter 9.1 "The von Neumann growth model" (pages 277–299): Yinyu Ye. Interior point algorithms: Theory and analysis. Wiley. 1997. [51] Screpanti, Ernesto; Zamagni, Stefano (1993). An Outline of the History of Economic Thought. New York: Oxford University Press. pp. 288–290. ISBN 0-19-828370-9. OCLC 57281275. [52] David Gale. The theory of linear economic models. McGraw-Hill, New York, 1960. [53] Morgenstern, Oskar; Thompson, Gerald L. (1976). Mathematical theory of expanding and contracting economies. Lexington Books. Lexington, Massachusetts: D. C. Heath and Company. pp. xviii+277. [54] " The Nature of Mathematical Programming (http:/ / glossary. computing. society. informs. org/ index. php?page=nature. html)", Mathematical Programming Glossary, INFORMS Computing Society. [55] Schmedders, Karl (2008). "numerical optimization methods in economics", The New Palgrave Dictionary of Economics, 2nd Edition, v. 6, pp. 138-57. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_N000148& edition=current& q=optimization& topicid=& result_number=1) [56] Robbins, Lionel (1935, 2nd ed.). An Essay on the Nature and Significance of Economic Science, Macmillan, p. 16. [57] Blume, Lawrence E. (2008). "duality", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_D000196) [58] Dixit, A. K. ([1976] 1990). Optimization in Economic Theory, 2nd ed., Oxford. Description (http:/ / books. google. com/ books?id=dHrsHz0VocUC& pg=find& pg=PA194=false#v=onepage& q& f=false) and contents preview (http:/ / books. google. com/ books?id=dHrsHz0VocUC& pg=PR7& lpg=PR6& dq=false& lr=#v=onepage& q=false& f=false). Mathematical economics 25 [59] • Samuelson, Paul A., 1998. "How Foundations Came to Be", Journal of Economic Literature, 36(3), pp. 1375 (http:/ / www. jstor. org/ pss/ 2564803)–1386. • _____ (1970). "Maximum Principles in Analytical Economics" (http:/ / www. nobelprize. org/ nobel_prizes/ economics/ laureates/ 1970/ samuelson-lecture. pdf), Nobel Prize lecture. [60] • Allan M. Feldman (3008). "welfare economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_W000050& edition=current& q=fundamental theorems & topicid=& result_number=3). • Mas-Colell, Andreu, Michael D. Whinston, and Jerry R. Green (1995), Microeconomic Theory, Chapter 16. Oxford University Press, ISBN 0-19-510268-1. Description (http:/ / www. oup. com/ us/ catalog/ general/ subject/ Economics/ MicroeconomicTheory/ ?view=usa& ci=9780195073409) and contents (http:/ / www. oup. com/ us/ catalog/ general/ subject/ Economics/ MicroeconomicTheory/ ?view=usa& sf=toc& ci=9780195073409). [61] • Geanakoplos, John ([1987] 2008). "Arrow–Debreu model of general equilibrium", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_A000133& edition=current& q=optimization& topicid=& result_number=20). • Arrow, Kenneth J., and Gérard Debreu (1954). "Existence of an Equilibrium for a Competitive Economy", Econometrica 22(3), pp. 265 (http:/ / www. jstor. org/ pss/ 1907353)-290. [62] • Scarf, Herbert E. (2008). "computation of general equilibria", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000573& q=computational economics& topicid=& result_number=3) • Kubler, Felix (2008). "computation of general equilibria (new developments)", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000564& q=computational economics& topicid=& result_number=2) [63] Nicola, p. 133 [64] Dorfman, Robert, Paul A. Samuelson, and Robert M. Solow (1958). Linear Programming and Economic Analysis. McGraw–Hill. Chapter-preview links. (http:/ / books. google. com/ books?hl=en& lr=& id=k5_vzaCNQP4C& oi=fnd& pg=PP11& dq=& ots=ticEpVKqDz& sig=FEIiGNmox4KDifmeKEfir7pZZNw#v=onepage& q& f=false) [65] M. Padberg, Linear Optimization and Extensions, Second Edition, Springer-Verlag, 1999. [66] Dantzig, George B. ([1987] 2008). "linear programming", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_L000106& edition=current& q=Linear programming& topicid=& result_number=1). [67] • Intriligator, Michael D. (2008). "nonlinear programming", The New Palgrave Dictionary of Economics, 2nd Edition. TOC (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_N000083& edition=current& q=non-linear programming& topicid=& result_number=3). • Blume, Lawrence E. (2008). "convex programming", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000348& edition=current& q=optimization& topicid=& result_number=4). • Kuhn, H. W.; Tucker, A. W. (1951). "Nonlinear programming". Proceedings of 2nd Berkeley Symposium. Berkeley: University of California Press. pp. 481–492. [68] • Bertsekas, Dimitri P. (1999). Nonlinear Programming (Second ed.). Cambridge, Massachusetts.: Athena Scientific. ISBN 1-886529-00-0. • Vapnyarskii, I.B. (2001), "Lagrange multipliers" (http:/ / www. encyclopediaofmath. org/ index. php?title=L/ l057190), in Hazewinkel, Michiel, Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4, . • Lasdon, Leon S. (1970). Optimization theory for large systems. Macmillan series in operations research. New York: The Macmillan Company. pp. xi+523. MR337317. • Lasdon, Leon S. (2002). Optimization theory for large systems (reprint of the 1970 Macmillan ed.). Mineola, New York: Dover Publications, Inc.. pp. xiii+523. MR1888251. • Hiriart-Urruty, Jean-Baptiste; Lemaréchal, Claude (1993). "XII Abstract duality for practitioners". Convex analysis and minimization algorithms, Volume II: Advanced theory and bundle methods. Grundlehren der Mathematischen Wissenschaften [Fundamental Principles of Mathematical Sciences]. 306. Berlin: Springer-Verlag. pp. 136–193 (and Bibliographical comments on pp. 334–335). ISBN 3-540-56852-2. [69] Lemaréchal, Claude (2001). "Lagrangian relaxation". In Michael Jünger and Denis Naddef. Computational combinatorial optimization: Papers from the Spring School held in Schloß Dagstuhl, May 15–19, 2000. Lecture Notes in Computer Science. 2241. Berlin: Springer-Verlag. pp. 112–156. doi:10.1007/3-540-45586-8_4. ISBN 3-540-42877-1. MR1900016.doi:10.1007/3-540-45586-8_4. [70] Pontryagin, L. S.; Boltyanski, V. G., Gamkrelidze, R. V., Mischenko, E. F. (1962). The Mathematical Theory of Optimal Processes (http:/ / books. google. com/ books?id=kwzq0F4cBVAC& printsec=frontcover#v=onepage& q& f=false). New York: Wiley. ISBN 68981. [71] • Zelikin, M. I. ([1987] 2008). "Pontryagin's principle of optimality", The New Palgrave Dictionary of Economics, 2nd Edition. Preview link (http:/ / www. dictionaryofeconomics. com/ search_results?q="Pontryagin's+ principle+ of+ optimality"& edition=current& button_search=GO). • Martos, Béla (1987). "control and coordination of economic activity", The New Palgrave: A Dictionary of Economics. Description link (http:/ / www. dictionaryofeconomics. com/ search_results?q=control+ coordination+ Martos+ & field=content& edition=all& topicid=). • Brock, W. A. (1987). "optimal control and economic dynamics", The New Palgrave: A Dictionary of Economics. Outline (http:/ / www. dictionaryofeconomics. com/ article?id=pde1987_X001613). • Shell, K., ed. (1967). Essays on the Theory of Optimal Economic Growth. Cambridge, Massachusetts: The MIT Press. ISBN 0-262-19036-2.] [72] Stokey, Nancy L. and Robert E. Lucas with Edward Prescott (1989). Recursive Methods in Economic Dynamics, Harvard University Press, chapter 5. Desecription (http:/ / www. hup. harvard. edu/ catalog. php?isbn=9780674750968) and chapter-preview links (http:/ / books. Mathematical economics 26 google. com/ books?hl=en& lr=& id=tWYo0QolyLAC& oi=fnd& pg=PR11& dq=#v=onepage& q=f=false& f=false). [73] Malliaris, A.G. (2008). "stochastic optimal control", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_S000269& edition=& field=keyword& q=Taylor's th& topicid=& result_number=1). [74] • Arrow, K. J.; Kurz, M. (1970). Public Investment, the Rate of Return, and Optimal Fiscal Policy. Baltimore, Maryland: The Johns Hopkins Press. ISBN 0-8018-1124-4. Abstract. (http:/ / md1. csa. com/ partners/ viewrecord. php?requester=gs& collection=ENV& recid=7107596& q=& uid=788819967& setcookie=yes) • Sethi, S. P.; Thompson, G. L. (2000). Optimal Control Theory: Applications to Management Science and Economics, Second Edition. New York: Springer. ISBN 0-7923-8608-6. Scroll to chapter-preview links. (http:/ / books. google. com/ books?id=gLMmaLYRy4QC& printsec=frontcover& source=gbs_atb#v=onepage& q& f=false) [75] Andrew McLennan, 2008. "fixed point theorems", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_F000135& edition=current& q=). [76] Weintraub, E. Roy (1977). "General Equilibrium Theory" (http:/ / books. google. com/ ?id=JDqAAAAAIAAJ). In Weintraub, Sidney. Modern Economic Thought. University of Pennsylvania Press. pp. 107–109. ISBN 0-8122-7712-0. . • Arrow, Kenneth J.; Debreu, Gérard (1954). "Existence of an equilibrium for a competitive economy". Econometrica (The Econometric Society) 22 (3): 265–290. doi:10.2307/1907353. ISSN 0012-9682. JSTOR 1907353. [77] Kantorovich, Leonid, and Victor Polterovich (2008). "Functional analysis", in S. Durlauf and L. Blume, ed., The New Palgrave Dictionary of Economics, 2nd Edition. Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_F000236), ed., Palgrave Macmillan. [78] Kantorovich, L. V (1990). ""My journey in science (supposed report to the Moscow Mathematical Society)" [expanding Russian Math. Surveys 42 (1987), no. 2, pp. 233–270]". In Lev J. Leifman. Functional analysis, optimization, and mathematical economics: A collection of papers dedicated to the memory of Leonid Vitalʹevich Kantorovich. New York: The Clarendon Press, Oxford University Press. pp. 8–45. ISBN 0-19-505729-5. MR898626. [79] Page 406: Polyak, B. T. (2002). "History of mathematical programming in the USSR: Analyzing the phenomenon (Chapter 3 The pioneer: L. V. Kantorovich, 1912–1986, pp. 405–407)". Mathematical Programming 91 (ISMP 2000, Part 1 (Atlanta, GA)): pp. 401–416. doi:10.1007/s101070100258. MR1888984. [80] "Leonid Vitaliyevich Kantorovich — Prize Lecture ("Mathematics in economics: Achievements, difficulties, perspectives")" (http:/ / nobelprize. org/ nobel_prizes/ economics/ laureates/ 1975/ kantorovich-lecture. html). Nobelprize.org. . Retrieved 12 Dec 2010. [81] Aliprantis, Charalambos D.; Brown, Donald J.; Burkinshaw, Owen (1990). Existence and optimality of competitive equilibria. Berlin: Springer–Verlag. pp. xii+284. ISBN 3-540-52866-0. MR1075992. [82] Rockafellar, R. Tyrrell. Conjugate duality and optimization. Lectures given at the Johns Hopkins University, Baltimore, Maryland, June, 1973. Conference Board of the Mathematical Sciences Regional Conference Series in Applied Mathematics, No. 16. Society for Industrial and Applied Mathematics, Philadelphia, Pa., 1974. vi+74 pp. [83] Lester G. Telser and Robert L. Graves Functional Analysis in Mathematical Economics: Optimization Over Infinite Horizons 1972. University of Chicago Press, 1972, ISBN 978-0-226-79190-6. [84] Neumann, John von, and Oskar Morgenstern (1944) Theory of Games and Economic Behavior, Princeton. [85] Mas-Colell, Andreu (1985). The Theory of general economic equilibrium: A differentiable approach. Econometric Society monographs. Cambridge UP. ISBN 0-521-26514-2. MR1113262. [86] Yves Balasko. Foundations of the Theory of General Equilibrium, 1988, ISBN 0-12-076975-1. [87] Creedy, John (2008). "Francis Ysidro (1845–1926)", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_E000041& edition=current& q=edgeworth& topicid=& result_number=3). [88] • Nash, John F., Jr. (1950). "The Bargaining Problem", Econometrica, 18(2), pp. 155-162 (http:/ / www. stern. nyu. edu/ networks/ Nash_The_Bargaining_Problem. pdf). • Serrano, Roberto (2008). "bargaining", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_B000073& edition=current& q=bargaining& topicid=& result_number=1). [89] • Smith,Vernon L. (1992). "Game Theory and Experimental Economics: Beginnings and Early Influences", in E. R. Weintraub, ed., Towards a History of Game Theory, pp. 241- (http:/ / books. google. com/ books?hl=en& lr=& id=9CHY2Gozh1MC& oi=fnd& pg=PA241& ots=onepage& q& f=false#v=onepage& q& f=false*) 282. • _____ (2001). "Experimental Economics", International Encyclopedia of the Social & Behavioral Sciences, pp. 5100-5108. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ B0080430767022324) per sect. 1.1 & 2.1. • Plott, Charles R., and Vernon L. Smith, ed. (2008). Handbook of Experimental Economics Results, v. 1, Elsevier, Part 4, Games, ch. 45-66 preview links (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL& _hubEid=1-s2. 0-S1574072207X00015& _cid=277334& _pubType=HS& _auth=y& _acct=C000228598& _version=1& _urlVersion=0& _userid=10& md5=49f8b6d5e3024eac39ed5fad351fe568). • Shubik, Martin (2002). "Game Theory and Experimental Gaming", in R. Aumann and S. Hart, ed., Handbook of Game Theory with Economic Applications, Elsevier, v. 3, pp. 2327-2351. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574000502030254). [90] From The New Palgrave Dictionary of Economics (2008), 2nd Edition: • Gul, Faruk. "behavioural economics and game theory." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_G000210& q=Behavioral economics & topicid=& result_number=2) • Camerer, Colin F. "behavioral game theory." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_B000302& q=Behavioral economics & topicid=& result_number=13) Mathematical economics 27 [91] • Rasmusen, Eric (2007). Games and Information, 4th ed. Description (http:/ / www. wiley. com/ WileyCDA/ WileyTitle/ productCd-EHEP001009. html) and chapter-preview links. (http:/ / books. google. com/ books?id=5XEMuJwnBmUC& printsec=fnd& pg=PR5& dq=gbs_atb#v=onepage& q& f=false) • Aumann, R., and S. Hart, ed. (1992, 2002). Handbook of Game Theory with Economic Applications v. 1, links at ch. 3-6 (http:/ / www. sciencedirect. com/ science/ handbooks/ 15740005/ 1) and v. 3, ch. 43 (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574000502030060). [92] • Tirole, Jean (1988). The Theory of Industrial Organization, MIT Press. Description (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?ttype=2& tid=8224) and chapter-preview links, pp. vii-ix (http:/ / books. google. com/ books?id=HIjsF0XONF8C& printsec=find& pg=PR7=onepage& q& f=false#v=onepage& q& f=false), "General Organization", pp. 5-6 (http:/ / books. google. com/ books?id=HIjsF0XONF8C& dq=find& pg=PA5& source=bn& hl=en& ei=e2vES-H-O8T68Abxp_GyDw& sa=X& oi=book_result& ct=result& resnum=4& ved=0CB8Q6AEwAw#v=onepage& q& f=false), and "Non-Cooperative Game Theory: A User's Guide Manual,' " ch. 11, pp. 423-59 (http:/ / books. google. com/ books?id=HIjsF0XONF8C& dq=find& pg=PA423& source=bn& hl=en& ei=e2vES-H-O8T68Abxp_GyDw& sa=X& oi=book_result& ct=result& resnum=4& ved=0CB8Q6AEwAw#v=onepage& q& f=false). • Bagwell, Kyle, and Asher Wolinsky (2002). "Game theory and Industrial Organization", ch. 49, Handbook of Game Theory with Economic Applications, v. 3, pp. 1851 (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574000502030126)-1895. [93] • Shubik, Martin (1981). "Game Theory Models and Methods in Political Economy", in Handbook of Mathematical Economics,, v. 1, pp. 285 (http:/ / www. sciencedirect. com/ science?_ob=ArticleURL& _udi=B7P5Y-4FDF0FN-C& _user=10& _coverDate=01/ 01/ 1981& _rdoc=11& _fmt=high& _orig=browse& _origin=browse& _zone=rslt_list_item& _srch=doc-info(#toc#24615#1981#999989999#565707#FLP#display#Volume)& _cdi=24615& _sort=d& _docanchor=& _ct=14& _acct=C000050221& _version=1& _urlVersion=0& _userid=10& md5=cb34198ec88c9ab8fa59af6d5634e9cf& searchtype=a)-330. [94] • The New Palgrave Dictionary of Economics (2008), 2nd Edition: Myerson, Roger B. "mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000132& edition=current& q=mechanism design& topicid=& result_number=3) _____. "revelation principle." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_R000137& edition=current& q=moral& topicid=& result_number=1) Sandholm, Tuomas. "computing in mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000563& edition=& field=keyword& q=algorithmic mechanism design& topicid=& result_number=1) • Nisan, Noam, and Amir Ronen (2001). "Algorithmic Mechanism Design", Games and Economic Behavior, 35(1-2), pp. 166–196 (http:/ / www. cs. cmu. edu/ ~sandholm/ cs15-892F09/ Algorithmic mechanism design. pdf). • Nisan, Noam, et al., ed. (2007). Algorithmic Game Theory, Cambridge University Press. Description (http:/ / www. cup. cam. ac. uk/ asia/ catalogue/ catalogue. asp?isbn=9780521872829). [95] • Halpern, Joseph Y. (2008). "computer science and game theory", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000566& edition=current& q=& topicid=& result_number=1). • Shoham, Yoav (2008). "Computer Science and Game Theory", Communications of the ACM, 51(8), pp. 75-79 (http:/ / www. robotics. stanford. edu/ ~shoham/ www papers/ CSGT-CACM-Shoham. pdf). • Roth,Alvin E. (2002). "The Economist as Engineer: Game Theory, Experimentation, and Computation as Tools for Design Economics", Econometrica, 70(4), pp. 1341–1378 (http:/ / kuznets. fas. harvard. edu/ ~aroth/ papers/ engineer. pdf). [96] • Kirman, Alan (2008). "economy as a complex system", The New Palgrave Dictionary of Economics , 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_E000246& edition=current& q=Computational economics& topicid=& result_number=20). • Tesfatsion, Leigh (2003). "Agent-based Computational Economics: Modeling Economies as Complex Adaptive Systems", Information Sciences, 149(4), pp. 262-268 (http:/ / copper. math. buffalo. edu/ urgewiki/ uploads/ Literature/ Tesfatsion2002. pdf). [97] Scott E. Page (2008), "agent-based models", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_A000218& edition=current& q=agent-based computational modeling& topicid=& result_number=1). [98] • Holland, John H., and John H. Miller (1991). "Artificial Adaptive Agents in Economic Theory", American Economic Review, 81(2), [pp. 365-370 (http:/ / www. santafe. edu/ media/ workingpapers/ 91-05-025. pdf) p. 366. • Arthur, W. Brian, 1994. "Inductive Reasoning and Bounded Rationality", American Economic Review, 84(2), pp. 406-411 (http:/ / www-personal. umich. edu/ ~samoore/ bit885f2011/ arthur-inductive. pdf). • Schelling, Thomas C. (1978 [2006]). Micromotives and Macrobehavior, Norton. Description (http:/ / books. wwnorton. com/ books/ 978-0-393-32946-9/ ), preview (http:/ / books. google. com/ books?id=DenWKRgqzWMC& printsec=find& pg=PA1=#v=onepage& q& f=false). • Sargent, Thomas J. (1994). Bounded Rationality in Macroeconomics, Oxford. Description (http:/ / www. oup. com/ us/ catalog/ general/ subject/ Economics/ MacroeconomicTheory/ ?view=usa& ci=9780198288695) and chapter-preview 1st-page links (http:/ / www. questia. com/ library/ book/ bounded-rationality-in-macroeconomics-thomas-j-sargent-by-thomas-j-sargent. jsp). [99] • Judd, Kenneth L. (2006). "Computationally Intensive Analyses in Economics", Handbook of Computational Economics, v. 2, ch. 17, Introduction, p. 883. [Pp. 881- (http:/ / books. google. com/ books?hl=en& lr=& id=6ITfRkNmKQcC& oi=fnd& pg=PA881& ots=2j0cCBB5S6& sig=a1DlAKMWcxFQZwSkGVVp2zlHIb8#v=onepage& q& f=false) 893. Pre-pub PDF (http:/ / www2. econ. iastate. edu/ tesfatsi/ Judd. finalrev. pdf). • _____ (1998). Numerical Methods in Economics, MIT Press. Links to description (http:/ / mitpress. mit. edu/ catalog/ item/ default. Mathematical economics 28 asp?ttype=2& tid=3257) and chapter previews (http:/ / books. google. com/ books?id=9Wxk_z9HskAC& pg=PR7& source=gbs_toc_r& cad=3#v=onepage& q& f=false). [100] • Tesfatsion, Leigh (2002). "Agent-Based Computational Economics: Growing Economies from the Bottom Up", Artificial Life, 8(1), pp.55-82. Abstract (http:/ / www. mitpressjournals. org/ doi/ abs/ 10. 1162/ 106454602753694765) and pre-pub PDF (http:/ / www. econ. brown. edu/ fac/ Peter_Howitt/ SummerSchool/ Agent. pdf). • _____ (1997). "How Economists Can Get Alife", in W. B. Arthur, S. Durlauf, and D. Lane, eds., The Economy as an Evolving Complex System, II, pp. 533-564. Addison-Wesley. Pre-pub PDF (http:/ / ageconsearch. umn. edu/ bitstream/ 18196/ 1/ er37. pdf). [101] Tesfatsion, Leigh (2006), "Agent-Based Computational Economics: A Constructive Approach to Economic Theory", ch. 16, Handbook of Computational Economics, v. 2, part 2, ACE study of economic system. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020162) and pre-pub PDF (http:/ / econ2. econ. iastate. edu/ tesfatsi/ hbintlt. pdf). [102] Axelrod, Robert (1997). The Complexity of Cooperation: Agent-Based Models of Competition and Collaboration, Princeton. Description (http:/ / press. princeton. edu/ titles/ 6144. html), contents (http:/ / press. princeton. edu/ titles/ 6144. html#TOC), and preview (http:/ / books. google. com/ books?id=J0dgRGMdjmQC& printsec=find& pg=PR11#v=onepage& q& f=false). [103] • Leombruni, Roberto, and Matteo Richiardi, ed. (2004), Industry and Labor Dynamics: The Agent-Based Computational Economics Approach. World Scientific Publishing ISBN 981-256-100-5. Description (http:/ / www. worldscibooks. com/ economics/ 5706. html) and chapter-preview links (http:/ / books. google. com/ books?id=P5O7A5D55nQC& printsec=fond& pg=PR5#v=onepage& q& f=false). • Epstein, Joshua M. (2006). "Growing Adaptive Organizations: An Agent-Based Computational Approach", in Generative Social Science: Studies in Agent-Based Computational Modeling, pp. 309 - (http:/ / books. google. com/ books?hl=en& lr=& id=543OS3qdxBYC& oi=fnd& pg=PA326& dq=false#v=onepage& q=false& f=false) 344. Description (http:/ / press. princeton. edu/ titles/ 8277. html) and abstract (http:/ / www. santafe. edu/ research/ working-papers/ abstract/ 99895b6465e8b87656612f8e3570b34c/ ). [104] Klosa, Tomas B., and Bart Nooteboom, 2001. "Agent-based Computational Transaction Cost Economics", Journal of Economic Dynamics and Control 25(3–4), pp. 503–52. Abstract. (http:/ / www. sciencedirect. com/ science/ article/ pii/ S0165188900000348) [105] Axtell, Robert (2005). "The Complexity of Exchange", Economic Journal, 115(504, Features), pp. F193-F210 (http:/ / econfaculty. gmu. edu/ pboettke/ workshop/ archives/ f05/ Axtell. pdf). [106] • The New Palgrave Dictionary of Economics (2008), 2nd Edition: Myerson, Roger B. "mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_M000132& edition=current& q=mechanism design& topicid=& result_number=3) _____. "revelation principle." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_R000137& edition=current& q=moral& topicid=& result_number=1) Sandholm, Tuomas. "computing in mechanism design." Abstract. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000563& edition=& field=keyword& q=algorithmic mechanism design& topicid=& result_number=1) • Nisan, Noam, and Amir Ronen (2001). "Algorithmic Mechanism Design", Games and Economic Behavior, 35(1-2), pp. 166–196 (http:/ / www. cs. cmu. edu/ ~sandholm/ cs15-892F09/ Algorithmic mechanism design. pdf). • Nisan, Noam, et al., ed. (2007). Algorithmic Game Theory, Cambridge University Press. Description (http:/ / www. cup. cam. ac. uk/ asia/ catalogue/ catalogue. asp?isbn=9780521872829). [107] Sandholm, Tuomas W., and Victor R. Lesser (2001)."Leveled Commitment Contracts and Strategic Breach", Games and Economic Behavior, 35(1-2), pp. 212-270 (http:/ / www. cs. cmu. edu/ afs/ . cs. cmu. edu/ Web/ People/ sandholm/ leveled. geb. pdf). [108] • Colander, David, Peter Howitt, Alan Kirman, Axel Leijonhufvud, and Perry Mehrling (2008). "Beyond DSGE Models: Toward an Empirically Based Macroeconomics", American Economic Review, 98(2), pp. 236 (http:/ / www. jstor. org/ pss/ 29730026)-240. Pre-pub PDF (http:/ / www. econ. brown. edu/ fac/ peter_howitt/ publication/ complex macro6. pdf). • Sargent, Thomas J. (1994). Bounded Rationality in Macroeconomics, Oxford. Description (http:/ / www. oup. com/ us/ catalog/ general/ subject/ Economics/ MacroeconomicTheory/ ?view=usa& ci=9780198288695) and chapter-preview 1st-page links (http:/ / www. questia. com/ library/ book/ bounded-rationality-in-macroeconomics-thomas-j-sargent-by-thomas-j-sargent. jsp). [109] Tesfatsion, Leigh (2006), "Agent-Based Computational Economics: A Constructive Approach to Economic Theory", ch. 16, Handbook of Computational Economics, v. 2, pp. 832-865. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020162) and pre-pub PDF (http:/ / econ2. econ. iastate. edu/ tesfatsi/ hbintlt. pdf). [110] Smith, Vernon L. (2008). "experimental economics", The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_E000277& q=experimental & topicid=& result_number=2). [111] Duffy, John (2006). "Agent-Based Models and Human Subject Experiments", ch. 19, Handbook of Computational Economics, v.2, pp. 949–101. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020198). [112] • Namatame, Akira, and Takao Terano (2002). "The Hare and the Tortoise: Cumulative Progress in Agent-based Simulation", in Agent-based Approaches in Economic and Social Complex Systems. pp. 3- (http:/ / books. google. com/ books?id=nTkwxVUYvfIC& printsec=find& pg=PA3=onepage& q& f=false#v=onepage& q& f=false) 14, IOS Press. Description (http:/ / www. iospress. nl/ book/ agent-based-approaches-in-economic-and-social-complex-systems-2/ ). • Fagiolo, Giorgio, Alessio Moneta, and Paul Windrum (2007). "A Critical Guide to Empirical Validation of Agent-Based Models in Economics: Methodologies, Procedures, and Open Problems", Computational Economics, 30, pp. 195 (http:/ / www. springerlink. com/ content/ t683473172528275/ )–226. [113] • Tesfatsion, Leigh (2006). "Agent-Based Computational Economics: A Constructive Approach to Economic Theory", ch. 16, Handbook of Computational Economics, v. 2, [pp. 831-880] sect. 5. Abstract (http:/ / www. sciencedirect. com/ science/ article/ pii/ S1574002105020162) Mathematical economics 29 and pre-pub PDF (http:/ / econ2. econ. iastate. edu/ tesfatsi/ hbintlt. pdf). • Judd, Kenneth L. (2006). "Computationally Intensive Analyses in Economics", Handbook of Computational Economics, v. 2, ch. 17, pp. 881- (http:/ / books. google. com/ books?hl=en& lr=& id=6ITfRkNmKQcC& oi=fnd& pg=PA881& ots=2j0cCBB5S6& sig=a1DlAKMWcxFQZwSkGVVp2zlHIb8#v=onepage& q& f=false) 893. Pre-pub PDF (http:/ / www2. econ. iastate. edu/ tesfatsi/ Judd. finalrev. pdf). • Tesfatsion, Leigh, and Kenneth L. Judd, ed. (2006). Handbook of Computational Economics, v. 2. Description (http:/ / www. elsevier. com/ wps/ find/ bookdescription. cws_home/ 660847/ description#description) & and chapter-preview links. (http:/ / www. sciencedirect. com/ science?_ob=PublicationURL& _hubEid=1-s2. 0-S1574002105X02003& _cid=273377& _pubType=HS& _auth=y& _acct=C000228598& _version=1& _urlVersion=0& _userid=10& md5=e4757b4f65755ed6340a11fee9615200) [114] Brockhaus, Oliver; Farkas, Michael; Ferraris, Andrew; Long, Douglas; Overhaus, Marcus (2000). Equity Derivatives and Market Risk Models (http:/ / books. google. com/ ?id=JGZPAAAAMAAJ). Risk Books. pp. 13–17. ISBN 978-1-899332-87-8. . Retrieved 2008-08-17. [115] Liner, Gaines H. (2002). "Core Journals in Economics". Economic Inquiry (Oxford University Press) 40 (1): 140. doi:10.1093/ei/40.1.138. [116] Stigler, George J.; Stigler, Steven J.; Friedland, Claire (April 1995). "The Journals of Economics". The Journal of Political Economy (The University of Chicago Press) 103 (2): 339. doi:10.1086/261986. ISSN 0022-3808. JSTOR 2138643. [117] Stigler et al. reviewed journal articles in core economic journals (as defined by the authors but meaning generally non-specialist journals) throughout the 20th century. Journal articles which at any point used geometric representation or mathematical notation were noted as using that level of mathematics as its "highest level of mathematical technique". The authors refer to "verbal techniques" as those which conveyed the subject of the piece without notation from geometry, algebra or calculus. [118] Stigler et al., p. 342 [119] Sutter, Daniel and Rex Pjesky. "Where Would Adam Smith Publish Today?: The Near Absence of Math-free Research in Top Journals" (May 2007). (http:/ / econjwatch. org/ issues/ volume-4-number-1-may-2007) [120] Arrow, Kenneth J. (April 1960). "The Work of Ragnar Frisch, Econometrician". Econometrica (Blackwell Publishing) 28 (2): 175–192. doi:10.2307/1907716. ISSN 0012-9682. JSTOR 1907716. [121] Bjerkholt, Olav (July 1995). "Ragnar Frisch, Editor of Econometrica 1933-1954". Econometrica (Blackwell Publishing) 63 (4): 755–765. doi:10.2307/2171799. ISSN 0012-9682. JSTOR 1906940. [122] Lange, Oskar (1945). "The Scope and Method of Economics". Review of Economic Studies (The Review of Economic Studies Ltd.) 13 (1): 19–32. doi:10.2307/2296113. ISSN 0034-6527. JSTOR 2296113. [123] Aldrich, John (January 1989). "Autonomy". Oxford Economic Papers (Oxford University Press) 41 (1, History and Methodology of Econometrics): 15–34. ISSN 0030-7653. JSTOR 2663180. [124] Epstein, Roy J. (1987). A History of Econometrics. Contributions to Economic Analysis. North-Holland. pp. 13–19. ISBN 978-0-444-70267-8. OCLC 230844893. [125] Colander, David C. (2004). "The Strange Persistence of the IS-LM Model". History of Political Economy (Duke University Press) 36 (Annual Supplement): 305–322. doi:10.1215/00182702-36-Suppl_1-305. ISSN 0018-2702. [126] Brems, Hans (Oct., 1975). "Marshall on Mathematics". Journal of Law and Economics (University of Chicago Press) 18 (2): 583–585. doi:10.1086/466825. ISSN 0022-2186. JSTOR 725308. [127] Frigg, R.; Hartman, S. (February 27, 2006). Edward N. Zalta. ed. Models in Science (http:/ / plato. stanford. edu/ entries/ models-science/ #OntWhaMod). Stanford Encyclopedia of Philosophy. Stanford, California: The Metaphysics Research Lab. ISSN 1095-5054. . Retrieved 2008-08-16. [128] Hayek, Friedrich (September 1945). "The Use of Knowledge in Society". American Economic Review 35 (4): 519–530. JSTOR 1809376. [129] Heilbroner, Robert (May–June 1999). "The end of the Dismal Science?" (http:/ / findarticles. com/ p/ articles/ mi_m1093/ is_3_42/ ai_54682627/ print). Challenge Magazine. . [130] Beed & Owen, 584 [131] Boland, L. A. (2007). "Seven Decades of Economic Methodology" (http:/ / books. google. com/ ?id=w-BEoTj0axoC). In I. C. Jarvie, K. Milford, D.W. Miller. Karl Popper:A Centenary Assessment. London: Ashgate Publishing. pp. 219. ISBN 978-0-7546-5375-2. . Retrieved 2008-06-10. [132] Beed, Clive; Kane, Owen (1991). "What Is the Critique of the Mathematization of Economics?". Kyklos 44 (4): 581–612. doi:10.1111/j.1467-6435.1991.tb01798.x. [133] Friedman, Milton (1953). Essays in Positive Economics (http:/ / books. google. com/ ?id=rSGekjfpf4cC). Chicago: University of Chicago Press. pp. 30, 33, 41. ISBN 978-0-226-26403-5. . [134] Keynes, John Maynard (1936). The General Theory of Employment, Interest and Money (http:/ / www. marxists. org/ reference/ subject/ economics/ keynes/ general-theory/ ch21. htm). Cambridge: Macmillan. pp. 297. ISBN 0-333-10729-2. . [135] Paul A. Samuelson (1952). "Economic Theory and Mathematics — An Appraisal", American Economic Review, 42(2), pp. 56, 64-65 (http:/ / cowles. econ. yale. edu/ P/ cp/ p00b/ p0061. pdf) (press +). [136] D.W. Bushaw and R.W. Clower (1957). Introduction to Mathematical Economics, p. vii. (http:/ / babel. hathitrust. org/ cgi/ pt?id=uc1. b3513586;page=root;view=image;size=100;seq=11;num=vii) [137] Solow, Robert M. (20 March 1988). "The Wide, Wide World Of Wealth (The New Palgrave: A Dictionary of Economics'. Edited by John Eatwell, Murray Milgate and Peter Newman. Four volumes. 4,103 pp. New York: Stockton Press. $650)" (http:/ / www. nytimes. com/ 1988/ 03/ 20/ books/ the-wide-wide-world-of-wealth. html?scp=1). New York Times. . Mathematical economics 30 External links • Journal of Mathematical Economics Aims & Scope (http://www.elsevier.com/wps/find/journaldescription. cws_home/505577/description#description) • Mathematical Economics and Financial Mathematics (http://www.dmoz.org/Science/Math/Applications/ Mathematical_Economics_and_Financial_Mathematics//) at the Open Directory Project Statistics Statistics is the study of the collection, organization, analysis, interpretation, and presentation of data.[1][2] It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments.[1] A statistician is someone who is particularly well-versed in the ways of thinking necessary to successful apply statistical analysis. Such people often gain experience through working in any of a wide number of fields. A discipline called mathematical statistics studies statistics mathematically. The word statistics, when referring to the scientific discipline, is singular, as in "Statistics is an art."[3] This should not be confused with the word statistic, referring to a quantity (such as mean or median) calculated from a set of data,[4] whose plural is statistics ("this statistic seems wrong" or "these statistics are misleading"). Scope Some consider statistics a mathematical body of science that pertains to the collection, analysis, interpretation or explanation, and presentation of data,[5] while others consider it a branch of mathematics[6] concerned with collecting and interpreting data. Because of its empirical roots and its focus on applications, statistics is usually considered a distinct mathematical science rather than a branch of mathematics.[7][8] Much of statistics is non-mathematical: ensuring that data More probability density is found the closer one gets to the expected (mean) value in a collection is undertaken in a way that normal distribution. Statistics used in standardized testing assessment are shown. The produces valid conclusions; coding and scales include standard deviations, cumulative percentages, percentile equivalents, archiving data so that information is Z-scores, T-scores, standard nines, and percentages in standard nines. retained and made useful for international comparisons of official statistics; reporting of results and summarised data (tables and graphs) in ways comprehensible to those must use them; implementing procedures that ensure the privacy of census information. Statisticians improve data quality by developing specific experiment designs and survey samples. Statistics itself also provides tools for prediction and forecasting the use of data and statistical models. Statistics is applicable to a wide variety of academic disciplines, including natural and social sciences, government, and business. Statistical consultants can help organizations and companies that don't have in-house expertise relevant to their particular questions. Statistics 31 Statistical methods can summarize or describe a collection of data. This is called descriptive statistics. This is particularly useful in communicating the results of experiments and research. In addition, data patterns may be modeled in a way that accounts for randomness and uncertainty in the observations. These models can be used to draw inferences about the process or population under study—a practice called inferential statistics. Inference is a vital element of scientific advance, since it provides a way to draw conclusions from data that are subject to random variation. To prove the propositions being investigated further, the conclusions are tested as well, as part of the scientific method. Descriptive statistics and analysis of the new data tend to provide more information as to the truth of the proposition. "Applied statistics" comprises descriptive statistics and the application of inferential statistics.[9] Theoretical statistics concerns both the logical arguments underlying justification of approaches to statistical inference, as well encompassing mathematical statistics. Mathematical statistics includes not only the manipulation of probability distributions necessary for deriving results related to methods of estimation and inference, but also various aspects of computational statistics and the design of experiments. Statistics is closely related to probability theory, with which it is often grouped. The difference is, roughly, that probability theory starts from the given parameters of a total population to deduce probabilities that pertain to samples. Statistical inference, however, moves in the opposite direction—inductively inferring from samples to the parameters of a larger or total population. History Statistical methods date back at least to the 5th century BC. The earliest known writing on statistics appears in a 9th century book entitled Manuscript on Deciphering Cryptographic Messages, written by Al-Kindi. In this book, Al-Kindi provides a detailed description of how to use statistics and frequency analysis to decipher encrypted messages. This was the birth of both statistics and cryptanalysis, according to the Saudi engineer Ibrahim Al-Kadi.[10][11] The Nuova Cronica, a 14th century history of Florence by the Florentine banker and official Giovanni Villani, includes much statistical information on population, ordinances, commerce, education, and religious facilities, and has been described as the first introduction of statistics as a positive element in history.[12] Some scholars pinpoint the origin of statistics to 1663, with the publication of Natural and Political Observations upon the Bills of Mortality by John Graunt.[13] Early applications of statistical thinking revolved around the needs of states to base policy on demographic and economic data, hence its stat- etymology. The scope of the discipline of statistics broadened in the early 19th century to include the collection and analysis of data in general. Today, statistics is widely employed in government, business, and natural and social sciences. Its mathematical foundations were laid in the 17th century with the development of the probability theory by Blaise Pascal and Pierre de Fermat. Probability theory arose from the study of games of chance. The method of least squares was first described by Carl Friedrich Gauss around 1794. The use of modern computers has expedited large-scale statistical computation, and has also made possible new methods that are impractical to perform manually. Overview In applying statistics to a scientific, industrial, or societal problem, it is necessary to begin with a population or process to be studied. Populations can be diverse topics such as "all persons living in a country" or "every atom composing a crystal". A population can also be composed of observations of a process at various times, with the data from each observation serving as a different member of the overall group. Data collected about this kind of "population" constitutes what is called a time series. Statistics 32 For practical reasons, a chosen subset of the population called a sample is studied—as opposed to compiling data about the entire group (an operation called census). Once a sample that is representative of the population is determined, data is collected for the sample members in an observational or experimental setting. This data can then be subjected to statistical analysis, serving two related purposes: description and inference. • Descriptive statistics summarize the population data by describing what was observed in the sample numerically or graphically. Numerical descriptors include mean and standard deviation for continuous data types (like heights or weights), while frequency and percentage are more useful in terms of describing categorical data (like race). • Inferential statistics uses patterns in the sample data to draw inferences about the population represented, accounting for randomness. These inferences may take the form of: answering yes/no questions about the data (hypothesis testing), estimating numerical characteristics of the data (estimation), describing associations within the data (correlation) and modeling relationships within the data (for example, using regression analysis). Inference can extend to forecasting, prediction and estimation of unobserved values either in or associated with the population being studied; it can include extrapolation and interpolation of time series or spatial data, and can also include data mining.[14] "... it is only the manipulation of uncertainty that interests us. We are not concerned with the matter that is uncertain. Thus we do not study the mechanism of rain; only whether it will rain." [15] Dennis Lindley, 2000 The concept of correlation is particularly noteworthy for the potential confusion it can cause. Statistical analysis of a data set often reveals that two variables (properties) of the population under consideration tend to vary together, as if they were connected. For example, a study of annual income that also looks at age of death might find that poor people tend to have shorter lives than affluent people. The two variables are said to be correlated; however, they may or may not be the cause of one another. The correlation phenomena could be caused by a third, previously unconsidered phenomenon, called a lurking variable or confounding variable. For this reason, there is no way to immediately infer the existence of a causal relationship between the two variables. (See Correlation does not imply causation.) To use a sample as a guide to an entire population, it is important that it truly representat that overall population. Representative sampling assures that inferences and conclusions can safely extend from the sample to the population as a whole. A major problem lies in determining the extent that the sample chosen is actually representative. Statistics offers methods to estimate and correct for any random trending within the sample and data collection procedures. There are also methods of experimental design for experiments that can lessen these issues at the outset of a study, strengthening its capability to discern truths about the population. Randomness is studied using the mathematical discipline of probability theory. Probability is used in "mathematical statistics" (alternatively, "statistical theory") to study the sampling distributions of sample statistics and, more generally, the properties of statistical procedures. The use of any statistical method is valid when the system or population under consideration satisfies the assumptions of the method. Misuse of statistics can produce subtle, but serious errors in description and interpretation—subtle in the sense that even experienced professionals make such errors, and serious in the sense that they can lead to devastating decision errors. For instance, social policy, medical practice, and the reliability of structures like bridges all rely on the proper use of statistics. See below for further discussion. Even when statistical techniques are correctly applied, the results can be difficult to interpret for those lacking expertise. The statistical significance of a trend in the data—which measures the extent to which a trend could be caused by random variation in the sample—may or may not agree with an intuitive sense of its significance. The set of basic statistical skills (and skepticism) that people need to deal with information in their everyday lives properly is referred to as statistical literacy. Statistics 33 Statistical methods Experimental and observational studies A common goal for a statistical research project is to investigate causality, and in particular to draw a conclusion on the effect of changes in the values of predictors or independent variables on dependent variables or response. There are two major types of causal statistical studies: experimental studies and observational studies. In both types of studies, the effect of differences of an independent variable (or variables) on the behavior of the dependent variable are observed. The difference between the two types lies in how the study is actually conducted. Each can be very effective. An experimental study involves taking measurements of the system under study, manipulating the system, and then taking additional measurements using the same procedure to determine if the manipulation has modified the values of the measurements. In contrast, an observational study does not involve experimental manipulation. Instead, data are gathered and correlations between predictors and response are investigated. Experiments The basic steps of a statistical experiment are: 1. Planning the research, including finding the number of replicates of the study, using the following information: preliminary estimates regarding the size of treatment effects, alternative hypotheses, and the estimated experimental variability. Consideration of the selection of experimental subjects and the ethics of research is necessary. Statisticians recommend that experiments compare (at least) one new treatment with a standard treatment or control, to allow an unbiased estimate of the difference in treatment effects. 2. Design of experiments, using blocking to reduce the influence of confounding variables, and randomized assignment of treatments to subjects to allow unbiased estimates of treatment effects and experimental error. At this stage, the experimenters and statisticians write the experimental protocol that shall guide the performance of the experiment and that specifies the primary analysis of the experimental data. 3. Performing the experiment following the experimental protocol and analyzing the data following the experimental protocol. 4. Further examining the data set in secondary analyses, to suggest new hypotheses for future study. 5. Documenting and presenting the results of the study. Experiments on human behavior have special concerns. The famous Hawthorne study examined changes to the working environment at the Hawthorne plant of the Western Electric Company. The researchers were interested in determining whether increased illumination would increase the productivity of the assembly line workers. The researchers first measured the productivity in the plant, then modified the illumination in an area of the plant and checked if the changes in illumination affected productivity. It turned out that productivity indeed improved (under the experimental conditions). However, the study is heavily criticized today for errors in experimental procedures, specifically for the lack of a control group and blindness. The Hawthorne effect refers to finding that an outcome (in this case, worker productivity) changed due to observation itself. Those in the Hawthorne study became more productive not because the lighting was changed but because they were being observed. Statistics 34 Observational study An example of an observational study is one that explores the correlation between smoking and lung cancer. This type of study typically uses a survey to collect observations about the area of interest and then performs statistical analysis. In this case, the researchers would collect observations of both smokers and non-smokers, perhaps through a case-control study, and then look for the number of cases of lung cancer in each group. Levels of measurement There are four main levels of measurement used in statistics: nominal, ordinal, interval, and ratio.[16] Each of these have different degrees of usefulness in statistical research. Ratio measurements have both a meaningful zero value and the distances between different measurements defined; they provide the greatest flexibility in statistical methods that can be used for analyzing the data. Interval measurements have meaningful distances between measurements defined, but the zero value is arbitrary (as in the case with longitude and temperature measurements in Celsius or Fahrenheit). Ordinal measurements have imprecise differences between consecutive values, but have a meaningful order to those values. Nominal measurements have no meaningful rank order among values. Because variables conforming only to nominal or ordinal measurements cannot be reasonably measured numerically, sometimes they are grouped together as categorical variables, whereas ratio and interval measurements are grouped together as quantitative variables, which can be either discrete or continuous, due to their numerical nature. Key terms used in statistics Null hypothesis Interpretation of statistical information can often involve the development of a null hypothesis in that the assumption is that whatever is proposed as a cause has no effect on the variable being measured. The best illustration for a novice is the predicament encountered by a jury trial. The null hypothesis, H0, asserts that the defendant is innocent, whereas the alternative hypothesis, H1, asserts that the defendant is guilty. The indictment comes because of suspicion of the guilt. The H0 (status quo) stands in opposition to H1 and is maintained unless H1 is supported by evidence"beyond a reasonable doubt". However,"failure to reject H0" in this case does not imply innocence, but merely that the evidence was insufficient to convict. So the jury does not necessarily accept H0 but fails to reject H0. While one can not "prove" a null hypothesis one can test how close it is to being true with a power test, which tests for type II errors. Error Working from a null hypothesis two basic forms of error are recognized: • Type I errors where the null hypothesis is falsely rejected giving a "false positive". • Type II errors where the null hypothesis fails to be rejected and an actual difference between populations is missed giving a false negative. Error also refers to the extent to which individual observations in a sample differ from a central value, such as the sample or population mean. Many statistical methods seek to minimize the mean-squared error, and these are called "methods of least squares." Measurement processes that generate statistical data are also subject to error. Many of these errors are classified as random (noise) or systematic (bias), but other important types of errors (e.g., blunder, such as when an analyst reports incorrect units) can also be important. Statistics 35 Interval estimation Most studies only sample part of a population , so results don't fully represent the whole population. Any estimates obtained from the sample only approximate the population value. Confidence intervals allow statisticians to express how closely the sample estimate matches the true value in the whole population. Often they are expressed as 95% confidence intervals. Formally, a 95% confidence interval for a value is a range where, if the sampling and analysis were repeated under the same conditions (yielding a different dataset), the interval would include the true (population) value 95% of the time. This does not imply that the probability that the true value is in the confidence interval is 95%. From the frequentist perspective, such a claim does not even make sense, as the true value is not a random variable. Either the true value is or is not within the given interval. However, it is true that, before any data are sampled and given a plan for how to construct the confidence interval, the probability is 95% that the yet-to-be-calculated interval will cover the true value: at this point, the limits of the interval are yet-to-be-observed random variables. One approach that does yield an interval that can be interpreted as having a given probability of containing the true value is to use a credible interval from Bayesian statistics: this approach depends on a different way of interpreting what is meant by "probability", that is as a Bayesian probability. Significance Statistics rarely give a simple Yes/No type answer to the question asked of them. Interpretation often comes down to the level of statistical significance applied to the numbers and often refers to the probability of a value accurately rejecting the null hypothesis (sometimes referred to as the p-value). Referring to statistical significance does not necessarily mean that the overall result is significant in real world terms. For example, in a large study of a drug it may be shown that the drug has a statistically significant but very small beneficial effect, such that the drug is unlikely to help the patient noticeably. Criticisms arise because the hypothesis testing approach forces one hypothesis (the null hypothesis) to be "favored," and can also seem to exaggerate the importance of minor differences in large studies. A difference that is highly statistically significant can still be of no practical significance, but it is possible to properly formulate tests in account for this. (See also criticism of hypothesis testing.) One response involves going beyond reporting only the significance level to include the p-value when reporting whether a hypothesis is rejected or accepted. The p-value, however, does not indicate the size of the effect. A better and increasingly common approach is to report confidence intervals. Although these are produced from the same calculations as those of hypothesis tests or p-values, they describe both the size of the effect and the uncertainty surrounding it. Examples Some well-known statistical tests and procedures are: • Analysis of variance (ANOVA) • Chi-squared test • Correlation • Factor analysis • Mann–Whitney U • Mean square weighted deviation (MSWD) • Pearson product-moment correlation coefficient • Regression analysis • Spearman's rank correlation coefficient • Student's t-test • Time series analysis Statistics 36 Specialized disciplines Statistical techniques are used in a wide range of types of scientific and social research, including: biostatistics, computational biology, computational sociology, network biology, social science, sociology and social research. Some fields of inquiry use applied statistics so extensively that they have specialized terminology. These disciplines include: • Actuarial science (assesses risk in the insurance and finance industries) • Applied information economics • Biostatistics • Business statistics • Chemometrics (for analysis of data from chemistry) • Data mining (applying statistics and pattern recognition to discover knowledge from data) • Demography • Econometrics • Energy statistics • Engineering statistics • Epidemiology • Geography and Geographic Information Systems, specifically in Spatial analysis • Image processing • Psychological statistics • Reliability engineering • Social statistics In addition, there are particular types of statistical analysis that have also developed their own specialised terminology and methodology: • Bootstrap & Jackknife Resampling • Multivariate statistics • Statistical classification • Statistical surveys • Structured data analysis (statistics) • Structural equation modelling • Survival analysis • Statistics in various sports, particularly baseball and cricket Statistics form a key basis tool in business and manufacturing as well. It is used to understand measurement systems variability, control processes (as in statistical process control or SPC), for summarizing data, and to make data-driven decisions. In these roles, it is a key tool, and perhaps the only reliable tool. Statistics 37 Statistical computing The rapid and sustained increases in computing power starting from the second half of the 20th century have had a substantial impact on the practice of statistical science. Early statistical models were almost always from the class of linear models, but powerful computers, coupled with suitable numerical algorithms, caused an increased interest in nonlinear models (such as neural networks) as well as the creation of new types, such as generalized linear models and multilevel models. Increased computing power has also led to the growing popularity of computationally intensive methods based on resampling, such as permutation tests and the bootstrap, gretl, an example of an open source statistical package while techniques such as Gibbs sampling have made use of Bayesian models more feasible. The computer revolution has implications for the future of statistics with new emphasis on "experimental" and "empirical" statistics. A large number of both general and special purpose statistical software are now available. Misuse There is a general perception that statistical knowledge is all-too-frequently intentionally misused by finding ways to interpret only the data that are favorable to the presenter.[17] A mistrust and misunderstanding of statistics is associated with the quotation, "There are three kinds of lies: lies, damned lies, and statistics". Misuse of statistics can be both inadvertent and intentional, and the book How to Lie With Statistics[17] outlines a range of considerations. In an attempt to shed light on the use and misuse of statistics, reviews of statistical techniques used in particular fields are conducted (e.g. Warne, Lazo, Ramos, and Ritter (2012)).[18] Ways to avoid misuse of statistics include using proper diagrams and avoiding bias.[19] "The misuse occurs when such conclusions are held to be representative of the universe by those who either deliberately or unconsciously overlook the sampling bias.[20]" Bar graphs are arguably the easiest diagrams to use and understand, and they can be made either with simple computer programs or hand drawn.[19] Unfortunately, most people do not look for bias or errors, so they do not see them. Thus, we believe something true that is not well-represented.[20] To make data gathered from statistics believable and accurate, the sample taken must be representative of the whole.[21] As Huff's book states,"The dependability of a sample can be destroyed by [bias]… allow yourself some degree of skepticism."[22] Statistics 38 Statistics applied to mathematics or the arts Traditionally, statistics was concerned with drawing inferences using a semi-standardized methodology that was "required learning" in most sciences. This has changed with use of statistics in non-inferential contexts. What was once considered a dry subject, taken in many fields as a degree-requirement, is now viewed enthusiastically. Initially derided by some mathematical purists, it is now considered essential methodology in certain areas. • In number theory, scatter plots of data generated by a distribution function may be transformed with familiar tools used in statistics to reveal underlying patterns, which may then lead to hypotheses. • Methods of statistics including predictive methods in forecasting are combined with chaos theory and fractal geometry to create video works that are considered to have great beauty. • The process art of Jackson Pollock relied on artistic experiments whereby underlying distributions in nature were artistically revealed. With the advent of computers, methods of statistics were applied to formalize such distribution driven natural processes, in order to make and analyze moving video art. • Methods of statistics may be used predicatively in performance art, as in a card trick based on a Markov process that only works some of the time, the occasion of which can be predicted using statistical methodology. • Statistics can be used to predicatively create art, as in the statistical or stochastic music invented by Iannis Xenakis, where the music is performance-specific. Though this type of artistry does not always come out as expected, it does behave in ways that are predictable and tunable using statistics. References [1] Dodge, Y. (2006) The Oxford Dictionary of Statistical Terms, OUP. ISBN 0-19-920613-9 [2] The Free Online Dictionary (http:/ / www. thefreedictionary. com/ dict. asp?Word=statistics) [3] "Statistics" (http:/ / www. merriam-webster. com/ dictionary/ statistics). Merriam-Webster Online Dictionary. . [4] "Statistic" (http:/ / www. merriam-webster. com/ dictionary/ statistic). Merriam-Webster Online Dictionary. . [5] Moses, Lincoln E. (1986) Think and Explain with Statistics, Addison-Wesley, ISBN 978-0-201-15619-5 . pp. 1–3 [6] Hays, William Lee, (1973) Statistics for the Social Sciences, Holt, Rinehart and Winston, p.xii, ISBN 978-0-03-077945-9 [7] Moore, David (1992). "Teaching Statistics as a Respectable Subject". In F. Gordon and S. Gordon. Statistics for the Twenty-First Century. Washington, DC: The Mathematical Association of America. pp. 14–25. ISBN 978-0-88385-078-7. [8] Chance, Beth L.; Rossman, Allan J. (2005). "Preface" (http:/ / www. rossmanchance. com/ iscam/ preface. pdf). Investigating Statistical Concepts, Applications, and Methods. Duxbury Press. ISBN 978-0-495-05064-3. . [9] Anderson, D.R.; Sweeney, D.J.; Williams, T.A.. (1994) Introduction to Statistics: Concepts and Applications, pp. 5–9. West Group. ISBN 978-0-314-03309-3 [10] Al-Kadi, Ibrahim A. (1992) "The origins of cryptology: The Arab contributions”, Cryptologia, 16(2) 97–126. doi:10.1080/0161-119291866801 [11] Singh, Simon (2000). The code book : the science of secrecy from ancient Egypt to quantum cryptography (1st Anchor Books ed.). New York: Anchor Books. ISBN 0-385-49532-3. [12] Villani, Giovanni. Encyclopædia Britannica. Encyclopædia Britannica 2006 Ultimate Reference Suite DVD. Retrieved on 2008-03-04. [13] Willcox, Walter (1938) "The Founder of Statistics". Review of the International Statistical Institute 5(4):321–328. JSTOR 1400906 [14] Breiman, Leo (2001). "Statistical Modelling: the two cultures". Statistical Science 16 (3): 199–231. doi:10.1214/ss/1009213726. MR1874152. CiteSeerX: 10.1.1.156.4933 (http:/ / citeseerx. ist. psu. edu/ viewdoc/ summary?doi=10. 1. 1. 156. 4933). [15] Lindley, D. (2000). "The Philosophy of Statistics". Journal of the Royal Statistical Society, Series D (The Statistician) 49 (3): 293–337. doi:10.1111/1467-9884.00238. JSTOR 2681060. [16] Thompson, B. (2006). Foundations of behavioral statistics. New York, NY: Guilford Press. [17] Huff, Darrell (1954) How to Lie With Statistics, WW Norton & Company, Inc. New York, NY. ISBN 0-393-31072-8 [18] Warne, R. Lazo, M., Ramos, T. and Ritter, N. (2012). Statistical Methods Used in Gifted Education Journals, 2006–2010. Gifted Child Quarterly, 56(3) 134–149. doi:10.1177/0016986212444122 [19] Encyclopedia of Archaeology. Credo Reference: Oxford: Elsevier Science. 2008. [20] Cohen, Jerome B. (December 1938). "Misuse of Statistics". Journal of the American Statistical Association (JSTOR) 33 (204): 657–674. doi:10.1080/01621459.1938.10502344. [21] Freund, J. F. (1988). "Modern Elementary Statistics". Credo Reference. [22] Huff, Darrell; Irving Geis (1954). How to Lie with Statistics. New York: Norton. "The dependability of a sample can be destroyed by [bias]… allow yourself some degree of skepticism." Bayesian inference 39 Bayesian inference In statistics, Bayesian inference is a method of inference in which Bayes' rule is used to update the probability estimate for a hypothesis as additional evidence is learned. Bayesian updating is an important technique throughout statistics, and especially in mathematical statistics: For some cases, exhibiting a Bayesian derivation for a statistical method automatically ensures that the method works as well as any competing method. Bayesian updating is especially important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a range of fields including science, engineering, medicine, and law. In the philosophy of decision theory, Bayesian inference is closely related to discussions of subjective probability, often called "Bayesian probability." Bayesian probability provides a rational method for updating beliefs;[1][2] however, non-Bayesian updating rules are compatible with rationality, according to philosophers Ian Hacking and Bas van Fraassen. Introduction to Bayes' rule Formal Bayesian inference derives the posterior probability as a consequence of two antecedents, a prior probability and a "likelihood function" derived from a probability model for the data to be observed. Bayesian inference computes the posterior probability according to Bayes' rule: where • means given. • stands for any hypothesis whose probability may be affected by data (called evidence below). Often there are competing hypotheses, from which one chooses the most probable. • the evidence corresponds to data that were not used in computing the prior probability. • , the prior probability, is the probability of before is observed. This indicates one's preconceived beliefs about how likely different hypotheses are, absent evidence regarding the instance under study. • , the posterior probability, is the probability of given , i.e., after is observed. This tells us what we want to know: the probability of a hypothesis given the observed evidence. • , the probability of observing given , is also known as the likelihood. It indicates the compatibility of the evidence with the given hypothesis. • is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered. (This can be seen by the fact that the hypothesis does not appear anywhere in the symbol, unlike for all the other factors.) This means that this factor does not enter into determining the relative probabilities of different hypotheses. Note that what affects the value of for different values of is only the factors and , which both appear in the numerator, and hence the posterior probability is proportional to both. In words: • (more exactly) The posterior probability of a hypothesis is determined by a combination of the inherent likeliness of a hypothesis (the prior) and the compatibility of the observed evidence with the hypothesis (the likelihood). • (more concisely) Posterior is proportional to prior times likelihood. Note that Bayes' rule can also be written as follows: Bayesian inference 40 where the factor represents the impact of on the probability of . Informal Rationally, Bayes' rule makes a great deal of sense. If the evidence doesn't match up with a hypothesis, one should reject the hypothesis. But if a hypothesis is extremely unlikely a priori, one should also reject it, even if the evidence does appear to match up. For example, imagine that I have various hypotheses about the nature of a newborn baby of a friend, including: • : the baby is a brown-haired boy. • : the baby is a blond-haired girl. • : the baby is a dog. Then consider two scenarios: 1. I'm presented with evidence in the form of a picture of a blond-haired baby girl. I find this evidence supports and opposes and . 2. I'm presented with evidence in the form of a picture of a baby dog. Although this evidence, treated in isolation, supports , my prior belief in this hypothesis (that a human can give birth to a dog) is extremely small, so the posterior probability is nevertheless small. The critical point about Bayesian inference, then, is that it provides a principled way of combining new evidence with prior beliefs, through the application of Bayes' rule. (Contrast this with frequentist inference, which relies only on the evidence as a whole, with no reference to prior beliefs.) Furthermore, Bayes' rule can be applied iteratively: after observing some evidence, the resulting posterior probability can then be treated as a prior probability, and a new posterior probability computed from new evidence. This allows for Bayesian principles to be applied to various kinds of evidence, whether viewed all at once or over time. This procedure is termed Bayesian updating. Bayesian updating Bayesian updating is widely used and computationally convenient. However, it is not the only updating rule that might be considered "rational." Ian Hacking noted that traditional "Dutch book" arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. Hacking wrote[3] "And neither the Dutch book argument, nor any other in the personalist arsenal of proofs of the probability axioms, entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour." Indeed, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics" following the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the case where the evidence itself is assigned a probability [4]). The additional hypotheses needed to uniquely require Bayesian updating have been deemed to be substantial, complicated, and unsatisfactory.[5] Bayesian inference 41 Formal description of Bayesian inference Definitions • , a data point in general. This may in fact be a vector of values. • , the parameter of the data point's distribution, i.e. . This may in fact be a vector of parameters. • , the hyperparameter of the parameter, i.e. . This may in fact be a vector of hyperparameters. • , a set of observed data points, i.e. . • , a new data point whose distribution is to be predicted. Bayesian inference • The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. . • The prior distribution might not be easily determined. In this case, we can use the Jeffreys prior to obtain the posterior distribution before updating them with newer observations. • The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. . This is also termed the likelihood, especially when viewed as a function of the parameter(s), sometimes written . • The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. . • The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference: Note that this is expressed in words as "posterior is proportional to prior times likelihood", or sometimes as "posterior = prior times likelihood, over evidence". Bayesian prediction • The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior: • The prior predictive distribution is the distribution of a new data point, marginalized over the prior: Bayesian theory calls for the use of the posterior predictive distribution to do predictive inference, i.e. to predict the distribution of a new, unobserved data point. That is, instead of a fixed point as a prediction, a distribution over possible points is returned. Only this way is the entire posterior distribution of the parameter(s) used. By comparison, prediction in frequentist statistics often involves finding an optimum point estimate of the parameter(s) — e.g. by maximum likelihood or maximum a posteriori estimation (MAP) — and then plugging this estimate into the formula for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value of the parameter, and hence will underestimate the variance of the predictive distribution. (In some instances, frequentist statistics can work around this problem. For example, confidence intervals and prediction intervals in frequentist statistics when constructed from a normal distribution with unknown mean and variance are constructed using a Student's t-distribution. This correctly estimates the variance, due to the fact that (1) the average of normally distributed random variables is also normally distributed; (2) the predictive distribution of a normally distributed data point with unknown mean and variance, using conjugate or uninformative priors, has a Bayesian inference 42 Student's t-distribution. In Bayesian statistics, however, the posterior predictive distribution can always be determined exactly — or at least, to an arbitrary level of precision, when numerical methods are used.) Note that both types of predictive distributions have the form of a compound probability distribution (as does the marginal likelihood). In fact, if the prior distribution is a conjugate prior, and hence the prior and posterior distributions come from the same family, it can easily be seen that both prior and posterior predictive distributions also come from the same family of compound distributions. The only difference is that the posterior predictive distribution uses the updated values of the hyperparameters (applying the Bayesian update rules given in the conjugate prior article), while the prior predictive distribution uses the values of the hyperparameters that appear in the prior distribution. Inference over exclusive and exhaustive possibilities If evidence is simultaneously used to update belief over a set of exclusive and exhaustive propositions, Bayesian inference may be thought of as acting on this belief distribution as a whole. General formulation Suppose a process is generating independent and identically distributed events , but the probability distribution is unknown. Let the event space represent the current state of belief for this process. Each model is represented by event . The conditional probabilities are specified to define the models. is the degree of belief in . Before the first inference step, is a set of initial prior probabilities. These must sum to 1, but are otherwise arbitrary. Suppose that the process is observed to generate . For each , the prior is updated to the posterior [6] . From Bayes' theorem: Diagram illustrating event space in general formulation of Bayesian inference. Although this diagram shows discrete models and events, the Upon observation of further evidence, this procedure may be repeated. continuous case may be visualized similarly using probability densities. Multiple observations For a set of independent and identically distributed observations , it may be shown that repeated application of the above is equivalent to Where This may be used to optimize practical calculations. Bayesian inference 43 Parametric formulation By parametrizing the space of models, the belief in all models may be updated in a single step. The distribution of belief over the model space may then be thought of as a distribution of belief over the parameter space. The distributions in this section are expressed as continuous, represented by probability densities, as this is the usual situation. The technique is however equally applicable to discrete distributions. Let the vector span the parameter space. Let the initial prior distribution over be , where is a set of parameters to the prior itself, or hyperparameters. Let be a set of independent and identically distributed event observations, where all are distributed as for some . Bayes' theorem is applied to find the posterior distribution over : Where Mathematical properties Interpretation of factor . That is, if the model were true, the evidence would be more likely than is predicted by the current state of belief. The reverse applies for a decrease in belief. If the belief does not change, . That is, the evidence is independent of the model. If the model were true, the evidence would be exactly as likely as predicted by the current state of belief. Cromwell's rule If then . If , then . This can be interpreted to mean that hard convictions are insensitive to counter-evidence. The former follows directly from Bayes' theorem. The latter can be derived by applying the first rule to the event "not " in place of " ," yielding "if , then ," from which the result immediately follows. Asymptotic behaviour of posterior Consider the behaviour of a belief distribution as it is updated a large number of times with independent and identically distributed trials. For sufficiently nice prior probabilities, the Bernstein-von Mises theorem gives that in the limit of infinite trials and the posterior converges to a Gaussian distribution independent of the initial prior under some conditions firstly outlined and rigorously proven by Joseph Leo Doob in 1948, namely if the random variable in consideration has a finite probability space. The more general results were obtained later by the statistician David A. Freedman who published in two seminal research papers in 1963 and 1965 when and under what circumstances the asymptotic behaviour of posterior is guaranteed. His 1963 paper treats, like Doob (1949), the finite case and comes to a satisfactory conclusion. However, if the random variable has an infinite but countable probability space (i.e. corresponding to a die with infinite many faces) the 1965 paper demonstrates that for a dense subset of priors the Bernstein-von Mises theorem is not applicable. In this case there is almost surely no asymptotic convergence. Later in the eighties and nineties Freedman and Persi Diaconis continued to work on the case of infinite countable Bayesian inference 44 probability spaces.[7] To summarise, there may be insufficient trials to suppress the effects of the initial choice, and especially for large (but finite) systems the convergence might be very slow. Conjugate priors In paramaterized form, the prior distribution is often assumed to come from a family of distributions called conjugate priors. The usefulness of a conjugate prior is that the corresponding posterior distribution will be in the same family, and the calculation may be expressed in closed form. Estimates of parameters and predictions It is often desired to use a posterior distribution to estimate a parameter or variable. Several methods of Bayesian estimation select measurements of central tendency from the posterior distribution. First, when the parameter space has two dimensions or more, there exists a unique median of the posterior distribution. For one-dimensional problems, a unique median exists for practical continuous problems. The posterior median is attractive as a robust estimator.[8] Second, if there exists a finite mean for the posterior distribution, then the posterior mean is a method of estimation. Third, taking a value with the greatest probability defines any set of maximum a posteriori (MAP) estimates: There are examples where no maximum is attained, in which case the set of MAP estimates is empty. There are other methods of estimation that minimize the posterior risk (expected-posterior loss) with respect to a loss function, and these are of interest to statistical decision theory using the sampling distribution ("frequentist statistics"). The posterior predictive distribution of a new observation (that is exchangeable with previous observations) is determined by Examples Probability of a hypothesis Suppose there are two full bowls of cookies. Bowl #1 has 10 chocolate chip and 30 plain cookies, while bowl #2 has 20 of each. Our friend Fred picks a bowl at random, and then picks a cookie at random. We may assume there is no reason to believe Fred treats one bowl differently from another, likewise for the cookies. The cookie turns out to be a plain one. How probable is it that Fred picked it out of bowl #1? Intuitively, it seems clear that the answer should be more than a half, since there are more plain cookies in bowl #1. The precise answer is given by Bayes' theorem. Let correspond to bowl #1, and to bowl #2. It is given that the bowls are identical from Fred's point of view, thus , and the two must add up to 1, so both are equal to 0.5. The event is the observation of a plain cookie. From the contents of the bowls, we know that and . Bayes' formula then yields Bayesian inference 45 Before we observed the cookie, the probability we assigned for Fred having chosen bowl #1 was the prior probability, , which was 0.5. After observing the cookie, we must revise the probability to , which is 0.6. Making a prediction An archaeologist is working at a site thought to be from the medieval period, between the 11th century to the 16th century. However, it is uncertain exactly when in this period the site was inhabited. Fragments of pottery are found, some of which are glazed and some of which are decorated. It is expected that if the site were inhabited during the early medieval period, then 1% of the pottery would be glazed and 50% of its area decorated, whereas if it had been inhabited in the late medieval period then 81% would be glazed and 5% of its area decorated. How Example results for archaeology example. This simulation was generated using c=15.2. confident can the archaeologist be in the date of inhabitation as fragments are unearthed? The degree of belief in the continuous variable (century) is to be calculated, with the discrete set of events as evidence. Assuming linear variation of glaze and decoration with time, and that these variables are independent, Assume a uniform prior of , and that trials are independent and identically distributed. When a new fragment of type is discovered, Bayes' theorem is applied to update the degree of belief for each : A computer simulation of the changing belief as 50 fragments are unearthed is shown on the graph. In the simulation, the site was inhabited around 1520, or . By calculating the area under the relevant portion of the graph for 50 trials, the archaeologist can say that there is practically no chance the site was inhabited in the 11th and 12th centuries, about 1% chance that it was inhabited during the 13th century, 63% chance during the 14th century and 36% during the 15th century. Note that the Bernstein-von Mises theorem asserts here the asymptotic convergence to the "true" distribution because the probability space corresponding to the discrete set of events is finite (see above section on asymptotic behaviour of the posterior). Bayesian inference 46 In frequentist statistics and decision theory A decision-theoretic justification of the use of Bayesian inference was given by Abraham Wald, who proved that every Bayesian procedure is admissible. Conversely, every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.[9] Wald characterized admissible procedures as Bayesian procedures (and limits of Bayesian procedures), making the Bayesian formalism a central technique in such areas of frequentist inference as parameter estimation, hypothesis testing, and computing confidence intervals.[10] For example: • "Under some conditions, all admissible procedures are either Bayes procedures or limits of Bayes procedures (in various senses). These remarkable results, at least in their original form, are due essentially to Wald. They are useful because the property of being Bayes is easier to analyze than admissibility."[9] • "In decision theory, a quite general method for proving admissibility consists in exhibiting a procedure as a unique Bayes solution."[11] • "In the first chapters of this work, prior distributions with finite support and the corresponding Bayes procedures were used to establish some of the main theorems relating to the comparison of experiments. Bayes procedures with respect to more general prior distributions have played a very important role in the development of statistics, including its asymptotic theory." "There are many problems where a glance at posterior distributions, for suitable priors, yields immediately interesting information. Also, this technique can hardly be avoided in sequential analysis."[12] • "A useful fact is that any Bayes decision rule obtained by taking a proper prior over the whole parameter space must be admissible"[13] • "An important area of investigation in the development of admissibility ideas has been that of conventional sampling-theory procedures, and many interesting results have been obtained."[14] Applications Computer applications Bayesian inference has applications in artificial intelligence and expert systems. Bayesian inference techniques have been a fundamental part of computerized pattern recognition techniques since the late 1950s. There is also an ever growing connection between Bayesian methods and simulation-based Monte Carlo techniques since complex models cannot be processed in closed form by a Bayesian analysis, while a graphical model structure may allow for efficient simulation algorithms like the Gibbs sampling and other Metropolis–Hastings algorithm schemes.[15] Recently Bayesian inference has gained popularity amongst the phylogenetics community for these reasons; a number of applications allow many demographic and evolutionary parameters to be estimated simultaneously. In the areas of population genetics and dynamical systems theory, approximate Bayesian computation (ABC) is also becoming increasingly popular. As applied to statistical classification, Bayesian inference has been used in recent years to develop algorithms for identifying e-mail spam. Applications which make use of Bayesian inference for spam filtering include CRM114, DSPAM, Bogofilter, SpamAssassin, SpamBayes, and Mozilla. Spam classification is treated in more detail in the article on the naive Bayes classifier. Solomonoff's Inductive inference is the theory of prediction based on observations; for example, predicting the next symbol based upon a given series of symbols. The only assumption is that the environment follows some unknown but computable probability distribution. It is a formal inductive framework that combines two well-studied principles of inductive inference: Bayesian statistics and Occam’s Razor.[16] Solomonoff's universal prior probability of any prefix p of a computable sequence x is the sum of the probabilities of all programs (for a universal computer) that compute something starting with p. Given some p and any computable but unknown probability distribution from Bayesian inference 47 which x is sampled, the universal prior and Bayes' theorem can be used to predict the yet unseen parts of x in optimal fashion. [17][18] In the courtroom Bayesian inference can be used by jurors to coherently accumulate the evidence for and against a defendant, and to see whether, in totality, it meets their personal threshold for 'beyond a reasonable doubt'.[19][20][21] Bayes' theorem is applied successively to all evidence presented, with the posterior from one stage becoming the prior for the next. The benefit of a Bayesian approach is that it gives the juror an unbiased, rational mechanism for combining evidence. It may be appropriate to explain Bayes' theorem to jurors in odds form, as betting odds are more widely understood than probabilities. Alternatively, a logarithmic approach, replacing multiplication with addition, might be easier for a jury to handle. If the existence of the crime is not in doubt, only the identity of the culprit, it has been suggested that the prior should be uniform over the qualifying population.[22] For example, if 1000 people could have committed the crime, the prior probability of guilt would be 1/1000. The use of Bayes' theorem by jurors is controversial. In the United Kingdom, a defence expert witness explained Bayes' theorem to the jury in R v Adams. The jury convicted, but the case went to appeal on the basis that no means of accumulating evidence had been provided for jurors who did not wish to use Bayes' theorem. The Court of Appeal upheld the conviction, but it also gave the opinion that "To introduce Bayes' Theorem, or any similar method, into a criminal trial plunges the jury into inappropriate and unnecessary realms of theory and Adding up evidence. complexity, deflecting them from their proper task." Gardner-Medwin[23] argues that the criterion on which a verdict in a criminal trial should be based is not the probability of guilt, but rather the probability of the evidence, given that the defendant is innocent (akin to a frequentist p-value). He argues that if the posterior probability of guilt is to be computed by Bayes' theorem, the prior probability of guilt must be known. This will depend on the incidence of the crime, which is an unusual piece of evidence to consider in a criminal trial. Consider the following three propositions: A The known facts and testimony could have arisen if the defendant is guilty B The known facts and testimony could have arisen if the defendant is innocent C The defendant is guilty. Gardner-Medwin argues that the jury should believe both A and not-B in order to convict. A and not-B implies the truth of C, but the reverse is not true. It is possible that B and C are both true, but in this case he argues that a jury should acquit, even though they know that they will be letting some guilty people go free. See also Lindley's paradox. Bayesian inference 48 Bayesian Epistemology Bayesian Epistemology is an epistemological movement that uses techniques of Bayesian inference as a means of justifying the rules of inductive logic. Karl Popper and David Miller have rejected the alleged rationality of Bayesianism, ie., using Bayes rule to make epistemological inferences:[24] It is prone to the same vicious circle as any other justificationist epistemology, because it presupposes what it attempts to justify. According to this view, a rational interpretation of Bayesian inference would see it merely as a probabilistic version of falsification, rejecting the belief, commonly held by Bayesianists, that high likelyhood achieved by a series of Bayesian updates would prove the hypothesis beyond any reasonable doubt, or even with likelyhood greater than 0. Other • The scientific method is sometimes interpreted as an application of Bayesian inference. In this view, Bayes' rule guides (or should guide) the updating of probabilities about hypotheses conditional on new observations or experiments.[25] • Bayesian search theory is used to search for lost objects. • Bayesian inference in phylogeny • Bayesian tool for methylation analysis Bayes and Bayesian inference The problem considered by Bayes in Proposition 9 of his essay, "An Essay towards solving a Problem in the Doctrine of Chances", is the posterior distribution for the parameter a (the success rate) of the binomial distribution. What is "Bayesian" about Proposition 9 is that Bayes presented it as a probability for the parameter . That is, not only can one compute probabilities for experimental outcomes, but also for the parameter which governs them, and the same algebra is used to make inferences of either kind. Interestingly, Bayes actually states his question in a way that might make the idea of assigning a probability distribution to a parameter palatable to a frequentist. He supposes that a billiard ball is thrown at random onto a billiard table, and that the probabilities p and q are the probabilities that subsequent billiard balls will fall above or below the first ball. By making the binomial parameter depend on a random event, he cleverly escapes a philosophical quagmire that was an issue he most likely was not even aware of. History The term Bayesian refers to Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem. However, it was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence.[26] Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes[27]). After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.[27] In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objective or "non-informative" current, the statistical analysis depends on only the model assumed, the data analyzed,[28] and the method assigning the prior, which differs from one objective Bayesian to another objective Bayesian. In the subjective or "informative" current, the specification of the prior depends on the belief (that is, propositions on which the analysis is prepared to act), which can summarize information from experts, previous studies, etc. In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an Bayesian inference 49 increasing interest in nonstandard, complex applications.[29] Despite growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics.[30] Nonetheless, Bayesian methods are widely accepted and used, such as for example in the field of machine learning.[31] Notes [1] Stanford encyclopedia of philosophy; Bayesian Epistemology; http:/ / plato. stanford. edu/ entries/ epistemology-bayesian [2] Gillies, Donald (2000); "Philosophical Theories of Probability"; Routledge; Chapter 4 "The subjective theory" [3] Hacking (1967, Section 3, page 316), Hacking (1988, page 124) [4] http:/ / plato. stanford. edu/ entries/ bayes-theorem/ [5] van Fraassen, B. (1989) Laws and Symmetry, Oxford University Press. ISBN 0-19-824860-1 [6] Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Rubin, Donald B. (2003). Bayesian Data Analysis, Second Edition. Boca Raton, FL: Chapman and Hall/CRC. ISBN 1-58488-388-X. [7] Larry Wasserman et alia, JASA 2000. [8] Sen, Pranab K.; Keating, J. P.; Mason, R. L. (1993). Pitman's measure of closeness: A comparison of statistical estimators. Philadelphia: SIAM. [9] Bickel & Doksum (2001, page 32) [10] * Kiefer, J. and Schwartz, R. (1965). "Admissible Bayes character of T2-, R2-, and other fully invariant tests for multivariate normal problems". Annals of Mathematical Statistics 36: 747–770. doi:10.1214/aoms/1177700051. • Schwartz, R. (1969). "Invariant proper Bayes tests for exponential families". Annals of Mathematical Statistics 40: 270–283. doi:10.1214/aoms/1177697822. • Hwang, J. T. and Casella, George (1982). "Minimax confidence sets for the mean of a multivariate normal distribution". Annals of Statistics 10: 868–881. doi:10.1214/aos/1176345877. [11] Lehmann, Erich (1986). Testing Statistical Hypotheses (Second ed.). (see page 309 of Chapter 6.7 "Admissibilty", and pages 17–18 of Chapter 1.8 "Complete Classes" [12] Le Cam, Lucien (1986). Asymptotic Methods in Statistical Decision Theory. Springer-Verlag. ISBN 0-387-96307-3. (From "Chapter 12 Posterior Distributions and Bayes Solutions", page 324) [13] Cox, D. R. and Hinkley, D. V (1974). Theoretical Statistics. Chapman and Hall. ISBN 0-04-121537-0. page 432 [14] Cox, D. R. and Hinkley, D. V (1974). Theoretical Statistics. Chapman and Hall. ISBN 0-04-121537-0. page 433) [15] Jim Albert (2009). Bayesian Computation with R, Second edition. New York, Dordrecht, etc.: Springer. ISBN 978-0-387-92297-3. [16] Samuel Rathmanner and Marcus Hutter. A philosophical treatise of universal induction. Entropy, 13(6):1076–1136, 2011. [17] "The problem of old evidence" ,in §5 of "On universal prediction and Bayesian confirmation", M Hutter - Theoretical Computer Science, 2007 - Elsevier http:/ / arxiv. org/ pdf/ 0709. 1516 [18] "Raymond J. Solomonoff", Peter Gacs, Paul M. B. Vitanyi, 2011 cs.bu.edu, http:/ / citeseerx. ist. psu. edu/ viewdoc/ download?doi=10. 1. 1. 186. 8268& rep=rep1& type=pdf [19] Dawid, A.P. and Mortera, J. (1996) "Coherent analysis of forensic identification evidence". Journal of the Royal Statistical Society, Series B, 58,425–443. [20] Foreman, L.A; Smith, A.F.M. and Evett, I.W. (1997). "Bayesian analysis of deoxyribonucleic acid profiling data in forensic identification applications (with discussion)". Journal of the Royal Statistical Society, Series A, 160, 429–469. [21] Robertson, B. and Vignaux, G.A. (1995) Interpreting Evidence: Evaluating Forensic Science in the Courtroom. John Wiley and Sons. Chichester. ISBN 978-0-471-96026-3 [22] Dawid, A.P. (2001) "Bayes' Theorem and Weighing Evidence by Juries"; http:/ / 128. 40. 111. 250/ evidence/ content/ dawid-paper. pdf [23] Gardner-Medwin, A. (2005) "What probability should the jury address?". Significance, 2 (1), March 2005 [24] David Miller: Critical Rationalism [25] Howson & Urbach (2005), Jaynes (2003) [26] Stephen M. Stigler (1986) The history of statistics. Harvard University press. Chapter 3. [27] Stephen. E. Fienberg, (2006) "When did Bayesian Inference become "Bayesian"? (http:/ / ba. stat. cmu. edu/ journal/ 2006/ vol01/ issue01/ fienberg. pdf) Bayesian Analysis, 1 (1), 1–40. See page 5. [28] JM. Bernardo (2005), "Reference analysis", Handbook of statistics, 25, 17–90 [29] Wolpert, RL. (2004) A conversation with James O. Berger, Statistical science, 9, 205–218 [30] José M. Bernardo (2006) A Bayesian mathematical statistics primer (http:/ / www. ime. usp. br/ ~abe/ ICOTS7/ Proceedings/ PDFs/ InvitedPapers/ 3I2_BERN. pdf). ICOTS-7 [31] Bishop, C.M. (2007) Pattern Recognition and Machine Learning. Springer, 2007 Bayesian inference 50 References • Aster, Richard; Borchers, Brian, and Thurber, Clifford (2012). Parameter Estimation and Inverse Problems, Second Edition, Elsevier. ISBN 0123850487, ISBN 978-0123850485 • Bickel, Peter J. and Doksum, Kjell A. (2001). Mathematical Statistics, Volume 1: Basic and Selected Topics (Second (updated printing 2007) ed.). Pearson Prentice–Hall. ISBN 0-13-850363-X. • Box, G.E.P. and Tiao, G.C. (1973) Bayesian Inference in Statistical Analysis, Wiley, ISBN 0-471-57428-7 • Edwards, Ward (1968). "Conservatism in Human Information Processing". In Kleinmuntz, B. Formal Representation of Human Judgment. Wiley. • Edwards, Ward (1982). "Conservatism in Human Information Processing (excerpted)". In Daniel Kahneman, Paul Slovic and Amos Tversky. Judgment under uncertainty: Heuristics and biases. Cambridge University Press. • Jaynes E.T. (2003) Probability Theory: The Logic of Science, CUP. ISBN 978-0-521-59271-0 ( Link to Fragmentary Edition of March 1996 (http://www-biba.inrialpes.fr/Jaynes/prob.html)). • Howson, C. and Urbach, P. (2005). Scientific Reasoning: the Bayesian Approach (3rd ed.). Open Court Publishing Company. ISBN 978-0-8126-9578-6. • Phillips, L.D.; Edwards, W. (October 2008). "Chapter 6: Conservatism in a simple probability inference task (Journal of Experimental Psychology (1966) 72: 346-354)". In Jie W. Weiss and David J. Weiss. A Science of Decision Making:The Legacy of Ward Edwards. Oxford University Press. pp. 536. ISBN 978-0-19-532298-9. Further reading Elementary The following books are listed in ascending order of probabilistic sophistication: • Colin Howson and Peter Urbach (2005). Scientific Reasoning: the Bayesian Approach (3rd ed.). Open Court Publishing Company. ISBN 978-0-8126-9578-6. • Berry, Donald A. (1996). Statistics: A Bayesian Perspective. Duxbury. ISBN 0-534-23476-3. • Morris H. DeGroot and Mark J. Schervish (2002). Probability and Statistics (third ed.). Addison-Wesley. ISBN 978-0-201-52488-8. • Bolstad, William M. (2007) Introduction to Bayesian Statistics: Second Edition, John Wiley ISBN 0-471-27020-2 • Winkler, Robert L, Introduction to Bayesian Inference and Decision, 2nd Edition (2003) ISBN 0-9647938-4-9 • Lee, Peter M. Bayesian Statistics: An Introduction. Fourth Edition (2012), John Wiley ISBN 978-1-1183-3257-3 • Carlin, Bradley P. and Louis, Thomas A. (2008). Bayesian Methods for Data Analysis, Third Edition. Boca Raton, FL: Chapman and Hall/CRC. ISBN 1-58488-697-8. • Gelman, Andrew; Carlin, John B.; Stern, Hal S.; Rubin, Donald B. (2003). Bayesian Data Analysis, Second Edition. Boca Raton, FL: Chapman and Hall/CRC. ISBN 1-58488-388-X. Intermediate or advanced • Berger, James O (1985). Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics (Second ed.). Springer-Verlag. ISBN 0-387-96098-8. • Bernardo, José M.; Smith, Adrian F. M. (1994). Bayesian Theory. Wiley. • DeGroot, Morris H., Optimal Statistical Decisions. Wiley Classics Library. 2004. (Originally published (1970) by McGraw-Hill.) ISBN 0-471-68029-X. • Schervish, Mark J. (1995). Theory of statistics. Springer-Verlag. ISBN 0-387-94546-6. • Jaynes, E.T. (1998) Probability Theory: The Logic of Science (http://www-biba.inrialpes.fr/Jaynes/prob. html). • O'Hagan, A. and Forster, J. (2003) Kendall's Advanced Theory of Statistics, Volume 2B: Bayesian Inference. Arnold, New York. ISBN 0-340-52922-9. Bayesian inference 51 • Robert, Christian P (2001). The Bayesian Choice – A Decision-Theoretic Motivation (second ed.). Springer. ISBN 0-387-94296-3. • Glenn Shafer and Pearl, Judea, eds. (1988) Probabilistic Reasoning in Intelligent Systems, San Mateo, CA: Morgan Kaufmann. External links • Hazewinkel, Michiel, ed. (2001), "Bayesian approach to statistical problems" (http://www.encyclopediaofmath. org/index.php?title=p/b015390), Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4 • Bayesian Statistics (http://www.scholarpedia.org/article/Bayesian_statistics) from Scholarpedia. • Introduction to Bayesian probability (http://www.dcs.qmw.ac.uk/~norman/BBNs/BBNs.htm) from Queen Mary University of London • Mathematical notes on Bayesian statistics and Markov chain Monte Carlo (http://webuser.bus.umich.edu/ plenk/downloads.htm) • Bayesian reading list (http://cocosci.berkeley.edu/tom/bayes.html), categorized and annotated by Tom Griffiths (http://psychology.berkeley.edu/faculty/profiles/tgriffiths.html). • Stanford Encyclopedia of Philosophy: "Inductive Logic" (http://plato.stanford.edu/entries/logic-inductive/) • Bayesian Confirmation Theory (http://faculty-staff.ou.edu/H/James.A.Hawthorne-1/ Hawthorne--Bayesian_Confirmation_Theory.pdf) • What is Bayesian Learning? (http://www.faqs.org/faqs/ai-faq/neural-nets/part3/section-7.html) Statistical hypothesis testing A statistical hypothesis test is a method of making decisions using data from a scientific study. In statistics, a result is called statistically significant if it has been predicted as unlikely to have occurred by chance alone, according to a pre-determined threshold probability, the significance level. The phrase "test of significance" was coined by statistician Ronald Fisher.[1] These tests are used in determining what outcomes of a study would lead to a rejection of the null hypothesis for a pre-specified level of significance; this can help to decide whether results contain enough information to cast doubt on conventional wisdom, given that conventional wisdom has been used to establish the null hypothesis. The critical region of a hypothesis test is the set of all outcomes which cause the null hypothesis to be rejected in favor of the alternative hypothesis. Statistical hypothesis testing is sometimes called confirmatory data analysis, in contrast to exploratory data analysis, which may not have pre-specified hypotheses. Statistical hypothesis testing is a key technique of frequentist statistical inference. Statistical hypothesis tests define a procedure which controls (fixes) the probability of incorrectly deciding that a default position (null hypothesis) is incorrect based on how likely it would be for a set of observations to occur if the null hypothesis were true. Note that this probability of making an incorrect decision is not the probability that the null hypothesis is true, nor whether any specific alternative hypothesis is true. This contrasts with other possible techniques of decision theory in which the null and alternative hypothesis are treated on a more equal basis. One naive Bayesian approach to hypothesis testing is to base decisions on the posterior probability,[2][3] but this fails when comparing point and continuous hypotheses. Other approaches to decision making, such as Bayesian decision theory, attempt to balance the consequences of incorrect decisions across all possibilities, rather than concentrating on a single null hypothesis. A number of other approaches to reaching a decision based on data are available via decision theory and optimal decisions, some of which have desirable properties, yet hypothesis testing is a dominant approach to data analysis in many fields of science. Extensions to the theory of hypothesis testing include the study of the power of tests, which refers to the probability of correctly rejecting the null hypothesis when a given state of nature exists. Such considerations can be used for the purpose of sample size determination prior to the collection of data. Statistical hypothesis testing 52 In a famous example of hypothesis testing, known as the Lady tasting tea example,[4] a female colleague of Fisher claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups, four of each variety, in random order. One could then ask what the probability was for her getting the number she got correct, but just by chance. The null hypothesis was that the Lady had no such ability. The test statistic was a simple count of the number of successes in selecting the 4 cups. The critical region was the single case of 4 successes of 4 possible based on a conventional probability criterion (< 5%; 1 of 70 ≈ 1.4%). Fisher asserted that no alternative hypothesis was (ever) required. The lady correctly identified every cup,[5] which would be considered a statistically significant result. The testing process In the statistical literature, statistical hypothesis testing plays a fundamental role.[6] The usual line of reasoning is as follows: 1. There is an initial research hypothesis of which the truth is unknown. 2. The first step is to state the relevant null and alternative hypotheses. This is important as mis-stating the hypotheses will muddy the rest of the process. Specifically, the null hypothesis allows to attach an attribute: it should be chosen in such a way that it allows us to conclude whether the alternative hypothesis can either be accepted or stays undecided as it was before the test.[7] 3. The second step is to consider the statistical assumptions being made about the sample in doing the test; for example, assumptions about the statistical independence or about the form of the distributions of the observations. This is equally important as invalid assumptions will mean that the results of the test are invalid. 4. Decide which test is appropriate, and state the relevant test statistic T. 5. Derive the distribution of the test statistic under the null hypothesis from the assumptions. In standard cases this will be a well-known result. For example the test statistic may follow a Student's t distribution or a normal distribution. 6. Select a significance level (α), a probability threshold below which the null hypothesis will be rejected. Common values are 5% and 1%. 7. The distribution of the test statistic under the null hypothesis partitions the possible values of T into those for which the null-hypothesis is rejected, the so called critical region, and those for which it is not. The probability of the critical region is α. 8. Compute from the observations the observed value tobs of the test statistic T. 9. Decide to either reject the null hypothesis in favor of the alternative or not reject it. The decision rule is to reject the null hypothesis H0 if the observed value tobs is in the critical region, and to accept or "fail to reject" the hypothesis otherwise. An alternative process is commonly used: • Compute from the observations the observed value tobs of the test statistic T. 2. From the statistic calculate a probability of the observation under the null hypothesis (the p-value). 3. Reject the null hypothesis in favor of the alternative or not reject it. The decision rule is to reject the null hypothesis if and only if the p-value is less than the significance level (the selected probability) threshold. The two processes are equivalent.[8] The former process was advantageous in the past when only tables of test statistics at common probability thresholds were available. It allowed a decision to be made without the calculation of a probability. It was adequate for classwork and for operational use, but it was deficient for reporting results. The latter process relied on extensive tables or on computational support not always available. The explicit calculation of a probability is useful for reporting. The calculations are now trivially performed with appropriate software. The difference in the two processes applied to the Radioactive suitcase example: Statistical hypothesis testing 53 • "The Geiger-counter reading is 10. The limit is 9. Check the suitcase." • "The Geiger-counter reading is high; 97% of safe suitcases have lower readings. The limit is 95%. Check the suitcase." The former report is adequate, the latter gives a more detailed explanation of the data and the reason why the suitcase is being checked. It is important to note the philosophical difference between accepting the null hypothesis and simply failing to reject it. The "fail to reject" terminology highlights the fact that the null hypothesis is assumed to be true from the start of the test; if there is a lack of evidence against it, it simply continues to be assumed true. The phrase "accept the null hypothesis" may suggest it has been proved simply because it has not been disproved, a logical fallacy known as the argument from ignorance. Unless a test with particularly high power is used, the idea of "accepting" the null hypothesis may be dangerous. Nonetheless the terminology is prevalent throughout statistics, where its meaning is well understood. Alternatively, if the testing procedure forces us to reject the null hypothesis (H0), we can accept the alternative hypothesis (H1) and we conclude that the research hypothesis is supported by the data. This fact expresses that our procedure is based on probabilistic considerations in the sense we accept that using another set of data could lead us to a different conclusion. The processes described here are perfectly adequate for computation. They seriously neglect the design of experiments considerations.[9][10] It is particularly critical that appropriate sample sizes be estimated before conducting the experiment. Interpretation If the p-value is less than the required significance level (equivalently, if the observed test statistic is in the critical region), then we say the null hypothesis is rejected at the given level of significance. Rejection of the null hypothesis is a conclusion. This is like a "guilty" verdict in a criminal trial - the evidence is sufficient to reject innocence, thus proving guilt. We might accept the alternative hypothesis (and the research hypothesis). If the p-value is not less than the required significance level (equivalently, if the observed test statistic is outside the critical region), then the test has no result. The evidence is insufficient to support a conclusion. (This is like a jury that fails to reach a verdict.) The researcher typically gives extra consideration to those cases where the p-value is close to the significance level. In the Lady tasting tea example (above), Fisher required the Lady to properly categorize all of the cups of tea to justify the conclusion that the result was unlikely to result from chance. He defined the critical region as that case alone. The region was defined by a probability (that the null hypothesis was correct) of less than 5%. Whether rejection of the null hypothesis truly justifies acceptance of the research hypothesis depends on the structure of the hypotheses. Rejecting the hypothesis that a large paw print originated from a bear does not immediately prove the existence of Bigfoot. Hypothesis testing emphasizes the rejection which is based on a probability rather than the acceptance which requires extra steps of logic. Statistical hypothesis testing 54 Use and Importance Statistics are helpful in analyzing most collections of data. This is equally true of hypothesis testing which can justify conclusions even when no scientific theory exists. In the Lady tasting tea example, it was "obvious" that no difference existed between (milk poured into tea) and (tea poured into milk). The data contradicted the "obvious". Real world applications of hypothesis testing include:[11] • Testing whether more men than women suffer from nightmares • Establishing authorship of documents • Evaluating the effect of the full moon on behavior • Determining the range at which a bat can detect an insect by echo • Deciding whether hospital carpeting results in more infections • Selecting the best means to stop smoking • Checking whether bumper stickers reflect car owner behavior • Testing the claims of handwriting analysts Statistical hypothesis testing plays an important role in the whole of statistics and in statistical inference. For example, Lehmann (1992) in a review of the fundamental paper by Neyman and Pearson (1933) says: "Nevertheless, despite their shortcomings, the new paradigm formulated in the 1933 paper, and the many developments carried out within its framework continue to play a central role in both the theory and practice of statistics and can be expected to do so in the foreseeable future". Significance testing has been the favored statistical tool in some experimental social sciences (over 90% of articles in the Journal of Applied Psychology during the early 1990s).[12] Other fields have favored the estimation of parameters (e.g., effect size). Cautions "If the government required statistical procedures to carry warning labels like those on drugs, most inference methods would have long labels indeed."[13] This caution applies to hypothesis tests and alternatives to them. The successful hypothesis test is associated with a probability and a type-I error rate. The conclusion might be wrong. The conclusion of the test is only as solid as the sample upon which it is based. The design of the experiment is critical. A number of unexpected effects have been observed including: • The Clever Hans effect. A horse appeared to be capable of doing simple arithmetic. • The Hawthorne effect. Industrial workers were more productive in better illumination, and most productive in worse. • The Placebo effect. Pills with no medically active ingredients were remarkably effective. A statistical analysis of misleading data produces misleading conclusions. The issue of data quality can be more subtle. In forecasting for example, there is no agreement on a measure of forecast accuracy. In the absence of a consensus measurement, no decision based on measurements will be without controversy. The book How to Lie with Statistics[14][15] is the most popular book on statistics ever published.[16] It does not much consider hypothesis testing, but its cautions are applicable, including: Many claims are made on the basis of samples too small to convince. If a report does not mention sample size, be doubtful. Hypothesis testing acts as a filter of statistical conclusions; Only those results meeting a probability threshold are publishable. Economics also acts as a publication filter; Only those results favorable to the author and funding source may be submitted for publication. The impact of filtering on publication is termed publication bias. A related problem is that of multiple testing (sometimes linked to data mining), in which a variety of tests for a variety of possible effects are applied to a single data set and only those yielding a significant result are reported. These are often dealt with by using multiplicity correction procedures that control the family wise error rate (FWER) or the Statistical hypothesis testing 55 false discovery rate (FDR). Those making critical decisions based on the results of a hypothesis test are prudent to look at the details rather than the conclusion alone. In the physical sciences most results are fully accepted only when independently confirmed. The general advice concerning statistics is, "Figures never lie, but liars figure" (anonymous). Examples The following examples should solidify these ideas. Analogy – Courtroom trial A statistical test procedure is comparable to a criminal trial; a defendant is considered not guilty as long as his or her guilt is not proven. The prosecutor tries to prove the guilt of the defendant. Only when there is enough charging evidence the defendant is convicted. In the start of the procedure, there are two hypotheses : "the defendant is not guilty", and : "the defendant is guilty". The first one is called null hypothesis, and is for the time being accepted. The second one is called alternative (hypothesis). It is the hypothesis one hopes to support. The hypothesis of innocence is only rejected when an error is very unlikely, because one doesn't want to convict an innocent defendant. Such an error is called error of the first kind (i.e. the conviction of an innocent person), and the occurrence of this error is controlled to be rare. As a consequence of this asymmetric behaviour, the error of the second kind (acquitting a person who committed the crime), is often rather large. H0 is true H1 is true Truly not guilty Truly guilty Accept Null Right decision Wrong Hypothesis decision Acquittal Type II Error Reject Null Hypothesis Wrong decision Right decision Conviction Type I Error A criminal trial can be regarded as either or both of two decision processes: guilty vs not guilty or evidence vs a threshold ("beyond a reasonable doubt"). In one view, the defendant is judged; in the other view the performance of the prosecution (which bears the burden of proof) is judged. A hypothesis test can be regarded as either a judgment of a hypothesis or as a judgment of evidence. Example 1 – Philosopher's beans The following example was produced by a philosopher describing scientific methods generations before hypothesis testing was formalized and popularized.[17] Few beans of this handful are white. Most beans in this bag are white. Therefore: Probably, these beans were taken from another bag. This is an hypothetical [sic] inference. The beans in the bag are the population. The handful are the sample. The null hypothesis is that the sample originated from the population. The criterion for rejecting the null-hypothesis is the "obvious" difference in appearance (an informal difference in the mean). The interesting result is that consideration of a real population and a real sample produced an imaginary bag. The philosopher was considering logic rather than probability. To be a real statistical hypothesis test, this example requires the formalities of a probability calculation and a comparison of that probability to a standard. Statistical hypothesis testing 56 A simple generalization of the example considers a mixed bag of beans and a handful that contain either very few or very many white beans. The generalization considers both extremes. It requires more calculations and more comparisons to arrive at a formal answer, but the core philosophy is unchanged; If the composition of the handful is greatly different that of the bag, then the sample probably originated from another bag. The original example is termed a one-sided or a one-tailed test while the generalization is termed a two-sided or two-tailed test. Example 2 – Clairvoyant card game A person (the subject) is tested for clairvoyance. He is shown the reverse of a randomly chosen playing card 25 times and asked which of the four suits it belongs to. The number of hits, or correct answers, is called X. As we try to find evidence of his clairvoyance, for the time being the null hypothesis is that the person is not clairvoyant. The alternative is, of course: the person is (more or less) clairvoyant. If the null hypothesis is valid, the only thing the test person can do is guess. For every card, the probability (relative frequency) of any single suit appearing is 1/4. If the alternative is valid, the test subject will predict the suit correctly with probability greater than 1/4. We will call the probability of guessing correctly p. The hypotheses, then, are: • null hypothesis (just guessing) and • alternative hypothesis (true clairvoyant). When the test subject correctly predicts all 25 cards, we will consider him clairvoyant, and reject the null hypothesis. Thus also with 24 or 23 hits. With only 5 or 6 hits, on the other hand, there is no cause to consider him so. But what about 12 hits, or 17 hits? What is the critical number, c, of hits, at which point we consider the subject to be clairvoyant? How do we determine the critical value c? It is obvious that with the choice c=25 (i.e. we only accept clairvoyance when all cards are predicted correctly) we're more critical than with c=10. In the first case almost no test subjects will be recognized to be clairvoyant, in the second case, a certain number will pass the test. In practice, one decides how critical one will be. That is, one decides how often one accepts an error of the first kind – a false positive, or Type I error. With c = 25 the probability of such an error is: and hence, very small. The probability of a false positive is the probability of randomly guessing correctly all 25 times. Being less critical, with c=10, gives: Thus, c = 10 yields a much greater probability of false positive. Before the test is actually performed, the maximum acceptable probability of a Type I error (α) is determined. Typically, values in the range of 1% to 5% are selected. (If the maximum acceptable error rate is zero, an infinite number of correct guesses is required.) Depending on this Type 1 error rate, the critical value c is calculated. For example, if we select an error rate of 1%, c is calculated thus: From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of a Type II error, a false negative. For the above example, we select: . Statistical hypothesis testing 57 Example 3 – Radioactive suitcase As an example, consider determining whether a suitcase contains some radioactive material. Placed under a Geiger counter, it produces 10 counts per minute. The null hypothesis is that no radioactive material is in the suitcase and that all measured counts are due to ambient radioactivity typical of the surrounding air and harmless objects. We can then calculate how likely it is that we would observe 10 counts per minute if the null hypothesis were true. If the null hypothesis predicts (say) on average 9 counts per minute and a standard deviation of 1 count per minute, then we say that the suitcase is compatible with the null hypothesis (this does not guarantee that there is no radioactive material, just that we don't have enough evidence to suggest there is). On the other hand, if the null hypothesis predicts 3 counts per minute and a standard deviation of 1 count per minute, then the suitcase is not compatible with the null hypothesis, and there are likely other factors responsible to produce the measurements. The test does not directly assert the presence of radioactive material. A successful test asserts that the claim of no radioactive material present is unlikely given the reading (and therefore ...). The double negative (disproving the null hypothesis) of the method is confusing, but using a counter-example to disprove is standard mathematical practice. The attraction of the method is its practicality. We know (from experience) the expected range of counts with only ambient radioactivity present, so we can say that a measurement is unusually large. Statistics just formalizes the intuitive by using numbers instead of adjectives. We probably do not know the characteristics of the radioactive suitcases; We just assume that they produce larger readings. To slightly formalize intuition: Radioactivity is suspected if the Geiger-count with the suitcase is among or exceeds the greatest (5% or 1%) of the Geiger-counts made with ambient radiation alone. This makes no assumptions about the distribution of counts. Many ambient radiation observations are required to obtain good probability estimates for rare events. The test described here is more fully the null-hypothesis statistical significance test. The null hypothesis represents what we would believe by default, before seeing any evidence. Statistical significance is a possible finding of the test, declared when the observed sample is unlikely to have occurred by chance if the null hypothesis were true. The name of the test describes its formulation and its possible outcome. One characteristic of the test is its crisp decision: to reject or not reject the null hypothesis. A calculated value is compared to a threshold, which is determined from the tolerable risk of error. Definition of terms The following definitions are mainly based on the exposition in the book by Lehmann and Romano:[6] Statistical hypothesis A statement about the parameters describing a population (not a sample). Statistic A value calculated from a sample, often to summarize the sample for comparison purposes. Simple hypothesis Any hypothesis which specifies the population distribution completely. Composite hypothesis Any hypothesis which does not specify the population distribution completely. Null hypothesis (H0) A simple hypothesis associated with a contradiction to a theory one would like to prove. Alternative hypothesis (H1) A hypothesis (often composite) associated with a theory one would like to prove. Statistical test Statistical hypothesis testing 58 A procedure whose inputs are samples and whose result is a hypothesis. Region of acceptance The set of values of the test statistic for which we fail to reject the null hypothesis. Region of rejection / Critical region The set of values of the test statistic for which the null hypothesis is rejected. Critical value The threshold value delimiting the regions of acceptance and rejection for the test statistic. Power of a test (1 − β) The test's probability of correctly rejecting the null hypothesis. The complement of the false negative rate, β. Power is termed sensitivity in biostatistics. ("This is a sensitive test. Because the result is negative, we can confidently say that the patient does not have the condition.") See sensitivity and specificity and Type I and type II errors for exhaustive definitions. Size / Significance level of a test (α) For simple hypotheses, this is the test's probability of incorrectly rejecting the null hypothesis. The false positive rate. For composite hypotheses this is the upper bound of the probability of rejecting the null hypothesis over all cases covered by the null hypothesis. The complement of the false positive rate, (1 − α), is termed specificity in biostatistics. ("This is a specific test. Because the result is positive, we can confidently say that the patient has the condition.") See sensitivity and specificity and Type I and type II errors for exhaustive definitions. p-value The probability, assuming the null hypothesis is true, of observing a result at least as extreme as the test statistic. Statistical significance test A predecessor to the statistical hypothesis test (see the Origins section). An experimental result was said to be statistically significant if a sample was sufficiently inconsistent with the (null) hypothesis. This was variously considered common sense, a pragmatic heuristic for identifying meaningful experimental results, a convention establishing a threshold of statistical evidence or a method for drawing conclusions from data. The statistical hypothesis test added mathematical rigor and philosophical consistency to the concept by making the alternative hypothesis explicit. The term is loosely used to describe the modern version which is now part of statistical hypothesis testing. Conservative test A test is conservative if, when constructed for a given nominal significance level, the true probability of incorrectly rejecting the null hypothesis is never greater than the nominal level. Exact test A test in which the significance level or critical value can be computed exactly, i.e., without any approximation. In some contexts this term is restricted to tests applied to categorical data and to permutation tests, in which computations are carried out by complete enumeration of all possible outcomes and their probabilities. A statistical hypothesis test compares a test statistic (z or t for examples) to a threshold. The test statistic (the formula found in the table below) is based on optimality. For a fixed level of Type I error rate, use of these statistics minimizes Type II error rates (equivalent to maximizing power). The following terms describe tests in terms of such optimality: Most powerful test Statistical hypothesis testing 59 For a given size or significance level, the test with the greatest power (probability of rejection) for a given value of the parameter(s) being tested, contained in the alternative hypothesis. Uniformly most powerful test (UMP) A test with the greatest power for all values of the parameter(s) being tested, contained in the alternative hypothesis. Common test statistics One-sample tests are appropriate when a sample is being compared to the population from a hypothesis. The population characteristics are known from theory or are calculated from the population. Two-sample tests are appropriate for comparing two samples, typically experimental and control samples from a scientifically controlled experiment. Paired tests are appropriate for comparing two samples where it is impossible to control important variables. Rather than comparing two sets, members are paired between samples so the difference between the members becomes the sample. Typically the mean of the differences is then compared to zero. Z-tests are appropriate for comparing means under stringent conditions regarding normality and a known standard deviation. T-tests are appropriate for comparing means under relaxed conditions (less is assumed). Tests of proportions are analogous to tests of means (the 50% proportion). Chi-squared tests use the same calculations and the same probability distribution for different applications: • Chi-squared tests for variance are used to determine whether a normal population has a specified variance. The null hypothesis is that it does. • Chi-squared tests of independence are used for deciding whether two variables are associated or are independent. The variables are categorical rather than numeric. It can be used to decide whether left-handedness is correlated with libertarian politics (or not). The null hypothesis is that the variables are independent. The numbers used in the calculation are the observed and expected frequencies of occurrence (from contingency tables). • Chi-squared goodness of fit tests are used to determine the adequacy of curves fit to data. The null hypothesis is that the curve fit is adequate. It is common to determine curve shapes to minimize the mean square error, so it is appropriate that the goodness-of-fit calculation sums the squared errors. F-tests (analysis of variance, ANOVA) are commonly used when deciding whether groupings of data by category are meaningful. If the variance of test scores of the left-handed in a class is much smaller than the variance of the whole class, then it may be useful to study lefties as a group. The null hypothesis is that two variances are the same - so the proposed grouping is not meaningful. In the table below, the symbols used are defined at the bottom of the table. Many other tests can be found in other articles. Statistical hypothesis testing 60 Name Formula Assumptions or notes One-sample z-test (Normal population or n > 30) and σ known. (z is the distance from the mean in relation to the standard deviation of the mean). For non-normal distributions it is possible to calculate a minimum proportion of a population that falls within k standard deviations for any k (see: Chebyshev's inequality). Two-sample z-test Normal population and independent observations and σ1 and σ2 are known One-sample t-test (Normal population or n > 30) and unknown Paired t-test (Normal population of differences or n > 30) and unknown or small sample size n < 30 Two-sample pooled (Normal populations or n1 + n2 > 40) and independent observations and σ1 t-test, equal variances = σ2 unknown [18] Two-sample (Normal populations or n1 + n2 > 40) and independent observations and σ1 unpooled t-test, ≠ σ2 both unknown unequal variances [18] One-proportion z-test n .p0 > 10 and n (1 − p0) > 10 and it is a SRS (Simple Random Sample), see notes. Two-proportion n1 p1 > 5 and n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and z-test, pooled for independent observations, see notes. Two-proportion n1 p1 > 5 and n1(1 − p1) > 5 and n2 p2 > 5 and n2(1 − p2) > 5 and z-test, unpooled for independent observations, see notes. Chi-squared test for Normal population variance Chi-squared test for df = k - 1 - # parameters estimated, and one of these must hold. • All goodness of fit [19] expected counts are at least 5. • All expected counts are > 1 and no more than 20% of expected counts are [20] less than 5 Two-sample F test Normal populations for equality of Arrange so and reject H0 for variances [21] In general, the subscript 0 indicates a value taken from the null hypothesis, H0, which should be used as much as possible in constructing its test statistic. ... Definitions of other symbols: Statistical hypothesis testing 61 • , the probability of Type I error (rejecting a null • = sample variance • = x/n = sample proportion, unless hypothesis when it is in fact true) specified otherwise • = sample size • = sample 1 standard deviation • = hypothesized population proportion • = sample 1 size • = sample 2 standard deviation • = proportion 1 • = sample 2 size • = t statistic • = proportion 2 • = sample mean • = degrees of freedom • = hypothesized difference in proportion • = hypothesized population mean • = sample mean of differences • = minimum of n1 and n2 • = population 1 mean • = hypothesized population • mean difference • = population 2 mean • = standard deviation of • differences • = population standard deviation • = Chi-squared statistic • = F statistic • = population variance • = sample standard deviation • = sum (of k numbers) Statistical hypothesis testing 62 Origins and Early Controversy Hypothesis testing is largely the product of Ronald Fisher, and was further developed by Jerzy Neyman, Karl Pearson and (son) Egon Pearson. Ronald Fisher, genius mathematician and biologist described by Richard Dawkins as "the greatest biologist since Darwin", began his life in statistics as a Bayesian (Zabell 1992), but Fisher soon grew disenchanted with the subjectivity involved, and sought to provide a more "objective" approach to inductive inference.[22] According to Stephen M. Stigler: "Fisher was an agricultural statistician who emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an inconsistent hybrid of the Fisher vs Neyman/Pearson formulation, methods and terminology developed in the early 20th century. While hypothesis testing was popularized early in the 20th century, evidence of its use can be found much earlier. In the 1770s Laplace considered the statistics of almost half a million births. The statistics showed an excess of boys compared to girls. He concluded by calculation of a p-value that the excess was a real, but unexplained, effect."[23] Fisher popularized the "significance test". He required a null-hypothesis (corresponding to a population frequency distribution) and a sample. His (now familiar) calculations determined whether to reject the null-hypothesis or not. The likely originator of the Significance testing did not utilize an alternative hypothesis so there was no confused yet widespread "hybrid" concept of a Type II error. method of hypothesis testing, as well as the use of "nil" null The p-value was devised as an informal, but objective, index meant to help a hypotheses, is E.F. Lindquist in researcher determine (based on other knowledge) whether to modify future his statistics textbook: Lindquist, experiments or strengthen one's faith in the null hypothesis.[24] Hypothesis testing E.F. (1940) Statistical Analysis In (and Type I/II errors) were devised by Neyman and Pearson as a more objective Educational Research. Boston: Houghton Mifflin. alternative to Fisher's p-value, also meant to determine researcher behaviour, but without requiring any inductive inference by the researcher.[25][26] Neither strategy is meant to provide any way of drawing conclusions from a single experiment.[25][27] Both strategies were meant to assess the results of experiments that were replicated multiple times.[28] Neyman & Pearson considered a different problem (which they called "hypothesis testing"). They initially considered two simple hypotheses (both with frequency distributions). They calculated two probabilities and typically selected the hypothesis associated with the higher probability (the hypothesis more likely to have generated the sample). Their method always selected a hypothesis. It also allowed the calculation of both types of error probabilities. Fisher and Neyman/Pearson clashed bitterly. Neyman/Pearson considered their formulation to be an improved generalization of significance testing.(The defining paper was abstract. Mathematicians have generalized and refined the theory for three generations.) Fisher thought that it was not applicable to scientific research because often, during the course of the experiment, it is discovered that the initial assumptions about the null hypothesis are questionable due to unexpected sources of error. He believed that the use of rigid reject/accept decisions based on models formulated before data is collected was incompatible with this common scenario faced by scientists and attempts to apply this method to scientific research would lead to mass confusion. Statistical hypothesis testing 63 Fisher wrote in 1955 that "the conclusions drawn by a scientific worker from a test of significance are provisional, and involve an intelligent attempt to understand the experimental situation," [24] and later in 1958, "We are quite in danger of sending highly-trained and highly intelligent young men out into the world with tables of erroneous numbers under their arms, and with a dense fog in the place where their brains ought to be. In this century, of course, they will be working on guided missiles and advising the medical profession on the control of disease, and there is no limit to the extent to which they could impede every sort of national effort."[29] Further, Fisher taught that his method was only applicable if little was known about the problem at hand and "no relevant sub-set [of the population] should be recognizable". He wrote that if "there were knowledge a priori of the distribution of [the population mean]... the method of Bayes would give a probability statement...[that] would supersede the fiducial value [the result of a significance test]... therefore the first condition [of significance testing] is that there should be no knowledge a priori." [29] In the same paper regarding the use of statistics in science Fisher complained: "it would be a mistake to think that mathematicians as such are particularly good at the inductive logical processes which are needed in improving or knowledge of the natural world" To clarify Fisher's view on on the use of Bayesian probability for scientific inference: He often praised Thomas Bayes for his insight into the special case in which one has prior information about the problem at hand, but he believed that "the essential information on which Bayes' theorem is based" [a prior probability distribution], was often "in reality absent",[30] and thus the theorem was inapplicable. Early writers on probability (especially Pierre Laplace) advocated use of the "principle of indifference" under these circumstances, which (in the opinion of Fisher, arbitrarily) claims that in the absence of contrary information one should assume all possible results are equally probable (i.e. use a uniform prior distribution). He saw such an assumption as completely unjustified, leading to statements such as: "Perhaps we were lucky in England in having the whole mass of fallacious rubbish put out of sight until we had time to think about probability in concrete terms and in relation, above all, to the purposes for which we wanted the idea in the natural sciences."[29] The modern version of hypothesis testing is a hybrid of the two approaches that resulted from confusion by writers of statistical textbooks (as predicted by Fisher) beginning in the 1940s.[31] (But signal detection, for example, still uses the Neyman/Pearson formulation.) Great conceptual differences and many caveats in addition to those mentioned above were ignored. Neyman and Pearson provided the stronger terminology, the more rigorous mathematics and the more consistent philosophy, but the subject taught today in introductory statistics has more similarities with Fisher's method than theirs.[32] This history explains the inconsistent terminology (example: the null hypothesis is never accepted, but there is a region of acceptance). Sometime around 1940,[31] in an apparent effort to provide researchers with a "non-controversial"[33] way to have their cake and eat it too, the authors of statistical text books began anonymously combining these two strategies by using the p-value in place of the test statistic (or data) to test against the Neyman-Pearson "significance level".[31] Thus, researchers were encouraged to infer the strength of their data against some null hypothesis using p-values, while also thinking they are retaining the post-data collection objectivity provided by hypothesis testing. It then became customary for the null hypothesis, which was originally some realistic research hypothesis, to be used almost solely as a strawman "nil" hypothesis (one where a treatment has no effect, regardless of the context).[34] A comparison between Fisherian, frequentist (Neyman-Pearson). Statistical hypothesis testing 64 Fisher’s null hypothesis testing Neyman–Pearson decision theory 1. Set up a statistical null hypothesis. The null need not be a 1. Set up two statistical hypotheses, H1 and H2, and decide about α, β, and sample nil hypothesis (i.e., zero difference). size before the experiment, based on subjective cost-benefit considerations. These define a rejection region for each hypothesis. 2. Report the exact level of significance (e.g., p = 0.051 or p = 2. If the data falls into the rejection region of H1, accept H2; otherwise accept H1. 0.049). Do not use a conventional 5% level, and do not talk Note that accepting a hypothesis does not mean that you believe in it, but only that about accepting or rejecting hypotheses. you act as if it were true. 3. Use this procedure only if little is known about the problem 3. The usefulness of the procedure is limited among others to situations where you at hand, and only to draw provisional conclusions in the have a disjunction of hypotheses (e.g., either μ1 =8 or μ2 = 10 is true) and where context of an attempt to understand the experimental you can make meaningful cost-benefit trade-offs for choosing alpha and beta. situation. Ongoing Controversy Given the prevalence of significance testing in published research literature, as well as the emergence of alternative approaches (namely, Bayesian statistics), hypothesis testing has been the subject of ongoing debate since its initial formulations by Fisher and Neyman/Pearson. The volume of criticism and rebuttal has filled books with language seldom used in the scholarly debate of a dry subject.[35][36][37][38] Nickerson surveyed the issues in the year 2000.[39] He included 300 references and reported 20 criticisms and almost as many recommendations, alternatives and supplements. Authors of statistical textbooks have largely ignored this controversy,.[40] According to psychologist Gerd Gigerenzer there is evidence that the majority of psychologists teaching statistics might not understand the nature of a p-value[41] and may be unaware of the controversy. According to Hubbard and Bayarri, researchers frequently misinterpret the p-value, viewing it "simultaneously in Neyman–Pearson terms as a deductive assessment of error in long-run repeated sampling situations, and in a Fisherian sense as a measure of inductive evidence in a single study. In fact, a p value from a significance test has no place in the Neyman–Pearson hypothesis testing framework. Contrary to popular misconception, p’s and α’s are not the same thing; they measure different concepts.[42] One set of criticisms focuses on the logic underlying hypothesis testing. Null hypothesis significance testing, as commonly performed, is neither based on a Fisherian strategy of inductive reasoning nor the Neyman-Pearson strategy of inductive behaviour. Although both strategies have merit on their own, the current practice has arisen due an erroneous combination of the two strategies that has no logical or mathematical basis.[43] Several criticisms of hypothesis testing relate to how the technique is commonly used in practice (rather than the fundamental statistical or philosophical implications of hypothesis testing per se). Some of these criticisms therefore aim to influence statistics education as much as critique the method. According to Cohen, "we, as teachers, consultants, authors, and otherwise perpetrators of quantitative methods, are responsible for the ritualization of null hypothesis significance testing (NHST, I resisted the temptation to call it statistical hypothesis inference testing) to the point of meaninglessness and beyond. I argue herein that NHST has not only failed to support the advance of psychology as a science but also has seriously impeded it [...] What's wrong with NHST? Well, among many other things, it does not tell us what we want to know," (the probability a hypothesis is true) "and we so much want to know what we want to know that, out of desperation, we nevertheless believe that it does!"[44] Importantly, the choice of one-tailed or two-tailed hypothesis tests, as well as the choice of significance level, can be seen as a subjective choice by the researcher, potentially to lower (or raise) the threshold for finding significant test statistics.[45] Often the arbitrary significance level of .05 is chosen because it is commonly used in published research, and it is commonly used because it is often chosen. Although this is not a problem with significance testing per se. One related view is that proper use of hypothesis testing requires penalizing researchers for making multiple comparisons (i.e., conducting multiple, distinct hypothesis tests on the same data).[46][47] As with any type of data analysis, data mining can introduce problems. Statistical hypothesis testing 65 Additionally, many commonly used tests (e.g., ANOVA) require potentially implausible assumptions about characteristics of the sample and/or population (e.g. normality, equal Ns, sphericity, equal variances), or require complicated adjustments to account for violations of these assumptions that have only been validated under a small subset of possible types of violations.[48] In practice, it is sometimes difficult to verify that these assumptions hold (e.g., in the case of small sample size) and multiple violations (e.g., of normality and homoscedasticity) may interact unpredictably. By contrast, nonparametric statistics aims to minimize the necessary assumptions about the distributions of test statistics, while Bayesian statistics simply makes distributional assumptions more explicit rather than eliminating them. As with Bayesian methods, the result of a significance test depends in part on the sample size used and the precision of measurement.[49] Accordingly, very large samples are more likely to produce significant results just as they would dramatically influence the shape of the posterior distribution in a Bayesian analysis. Another set of concerns relates to the interpretation of statistical hypothesis tests. As one example, concerns have been expressed about the prevalence of "no effect" null hypotheses (e.g., those where any difference between two group means is interpreted as meaningful) rather than hypothesis tests with larger or smaller null values.[50] There is no statistical requirement that a null hypothesis be "no effect," but this is the default setting in most statistical software (and consequently, most common in practice). Using "no effect" as a default could be considered a strawman.[51] According to Savage (1957), "Null hypotheses of no difference are usually known to be false before the data are collected; when they are, their rejection or acceptance simply reflects the size of the sample and the power of the test, and is not a contribution to science."[49] Because hypothesis testing looks only at the sample data and typically incorporates no other information, statistically significant results may be substantively unimportant (e.g., detecting an effect known to exist due to considerable previous research). However, hypothesis tests can be constructed such that a null hypothesis reflects the results of prior findings, thereby testing whether the new data suggest an effect different from that found before. And, meta-analysis can be used to aggregate the results of multiple studies to draw inferences about the distribution of effects. In general, statistical significance does not imply practical significance, in that a statistically significant difference may have a trivial effect size, especially when using large samples. For example, in one NAEP analysis of 100,000 students, male science test scores were found to be significantly higher than those of females,[52] suggesting that there is a small difference between males and females, but the substantive size of that difference is trivial. Indeed, in this example, the mean difference was only 4 out of 300 points,[53] implying heavily overlapped distributions of scores, illustrating that differences in expected values of distributions may overshadow the shape of those distributions relative to one another. In this example, a hypothesis test could have been constructed to look for an effect size of substantive (rather than just statistical) significance. Part of the concern with hypothesis testing relates to how much emphasis is placed on p-values. P-values are widely interpreted as error rates, evidence against the null hypothesis, or even evidence for the research hypothesis. These misconceptions continue to persist despite decades of effort to eradicate them.[54] When interpreted in the common, psuedo-Fisherian sense, P-values can overstate the evidence against the null by an order of magnitude[55][56] Researchers' focus on p-values can be particularly problematic as it relates to the publication of research findings, particularly when statistical significance is used as a criterion for publication, which can yield publication bias. When publication is contingent on p-values, several problems emerge. Published Type I errors are difficult to correct. Published effect sizes are also biased upward (because achieving statistical significance requires effect sizes that are sufficiently large).[57] Type II errors (false negatives) might also be common due to insufficient power to detect small effects; but both Bayesian methods and meta-analysis can address this problem when the results of several underpowered studies are available. A focus on p-values also emerges in other areas. For example, some countries publish unusually high proportions of positive results.[58] and ‘‘positive’’ results increase down the hierarchy of the sciences.[59] Some research suggests that negative results are disappearing from most disciplines and countries[60] due to significance being tied to the publishability of research. Flexibility in data collection, analysis, and reporting can increase false-positive rates.[61] Statistical hypothesis testing 66 P-values in the psychology literature are much more common immediately below .05 than expected.[62] Controversy over significance testing, and its effects on publication bias in particular, has produced several results.[41] The American Psychological Association has strengthened its statistical reporting requirements after review,[63] medical journal publishers have recognized the obligation to publish some results that are not statistically significant to combat publication bias[64] and a journal (Journal of Articles in Support of the Null Hypothesis) has been created to publish such results exclusively.[65] Textbooks have added some cautions[66] and increased coverage of the tools necessary to estimate the size of the sample required to produce significant results. Major organizations have not abandoned use of significance tests although some have discussed doing so.[63] Alternative to significance testing However, the numerous criticisms of significance testing do not lead to a single alternative or even to a unified set of alternatives. A unifying position of critics is that statistics should not lead to a conclusion or a decision but to a probability or to an estimated value with a confidence interval rather than to an accept-reject decision regarding a particular hypothesis. There is some consensus that the hybrid testing procedure[67] that is commonly used is fundamentally flawed. It is unlikely that the controversy surrounding significance testing will be resolved in the near future. Its supposed flaws and unpopularity do not eliminate the need for an objective and transparent means of reaching conclusions regarding studies that produce statistical results. Critics have not unified around an alternative. Some of them have, however, suggested reforms for statistical and marketing research education to include a more thorough analysis of the meaning of statistical significance.[68] Other forms of reporting confidence or uncertainty could probably grow in popularity. Some recent work includes reconstruction and defense of Neyman–Pearson testing.[67] One strong critic of significance testing suggested a list of reporting alternatives:[69] effect sizes for importance, prediction intervals for confidence, replications and extensions for replicability, meta-analyses for generality. None of these suggested alternatives produces a conclusion/decision. Lehmann said that hypothesis testing theory can be presented in terms of conclusions/decisions, probabilities, or confidence intervals. "The distinction between the ... approaches is largely one of reporting and interpretation."[70] On one "alternative" there is no disagreement: Fisher himself said,[4] "In relation to the test of significance, we may say that a phenomenon is experimentally demonstrable when we know how to conduct an experiment which will rarely fail to give us a statistically significant result." Cohen, an influential critic of significance testing, concurred,[71] "...don't look for a magic alternative to NHST [null hypothesis significance testing] ... It doesn't exist." "...given the problems of statistical induction, we must finally rely, as have the older sciences, on replication." The "alternative" to significance testing is repeated testing. The easiest way to decrease statistical uncertainty is by obtaining more data, whether by increased sample size or by repeated tests. Nickerson claimed to have never seen the publication of a literally replicated experiment in psychology.[39] However, an indirect approach to replication is meta-analysis. Bayesian inference is one alternative to significance testing. For example, Bayesian parameter estimation can provide rich information about the data from which researchers can draw inferences, while using uncertain priors that exert only minimal influence on the results when enough data is available. Psychologist Kruschke, John K. has suggested Bayesian estimation as an alternative for the t-test.[72] Alternatively two competing models/hypothesis can be compared using Bayes factors.[73] Bayesian methods could be criticized for requiring information that is seldom available in the cases where significance testing is most heavily used. Advocates of a Bayesian approach sometimes claim that the goal of a researcher is most often to objectively assess the probability that a hypothesis is true based on the data they have collected. Neither Fisher's significance testing, nor Neyman-Pearson hypothesis testing can provide this information, and do not claim to. The probability a hypothesis is true can only be derived from use of Bayes' Theorem, which was unsatisfactory to both the Fisher and Neyman-Pearson camps due to the explicit use of subjectivity in the form of the prior probability.[25][74] Fisher's Statistical hypothesis testing 67 strategy is to sidestep this with the p-value (an objective index based on the data alone) followed by inductive inference, while Neyman-Pearson devised their approach of inductive behaviour. Education Statistics is increasingly being taught in schools with hypothesis testing being one of the elements taught.[75][76] Many conclusions reported in the popular press (political opinion polls to medical studies) are based on statistics. An informed public should understand the limitations of statistical conclusions[77][78] and many college fields of study require a course in statistics for the same reason.[77][78] An introductory college statistics class places much emphasis on hypothesis testing - perhaps half of the course. Such fields as literature and divinity now include findings based on statistical analysis (see the Bible Analyzer). An introductory statistics class teaches hypothesis testing as a cookbook process. Hypothesis testing is also taught at the postgraduate level. Statisticians learn how to create good statistical test procedures (like z, Student's t, F and chi-squared). Statistical hypothesis testing is considered a mature area within statistics,[79] but a limited amount of development continues. References [1] R. A. Fisher (1925).Statistical Methods for Research Workers, Edinburgh: Oliver and Boyd, 1925, p.43. [2] Schervish, M (1996) Theory of Statistics, p. 218. Springer ISBN 0-387-94546-6 [3] Kaye, David H.; Freedman, David A. (2011). "Reference Guide on Statistics" (http:/ / www. nap. edu/ openbook. php?record_id=13163& page=211). Reference manual on scientific evidence (3rd ed.). Eagan, MN Washington, D.C: West National Academies Press. p. 259. ISBN 978-0-309-21421-6. . [4] Fisher, Sir Ronald A. (1956) [1935]. "Mathematics of a Lady Tasting Tea" (http:/ / books. google. com/ ?id=oKZwtLQTmNAC& pg=PA1512& dq="mathematics+ of+ a+ lady+ tasting+ tea"). In James Roy Newman. The World of Mathematics, volume 3 [Design of Experiments]. Courier Dover Publications. ISBN 978-0-486-41151-4. . Originally from Fisher's book Design of Experiments. [5] Box, Joan Fisher (1978). R.A. Fisher, The Life of a Scientist. New York: Wiley. p. 134. ISBN 0-471-09300-9. [6] Lehmann, E.L.; Romano, Joseph P. (2005). Testing Statistical Hypotheses (3E ed.). New York: Springer. ISBN 0-387-98864-5. [7] Adèr,J.H. (2008). Chapter 12: Modelling. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research Methods: A consultant's companion (pp. 183–209). Huizen, The Netherlands: Johannes van Kessel Publishing [8] Triola, Mario (2001). Elementary statistics (8 ed.). Boston: Addison-Wesley. p. 388. ISBN 0-201-61477-4. [9] Hinkelmann, Klaus and Kempthorne, Oscar (2008). Design and Analysis of Experiments. I and II (Second ed.). Wiley. ISBN 978-0-470-38551-7. [10] Montgomery, Douglas (2009). Design and analysis of experiments. Hoboken, NJ: Wiley. ISBN 978-0-470-12866-4. [11] Richard J. Larsen, Donna Fox Stroup (1976). Statistics in the Real World. [12] Hubbard, R.; Parsa, A. R.; Luthy, M. R. (1997). "The spread of statistical significance testing in psychology: The case of the Journal of Applied Psychology". Theory and Psychology 7 (4): 545–554. doi:10.1177/0959354397074006. [13] Moore, David (2003). Introduction to the practice of statistics. New York: W.H. Freeman and Co. p. 426. ISBN 9780716796572. [14] Huff, Darrell (1993). How to lie with statistics. New York: Norton. ISBN 0-393-31072-8. [15] Huff, Darrell (1991). How to lie with statistics. London: Penguin Books. ISBN 0-14-013629-0. [16] "Over the last fifty years, How to Lie with Statistics has sold more copies than any other statistical text." J. M. Steele. " Darrell Huff and Fifty Years of How to Lie with Statistics (http:/ / www-stat. wharton. upenn. edu/ ~steele/ Publications/ PDF/ TN148. pdf). Statistical Science, 20 (3), 2005, 205–209. [17] C. S. Peirce (August 1878). "Illustrations of the Logic of Science VI: Deduction, Induction, and Hypothesis" (http:/ / en. wikisource. org/ w/ index. php?oldid=3592335). Popular Science Monthly 13. . Retrieved 30 March 2012. [18] NIST handbook: Two-Sample t-Test for Equal Means (http:/ / www. itl. nist. gov/ div898/ handbook/ eda/ section3/ eda353. htm) [19] Steel, R.G.D, and Torrie, J. H., Principles and Procedures of Statistics with Special Reference to the Biological Sciences., McGraw Hill, 1960, page 350. [20] Weiss, Neil A. (1999). Introductory Statistics (5th ed.). pp. 802. ISBN 0-201-59877-9. [21] NIST handbook: F-Test for Equality of Two Standard Deviations (http:/ / www. itl. nist. gov/ div898/ handbook/ eda/ section3/ eda359. htm) (Testing standard deviations the same as testing variances) [22] Raymond Hubbard, M.J. Bayarri, P Values are not Error Probabilities (http:/ / ftp. isds. duke. edu/ WorkingPapers/ 03-26. pdf). A working paper that explains the difference between Fisher's evidential p-value and the Neyman–Pearson Type I error rate . [23] Stigler, Stephen M. (1986). The history of statistics : the measurement of uncertainty before 1900. Cambridge, Mass: Belknap Press of Harvard University Press. p. 134. ISBN 0-674-40340-1. [24] Fisher, R (1955). "Statistical Methods and Scientific Induction" (http:/ / www. phil. vt. edu/ dmayo/ PhilStatistics/ Triad/ Fisher 1955. pdf). Journal of the Royal Statistical Society. Series B (Methodological) 17 (1): 69–78. . Statistical hypothesis testing 68 [25] Neyman, J (January 1, 1933). "On the Problem of the most Efficient Tests of Statistical Hypotheses.". Phil. Trans. R. Soc. Lond. A 231 (694-706): 289–337. doi:10.1098/rsta.1933.0009. [26] Goodman, S N (June 15, 1999). "Toward evidence-based medical statistics. 1: The P Value Fallacy." (http:/ / annals. org/ article. aspx?articleid=712762). Ann Intern Med 130 (12): 995–1004. . [27] Fisher, R A (1926). "The Arrangement of Field Experiments." (http:/ / digital. library. adelaide. edu. au/ dspace/ bitstream/ 2440/ 15191/ 1/ 48. pdf). Journal of the Ministry of Agriculture of Great Britain 33 (N/A): 503–513. . [28] Sterne, J A C (August 2001). "Sifting the evidence--what's wrong with significance tests?" (http:/ / www. physther. org/ content/ 81/ 8/ 1464. full. pdf+ html). Physical Therapy 81 (8): 1464–1469. . [29] Fisher, R (1958). "The Nature of Probability" (http:/ / www. york. ac. uk/ depts/ maths/ histstat/ fisher272. pdf). Centennial Review 2 (N/A): 261–274. . [30] Fisher, R (1930). "Inverse Probability" (http:/ / www. cse. buffalo. edu/ ~peter/ refs/ Historical/ Fisher_1930. pdf). Proceedings of the Cambridge Philosophical Society 26 (N/A): 528–535. . [31] Halpin, P F (Winter 2006). "Inductive Inference or Inductive Behavior: Fisher and Neyman: Pearson Approaches to Statistical Testing in Psychological Research (1940-1960)" (http:/ / www. jstor. org/ stable/ 20445367). The American Journal of Psychology 119 (4): 625–653. . [32] Gigerenzer, Gerd; Zeno Swijtink, Theodore Porter, Lorraine Daston, John Beatty, Lorenz Kruger (1990). "Part 3: The inference experts". The Empire of Chance: How Probability Changed Science and Everyday Life. Cambridge University Press. pp. 70–122. ISBN 978-0-521-39838-1. [33] Gigerenzer, G (November 2004). "Mindless statistics". The Journal of Socio-Economics 33 (5): 587–606. doi:10.1016/j.socec.2004.09.033. [34] Loftus, G R (1991). "On the Tyranny of Hypothesis Testing in the Social Sciences" (https:/ / www. ics. uci. edu/ ~sternh/ courses/ 210/ loftus91_tyranny. pdf). Contemporary Psychology 36 (2): 102–105. . [35] Harlow, Lisa Lavoie; Stanley A. Mulaik; James H. Steiger, ed. (1997). What If There Were No Significance Tests?. Lawrence Erlbaum Associates. ISBN 978-0-8058-2634-0. [36] Morrison, Denton; Henkel, Ramon, ed. (2006) [1970]. The Significance Test Controversy. AldineTransaction. ISBN 0-202-30879-0. [37] McCloskey, Deirdre N.; Stephen T. Ziliak (2008). The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. University of Michigan Press. ISBN 0-472-05007-9. [38] Chow, Siu L. (1997). Statistical Significance: Rationale, Validity and Utility. ISBN 0-7619-5205-5. [39] Nickerson, Raymond S. (2000). "Null Hypothesis Significance Tests: A Review of an Old and Continuing Controversy". Psychological Methods 5 (2): 241–301. doi:10.1037/1082-989X.5.2.241. PMID 10937333. [40] Gliner, J A (2002). "Problems With Null Hypothesis Significance Testing (NHST): What Do the Textbooks Say?". The Journal of Experimental Education 71 (1): 83–92. doi:10.1080/00220970209602058. [41] Gigerenzer, G (2004). "The Null Ritual What You Always Wanted to Know About Significant Testing but Were Afraid to Ask" (http:/ / library. mpib-berlin. mpg. de/ ft/ gg/ GG_Null_2004. pdf). The SAGE Handbook of Quantitative Methodology for the Social Sciences N/A (N/A): 391–408. doi:10.4135/9781412986311. . [42] Raymond Hubbard, M.J. Bayarri, P Values are not Error Probabilities (http:/ / ftp. isds. duke. edu/ WorkingPapers/ 03-26. pdf). A working paper that explains the difference between Fisher's evidential p-value and the Neyman–Pearson Type I error rate . [43] Raymond Hubbard, M.J. Bayarri, P Values are not Error Probabilities (http:/ / ftp. isds. duke. edu/ WorkingPapers/ 03-26. pdf). A working paper that explains the difference between Fisher's evidential p-value and the Neyman–Pearson Type I error rate . [44] Cohen, J (December 1994). "The Earth Is Round (p < .05)" (http:/ / ist-socrates. berkeley. edu/ ~maccoun/ PP279_Cohen1. pdf). American Psychologist 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997. . [45] Bakan, D (December 1966). "The Test of Significance in Psychological Research". Psychological Bulletin 66 (6): 423–437. doi:10.1037/h0020412. [46] Kruschke, J K (September/October 2010). "Bayesian Data Analysis". Wiley Interdisciplinary Reviews: Cognitive Science 1 (5): 658–676. doi:10.1002/wcs.72. [47] Rothman, K J (January 1990). "No Adjustments Are Needed for Multiple Comparisons" (http:/ / www. jstor. org/ stable/ 20065622). Epidemiology 1 (1): 43–46. . [48] Nester, M R (1996). "An Applied Statistician's Creed". Journal of the Royal Statistical Society. Series C (Applied Statistics) 45 (4): 401–410. JSTOR 2986064. [49] Savage, R (September 1957). "Nonparametric Statistics" (http:/ / www. jstor. org/ stable/ 2280903). Journal of the American Statistical Association 52 (279): 331–344. . [50] Cohen, J (December 1994). "The Earth Is Round (p < .05)" (http:/ / ist-socrates. berkeley. edu/ ~maccoun/ PP279_Cohen1. pdf). American Psychologist 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997. . [51] Meehl, P (June 1967). "Theory-Testing in Psychology and Physics: A Methodological Paradox" (http:/ / www. psych. ucsb. edu/ ~janusonis/ meehl1967. pdf). Philosophy of Science 34 (2): 103–115. . [52] "Male and Female Students Make Gains Since 2000 at Grade 4; Males Outperform Females at all Three Grades in 2005" (http:/ / nationsreportcard. gov/ science_2005/ s0110. asp). The Nation's Report Card. U.S. Department of Education. . Retrieved 16 September 2012. [53] Hyde, Janet Shibley; Marcia C. Linn (27 October 2006). "Gender similarities in mathematics and science" (http:/ / www. sciencemag. org/ content/ 314/ 5799/ 599. summary). Science 314 (5799): 599–600. doi:10.1126/science.1132154. PMID 17068246. . Retrieved 16 September 2012. Statistical hypothesis testing 69 [54] Goodman, Steven (July 2008). "A Dirty Dozen: Twelve P-Value Misconceptions". Seminars in Hematology 45 (3): 135–140. doi:10.1053/j.seminhematol.2008.04.003. PMID 18582619. [55] Berger, J O (March 1987). "Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence". Journal of the American Statistical Association 82 (397): 112–122. JSTOR 2289131. [56] Goodman, S N (June 15, 1999). "Toward evidence-based medical statistics. 2: The Bayes factor." (http:/ / annals. org/ article. aspx?articleid=712763). Ann Intern Med 130 (12): 1005–1013. doi:10.7326/0003-4819-130-12-199906150-00009. PMID 10383350. . [57] Sutton, A J; Duval, SJ; Tweedie, RL; Abrams, KR; Jones, DR (10 June 2000). "Empirical assessment of effect of publication bias on meta-analyses". BMJ 320 (7249): 1574–1577. doi:10.1136/bmj.320.7249.1574. PMC 27401. PMID 10845965. [58] Vickers, A (April 1998). "Do Certain Countries Produce Only Positive Results? A Systematic Review of Controlled Trials" (http:/ / dx. doi. org/ 10. 1016/ S0197-2456(97)00150-5). Controlled Clinical Trials 19 (2): 159–166. . [59] Fanelli, D (April 7, 2010). "‘‘Positive’’ Results Increase Down the Hierarchy of the Sciences". PLOS ONE 5 (4:e10068): N/A. doi:10.1371/journal.pone.0010068. [60] Fanelli, D (January 2012volume=90). "Negative results are disappearing from most disciplines and countries". Scientometrics (3): 891–904. doi:10.1007/s11192-011-0494-7. [61] Simmons, J P (2011). "Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant". Psychological Science 22 (11): 1359–1366. doi:10.1177/0956797611417632. [62] Masicampo, E J (August 2, 2012). "A peculiar prevalence of p values just below .05" (http:/ / dx. doi. org/ 10. 1080/ 17470218. 2012. 711335). The Quarterly Journal of Experimental Psychology 65 (11): 2271–2279. . [63] Wilkinson, Leland (1999). "Statistical Methods in Psychology Journals; Guidelines and Explanations". American Psychologist 54 (8): 594–604. doi:10.1037/0003-066X.54.8.594. [64] "ICMJE: Obligation to Publish Negative Studies" (http:/ / www. icmje. org/ publishing_1negative. html). . Retrieved 3 September 2012. "Editors should seriously consider for publication any carefully done study of an important question, relevant to their readers, whether the results for the primary or any additional outcome are statistically significant. Failure to submit or publish findings because of lack of statistical significance is an important cause of publication bias." [65] Journal of Articles in Support of the Null Hypothesis website: JASNH homepage (http:/ / www. jasnh. com/ ). Volume 1 number 1 was published in 2002, and all articles are on psychology-related subjects. [66] Howell, David (2002). Statistical Methods for Psychology (5 ed.). Duxbury. p. 94. ISBN 0-534-37770-X. [67] Mayo, D. G.; Spanos, A. (2006). "Severe Testing as a Basic Concept in a Neyman-Pearson Philosophy of Induction". The British Journal for the Philosophy of Science 57 (2): 323. doi:10.1093/bjps/axl003. [68] Hubbard, R.; Armstrong, J. S. (2006). "Why We Don't Really Know What Statistical Significance Means: Implications for Educators". Journal of Marketing Education 28 (2): 114. doi:10.1177/0273475306288399. Preprint (http:/ / marketing. wharton. upenn. edu/ ideas/ pdf/ Armstrong/ StatisticalSignificance. pdf) [69] Armstrong, J. Scott (2007). "Significance tests harm progress in forecasting" (http:/ / repository. upenn. edu/ cgi/ viewcontent. cgi?article=1104& context=marketing_papers& sei-redir=1& referer=http:/ / scholar. google. com/ scholar?q=Significance+ tests+ harm+ progress+ in+ forecasting& hl=en& btnG=Search& as_sdt=1%2C5& as_sdtp=on#search="Significance tests harm progress forecasting"). International Journal of Forecasting 23 (2): 321–327. doi:10.1016/j.ijforecast.2007.03.004. . [70] E. L. Lehmann (1997). "Testing Statistical Hypotheses: The Story of a Book". Statistical Science 12 (1): 48–52. doi:10.1214/ss/1029963261. [71] Jacob Cohen (December 1994). "The Earth Is Round (p < .05)". American Psychologist 49 (12): 997–1003. doi:10.1037/0003-066X.49.12.997. This paper lead to the review of statistical practices by the APA. Cohen was a member of the Task Force that did the review. [72] Kruschke, J K (July 9, 2012). "Bayesian Estimation Supersedes the t test". Journal of Experimental Psychology: General. N/A (N/A): N/A. doi:10.1037/a0029146. [73] Kass, R E (1993). Bayes factors and model uncertainty (http:/ / www. stat. washington. edu/ research/ reports/ 1993/ tr254. pdf). N/A. pp. N/A. . [74] Aldrich, J (2008). "R. A. Fisher on Bayes and Bayes' theorem" (http:/ / ba. stat. cmu. edu/ journal/ 2008/ vol03/ issue01/ aldrich. pdf). Bayesian Analysis 3 (1): 161–170. . [75] Mathematics > High School: Statistics & Probability > Introduction (http:/ / www. corestandards. org/ the-standards/ mathematics/ hs-statistics-and-probability/ introduction/ ) Common Core State Standards Initiative (relates to USA students) [76] College Board Tests > AP: Subjects > Statistics (http:/ / www. collegeboard. com/ student/ testing/ ap/ sub_stats. html) The College Board (relates to USA students) [77] Huff, Darrell (1993). How to lie with statistics. New York: Norton. p. 8. ISBN 0-393-31072-8. 'Statistical methods and statistical terms are necessary in reporting the mass data of social and economic trends, business conditions, "opinion" polls, the census. But without writers who use the words with honesty and readers who know what they mean, the result can only be semantic nonsense.' [78] Snedecor, George W.; Cochran, William G. (1967). Statistical Methods (6 ed.). Ames, Iowa: Iowa State University Press. p. 3. "...the basic ideas in statistics assist us in thinking clearly about the problem, provide some guidance about the conditions that must be satisfied if sound inferences are to be made, and enable us to detect many inferences that have no good logical foundation." [79] Lehmann, E. L. (1997). "Testing statistical hypotheses: The story of a book". Statistical Science 12: 48. doi:10.1214/ss/1029963261. Statistical hypothesis testing 70 Further reading • Lehmann E.L. (1992) "Introduction to Neyman and Pearson (1933) On the Problem of the Most Efficient Tests of Statistical Hypotheses". In: Breakthroughs in Statistics, Volume 1, (Eds Kotz, S., Johnson, N.L.), Springer-Verlag. ISBN 0-387-94037-5 (followed by reprinting of the paper) • Neyman, J.; Pearson, E.S. (1933). "On the Problem of the Most Efficient Tests of Statistical Hypotheses". Phil. Trans. R. Soc., Series A 231 (694–706): 289–337. doi:10.1098/rsta.1933.0009. External links • Hazewinkel, Michiel, ed. (2001), "Statistical hypotheses, verification of" (http://www.encyclopediaofmath.org/ index.php?title=p/s087400), Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4 • Wilson González, Georgina; Karpagam Sankaran (September 10, 1997). "Hypothesis Testing" (http://www.cee. vt.edu/ewr/environmental/teach/smprimer/hypotest/ht.html). Environmental Sampling & Monitoring Primer. Virginia Tech. • Bayesian critique of classical hypothesis testing (http://www.cs.ucsd.edu/users/goguen/courses/275f00/stat. html) • Critique of classical hypothesis testing highlighting long-standing qualms of statisticians (http://www.npwrc. usgs.gov/resource/methods/statsig/stathyp.htm) • Dallal GE (2007) The Little Handbook of Statistical Practice (http://www.tufts.edu/~gdallal/LHSP.HTM) (A good tutorial) • References for arguments for and against hypothesis testing (http://core.ecu.edu/psyc/wuenschk/StatHelp/ NHST-SHIT.htm) • Statistical Tests Overview: (http://www.wiwi.uni-muenster.de/ioeb/en/organisation/pfaff/ stat_overview_table.html) How to choose the correct statistical test • An Interactive Online Tool to Encourage Understanding Hypothesis Testing (http://wasser.heliohost.org/ ?l=en) • A non mathematical way to understand Hypothesis Testing (http://simplifyingstats.com/data/ HypothesisTesting.pdf) Estimation theory 71 Estimation theory Estimation theory is a branch of statistics and signal processing that deals with estimating the values of parameters based on measured/empirical data that has a random component. The parameters describe an underlying physical setting in such a way that their value affects the distribution of the measured data. An estimator attempts to approximate the unknown parameters using the measurements. For example, it is desired to estimate the proportion of a population of voters who will vote for a particular candidate. That proportion is the parameter sought; the estimate is based on a small random sample of voters. Or, for example, in radar the goal is to estimate the range of objects (airplanes, boats, etc.) by analyzing the two-way transit timing of received echoes of transmitted pulses. Since the reflected pulses are unavoidably embedded in electrical noise, their measured values are randomly distributed, so that the transit time must be estimated. In estimation theory, it is assumed the measured data is random with probability distribution dependent on the parameters of interest. For example, in electrical communication theory, the measurements which contain information regarding the parameters of interest are often associated with a noisy signal. Without randomness, or noise, the problem would be deterministic and estimation would not be needed. Estimation process The entire purpose of estimation theory is to arrive at an estimator — preferably an easily implementable one. The estimator takes the measured data as input and produces an estimate of the parameters. It is also preferable to derive an estimator that exhibits optimality. Estimator optimality usually refers to achieving minimum average error over some class of estimators, for example, a minimum variance unbiased estimator. In this case, the class is the set of unbiased estimators, and the average error measure is variance (average squared error between the value of the estimate and the parameter). However, optimal estimators do not always exist. These are the general steps to arrive at an estimator: • In order to arrive at a desired estimator, it is first necessary to determine a probability distribution for the measured data, and the distribution's dependence on the unknown parameters of interest. Often, the probability distribution may be derived from physical models that explicitly show how the measured data depends on the parameters to be estimated, and how the data is corrupted by random errors or noise. In other cases, the probability distribution for the measured data is simply "assumed", for example, based on familiarity with the measured data and/or for analytical convenience. • After deciding upon a probabilistic model, it is helpful to find the theoretically achievable (optimal) precision available to any estimator based on this model. The Cramér–Rao bound is useful for this. • Next, an estimator needs to be developed, or applied (if an already known estimator is valid for the model). There are a variety of methods for developing estimators; maximum likelihood estimators are often the default although they may be hard to compute or even fail to exist. If possible, the theoretical performance of the estimator should be derived and compared with the optimal performance found in the last step. • Finally, experiments or simulations can be run using the estimator to test its performance. After arriving at an estimator, real data might show that the model used to derive the estimator is incorrect, which may require repeating these steps to find a new estimator. A non-implementable or infeasible estimator may need to be scrapped and the process started anew. In summary, the estimator estimates the parameters of a physical model based on measured data. Estimation theory 72 Basics To build a model, several statistical "ingredients" need to be known. These are needed to ensure the estimator has some mathematical tractability instead of being based on "good feel". The first is a set of statistical samples taken from a random vector (RV) of size N. Put into a vector, Secondly, there are the corresponding M parameters which need to be established with their continuous probability density function (pdf) or its discrete counterpart, the probability mass function (pmf) It is also possible for the parameters themselves to have a probability distribution (e.g., Bayesian statistics). It is then necessary to define the Bayesian probability After the model is formed, the goal is to estimate the parameters, commonly denoted , where the "hat" indicates the estimate. One common estimator is the minimum mean squared error estimator, which utilizes the error between the estimated parameters and the actual value of the parameters as the basis for optimality. This error term is then squared and minimized for the MMSE estimator. Estimators Commonly used estimators and estimation methods, and topics related to them: • Maximum likelihood estimators • Bayes estimators • Method of moments estimators • Cramér–Rao bound • Minimum mean squared error (MMSE), also known as Bayes least squared error (BLSE) • Maximum a posteriori (MAP) • Minimum variance unbiased estimator (MVUE) • Best linear unbiased estimator (BLUE) • Unbiased estimators — see estimator bias. • Particle filter • Markov chain Monte Carlo (MCMC) • Kalman filter, and its various derivatives • Wiener filter Estimation theory 73 Examples Unknown constant in additive white Gaussian noise Consider a received discrete signal, , of independent samples that consists of an unknown constant with additive white Gaussian noise (AWGN) with known variance (i.e., ). Since the variance is known then the only unknown parameter is . The model for the signal is then Two possible (of many) estimators are: • • which is the sample mean Both of these estimators have a mean of , which can be shown through taking the expected value of each estimator and At this point, these two estimators would appear to perform the same. However, the difference between them becomes apparent when comparing the variances. and It would seem that the sample mean is a better estimator since its variance is lower for every N>1. Maximum likelihood Continuing the example using the maximum likelihood estimator, the probability density function (pdf) of the noise for one sample is and the probability of becomes ( can be thought of a ) By independence, the probability of becomes Taking the natural logarithm of the pdf Estimation theory 74 and the maximum likelihood estimator is Taking the first derivative of the log-likelihood function and setting it to zero This results in the maximum likelihood estimator which is simply the sample mean. From this example, it was found that the sample mean is the maximum likelihood estimator for samples of a fixed, unknown parameter corrupted by AWGN. Cramér–Rao lower bound To find the Cramér–Rao lower bound (CRLB) of the sample mean estimator, it is first necessary to find the Fisher information number and copying from above Taking the second derivative and finding the negative expected value is trivial since it is now a deterministic constant Finally, putting the Fisher information into results in Comparing this to the variance of the sample mean (determined previously) shows that the sample mean is equal to the Cramér–Rao lower bound for all values of and . In other words, the sample mean is the (necessarily unique) efficient estimator, and thus also the minimum variance unbiased estimator (MVUE), in addition to being the maximum likelihood estimator. Estimation theory 75 Maximum of a uniform distribution One of the simplest non-trivial examples of estimation is the estimation of the maximum of a uniform distribution. It is used as a hands-on classroom exercise and to illustrate basic principles of estimation theory. Further, in the case of estimation based on a single sample, it demonstrates philosophical issues and possible misunderstandings in the use of maximum likelihood estimators and likelihood functions. Given a discrete uniform distribution with unknown maximum, the UMVU estimator for the maximum is given by where m is the sample maximum and k is the sample size, sampling without replacement.[1][2] This problem is commonly known as the German tank problem, due to application of maximum estimation to estimates of German tank production during World War II. The formula may be understood intuitively as: "The sample maximum plus the average gap between observations in the sample", the gap being added to compensate for the negative bias of the sample maximum as an estimator for the population maximum.[3] This has a variance of[1] so a standard deviation of approximately , the (population) average size of a gap between samples; compare above. This can be seen as a very simple case of maximum spacing estimation. The sample maximum is the maximum likelihood estimator for the population maximum, but, as discussed above, it is biased. Applications Numerous fields require the use of estimation theory. Some of these fields include (but are by no means limited to): • Interpretation of scientific experiments • Signal processing • Clinical trials • Opinion polls • Quality control • Telecommunications • Project management • Software engineering • Control theory (in particular Adaptive control) • Network intrusion detection system • Orbit determination Measured data are likely to be subject to noise or uncertainty and it is through statistical probability that optimal solutions are sought to extract as much information from the data as possible. Estimation theory 76 Notes [1] Johnson, Roger (1994), "Estimating the Size of a Population", Teaching Statistics (http:/ / www. rsscse. org. uk/ ts/ index. htm) 16 (2 (Summer)): 50, doi:10.1111/j.1467-9639.1994.tb00688.x [2] Johnson, Roger (2006), "Estimating the Size of a Population" (http:/ / www. rsscse. org. uk/ ts/ gtb/ johnson. pdf), Getting the Best from Teaching Statistics (http:/ / www. rsscse. org. uk/ ts/ gtb/ contents. html), [3] The sample maximum is never more than the population maximum, but can be less, hence it is a biased estimator: it will tend to underestimate the population maximum. References References • Theory of Point Estimation by E.L. Lehmann and G. Casella. (ISBN 0387985026) • Systems Cost Engineering by Dale Shermon. (ISBN 978-0-566-08861-2) • Mathematical Statistics and Data Analysis by John Rice. (ISBN 0-534-209343) • Fundamentals of Statistical Signal Processing: Estimation Theory by Steven M. Kay (ISBN 0-13-345711-7) • An Introduction to Signal Detection and Estimation by H. Vincent Poor (ISBN 0-387-94173-8) • Detection, Estimation, and Modulation Theory, Part 1 by Harry L. Van Trees (ISBN 0-471-09517-6; website (http://gunston.gmu.edu/demt/demtp1/)) • Optimal State Estimation: Kalman, H-infinity, and Nonlinear Approaches by Dan Simon website (http:// academic.csuohio.edu/simond/estimation/) • Ali H. Sayed, Adaptive Filters, Wiley, NJ, 2008, ISBN 978-0-470-25388-5. • Ali H. Sayed, Fundamentals of Adaptive Filtering, Wiley, NJ, 2003, ISBN 0-471-46126-1. • Thomas Kailath, Ali H. Sayed, and Babak Hassibi, Linear Estimation, Prentice-Hall, NJ, 2000, ISBN 978-0-13-022464-4. • Babak Hassibi, Ali H. Sayed, and Thomas Kailath, Indefinite Quadratic Estimation and Control: A Unified Approach to H2 and Hoo Theories, Society for Industrial & Applied Mathematics (SIAM), PA, 1999, ISBN 978-0-89871-411-1. • V.G.Voinov, M.S.Nikulin, "Unbiased estimators and their applications. Vol.1: Univariate case", Kluwer Academic Publishers, 1993, ISBN 0-7923-2382-3. • V.G.Voinov, M.S.Nikulin, "Unbiased estimators and their applications. Vol.2: Multivariate case", Kluwer Academic Publishers, 1996, ISBN 0-7923-3939-8. Semiparametric model 77 Semiparametric model In statistics a semiparametric model is a model that has parametric and nonparametric components. A model is a collection of distributions: indexed by a parameter . • A parametric model is one in which the indexing parameter is a finite-dimensional vector (in -dimensional Euclidean space for some integer ); i.e. the set of possible values for is a subset of , or . In this case we say that is finite-dimensional. • In nonparametric models, the set of possible values of the parameter is a subset of some space, not necessarily finite dimensional. For example, we might consider the set of all distributions with mean 0. Such spaces are vector spaces with topological structure, but may not be finite dimensional as vector spaces. Thus, for some possibly infinite dimensional space . • In semiparametric models, the parameter has both a finite dimensional component and an infinite dimensional component (often a real-valued function defined on the real line). Thus the parameter space in a semiparametric model satisfies , where is an infinite dimensional space. It may appear at first that semiparametric models include nonparametric models, since they have an infinite dimensional as well as a finite dimensional component. However, a semiparametric model is considered to be "smaller" than a completely nonparametric model because we are often interested only in the finite-dimensional component of . That is, we are not interested in estimating the infinite-dimensional component. In nonparametric models, by contrast, the primary interest is in estimating the infinite dimensional parameter. Thus the estimation task is statistically harder in nonparametric models. These models often use smoothing or kernels. Example A well-known example of a semiparametric model is the Cox proportional hazards model. If we are interested in studying the time to an event such as death due to cancer or failure of a light bulb, the Cox model specifies the following distribution function for : where is a known function of time (the covariate vector at time ), and and are unknown parameters. . Here is finite dimensional and is of interest; is an unknown non-negative function of time (known as the baseline hazard function) and is often a nuisance parameter. The collection of possible candidates for is infinite dimensional. Non-parametric statistics 78 Non-parametric statistics In statistics, the term non-parametric statistics has at least two different meanings: 1. The first meaning of non-parametric covers techniques that do not rely on data belonging to any particular distribution. These include, among others: • distribution free methods, which do not rely on assumptions that the data are drawn from a given probability distribution. As such it is the opposite of parametric statistics. It includes non-parametric statistical models, inference and statistical tests. • non-parametric statistics (in the sense of a statistic over data, which is defined to be a function on a sample that has no dependency on a parameter), whose interpretation does not depend on the population fitting any parametrized distributions. Statistics based on the ranks of observations are one example of such statistics and these play a central role in many non-parametric approaches. 2. The second meaning of non-parametric covers techniques that do not assume that the structure of a model is fixed. Typically, the model grows in size to accommodate the complexity of the data. In these techniques, individual variables are typically assumed to belong to parametric distributions, and assumptions about the types of connections among variables are also made. These techniques include, among others: • non-parametric regression, which refers to modeling where the structure of the relationship between variables is treated non-parametrically, but where nevertheless there may be parametric assumptions about the distribution of model residuals. • non-parametric hierarchical Bayesian models, such as models based on the Dirichlet process, which allow the number of latent variables to grow as necessary to fit the data, but where individual variables still follow parametric distributions and even the process controlling the rate of growth of latent variables follows a parametric distribution. Applications and purpose Non-parametric methods are widely used for studying populations that take on a ranked order (such as movie reviews receiving one to four stars). The use of non-parametric methods may be necessary when data have a ranking but no clear numerical interpretation, such as when assessing preferences. In terms of levels of measurement, non-parametric methods result in "ordinal" data. As non-parametric methods make fewer assumptions, their applicability is much wider than the corresponding parametric methods. In particular, they may be applied in situations where less is known about the application in question. Also, due to the reliance on fewer assumptions, non-parametric methods are more robust. Another justification for the use of non-parametric methods is simplicity. In certain cases, even when the use of parametric methods is justified, non-parametric methods may be easier to use. Due both to this simplicity and to their greater robustness, non-parametric methods are seen by some statisticians as leaving less room for improper use and misunderstanding. The wider applicability and increased robustness of non-parametric tests comes at a cost: in cases where a parametric test would be appropriate, non-parametric tests have less power. In other words, a larger sample size can be required to draw conclusions with the same degree of confidence. Non-parametric statistics 79 Non-parametric models Non-parametric models differ from parametric models in that the model structure is not specified a priori but is instead determined from data. The term non-parametric is not meant to imply that such models completely lack parameters but that the number and nature of the parameters are flexible and not fixed in advance. • A histogram is a simple nonparametric estimate of a probability distribution • Kernel density estimation provides better estimates of the density than histograms. • Nonparametric regression and semiparametric regression methods have been developed based on kernels, splines, and wavelets. • Data envelopment analysis provides efficiency coefficients similar to those obtained by multivariate analysis without any distributional assumption. Methods Non-parametric (or distribution-free) inferential statistical methods are mathematical procedures for statistical hypothesis testing which, unlike parametric statistics, make no assumptions about the probability distributions of the variables being assessed. The most frequently used tests include • Anderson–Darling test • Statistical Bootstrap Methods • Cochran's Q • Cohen's kappa • Friedman two-way analysis of variance by ranks • Kaplan–Meier • Kendall's tau • Kendall's W • Kolmogorov–Smirnov test • Kruskal-Wallis one-way analysis of variance by ranks • Kuiper's test • Logrank Test • Mann–Whitney U or Wilcoxon rank sum test • McNemar's test • median test • Pitman's permutation test • Rank products • Siegel–Tukey test • sign test • Spearman's rank correlation coefficient • Wald–Wolfowitz runs test • Wilcoxon signed-rank test. Non-parametric statistics 80 General references • Corder, G.W. & Foreman, D.I. (2009) Nonparametric Statistics for Non-Statisticians: A Step-by-Step Approach, Wiley ISBN 978-0-470-45461-9 • Gibbons, Jean Dickinson and Chakraborti, Subhabrata (2003) Nonparametric Statistical Inference, 4th Ed. CRC ISBN 0-8247-4052-1 • Hettmansperger, T. P.; McKean, J. W. (1998). Robust nonparametric statistical methods. Kendall's Library of Statistics. 5 (First ed.). London: Edward Arnold. pp. xiv+467 pp.. ISBN 0-340-54937-8, 0-471-19479-4. MR1604954. • Wasserman, Larry (2007) All of nonparametric statistics, Springer. ISBN 0-387-25145-6 • Bagdonavicius, V., Kruopis, J., Nikulin, M.S. (2011). "Non-parametric tests for complete data", ISTE&WILEY: London&Hoboken. ISBN 978-1-84821-269-5 Simulation Simulation is the imitation of the operation of a real-world process or system over time.[1] The act of simulating something first requires that a model be developed; this model represents the key characteristics or behaviors of the selected physical or abstract system or process. The model represents the system itself, whereas the simulation represents the operation of the system over time. Simulation is used in many contexts, such as simulation of technology for performance optimization, safety engineering, testing, training, education, and video Wooden mechanical horse simulator during WWI. games. Training simulators include flight simulators for training aircraft pilots to provide them with a lifelike experience. Simulation is also used with scientific modelling of natural systems or human systems to gain insight into their functioning.[2] Simulation can be used to show the eventual real effects of alternative conditions and courses of action. Simulation is also used when the real system cannot be engaged, because it may not be accessible, or it may be dangerous or unacceptable to engage, or it is being designed but not yet built, or it may simply not exist.[3] Key issues in simulation include acquisition of valid source information about the relevant selection of key characteristics and behaviours, the use of simplifying approximations and assumptions within the simulation, and fidelity and validity of the simulation outcomes. Simulation 81 Classification and terminology Historically, simulations used in different fields developed largely independently, but 20th century studies of Systems theory and Cybernetics combined with spreading use of computers across all those fields have led to some unification and a more systematic view of the concept. Physical simulation refers to simulation in which physical objects are substituted for the real thing (some circles[4] use the term for computer simulations modelling selected laws of physics, but this article doesn't). These physical objects are often chosen because they are smaller or cheaper than the actual object or system. Interactive simulation is a special kind of physical simulation, often referred to as a human in the loop simulation, in which physical simulations include human operators, such as in a flight simulator or a driving simulator. Human in the loop simulations can include a computer simulation as a so-called synthetic environment.[5] Human-in-the-loop simulation of outer space. Computer simulation A computer simulation (or "sim") is an attempt to model a real-life or hypothetical situation on a computer so that it can be studied to see how the system works. By changing variables in the simulation, predictions may be made about the behaviour of the system. It is a tool to virtually investigate the behaviour of the system under study.[1] Computer simulation has become a useful part of modeling many natural systems in physics, chemistry and biology,[6] and human Visualization of a direct numerical simulation systems in economics and social science (the computational sociology) model. as well as in engineering to gain insight into the operation of those systems. A good example of the usefulness of using computers to simulate can be found in the field of network traffic simulation. In such simulations, the model behaviour will change each simulation according to the set of initial parameters assumed for the environment. Traditionally, the formal modeling of systems has been via a mathematical model, which attempts to find analytical solutions enabling the prediction of the behaviour of the system from a set of parameters and initial conditions. Computer simulation is often used as an adjunct to, or substitution for, modeling systems for which simple closed form analytic solutions are not possible. There are many different types of computer simulation, the common feature they all share is the attempt to generate a sample of representative scenarios for a model in which a complete enumeration of all possible states would be prohibitive or impossible. Several software packages exist for running computer-based simulation modeling (e.g. Monte Carlo simulation, stochastic modeling, multimethod modeling) that makes all the modeling almost effortless. Modern usage of the term "computer simulation" may encompass virtually any computer-based representation. Simulation 82 Computer science In computer science, simulation has some specialized meanings: Alan Turing used the term "simulation" to refer to what happens when a universal machine executes a state transition table (in modern terminology, a computer runs a program) that describes the state transitions, inputs and outputs of a subject discrete-state machine. The computer simulates the subject machine. Accordingly, in theoretical computer science the term simulation is a relation between state transition systems, useful in the study of operational semantics. Less theoretically, an interesting application of computer simulation is to simulate computers using computers. In computer architecture, a type of simulator, typically called an emulator, is often used to execute a program that has to run on some inconvenient type of computer (for example, a newly designed computer that has not yet been built or an obsolete computer that is no longer available), or in a tightly controlled testing environment (see Computer architecture simulator and Platform virtualization). For example, simulators have been used to debug a microprogram or sometimes commercial application programs, before the program is downloaded to the target machine. Since the operation of the computer is simulated, all of the information about the computer's operation is directly available to the programmer, and the speed and execution of the simulation can be varied at will. Simulators may also be used to interpret fault trees, or test VLSI logic designs before they are constructed. Symbolic simulation uses variables to stand for unknown values. In the field of optimization, simulations of physical processes are often used in conjunction with evolutionary computation to optimize control strategies... Simulation in education and training Simulation is extensively used for educational purposes. It is frequently used by way of adaptive hypermedia. Simulation is often used in the training of civilian and military personnel.[7] This usually occurs when it is prohibitively expensive or simply too dangerous to allow trainees to use the real equipment in the real world. In such situations they will spend time learning valuable lessons in a "safe" virtual environment yet living a lifelike experience (or at least it is the goal). Often the convenience is to permit mistakes during training for a safety-critical system. For example, in simSchool [8] teachers practice classroom management and teaching techniques on simulated students, which avoids "learning on the job" that can damage real students. There is a distinction, though, between simulations used for training and Instructional simulation. Training simulations typically come in one of three categories:[9] • "live" simulation (where actual players use genuine systems in a real environment); • "virtual" simulation (where actual players use simulated systems in a synthetic environment [5]), or • "constructive" simulation (where simulated players use simulated systems in a synthetic environment). Constructive simulation is often referred to as "wargaming" since it bears some resemblance to table-top war games in which players command armies of soldiers and equipment that move around a board. In standardized tests, "live" simulations are sometimes called "high-fidelity", producing "samples of likely performance", as opposed to "low-fidelity", "pencil-and-paper" simulations producing only "signs of possible performance",[10] but the distinction between high, moderate and low fidelity remains relative, depending on the context of a particular comparison. Simulations in education are somewhat like training simulations. They focus on specific tasks. The term 'microworld' is used to refer to educational simulations which model some abstract concept rather than simulating a realistic object or environment, or in some cases model a real world environment in a simplistic way so as to help a learner develop an understanding of the key concepts. Normally, a user can create some sort of construction within the microworld that will behave in a way consistent with the concepts being modeled. Seymour Papert was one of the first to advocate the value of microworlds, and the Logo programming environment developed by Papert is one of the most famous microworlds. As another example, the Global Challenge Award online STEM learning web site Simulation 83 uses microworld simulations to teach science concepts related to global warming and the future of energy. Other projects for simulations in educations are Open Source Physics, NetSim etc. Management games (or business simulations) have been finding favour in business education in recent years.[11] Business simulations that incorporate a dynamic model enable experimentation with business strategies in a risk free environment and provide a useful extension to case study discussions. Social simulations may be used in social science classrooms to illustrate social and political processes in anthropology, economics, history, political science, or sociology courses, typically at the high school or university level. These may, for example, take the form of civics simulations, in which participants assume roles in a simulated society, or international relations simulations in which participants engage in negotiations, alliance formation, trade, diplomacy, and the use of force. Such simulations might be based on fictitious political systems, or be based on current or historical events. An example of the latter would be Barnard College's Reacting to the Past series of historical educational games.[12] The National Science Foundation has also supported the creation of reacting games that address science and math education.[13] In recent years, there has been increasing use of social simulations for staff training in aid and development agencies. The Carana simulation, for example, was first developed by the United Nations Development Programme, and is now used in a very revised form by the World Bank for training staff to deal with fragile and conflict-affected countries.[14] Common user interaction systems for virtual simulations Virtual simulations represent a specific category of simulation that utilizes simulation equipment to create a simulated world for the user. Virtual simulations allow users to interact with a virtual world. Virtual worlds operate on platforms of integrated software and hardware components. In this manner, the system can accept input from the user (e.g., body tracking, voice/sound recognition, physical controllers) and produce output to the user (e.g., visual display, aural display, haptic display) .[15] Virtual Simulations use the aforementioned modes of interaction to produce a sense of immersion for the user. Virtual simulation input hardware There is a wide variety of input hardware available to accept user input for virtual simulations. The following list briefly describes several of them: Body tracking The motion capture method is often used to record the user’s movements and translate the captured data into inputs for the virtual simulation. For example, if a user physically turns their head, the motion would be captured by the simulation hardware in some way and translated to a corresponding shift in view within the simulation. • Capture suits and/or gloves may be used Motorcycle simulator of Bienal do Automóvel exhibition, in Belo Horizonte, to capture movements of users body Brazil. parts. The systems may have sensors Simulation 84 incorporated inside them to sense movements of different body parts (e.g., fingers). Alternatively, these systems may have exterior tracking devices or marks that can be detected by external ultrasound, optical receivers or electromagnetic sensors. Internal inertial sensors are also available on some systems. The units may transmit data either wirelessly or through cables. • Eye trackers can also be used to detect eye movements so that the system can determine precisely where a user is looking at any given instant. Physical controllers Physical controllers provide input to the simulation only through direct manipulation by the user. In virtual simulations, tactile feedback from physical controllers is highly desirable in a number of simulation environments. • Omni directional treadmills can be used to capture the users locomotion as they walk or run. • High fidelity instrumentation such as instrument panels in virtual aircraft cockpits provides users with actual controls to raise the level of immersion. For example, pilots can use the actual global positioning system controls from the real device in a simulated cockpit to help them practice procedures with the actual device in the context of the integrated cockpit system. Voice/sound recognition This form of interaction may be used either to interact with agents within the simulation (e.g., virtual people) or to manipulate objects in the simulation (e.g., information). Voice interaction presumably increases the level of immersion for the user. • Users may use headsets with boom microphones, lapel microphones or the room may be equipped with strategically located microphones. Current research into user input systems Research in future input systems hold a great deal of promise for virtual simulations. Systems such as brain-computer interfaces (BCIs)Brain-computer interface offer the ability to further increase the level of immersion for virtual simulation users. Lee, Keinrath, Scherer, Bischof, Pfurtscheller [16] proved that naïve subjects could be trained to use a BCI to navigate a virtual apartment with relative ease. Using the BCI, the authors found that subjects were able to freely navigate the virtual environment with relatively minimal effort. It is possible that these types of systems will become standard input modalities in future virtual simulation systems. Simulation is a one of the part of an engineering students and also imp for main electrical students its come in form of education purpose. Virtual simulation output hardware There is a wide variety of output hardware available to deliver stimulus to users in virtual simulations. The following list briefly describes several of them: Visual display Visual displays provide the visual stimulus to the user. • Stationary displays can vary from a conventional desktop display to 360-degree wrap around screens to stereo three-dimensional screens. Conventional desktop displays can vary in size from 15 to 60+ inches. Wrap around screens are typically utilized in what is known as a Cave Automatic Virtual Environment (CAVE) Cave Automatic Virtual Environment. Stereo three-dimensional screens produce three-dimensional images either with or without special glasses—depending on the design. • Head mounted displays (HMDs) have small displays that are mounted on headgear worn by the user. These systems are connected directly into the virtual simulation to provide the user with a more immersive experience. Weight, update rates and field of view are some of the key variables that differentiate HMDs. Naturally, heavier HMDs are undesirable as they cause fatigue over time. If the update rate is too slow, the system is unable to update the displays fast enough to correspond with a quick head turn by the user. Slower update rates tend to cause simulation sickness and disrupt the sense of immersion. Field of view or the angular extent of the world that is seen at a given moment Field of view can vary from system to system and has been found to affect the users sense of immersion. Simulation 85 Aural display Several different types of audio systems exist to help the user hear and localize sounds spatially. Special software can be used to produce 3D audio effects 3D audio to create the illusion that sound sources are placed within a defined three-dimensional space around the user. • Stationary conventional speaker systems may be used provide dual or multi-channel surround sound. However, external speakers are not as effective as headphones in producing 3D audio effects.[15] • Conventional headphones offer a portable alternative to stationary speakers. They also have the added advantages of masking real world noise and facilitate more effective 3D audio sound effects.[15] Haptic display These displays provide sense of touch to the user Haptic technology. This type of output is sometimes referred to as force feedback. • Tactile tile displays use different types of actuators such as inflatable bladders, vibrators, low frequency sub-woofers, pin actuators and/or thermo-actuators to produce sensations for the user. • End effector displays can respond to users inputs with resistance and force.[15] These systems are often used in medical applications for remote surgeries that employ robotic instruments.[17] Vestibular display These displays provide a sense of motion to the user Motion simulator. They often manifest as motion bases for virtual vehicle simulation such as driving simulators or flight simulators. Motion bases are fixed in place but use actuators to move the simulator in ways that can produce the sensations pitching, yawing or rolling. The simulators can also move in such a way as to produce a sense of acceleration on all axes (e.g., the motion base can produce the sensation of falling). Clinical healthcare simulators Medical simulators are increasingly being developed and deployed to teach therapeutic and diagnostic procedures as well as medical concepts and decision making to personnel in the health professions. Simulators have been developed for training procedures ranging from the basics such as blood draw, to laparoscopic surgery [18] and trauma care. They are also important to help on prototyping new devices[19] for biomedical engineering problems. Currently, simulators are applied to research and develop tools for new therapies,[20] treatments[21] and early diagnosis[22] in medicine. Many medical simulators involve a computer connected to a plastic simulation of the relevant anatomy. Sophisticated simulators of this type employ a life size mannequin that responds to injected drugs and can be programmed to create simulations of life-threatening emergencies. In other simulations, visual components of the procedure are reproduced by computer graphics techniques, while touch-based components are reproduced by haptic feedback devices combined with physical simulation routines computed in response to the user's actions. Medical simulations of this sort will often use 3D CT or MRI scans of patient data to enhance realism. Some medical simulations are developed to be widely distributed (such as web-enabled simulations [23] and procedural simulations [24] that can be viewed via standard web browsers) and can be interacted with using standard computer interfaces, such as the keyboard and mouse. Another important medical application of a simulator — although, perhaps, denoting a slightly different meaning of simulator — is the use of a placebo drug, a formulation that simulates the active drug in trials of drug efficacy (see Placebo (origins of technical term)). Improving patient safety Patient safety is a concern in the medical industry. Patients have been known to suffer injuries and even death due to management error, and lack of using best standards of care and training. According to Building a National Agenda for Simulation-Based Medical Education (Eder-Van Hook, Jackie, 2004), “A health care provider’s ability to react prudently in an unexpected situation is one of the most critical factors in creating a positive outcome in medical emergency, regardless of whether it occurs on the battlefield, freeway, or hospital emergency room.” simulation. Simulation 86 Eder-Van Hook (2004) also noted that medical errors kill up to 98,000 with an estimated cost between $37 and $50 million and $17 to $29 billion for preventable adverse events dollars per year. “Deaths due to preventable adverse events exceed deaths attributable to motor vehicle accidents, breast cancer, or AIDS” Eder-Van Hook (2004). With these types of statistics it is no wonder that improving patient safety is a prevalent concern in the industry. Innovative simulation training solutions are now being used to train medical professionals in an attempt to reduce the number of safety concerns that have adverse effects on the patients. However, according to the article Does Simulation Improve Patient Safety? Self-efficacy, Competence, Operational Performance, and Patient Safety (Nishisaki A., Keren R., and Nadkarni, V., 2007), the jury is still out. Nishisaki states that “There is good evidence that simulation training improves provider and team self-efficacy and competence on manikins. There is also good evidence that procedural simulation improves actual operational performance in clinical settings.[25] However, no evidence yet shows that crew resource management training through simulation, despite its promise, improves team operational performance at the bedside. Also, no evidence to date proves that simulation training actually improves patient outcome. Even so, confidence is growing in the validity of medical simulation as the training tool of the future.” This could be because there are not enough research studies yet conducted to effectively determine the success of simulation initiatives to improve patient safety. Examples of [recently implemented] research simulations used to improve patient care [and its funding] can be found at Improving Patient Safety through Simulation Research (US Department of Human Health Services) http://www.ahrq.gov/qual/simulproj.htm. One such attempt to improve patient safety through the use of simulations training is pediatric care to deliver just-in-time service or/and just-in-place. This training consists of 20 minutes of simulated training just before workers report to shift. It is hoped that the recentness of the training will increase the positive and reduce the negative results that have generally been associated with the procedure. The purpose of this study is to determine if just-in-time training improves patient safety and operational performance of orotracheal intubation and decrease occurrences of undesired associated events and “to test the hypothesis that high fidelity simulation may enhance the training efficacy and patient safety in simulation settings.” The conclusion as reported in Abstract P38: Just-In-Time Simulation Training Improves ICU Physician Trainee Airway Resuscitation Participation without Compromising Procedural Success or Safety (Nishisaki A., 2008), were that simulation training improved resident participation in real cases; but did not sacrifice the quality of service. It could be therefore hypothesized that by increasing the number of highly trained residents through the use of simulation training, that the simulation training does in fact increase patient safety. This hypothesis would have to be researched for validation and the results may or may not generalize to other situations. History of simulation in healthcare The first medical simulators were simple models of human patients.[26] Since antiquity, these representations in clay and stone were used to demonstrate clinical features of disease states and their effects on humans. Models have been found from many cultures and continents. These models have been used in some cultures (e.g., Chinese culture) as a "diagnostic" instrument, allowing women to consult male physicians while maintaining social laws of modesty. Models are used today to help students learn the anatomy of the musculoskeletal system and organ systems.[26] Simulation 87 Type of models Active models Active models that attempt to reproduce living anatomy or physiology are recent developments. The famous “Harvey” mannequin was developed at the University of Miami and is able to recreate many of the physical findings of the cardiology examination, including palpation, auscultation, and electrocardiography.[27] Interactive models More recently, interactive models have been developed that respond to actions taken by a student or physician.[27] Until recently, these simulations were two dimensional computer programs that acted more like a textbook than a patient. Computer simulations have the advantage of allowing a student to make judgments, and also to make errors. The process of iterative learning through assessment, evaluation, decision making, and error correction creates a much stronger learning environment than passive instruction. Computer simulators Simulators have been proposed as an ideal tool for assessment of students for clinical skills.[28] For patients, "cybertherapy" can be used for sessions simulating traumatic expericences, from fear of heights to social anxiety.[29] Programmed patients and simulated clinical situations, including mock disaster drills, have been used extensively for education and evaluation. These “lifelike” simulations are expensive, and lack reproducibility. A fully functional "3Di" simulator would be the most specific tool available for teaching and measurement of clinical skills. Gaming platforms have been applied to create 3DiTeams learner is percussing the patient's chest in virtual field hospital these virtual medical environments to create an interactive method for learning and application of information in a clinical context.[30][31] Immersive disease state simulations allow a doctor or HCP to experience what a disease actually feels like. Using sensors and transducers symptomatic effects can be delivered to a participant allowing them to experience the patients disease state. Such a simulator meets the goals of an objective and standardized examination for clinical competence.[32] This system is superior to examinations that use "standard patients" because it permits the quantitative measurement of competence, as well as reproducing the same objective findings.[33] Simulation in entertainment Simulation in entertainment encompasses many large and popular industries such as film, television, video games (including serious games) and rides in theme parks. Although modern simulation is thought to have its roots in training and the military, in the 20th century it also became a conduit for enterprises which were more hedonistic in nature. Advances in technology in the 1980s and 1990s caused simulation to become more widely used and it began to appear in movies such as Jurassic Park (1993) and in computer-based games such as Atari’s Battlezone (1980). Simulation 88 History Early history (1940’s and 50’s) The first simulation game may have been created as early as 1947 by Thomas T. Goldsmith Jr. and Estle Ray Mann. This was a straightforward game that simulated a missile being fired at a target. The curve of the missile and its speed could be adjusted using several knobs. In 1958 a computer game called “Tennis for Two” was created by Willy Higginbotham which simulated a tennis game between two players who could both play at the same time using hand controls and was displayed on an oscilloscope.[34] This was one of the first electronic video games to use a graphical display. Modern simulation (1980’s-present) Advances in technology in the 1980s made the computer more affordable and more capable than they were in previous decades [35] which facilitated the rise of computer such as the Xbox gaming. The first video game consoles released in the 1970s and early 1980s fell prey to the industry crash in 1983, but in 1985, Nintendo released the Nintendo Entertainment System (NES) which became the best selling console in video game history.[36] In the 1990s, computer games became widely popular with the release of such game as The Sims and Command & Conquer and the still increasing power of desktop computers. Today, computer simulation games such as World of Warcraft are played by millions of people around the world. Computer-generated imagery was used in film to simulate objects as early as 1976, though in 1982, the film Tron was the first film to use computer-generated imagery for more than a couple of minutes. However, the commercial failure of the movie may have caused the industry to step away from the technology.[37] In 1993, the film Jurassic Park became the first popular film to use computer-generated graphics extensively, integrating the simulated dinosaurs almost seamlessly into live action scenes. This event transformed the film industry; in 1995, the film Toy Story was the first film to use only computer-generated images and by the new millennium computer generated graphics were the leading choice for special effects in films.[38] Simulators have been used for entertainment since the Link Trainer in the 1930s.[39] The first modern simulator ride to open at a theme park was Disney’s Star Tours in 1987 soon followed by Universal’s The Funtastic World of Hanna-Barbera in 1990 which was the first ride to be done entirely with computer graphics.[40] Examples of entertainment simulation Computer and video games Simulation games, as opposed to other genres of video and computer games, represent or simulate an environment accurately. Moreover, they represent the interactions between the playable characters and the environment realistically. These kinds of games are usually more complex in terms of game play.[41] Simulation games have become incredibly popular among people of all ages.[42] Popular simulation games include SimCity, Tiger Woods PGA Tour and Virtonomics. There are also Flight Simulation and Driving Simulation games. Film Computer-generated imagery is “the application of the field of 3D computer graphics to special effects”. This technology is used for visual effects because they are high in quality, controllable, and can create effects that would not be feasible using any other technology either because of cost, resources or safety.[43] Computer-generated graphics can be seen in many live action movies today, especially those of the action genre. Further, computer generated imagery has almost completely supplanted hand-drawn animation in children's movies which are increasingly computer-generated only. Examples of movies that use computer-generated imagery include Finding Nemo, 300 and Iron Man. Simulation 89 Theme park rides Simulator rides are the progeny of military training simulators and commercial simulators, but they are different in a fundamental way. While military training simulators react realistically to the input of the trainee in real time, ride simulators only feel like they move realistically and move according to prerecorded motion scripts.[40] One of the first simulator rides, Star Tours, which cost $32 million, used a hydraulic motion based cabin. The movement was programmed by a joystick. Today’s simulator rides, such as The Amazing Adventures of Spider-Man include elements to increase the amount of immersion experienced by the riders such as: 3D imagery, physical effects (spraying water or producing scents), and movement through an environment.[44] Examples of simulation rides include Mission Space and The Simpsons Ride. There are many simulation rides at themeparks like Disney, Universal etc., Examples are Flint Stones, Earth Quake, Time Machine, King Kong. Simulation and manufacturing Manufacturing represents one of the most important applications of Simulation. This technique represents a valuable tool used by engineers when evaluating the effect of capital investment in equipments and physical facilities like factory plants, warehouses, and distribution centers. Simulation can be used to predict the performance of an existing or planned system and to compare alternative solutions for a particular design problem.[45] Another important goal of manufacturing-simulations is to quantify system performance. Common measures of system performance include the following:[46] • Throughput under average and peak loads; • System cycle time (how long it take to produce one part); • Utilization of resource, labor, and machines; • Bottlenecks and choke points; • Queuing at work locations; • Queuing and delays caused by material-handling devices and systems; • WIP storages needs; • Staffing requirements; • Effectiveness of scheduling systems; • Effectiveness of control systems. More examples in various areas Automobile simulator An automobile simulator provides an opportunity to reproduce the characteristics of real vehicles in a virtual environment. It replicates the external factors and conditions with which a vehicle interacts enabling a driver to feel as if they are sitting in the cab of their own vehicle. Scenarios and events are replicated with sufficient reality to ensure that drivers become fully immersed in the experience rather than simply viewing it as an educational experience. The simulator provides a constructive experience for the novice driver and enables more complex exercises to be undertaken by the more A soldier tests out a heavy-wheeled-vehicle mature driver. For novice drivers, truck simulators provide an driver simulator. opportunity to begin their career by applying best practice. For mature drivers, simulation provides the ability to enhance good driving or to detect poor practice and to suggest the necessary steps for remedial action. For companies, it provides an opportunity to educate staff in the driving skills Simulation 90 that achieve reduced maintenance costs, improved productivity and, most importantly, to ensure the safety of their actions in all possible situations. Biomechanics simulators An open-source simulation platform for creating dynamic mechanical models built from combinations of rigid and deformable bodies, joints, constraints, and various force actuators. It is specialized for creating biomechanical models of human anatomical structures, with the intention to study their function and eventually assist in the design and planning of medical treatment. A biomechanics simulator is used to analyze walking dynamics, study sports performance, simulate surgical procedures, analyze joint loads, design medical devices, and animate human and animal movement. A neuromechanical simulator that combines biomechanical and biologically realistic neural network simulation. It allows the user to test hypotheses on the neural basis of behavior in a physically accurate 3-D virtual environment. City and urban simulation A city simulator can be a city-building game but can also be a tool used by urban planners to understand how cities are likely to evolve in response to various policy decisions. AnyLogic is an example of modern, large-scale urban simulators designed for use by urban planners. City simulators are generally agent-based simulations with explicit representations for land use and transportation. UrbanSim and LEAM are examples of large-scale urban simulation models that are used by metropolitan planning agencies and military bases for land use and transportation planning. Classroom of the future The "classroom of the future" will probably contain several kinds of simulators, in addition to textual and visual learning tools. This will allow students to enter the clinical years better prepared, and with a higher skill level. The advanced student or postgraduate will have a more concise and comprehensive method of retraining — or of incorporating new clinical procedures into their skill set — and regulatory bodies and medical institutions will find it easier to assess the proficiency and competency of individuals. The classroom of the future will also form the basis of a clinical skills unit for continuing education of medical personnel; and in the same way that the use of periodic flight training assists airline pilots, this technology will assist practitioners throughout their career. The simulator will be more than a "living" textbook, it will become an integral a part of the practice of medicine. The simulator environment will also provide a standard platform for curriculum development in institutions of medical education. Communication satellite simulation Modern satellite communications systems (SatCom) are often large and complex with many interacting parts and elements. In addition, the need for broadband connectivity on a moving vehicle has increased dramatically in the past few years for both commercial and military applications. To accurately predict and deliver high quality of service, satcom system designers have to factor in terrain as well as atmospheric and meteorological conditions in their planning. To deal with such complexity, system designers and operators increasingly turn towards computer models of their systems to simulate real world operational conditions and gain insights in to usability and requirements prior to final product sign-off. Modeling improves the understanding of the system by enabling the SatCom system designer or planner to simulate real world performance by injecting the models with multiple hypothetical atmospheric and environmental conditions. Simulation is often used in the training of civilian and military personnel.[7] This usually occurs when it is prohibitively expensive or simply too dangerous to allow trainees to use the real equipment in the real world. In such situations they will spend time learning valuable lessons in a "safe" Simulation 91 virtual environment yet living a lifelike experience (or at least it is the goal). Often the convenience is to permit mistakes during training for a safety-critical system. For example, in simSchool teachers practice classroom management and teaching techniques on simulated students, which avoids "learning on the job" that can damage real students. Digital Lifecycle Simulation Simulation solutions are being increasingly integrated with CAx (CAD, CAM, CAE....) solutions and processes. The use of simulation throughout the product lifecycle, especially at the earlier concept and design stages, has the potential of providing substantial benefits. These benefits range from direct cost issues such as reduced prototyping and shorter time-to-market, to better performing products and higher margins. However, for some companies, simulation has not provided the expected benefits. The research firm Aberdeen Group has found that nearly all Simulation of airflow over an engine best-in-class manufacturers use simulation early in the design process as compared to 3 or 4 laggards who do not. The successful use of simulation, early in the lifecycle, has been largely driven by increased integration of simulation tools with the entire CAD, CAM and PLM solution-set. Simulation solutions can now function across the extended enterprise in a multi-CAD environment, and include solutions for managing simulation data and processes and ensuring that simulation results are made part of the product lifecycle history. The ability to use simulation across the entire lifecycle has been enhanced through improved user interfaces such as tailorable user interfaces and "wizards" which allow all appropriate PLM participants to take part in the simulation process. Disaster preparedness and simulation training Simulation training has become a method for preparing people for disasters. Simulations can replicate emergency situations and track how learners respond thanks to a lifelike experience. Disaster preparedness simulations can involve training on how to handle terrorism attacks, natural disasters, pandemic outbreaks, or other life-threatening emergencies. One organization that has used simulation training for disaster preparedness is CADE (Center for Advancement of Distance Education). CADE[47] has used a video game to prepare emergency workers for multiple types of attacks. As reported by News-Medical.Net, ”The video game is the first in a series of simulations to address bioterrorism, pandemic flu, smallpox and other disasters that emergency personnel must prepare for.[48]” Developed by a team from the University of Illinois at Chicago (UIC), the game allows learners to practice their emergency skills in a safe, controlled environment. The Emergency Simulation Program (ESP) at the British Columbia Institute of Technology (BCIT), Vancouver, British Columbia, Canada is another example of an organization that uses simulation to train for emergency situations. ESP uses simulation to train on the following situations: forest fire fighting, oil or chemical spill response, earthquake response, law enforcement, municipal fire fighting, hazardous material handling, military training, and response to terrorist attack [49] One feature of the simulation system is the implementation of “Dynamic Run-Time Clock,” which allows simulations to run a 'simulated' time frame, 'speeding up' or 'slowing down' time as desired”[49] Additionally, the system allows session recordings, picture-icon based navigation, file storage of individual simulations, multimedia components, and launch external applications. At the University of Québec in Chicoutimi, a research team at the outdoor research and expertise laboratory (Laboratoire d'Expertise et de Recherche en Plein Air - LERPA) specializes in using wilderness backcountry accident simulations to verify emergency response coordination. Simulation 92 Instructionally, the benefits of emergency training through simulations are that learner performance can be tracked through the system. This allows the developer to make adjustments as necessary or alert the educator on topics that may require additional attention. Other advantages are that the learner can be guided or trained on how to respond appropriately before continuing to the next emergency segment—this is an aspect that may not be available in the live-environment. Some emergency training simulators also allows for immediate feedback, while other simulations may provide a summary and instruct the learner to engage in the learning topic again. In a live-emergency situation, emergency responders do not have time to waste. Simulation-training in this environment provides an opportunity for learners to gather as much information as they can and practice their knowledge in a safe environment. They can make mistakes without risk of endangering lives and be given the opportunity to correct their errors to prepare for the real-life emergency. Economics simulation In economics and especially macroeconomics, the effects of proposed policy actions, such as fiscal policy changes or monetary policy changes, are simulated to judge their desirability. A mathematical model of the economy, having been fitted to historical economic data, is used as a proxy for the actual economy; proposed values of government spending, taxation, open market operations, etc. are used as inputs to the simulation of the model, and various variables of interest such as the inflation rate, the unemployment rate, the balance of trade deficit, the government budget deficit, etc. are the outputs of the simulation. The simulated values of these variables of interest are compared for different proposed policy inputs to determine which set of outcomes is most desirable. Engineering, technology or process simulation Simulation is an important feature in engineering systems or any system that involves many processes. For example in electrical engineering, delay lines may be used to simulate propagation delay and phase shift caused by an actual transmission line. Similarly, dummy loads may be used to simulate impedance without simulating propagation, and is used in situations where propagation is unwanted. A simulator may imitate only a few of the operations and functions of the unit it simulates. Contrast with: emulate.[50] Most engineering simulations entail mathematical modeling and computer assisted investigation. There are many cases, however, where mathematical modeling is not reliable. Simulation of fluid dynamics problems often require both mathematical and physical simulations. In these cases the physical models require dynamic similitude. Physical and chemical simulations have also direct realistic uses, rather than research uses; in chemical engineering, for example, process simulations are used to give the process parameters immediately used for operating chemical plants, such as oil refineries. Equipment simulation Due to the dangerous and expensive nature of training on heavy equipment, simulation has become a common solution across many industries. Types of simulated equipment include cranes, mining reclaimers and construction equipment, among many others. Often the simulation units will include pre-built scenarios by which to teach trainees, as well as the ability to customize new scenarios. Such equipment simulators are intended to create a safe and cost effective alternative to training on live equipment.[51] Ergonomics simulation Ergonomic simulation involves the analysis of virtual products or manual tasks within a virtual environment. In the engineering process, the aim of ergonomics is to develop and to improve the design of products and work environments.[52] Ergonomic simulation utilizes an anthropometric virtual representation of the human, commonly referenced as a mannequin or Digital Human Models (DHMs), to mimic the postures, mechanical loads, and performance of a human operator in a simulated environment such as an airplane, automobile, or manufacturing Simulation 93 facility. DHMs are recognized as evolving and valuable tool for performing proactive ergonomics analysis and design.[53] The simulations employ 3D-graphics and physics-based models to animate the virtual humans. Ergonomics software uses inverse kinematics (IK) capability for posing the DHMs.[52] Several ergonomic simulation tools have been developed including Jack, SAFEWORK, RAMSIS, and SAMMIE. The software tools typically calculate biomechanical properties including individual muscle forces, joint forces and moments. Most of these tools employ standard ergonomic evaluation methods such as the NIOSH lifting equation and Rapid Upper Limb Assessment (RULA). Some simulations also analyze physiological measures including metabolism, energy expenditure, and fatigue limits Cycle time studies, design and process validation, user comfort, reachability, and line of sight are other human-factors that may be examined in ergonomic simulation packages.[54] Modeling and simulation of a task can be performed by manually manipulating the virtual human in the simulated environment. Some ergonomics simulation software permits interactive, real-time simulation and evaluation through actual human input via motion capture technologies. However, motion capture for ergonomics requires expensive equipment and the creation of props to represent the environment or product. Some applications of ergonomic simulation in include analysis of solid waste collection, disaster management tasks, interactive gaming,[55] automotive assembly line,[56] virtual prototyping of rehabilitation aids,[57] and aerospace product design.[58] Ford engineers use ergonomics simulation software to perform virtual product design reviews. Using engineering data, the simulations assist evaluation of assembly ergonomics. The company uses Siemen’s Jack and Jill ergonomics simulation software in improving worker safety and efficiency, without the need to build expensive prototypes.[59] Finance simulation In finance, computer simulations are often used for scenario planning. Risk-adjusted net present value, for example, is computed from well-defined but not always known (or fixed) inputs. By imitating the performance of the project under evaluation, simulation can provide a distribution of NPV over a range of discount rates and other variables. Simulations are frequently used in financial training to engage participants in experiencing various historical as well as fictional situations. There are stock market simulations, portfolio simulations, risk management simulations or models and forex simulations. Such simulations are typically based on stochastic asset models. Using these simulations in a training program allows for the application of theory into a something akin to real life. As with other industries, the use of simulations can be technology or case-study driven. Flight simulation Flight Simulation Training Devices (FSTD) are used to train pilots on the ground. In comparison to training in an actual aircraft, simulation based training allows for the training of maneuvers or situations that may be impractical (or even dangerous) to perform in the aircraft, while keeping the pilot and instructor in a relatively low-risk environment on the ground. For example, electrical system failures, instrument failures, hydraulic system failures, and even flight control failures can be simulated without risk to the pilots or an aircraft. Instructors can also provide students with a higher concentration of training tasks in a given period of time than is usually possible in the aircraft. For example, conducting multiple instrument approaches in the actual aircraft may require significant time spent repositioning the aircraft, while in a simulation, as soon as one approach has been completed, the instructor can immediately preposition the simulated aircraft to an ideal (or less than ideal) location from which to begin the next approach. Flight simulation also provides an economic advantage over training in an actual aircraft. Once fuel, maintenance, and insurance costs are taken into account, the operating costs of an FSTD are usually substantially lower than the operating costs of the simulated aircraft. For some large transport category airplanes, the operating costs may be several times lower for the FSTD than the actual aircraft. Simulation 94 Some people who use simulator software, especially flight simulator software, build their own simulator at home. Some people — to further the realism of their homemade simulator — buy used cards and racks that run the same software used by the original machine. While this involves solving the problem of matching hardware and software — and the problem that hundreds of cards plug into many different racks — many still find that solving these problems is well worthwhile. Some are so serious about realistic simulation that they will buy real aircraft parts, like complete nose sections of written-off aircraft, at aircraft boneyards. This permits people to simulate a hobby that they are unable to pursue in real life. Marine simulators Bearing resemblance to flight simulators, marine simulators train ships' personnel. The most common marine simulators include: • Ship's bridge simulators • Engine room simulators • Cargo handling simulators • Communication / GMDSS simulators • ROV simulators Simulators like these are mostly used within maritime colleges, training institutions and navies. They often consist of a replication of a ships' bridge, with operating console(s), and a number of screens on which the virtual surroundings are projected. Military simulations Military simulations, also known informally as war games, are models in which theories of warfare can be tested and refined without the need for actual hostilities. They exist in many different forms, with varying degrees of realism. In recent times, their scope has widened to include not only military but also political and social factors (for example, the NationLab series of strategic exercises in Latin America).[60] While many governments make use of simulation, both individually and collaboratively, little is known about the model's specifics outside professional circles. Payment and securities settlement system simulations Simulation techniques have also been applied to payment and securities settlement systems. Among the main users are central banks who are generally responsible for the oversight of market infrastructure and entitled to contribute to the smooth functioning of the payment systems. Central banks have been using payment system simulations to evaluate things such as the adequacy or sufficiency of liquidity available ( in the form of account balances and intraday credit limits) to participants (mainly banks) to allow efficient settlement of payments.[61][62] The need for liquidity is also dependent on the availability and the type of netting procedures in the systems, thus some of the studies have a focus on system comparisons.[63] Another application is to evaluate risks related to events such as communication network breakdowns or the inability of participants to send payments (e.g. in case of possible bank failure).[64] This kind of analysis falls under the concepts of Stress testing or scenario analysis. A common way to conduct these simulations is to replicate the settlement logics of the real payment or securities settlement systems under analysis and then use real observed payment data. In case of system comparison or system development, naturally also the other settlement logics need to be implemented. To perform stress testing and scenario analysis, the observed data needs to be altered, e.g. some payments delayed or removed. To analyze the levels of liquidity, initial liquidity levels are varried. System comparisons (benchmarking)or evaluations of new netting algorithms or rules are performed by running simulations with a fixed set of data and varying only the system setups. Simulation 95 Inference is usually done by comparing the benchmark simulation results to the results of altered simulation setups by comparing indicators such as unsettled transactions or settlement delays. Robotics simulators A robotics simulator is used to create embedded applications for a specific (or not) robot without being dependent on the 'real' robot. In some cases, these applications can be transferred to the real robot (or rebuilt) without modifications. Robotics simulators allow reproducing situations that cannot be 'created' in the real world because of cost, time, or the 'uniqueness' of a resource. A simulator also allows fast robot prototyping. Many robot simulators feature physics engines to simulate a robot's dynamics. Production simulation Simulations of production systems is used mainly to examine the effect of improvements or investments in a production system. Most often this is done using a static spreadsheet with process times and transportation times. For more sophisticated simulations Discrete Event Simulation (DES) is used with the advantages to simulate dynamics in the production system. A production system is very much dynamic depending on variations in manufacturing processes, assembly times, machine set-ups, breaks, breakdowns and small stoppages.[65] There are lots of programs commonly used for discrete event simulation. They differ in usability and markets but do often share the same foundation. There is an academic project investigating the possibilities to use production simulation software for ecology labeling, named EcoProIT. Sales process simulators Simulations are useful in modeling the flow of transactions through business processes, such as in the field of sales process engineering, to study and improve the flow of customer orders through various stages of completion (say, from an initial proposal for providing goods/services through order acceptance and installation). Such simulations can help predict the impact of how improvements in methods might impact variability, cost, labor time, and the quantity of transactions at various stages in the process. A full-featured computerized process simulator can be used to depict such models, as can simpler educational demonstrations using spreadsheet software, pennies being transferred between cups based on the roll of a die, or dipping into a tub of colored beads with a scoop.[66] Sports Simulation In sports, computer simulations are often done to predict the outcome of games and the performance of individual players. They attempt to recreate the game through models built from statistics. The increase in technology has allowed anyone with knowledge of programming the ability to run simulations of their models. The simulations are built from a series of mathematical algorithms, or models, and can vary with accuracy. Accuscore, which is licensed by companies such as ESPN, is a well known simulation program for all major sports. It offers detailed analysis of games through simulated betting lines, projected point totals and overall probabilities. With the increased interest in fantasy sports simulation models that predict individual player performance have gained popularity. Companies like What If Sports and StatFox specialize in not only using their simulations for predicting game results, but how well individual players will do as well. Many people use models to determine who to start in their fantasy leagues. Another way simulations are helping the sports field is in the use of biomechanics. Models are derived and simulations are run from data received from sensors attached to athletes and video equipment. Sports biomechanics aided by simulation models answer questions regarding training techniques such as: the effect of fatigue on throwing performance (height of throw) and biomechanical factors of the upper limbs (reactive strength index; hand contact time).[67] Simulation 96 Computer simulations allow us to take models which before were too complex to run, and give us answers. Simulations have proven to be some of our best insights into both play performance and team predictability. Space shuttle countdown simulation Simulation is used at Kennedy Space Center (KSC) to train and certify Space Shuttle engineers during simulated launch countdown operations. The Space Shuttle engineering community participates in a launch countdown integrated simulation before each shuttle flight. This simulation is a virtual simulation where real people interact with simulated Space Shuttle vehicle and Ground Support Equipment (GSE) hardware. The Shuttle Final Countdown Phase Simulation, also known as S0044, involves countdown processes Firing Room 1 configured for space shuttle launches that integrate many of the Space Shuttle vehicle and GSE systems. Some of the Shuttle systems integrated in the simulation are the main propulsion system, main engines, solid rocket boosters, ground liquid hydrogen and liquid oxygen, external tank, flight controls, navigation, and avionics.[68] The high-level objectives of the Shuttle Final Countdown Phase Simulation are: • To demonstrate Firing Room final countdown phase operations. • To provide training for system engineers in recognizing, reporting and evaluating system problems in a time critical environment. • To exercise the launch teams ability to evaluate, prioritize and respond to problems in an integrated manner within a time critical environment. • To provide procedures to be used in performing failure/recovery testing of the operations performed in the final countdown phase.[69] The Shuttle Final Countdown Phase Simulation takes place at the Kennedy Space Center Launch Control Center Firing Rooms. The firing room used during the simulation is the same control room where real launch countdown operations are executed. As a result, equipment used for real launch countdown operations is engaged. Command and control computers, application software, engineering plotting and trending tools, launch countdown procedure documents, launch commit criteria documents, hardware requirement documents, and any other items used by the engineering launch countdown teams during real launch countdown operations are used during the simulation. The Space Shuttle vehicle hardware and related GSE hardware is simulated by mathematical models (written in Shuttle Ground Operations Simulator (SGOS) modeling language [70]) that behave and react like real hardware. During the Shuttle Final Countdown Phase Simulation, engineers command and control hardware via real application software executing in the control consoles – just as if they were commanding real vehicle hardware. However, these real software applications do not interface with real Shuttle hardware during simulations. Instead, the applications interface with mathematical model representations of the vehicle and GSE hardware. Consequently, the simulations bypass sensitive and even dangerous mechanisms while providing engineering measurements detailing how the hardware would have reacted. Since these math models interact with the command and control application software, models and simulations are also used to debug and verify the functionality of application software.[71] Simulation 97 Satellite navigation simulators The only true way to test GNSS receivers (commonly known as Sat-Nav's in the commercial world)is by using an RF Constellation Simulator. A receiver that may for example be used on an aircraft, can be tested under dynamic conditions without the need to take it on a real flight. The test conditions can be repeated exactly, and there is full control over all the test parameters. this is not possible in the 'real-world' using the actual signals. For testing receivers that will use the new Galileo (satellite navigation) there is no alternative, as the real signals do not yet exist. Weather simulation Predicting weather conditions by extrapolating/interpolating previous data is one of the real use of simulation. Most of the weather forecats use this information published by Weather buereaus. This kind of simulations help in predicting and forewarning about extreme weather conditions like the path of an active hurricane/cyclone. Numerical weather prediction for forecasting involves complicated numeric computer models to predict weather accurately by taking many parameters in to account. Simulation and games Strategy games — both traditional and modern — may be viewed as simulations of abstracted decision-making for the purpose of training military and political leaders (see History of Go for an example of such a tradition, or Kriegsspiel for a more recent example). Many other video games are simulators of some kind. Such games can simulate various aspects of reality, from business, to government, to construction, to piloting vehicles (see above). Historical usage Historically, the word had negative connotations: …for Distinction Sake, a Deceiving by Words, is commonly called a Lye, and a Deceiving by Actions, Gestures, or Behavior, is called Simulation… —Robert South, South, 1697, p.525 However, the connection between simulation and dissembling later faded out and is now only of linguistic interest.[72] References [1] J. Banks, J. Carson, B. Nelson, D. Nicol (2001). Discrete-Event System Simulation. Prentice Hall. p. 3. ISBN 0-13-088702-1. [2] In the words of the Simulation article (http:/ / www. modelbenders. com/ encyclopedia/ encyclopedia. html) in Encyclopedia of Computer Science, "designing a model of a real or imagined system and conducting experiments with that model". [3] Sokolowski, J.A., Banks, C.M. (2009). Principles of Modeling and Simulation. Hoboken, NJ: Wiley. p. 6. ISBN 978-0-470-28943-3. [4] For example in computer graphics SIGGRAPH 2007 | For Attendees | Papers (http:/ / www. siggraph. org/ s2007/ attendees/ papers/ 12. html) Doc:Tutorials/Physics/BSoD - BlenderWiki (http:/ / wiki. blender. org/ index. php/ BSoD/ Physical_Simulation). [5] Thales defines synthetic environment as "the counterpart to simulated models of sensors, platforms and other active objects" for "the simulation of the external factors that affect them" (http:/ / www. thalesresearch. com/ Default. aspx?tabid=181) while other vendors use the term for more visual, virtual reality-style simulators (http:/ / www. cae. com/ www2004/ Products_and_Services/ Civil_Simulation_and_Training/ Simulation_Equipment/ Visual_Solutions/ Synthetic_Environments/ index. shtml). [6] For a popular research project in the field of biochemistry where "computer simulation is particularly well suited to address these questions" Folding@home - Main (http:/ / folding. stanford. edu/ Pande/ Main), see Folding@Home. [7] For an academic take on a training simulator, see e.g. Towards Building an Interactive, Scenario-based Training Simulator (http:/ / gel. msu. edu/ magerko/ papers/ 11TH-CGF-058. pdf), for medical application Medical Simulation Training Benefits (http:/ / www. immersion. com/ medical/ benefits1. php) as presented by a simulator vendor and for military practice A civilian's guide to US defense and security assistance to Latin America and the Caribbean (http:/ / ciponline. org/ facts/ exe. htm) published by Center for International Policy. [8] http:/ / www. simschool. org [9] Classification used by the Defense Modeling and Simulation Office. Simulation 98 [10] "High Versus Low Fidelity Simulations: Does the Type of Format Affect Candidates' Performance or Perceptions?" (http:/ / www. ipmaac. org/ conf/ 03/ havighurst. pdf) [11] For example All India management association (http:/ / www. aima-ind. org/ ) maintains that playing to win, participants "imbibe new forms of competitive behavior that are ideal for today's highly chaotic business conditions" (http:/ / www. aima-ind. org/ management_games. asp) and IBM claims that "the skills honed playing massive multiplayer dragon-slaying games like World of Warcraft can be useful when managing modern multinationals". [12] "Reacting to the Past Home Page" (http:/ / www. barnard. columbia. edu/ reacting/ ) [13] https:/ / sites. google. com/ site/ reactingscience/ Reacting to the Past:STEM Games [14] "Carana," at 'PaxSims' blog, 27 January 2009 (http:/ / paxsims. wordpress. com/ 2009/ 01/ 27/ carana/ ) [15] Sherman, W.R., Craig, A.B. (2003). Understanding Virtual Reality. San Francisco, CA: Morgan Kaufmann. ISBN 978-1-55860-353-0. [16] Leeb, R., Lee, F., Keinrath, C., Schere, R., Bischof, H., Pfurtscheller, G. (2007). "Brain-Computer Communication: Motivation, Aim, and Impact of Exploring a Virtual Apartment". IEEE Transactions on Neural Systems and Rehabilitation Engineering 15 (4): 473–481. doi:10.1109/TNSRE.2007.906956. [17] Zahraee, A.H., Szewczyk, J., Paik, J.K., Guillaume, M. (2010). Robotic hand-held surgical device: evaluation of end-effector’s kinematics and development of proof-of-concept prototypes. Proceedings of the 13th International Conference on Medical Image Computing and Computer Assisted Intervention, Beijing, China. [18] Ahmed K, Keeling AN, Fakhry M, Ashrafian H, Aggarwal R, Naughton PA, Darzi A, Cheshire N, et al. (January 2010). "Role of Virtual Reality Simulation in Teaching and Assessing Technical Skills in Endovascular Intervention". J Vasc Interv Radiol 21. [19] Narayan, Roger; Kumta, Prashant; Sfeir, Charles; Lee, Dong-Hyun; Choi, Daiwon; Olton, Dana (October 2004). "Nanostructured ceramics in medical devices: Applications and prospects" (http:/ / www. ingentaconnect. com/ content/ tms/ jom/ 2004/ 00000056/ 00000010/ art00011). JOM 56 (10): 38–43. Bibcode 2004JOM....56j..38N. doi:10.1007/s11837-004-0289-x. PMID 11196953. . [20] Couvreur P, Vauthier C (July 2006). "Nanotechnology: intelligent design to treat complex disease". Pharm. Res. 23 (7): 1417–50. doi:10.1007/s11095-006-0284-8. PMID 16779701. [21] Hede S, Huilgol N (2006). ""Nano": the new nemesis of cancer" (http:/ / www. cancerjournal. net/ article. asp?issn=0973-1482;year=2006;volume=2;issue=4;spage=186;epage=195;aulast=Hede). J Cancer Res Ther 2 (4): 186–95. doi:10.4103/0973-1482.29829. PMID 17998702. . [22] Leary SP, Liu CY, Apuzzo ML (June 2006). "Toward the emergence of nanoneurosurgery: part III—nanomedicine: targeted nanotherapy, nanosurgery, and progress toward the realization of nanoneurosurgery" (http:/ / meta. wkhealth. com/ pt/ pt-core/ template-journal/ lwwgateway/ media/ landingpage. htm?issn=0148-396X& volume=58& issue=6& spage=1009). Neurosurgery 58 (6): 1009–26; discussion 1009–26. doi:10.1227/01.NEU.0000217016.79256.16. PMID 16723880. . [23] http:/ / vam. anest. ufl. edu/ wip. html [24] http:/ / onlinelibrary. wiley. com/ doi/ 10. 1111/ j. 1445-2197. 2010. 05349. x/ abstract [25] Nishisaki A, Keren R, Nadkarni V (June 2007). "Does simulation improve patient safety? Self-efficacy, competence, operational performance, and patient safety" (http:/ / linkinghub. elsevier. com/ retrieve/ pii/ S1932-2275(07)00025-0). Anesthesiol Clin 25 (2): 225–36. doi:10.1016/j.anclin.2007.03.009. PMID 17574187. . [26] Meller, G. (1997). "A Typology of Simulators for Medical Education" (http:/ / www. medsim. com/ profile/ article1. html). Journal of Digital Imaging. . [27] Cooper Jeffery B, Taqueti VR (2008-12). "A brief history of the development of mannequin simulators for clinical education and training" (http:/ / pmj. bmj. com/ content/ 84/ 997/ 563. long). Postgrad Med J. 84 (997): 563–570. doi:10.1136/qshc.2004.009886. PMID 19103813. . Retrieved 2011-05-24. [28] Murphy D, Challacombe B, Nedas T, Elhage O, Althoefer K, Seneviratne L, Dasgupta P. (May 2007). "[Equipment and technology in robotics]" (in Spanish; Castilian). Arch. Esp. Urol. 60 (4): 349–55. PMID 17626526. [29] "In Cybertherapy, Avatars Assist With Healing" (http:/ / www. nytimes. com/ 2010/ 11/ 23/ science/ 23avatar. html?_r=1& ref=science). New York Times. 2010-11-22. . Retrieved 2010-11-23. [30] Dagger, Jacob (May–June 2008). Update: "The New Game Theory" (http:/ / www. dukemagazine. duke. edu/ dukemag/ issues/ 050608/ depupd. html). 94. Duke Magazine. . Retrieved 2011-02-08. [31] Steinberg, Scott (2011-01-31). "How video games can make you smarter" (http:/ / articles. cnn. com/ 2011-01-31/ tech/ video. games. smarter. steinberg_1_video-games-interactive-simulations-digital-world?_s=PM:TECH). Cable News Network (CNN Tech). . Retrieved 2011-02-08. [32] Vlaovic PD, Sargent ER, Boker JR, et al. (2008). "Immediate impact of an intensive one-week laparoscopy training program on laparoscopic skills among postgraduate urologists" (http:/ / openurl. ingenta. com/ content/ nlm?genre=article& issn=1086-8089& volume=12& issue=1& spage=1& aulast=Vlaovic). JSLS 12 (1): 1–8. PMC 3016039. PMID 18402731. . [33] Leung J, Foster E (April 2008). "How do we ensure that trainees learn to perform biliary sphincterotomy safely, appropriately, and effectively?" (http:/ / www. current-reports. com/ article_frame. cfm?PubID=GR10-2-2-03& Type=Abstract). Curr Gastroenterol Rep 10 (2): 163–8. doi:10.1007/s11894-008-0038-3. PMID 18462603. . [34] http:/ / www. pong-story. com/ intro. htm Archived (http:/ / www. webcitation. org/ 5seElSgBC) 10 September 2010 at WebCite [35] History of Computers 1980 (http:/ / homepages. vvm. com/ ~jhunt/ compupedia/ History of Computers/ history_of_computers_1980. htm) [36] "Video Game Console Timeline - Video Game History - Xbox 360 - TIME Magazine" (http:/ / www. time. com/ time/ covers/ 1101050523/ console_timeline/ ). Time. 2005-05-23. . Retrieved 2010-05-23. Simulation 99 [37] TRON - The 1982 Movie (http:/ / design. osu. edu/ carlson/ history/ tron. html) [38] http:/ / www. beanblossom. in. us/ larryy/ cgi. html [39] Link Trainer Restoration (http:/ / www. starksravings. com/ linktrainer/ linktrainer. htm) [40] http:/ / www. trudang. com/ simulatr/ simulatr. html [41] Simulation - General Information | Open-Site.org (http:/ / open-site. org/ Games/ Video_Games/ Simulation) [42] Video Games in the US Market Research | IBISWorld (http:/ / www. ibisworld. com/ industry/ retail. aspx?indid=2003& chid=1) [43] Computer-generated imagery (http:/ / www. sciencedaily. com/ articles/ c/ computer-generated_imagery. htm) [44] Bringing Spidey to Life: Kleiser-Walczak Construction Company (http:/ / www. awn. com/ mag/ issue4. 02/ 4. 02pages/ kenyonspiderman. php3) [45] Benedettini, O., Tjahjono, B. (2008). "Towards an improved tool to facilitate simulation modeling of complex manufacturing systems". International Journal of Advanced Manufacturing Technology 43 (1/2): 191–9. doi:10.1007/s00170-008-1686-z. [46] Banks, J., Carson J., Nelson B.L., Nicol, D. (2005). Discrete-event system simulation (4th ed.). Upper Saddle River, NJ: Pearson Prentice Hall. ISBN 978-0-13-088702-3. [47] CADE- http:/ / www. uic. edu/ sph/ cade/ [48] News-Medical.Net article- http:/ / www. news-medical. net/ news/ 2005/ 10/ 27/ 14106. aspx [49] Emergency Response Training (http:/ / www. straylightmm. com/ ) [50] Federal Standard 1037C [51] GlobalSim | World Class Training Simulators | Crane Simulator (http:/ / www. globalsim. com) [52] Reed, M. P., Faraway, J., Chaffin, D. B., & Martin, B. J. (2006). The HUMOSIM Ergonomics Framework: A new approach to digital human simulation for ergonomic analysis. SAE Technical Paper, 01-2365 [53] Chaffin, D. B. (2007). Human motion simulation for vehicle and workplace design. Human Factors and Ergonomics in Manufacturing & Service Industries,17(5), 475-484 [54] Jack and Process Simulate Human: Siemens PLM Software (http:/ / www. plm. automation. siemens. com/ en_us/ products/ tecnomatix/ assembly_planning/ jack/ index. shtml) [55] Bush, P. M., Gaines, S., Gammoh, F., & Wooden, S. A Comparison of Software Tools for Occupational Biomechanics and Ergonomic Research. [56] Niu, J. W., Zhang, X. W., Zhang, X., & Ran, L. H. (2010, December). Investigation of ergonomics in automotive assembly line using Jack. InIndustrial Engineering and Engineering Management (IEEM), 2010 IEEE International Conference on (pp. 1381-1385). IEEE. [57] Beitler, Matthew T., Harwin, William S., & Mahoney, Richard M. (1996) In Proceedings of the virtual prototyping of rehabilitation aids, RESNA 96, pp. 360–363. [58] G.R. Bennett. The application of virtual prototyping in development of complex aerospace products. Virtual Prototyping Journal, 1 (1) (1996), pp. 13–20 [59] From the floor of the 2012 Chicago Auto Show: Automation World shows how Ford uses the power of simulation « Siemens PLM Software Blog (http:/ / blog. industrysoftware. automation. siemens. com/ blog/ 2012/ 03/ 21/ floor-2012-chicago-auto-show-automation-world-shows-ford-power-simulation/ ) [60] The Economist provides a current (as of 2012) survey of public projects attempting to simulate some theories in "The science of civil war: What makes heroic strife" (http:/ / www. economist. com/ node/ 21553006). [61] Leinonen (ed.): Simulation studies of liquidity needs, risks and efficiency in payment networks (Bank of Finland Studies E:39/2007) Simulation publications (http:/ / pss. bof. fi/ Pages/ Publications. aspx) [62] Neville Arjani: Examining the Trade-Off between Settlement Delay and Intraday Liquidity in Canada's LVTS: A Simulation Approach (Working Paper 2006-20, Bank of Canada) Simulation publications (http:/ / pss. bof. fi/ Pages/ Publications. aspx) [63] Johnson, K. - McAndrews, J. - Soramäki, K. 'Economizing on Liquidity with Deferred Settlement Mechanisms' (Reserve Bank of New York Economic Policy Review, December 2004) [64] H. Leinonen (ed.): Simulation analyses and stress testing of payment networks (Bank of Finland Studies E:42/2009) Simulation publications (http:/ / pss. bof. fi/ Pages/ Publications. aspx) [65] Ulf, Eriksson (2005). Diffusion of Discrete Event Simulation in Swedish Industry. Gothenburg: Doktorsavhandlingar vid Chalmers tekniska högskola. ISBN 91-7291-577-3. [66] Paul H. Selden (1997). Sales Process Engineering: A Personal Workshop. Milwaukee, WI: ASQ Quality Press. ISBN 978-0-87389-418-0. [67] Harrison, Andrew J (2011). "Throwing and catching movements exhibit post-activation potentiation effects following fatigue". Sports Biomechanics 10 (3): 185–196. doi:10.1080/TNSRE10.1080/14763141.2011.592544. [68] Sikora, E.A. (2010, July 27). Space Shuttle Main Propulsion System expert, John F. Kennedy Space Center. Interview. [69] Shuttle Final Countdown Phase Simulation. National Aeronautics and Space Administration KSC Document # RTOMI S0044, Revision AF05, 2009. [70] Shuttle Ground Operations Simulator (SGOS) Summary Description Manual. National Aeronautics and Space Administration KSC Document # KSC-LPS-SGOS-1000, Revision 3 CHG-A, 1995. [71] Math Model Main Propulsion System (MPS) Requirements Document, National Aeronautics and Space Administration KSC Document # KSCL-1100-0522, Revision 9, June 2009. [72] South, in the passage quoted, was speaking of the differences between a falsehood and an honestly mistaken statement; the difference being that in order for the statement to be a lie the truth must be known, and the opposite of the truth must have been knowingly uttered. And, from Simulation 100 this, to the extent to which a lie involves deceptive words, a simulation involves deceptive actions, deceptive gestures, or deceptive behavior. Thus, it would seem, if a simulation is false, then the truth must be known (in order for something other than the truth to be presented in its stead); and, for the simulation to simulate. Because, otherwise, one would not know what to offer up in simulation. Bacon’s essay Of Simulation and Dissimulation (http:/ / www. authorama. com/ essays-of-francis-bacon-7. html) expresses somewhat similar views; it is also significant that Samuel Johnson thought so highly of South's definition, that he used it in the entry for simulation in his Dictionary of the English Language. Further reading • C. Aldrich (2003). Learning by Doing : A Comprehensive Guide to Simulations, Computer Games, and Pedagogy in e-Learning and Other Educational Experiences. San Francisco: Pfeifer — John Wiley & Sons. ISBN 978-0-7879-7735-1. • C. Aldrich (2004). Simulations and the future of learning: an innovative (and perhaps revolutionary) approach to e-learning. San Francisco: Pfeifer — John Wiley & Sons. ISBN 978-0-7879-6962-2. • Steve Cohen (2006). Virtual Decisions. Mahwah, NJ: Lawrence Erlbaum Associates. ISBN 978-0-8058-4994-3. • R. Frigg, S. Hartmann (2007). "Models in Science" (http://plato.stanford.edu/entries/models-science/). Stanford Encyclopedia of Philosophy. • S. Hartmann (1996). "The World as a Process: Simulations in the Natural and Social Sciences" (http:// philsci-archive.pitt.edu/archive/00002412/). In R. Hegselmann, et al.. Modelling and Simulation in the Social Sciences from the Philosophy of Science Point of View. Theory and Decision Library. Dordrecht: Kluwer. pp. 77–100. • J.P. Hertel (2002). Using Simulations to Promote Learning in Higher Education. Sterling, Virginia: Stylus. ISBN 978-1-57922-052-5. • P. Humphreys (2004). Extending Ourselves: Computational Science, Empiricism, and Scientific Method. Oxford: Oxford University Press. ISBN 978-0-19-515870-0. • F. Percival, S. Lodge, D. Saunders (1993). The Simulation and Gaming Yearbook: Developing Transferable Skills in Education and Training. London: Kogan Page. • James J. Nutaro, Building Software for Simulation: Theory and Algorithms, with Applications in C++. Wiley, 2010. • D. Saunders, ed. (2000). The International Simulation and Gaming Research Yearbook. London: Kogan Page. • Roger D. Smith: Simulation Article (http://www.modelbenders.com/encyclopedia/encyclopedia.html), Encyclopedia of Computer Science, Nature Publishing Group, ISBN 978-0-333-77879-1. • Roger D. Smith: "Simulation: The Engine Behind the Virtual World" (http://www.modelbenders.com/ Bookshop/techpapers.html), eMatter, December, 1999. • R. South (1688). "A Sermon Delivered at Christ-Church, Oxon., Before the University, Octob. 14. 1688: Prov. XII.22 Lying Lips are abomination to the Lord", pp. 519–657 in South, R., Twelve Sermons Preached Upon Several Occasions (Second Edition), Volume I, Printed by S.D. for Thomas Bennet, (London), 1697. • Gabriel A. Wainer (2009) Discrete-Event Modeling and Simulation: a practitioner's approach (http://cell-devs. sce.carleton.ca/mediawiki/index.php/Main_Page). CRC Press, 2009. • Eric Winsberg (1999) Sanctioning Models: The epistemology of simulation (http://www.cas.usf.edu/~ewinsb/ SiC_Eric_Winsberg.pdf), in Sismondo, Sergio and Snait Gissis (eds.) (1999), Modeling and Simulation. Special Issue of Science in Context 12. • Eric Winsberg (2001). "Simulations, Models and Theories: Complex Physical Systems and their Representations". Philosophy of Science 68: 442–454. • Eric Winsberg (2003). "Simulated Experiments: Methodology for a Virtual World" (http://www.cas.usf.edu/ ~ewinsb/methodology.pdf) (PDF). Philosophy of Science 70: 105–125. doi:10.1086/367872. • Joseph Wolfe, David Crookall (1998). "Developing a scientific knowledge of simulation/gaming" (http://sag. sagepub.com/cgi/reprint/29/1/7). Simulation & Gaming: an International Journal of Theory, Design and Research 29 (1): 7–19. Simulation 101 • Ellen K. Levy (2004). "Synthetic Lighting: Complex Simulations of Nature". Photography Quarterly (88): 5–9. • Paul Humphreys and Cyrille Imbert, ed. (2012). Models, Simulations and Representations. London and New York: Routledge. ISBN 978-0-203-80841-2 (ebk). • World Simulations External links • Bibliographies containing more references (http://www.unice.fr/sg/resources/bibliographies.htm) to be found on the website of the journal Simulation & Gaming (http://www.unice.fr/sg/). Probability distribution In probability and statistics, a probability distribution assigns a probability to each of the possible outcomes of a random experiment. Examples are found in experiments whose sample space is non-numerical, where the distribution would be a categorical distribution; experiments whose sample space is encoded by discrete random variables, where the distribution is a probability mass function; and experiments with sample spaces encoded by continuous random variables, where the distribution is a probability density function. More complex experiments, such as those involving stochastic processes defined in continuous-time, may demand the use of more general probability measures. In applied probability, a probability distribution can be specified in a number of different ways, often chosen for mathematical convenience: • by supplying a valid probability mass function or probability density function • by supplying a valid cumulative distribution function or survival function • by supplying a valid hazard function • by supplying a valid characteristic function • by supplying a rule for constructing a new random variable from other random variables whose joint probability distribution is known. Important and commonly encountered probability distributions include the binomial distribution, the hypergeometric distribution, and the normal distribution. Introduction To define probability distributions for the simplest cases, one needs to distinguish between discrete and continuous random variables. In the discrete case, one can easily assign a probability to each possible value: for example, when throwing a die, each of the six values 1 to 6 has the probability 1/6. In contrast, when a random variable takes values from a continuum, probabilities are nonzero only if they refer to finite intervals: in quality control one might demand that the probability of a "500 g" package containing between 490 g and 510 g should be no less than 98%. Discrete probability distribution for the sum of two dice. Probability distribution 102 If the random variable is real-valued (or more generally, if a total order is defined for its possible values), the cumulative distribution function gives the probability that the random variable is no larger than a given value; in the real-valued case it is the integral of the density. Terminology Normal distribution, also called Gaussian or "bell curve", the most As probability theory is used in quite diverse important continuous random distribution. applications, terminology is not uniform and sometimes confusing. The following terms are used for non-cumulative probability distribution functions: • Probability mass, Probability mass function, p.m.f.: for discrete random variables. • Categorical distribution: for discrete random variables with a finite set of values. • Probability density, Probability density function, p.d.f: Most often reserved for continuous random variables. The following terms are somewhat ambiguous as they can refer to non-cumulative or cumulative distributions, depending on authors' preferences: • Probability distribution function: Continuous or discrete, non-cumulative or cumulative. • Probability function: Even more ambiguous, can mean any of the above, or anything else. Finally, • Probability distribution: Either the same as probability distribution function. Or understood as something more fundamental underlying an actual mass or density function. Basic terms • Mode: most frequently occurring value in a distribution • Tail: region of least frequently occurring values in a distribution • Support: the smallest closed set whose complement has probability zero. It may be understood as the points or elements that are actual members of the distribution. Discrete probability distribution A discrete probability distribution shall be understood as a probability distribution characterized by a probability mass function. Thus, the distribution of a random variable X is discrete, and X is then called a discrete random variable, if The probability mass function of a discrete as u runs through the set of all possible values of X. It follows that such probability distribution. The probabilities of the a random variable can assume only a finite or countably infinite singletons {1}, {3}, and {7} are respectively 0.2, number of values. 0.5, 0.3. A set not containing any of these points has probability zero. In cases more frequently considered, this set of possible values is a topologically discrete set in the sense that all its points are isolated points. But there are discrete random variables for which this countable set is dense on the real line (for example, a distribution over rational numbers). Probability distribution 103 Among the most well-known discrete probability distributions that are used for statistical modeling are the Poisson distribution, the Bernoulli distribution, the binomial distribution, the geometric distribution, and the negative binomial distribution. In addition, the discrete uniform distribution is commonly used in computer programs that make The cdf of a discrete probability distribution, ... equal-probability random selections between a number of choices. Cumulative density Equivalently to the above, a discrete random variable can be defined as a random variable whose cumulative distribution function (cdf) increases only by jump discontinuities—that is, its cdf increases only ... of a continuous probability distribution, ... where it "jumps" to a higher value, and is constant between those jumps. The points where jumps occur are precisely the values which the random variable may take. The number of such jumps may be finite or countably infinite. The set of locations of such jumps need not be topologically discrete; for example, the cdf might jump at each rational number. ... of a distribution which has both a continuous part and a discrete part. Delta-function representation Consequently, a discrete probability distribution is often represented as a generalized probability density function involving Dirac delta functions, which substantially unifies the treatment of continuous and discrete distributions. This is especially useful when dealing with probability distributions involving both a continuous and a discrete part. Indicator-function representation For a discrete random variable X, let u0, u1, ... be the values it can take with non-zero probability. Denote These are disjoint sets, and by formula (1) It follows that the probability that X takes any value except for u0, u1, ... is zero, and thus one can write X as except on a set of probability zero, where is the indicator function of A. This may serve as an alternative definition of discrete random variables. Continuous probability distribution A continuous probability distribution is a probability distribution that has a probability density function. Mathematicians also call such a distribution absolutely continuous, since its cumulative distribution function is absolutely continuous with respect to the Lebesgue measure λ. If the distribution of X is continuous, then X is called a continuous random variable. There are many examples of continuous probability distributions: normal, uniform, chi-squared, and others. Probability distribution 104 Intuitively, a continuous random variable is the one which can take a continuous range of values — as opposed to a discrete distribution, where the set of possible values for the random variable is at most countable. While for a discrete distribution an event with probability zero is impossible (e.g. rolling 3½ on a standard die is impossible, and has probability zero), this is not so in the case of a continuous random variable. For example, if one measures the width of an oak leaf, the result of 3½ cm is possible, however it has probability zero because there are uncountably many other potential values even between 3 cm and 4 cm. Each of these individual outcomes has probability zero, yet the probability that the outcome will fall into the interval (3 cm, 4 cm) is nonzero. This apparent paradox is resolved by the fact that the probability that X attains some value within an infinite set, such as an interval, cannot be found by naively adding the probabilities for individual values. Formally, each value has an infinitesimally small probability, which statistically is equivalent to zero. Formally, if X is a continuous random variable, then it has a probability density function ƒ(x), and therefore its probability of falling into a given interval, say [a, b] is given by the integral In particular, the probability for X to take any single value a (that is a ≤ X ≤ a) is zero, because an integral with coinciding upper and lower limits is always equal to zero. The definition states that a continuous probability distribution must possess a density, or equivalently, its cumulative distribution function be absolutely continuous. This requirement is stronger than simple continuity of the cdf, and there is a special class of distributions, singular distributions, which are neither continuous nor discrete nor their mixture. An example is given by the Cantor distribution. Such singular distributions however are never encountered in practice. Note on terminology: some authors use the term "continuous distribution" to denote the distribution with continuous cdf. Thus, their definition includes both the (absolutely) continuous and singular distributions. By one convention, a probability distribution is called continuous if its cumulative distribution function is continuous and, therefore, the probability measure of singletons for all . Another convention reserves the term continuous probability distribution for absolutely continuous distributions. These distributions can be characterized by a probability density function: a non-negative Lebesgue integrable function defined on the real numbers such that Discrete distributions and some continuous distributions (like the Cantor distribution) do not admit such a density. Probability distributions of scalar random variables The following applies to all types of scalar random variables. Because a probability distribution Pr on the real line is determined by the probability of a scalar random variable X being in a half-open interval (-∞, x], the probability distribution is completely characterized by its cumulative distribution function: Some properties • The probability distribution of the sum of two independent random variables is the convolution of each of their distributions. • Probability distributions are not a vector space – they are not closed under linear combinations, as these do not preserve non-negativity or total integral 1 – but they are closed under convex combination, thus forming a convex subset of the space of functions (or measures). Probability distribution 105 Kolmogorov definition In the measure-theoretic formalization of probability theory, a random variable is defined as a measurable function X from a probability space to measurable space . A probability distribution is the pushforward measure, P, satisfying X*P = PX −1 on . Random number generation A frequent problem in statistical simulations (the Monte Carlo method) is the generation of pseudo-random numbers that are distributed in a given way. Most algorithms are based on a pseudorandom number generator that produces numbers X that are uniformly distributed in the interval [0,1). These random variates X are then transformed via some algorithm to create a new random variate having the required probability distribution. Applications The concept of the probability distribution and the random variables which they describe underlies the mathematical discipline of probability theory, and the science of statistics. There is spread or variability in almost any value that can be measured in a population (e.g. height of people, durability of a metal, sales growth, traffic flow, etc.); almost all measurements are made with some intrinsic error; in physics many processes are described probabilistically, from the kinetic properties of gases to the quantum mechanical description of fundamental particles. For these and many other reasons, simple numbers are often inadequate for describing a quantity, while probability distributions are often more appropriate. As a more specific example of an application, the cache language models and other statistical language models used in natural language processing to assign probabilities to the occurrence of particular words and word sequences do so by means of probability distributions. Common probability distributions The following is a list of some of the most common probability distributions, grouped by the type of process that they are related to. For a more complete list, see list of probability distributions, which groups by the nature of the outcome being considered (discrete, continuous, multivariate, etc.) Note also that all of the univariate distributions below are singly peaked; that is, it is assumed that the values cluster around a single point. In practice, actually observed quantities may cluster around multiple values. Such quantities can be modeled using a mixture distribution. Related to real-valued quantities that grow linearly (e.g. errors, offsets) • Normal distribution (Gaussian distribution), for a single such quantity; the most common continuous distribution Related to positive real-valued quantities that grow exponentially (e.g. prices, incomes, populations) • Log-normal distribution, for a single such quantity whose log is normally distributed • Pareto distribution, for a single such quantity whose log is exponentially distributed; the prototypical power law distribution Related to real-valued quantities that are assumed to be uniformly distributed over a (possibly unknown) region • Discrete uniform distribution, for a finite set of values (e.g. the outcome of a fair die) • Continuous uniform distribution, for continuously distributed values Probability distribution 106 Related to Bernoulli trials (yes/no events, with a given probability) • Basic distributions: • Bernoulli distribution, for the outcome of a single Bernoulli trial (e.g. success/failure, yes/no) • Binomial distribution, for the number of "positive occurrences" (e.g. successes, yes votes, etc.) given a fixed total number of independent occurrences • Negative binomial distribution, for binomial-type observations but where the quantity of interest is the number of failures before a given number of successes occurs • Geometric distribution, for binomial-type observations but where the quantity of interest is the number of failures before the first success; a special case of the negative binomial distribution • Related to sampling schemes over a finite population: • Hypergeometric distribution, for the number of "positive occurrences" (e.g. successes, yes votes, etc.) given a fixed number of total occurrences, using sampling without replacement • Beta-binomial distribution, for the number of "positive occurrences" (e.g. successes, yes votes, etc.) given a fixed number of total occurrences, sampling using a Polya urn scheme (in some sense, the "opposite" of sampling without replacement) Related to categorical outcomes (events with K possible outcomes, with a given probability for each outcome) • Categorical distribution, for a single categorical outcome (e.g. yes/no/maybe in a survey); a generalization of the Bernoulli distribution • Multinomial distribution, for the number of each type of categorical outcome, given a fixed number of total outcomes; a generalization of the binomial distribution • Multivariate hypergeometric distribution, similar to the multinomial distribution, but using sampling without replacement; a generalization of the hypergeometric distribution Related to events in a Poisson process (events that occur independently with a given rate) • Poisson distribution, for the number of occurrences of a Poisson-type event in a given period of time • Exponential distribution, for the time before the next Poisson-type event occurs Useful for hypothesis testing related to normally distributed outcomes • Chi-squared distribution, the distribution of a sum of squared standard normal variables; useful e.g. for inference regarding the sample variance of normally distributed samples (see chi-squared test) • Student's t distribution, the distribution of the ratio of a standard normal variable and the square root of a scaled chi squared variable; useful for inference regarding the mean of normally distributed samples with unknown variance (see Student's t-test) • F-distribution, the distribution of the ratio of two scaled chi squared variables; useful e.g. for inferences that involve comparing variances or involving R-squared (the squared correlation coefficient) Probability distribution 107 Useful as conjugate prior distributions in Bayesian inference • Beta distribution, for a single probability (real number between 0 and 1); conjugate to the Bernoulli distribution and binomial distribution • Gamma distribution, for a non-negative scaling parameter; conjugate to the rate parameter of a Poisson distribution or exponential distribution, the precision (inverse variance) of a normal distribution, etc. • Dirichlet distribution, for a vector of probabilities that must sum to 1; conjugate to the categorical distribution and multinomial distribution; generalization of the beta distribution • Wishart distribution, for a symmetric non-negative definite matrix; conjugate to the inverse of the covariance matrix of a multivariate normal distribution; generalization of the gamma distribution References • B. S. Everitt: The Cambridge Dictionary of Statistics, Cambridge University Press, Cambridge (3rd edition, 2006). ISBN 0-521-69027-7 • Bishop: Pattern Recognition and Machine Learning, Springer, ISBN 0-387-31073-8 External links • Hazewinkel, Michiel, ed. (2001), "Probability distribution" (http://www.encyclopediaofmath.org/index. php?title=p/p074900), Encyclopedia of Mathematics, Springer, ISBN 978-1-55608-010-4 Single-equation methods (econometrics) A variety of methods are used in econometrics to estimate models consisting of a single equation. The oldest and still the most commonly used is the ordinary least squares method used to estimate linear regressions. A variety of methods are available to estimate non-linear models. A particularly important class of non-linear models are those used to estimate relationships where the dependent variable is discrete, truncated or censored. These include logit, probit and Tobit models. Single equation methods may be applied to time-series, cross section or panel data. Cross-sectional study 108 Cross-sectional study Cross-sectional studies (also known as cross-sectional analyses, transversal studies, prevalence study) form a class of research methods that involve observation of all of a population, or a representative subset, at one specific point in time. They differ from case-control studies in that they aim to provide data on the entire population under study, whereas case-control studies typically include only individuals with a specific characteristic, with a sample, often a tiny minority, of the rest of the population. Cross-sectional studies are descriptive studies (neither longitudinal nor experimental). Unlike case-control studies, they can be used to describe, not only the Odds ratio, but also absolute risks and relative risks from prevalences (sometimes called prevalence risk ratio, or PRR).[1][2] They may be used to describe some feature of the population, such as prevalence of an illness, or they may support inferences of cause and effect. Longitudinal studies differ from both in making a series of observations more than once on members of the study population over a period of time. Cross-sectional studies in medicine Cross-sectional studies involve data collected at a defined time. They are often used to assess the prevalence of acute or chronic conditions, or to answer questions about the causes of disease or the results of medical intervention. They may also be described as censuses. Cross-sectional studies may involve special data collection, including questions about the past, but they often rely on data originally collected for other purposes. They are moderately expensive, and are not suitable for the study of rare diseases. Difficulty in recalling past events may also contribute bias. Use of routine data: large scale, low cost to the researcher The use of routinely collected data allows large cross-sectional studies to be made at little or no expense. This is a major advantage over other forms of epidemiological study. A natural progression has been suggested from cheap cross-sectional studies of routinely collected data which suggest hypotheses, to case-control studies testing them more specifically, then to cohort studies and trials which cost much more and take much longer, but may give stronger evidence. In a cross-sectional survey, a specific group is looked at to see if an activity, say alcohol consumption, is related to the health effect being investigated, say cirrhosis of the liver. If alcohol use is correlated with cirrhosis of the liver, this would support the hypothesis that alcohol use may cause cirrhosis. Routine data not designed to answer the specific question However, routinely collected data does not normally describe which variable is the cause and which the effect. Cross-sectional studies using data originally collected for other purposes are often unable to include data on confounding factors, other variables that affect the relationship between the putative cause and effect. For example, data only on present alcohol consumption and cirrhosis would not allow the role of past alcohol consumption, or of other causes, to be explored. Most case-control studies collect specifically designed data on all participants, including data fields designed to allow the hypothesis of interest to be tested. However, in issues where strong personal feelings may be involved, specific questions may be a source of bias. For example, past alcohol consumption may be incorrectly reported by an individual wishing to reduce their personal feelings of guilt. Such bias may be less in routinely collected statistics, or effectively eliminated if the observations are made by third parties, for example taxation records of alcohol by area. Cross-sectional study 109 Aggregated data and the "ecological fallacy" Cross-sectional studies can contain individual-level data (one record per individual, for example, in national health surveys). However, in modern epidemiology it may be impossible to survey the entire population of interest, so cross-sectional studies often involve secondary analysis of data collected for another purpose. In many such cases, no individual records are available to the researcher, and group-level information must be used. Major sources of such data are often large institutions like the Census Bureau or the Centers for Disease Control in the United States. Recent census data is not provided on individuals - in the UK individual census data is released only after a century. Instead data are aggregated, usually by administrative area. Inferences about individuals based on aggregate data are weakened by the ecological fallacy. Also consider the potential for committing the "atomistic fallacy" where assumptions about aggregated counts are made based on the aggregation of individual level data (such as averaging census tracts to calculate a county average). For example, it might be true that there is no correlation between infant mortality and family income at the city level, while still being true that there is a strong relationship between infant mortality and family income at the individual level. All aggregate statistics are subject to compositional effects, so that what matters is not only the individual-level relationship between income and infant mortality, but also the proportions of low, middle, and high income individuals in each city. Because case-control studies are usually based on individual-level data, they do not have this problem. References [1] "When to use the odds ratio or the relative risk?", by Carsten Oliver Schmidt, Thomas Kohlmann, Int J Public Health 53 (2008) 165–167 1661-8556/08/030165-3 DOI 10.1007/s000 -00 -7068-3 © Birkhäuser Verlag, Basel, 2008 link (http:/ / www. ncbi. nlm. nih. gov/ pubmed/ 19127890) [2] Letters to the Editor, "Odds Ratio or Relative Risk for Cross-Sectional Data?", From JAMES LEE, International Journal of Epidemiology, International Epktemiotogical Association 1994, Vol. 23, No. 1. link (http:/ / ije. oxfordjournals. org/ content/ 23/ 1/ 201. long) • Epidemiology for the Uninitiated by Coggon, Rose, and Barker, Chapter 8, "Case-control and cross-sectional studies", BMJ (British Medical Journal) Publishing, 1997 (http://resources.bmj.com/bmj/readers/readers/ epidemiology-for-the-uninitiated/8-case-control-and-cross-sectional-studies) • Research Methods Knowledge Base by William M. K. Trochim, Web Center for Social Research Methods, copyright 2006 (http://www.socialresearchmethods.net/kb/timedim.php) • Cross-Sectional Design by Michelle A. Saint-Germain (http://www.csulb.edu/~msaintg/ppa696/696preex. htm#Cross-Sectional Design) External links • Study Design Tutorial (http://www.vet.cornell.edu/imaging/tutorial/4studydesigns/crosssectional.html) Cornell University College of Veterinary Medicine Spatial analysis 110 Spatial analysis Spatial analysis or spatial statistics includes any of the formal techniques which study entities using their topological, geometric, or geographic properties. The phrase properly refers to a variety of techniques, many still in their early development, using different analytic approaches and applied in fields as diverse as astronomy, with its studies of the placement of galaxies in the cosmos, to chip fabrication engineering, with its use of 'place and route' algorithms to build complex wiring structures. The phrase is often used in a more restricted sense to describe techniques applied to structures at the human scale, most notably in the analysis of geographic data. The phrase is even sometimes used to refer to a specific technique in a single area of research, for Map by Dr. John Snow of London, showing clusters of cholera cases in the 1854 example, to describe geostatistics. Broad Street cholera outbreak. This was one of the first uses of map-based spatial analysis. Complex issues arise in spatial analysis, many of which are neither clearly defined nor completely resolved, but form the basis for current research. The most fundamental of these is the problem of defining the spatial location of the entities being studied. For example, a study on human health could describe the spatial position of humans with a point placed where they live, or with a point located where they work, or by using a line to describe their weekly trips; each choice has dramatic effects on the techniques which can be used for the analysis and on the conclusions which can be obtained. Other issues in spatial analysis include the limitations of mathematical knowledge, the assumptions required by existing statistical techniques, and problems in computer based calculations. Classification of the techniques of spatial analysis is difficult because of the large number of different fields of research involved, the different fundamental approaches which can be chosen, and the many forms the data can take. The history of the Spatial Analysis Spatial analysis can perhaps be considered to have arisen with the early attempts at cartography and surveying but many fields have contributed to its rise in modern form. Biology contributed through botanical studies of global plant distributions and local plant locations, ethological studies of animal movement, landscape ecological studies of vegetation blocks, ecological studies of spatial population dynamics, and the study of biogeography. Epidemiology contributed with early work on disease mapping, notably [Dr. Snow's work] mapping an outbreak of cholera, with research on mapping the spread of disease and with locational studies for health care delivery. Statistics has contributed greatly through work in spatial statistics. Economics has contributed notably through spatial econometrics. Geographic information system is currently a major contributor due to the importance of geographic software in the modern analytic toolbox. Remote sensing has contributed extensively in morphometric and clustering analysis. Computer science has contributed extensively through the study of algorithms, notably in computational geometry. Mathematics continues to provide the fundamental tools for analysis and to reveal the complexity of the Spatial analysis 111 spatial realm, for example, with recent work on fractals and scale invariance. Scientific modelling provides a useful framework for new approaches. Fundamental issues in spatial analysis Spatial analysis confronts many fundamental issues in the definition of its objects of study, in the construction of the analytic operations to be used, in the use of computers for analysis, in the limitations and particularities of the analyses which are known, and in the presentation of analytic results. Many of these issues are active subjects of modern research. Common errors often arise in spatial analysis, some due to the mathematics of space, some due to the particular ways data are presented spatially, some due to the tools which are available. Census data, because it protects individual privacy by aggregating data into local units, raises a number of statistical issues. The fractal nature of coastline makes precise measurements of its length difficult if not impossible. A computer software fitting straight lines to the curve of a coastline, can easily calculate the lengths of the lines which it defines. However these straight lines may have no inherent meaning in the real world, as was shown for the coastline of Britain. These problems represent a challenge in spatial analysis because of the power of maps as media of presentation. When results are presented as maps, the presentation combines spatial data which are generally accurate with analytic results which may be inaccurate, leading to an impression that analytic results are more accurate than the data would indicate.[1] Spatial characterization The definition of the spatial presence of an entity constrains the possible analysis which can be applied to that entity and influences the final conclusions that can be reached. While this property is fundamentally true of all analysis, it is particularly important in spatial analysis because the tools to define and study entities favor specific characterizations of the entities being studied. Statistical techniques favor the spatial definition of objects as points because there are very few statistical techniques which operate directly on line, area, or volume elements. Computer tools favor the spatial definition of objects as homogeneous and separate elements because of the limited number of database elements and computational structures available, and the ease with which these primitive structures can be created. Spatial dependency or auto-correlation Spread of bubonic plague in medieval Europe. The colors indicate Spatial dependency is the co-variation of properties the spatial distribution of plague outbreaks over time. Possibly due to the limitations of printing or for a host of other reasons, the within geographic space: characteristics at proximal cartographer selected a discrete number of colors to characterize locations appear to be correlated, either positively or (and simplify) reality. negatively. Spatial dependency leads to the spatial autocorrelation problem in statistics since, like temporal autocorrelation, this violates standard statistical techniques that assume independence among observations. For example, regression analyses that do not compensate for spatial dependency can have unstable parameter estimates and yield unreliable significance tests. Spatial regression models (see below) capture these relationships and do not suffer from these weaknesses. It is also appropriate to view spatial Spatial analysis 112 dependency as a source of information rather than something to be corrected.[2] Locational effects also manifest as spatial heterogeneity, or the apparent variation in a process with respect to location in geographic space. Unless a space is uniform and boundless, every location will have some degree of uniqueness relative to the other locations. This affects the spatial dependency relations and therefore the spatial process. Spatial heterogeneity means that overall parameters estimated for the entire system may not adequately describe the process at any given location. Scaling Spatial measurement scale is a persistent issue in spatial analysis; more detail is available at the modifiable areal unit problem (MAUP) topic entry. Landscape ecologists developed a series of scale invariant metrics for aspects of ecology that are fractal in nature. In more general terms, no scale independent method of analysis is widely agreed upon for spatial statistics. Sampling Spatial sampling involves determining a limited number of locations in geographic space for faithfully measuring phenomena that are subject to dependency and heterogeneity. Dependency suggests that since one location can predict the value of another location, we do not need observations in both places. But heterogeneity suggests that this relation can change across space, and therefore we cannot trust an observed degree of dependency beyond a region that may be small. Basic spatial sampling schemes include random, clustered and systematic. These basic schemes can be applied at multiple levels in a designated spatial hierarchy (e.g., urban area, city, neighborhood). It is also possible to exploit ancillary data, for example, using property values as a guide in a spatial sampling scheme to measure educational attainment and income. Spatial models such as autocorrelation statistics, regression and interpolation (see below) can also dictate sample design. Common errors in spatial analysis The fundamental issues in spatial analysis lead to numerous problems in analysis including bias, distortion and outright errors in the conclusions reached. These issues are often interlinked but various attempts have been made to separate out particular issues from each other. Length In a paper by Benoit Mandelbrot on the coastline of Britain it was shown that it is inherently nonsensical to discuss certain spatial concepts despite an inherent presumption of the validity of the concept. Lengths in ecology depend directly on the scale at which they are measured and experienced. So while surveyors commonly measure the length of a river, this length only has meaning in the context of the relevance of the measuring technique to the question under study. Spatial analysis 113 Britain measured using a long Britain measured using a medium Britain measured using a short yardstick yardstick yardstick Locational fallacy The locational fallacy refers to error due to the particular spatial characterization chosen for the elements of study, in particular choice of placement for the spatial presence of the element. Spatial characterizations may be simplistic or even wrong. Studies of humans often reduce the spatial existence of humans to a single point, for instance their home address. This can easily lead to poor analysis, for example, when considering disease transmission which can happen at work or at school and therefore far from the home. The spatial characterization may implicitly limit the subject of study. For example, the spatial analysis of crime data has recently become popular but these studies can only describe the particular kinds of crime which can be described spatially. This leads to many maps of assault but not to any maps of embezzlement with political consequences in the conceptualization of crime and the design of policies to address the issue. Atomic fallacy This describes errors due to treating elements as separate 'atoms' outside of their spatial context. Ecological fallacy The ecological fallacy describes errors due to performing analyses on aggregate data when trying to reach conclusions on the individual units. Errors occur in part from spatial aggregation. For example a pixel represents the average surface temperatures within an area. Ecological fallacy would be to assume that all points within the area have the same temperature. This topic is closely related to the modifiable areal unit problem. Spatial analysis 114 Solutions to the fundamental issues Geographic space A mathematical space exists whenever we have a set of observations and quantitative measures of their attributes. For example, we can represent individuals' income or years of education within a coordinate system where the location of each individual can be specified with respect to both dimensions. The distances between individuals within this space is a quantitative measure of their differences with respect to income and education. However, in spatial analysis we are concerned with specific types of mathematical spaces, namely, geographic space. In geographic space, the observations correspond to locations in a spatial measurement framework that captures their proximity in the real world. The locations in a spatial measurement framework often represent locations on the surface of the Earth, but this is not Manhattan distance versus Euclidean distance: The red, strictly necessary. A spatial measurement framework can also blue, and yellow lines have the same length (12) in capture proximity with respect to, say, interstellar space or within both Euclidean and taxicab geometry. In Euclidean a biological entity such as a liver. The fundamental tenet is geometry, the green line has length 6×√2 ≈ 8.48, and is the unique shortest path. In taxicab geometry, the green Tobler's First Law of Geography: if the interrelation between line's length is still 12, making it no shorter than any entities increases with proximity in the real world, then other path shown. representation in geographic space and assessment using spatial analysis techniques are appropriate. The Euclidean distance between locations often represents their proximity, although this is only one possibility. There are an infinite number of distances in addition to Euclidean that can support quantitative analysis. For example, "Manhattan" (or "Taxicab") distances where movement is restricted to paths parallel to the axes can be more meaningful than Euclidean distances in urban settings. In addition to distances, other geographic relationships such as connectivity (e.g., the existence or degree of shared borders) and direction can also influence the relationships among entities. It is also possible to compute minimal cost paths across a cost surface; for example, this can represent proximity among locations when travel must occur across rugged terrain. Types of spatial analysis Spatial data comes in many varieties and it is not easy to arrive at a system of classification that is simultaneously exclusive, exhaustive, imaginative, and satisfying. -- G. Upton & B. Fingelton[3] Spatial data analysis Urban and Regional Studies deal with large tables of spatial data obtained from censuses and surveys. It is necessary to simplify the huge amount of detailed information in order to extract the main trends. Multivariate analysis (or Factor_analysis Factor analysis, FA) allows a change of variables, transforming the many variables of the census, usually correlated between themselves, into fewer independent "Factors" or "Principal Components" which are, actually, the eigenvectors of the data correlation matrix weighted by the inverse of their eigenvalues. This change of variables has two main advantages : 1. Since information is concentrated on the first new factors, it is possible to keep only a few of them while losing only a small amount of information ; mapping them produces fewer and more significant maps Spatial analysis 115 2. The factors, actually the eigenvectors, are orthogonal by construction, i.e. not correlated. In most cases, the dominant factor (with the largest eigenvalue) is the Social Component, separating rich and poor in the city. Since factors are not-correlated, other smaller processes than social status, which would have remained hidden otherwise, appear on the second, third, … factors. Factor analysis depends on measuring distances between observations : the choice of a significant metric is crucial. The Euclidean metric (Principal Component Analysis), the Chi-Square distance (Correspondence Analysis) or the Generalized Mahalanobis distance (Discriminant Analysis ) are among the more widely used.[4] More complicated models, using communalities or rotations have been proposed.[5] Using multivariate methods in spatial analysis began really in the 1950s (although some examples go back to the beginning of the century) and culminated in the 1970s, with the increasing power and accessibility of computers. Already in 1948, in a seminal publication, two sociologists, Bell and Shevky,[6] had shown that most city populations in the USA and in the world could be represented with three independent factors : 1- the « socio-economic status » opposing rich and poor districts and distributed in sectors running along highways from the city center, 2- the « life cycle », i.e. the age structure of households, distributed in concentric circles, and 3- « race and ethnicity », identifying patches of migrants located within the city. In 1961, in a groundbreaking study, British geographers used FA to classify British towns.[7] Brian J Berry, at the University of Chicago, and his students made a wide use of the method,[8] applying it to most important cities in the world and exhibiting common social structures.[9] The use of Factor Analysis in Geography, made so easy by modern computers, has been very wide but not always very wise.[10] Since the vectors extracted are determined by the data matrix, it is not possible to compare factors obtained from different censuses. A solution consists in fusing together several census matrices in a unique table which, then, may be analyzed. This, however, assumes that the definition of the variables has not changed over time and produces very large tables, difficult to manage. A better solution, proposed by psychometricians,[11] groups the data in a « cubic matrix », with three entries (for instance, locations, variables, time periods). A Three-Way Factor Analysis produces then three groups of factors related by a small cubic « core matrix ».[12] This method, which exhibits data evolution over time, has not been widely used in geography.[13] In Los Angeles,[14] however, it has exhibited the role, traditionally ignored, of Downtown as an organizing center for the whole city during several decades. Spatial autocorrelation Spatial autocorrelation statistics measure and analyze the degree of dependency among observations in a geographic space. Classic spatial autocorrelation statistics include Moran's , Geary's , Getis's and the standard deviational ellipse. These statistics require measuring a spatial weights matrix that reflects the intensity of the geographic relationship between observations in a neighborhood, e.g., the distances between neighbors, the lengths of shared border, or whether they fall into a specified directional class such as "west". Classic spatial autocorrelation statistics compare the spatial weights to the covariance relationship at pairs of locations. Spatial autocorrelation that is more positive than expected from random indicate the clustering of similar values across geographic space, while significant negative spatial autocorrelation indicates that neighboring values are more dissimilar than expected by chance, suggesting a spatial pattern similar to a chess board. Spatial autocorrelation statistics such as Moran's and Geary's are global in the sense that they estimate the overall degree of spatial autocorrelation for a dataset. The possibility of spatial heterogeneity suggests that the estimated degree of autocorrelation may vary significantly across geographic space. Local spatial autocorrelation statistics provide estimates disaggregated to the level of the spatial analysis units, allowing assessment of the dependency relationships across space. statistics compare neighborhoods to a global average and identify local regions of strong autocorrelation. Local versions of the and statistics are also available. Spatial analysis 116 Spatial interpolation Spatial interpolation methods estimate the variables at unobserved locations in geographic space based on the values at observed locations. Basic methods include inverse distance weighting: this attenuates the variable with decreasing proximity from the observed location. Kriging is a more sophisticated method that interpolates across space according to a spatial lag relationship that has both systematic and random components. This can accommodate a wide range of spatial relationships for the hidden values between observed locations. Kriging provides optimal estimates given the hypothesized lag relationship, and error estimates can be mapped to determine if spatial patterns exist. Spatial regression Spatial regression methods capture spatial dependency in regression analysis, avoiding statistical problems such as unstable parameters and unreliable significance tests, as well as providing information on spatial relationships among the variables involved. Depending on the specific technique, spatial dependency can enter the regression model as relationships between the independent variables and the dependent, between the dependent variables and a spatial lag of itself, or in the error terms. Geographically weighted regression (GWR) is a local version of spatial regression that generates parameters disaggregated by the spatial units of analysis.[15] This allows assessment of the spatial heterogeneity in the estimated relationships between the independent and dependent variables. The use of Markov Chain Monte Carlo (MCMC) methods can allow the estimation of complex functions, such as Poisson-Gamma-CAR, Poisson-lognormal-SAR, or Overdispersed logit models. Spatial interaction Spatial interaction or "gravity models" estimate the flow of people, material or information between locations in geographic space. Factors can include origin propulsive variables such as the number of commuters in residential areas, destination attractiveness variables such as the amount of office space in employment areas, and proximity relationships between the locations measured in terms such as driving distance or travel time. In addition, the topological, or connective, relationships between areas must be identified, particularly considering the often conflicting relationship between distance and topology; for example, two spatially close neighborhoods may not display any significant interaction if they are separated by a highway. After specifying the functional forms of these relationships, the analyst can estimate model parameters using observed flow data and standard estimation techniques such as ordinary least squares or maximum likelihood. Competing destinations versions of spatial interaction models include the proximity among the destinations (or origins) in addition to the origin-destination proximity; this captures the effects of destination (origin) clustering on flows. Computational methods such as artificial neural networks can also estimate spatial interaction relationships among locations and can handle noisy and qualitative data. Simulation and modeling Spatial interaction models are aggregate and top-down: they specify an overall governing relationship for flow between locations. This characteristic is also shared by urban models such as those based on mathematical programming, flows among economic sectors, or bid-rent theory. An alternative modeling perspective is to represent the system at the highest possible level of disaggregation and study the bottom-up emergence of complex patterns and relationships from behavior and interactions at the individual level. Complex adaptive systems theory as applied to spatial analysis suggests that simple interactions among proximal entities can lead to intricate, persistent and functional spatial entities at aggregate levels. Two fundamentally spatial simulation methods are cellular automata and agent-based modeling. Cellular automata modeling imposes a fixed spatial framework such as grid cells and specifies rules that dictate the state of a cell based on the states of its neighboring cells. As time progresses, spatial patterns emerge as cells change states based on their neighbors; this Spatial analysis 117 alters the conditions for future time periods. For example, cells can represent locations in an urban area and their states can be different types of land use. Patterns that can emerge from the simple interactions of local land uses include office districts and urban sprawl. Agent-based modeling uses software entities (agents) that have purposeful behavior (goals) and can react, interact and modify their environment while seeking their objectives. Unlike the cells in cellular automata, agents can be mobile with respect to space. For example, one could model traffic flow and dynamics using agents representing individual vehicles that try to minimize travel time between specified origins and destinations. While pursuing minimal travel times, the agents must avoid collisions with other vehicles also seeking to minimize their travel times. Cellular automata and agent-based modeling are complementary modeling strategies. They can be integrated into a common geographic automata system where some agents are fixed while others are mobile. Multiple-Point Geostatistics (MPS) Spatial analysis of a conceptual geological model is the main purpose of any MPS algorithm. The method analyzes the spatial statistics of the geological model, called the training image, and generates realizations of the phenomena that honor those input multiple-point statistics. One of the recent technique to accomplish this task is the pattern-based method of Honarkhah.[16] In this method, a distance-based approach is employed to analyze the patterns in the training image. This allows the reproduction of the multiple-point statistics, and the complex geometrical features of the given image. The final generated realizations of this, so called random field, can be used to quantify spatial uncertainty. In the recent method, Tahmasebi et al.[17] used a cross-correlation function to better spatial pattern reproduction and they presented the CCSIM algorithm. This method is able to quantify the spatial connectivity, variability and uncertainty. Furthermore, the method is not sensitive to any type of data and is able to simulate both categorical and continuous scenarios. Geographic information science and spatial analysis Geographic information systems (GIS) and the underlying geographic information science that advances these technologies have a strong influence on spatial analysis. The increasing ability to capture and handle geographic data means that spatial analysis is occurring within increasingly data-rich environments. Geographic data capture systems include remotely sensed imagery, environmental monitoring systems such as intelligent transportation systems, and location-aware technologies such as mobile devices that can report location in near-real time. GIS provide platforms for managing these data, computing spatial relationships such as distance, connectivity and directional relationships between spatial units, and visualizing both the raw data and spatial analytic results within a cartographic context. Spatial analysis 118 Content • Spatial location: Transfer positioning information of space objects with the help of space coordinate system. Projection transformation theory is the foundation of spatial object representation. • Spatial distribution: the similar spatial object groups positioning information, This flow map of Napoleon's ill-fated march on Moscow is an early and celebrated including distribution, trends, contrast example of geovisualization. It shows the army's direction as it traveled, the places etc.. the troops passed through, the size of the army as troops died from hunger and • Spatial form: the geometric shape of the wounds, and the freezing temperatures they experienced. spatial objects • Spatial space: the space objects' approaching degree • Spatial relationship: relationship between spatial objects, including topological, orientation, similarity, etc.. Geovisualization (GVis) combines scientific visualization with digital cartography to support the exploration and analysis of geographic data and information, including the results of spatial analysis or simulation. GVis leverages the human orientation towards visual information processing in the exploration, analysis and communication of geographic data and information. In contrast with traditional cartography, GVis is typically three or four-dimensional (the latter including time) and user-interactive. Geographic knowledge discovery (GKD) is the human-centered process of applying efficient computational tools for exploring massive spatial databases. GKD includes geographic data mining, but also encompasses related activities such as data selection, data cleaning and pre-processing, and interpretation of results. GVis can also serve a central role in the GKD process. GKD is based on the premise that massive databases contain interesting (valid, novel, useful and understandable) patterns that standard analytical techniques cannot find. GKD can serve as a hypothesis-generating process for spatial analysis, producing tentative patterns and relationships that should be confirmed using spatial analytical techniques. Spatial Decision Support Systems (SDSS) take existing spatial data and use a variety of mathematical models to make projections into the future. This allows urban and regional planners to test intervention decisions prior to implementation. References [1] Mark Monmonier How to Lie with Maps University of Chicago Press, 1996. [2] De Knegt; H.J.; F. van Langevelde; M.B. Coughenour; A.K. Skidmore; W.F. de Boer; I.M.A. Heitkönig; N.M. Knox; R. Slotow; C. van der Waal and H.H.T. Prins (2010). Spatial autocorrelation and the scaling of species–environment relationships. Ecology 91: 2455–2465. doi:10.1890/09-1359.1 [3] Graham J. Upton & Bernard Fingelton: Spatial Data Analysis by Example Volume 1: Point Pattern and Quantitative Data John Wiley & Sons, New York. 1985. [4] Harman H H (1960) Modern Factor Analysis, University of Chicago Press [5] Rummel R J (1970) Applied Factor Analysis. Evanston, ILL: Northwestern University Press. [6] Bell W & E Shevky (1955) Social Area Analysis, Stanford University Press [7] Moser C A & W Scott (1961) British Towns ; A Statistical Study of their Social and Economic Differences, Oliver & Boyd, London. [8] Berry B J & F Horton (1971) Geographic Perspectives on Urban Systems, John Wiley, N-Y. [9] Berry B J & K B Smith eds (1972) City Classification Handbook : Methods and Applications, John Wiley, N-Y. [10] Ciceri M-F (1974) Méthodes d’analyse multivariée dans la géographie anglo-saxonne, Université de Paris-1 ; free download on http:/ / www-ohp. univ-paris1. fr [11] Tucker L R (1964) « The extension of Factor Analysis to three-dimensional matrices », in Frederiksen N & H Gulliksen eds, Contributions to Mathematical Psychology, Holt, Reinhart and Winston, N-Y. Spatial analysis 119 [12] R. Coppi & S. Bolasco, eds. (1989), Multiway data analysis, Elsevier, Amsterdam. [13] Cant, R.G. (1971) « Changes in the location of manufacturing in New Zealand 1957-1968: An application of three-mode factor analysis. », New Zealand Geographer, 27, 38-55. [14] Marchand B (1986) The Emergence of Los Angeles, 1940-1970, Pion Ltd, London [15] Fotheringham, A. S., Charlton, M. E., & Brunsdon, C. (1998). Geographically weighted regression: a natural evolution of the expansion method for spatial data analysis. Environment and Planning A, 30(11), 1905-1927. doi:10.1068/a301905 [16] Honarkhah, M and Caers, J, 2010, Stochastic Simulation of Patterns Using Distance-Based Pattern Modeling (http:/ / dx. doi. org/ 10. 1007/ s11004-010-9276-7), Mathematical Geosciences, 42: 487 - 517 [17] Tahmasebi, P., Hezarkhani, A., Sahimi, M., 2012, Multiple-point geostatistical modeling based on the cross-correlation functions (http:/ / www. springerlink. com/ content/ 0150455247264825/ fulltext. pdf), Computational Geosciences, 16(3):779-79742 Further reading • Abler, R., J. Adams, and P. Gould (1971) Spatial Organization–The Geographer's View of the World, Englewood Cliffs, NJ: Prentice-Hall. • Anselin, L. (1995) "Local indicators of spatial association – LISA". Geographical Analysis, 27, 93–115 (http:// www.drs.wisc.edu/documents/articles/curtis/cesoc977/Anselin1995.pdf). • Banerjee, S., B.P. Carlin and A.E. Gelfand (2004). Hierarchical Modeling and Analysis for Spatial Data. Taylor and Francis: Chapman and Hall/CRC Press. • Benenson, I. and P. M. Torrens. (2004). Geosimulation: Automata-Based Modeling of Urban Phenomena. Wiley. • Fotheringham, A. S., C. Brunsdon and M. Charlton (2000) Quantitative Geography: Perspectives on Spatial Data Analysis, Sage. • Fotheringham, A. S. and M. E. O'Kelly (1989) Spatial Interaction Models: Formulations and Applications, Kluwer Academic • Fotheringham, A. S. and P. A. Rogerson (1993) "GIS and spatial analytical problems". International Journal of Geographical Information Systems, 7, 3–19. • Goodchild, M. F. (1987) "A spatial analytical perspective on geographical information systems". International Journal of Geographical Information Systems, 1, 327–44 (http://www.geog.ucsb.edu/~good/papers/95.pdf). • MacEachren, A. M. and D. R. F. Taylor (eds.) (1994) Visualization in Modern Cartography, Pergamon. • Levine, N. (2010). CrimeStat: A Spatial Statistics Program for the Analysis of Crime Incident Locations. Version 3.3. Ned Levine & Associates, Houston, TX and the National Institute of Justice, Washington, DC. Ch. 1-17 + 2 update chapters (http://www.icpsr.umich.edu/CrimeStat) • Miller, H. J. (2004) "Tobler's First Law and spatial analysis". Annals of the Association of American Geographers, 94, 284–289. • Miller, H. J. and J. Han (eds.) (2001) Geographic Data Mining and Knowledge Discovery, Taylor and Francis. • O'Sullivan, D. and D. Unwin (2002) Geographic Information Analysis, Wiley. • Parker, D. C., S. M. Manson, M.A. Janssen, M. J. Hoffmann and P. Deadman (2003) "Multi-agent systems for the simulation of land-use and land-cover change: A review". Annals of the Association of American Geographers, 93, 314–337 (http://uwf.edu/zhu/evr6930/17.pdf). • White, R. and G. Engelen (1997) "Cellular automata as the basis of integrated dynamic regional modelling". Environment and Planning B: Planning and Design, 24, 235–246. • Scheldeman, X. & van Zonneveld, M. (2010). Training Manual on Spatial Analysis of Plant Diversity and Distribution (http://www.bioversityinternational.org/training/training_materials/gis_manual.html). Bioversity International. • Fisher MM, Leung Y (2001) Geocomputational Modelling: techniques and applications. Springer Verlag, Berlin • Fotheringham S, Clarke G, Abrahart B (1997) Geocomputation and GIS. Transactions in GIS 2:199-200 • Openshaw S and Abrahart RJ (2000) GeoComputation. CRC Press • Diappi L (2004) Evolving Cities: Geocomputation in Territorial Planning. Ashgate, England • Longley PA, Brooks SM, McDonnell R, Macmillan B (1998), Geocomputation, a primer. John Wiley and Sons, Chichester Spatial analysis 120 • Ehlen J, Caldwell DR and Harding S (2002) GeoComputation: what is it?,Comput Environ and Urban Syst 26:257-265 • Gahegan M (1999) What is Geocomputation? Transaction in GIS 3:203-206 • Murgante B., Borruso G., Lapucci A. (2009) "Geocomputation and Urban Planning" Studies in Computational Intelligence, Vol. 176. Springer-Verlag, Berlin. • Fischer M., Leung Y.(2010) "GeoComputational Modelling: Techniques and Applications" Advances in Spatial Science. Springer-Verlag, Berlin. • Murgante B., Borruso G., Lapucci A. (2011) "Geocomputation, Sustainability and Environmental Planning" Studies in Computational Intelligence, Vol. 348. Springer-Verlag, Berlin. • Tahmasebi, P., Hezarkhani, A., Sahimi, M., 2012, Multiple-point geostatistical modeling based on the cross-correlation functions (http://www.springerlink.com/content/0150455247264825/fulltext.pdf), Computational Geosciences, 16(3):779-79742. External links • ICA Commission on Geospatial Analysis and Modeling (https://sites.google.com/site/commissionofica/) • An educational resource about spatial statistics and geostatistics (http://www.ai-geostats.org/) • A comprehensive guide to principles, techniques & software tools (http://www.spatialanalysisonline.com/) • Social and Spatial Inequalities (http://sasi.group.shef.ac.uk/) • National Center for Geographic Information and Analysis (NCGIA) (http://www.ncgia.ucsb.edu/) • International Cartographic Association (ICA) (http://www.icaci.org), the world body for mapping and GIScience professionals Effect size In statistics, an effect size is a measure of the strength of a phenomenon[1] (for example, the relationship between two variables in a statistical population) or a sample-based estimate of that quantity. An effect size calculated from data is a descriptive statistic that conveys the estimated magnitude of a relationship without making any statement about whether the apparent relationship in the data reflects a true relationship in the population. In that way, effect sizes complement inferential statistics such as p-values. Among other uses, effect size measures play an important role in meta-analysis studies that summarize findings from a specific area of research, and in statistical power analyses. The concept of effect size appears already in everyday language. For example, a weight loss program may boast that it leads to an average weight loss of 30 pounds. In this case, 30 pounds is an indicator of the claimed effect size. Another example is that a tutoring program may claim that it raises school performance by one letter grade. This grade increase is the claimed effect size of the program. These are both examples of "absolute effect sizes", meaning that they convey the average difference between two groups without any discussion of the variability within the groups. For example, if the weight loss program results in an average loss of 30 pounds, it is possible that every participant loses exactly 30 pounds, or half the participants lose 60 pounds and half lose no weight at all. Reporting effect sizes is considered good practice when presenting empirical research findings in many fields.[2][3] The reporting of effect sizes facilitates the interpretation of the substantive, as opposed to the statistical, significance of a research result.[4] Effect sizes are particularly prominent in social and medical research. Relative and absolute measures of effect size convey different information, and can be used complementarily. A prominent task force in the psychology research community expressed the following recommendation: Always present effect sizes for primary outcomes...If the units of measurement are meaningful on a practical level (e.g., number of cigarettes smoked per day), then we usually prefer an unstandardized measure Effect size 121 (regression coefficient or mean difference) to a standardized measure (r or d). — L. Wilkinson and APA Task Force on Statistical Inference (1999, p. 599) Overview Population and sample effect sizes The term effect size can refer to a statistic calculated from a sample of data, or to a parameter of a hypothetical statistical population. Conventions for distinguishing sample from population effect sizes follow standard statistical practices — one common approach is to use Greek letters like ρ to denote population parameters and Latin letters like r to denote the corresponding statistic; alternatively, a "hat" can be placed over the population parameter to denote the statistic, e.g. with being the estimate of the parameter . As in any statistical setting, effect sizes are estimated with error, and may be biased unless the effect size estimator that is used is appropriate for the manner in which the data were sampled and the manner in which the measurements were made. An example of this is publication bias, which occurs when scientists only report results when the estimated effect sizes are large or are statistically significant. As a result, if many researchers are carrying out studies under low statistical power, the reported results are biased to be stronger than true effects, if any.[5] Another example where effect sizes may be distorted is in a multiple trial experiment, where the effect size calculation is based on the averaged or aggregated response across the trials.[6] Relationship to test statistics Sample-based effect sizes are distinguished from test statistics used in hypothesis testing, in that they estimate the strength of an apparent relationship, rather than assigning a significance level reflecting whether the relationship could be due to chance. The effect size does not determine the significance level, or vice-versa. Given a sufficiently large sample size, a statistical comparison will always show a significant difference unless the population effect size is exactly zero. For example, a sample Pearson correlation coefficient of 0.1 is strongly statistically significant if the sample size is 1000. Reporting only the significant p-value from this analysis could be misleading if a correlation of 0.1 is too small to be of interest in a particular application. Standardized and unstandardized effect sizes The term effect size can refer to a standardized measures of effect (such as r, Cohen's d, and odds ratio), or to an unstandardized measure (e.g., the raw difference between group means and unstandardized regression coefficients). Standardized effect size measures are typically used when the metrics of variables being studied do not have intrinsic meaning (e.g., a score on a personality test on an arbitrary scale), when results from multiple studies are being combined, when some or all of the studies use different scales, or when it is desired to convey the size of an effect relative to the variability in the population. In meta-analysis, standardized effect sizes are used as a common measure that can be calculated for different studies and then combined into an overall summary. Effect size 122 Types Effect Sizes Based on "Variance Explained" These effect sizes estimate the amount of the variance within an experiment that is "explained" or "accounted for" by the experiment's model. Pearson r (correlation) Pearson's correlation, often denoted r and introduced by Karl Pearson, is widely used as an effect size when paired quantitative data are available; for instance if one were studying the relationship between birth weight and longevity. The correlation coefficient can also be used when the data are binary. Pearson's r can vary in magnitude from −1 to 1, with −1 indicating a perfect negative linear relation, 1 indicating a perfect positive linear relation, and 0 indicating no linear relation between two variables. Cohen gives the following guidelines for the social sciences:[7][8] Effect size r Small 0.10 Medium 0.30 Large 0.50 Coefficient of determination A related effect size is r², the coefficient of determination (also referred to as "r-squared"), calculated as the square of the Pearson correlation r. In the case of paired data, this is a measure of the proportion of variance shared by the two variables, and varies from 0 to 1. For example, with an r of 0.21 the coefficient of determination is 0.0441, meaning that 4.4% of the variance of either variable is shared with the other variable. The r² is always positive, so does not convey the direction of the correlation between the two variables. Cohen's ƒ2 Cohen's ƒ2 is one of several effect size measures to use in the context of an F-test for ANOVA or multiple regression. Note that it estimates for the sample rather than the population and is biased (overestimates effect size for the ANOVA). The ƒ2 effect size measure for multiple regression is defined as: where R2 is the squared multiple correlation. The effect size measure for hierarchical multiple regression is defined as: where R2A is the variance accounted for by a set of one or more independent variables A, and R2AB is the combined variance accounted for by A and another set of one or more independent variables B. By convention, ƒ2A effect sizes of 0.02, 0.15, and 0.35 are termed small, medium, and large, respectively.[7] Cohen's can also be found for factorial analysis of variance (ANOVA, aka the F-test) working backwards using : In a balanced design (equivalent sample sizes across groups) of ANOVA, the corresponding population parameter of is Effect size 123 wherein μj denotes the population mean within the jth group of the total K groups, and σ the equivalent population standard deviations within each groups. SS is the sum of squares manipulation in ANOVA. An unbiased estimator for ANOVA would be based on Omega squared, which estimates for the population. Omega-squared, ω2 A more unbiased estimator of the variance explained in the population is omega-squared[9][10][11] This form of the formula is limited to between-subjects analysis with equal sample sizes in all cells,[11]. Since it is less biased (although still biased), ω2 is preferable to Cohen's ƒ2; however, it can be more inconvenient to calculate for complex analyses. A generalized form of the estimator has been published for between-subjects and within-subjects analysis, repeated measure, mixed design, and randomized block design experiments.[12] In addition, methods to calculate partial Omega2 for individual factors and combined factors in designs with up to three independent variables have been published.[12] Effect sizes based on means or distances between/among means A (population) effect size θ based on means usually considers the standardized mean difference between two populations[13]:78 where μ1 is the mean for one population, μ2 is the mean for the other population, and σ is a standard deviation based on either or both populations. In the practical setting the population values are typically not known and must be estimated from sample statistics. The several versions of effect sizes based on means differ with respect to which statistics are used. Plots of Gaussian densities illustrating various This form for the effect size resembles the computation for a t-test values of Cohen's d. statistic, with the critical difference that the t-test statistic includes a factor of . This means that for a given effect size, the significance level increases with the sample size. Unlike the t-test statistic, the effect size aims to estimate a population parameter, so is not affected by the sample size. Cohen's d Cohen's d is defined as the difference between two means divided by a standard deviation for the data Cohen's d is frequently used in estimating sample sizes. A lower Cohen's d indicates a necessity of larger sample sizes, and vice versa, as can subsequently be determined together with the additional parameters of desired significance level and statistical power.[14] What precisely the standard deviation s is was not originally made explicit by Jacob Cohen because he defined it (using the symbol "σ") as "the standard deviation of either population (since they are assumed equal)".[7]:20 Other authors make the computation of the standard deviation more explicit with the following definition for a pooled Effect size 124 standard deviation[15]:14 with two independent samples. This definition of "Cohen's d" is termed the maximum likelihood estimator by Hedges and Olkin,[13] and it is related to Hedges' g (see below) by a scaling[13]:82 Glass's Δ In 1976 Gene V. Glass proposed an estimator of the effect size that uses only the standard deviation of the second group[13]:78 The second group may be regarded as a control group, and Glass argued that if several treatments were compared to the control group it would be better to use just the standard deviation computed from the control group, so that effect sizes would not differ under equal means and different variances. Under an assumption of equal population variances a pooled estimate for σ is more precise. Hedges' g Hedges' g, suggested by Larry Hedges in 1981,[16] is like the other measures based on a standardized difference[13]:79 but its pooled standard deviation is computed slightly differently from Cohen's d As an estimator for the population effect size θ it is biased. However, this bias can be corrected for by multiplication with a factor Hedges and Olkin refer to this unbiased estimator as d,[13] but it is not the same as Cohen's d. The exact form for the correction factor J() involves the gamma function[13]:104 Effect size 125 Distribution of effect sizes based on means Provided that the data is Gaussian distributed a scaled Hedges' g, , follows a noncentral t-distribution with the noncentrality parameter and (n1 + n2 − 2) degrees of freedom. Likewise, the scaled Glass' Δ is distributed with n2 − 1 degrees of freedom. From the distribution it is possible to compute the expectation and variance of the effect sizes. In some cases large sample approximations for the variance are used. One suggestion for the variance of Hedges' unbiased estimator is[13]:86 Effect sizes for associations among categorical variables Phi (φ) Cramér's Phi (φc) Commonly-used measures of association for the chi-squared test are the Phi coefficient and Cramér's V. Phi is related to the point-biserial correlation coefficient and Cohen's d and estimates the extent of the relationship between two variables (2 x 2).[17] Cramér's Phi may be used with variables having more than two levels. Phi can be computed by finding the square root of the chi-squared statistic divided by the sample size. Similarly, Cramér's phi is computed by taking the square root of the chi-squared statistic divided by the sample size and the length of the minimum dimension (k is the smaller of the number of rows r or columns c). φc is the intercorrelation of the two discrete variables[18] and may be computed for any value of r or c. However, as chi-squared values tend to increase with the number of cells, the greater the difference between r and c, the more likely φc will tend to 1 without strong evidence of a meaningful correlation. Cramér's phi may also be applied to 'goodness of fit' chi-squared models (i.e. those where c=1). In this case it functions as a measure of tendency towards a single outcome (i.e. out of k outcomes). Odds ratio The odds ratio (OR) is another useful effect size. It is appropriate when the research question focuses on the degree of association between two binary variables. For example, consider a study of spelling ability. In a control group, two students pass the class for every one who fails, so the odds of passing are two to one (or 2/1 = 2). In the treatment group, six students pass for every one who fails, so the odds of passing are six to one (or 6/1 = 6). The effect size can be computed by noting that the odds of passing in the treatment group are three times higher than in the control group (because 6 divided by 2 is 3). Therefore, the odds ratio is 3. Odds ratio statistics are on a different scale than Cohen's d, so this '3' is not comparable to a Cohen's d of 3. Relative risk The relative risk (RR), also called risk ratio, is simply the risk (probability) of an event relative to some independent variable. This measure of effect size differs from the odds ratio in that it compares probabilities instead of odds, but asymptotically approaches the latter for small probabilities. Using the example above, the probabilities for those in the control group and treatment group passing is 2/3 (or 0.67) and 6/7 (or 0.86), respectively. The effect size can be computed the same as above, but using the probabilities instead. Therefore, the relative risk is 1.28. Since rather large probabilities of passing were used, there is a large difference between relative risk and odds ratio. Had failure Effect size 126 (a smaller probability) been used as the event (rather than passing), the difference between the two measures of effect size would not be so great. While both measures are useful, they have different statistical uses. In medical research, the odds ratio is commonly used for case-control studies, as odds, but not probabilities, are usually estimated.[19] Relative risk is commonly used in randomized controlled trials and cohort studies.[20] When the incidence of outcomes are rare in the study population (generally interpreted to mean less than 10%), the odds ratio is considered a good estimate of the risk ratio. However, as outcomes become more common, the odds ratio and risk ratio diverge, with the odds ratio overestimating or underestimating the risk ratio when the estimates are greater than or less than 1, respectively. When estimates of the incidence of outcomes are available, methods exist to convert odds ratios to risk ratios.[21][22] Confidence intervals by means of noncentrality parameters Confidence intervals of standardized effect sizes, especially Cohen's and , rely on the calculation of confidence intervals of noncentrality parameters (ncp). A common approach to construct the confidence interval of ncp is to find the critical ncp values to fit the observed statistic to tail quantiles α/2 and (1 − α/2). The SAS and R-package MBESS provides functions to find critical values of ncp. t-test for mean difference of single group or two related groups For a single group, M denotes the sample mean, μ the population mean, SD the sample's standard deviation, σ the population's standard deviation, and n is the sample size of the group. The t value is used to test the hypothesis on the difference between the mean and a baseline μbaseline. Usually, μbaseline is zero. In the case of two related groups, the single group is constructed by the differences in pair of samples, while SD and σ denote the sample's and population's standard deviations of differences rather than within original two groups. and Cohen's is the point estimate of So, Effect size 127 t-test for mean difference between two independent groups n1 or n2 are the respective sample sizes. wherein and Cohen's is the point estimate of So, One-way ANOVA test for mean difference across multiple independent groups One-way ANOVA test applies noncentral F distribution. While with a given population standard deviation , the same test question applies noncentral chi-squared distribution. For each j-th sample within i-th group Xi,j, denote While, So, both ncp(s) of F and equate In case of for K independent groups of same size, the total sample size is N := n·K. The t-test for a pair of independent groups is a special case of one-way ANOVA. Note that the noncentrality parameter of F is not comparable to the noncentrality parameter of the corresponding t. Actually, , and . Effect size 128 "Small", "medium", "large" Some fields using effect sizes apply words such as "small", "medium" and "large" to the size of the effect. Whether an effect size should be interpreted small, medium, or large depends on its substantial context and its operational definition. Cohen's conventional criteria small, medium, or big[7] are near ubiquitous across many fields. Power analysis or sample size planning requires an assumed population parameter of effect sizes. Many researchers adopt Cohen's standards as default alternative hypotheses. Russell Lenth criticized them as T-shirt effect sizes[23] This is an elaborate way to arrive at the same sample size that has been used in past social science studies of large, medium, and small size (respectively). The method uses a standardized effect size as the goal. Think about it: for a "medium" effect size, you'll choose the same n regardless of the accuracy or reliability of your instrument, or the narrowness or diversity of your subjects. Clearly, important considerations are being ignored here. "Medium" is definitely not the message! For Cohen's d an effect size of 0.2 to 0.3 might be a "small" effect, around 0.5 a "medium" effect and 0.8 to infinity, a "large" effect.[7]:25 (But note that the d might be larger than one) Cohen's text[7] anticipates Lenth's concerns: "The terms 'small,' 'medium,' and 'large' are relative, not only to each other, but to the area of behavioral science or even more particularly to the specific content and research method being employed in any given investigation....In the face of this relativity, there is a certain risk inherent in offering conventional operational definitions for these terms for use in power analysis in as diverse a field of inquiry as behavioral science. This risk is nevertheless accepted in the belief that more is to be gained than lost by supplying a common conventional frame of reference which is recommended for use only when no better basis for estimating the ES index is available." (p. 25) In an ideal world, researchers would interpret the substantive significance of their results by grounding them in a meaningful context or by quantifying their contribution to knowledge. Where this is problematic, Cohen's effect size criteria may serve as a last resort.[4] References [1] Kelley, Ken; Preacher, Kristopher J. (2012). "On Effect Size". Psychological Methods 17 (2): 137–152. doi:10.1037/a0028086. [2] Wilkinson, Leland; APA Task Force on Statistical Inference (1999). "Statistical methods in psychology journals: Guidelines and explanations". American Psychologist 54 (8): 594–604. doi:10.1037/0003-066X.54.8.594. [3] Nakagawa, Shinichi; Cuthill, Innes C (2007). "Effect size, confidence interval and statistical significance: a practical guide for biologists". Biological Reviews Cambridge Philosophical Society 82 (4): 591–605. doi:10.1111/j.1469-185X.2007.00027.x. PMID 17944619. [4] Ellis, Paul D. (2010). The Essential Guide to Effect Sizes: An Introduction to Statistical Power, Meta-Analysis and the Interpretation of Research Results. United Kingdom: Cambridge University Press. [5] Brand A, Bradley MT, Best LA, Stoica G (2008). "Accuracy of effect size estimates from published psychological research" (http:/ / mtbradley. com/ brandbradelybeststoicapdf. pdf). Perceptual and Motor Skills 106 (2): 645–649. doi:10.2466/PMS.106.2.645-649. PMID 18556917. . [6] Brand A, Bradley MT, Best LA, Stoica G (2011). "Multiple trials may yield exaggerated effect size estimates" (http:/ / www. ipsychexpts. com/ brand_et_al_(2011). pdf). The Journal of General Psychology 138 (1): 1–11. doi:10.1080/00221309.2010.520360. . [7] Jacob Cohen (1988). Statistical Power Analysis for the Behavioral Sciences (second ed.). Lawrence Erlbaum Associates. [8] Cohen, J (1992). "A power primer". Psychological Bulletin 112 (1): 155–159. doi:10.1037/0033-2909.112.1.155. PMID 19565683. [9] Bortz, 1999, p. 269f.; [10] Bühner & Ziegler (2009, p. 413f) [11] Tabachnick & Fidell (2007, p. 55) [12] Olejnik, S. & Algina, J. 2003. Generalized Eta and Omega Squared Statistics: Measures of Effect Size for Some Common Research Designs Psychological Methods. 8:(4)434-447. http:/ / cps. nova. edu/ marker/ olejnik2003. pdf [13] Larry V. Hedges & Ingram Olkin (1985). Statistical Methods for Meta-Analysis. Orlando: Academic Press. ISBN 0-12-336380-2. [14] Chapter 13 (http:/ / davidakenny. net/ statbook/ chapter_13. pdf), page 215, in: Kenny, David A. (1987). Statistics for the social and behavioral sciences. Boston: Little, Brown. ISBN 0-316-48915-8. [15] Joachim Hartung, Guido Knapp & Bimal K. Sinha (2008). Statistical Meta-Analysis with Application. Hoboken, New Jersey: Wiley. Effect size 129 [16] Larry V. Hedges (1981). "Distribution theory for Glass's estimator of effect size and related estimators". Journal of Educational Statistics 6 (2): 107–128. doi:10.3102/10769986006002107. [17] Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect-size indices: Problems with a commonly recommended formula. (http:/ / www. eric. ed. gov/ ERICWebPortal/ custom/ portlets/ recordDetails/ detailmini. jsp?_nfpb=true& _& ERICExtSearch_SearchValue_0=ED433353& ERICExtSearch_SearchType_0=no& accno=ED433353) Paper presented at the annual meeting of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353) [18] Sheskin, David J. (1997). Handbook of Parametric and Nonparametric Statistical Procedures. Boca Raton, Fl: CRC Press. [19] Deeks J (1998). "When can odds ratios mislead? : Odds ratios should be used only in case-control studies and logistic regression analyses". BMJ 317 (7166): 1155–6. PMC 1114127. PMID 9784470. [20] Medical University of South Carolina. Odds ratio versus relative risk (http:/ / www. musc. edu/ dc/ icrebm/ oddsratio. html). Accessed on: September 8, 2005. [21] Zhang, J.; Yu, K. (1998). "What's the relative risk? A method of correcting the odds ratio in cohort studies of common outcomes". JAMA: the Journal of the American Medical Association 280 (19): 1690–1691. doi:10.1001/jama.280.19.1690. PMID 9832001. [22] Greenland, S. (2004). "Model-based Estimation of Relative Risks and Other Epidemiologic Measures in Studies of Common Outcomes and in Case-Control Studies". American Journal of Epidemiology 160 (4): 301–305. doi:10.1093/aje/kwh221. PMID 15286014. [23] Russell V. Lenth. "Java applets for power and sample size" (http:/ / www. stat. uiowa. edu/ ~rlenth/ Power/ ). Division of Mathematical Sciences, the College of Liberal Arts or The University of Iowa. . Retrieved 2008-10-08. Further reading • Aaron, B., Kromrey, J. D., & Ferron, J. M. (1998, November). Equating r-based and d-based effect-size indices: Problems with a commonly recommended formula. Paper presented at the annual meeting of the Florida Educational Research Association, Orlando, FL. (ERIC Document Reproduction Service No. ED433353) (http:// www.eric.ed.gov/ERICWebPortal/contentdelivery/servlet/ERICServlet?accno=ED433353) • Bonett, D.G. (2008). Confidence intervals for standardized linear contrasts of means, Psychological Methods, 13, 99-109. • Bonett, D.G. (2009). Estimating standardized linear contrasts of means with desired precision, Psychological Methods"", 14, 1-5. • Cumming, G. and Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 530–572. • Kelley, K. (2007). Confidence intervals for standardized effect sizes: Theory, application, and implementation. Journal of Statistical Software, 20(8), 1-24. (http://www.jstatsoft.org/v20/i08/paper) • Lipsey, M.W., & Wilson, D.B. (2001). Practical meta-analysis. Sage: Thousand Oaks, CA. External links Online Applications • Free Effect Size Calculator for Multiple Regression (http://www.danielsoper.com/statcalc/calc05.aspx) • Free Effect Size Calculator for Hierarchical Multiple Regression (http://www.danielsoper.com/statcalc/calc13. aspx) • Copylefted Effect Size Confidence Interval R Code with RWeb service for t-test, ANOVA, regression, and RMSEA (http://xiaoxu.lxxm.com/ncp) Software • compute.es: Compute Effect Sizes (http://cran.r-project.org/web/packages/compute.es/index.html) (R package) • MBESS (http://cran.r-project.org/web/packages/MBESS/index.html) - One of R's packages providing confidence intervals of effect sizes based non-central parameters • MIX 2.0 (http://www.meta-analysis-made-easy.com) Software for professional meta-analysis in Excel. Many effect sizes available. • Effect Size Calculators (http://myweb.polyu.edu.hk/~mspaul/calculator/calculator.html) Calculate d and r from a variety of statistics. Effect size 130 • Free Effect Size Generator (http://www.clintools.com/victims/resources/software/effectsize/ effect_size_generator.html) - PC & Mac Software • Free GPower Software (http://www.psycho.uni-duesseldorf.de/aap/projects/gpower/) - PC & Mac Software • ES-Calc: a free add-on for Effect Size Calculation in ViSta 'The Visual Statistics System' (http://www.mdp. edu.ar/psicologia/vista/vista.htm). Computes Cohen's d, Glass's Delta, Hedges' g, CLES, Non-Parametric Cliff's Delta, d-to-r Conversion, etc. Further Explanations • Effect Size (ES) (http://www.uccs.edu/~faculty/lbecker/es.htm) • EffectSizeFAQ.com (http://effectsizefaq.com/) • Measuring Effect Size (http://davidmlane.com/hyperstat/effect_size.html) • Effect size for two independent groups (http://web.uccs.edu/lbecker/Psy590/es.htm#II.independent) • Effect size for two dependent groups (http://web.uccs.edu/lbecker/Psy590/es.htm#III.Effect size measures for two dependent) • Computing and Interpreting Effect size Measures with ViSta (http://www.tqmp.org/doc/vol5-1/p25-34.pdf) Time series In statistics, signal processing, pattern recognition, econometrics, mathematical finance, weather forecasting, earthquake prediction, electroencephalography, control engineering and communications engineering a time series is a sequence of data points, measured typically at successive points in time spaced at uniform time intervals. Examples of time series are the daily closing value of the Dow Jones index and the annual flow volume of the Nile River at Aswan. Time series are very frequently plotted via line charts. Time series analysis comprises methods for analyzing Time series: random data plus trend, with best-fit line and different time series data in order to extract meaningful statistics smoothings and other characteristics of the data. Time series forecasting is the use of a model to predict future values based on previously observed values. While regression analysis is often employed in such a way as to test theories that the current value of one time series affects the current value of another time series, this type of analysis of time series is not called "time series analysis". Time series data have a natural temporal ordering. This makes time series analysis distinct from other common data analysis problems, in which there is no natural ordering of the observations (e.g. explaining people's wages by reference to their respective education levels, where the individuals' data could be entered in any order). Time series analysis is also distinct from spatial data analysis where the observations typically relate to geographical locations (e.g. accounting for house prices by the location as well as the intrinsic characteristics of the houses). A stochastic model for a time series will generally reflect the fact that observations close together in time will be more closely related than observations further apart. In addition, time series models will often make use of the natural one-way ordering of time so that values for a given period will be expressed as deriving in some way from past values, rather than from future values (see time reversibility.) Methods for time series analyses may be divided into two classes: frequency-domain methods and time-domain methods. The former include spectral analysis and recently wavelet analysis; the latter include auto-correlation and Time series 131 cross-correlation analysis. Additionally time series analysis techniques may be divided into parametric and non-parametric methods. The parametric approaches assume that the underlying stationary stochastic process has a certain structure which can be described using a small number of parameters (for example, using an autoregressive or moving average model). In these approaches, the task is to estimate the parameters of the model that describes the stochastic process. By contrast, non-parametric approaches explicitly estimate the covariance or the spectrum of the process without assuming that the process has any particular structure. Additionally methods of time series analysis may be divided into linear and non-linear, univariate and multivariate. Time series analysis can be applied to: • real-valued, continuous data • discrete numeric data • discrete symbolic data (i.e. sequences of characters, such as letters and words in English language[1]). Analysis There are several types of motivation and data analysis available for time series which are appropriate for different purposes. Motivation In the context of statistics, econometrics, quantitative finance, seismology, meteorology, and geophysics the primary goal of time series analysis is forecasting. In the context of signal processing, control engineering and communication engineering it is used for signal detection and estimation, while in the context of data mining, pattern recognition and machine learning time series analysis can be used for clustering, classification, query by content, anomaly detection as well as forecasting. Exploratory analysis The clearest way to examine a regular time series manually is with a line chart such as the one shown for tuberculosis in the United States, made with a spreadsheet program. The number of cases was standardized to a rate per 100,000 and the percent change per year in this rate was calculated. The nearly steadily dropping line shows that the TB incidence was decreasing in most years, but the percent change in this rate varied by as much as +/- 10%, with 'surges' in 1975 and around the early 1990s. The use of both vertical axes allows the comparison of two time series in one graphic. Other techniques Tuberculosis incidence US 1953-2009 include: • Autocorrelation analysis to examine serial dependence • Spectral analysis to examine cyclic behaviour which need not be related to seasonality. For example, sun spot activity varies over 11 year cycles.[2][3] Other common examples include celestial phenomena, weather patterns, neural activity, commodity prices, and economic activity. • Separation into components representing trend, seasonality, slow and fast variation, and cyclical irregularity: see trend estimation and decomposition of time series Time series 132 Prediction and forecasting • Fully formed statistical models for stochastic simulation purposes, so as to generate alternative versions of the time series, representing what might happen over non-specific time-periods in the future • Simple or fully formed statistical models to describe the likely outcome of the time series in the immediate future, given knowledge of the most recent outcomes (forecasting). • Forecasting on time series is usually done using automated statistical software packages and programming languages, such as R (programming language), S (programming language), SAS (software), SPSS, Minitab and many others. Classification • Assigning time series pattern to a specific category, for example identify a word based on series of hand movements in sign language See main article: Statistical classification Regression analysis • Estimating future value of a signal based on its previous behavior, e.g. predict the price of AAPL stock based on its previous price movements for that hour, day or month, or predict position of Apollo 11 spacecraft at a certain future moment based on its current trajectory (i.e. time series of its previous locations).[4] • Regression analysis is usually based on statistical interpretation of time series properties in time domain, pioneered by statisticians George Box and Gwilym Jenkins in the 50s: see Box–Jenkins Signal Estimation • This approach is based on harmonic analysis and filtering of signals in the frequency domain using the Fourier transform, and spectral density estimation, the development of which was significantly accelerated during World War II by mathematician Norbert Wiener, electrical engineers Rudolf E. Kálmán, Dennis Gabor and others for filtering signals from noise and predicting signal values at a certain point in time. See Kalman Filter, Estimation theory and Digital Signal Processing Models Models for time series data can have many forms and represent different stochastic processes. When modeling variations in the level of a process, three broad classes of practical importance are the autoregressive (AR) models, the integrated (I) models, and the moving average (MA) models. These three classes depend linearly[5] on previous data points. Combinations of these ideas produce autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA) models. The autoregressive fractionally integrated moving average (ARFIMA) model generalizes the former three. Extensions of these classes to deal with vector-valued data are available under the heading of multivariate time-series models and sometimes the preceding acronyms are extended by including an initial "V" for "vector", as in VAR for vector autoregression. An additional set of extensions of these models is available for use where the observed time-series is driven by some "forcing" time-series (which may not have a causal effect on the observed series): the distinction from the multivariate case is that the forcing series may be deterministic or under the experimenter's control. For these models, the acronyms are extended with a final "X" for "exogenous". Non-linear dependence of the level of a series on previous data points is of interest, partly because of the possibility of producing a chaotic time series. However, more importantly, empirical investigations can indicate the advantage of using predictions derived from non-linear models, over those from linear models, as for example in nonlinear autoregressive exogenous models. Time series 133 Among other types of non-linear time series models, there are models to represent the changes of variance over time (heteroskedasticity). These models represent autoregressive conditional heteroskedasticity (ARCH) and the collection comprises a wide variety of representation (GARCH, TARCH, EGARCH, FIGARCH, CGARCH, etc.). Here changes in variability are related to, or predicted by, recent past values of the observed series. This is in contrast to other possible representations of locally varying variability, where the variability might be modelled as being driven by a separate time-varying process, as in a doubly stochastic model. In recent work on model-free analyses, wavelet transform based methods (for example locally stationary wavelets and wavelet decomposed neural networks) have gained favor. Multiscale (often referred to as multiresolution) techniques decompose a given time series, attempting to illustrate time dependence at multiple scales. See also Markov switching multifractal (MSMF) techniques for modeling volatility evolution. Notation A number of different notations are in use for time-series analysis. A common notation specifying a time series X that is indexed by the natural numbers is written X = {X1, X2, ...}. Another common notation is Y = {Yt: t ∈ T}, where T is the index set. Conditions There are two sets of conditions under which much of the theory is built: • Stationary process • Ergodic process However, ideas of stationarity must be expanded to consider two important ideas: strict stationarity and second-order stationarity. Both models and applications can be developed under each of these conditions, although the models in the latter case might be considered as only partly specified. In addition, time-series analysis can be applied where the series are seasonally stationary or non-stationary. Situations where the amplitudes of frequency components change with time can be dealt with in time-frequency analysis which makes use of a time–frequency representation of a time-series or signal.[6] Models The general representation of an autoregressive model, well known as AR(p), is where the term εt is the source of randomness and is called white noise. It is assumed to have the following characteristics: • • • With these assumptions, the process is specified up to second-order moments and, subject to conditions on the coefficients, may be second-order stationary. If the noise also has a normal distribution, it is called normal or Gaussian white noise. In this case, the AR process may be strictly stationary, again subject to conditions on the coefficients. Tools for investigating time-series data include: Time series 134 • Consideration of the autocorrelation function and the spectral density function (also cross-correlation functions and cross-spectral density functions) • Scaled cross- and auto-correlation functions[7] • Performing a Fourier transform to investigate the series in the frequency domain • Use of a filter to remove unwanted noise • Principal components analysis (or empirical orthogonal function analysis) • Singular spectrum analysis • "Structural" models: • General State Space Models • Unobserved Components Models • Machine Learning • Artificial neural networks • Support Vector Machine • Fuzzy Logic • Hidden Markov model • Control chart • Shewhart individuals control chart • CUSUM chart • EWMA chart • Detrended fluctuation analysis • Dynamic time warping • Dynamic Bayesian network • Time-frequency analysis techniques: • Fast Fourier Transform • Continuous wavelet transform • Short-time Fourier transform • Chirplet transform • Fractional Fourier transform • Chaotic analysis • Correlation dimension • Recurrence plots • Recurrence quantification analysis • Lyapunov exponents • Entropy encoding Time series 135 Measures Time series metrics or features that can be used for time series classification or regression analysis[8]: • Univariate linear measures • Moment (mathematics) • Spectral band power • Spectral edge frequency • Accumulated Energy (signal processing) • Characteristics of the autocorrelation function • Hjorth parameters • FFT parameters • Autoregressive model parameters • Mann–Kendall test • Univariate non-linear measures • Measures based on the correlation sum • Correlation dimension • Correlation integral • Correlation density • Correlation entropy • Approximate Entropy[9] • Sample entropy • Fourier entropy • Wavelet entropy • Rényi entropy • Higher-order methods • Marginal predictability • Dynamical similarity index • State space dissimilarity measures • Lyapunov exponent • Permutation methods • Local flow • Other univariate measures • Algorithmic complexity • Kolmogorov complexity estimates • Hidden Markov Model states • Surrogate time series and surrogate correction • Loss of recurrence (degree of non-stationarity) • Bivariate linear measures • Maximum linear cross-correlation • Linear Coherence (signal processing) • Bivariate non-linear measures • Non-linear interdependence • Dynamical Entrainment (physics) • Measures for Phase synchronization • Similarity measures[10]: • Dynamic Time Warping Time series 136 • Hidden Markov Models • Edit distance • Total correlation • Newey–West estimator • Prais-Winsten transformation • Data as Vectors in a Metrizable Space • Minkowski distance • Mahalanobis distance • Data as Time Series with Envelopes • Global Standard Deviation • Local Standard Deviation • Windowed Standard Deviation • Data Interpreted as Stochastic Series • Pearson product-moment correlation coefficient • Spearman's rank correlation coefficient • Data Interpreted as a Probability Distribution Function • Kolmogorov-Smirnov test • Cramér-von Mises criterion References [1] Lin, Jessica and Keogh, Eamonn and Lonardi, Stefano and Chiu, Bill. A symbolic representation of time series, with implications for streaming algorithms. Proceedings of the 8th ACM SIGMOD workshop on Research issues in data mining and knowledge discovery, 2003. url: http:/ / doi. acm. org/ 10. 1145/ 882082. 882086 [2] Bloomfield, P. (1976). Fourier analysis of time series: An introduction. New York: Wiley. [3] Shumway, R. H. (1988). Applied statistical time series analysis. Englewood Cliffs, NJ: Prentice Hall. [4] Lawson, Charles L., Hanson, Richard, J. (1987). Solving Least Squares Problems. Society for Industrial and Applied Mathematics, 1987. [5] Gershenfeld, N. (1999). The nature of mathematical modeling. p.205-08 [6] Boashash, B. (ed.), (2003) Time-Frequency Signal Analysis and Processing: A Comprehensive Reference, Elsevier Science, Oxford, 2003 ISBN ISBN 0-08-044335-4 [7] Nikolić D, Muresan RC, Feng W, Singer W (2012) Scaled correlation analysis: a better way to compute a cross-correlogram. European Journal of Neuroscience, pp. 1–21, doi:10.1111/j.1460-9568.2011.07987.x http:/ / www. danko-nikolic. com/ wp-content/ uploads/ 2012/ 03/ Scaled-correlation-analysis. pdf [8] Mormann, Florian and Andrzejak, Ralph G. and Elger, Christian E. and Lehnertz, Klaus. 'Seizure prediction: the long and winding road. Brain, 2007,130 (2): 314-33.url : http:/ / brain. oxfordjournals. org/ content/ 130/ 2/ 314. abstract [9] Land, Bruce and Elias, Damian. Measuring the "Complexity" of a time series. URL: http:/ / www. nbb. cornell. edu/ neurobio/ land/ PROJECTS/ Complexity/ [10] Ropella, G.E.P.; Nag, D.A.; Hunt, C.A.; , "Similarity measures for automated comparison of in silico and in vitro experimental results," Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE , vol.3, no., pp. 2933- 2936 Vol.3, 17-21 Sept. 2003 doi: 10.1109/IEMBS.2003.1280532 URL: http:/ / ieeexplore. ieee. org/ stamp/ stamp. jsp?tp=& arnumber=1280532& isnumber=28615 Time series 137 Further reading • Bloomfield, P. (1976). Fourier analysis of time series: An introduction. New York: Wiley. • Box, George; Jenkins, Gwilym (1976), Time series analysis: forecasting and control, rev. ed., Oakland, California: Holden-Day • Brillinger, D. R. (1975). Time series: Data analysis and theory. New York: Holt, Rinehart. & Winston. • Brigham, E. O. (1974). The fast Fourier transform. Englewood Cliffs, NJ: Prentice-Hall. • Elliott, D. F., & Rao, K. R. (1982). Fast transforms: Algorithms, analyses, applications. New York: Academic Press. • Gershenfeld, Neil (2000), The nature of mathematical modeling, Cambridge: Cambridge Univ. Press, ISBN 978-0-521-57095-4, OCLC 174825352 • Hamilton, James (1994), Time Series Analysis, Princeton: Princeton Univ. Press, ISBN 0-691-04289-6 • Jenkins, G. M., & Watts, D. G. (1968). Spectral analysis and its applications. San Francisco: Holden-Day. • Priestley, M. B. (1981). Spectral Analysis and Time Series. London: Academic Press. ISBN 978-0-12-564901-8 • Shasha, D. (2004), High Performance Discovery in Time Series, Berlin: Springer, ISBN 0-387-00857-8 • Shumway, R. H. (1988). Applied statistical time series analysis. Englewood Cliffs, NJ: Prentice Hall. • Wiener, N.(1964). Extrapolation, Interpolation, and Smoothing of Stationary Time Series.The MIT Press. • Wei, W. W. (1989). Time series analysis: Univariate and multivariate methods. New York: Addison-Wesley. • Weigend, A. S., and N. A. Gershenfeld (Eds.) (1994) Time Series Prediction: Forecasting the Future and Understanding the Past. Proceedings of the NATO Advanced Research Workshop on Comparative Time Series Analysis (Santa Fe, May 1992) MA: Addison-Wesley. • Durbin J., and Koopman S.J. (2001) Time Series Analysis by State Space Methods. Oxford University Press. External links • A First Course on Time Series Analysis (http://statistik.mathematik.uni-wuerzburg.de/timeseries/) - an open source book on time series analysis with SAS • Introduction to Time series Analysis (Engineering Statistics Handbook) (http://www.itl.nist.gov/div898/ handbook/pmc/section4/pmc4.htm) - a practical guide to Time series analysis • MATLAB Toolkit for Computation of Multiple Measures on Time Series Data Bases (http://www.jstatsoft.org/ v33/i05/paper) • A Matlab tutorial on power spectra, wavelet analysis, and coherence (http://www.nbtwiki.net/doku. php?id=tutorial:power_spectra_wavelet_analysis_and_coherence) on website with many other tutorials. Panel data 138 Panel data In statistics and econometrics, the term panel data refers to multi-dimensional data frequently involving measurements over time. Panel data contain observations on multiple phenomena observed over multiple time periods for the same firms or individuals. In biostatistics, the term longitudinal data is often used instead,[1][2] wherein a subject or cluster constitutes a panel member or individual in a longitudinal study. (Somewhat confusingly, the Stata manual entry on the "xtset" command refers to a panel member simply as a "panel".) Time series and cross-sectional data are special cases of panel data that are in one dimension only (one panel member or individual for the former, one time point for the latter). Example balanced panel: unbalanced panel: In the example above, two data sets with a two-dimensional panel structure are shown, although the second data set might be a three-dimensional structure since it has three people. Individual characteristics (income, age, sex. educ) are collected for different persons and different years. In the left data set two persons (1, 2) are observed over three years (2003, 2004, 2005). Because each person is observed every year, the left-hand data set is called a balanced panel, whereas the data set on the right hand is called an unbalanced panel, since Person 1 is not observed in year 2005 and Person 3 is not observed in 2003 or 2005. Analysis of panel data A panel has the form where is the individual dimension and is the time dimension. A general panel data regression model is written as Different assumptions can be made on the precise structure of this general model. Two important models are the fixed effects model and the random effects model. The fixed effects model is denoted as are individual-specific, time-invariant effects (for example in a panel of countries this could include geography, climate etc.) and because we assume they are fixed over time, this is called the fixed-effects model. The random effects model assumes in addition that and that is, the two error components are independent from each other. Panel data 139 Data sets which have a panel design • German Socio-Economic Panel (SOEP) • Household, Income and Labour Dynamics in Australia Survey (HILDA) • British Household Panel Survey (BHPS) • Survey of Family Income and Employment (SoFIE) • Survey of Income and Program Participation (SIPP) • Lifelong Labour Market Database (LLMDB) • Panel Study of Income Dynamics (PSID) • Korean Labor and Income Panel Study (KLIPS) • Chinese Family Panel Studies (CFPS) • German Family Panel (pairfam) • National Longitudinal Surveys (NLSY) Notes [1] Peter J. Diggle, Patrick Heagerty, Kung-Yee Liang, and Scott L. Zeger, 2002. Analysis of Longitudinal Data. 2nd ed., Oxford University Press, p. 2. [2] Garrett M. Fitzmaurice, Nan M. Laird, and James H. Ware, 2004. Applied Longitudinal Analysis. John Wiley & Sons, Inc., p. 2. References Baltagi, Badi H., 2008. Econometric Analysis of Panel Data, John Wiley & Sons, Ltd. Hsiao, Cheng, 2003. Analysis of Panel Data, Cambridge University Press. Davies, A. and Lahiri, K., 2000. "Re-examining the Rational Expectations Hypothesis Using Panel Data on Multi-Period Forecasts," Analysis of Panels and Limited Dependent Variable Models, Cambridge University Press. Davies, A. and Lahiri, K., 1995. "A New Framework for Testing Rationality and Measuring Aggregate Shocks Using Panel Data," Journal of Econometrics 68: 205-227. Frees, E., 2004. Longitudinal and Panel Data, Cambridge University Press. External links • PSID (http://psidonline.isr.umich.edu/) • KLIPS (http://www.kli.re.kr/klips) • pairfam (http://www.pairfam.uni-bremen.de/en/study.html) Truncated regression model 140 Truncated regression model Truncated regression models arise in many applications of statistics, for example in econometrics, in cases where observations with values in the outcome variable below or above certain thresholds systematically excluded from the sample. Therefore, whole observations are missing, so that neither the dependent nor the independent variable is known. Truncated regression models are often confused with censored regression models where only the value of the dependent variable is clustered at a lower threshold, an upper threshold, or both, while the value for independent variables is available. Example One example of truncated samples come from historical military height records. Many armies imposed a minimum height requirement (MHR) on soldiers. This implies that men shorter than the MHR are not included in the sample. This implies that samples drawn from such records are perforce deficient i.e., incomplete, inasmuch as a substantial portion of the underlying population's height distribution is unavailable for analysis. Consequently, without proper statistical correction, any results obtained from such deficient samples, such as means, correlations, or regression coefficients are wrong (biased). In such a case truncated regression has the considerable advantage of immediately providing consistent and unbiased estimates of the coefficients of the independent variables, as well as their standard errors, thereby allowing for further statistical inference, such as the calculation of the t-values of the estimates. References • A'Hearn, Brian (2004) "A Restricted Maximum Likelihood Estimator for Truncated Height Samples." Economics and Human Biology 2 (1), 5–20. • Komlos, John (2004) "How to (and How Not to) Analyze Deficient Height Samples: an Introduction." Historical Methods, 37, No. 4, Fall, 160–173. • Wolynetz, M.S. (1979) "Maximum Likelihood estimation in a Linear model from Confined and Censored Normal Data". Journal of the Royal Statistical Society (Series C), 28(2), 195–206 Censored regression model 141 Censored regression model Censored regression models commonly arise in econometrics in cases where the variable of interest is only observable under certain conditions. A common example is labor supply. Data are frequently available on the hours worked by employees, and a labor supply model estimates the relationship between hours worked and characteristics of employees such as age, education and family status. However, such estimates undertaken using linear regression will be biased by the fact that for people who are unemployed it is not possible to observe the number of hours they would have worked had they had employment. Still we know age, education and family status for those observations. A model commonly used to deal with censored data is the Tobit model, including variations such as the Tobit Type II, Type III, and Type IV models. These and other censored regression models are often confused with truncated regression models. Truncated regression models are used for data where whole observations are missing so that the values for the dependent and the independent variables are unknown. Censored regression models are used for data where only the value for the dependent variable (hours of work in the example above) is unknown while the values of the independent variables (age, education, family status) are still available. Censored regression models are usually estimated using maximum likelihood estimation. The general validity of this approach has been shown by Schnedler (2005) who also provides a method to find the likelihood for a broad class of applications. References • Schnedler, Wendelin (2005). "Likelihood estimation for censored random vectors". Econometric Reviews 24 (2),195–217. Poisson regression 142 Poisson regression In statistics, Poisson regression is a form of regression analysis used to model count data and contingency tables. Poisson regression assumes the response variable Y has a Poisson distribution, and assumes the logarithm of its expected value can be modeled by a linear combination of unknown parameters. A Poisson regression model is sometimes known as a log-linear model, especially when used to model contingency tables. Poisson regression models are generalized linear models with the logarithm as the (canonical) link function, and the Poisson distribution function. Regression models If is a vector of independent variables, then the model takes the form where and . Sometimes this is written more compactly as where x is now an (n + 1)-dimensional vector consisting of n independent variables concatenated to some constant, usually 1. Here θ is simply a concatenated to b. Thus, when given a Poisson regression model θ and an input vector , the predicted mean of the associated Poisson distribution is given by If Yi are independent observations with corresponding values xi of the predictor variable, then θ can be estimated by maximum likelihood. The maximum-likelihood estimates lack a closed-form expression and must be found by numerical methods. The probability surface for maximum-likelihood Poisson regression is always convex, making Newton–Raphson or other gradient-based methods appropriate estimation techniques. Maximum likelihood-based parameter estimation Given a set of parameters θ and an input vector x, the mean of the predicted Poisson distribution, as stated above, is given by , and thus, the Poisson distribution's probability mass function is given by Now suppose we are given a data set consisting of m vectors , along with a set of m values . Then, for a given set of parameters θ, the probability of attaining this particular set of data is given by By the method of maximum likelihood, we wish to find the set of parameters θ that makes this probability as large as possible. To do this, the equation is first rewritten as a likelihood function in terms of θ: . Poisson regression 143 Note that the expression on the right hand side has not actually changed. A formula in this form is typically difficult to work with; instead, one uses the log-likelihood: . Notice that the parameters θ only appear in the first two terms of each term in the summation. Therefore, given that we are only interested in finding the best value for θ we may drop the yi! and simply write . To find a maximum, we need to solve an equation which has no closed-form solution. However, the negative log-likelhood, , is a convex function, and so standard convex optimization techniques such as gradient descent can be applied to find the optimal value of θ. Poisson regression in practice Poisson regression may be appropriate when the dependent variable is a count, for instance of events such as the arrival of a telephone call at a call centre. The events must be independent in the sense that the arrival of one call will not make another more or less likely, but the probability per unit time of events is understood to be related to covariates such as time of day. "Exposure" and offset Poisson regression may also be appropriate for rate data, where the rate is a count of events occurring to a particular unit of observation, divided by some measure of that unit's exposure. For example, biologists may count the number of tree species in a forest, and the rate would be the number of species per square kilometre. Demographers may model death rates in geographic areas as the count of deaths divided by person−years. More generally, event rates can be calculated as events per unit time, which allows the observation window to vary for each unit. In these examples, exposure is respectively unit area, person−years and unit time. In Poisson regression this is handled as an offset, where the exposure variable enters on the right-hand side of the equation, but with a parameter estimate (for log(exposure)) constrained to 1. which implies Offset in the case of a GLM in R can be achieved using the offset() function: glm.fit <- glm(y ~ offset(log(exposure)) + x, family=poisson(link=log) ) Overdispersion A characteristic of the Poisson distribution is that its mean is equal to its variance. In certain circumstances, it will be found that the observed variance is greater than the mean; this is known as overdispersion and indicates that the model is not appropriate. A common reason is the omission of relevant explanatory variables, or dependent observations. Under some circumstances, the problem of overdispersion can be solved by using a negative binomial distribution instead.[1][2] Another common problem with Poisson regression is excess zeros: if there are two processes at work, one determining whether there are zero events or any events, and a Poisson process determining how many events there are, there will be more zeros than a Poisson regression would predict. An example would be the distribution of Poisson regression 144 cigarettes smoked in an hour by members of a group where some individuals are non-smokers. Other generalized linear models such as the negative binomial model may function better in these cases. Use in survival analysis Poisson regression creates proportional hazards models, one class of survival analysis: see proportional hazards models for descriptions of Cox models. Tests of over-dispersion One method for testing for over dispersion in the data is to regress a variable zi against the predicted values of t estimated from the Poisson regression.[3] This test has three steps. 1. Estimate a Poisson regression of yi on xi and generate the predicted values ( ti ) 2. Calculate the zi variable 3. Regress zi against ti with ordinary least squares. In symbols where a is a constant and ei is a random variable with an expectation of zero. The null hypothesis being tested here is that the data are Poisson-distributed: in this case a = 0. Extensions Regularized Poisson regression When estimating the parameters for Poisson regression, one typically tries to find values for θ that maximize the likelihood of an expression of the form where m is the number of examples in the data set, and is the probability mass function of the Poisson distribution with the mean set to . Regularization can be added to this optimization problem by instead maximizing , for some positive constant . This technique, similar to ridge regression, can reduce overfitting. Implementations Some statistics packages include implementations of Poisson regression. • MATLAB Statistics Toolbox: Poisson regression can be performed using the "glmfit" and "glmval" functions.[4] • Microsoft Excel: Excel is not capable of doing Poisson regression by default. One of the Excel Add-ins for Poisson regression is XPost [5] • R: The function for fitting a generalized linear model in R is glm(), and can be used for Poisson Regression • SAS: Poisson regression in SAS is done by using GENMOD • SPSS: In SPSS, Poisson regression is done by using the GENLIN command • Stata: Stata has a procedure for Poisson regression named "poisson" • mPlus: mPlus allows for Poisson regression using the command COUNT IS when specifying the data Poisson regression 145 References • Cameron, A.C. and P.K. Trivedi (1998). Regression analysis of count data, Cambridge University Press. ISBN 0-521-63201-3 • Christensen, Ronald (1997). Log-linear models and logistic regression. Springer Texts in Statistics (Second ed.). New York: Springer-Verlag. pp. xvi+483. ISBN 0-387-98247-7. MR1633357. • Hilbe, J. M. (2007). Negative Binomial Regression, Cambridge University Press. ISBN 978-0-521-85772-7 [1] Paternoster R, Brame R (1997). "Multiple routes to delinquency? A test of developmental and general theories of crime". Criminology 35: 45–84. [2] Berk R, MacDonald J (2008). "Overdispersion and Poisson regression" (http:/ / www. crim. upenn. edu/ faculty/ papers/ berk/ regression. pdf). Journal of Quantitative Criminology 24: 269–284. . [3] https:/ / files. nyu. edu/ mrg217/ public/ count. pdf [4] http:/ / www. mathworks. com/ help/ toolbox/ stats/ glmfit. html [5] http:/ / www. indiana. edu/ ~jslsoc/ files_research/ xpost/ xpost. pdf Discrete choice In economics, discrete choice problems involve choices between two or more discrete alternatives, such as entering or not entering the labor market, or choosing between modes of transport. Such choices contrast with standard consumption models in which the quantity of each good consumed is assumed to be a continuous variable. In the continuous case, calculus methods (e.g. first-order conditions) can be used to determine the optimum, and demand can be modeled using regression analysis. On the other hand, discrete choice analysis examines situations in which the potential outcomes are discrete, such that the optimum is not characterized by standard first-order conditions. Loosely, regression analysis examines “how much” while discrete choice analysis examines “which.” However, discrete choice analysis can be and has been used to examine the chosen quantity in particular situations, such as the number of vehicles a household chooses to own [1] and the number of minutes of telecommunications service a customer decides to use.[2] Discrete choice models are statistical procedures that model choices made by people among a finite set of alternatives. The models have been used to examine, e.g., the choice of which car to buy,[1][3] where to go to college,[4] , which mode of transport (car, bus, rail) to take to work[5] among numerous other applications. Discrete choice models are also used to examine choices by organizations, such as firms or government agencies. In the discussion below, the decision-making unit is assumed to be a person, though the concepts are applicable more generally. Daniel McFadden won the Nobel prize in 2000 for his pioneering work in developing the theoretical basis for discrete choice. Discrete choice models statistically relate the choice made by each person to the attributes of the person and the attributes of the alternatives available to the person. For example, the choice of which car a person buys is statistically related to the person’s income and age as well as to price, fuel efficiency, size, and other attributes of each available car. The models estimate the probability that a person chooses a particular alternative. The models are often used to forecast how people’s choices will change under changes in demographics and/or attributes of the alternatives. Discrete choice 146 Applications • Marketing researchers use discrete choice models to study consumer demand and to predict competitive business responses, enabling choice modelers to solve a range of business problems, such as pricing, product development, and demand estimation problems.[1] • Transportation planners use discrete choice models to predict demand for planned transportation systems, such as which route a driver will take and whether someone will take rapid transit systems.[5][6] The first applications of discrete choice models were in transportation planning, and much of the most advanced research in discrete choice models is conducted by transportation researchers. • Energy forecasters and policymakers use discrete choice models for households’ and firms’ choice of heating system, appliance efficiency levels, and fuel efficiency level of vehicles.[7][8] • Environmental studies utilize discrete choice models to examine the recreators’ choice of, e.g., fishing or skiing site and to infer the value of amenities, such as campgrounds, fish stock, and warming huts, and to estimate the value of water quality improvements.[9] • Labor economists use discrete choice models to examine participation in the work force, occupation choice, and choice of college and training programs.[4] Common Features of Discrete Choice Models Discrete choice models take many forms, including: Binary Logit, Binary Probit, Multinomial Logit, Conditional Logit, Multinomial Probit, Nested Logit, Generalized Extreme Value Models, Mixed Logit, and Exploded Logit. All of these models have the features described below in common. Choice Set The choice set is the set of alternatives that are available to the person. For a discrete choice model, the choice set must meet three requirements: 1. The set of alternatives must be exhaustive, meaning that the set includes all possible alternatives. This requirement implies that the person necessarily does choose an alternative from the set. 2. The alternatives must be mutually exclusive, meaning that choosing one alternative means not choosing any other alternatives. This requirement implies that the person chooses only one alternative from the set. 3. The set must contain a finite number of alternatives. This third requirement distinguishes discrete choice analysis from regression analysis in which the dependent variable can (theoretically) take an infinite number of values. • Example: The choice set for a person deciding which mode of transport to take to work includes driving alone, carpooling, taking bus, etc. The choice set is complicated by the fact that a person can use multiple modes for a given trip, such as driving a car to a train station and then taking train to work. In this case, the choice set can include each possible combination of modes. Alternatively, the choice can be defined as the choice of “primary” mode, with the set consisting of car, bus, rail, and other (e.g. walking, bicycles, etc.). Note that “other” alternative is used to make the choice set exhaustive. Different people may have different choice sets, depending on their circumstances. For instance, Toyota-owned Scion is not sold in Canada as of 2009, so new car buyers in Canada face different choice sets from those of American consumers. Discrete choice 147 Defining Choice Probabilities A discrete choice model specifies the probability that a person chooses a particular alternative, with the probability expressed as a function of observed variables that relate to the alternatives and the person. In its general form, the probability that person n chooses alternative i is expressed as: where is a vector of attributes of alternative i faced by person n, is a vector of attributes of the other alternatives (other than i) faced by person n, is a vector of characteristics of person n, and is a set of parameters that relate variables to probabilities, which are estimated statistically. In the mode of transport example above, the attributes of modes (xni), such as travel time and cost, and the characteristics of consumer (sn), such as annual income, age, and gender, can be used to calculate choice probabilities. The attributes of the alternatives can differ over people; e.g., cost and time for travel to work by car, bus, and rail are different for each person depending on the location of home and work of that person. Properties: • Pni is between 0 and 1 • where J is the total number of alternatives. • Expected share choosing i where N is the number of people making the choice. Different models (i.e. different function G) have different properties. Prominent models are introduced below. Consumer Utility Discrete choice models can be derived from utility theory. This derivation is useful for three reasons: 1. It gives a precise meaning to the probabilities Pni 2. It motivates and distinguishes alternative model specifications, e.g., G’s. 3. It provides the theoretical basis for calculation of changes in consumer surplus (compensating variation) from changes in the attributes of the alternatives. Uni is the utility (or net benefit or well-being) that person n obtains from choosing alternative i. The behavior of the person is utility-maximizing: person n chooses the alternative that provides the highest utility. The choice of the person is designated by dummy variables, yni, for each alternative: Consider now the researcher who is examining the choice. The person’s choice depends on many factors, some of which the researcher observes and some of which the researcher does not. The utility that the person obtains from choosing an alternative is decomposed into a part that depends on variables that the researcher observes and a part that depends on variables that the researcher does not observe. In a linear form, this decomposition is expressed as where is a vector of observed variables relating to alternative i for person n that depends on attributes of the alternative, xni, interacted perhaps with attributes of the person, sn, such that it can be expressed as for some numerical function z, is a corresponding vector of coefficients of the observed variables, and captures the impact of all unobserved factors that affect the person’s choice. Discrete choice 148 The choice probability is then Given β, the choice probability is the probability that the random terms, εnj − εni (which are random from the researcher’s perspective, since the researcher does not observe them) are below the respective quantities . Different choice models (i.e. different specifications of G) arise from different distributions of εni for all i and different treatments of β. Properties of Discrete Choice Models Implied by Utility Theory Only differences matter The probability that a person chooses a particular alternative is determined by comparing the utility of choosing that alternative to the utility of choosing other alternatives: As the last term indicates, the choice probability depends only on the difference in utilities between alternatives, not on the absolute level of utilities. Equivalently, adding a constant to the utilities of all the alternatives does not change the choice probabilities. Scale must be normalized Since utility has no units, it is necessary to normalize the scale of utilities. The scale of utility is often defined by the variance of the error term in discrete choice models. This variance may differ depending on the characteristics of the dataset, such as when or where the data are collected. Normalization of the variance therefore affects the interpretation of parameters estimated across diverse datasets. Prominent Types of Discrete Choice Models Discrete choice models can first be classified according to the number of available alternatives. * Binomial choice models (dichotomous): 2 available alternatives * Multinomial choice models (polytomous): 3 or more available alternatives Multinomial choice models can further be classified according to the model specification: * Models, such as standard logit, that assume no correlation in unobserved factors over alternatives * Models that allow correlation in unobserved factors among alternatives In addition, specific forms of the models are available for examining rankings of alternatives (i.e., first choice, second choice, third choice, etc.) and for ratings data. Details for each model are provided in the following sections. Discrete choice 149 Binary Choice A. Logit with attributes of the person but no attributes of the alternatives Un is the utility (or net benefit) that person n obtains from taking an action (as opposed to not taking the action). The utility the person obtains from taking the action depends on the characteristics of the person, some of which are observed by the researcher and some are not: The person takes the action, yn = 1, if Un > 0. The unobserved term, εn, is assumed to have a logistic distribution. The specification is written succinctly as: • Un = βsn + εn • • ε ∼ Logistic, Then the probability of taking the action is B. Probit with attributes of the person but no attributes of the alternatives The description of the model is the same as model A, except the unobserved terms are distributed standard normal instead of logistic. • Un = βsn + εn • • ε ∼ Standard normal, Then the probability of taking the action is , where Φ() is cumulative distribution function of standard normal. C. Logit with variables that vary over alternatives Uni is the utility person n obtains from choosing alternative i. The utility of each alternative depends on the attributes of the alternatives interacted perhaps with the attributes of the person. The unobserved terms are assumed to have an extreme value distribution.[10] • Un1 = βzn1 + εn1, • Un2 = βzn2 + εn2, • iid extreme value, which gives this expression for the probability We can relate this specification to model A above, which is also binary logit. In particular, Pn1 can also be expressed as Discrete choice 150 Note that if two error terms are iid extreme value,[10] their difference is distributed logistic, which is the basis for the equivalence of the two specifications. D. Probit with variables that vary over alternatives The description of the model is the same as model C, except the difference of the two unobserved terms are distributed standard normal instead of logistic. Then the probability of taking the action is where Φ is the cumulative distribution function of standard normal. Multinomial Choice without Correlation Among Alternatives E. Logit with attributes of the person but no attributes of the alternatives The utility for all alternatives depends on the same variables, sn, but the coefficients are different for different alternatives: • Uni = βi'sn + εni, • Since only differences in utility matter, it is necessary to normalize for one alternative. Assuming , • εni ∼ iid extreme value [10] The choice probability takes the form where J is the total number of alternatives. F. Logit with variables that vary over alternatives (also called conditional logit) The utility for each alternative depends on attributes of that alternative, interacted perhaps with attributes of the person: • Uni = βzni + εni, • εni ∼ iid extreme value,[10] The choice probability takes the form where J is the total number of alternatives. Note that model E can be expressed in the same form as model F by appropriate respecification of variables. • Let be a dummy variable that identifies alternative k: • Multiply sn from model E with each of these dummies: . • Then, model F is obtained by using and , where J is the number of alternatives. Discrete choice 151 Multinomial Choice with Correlation Among Alternatives A standard logit model is not always suitable, since it assumes that there is no correlation in unobserved factors over alternatives. This lack of correlation translates into a particular pattern of substitution among alternatives that might not always be realistic in a given situation. This pattern of substitution is often called the Independence of Irrelevant Alternatives (IIA) property of standard logit models. See the Red Bus/Blue Bus example [11] or path choice example.[11] A number of models have been proposed to allow correlation over alternatives and more general substitution patterns: • Nested Logit Model - Captures correlations between alternatives by partitioning the choice set into 'nests' • Cross-nested Logit model[12] (CNL) - Alternatives may belong to more than one nest • C-logit Model[13] - Captures correlations between alternatives using 'commonality factor' • Paired Combinatorial Logit Model[14] - Suitable for route choice problems. • Generalised Extreme Value Model[15] - General class of model, derived from the random utility model[11] to which multinomial logit and nested logit belong • Conditional probit [16][17]- Allows full covariance among alternatives using a joint normal distribution. • Mixed logit [8][9][17]- Allows any form of correlation and substitution patterns.[18] When a mixed logit is with jointly normal random terms, the models is sometimes called "multinomial probit model with logit kernel"<ref name=benakiva-bierlaire-1999/.[19] Can be applied to route choice [20] The following sections describe Nested Logit, GEV, Probit, and Mixed Logit models in detail. G. Nested Logit and Generalized Extreme Value (GEV) models The model is the same as model F except that the unobserved component of utility is correlated over alternatives rather than being independent over alternatives. • Uni = βzni + εni, • The marginal distribution of each εni is extreme value,[10] but their joint distribution allows correlation among them. • The probability takes many forms depending on the pattern of correlation that is specified. See Generalized Extreme Value. H. Multinomial Probit The model is the same as model G except that the unobserved terms are distributed jointly normal, which allows any pattern of correlation and heteroscedasticity: • Uni = βzni + εni, • The choice probability is where is the joint normal density with mean zero and covariance . • The integral for this choice probability does not have a closed form, and so the probability is approximated by quadrature or simulation [21]. • When is the identity matrix (such that there is no correlation or heteroscedasticity), the model is called independent probit. Discrete choice 152 I. Mixed Logit Mixed Logit models have become increasingly popular in recent years for several reasons. First, the model allows β to be random in addition to ε. The randomness in β accommodates random taste variation over people and correlation across alternatives that generates flexible substitution patterns. Second, the advent in simulation has made approximation of the model fairly easy. In addition, McFadden and Train[] have shown that any true choice model can be approximated, to any degree of accuracy by a mixed logit with appropriate specification of explanatory variables and distribution of coefficients. • Uni = βzni + εni, • for any distribution , where is the set of distribution parameters (e.g. mean and variance) to be estimated, • εni ∼ iid extreme value,[10] The choice probability is where is logit probability evaluated at is the total number of alternatives. The integral for this choice probability does not have a closed form, so the probability is approximated by simulation [22] . Also see Mixed logit for further details. Model Applications The models described above are adapted to accommodate rankings and ratings data. Ranking of Alternatives In many situations, a person's ranking of alternatives is observed, rather than just their chosen alternative. For example, a person who has bought a new car might be asked what he/she would have bought if that car was not offered, which provides information on the person's second choice in addition to their first choice. Or, in a survey, a respondent might be asked: Example: Rank the following cell phone calling plans from your most preferred to your least preferred. * $60 per month for unlimited anytime minutes, two-year contract with $100 early termination fee * $30 per month for 400 anytime minutes, 3 cents per minute after 400 minutes, one-year contract with $125 early termination fee * $35 per month for 500 anytime minutes, 3 cents per minute after 500 minutes, no contract or early termination fee * $50 per month for 1000 anytime minutes, 5 cents per minute after 1000 minutes, two-year contract with $75 early termination fee The models described above can be adapted to account for rankings beyond the first choice. The most prominent model for rankings data is the exploded logit and its mixed version. Discrete choice 153 J. Exploded Logit Under the same assumptions as for a standard logit (model F), the probability for a ranking of the alternatives is a product of standard logits. The model is called "exploded logit" because the choice situation that is usually represented as one logit formula for the chosen alternative is expanded ("exploded") to have a separate logit formula for each ranked alternative. The exploded logit model is the product of standard logit models with the choice set decreasing as each alternative is ranked and leaves the set of available choices in the subsequent choice. Without loss of generality, the alternatives can be relabeled to represent the person's ranking, such that alternative 1 is the first choice, 2 the second choice, etc. The choice probability of ranking J alternatives as 1, 2, …, J is then As with standard logit, the exploded logit model assumes no correlation in unobserved factors over alternatives. The exploded logit can be generalized, in the same way as the standard logit is generalized, to accommodate correlations among alternatives and random taste variation. The "mixed exploded logit" model is obtained by probability of the ranking, given above, for Lni in the mixed logit model (model I). This model is also known in econometrics as the rank ordered logit model and it was introduced in that field by Beggs, Cardell and Hausman in 1981[23] · .[24] One application is the Combes et alii paper explaining the ranking of candidates to become professor.[24] Is is also known as Plackett–Luce model in biomedical literature.[24] Ratings Data In survey, respondents are often asked to give ratings, such as: Example: Please give your rating of how well the President is doing. 1: Very badly 2: Badly 3: Fine 4: Good 5: Very good Or, Example: On a 1-5 scale where 1 means disagree completely and 5 means agree completely, how much do you agree with the following statement. "The Federal government should do more to help people facing foreclosure on their homes." A multinomial discrete-choice model can examine the responses to these questions (model G, model H, model I). However, these models are derived under the concept that the respondent obtains some utility for each possible answer and gives the answer that provides the greatest utility. It might be more natural to think that the respondent has some latent measure or index associated with the question and answers in response to how high this measure is. Ordered logit and ordered probit models are derived under this concept. Discrete choice 154 K. Ordered Logit Let Un represent the strength of survey respondent n’s feelings or opinion on the survey subject. Assume that there are cutoffs of the level of the opinion in choosing particular response. For instance, in the example of the helping people facing foreclosure, the person chooses 1. 1, if Un < a 2. 2, if a < Un < b 3. 3, if b < Un < c 4. 4, if c < Un < d 5. 5, if Un > d, for some real numbers a, b, c, d. Defining Logistic, then the probability of each possible response is: and so on up to The parameters of the model are the coefficients β and the cut-off points a − d, one of which must be normalized for identification. When there are only two possible responses, the ordered logit is the same a binary logit (model A), with one cut-off point normalized to zero. L. Ordered Probit The description of the model is the same as model K, except the unobserved terms are distributed standard normal instead of logistic. Then the choice probabilities are • Prob(choosing 1) = Φ(a − βzn), • Prob(choosing 2) = Φ(b − βzn) − Φ(a − βzn), and so on. where Φ(.) is the cumulative distribution function of standard normal. Discrete choice 155 Textbooks for further reading • McFadden, Daniel L. (1984). Econometric analysis of qualitative response models. Handbook of Econometrics, Volume II. Chapter 24. Elsevier Science Publishers BV. • Ben-Akiva, M [25] and S. Lerman [26] (1985). Discrete Choice Analysis: Theory and Application to Travel Demand [27], MIT Press. • Hensher, D. [28], J. Rose [29], and W. Greene [30] (2005). Applied Choice Analysis: A Primer [31], Cambridge University Press. • Maddala, G. (1983). Limited-dependent and Qualitative Variables in Econometrics [32], Cambridge University Press. • Train, K. (2003). Discrete Choice Methods with Simulation [33], Cambridge University Press. Notes [1] Train, K. (1986). Qualitative Choice Analysis: Theory, Econometrics, and an Application to Automobile Demand, MIT Press, Chapter 8 (http:/ / emlab. berkeley. edu/ books/ choice. html). [2] Train, K. (1987). McFadden, D., and Ben-Akiva, M., “ The Demand for Local Telephone Service: A Fully Discrete Model of Residential Call Patterns and Service Choice (http:/ / elsa. berkeley. edu/ reprints/ misc/ demand. pdf), Rand Journal of Economics, Vol. 18, No. 1, pp109-123 [3] Train, K. and Winston, C. (http:/ / www. brookings. edu/ experts/ w/ winstonc. aspx/ ) (2007). “ Vehicle Choice Behavior and the Declining Market Share of US Automakers (http:/ / elsa. berkeley. edu/ ~train/ tw104. pdf),” International Economic Review, Vol. 48, No. 4, pp. 1469-1496. [4] Fuller, WC, Manski, C., and Wise, D. (1982). " New Evidence on the Economic Determinants of Post-secondary Schooling Choices (http:/ / www. jstor. org/ pss/ 145612)." Journal of Human Resources 17(4): 477-498. [5] Train, K. (1978). “ A Validation Test of a Disaggregate Mode Choice Model (http:/ / elsa. berkeley. edu/ ~train/ valtrb. pdf)”, Transportation Research, Vol. 12, pp. 167-174. [6] Ramming, M.S. (2001). “ Network Knowledge and Route Choice (http:/ / library. mit. edu/ item/ 001107149)”. Unpublished Ph.D. Thesis, Massachusetts Institute of Technology. MIT catalogue [7] Andrew Goett, Kathleen Hudson, and Train, K (2002). "Customer Choice Among Retail Energy Suppliers," Energy Journal, Vol. 21, No. 4, pp. 1-28. [8] David Revelt and Train, K (1998). " Mixed Logit with Repeated Choices: Households' Choices of Appliance Efficiency Level (http:/ / www. jstor. org/ stable/ pdfplus/ 2646846. pdf)," Review of Economics and Statistics, Vol. 80, No. 4, pp. 647-657 [9] Train, K (1998)." Recreation Demand Models with Taste Variation (http:/ / citeseerx. ist. psu. edu/ viewdoc/ summary?doi=10. 1. 1. 27. 4879)," Land Economics, Vol. 74, No. 2, pp. 230-239. [10] The density of the extreme value distribution is ƒ(εnj) = exp( − εnj)exp( − exp( − εnj)), and the cumulative distribution function is F(εnj) = exp( − exp( − εnj)). This distribution is also called the Gumbel or type I extreme value distribution, a special type of generalized extreme value distribution. [11] Ben-Akiva, M (http:/ / cee. mit. edu/ ben-akiva/ ) and Lerman, S (http:/ / cee. mit. edu/ lerman/ ) (1985). " Discrete Choice Analysis: Theory and Application to Travel Demand (Transportation Studies) (http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?tid=8271& ttype=2)", Massachusetts: MIT Press. [12] Vovsha, P. (1997). " Application of Cross-Nested Logit Model to Mode Choice in Tel Aviv, Israel, Metropolitan Area (http:/ / trb. metapress. com/ content/ l341607q38j850j7/ )," Transportation Research Record, 1607. [13] Cascetta, E., A. Nuzzolo, F. Russo, and A.Vitetta (1996). “ A Modified Logit Route Choice Model Overcoming Path Overlapping Problems: Specification and Some Calibration Results for Interurban Networks (http:/ / www2. informatik. hu-berlin. de/ alkox/ lehre/ lvws0809/ verkehr/ logit. pdf).” In J.B. Lesort (ed.), Transportation and Traffic Theory. Proceedings from the Thirteenth International Symposium on Transportation and Traffic Theory, Lyon, France, Pergamon pp. 697–711. . [14] Chu, C. (1989). “A Paired Combinatorial Logit Model for Travel Demand Analysis.” In Proceedings of the 5th World Conference on Transportation Research, 4, Ventura, CA, pp. 295–309. [15] McFadden, D. (1978). “ Modeling the Choice of Residential Location (http:/ / cowles. econ. yale. edu/ P/ cd/ d04b/ d0477. pdf).” In A. Karlqvist et al. (eds.), Spatial Interaction Theory and Residential Location, North Holland, Amsterdam pp. 75–96 [16] J. Hausman and D. Wise (1978). "A Conditional Probit Model for Qualitative Choice: Discrete Decisions Recognizing Interdependence and Heterogenous Preferences," Econometrica, Vol. 48, No. 2, pp. 403-426 [17] Train, K(2003). " Discrete Choice Methods with Simulation (http:/ / elsa. berkeley. edu/ books/ choice2. html)", Massachusetts: Cambridge University Press. [18] McFadden, D. and Train, K. (2000). “ Mixed MNL Models for Discrete Response (http:/ / elsa. berkeley. edu/ wp/ mcfadden1198/ mcfadden1198. pdf),” Journal of Applied Econometrics, Vol. 15, No. 5, pp. 447-470, Discrete choice 156 [19] M. Ben-Akiva (http:/ / cee. mit. edu/ ben-akiva/ ) and D. Bolduc (http:/ / www. ecn. ulaval. ca/ no_cache/ professeurs/ fiche_de_professeurs/ ?tx_fsgprofs_pi1[prof]=7& tx_fsgprofs_pi1[backPid]=60) (1996). “ Multinomial Probit with a Logit Kernel and a General Parametric Specification of the Covariance Structure (http:/ / elsa. berkeley. edu/ reprints/ misc/ multinomial. pdf).” Working Paper. [20] Bekhor, S. (http:/ / www. technion. ac. il/ ~civil/ bekhor/ ), Ben-Akiva, M. (http:/ / cee. mit. edu/ ben-akiva/ ), and M.S. Ramming (2002). “ Adaptation of Logit Kernel to Route Choice Situation (http:/ / trb. metapress. com/ content/ 126847136p81w0p3/ ).” Transportation Research Record, 1805, 78–85. [21] http:/ / elsa. berkeley. edu/ choice2/ ch5. pdf [22] http:/ / elsa. berkeley. edu/ choice2/ ch6. pdf [23] Beggs, S., Cardell, S., Hausman, J., 1981. Assessing the potential demand for electric cars. Journal of Econometrics 17 (1), 1–19 (September). [24] Pierre-Philippe Combes, Laurent Linnemer, Michael Visser, Publish or peer-rich? The role of skills and networks in hiring economics professors, Labour Economics, Volume 15, Issue 3, June 2008, Pages 423-441, ISSN 0927-5371, 10.1016/j.labeco.2007.04.003. (http:/ / www. sciencedirect. com/ science/ article/ pii/ S0927537107000413) [25] http:/ / cee. mit. edu/ ben-akiva/ [26] http:/ / cee. mit. edu/ lerman/ [27] http:/ / mitpress. mit. edu/ catalog/ item/ default. asp?tid=8271& ttype=2 [28] http:/ / www. econ. usyd. edu. au/ staff/ davidh/ [29] http:/ / www. econ. usyd. edu. au/ staff/ johnr/ [30] http:/ / pages. stern. nyu. edu/ ~wgreene/ [31] http:/ / books. google. com/ books?hl=en& lr=& id=8yZrtCCABAgC& oi=fnd& pg=PR17& dq=Applied+ Choice+ Analysis:+ A+ Primer& ots=RCKM2_nbA4& sig=tKOOWUvIF3QcF-z8vN0wyxR7_4w [32] http:/ / books. google. com/ books?hl=en& lr=& id=-Ji1ZaUg7gcC& oi=fnd& pg=PR11& dq=G. S. + Maddala,+ Limited-dependent+ and+ Qualitative+ Variables+ in+ Econometrics,+ New+ York+ :+ Cambridge+ University+ Press,+ 1983. + & ots=7d1s4GmQHK& sig=knQEH5Ew6d_T-OQTzYYetvoIaJo [33] http:/ / elsa. berkeley. edu/ books/ choice2. html References Simultaneous equations model Simultaneous equation models are a form of statistical model in the form of a set of linear simultaneous equations. They are often used in econometrics. Structural and reduced form Suppose there are m regression equations of the form where i is the equation number, and t = 1, …, T is the observation index. In these equations xit is the ki×1 vector of exogenous variables, yit is the dependent variable, y−i,t is the ni×1 vector of all other endogenous variables which enter the ith equation on the right-hand side, and uit are the error terms. The “−i” notation indicates that the vector y−i,t may contain any of the y’s except for yit (since it is already present on the left-hand side). The regression coefficients βi and γi are of dimensions ki×1 and ni×1 correspondingly. Vertically stacking the T observations corresponding to the ith equation, we can write each equation in vector form as where yi and ui are T×1 vectors, Xi is a T×ki matrix of exogenous regressors, and Y−i is a T×ni matrix of endogenous regressors on the right-hand side of the ith equation. Finally, we can move all endogenous variables to the left-hand side and write the m equations jointly in vector form as This representation is known as the structural form. In this equation Y = [y1 y2 … ym] is the T×m matrix of dependent variables. Each of the matrices Y−i is in fact an ni-columned submatrix of this Y. The m×m matrix Γ, Simultaneous equations model 157 which describes the relation between the dependent variables, has a complicated structure. It has ones on the diagonal, and all other elements of each column i are either the components of the vector −γi or zeros, depending on which columns of Y were included in the matrix Y−i. The T×k matrix X contains all exogenous regressors from all equations, but without repetitions (that is, matrix X should be of full rank). Thus, each Xi is a ki-columned submatrix of X. Matrix Β has size k×m, and each of its columns consists of the components of vectors βi and zeros, depending on which of the regressors from X were included or excluded from Xi. Finally, U = [u1 u2 … um] is a T×m matrix of the error terms. Postmultiplying the structural equation by Γ −1, the system can be written in the reduced form as This is already a simple general linear model, and it can be estimated for example by ordinary least squares. Unfortunately, the task of decomposing the estimated matrix into the individual factors Β and Γ −1 is quite complicated, and therefore the reduced form is more suitable for prediction but not inference. Assumptions Firstly, the rank of the matrix X of exogenous regressors must be equal to k, both in finite samples and in the limit as T → ∞ (this later requirement means that in the limit the expression should converge to a nondegenerate k×k matrix). Matrix Γ is also assumed to be non-degenerate. Secondly, error terms are assumed to be serially independent and identically distributed. That is, if the tth row of matrix U is denoted by u(t), then the sequence of vectors {u(t)} should be iid, with zero mean and some covariance matrix Σ (which is unknown). In particular, this implies that E[U] = 0, and E[U′U] = T Σ. Lastly, the identification conditions requires that the number of unknowns in this system of equations should not exceed the number of equations. More specifically, the order condition requires that for each equation ki + ni ≤ k, which can be phrased as “the number of excluded exogenous variables is greater or equal to the number of included endogenous variables”. The rank condition of identifiability is that rank(Πi0) = ni, where Πi0 is a (k − ki)×ni matrix which is obtained from Π by crossing out those columns which correspond to the excluded endogenous variables, and those rows which correspond to the included exogenous variables. Estimation Two-stages least squares (2SLS) The simplest and the most common[1] estimation method for the simultaneous equations model is the so-called two-stage least squares method, developed independently by Theil (1953) and Basmann (1957). It is an equation-by-equation technique, where the endogenous regressors on the right-hand side of each equation are being instrumented with the regressors X from all other equations. The method is called “two-stage” because it conducts estimation in two steps:[2] Step 1: Regress Y−i on X and obtain the predicted values ; Step 2: Estimate γi, βi by the ordinary least squares regression of yi on and Xi. th If the i equation in the model is written as where Zi is a T×(ni + ki) matrix of both endogenous and exogenous regressors in the ith equation, and δi is an (ni + ki)-dimensional vector of regression coefficients, then the 2SLS estimator of δi will be given by[3] where P = X (X ′X)−1X ′ is the projection matrix onto the linear space spanned by the exogenous regressors X. Simultaneous equations model 158 Indirect least squares Indirect least squares is an approach in econometrics where the coefficients in a simultaneous equations model are estimated from the reduced form model using ordinary least squares.[4][5] For this, the structural system of equations is transformed into the reduced form first. Once the coefficients are estimated the model is put back into the structural form. Limited information maximum likelihood (LIML) The “limited information” maximum likelihood method was suggested by Anderson & Rubin (1949). The explicit formula for this estimator is:[6] where M = I − X (X ′X)−1X ′, Mi = I − Xi (Xi′Xi)−1Xi′, and λ is the smallest characteristic root of the matrix Note that when λ = 1 the LIML estimator coincides with the 2SLS estimator. Three-stage least squares (3SLS) The three-stage least squares estimator was introduced by Zellner & Theil (1962). It combines two-stage least squares (2SLS) with seemingly unrelated regressions (SUR). Notes [1] Greene (2003, p. 398) [2] Greene (2003, p. 399) [3] Greene (2003, p. 399) [4] Park, S-B. (1974) "On Indirect Least Squares Estimation of a Simultaneous Equation System", The Canadian Journal of Statistics / La Revue Canadienne de Statistique, 2 (1), 75–82 JSTOR 3314964 [5] Vajda, S., Valko, P. Godfrey, K.R. (1987) "Direct and indirect least squares methods in continuous-time parameter estimation", Automatica, 23 (6), 707–718 doi:10.1016/0005-1098(87)90027-6 [6] Amemiya (1985, p. 235) References • Amemiya, Takeshi (1985). Advanced econometrics. Cambridge, Massachusetts: Harvard University Press. ISBN 0-674-00560-0. • Anderson, T.W.; Rubin, H. (1949). "Estimator of the parameters of a single equation in a complete system of stochastic equations". Annals of Mathematical Statistics 20 (1): 46–63. JSTOR 2236803. • Basmann, R.L. (1957). "A generalized classical method of linear estimation of coefficients in a structural equation". Econometrica 25 (1): 77–83. JSTOR 1907743. • Davidson, Russell; MacKinnon, James G. (1993). Estimation and inference in econometrics. Oxford University Press. ISBN 978-0-19-506011-9. • Greene, William H. (2002). Econometric analysis (5th ed.). Prentice Hall. ISBN 0-13-066198-9. • Zellner, A.; Theil, H. (1962). "Three-stage least squares: simultaneous estimation of simultaneous equations". Econometrica 30 (1): 54–78. JSTOR 1911287. Simultaneous equations model 159 External links • About.com:economics (http://economics.about.com/library/glossary/bldef-ils.htm) Online dictionary of economics, entry for ILS Survival analysis Survival analysis is a branch of statistics which deals with death in biological organisms and failure in mechanical systems. This topic is called reliability theory or reliability analysis in engineering, and duration analysis or duration modeling in economics or event history analysis in sociology. Survival analysis attempts to answer questions such as: what is the fraction of a population which will survive past a certain time? Of those that survive, at what rate will they die or fail? Can multiple causes of death or failure be taken into account? How do particular circumstances or characteristics increase or decrease the odds of survival? To answer such questions, it is necessary to define "lifetime". In the case of biological survival, death is unambiguous, but for mechanical reliability, failure may not be well-defined, for there may well be mechanical systems in which failure is partial, a matter of degree, or not otherwise localized in time. Even in biological problems, some events (for example, heart attack or other organ failure) may have the same ambiguity. The theory outlined below assumes well-defined events at specific times; other cases may be better treated by models which explicitly account for ambiguous events. More generally, survival analysis involves the modeling of time to event data; in this context, death or failure is considered an "event" in the survival analysis literature – traditionally only a single event occurs for each subject, after which the organism or mechanism is dead or broken. Recurring event or repeated event models relax that assumption. The study of recurring events is relevant in systems reliability, and in many areas of social sciences and medical research. General formulation Survival function The object of primary interest is the survival function, conventionally denoted S, which is defined as where t is some time, T is a random variable denoting the time of death, and "Pr" stands for probability. That is, the survival function is the probability that the time of death is later than some specified time t. The survival function is also called the survivor function or survivorship function in problems of biological survival, and the reliability function in mechanical survival problems. In the latter case, the reliability function is denoted R(t). Usually one assumes S(0) = 1, although it could be less than 1 if there is the possibility of immediate death or failure. The survival function must be non-increasing: S(u) ≤ S(t) if u ≥ t. This property follows directly because T>u implies T>t. This reflects the notion that survival to a later age is only possible if all younger ages are attained. Given this property, the lifetime distribution function and event density (F and f below) are well-defined. The survival function is usually assumed to approach zero as age increases without bound, i.e., S(t) → 0 as t → ∞, although the limit could be greater than zero if eternal life is possible. For instance, we could apply survival analysis to a mixture of stable and unstable carbon isotopes; unstable isotopes would decay sooner or later, but the stable isotopes would last indefinitely. Survival analysis 160 Lifetime distribution function and event density Related quantities are defined in terms of the survival function. The lifetime distribution function, conventionally denoted F, is defined as the complement of the survival function, If F is differentiable then the derivative , which is the density function of the lifetime distribution, is conventionally denoted f, The function f is sometimes called the event density; it is the rate of death or failure events per unit time. The survival function can be expressed in terms of distribution and density functions Similarly, a survival event density function can be defined as Hazard function and cumulative hazard function The hazard function, conventionally denoted , is defined as the event rate at time t conditional on survival until time t or later (that is, T ≥ t), Force of mortality is a synonym of hazard function which is used particularly in demography and actuarial science, where it is denoted by . The term hazard rate is another synonym. The hazard function must be non-negative, λ(t) ≥ 0, and its integral over must be infinite, but is not otherwise constrained; it may be increasing or decreasing, non-monotonic, or discontinuous. An example is the bathtub curve hazard function, which is large for small values of t, decreasing to some minimum, and thereafter increasing again; this can model the property of some mechanical systems to either fail soon after operation, or much later, as the system ages. The hazard function can alternatively be represented in terms of the cumulative hazard function, conventionally denoted : so transposing signs and exponentiating or differentiating (with the chain rule) The name "cumulative hazard function" is derived from the fact that which is the "accumulation" of the hazard over time. From the definition of , we see that it increases without bound as t tends to infinity (assuming that S(t) tends to zero). This implies that must not decrease too quickly, since, by definition, the cumulative hazard has to diverge. For example, is not the hazard function of any survival distribution, because its integral Survival analysis 161 converges to 1. Quantities derived from the survival distribution Future lifetime at a given time is the time remaining until death, given survival to age . Thus, it is in the present notation. The expected future lifetime is the expected value of future lifetime. The probability of death at or before age , given survival until age , is just Therefore the probability density of future lifetime is and the expected future lifetime is where the second expression is obtained using integration by parts. For , that is, at birth, this reduces to the expected lifetime. In reliability problems, the expected lifetime is called the mean time to failure, and the expected future lifetime is called the mean residual lifetime. As the probability of an individual surviving until age t or later is S(t), by definition, the expected number of survivors at age t out of an initial population of n newborns is n × S(t), assuming the same survival function for all individuals. Thus the expected proportion of survivors is S(t). If the survival of different individuals is independent, the number of survivors at age t has a binomial distribution with parameters n and S(t), and the variance of the proportion of survivors is S(t) × (1-S(t))/n. The age at which a specified proportion of survivors remain can be found by solving the equation S(t) = q for t, where q is the quantile in question. Typically one is interested in the median lifetime, for which q = 1/2, or other quantiles such as q = 0.90 or q = 0.99. One can also make more complex inferences from the survival distribution. In mechanical reliability problems, one can bring cost (or, more generally, utility) into consideration, and thus solve problems concerning repair or replacement. This leads to the study of renewal theory and reliability theory of aging and longevity. Censoring Censoring is a form of missing data problem which is common in survival analysis. Ideally, both the birth and death dates of a subject are known, in which case the lifetime is known. If it is known only that the date of death is after some date, this is called right censoring. Right censoring will occur for those subjects whose birth date is known but who are still alive when they are lost to follow-up or when the study ends. If a subject's lifetime is known to be less than a certain duration, the lifetime is said to be left-censored. It may also happen that subjects with a lifetime less than some threshold may not be observed at all: this is called truncation. Note that truncation is different from left censoring, since for a left censored datum, we know the subject exists, but for a truncated datum, we may be completely unaware of the subject. Truncation is also common. In a so-called delayed entry study, subjects are not observed at all until they have reached a certain age. For example, people may not be observed until they have reached the age to enter school. Any deceased subjects in the pre-school age group would be unknown. Left-truncated data is common in actuarial work for life insurance and pensions (Richards, 2010). Survival analysis 162 We generally encounter right-censored data. Left-censored data can occur when a person's survival time becomes incomplete on the left side of the follow-up period for the person. As an example, we may follow up a patient for any infectious disorder from the time of his or her being tested positive for the infection. We may never know the exact time of exposure to the infectious agent.[1] Fitting parameters to data Survival models can be usefully viewed as ordinary regression models in which the response variable is time. However, computing the likelihood function (needed for fitting parameters or making other kinds of inferences) is complicated by the censoring. The likelihood function for a survival model, in the presence of censored data, is formulated as follows. By definition the likelihood function is the conditional probability of the data given the parameters of the model. It is customary to assume that the data are independent given the parameters. Then the likelihood function is the product of the likelihood of each datum. It is convenient to partition the data into four categories: uncensored, left censored, right censored, and interval censored. These are denoted "unc.", "l.c.", "r.c.", and "i.c." in the equation below. For an uncensored datum, with equal to the age at death, we have For a left censored datum, such that the age at death is known to be less than , we have For a right censored datum, such that the age at death is known to be greater than , we have For an interval censored datum, such that the age at death is known to be less than and greater than , we have An important application where interval censored data arises is current status data, where the actual occurrence of an event is only known to the extent that it known not to occurred before observation time and to have occurred before the next. Non-parametric estimation The Nelson–Aalen estimator can be used to provide a non-parametric estimate of the cumulative hazard rate function. Distributions used in survival analysis • Exponential distribution • Weibull distribution • Exponential-logarithmic distribution References [1] Singh R, Mukhopadhyay K. Survival analysis in clinical trials: Basics and must know areas. Perspect Clin Res [serial online] 2011 [cited 2011 Nov 1];2:145-8. Available from: http:/ / www. picronline. org/ text. asp?2011/ 2/ 4/ 145/ 86872 Survival analysis 163 Sources • David Collett. Modelling Survival Data in Medical Research, Second Edition. Boca Raton: Chapman & Hall/CRC. 2003. ISBN 978-1-58488-325-8 • Regina Elandt-Johnson and Norman Johnson. Survival Models and Data Analysis. New York: John Wiley & Sons. 1980/1999. • J. D. Kalbfleisch and Ross L. Prentice. The statistical analysis of failure time data. New York: John Wiley & Sons. 1980 (1st ed.), 2002 (2nd ed.) ISBN 978-0-471-36357-6 • Jerald F. Lawless. Statistical Models and Methods for Lifetime Data, 2nd edition. John Wiley and Sons, Hoboken. 2003. • Terry Therneau. "A Package for Survival Analysis in S". http://www.mayo.edu/hsr/people/therneau/survival. ps, at: http://mayoresearch.mayo.edu/mayo/research/biostat/therneau.cfm • "Engineering Statistics Handbook", NIST/SEMATEK, (http://www.itl.nist.gov/div898/handbook/) • Rausand, M. and Hoyland, A. System Reliability Theory: Models, Statistical Methods, and Applications, John Wiley & Sons, Hoboken, 2004. See web site (http://www.ntnu.no/ross/books/srt). • Richards, S. J. A handbook of parametric survival models for actuarial use. Scandinavian Actuarial Journal (http:/ /www.informaworld.com/smpp/content~db=all~content=a926035258~frm=titlelink) Singh R, Mukhopadhyay K. Survival analysis in clinical trials: Basics and must know areas. Perspect Clin Res [serial online] 2011 [cited 2011 Nov 1];2:145-8. Available from: http:/ / www. picronline. org/ text. asp?2011/ 2/ 4/ 145/86872 External links • SOCR, Survival analysis applet (http://www.socr.ucla.edu/htmls/ana/Survival_Analysis.html) and interactive learning activity (http://wiki.stat.ucla.edu/socr/index.php/ SOCR_EduMaterials_AnalysisActivities_Survival). • Survival/Failure Time Analysis (http://www.statsoft.com/textbook/stsurvan.html) @ Statistics' Textbook Page (http://www.statsoft.com/textbook/) • Survival Analysis in R (http://www.netstorm.be/home/survival) Survey methodology 164 Survey methodology In statistics, survey methodology is the field that studies the sampling of individuals from a population with a view towards making statistical inferences about the population using the sample. Polls about public opinion, such as political beliefs, are reported in the news media in democracies. Other types of survey are used for scientific purposes. Surveys provide important information for all kinds of research fields, e.g., marketing research, psychology, health professionals and sociology.[1] A survey may focus on different topics such as preferences (e.g., for a presidential candidate), behavior (smoking and drinking behavior), or factual information (e.g., income), depending on its purpose. Since survey research is always based on a sample of the population, the success of the research is dependent on the representativeness of the population of concern (see also sampling (statistics) and survey sampling). Survey methodology seeks to identify principles about the design, collection, processing, and analysis of surveys in connection to the cost and quality of survey estimates. It focuses on improving quality within cost constraints, or alternatively, reducing costs for a fixed level of quality. Survey methodology is both a scientific field and a profession. Part of the task of a survey methodologist is making a large set of decisions about thousands of individual features of a survey in order to improve it.[2] The most important methodological challenges of a survey methodologist include making decisions on how to:[2] • Identify and select potential sample members. • Contact sampled individuals and collect data from those who are hard to reach (or reluctant to respond). • Evaluate and test questions. • Select the mode for posing questions and collecting responses. • Train and supervise interviewers (if they are involved). • Check data files for accuracy and internal consistency. • Adjust survey estimates to correct for identified errors. Selecting samples Survey samples can be broadly divided into two types: probability samples and non-probability samples. Stratified sampling is a method of probability sampling such that sub-populations within an overall population are identified and included in the sample selected in a balanced way. Modes of data collection There are several ways of administering a survey. The choice between administration modes is influenced by several factors, including 1) costs, 2) coverage of the target population, 3) flexibility of asking questions, 4) respondents' willingness to participate and 5) response accuracy. Different methods create mode effects that change how respondents answer, and different methods have different advantages. The most common modes of administration can be summarized as:[3] • Telephone • Mail (post) • Online surveys • Personal in-home surveys • Personal mall or street intercept survey • Hybrids of the above. Survey methodology 165 Cross-sectional and longitudinal surveys There is a distinction between one-time (cross-sectional) surveys, which involve a single questionnaire or interview administered to each sample member, and surveys which repeatedly collect information from the sample people over time. The latter are known as longitudinal surveys. Longitudinal surveys have considerable analytical advantages but they are also challenging to implement successfully. Consequently, specialist methods have been developed to select longitudinal samples, to collect data repeatedly, to keep track of sample members over time, to keep respondents motivated to participate, and to process and analyse longitudinal survey data [4] Response formats Usually, a survey consists of a number of questions that the respondent has to answer in a set format. A distinction is made between open-ended and closed-ended questions. An open-ended question asks the respondent to formulate his own answer, whereas a closed-ended question has the respondent pick an answer from a given number of options. The response options for a closed-ended question should be exhaustive and mutually exclusive. Four types of response scales for closed-ended questions are distinguished: • Dichotomous, where the respondent has two options • Nominal-polytomous, where the respondent has more than two unordered options • Ordinal-polytomous, where the respondent has more than two ordered options • (bounded)Continuous, where the respondent is presented with a continuous scale A respondent's answer to an open-ended question can be coded into a response scale afterwards,[3] or analysed using more qualitative methods. Advantages and disadvantages Advantages • They are relatively easy to administer. • Can be developed in less time compared with other data-collection methods. • Can be cost-effective. • Few 'experts' are required to develop a survey, which may increase the reliability of the survey data. • If conducted remotely, can reduce or obviate geographical dependence. • Useful in describing the characteristics of a large population assuming the sampling is valid. • Can be administered remotely via the Web, mobile devices, mail, e-mail, telephone, etc. • Efficient at collecting information from a large number of respondents. • Statistical techniques can be applied to the survey data to determine validity, reliability, and statistical significance even when analyzing multiple variables. • Many questions can be asked about a given topic giving considerable flexibility to the analysis. • Support both between and within-subjects study designs. • A wide range of information can be collected (e.g., attitudes, values, beliefs, and behaviour). • Because they are standardized, they are relatively free from several types of errors. • Surveys are ideal for scientific research studies because they provide all the participants with a standardized stimulus. With such high reliability obtained, the researcher’s own biases are eliminated Survey methodology 166 Disadvantages The reliability of survey data may depend on the following: • Respondents' motivation, honesty, memory, and ability to respond: • Respondents may not be motivated to give accurate answers. • Respondents may be motivated to give answers that present themselves in a favorable light. • Respondents may not be fully aware of their reasons for any given action. • Structured surveys, particularly those with closed ended questions, may have low validity when researching affective variables. • Self-selection bias. Although the individuals chosen to participate in surveys are often randomly sampled, errors due to non-response may exist (see also chapter 13 of Adér et al. (2008) for more information on how to deal with non-responders and biased data). That is, people who choose to respond on the survey may be different from those who do not respond, thus biasing the estimates. For example, polls or surveys that are conducted by calling a random sample of publicly available telephone numbers will not include the responses of people with unlisted telephone numbers, mobile (cell) phone numbers, people who are unable to answer the phone (e.g., because they normally sleep during the time of day the survey is conducted, because they are at work, etc.), people who do not answer calls from unknown or unfamiliar telephone numbers. Likewise, such a survey will include a disproportionate number of respondents who have traditional, land-line telephone service with listed phone numbers, and people who stay home much of the day and are much more likely to be available to participate in the survey (e.g., people who are unemployed, disabled, elderly, etc.). • Question design. Survey question answer-choices could lead to vague data sets because at times they are relative only to a personal abstract notion concerning "strength of choice". For instance the choice "moderately agree" may mean different things to different subjects, and to anyone interpreting the data for correlation. Even 'yes' or 'no' answers are problematic because subjects may for instance put "no" if the choice "only once" is not available. Nonresponse reduction The following ways have been recommended for reducing nonresponse[5] in telephone and face-to-face surveys:[6] • Advance letter. A short letter is sent in advance to inform the sampled respondents about the upcoming survey. The style of the letter should be personalized but not overdone. First, it announces that a phone call will be made/ or an interviewer wants to make an appointment to do the survey face-to-face. Second, the research topic will be described. Last, it allows both an expression of the surveyor's appreciation of cooperation and an opening to ask questions on the survey. • Training. The interviewers are thoroughly trained in how to ask respondents questions, how to work with computers and making schedules for callbacks to respondents who were not reached. • Short introduction. The interviewer should always start with a short instruction about him or herself. She/he should give her name, the institute she is working for, the length of the interview and goal of the interview. Also it can be useful to make clear that you are not selling anything: this has been shown to lead led to a slightly higher responding rate.[7] • Respondent-friendly survey questionnaire. The questions asked must be clear, non-offensive and easy to respond to for the subjects under study. Survey methodology 167 Other methods to increase response rates • brevity – single page if possible • financial incentives • paid in advance • paid at completion • non-monetary incentives • commodity giveaways (pens, notepads) • entry into a lottery, draw or contest • discount coupons • promise of contribution to charity • preliminary notification • foot-in-the-door techniques – start with a small inconsequential request • personalization of the request – address specific individuals • follow-up requests – multiple requests • emotional appeals • bids for sympathy • convince respondent that they can make a difference • guarantee anonymity • legal compulsion (certain government-run surveys) Interviewer effects Survey methodologists have devoted much effort to determine the extent to which interviewee responses are affected by physical characteristics of the interviewer. Main interviewer traits that have been demonstrated to influence survey responses are race [8] , gender [9] and relative body weight (BMI) .[10] These interviewer effects are particularly operant when questions are related to the interviewer trait. Hence, race of interviewer has been shown to affect responses to measures regarding racial attitudes ,[11] interviewer sex responses to questions involving gender issues ,[12] and interviewer BMI answers to eating and dieting-related questions .[13] While interviewer effects have been investigated mainly for face-to-face surveys, they have also been shown to exist for interview modes with no visual contact, such as telephone surveys and in video-enhanced web surveys. The explanation typically provided for interviewer effects is that of social desirability. Survey participants may attempt to project a positive self-image in an effort to conform to the norms they attribute to the interviewer asking questions. Notes [1] http:/ / whatisasurvey. info/ [2] Groves, R.M.; Fowler, F. J.; Couper, M.P.; Lepkowski, J.M.; Singer, E.; Tourangeau, R. (2009). Survey Methodology. New Jersey: John Wiley & Sons. ISBN 978-1-118-21134-2. [3] Mellenbergh, G.J. (2008). Chapter 9: Surveys. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research Methods: A consultant's companion (pp. 183–209). Huizen, The Netherlands: Johannes van Kessel Publishing. [4] Lynn, P. (2009) (Ed.) Methodology of Longitudinal Surveys. Wiley. ISBN 0-470-01871-2 [5] Lynn, P. (2008) "The problem of non-response", chapter 3, 35-55, in International Handbook of Survey Methodology (ed.s E.de Leeuw, J.Hox & D.Dillman). Erlbaum. ISBN 0-8058-5753-2 [6] Dillman, D.A. (1978) Mail and telephone surveys: The total design method. Wiley. ISBN 0-471-21555-4 [7] De Leeuw, E.D. (2001). "I am not selling anything: Experiments in telephone introductions". Kwantitatieve Methoden, 22, 41–48. [8] Hill, M.E (2002). "Race of the interviewer and perception of skin color: Evidence from the multi-city study of urban inequality" (http:/ / www. jstor. org/ stable/ 3088935). American Sociological Review 67 (1): 99–108. . [9] Flores-Macias, F.; Lawson, C. (2008). "Effects of interviewer gender on survey responses: Findings from a household survey in Mexico". International Journal of Public Opinion Research 20 (1): 100–110. doi:10.1093/ijpor/edn007. [10] Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B.; Van Strien, T. (2011). "BMI of interviewer effects". International Journal of Public Opinion Research 23 (4): 530–543. doi:10.1093/ijpor/edr026. Survey methodology 168 [11] Anderson, B.A.; Abramson, B.D. (1988). "The effects of the race of the interviewer on race-related attitudes of black respondents in SRC/CPS national election studies". Public Opinion Quarterly 52 (3): 1–28. doi:10.1086/269108. [12] Kane, E.W.; Macaulay, L.J. (1993). "Interviewer gender and gender attitudes". Public Opinion Quarterly 57 (1): 1–28. doi:10.1086/269352. [13] Eisinga, R.; Te Grotenhuis, M.; Larsen, J.K.; Pelzer, B.. "Interviewer BMI effects on under- and over-reporting of restrained eating. Evidence from a national Dutch face-to-face survey and a postal follow-up". International Journal of Public Health 57 (3): 643-647. doi:10.1007/s00038-011-0323-z. References • Abramson, J.J. and Abramson, Z.H. (1999).Survey Methods in Community Medicine: Epidemiological Research, Programme Evaluation, Clinical Trials (5th edition). London: Churchill Livingstone/Elsevier Health Sciences ISBN 0-443-06163-7 • Groves, R.M. (1989). Survey Errors and Survey Costs Wiley. ISBN 0-471-61171-9 • Ornstein, M.D. (1998). "Survey Research." Current Sociology 46(4): iii-136. • Shaughnessy, J. J., Zechmeister, E. B., & Zechmeister, J. S. (2006). Research Methods in Psychology (Seventh Edition ed.). McGraw–Hill Higher Education. ISBN 0-07-111655-9 (pp. 143–192) • Adèr, H. J., Mellenbergh, G. J., & Hand, D. J. (2008). Advising on research methods: A consultant's companion. Huizen, The Netherlands: Johannes van Kessel Publishing. • Dillman, D.A. (1978) Mail and telephone surveys: The total design method. New York: Wiley. ISBN 0-471-21555-4 Further reading • Andres, Lesley (2012). "Designing and Doing Survey Research" (http://www.uk.sagepub.com/books/ Book234957?siteId=sage-uk&prodTypes=any&q=andres). London: Sage. • Leung, Wai-Ching (2001) "Conducting a Survey" (http://archive.student.bmj.com/back_issues/0601/ education/187.html), in Student BMJ, (British Medical Journal, Student Edition), May 2001 External links • Surveys (http://www.dmoz.org/Science/Social_Sciences/Methodology/Survey/) at the Open Directory Project • OmniPHP(tm) SurveyEngine (http://surveyengine.sourceforge.net/) – An open source advanced survey development application that allows creating any type of web-based survey. • Nonprofit Research Collection on the Use of Surveys in Nonprofit Research (http://www.issuelab.org/closeup/ Jan_2009/) Published on IssueLab • Survey Question Bank (http://www.surveynet.ac.uk/sqb) • Designing surveys – a basic guide (http://www.snapsurveys.com/surveys/) Index (economics) 169 Index (economics) In economics and finance, an index is a statistical measure of changes in a representative group of individual data points. These data may be derived from any number of sources, including company performance, prices, productivity, and employment. Economic indices (index, plural) track economic health from different perspectives. Influential global financial indices such as the Global Dow, and the NASDAQ Composite track the performance of selected large and powerful companies in order to evaluate and predict economic trends. The Dow Jones Industrial Average and the S&P 500 primarily track U.S. markets, though some legacy international companies are included.[1] The Consumer Price Index tracks the variation in prices for different consumer goods and services over time in a constant geographical location, and is integral to calculations used to adjust salaries, bond interest rates, and tax thresholds for inflation. The GDP Deflator Index, or real GDP, measures the level of prices of all new, domestically produced, final goods and services in an economy.[2] Market performance indices include the labour market index/job index and proprietary stock market index investment instruments offered by brokerage houses. Some indices display market variations that cannot be captured in other ways. For example, the Economist provides a Big Mac Index that expresses the adjusted cost of a globally ubiquitous Big Mac as a percentage over or under the cost of a Big Mac in the U.S. in USD (estimated: $3.57).[3] The least relatively expensive Big Mac price occurs in Hong Kong, at a 52% reduction from U.S. prices, or $1.71 U.S. Such indices can be used to help forecast currency values. From this example, it would be assumed that Hong Kong currency is undervalued, and provides a currency investment opportunity. Index numbers An index number is an economic data figure reflecting price or quantity compared with a standard or base value.[4][5] The base usually equals 100 and the index number is usually expressed as 100 times the ratio to the base value. For example, if a commodity costs twice as much in 1970 as it did in 1960, its index number would be 200 relative to 1960. Index numbers are used especially to compare business activity, the cost of living, and employment. They enable economists to reduce unwieldy business data into easily understood terms. In economics, index numbers generally are time series summarising movements in a group of related variables. In some cases, however, index numbers may compare geographic areas at a point in time. An example is a country's purchasing power parity. The best-known index number is the consumer price index, which measures changes in retail prices paid by consumers. In addition, a cost-of-living index (COLI) is a price index number that measures relative cost of living over time.[6] In contrast to a COLI based on the true but unknown utility function, a superlative index number is an index number that can be calculated.[6] Thus, superlative index numbers are used to provide a fairly close approximation to the underlying cost-of-living index number in a wide range of circumstances.[6] There is a substantial body of economic analysis concerning the construction of index numbers, desirable properties of index numbers and the relationship between index numbers and economic theory. Index number problem The "index number problem" refers to the difficulty of constructing a valid index when both price and quantity change over time. For instance, in the construction of price indices for inflation, the nature of goods in the economy changes over time as well as their prices. A price index constructed in 1950 using a standard basket of goods based on 1950 consumption would not well represent the prices faced by consumers in 2000, as goods in some categories are no longer traded in 2000, new categories of goods have been introduced, and the relative spending on different categories of goods will change drastically. Furthermore, the goods in the basket may have changed in quality. There is no theoretically ideal solution to this problem. In practice for retail price indices, the "basket of goods" is updated incrementally every few years to reflect changes. Nevertheless, the fact remains that many economic indices Index (economics) 170 taken over the long term are not really like-for-like comparisons and this is an issue taken into account by researchers in economic history. Indices Provider: Dow Jones • Dow Jones Industrial Average Provider: Standard & Poor's • S&P 500 • S&P 400 • S&P 600 • S&P 1500 • S&P/ASX 200 • S&P/TSX Composite Index • S&P Global 1200 • S&P Custom Group of indices • S&P Leveraged Loan Index • Case–Shiller index Provider: Russell Investments • Russell 1000 Index • Russell 2000 Index • Russell 3000 Index • Russell Midcap Index • Russell Microcap Index • Russell Global Index • Russell Developed Index • Russell Europe Index • Russell Asia Pacific Index • Russell Emerging Markets Index Provider: Morgan Stanley Capital International • MSCI World Index • MSCI EAFE (Europe, Australasia, and Far East) Index Provider: Bombay Stock Exchange • BSE SENSEX Provider: Reuters • Reuters-CRB Commodities Index Provider: Markit • ABX • CDX / iTraxx • CMBX Provider: Historic Automobile Group • HAGI Top Index Index (economics) 171 References [1] http:/ / www. investopedia. com/ university/ indexes/ index1. asp [2] http:/ / www. politonomist. com/ gdp-deflator-and-measuring-inflation-00491/ [3] http:/ / www. oanda. com/ currency/ big-mac-index [4] Diewert, W. E., "Index Numbers", in Eatwell, John; Milgate, Murray; Newman, Peter, The New Palgrave: A Dictionary of Economics, 2, pp. 767–780 [5] Moulton, Brent R.; Smith, Jeffrey W., "Price Indices", in Newman, Peter; Milgate, Murray; Eatwell, John, The New Palgrave Dictionary of Money and Finance, 3, pp. 179–181 [6] Turvey, Ralph. (2004) Consumer Price Index Manual: Theory And Practice. (http:/ / books. google. com/ books?id=HOqcFW9b5VoC& pg=PA11& dq=Superlative+ index+ number& as_brr=3& sig=0pz8BjGjaNoB-HEPK8o09xOH57Q#PPA11,M1) Page 11. Publisher: International Labour Organization. ISBN 92-2-113699-X. Further reading • Robin Marris, Economic Arithmetic, (1958). External links • Humboldt Economic Index (http:/ / www. humboldt. edu/ ~indexhum/ ) • SG Index (http:/ / www. sgindex. • S&P Indices (http:/ / www2. standardandpoors. com/ portal/ site/ sp/ en/ us/ page. category/ indices/ com/ ) 2,3,1,0,0,0,0,0,0,0,0,0,0,0,0,0. html?lid=us_topnav_indicies) • Dow Jones Indexes (http:/ / www. djindexes. com/ ) Aggregate demand In macroeconomics, aggregate demand (AD) is the total demand for final goods and services in the economy (Y) at a given time and price level.[1] It is the amount of goods and services in the economy that will be purchased at all possible price levels.[2] This is the demand for the gross domestic product of a country when inventory levels are static. It is often called effective demand, though at other times this term is distinguished. It is often cited that the aggregate demand curve is downward sloping because at lower price levels a greater quantity is demanded. While this is correct at the microeconomic, single good level, at the aggregate level this is incorrect. The aggregate demand curve is in fact downward sloping as a result of three distinct effects: Pigou's wealth effect, the Keynes' interest rate effect and the Mundell-Fleming exchange-rate effect. Components An aggregate demand curve is the sum of individual demand curves for different sectors of the economy. The aggregate demand is usually described as a linear sum of four separable demand sources.[3] where • is consumption (may also be known as consumer spending) = , • is Investment, • is Government spending, • is Net export, • is total exports, and • is total imports = . These four major parts, which can be stated in either 'nominal' or 'real' terms, are: Aggregate demand 172 • personal consumption expenditures (C) or "consumption," demand by households and unattached individuals; its determination is described by the consumption function. The consumption function is C= a + (mpc)(Y-T) • a is autonomous consumption, mpc is the marginal propensity to consume, (Y-T) is the disposable income. • gross private domestic investment (I), such as spending by business firms on factory construction. This includes all private sector spending aimed at the production of some future consumable. • In Keynesian economics, not all of gross private domestic investment counts as part of aggregate demand. Much or most of the investment in inventories can be due to a short-fall in demand (unplanned inventory accumulation or "general over-production"). The Keynesian model forecasts a decrease in national output and income when there is unplanned investment. (Inventory accumulation would correspond to an excess supply of products; in the National Income and Product Accounts, it is treated as a purchase by its producer.) Thus, only the planned or intended or desired part of investment (Ip) is counted as part of aggregate demand. (So, I does not include the 'investment' in running up or depleting inventory levels.) • Investment is affected by the output and the interest rate (i). Consequently, we can write it as I(Y,i). Investment has positive relationship with the output and negative relationship with the interest rate. For example, an increase in the interest rate will cause aggregate demand to decline. Interest costs are part of the cost of borrowing and as they rise, both firms and households will cut back on spending. This shifts the aggregate demand curve to the left. This lowers equilibrium GDP below potential GDP. As production falls for many firms, they begin to lay off workers, and unemployment rises. The declining demand also lowers the price level. The economy is in recession. • gross government investment and consumption expenditures (G). • net exports (NX and sometimes (X-M)), i.e., net demand by the rest of the world for the country's output. In sum, for a single country at a given time, aggregate demand (D or AD) = C + Ip + G + (X-M). These macrovariables are constructed from varying types of microvariables from the price of each, so these variables are denominated in (real or nominal) currency terms. Aggregate demand 173 Aggregate demand curves Understanding of the aggregate demand curve depends on whether it is examined based on changes in demand as income changes, or as price change. Aggregate demand-aggregate supply model Sometimes, especially in textbooks, "aggregate demand" refers to an entire demand curve that looks like that in a typical Marshallian supply and demand diagram. Thus, that we could refer to an "aggregate quantity demanded" (Yd = C + Ip + G + NX in real or inflation-corrected terms) at any given aggregate average price level (such as the GDP deflator), P. In these diagrams, typically the Yd rises as the average price level (P) falls, as with the AD line in the diagram. The main theoretical reason for this is that if the nominal money supply (Ms) is constant, a falling P implies that the real money supply (M s/P)rises, encouraging lower interest rates and higher spending. This is often called the "Keynes effect." Carefully using ideas from the theory of Aggregate supply/demand graph supply and demand, aggregate supply can help determine the extent to which increases in aggregate demand lead to increases in real output or instead to increases in prices (inflation). In the diagram, an increase in any of the components of AD (at any given P) shifts the AD curve to the right. This increases both the level of real production (Y) and the average price level (P). But different levels of economic activity imply different mixtures of output and price increases. As shown, with very low levels of real gross domestic product and thus large amounts of unemployed resources, most economists of the Keynesian school suggest that most of the change would be in the form of output and employment increases. As the economy gets close to potential output (Y*), we would see more and more price increases rather than output increases as AD increases. Beyond Y*, this gets more intense, so that price increases dominate. Worse, output levels greater than Y* cannot be sustained for long. The AS is a short-term relationship here. If the economy persists in operating above potential, the AS curve will shift to the left, making the increases in real output transitory. At low levels of Y, the world is more complicated. First, most modern industrial economies experience few if any falls in prices. So the AS curve is unlikely to shift down or to the right. Second, when they do suffer price cuts (as in Japan), it can lead to disastrous deflation. Aggregate demand 174 If they do not result in greater aggregate demand, increasing business profits do not necessarily lead to increased economic growth. When businesses and banks have disincentive to spend accumulated capital, such as cash repatriation taxes from profits in overseas tax havens and interest on excess reserves paid to banks, increased profits can lead to decreasing growth.[4][5] Debt Red: corporate profits after tax and inventory A Post-Keynesian theory of aggregate demand emphasizes the role of valuation adjustment. Blue: nonresidential fixed investment (roughly speaking, business debt, which it considers a fundamental component of aggregate investment), both as fractions of U.S. GDP, demand;[6] the contribution of change in debt to aggregate demand is 1989-2012. referred to by some as the credit impulse.[7] Aggregate demand is spending, be it on consumption, investment, or other categories. Spending is related to income via: Income – Spending = Net Savings Rearranging this yields: Spending = Income – Net Savings = Income + Net Increase in Debt In words: what you spend is what you earn, plus what you borrow: if you spend $110 and earned $100, then you must have net borrowed $10; conversely if you spend $90 and earn $100, then you have net savings of $10, or have reduced debt by $10, for net change in debt of –$10. If debt grows or shrinks slowly as a percentage of GDP, its impact on aggregate demand is small; conversely, if debt is significant, then changes in the dynamics of debt growth can have significant impact on aggregate demand. Change in debt is tied to the level of debt:[6] if the overall debt level is 10% of GDP and 1% of loans are not repaid, this impacts GDP by 1% of 10% = 0.1% of GDP, which is statistical noise. Conversely, if the debt level is 300% of GDP and 1% of loans are not repaid, this impacts GDP by 1% of 300% = 3% of GDP, which is significant: a change of this magnitude will generally cause a recession. Similarly, changes in the repayment rate (debtors paying down their debts) impact aggregate demand in proportion to the level of debt. Thus, as the level of debt in an economy grows, the economy becomes more sensitive to debt dynamics, and credit bubbles are of macroeconomic concern. Since write-offs and savings rates both spike in recessions, both of which result in shrinkage of credit, the resulting drop in aggregate demand can worsen and perpetuate the recession in a vicious cycle. This perspective originates in, and is intimately tied to, the debt-deflation theory of Irving Fisher, and the notion of a credit bubble (credit being the flip side of debt), and has been elaborated in the Post-Keynesian school.[6] If the overall level of debt is rising each year, then aggregate demand exceeds Income by that amount. However, if the level of debt stops rising and instead starts falling (if "the bubble bursts"), then aggregate demand falls short of income, by the amount of net savings (largely in the form of debt repayment or debt writing off, such as in bankruptcy). This causes a sudden and sustained drop in aggregate demand, and this shock is argued to be the proximate cause of a class of economic crises, properly financial crises. Indeed, a fall in the level of debt is not necessary – even a slowing in the rate of debt growth causes a drop in aggregate demand (relative to the higher borrowing year).[8] These crises then end when credit starts growing again, either because most or all debts have been repaid or written off, or for other reasons as below. From the perspective of debt, the Keynesian prescription of government deficit spending in the face of an economic crisis consists of the government net dis-saving (increasing its debt) to compensate for the shortfall in private debt: it replaces private debt with public debt. Other alternatives include seeking to restart the growth of private debt ("reflate the bubble"), or slow or stop its fall; and debt relief, which by lowering or eliminating debt stops credit from contracting (as it cannot fall below zero) and allows debt to either stabilize or grow – this has the further effect of redistributing wealth from creditors (who write off debts) to debtors (whose debts are relieved). Aggregate demand 175 Criticisms Austrian theorist Henry Hazlitt argued that aggregate demand is a meaningless concept in economic analysis.[9] Friedrich Hayek, another Austrian theorist, argued that Keynes' study of the aggregate relations in an economy is fallacious, as recessions are caused by micro-economic factors.[10] References [1] Sexton, Robert; Fortura, Peter (2005). Exploring Economics. ISBN 0-17-641482-7. "This is the sum of the demand for all final goods and services in the economy. It can also be seen as the quantity of real GDP demanded at different price levels." [2] O'Sullivan, Arthur; Steven M. Sheffrin (2003). oi = Economics: Principles in action (http:/ / www. pearsonschool. com/ index. cfm?locator=PSZ3R9& PMDbSiteId=2781& PMDbSolutionId=6724& PMDbCategoryId=& PMDbProgramId=12881& level=4). Upper Saddle River, New Jersey 07458: Pearson Prentice Hall. pp. 307. ISBN 0-13-063085-3. oi =. [3] "aggregate demand (AD)" (http:/ / www. tutor2u. net/ economics/ content/ topics/ ad_as/ ad-as_notes. htm). Archived (http:/ / web. archive. org/ web/ 20071109072803/ http:/ / www. tutor2u. net/ economics/ content/ topics/ ad_as/ ad-as_notes. htm) from the original on 9 November 2007. . Retrieved 2007-11-04. [4] "Profits and Business Investment" (http:/ / krugman. blogs. nytimes. com/ 2013/ 02/ 09/ profits-and-business-investment/ ) Paul Krugman, New York Times", February 9, 2013 [5] "Still Say’s Law After All These Years" (http:/ / krugman. blogs. nytimes. com/ 2013/ 02/ 10/ still-says-law-after-all-these-years/ ) Paul Krugman, New York Times, February 10, 2013 [6] Debtwatch No 41, December 2009: 4 Years of Calling the GFC (http:/ / www. debtdeflation. com/ blogs/ 2009/ 12/ 01/ debtwatch-no-41-december-2009-4-years-of-calling-the-gfc/ ), Steve Keen, December 1, 2009 [7] Credit and Economic Recovery: Demystifying Phoenix Miracles (http:/ / ssrn. com/ paper=1595980), Michael Biggs, Thomas Mayer, Andreas Pick, March 15, 2010 [8] "However much you borrow and spend this year, if it is less than last year, it means your spending will go into recession." Dhaval Joshi, RAB Capital, quoted in Noughty boys on trading floor led us into debt-laden fantasy (http:/ / www. smh. com. au/ business/ noughty-boys-on-trading-floor-led-us-into-debtladen-fantasy-20091222-lbs4. html) [9] Hazlitt, Henry (1959). The Failure of the 'New Economics': An Analysis of the Keynesian Fallacies (http:/ / www. mises. org/ books/ failureofneweconomics. pdf). D. Van Nostrand. . [10] Hayek, Friedrich (1989). The Collected Works of F.A. Hayek. University of Chicago Press. p. 202. ISBN 978-0-226-32097-7. External links • Elmer G. Wiens: Classical & Keynesian AD-AS Model (http://www.egwald.ca/macroeconomics/keynesian. php) - An on-line, interactive model of the Canadian Economy. Operations research 176 Operations research Operations research, or operational research in British usage, is a discipline that deals with the application of advanced analytical methods to help make better decisions.[1] It is often considered to be a sub-field of mathematics.[2] The terms management science and decision science are sometimes used as more modern-sounding synonyms.[3] Employing techniques from other mathematical sciences, such as mathematical modeling, statistical analysis, and mathematical optimization, operations research arrives at optimal or near-optimal solutions to complex decision-making problems. Because of its emphasis on human-technology interaction and because of its focus on practical applications, operations research has overlap with other disciplines, notably industrial engineering and operations management, and draws on psychology and organization science. Operations Research is often concerned with determining the maximum (of profit, performance, or yield) or minimum (of loss, risk, or cost) of some real-world objective. Originating in military efforts before World War II, its techniques have grown to concern problems in a variety of industries.[4] Overview Operational research (OR) encompasses a wide range of problem-solving techniques and methods applied in the pursuit of improved decision-making and efficiency, such as simulation, mathematical optimization, queueing theory and other stochastic-process models, Markov decision processes, econometric methods, data envelopment analysis, neural networks, expert systems, decision analysis, and the analytic hierarchy process.[5] Nearly all of these techniques involve the construction of mathematical models that attempt to describe the system. Because of the computational and statistical nature of most of these fields, OR also has strong ties to computer science and analytics. Operational researchers faced with a new problem must determine which of these techniques are most appropriate given the nature of the system, the goals for improvement, and constraints on time and computing power. The major subdisciplines in modern operational research, as identified by the journal Operations Research,[6] are: • Computing and information technologies • Environment, energy, and natural resources • Financial engineering • Manufacturing, service sciences, and supply chain management • Marketing Engineering[7] • Policy modeling and public sector work • Revenue management • Simulation • Stochastic models • Transportation. History As a formal discipline, operational research originated in the efforts of military planners during World War II. In the decades after the war, the techniques began to be applied more widely to problems in business, industry and society. Since that time, operational research has expanded into a field widely used in industries ranging from petrochemicals to airlines, finance, logistics, and government, moving to a focus on the development of mathematical models that can be used to analyse and optimize complex systems, and has become an area of active academic and industrial research.[4] Operations research 177 Historical origins In the World War II era, operational research was defined as "a scientific method of providing executive departments with a quantitative basis for decisions regarding the operations under their control."[8] Other names for it included operational analysis (UK Ministry of Defence from 1962)[9] and quantitative management.[10] Prior to the formal start of the field, early work in operational research was carried out by individuals such as Charles Babbage. His research into the cost of transportation and sorting of mail led to England's universal "Penny Post" in 1840, and studies into the dynamical behaviour of railway vehicles in defence of the GWR's broad gauge.[11] Percy Bridgman brought operational research to bear on problems in physics in the 1920s and would later attempt to extend these to the social sciences.[12] The modern field of operational research arose during World War II. Modern operational research originated at the Bawdsey Research Station in the UK in 1937 and was the result of an initiative of the station's superintendent, A. P. Rowe. Rowe conceived the idea as a means to analyse and improve the working of the UK's early warning radar system, Chain Home (CH). Initially, he analysed the operating of the radar equipment and its communication networks, expanding later to include the operating personnel's behaviour. This revealed unappreciated limitations of the CH network and allowed remedial action to be taken.[13] Scientists in the United Kingdom including Patrick Blackett (later Lord Blackett OM PRS), Cecil Gordon, C. H. Waddington, Owen Wansbrough-Jones, Frank Yates, Jacob Bronowski and Freeman Dyson, and in the United States with George Dantzig looked for ways to make better decisions in such areas as logistics and training schedules. After the war it began to be applied to similar problems in industry. Second World War During the Second World War close to 1,000 men and women in Britain were engaged in operational research. About 200 operational research scientists worked for the British Army.[14] Patrick Blackett worked for several different organizations during the war. Early in the war while working for the Royal Aircraft Establishment (RAE) he set up a team known as the "Circus" which helped to reduce the number of anti-aircraft artillery rounds needed to shoot down an enemy aircraft from an average of over 20,000 at the start of the Battle of Britain to 4,000 in 1941.[15] In 1941 Blackett moved from the RAE to the Navy, after first working with RAF Coastal Command, in 1941 and then early in 1942 to the Admiralty.[16] Blackett's team at Coastal Command's Operational Research Section (CC-ORS) included two future Nobel prize winners and many other people who went on to be pre-eminent in their fields.[17] They undertook a number of crucial analyses that aided the war effort. Britain introduced the convoy system to reduce shipping losses, but while the principle of using warships to accompany merchant ships was generally accepted, it was unclear whether it was better for convoys to be small or large. Convoys travel at the speed of the slowest member, so small convoys can travel faster. It was also argued that small convoys would be harder for German U-boats to detect. On the other hand, large convoys could deploy more warships against an attacker. Blackett's staff showed that the losses suffered by convoys depended largely on the number of escort vessels present, rather than on the overall size of the convoy. Their conclusion, therefore, was that a few large convoys are more defensible than many small ones.[18] While performing an analysis of the methods used by RAF Coastal Command to hunt and destroy submarines, one of the analysts asked what colour the aircraft were. As most of them were from Bomber Command they were painted black for nighttime operations. At the suggestion of CC-ORS a test was run to see if that was the best colour to camouflage the aircraft for daytime operations in the grey North Atlantic skies. Tests showed that aircraft painted white were on average not spotted until they were 20% closer than those painted black. This change indicated that 30% more submarines would be attacked and sunk for the same number of sightings.[19] Other work by the CC-ORS indicated that on average if the trigger depth of aerial-delivered depth charges (DCs) were changed from 100 feet to 25 feet, the kill ratios would go up. The reason was that if a U-boat saw an aircraft Operations research 178 only shortly before it arrived over the target then at 100 feet the charges would do no damage (because the U-boat wouldn't have had time to descend as far as 100 feet), and if it saw the aircraft a long way from the target it had time to alter course under water so the chances of it being within the 20-foot kill zone of the charges was small. It was more efficient to attack those submarines close to the surface when the targets' locations were better known than to attempt their destruction at greater depths when their positions could only be guessed. Before the change of settings from 100 feet to 25 feet, 1% of submerged U-boats were sunk and 14% damaged. After the change, 7% were sunk and 11% damaged. (If submarines were caught on the surface, even if attacked shortly after submerging, the numbers rose to 11% sunk and 15% damaged). Blackett observed "there can be few cases where such a great operational gain had been obtained by such a small and simple change of tactics".[20] Bomber Command's Operational Research Section (BC-ORS), analysed a report of a survey carried out by RAF Bomber Command. For the survey, Bomber Command inspected all bombers returning from bombing raids over Germany over a particular period. All damage inflicted by German air defences was noted and the recommendation was given that armour be added in the most heavily damaged areas. This recommendation was not adopted because the fact that the aircraft returned with these areas damaged indicated these areas were NOT vital, and adding armor weight to non-vital areas where damage is acceptable negatively affects aircraft performance. Their suggestion to remove some of the crew so that an aircraft loss would result in fewer personnel loss was also rejected by RAF command. Blackett's team made the logical recommendation that the armour be placed in the areas which were completely untouched by damage in the bombers which returned. They reasoned that the survey was biased, since it only included aircraft that returned to Britain. The untouched areas of returning aircraft were probably vital areas, which, if hit, would result in the loss of the aircraft.[21] When Germany organised its air defences into the Kammhuber Line, it was realised by the British that if the RAF bombers were to fly in a bomber stream they could overwhelm the night fighters who flew in individual cells directed to their targets by ground controllers. It was then a matter of calculating the statistical loss from collisions against the statistical loss from night fighters to calculate how close the bombers should fly to minimise RAF losses.[22] The "exchange rate" ratio of output to input was a characteristic feature of operational research. By comparing the number of flying hours put in by Allied aircraft to the number of U-boat sightings in a given area, it was possible to redistribute aircraft to more productive patrol areas. Comparison of exchange rates established "effectiveness ratios" useful in planning. The ratio of 60 mines laid per ship sunk was common to several campaigns: German mines in British ports, British mines on German routes, and United Map of Kammhuber Line States mines in Japanese routes.[23] Operational research doubled the on-target bomb rate of B-29s bombing Japan from the Marianas Islands by increasing the training ratio from 4 to 10 percent of flying hours; revealed that wolf-packs of three United States submarines were the most effective number to enable all members of the pack to engage targets discovered on their individual patrol stations; revealed that glossy enamel paint was more effective camouflage for night fighters than traditional dull camouflage paint finish, and the smooth paint finish increased airspeed by reducing skin friction.[23] On land, the operational research sections of the Army Operational Research Group (AORG) of the Ministry of Supply (MoS) were landed in Normandy in 1944, and they followed British forces in the advance across Europe. They analysed, among other topics, the effectiveness of artillery, aerial bombing, and anti-tank shooting. Operations research 179 After World War II With expanded techniques and growing awareness of the field at the close of the war, operational research was no longer limited to only operational, but was extended to encompass equipment procurement, training, logistics and infrastructure. Problems addressed with operational research • Critical path analysis or project planning: identifying those processes in a complex project which affect the overall duration of the project • Floorplanning: designing the layout of equipment in a factory or components on a computer chip to reduce manufacturing time (therefore reducing cost) • Network optimization: for instance, setup of telecommunications networks to maintain quality of service during outages • Allocation problems • Facility location • Assignment Problems: • Assignment problem • Generalized assignment problem • Quadratic assignment problem • Weapon target assignment problem • Bayesian search theory : looking for a target • Optimal search • Routing, such as determining the routes of buses so that as few buses are needed as possible • Supply chain management: managing the flow of raw materials and products based on uncertain demand for the finished products • Efficient messaging and customer response tactics • Automation: automating or integrating robotic systems in human-driven operations processes • Globalization: globalizing operations processes in order to take advantage of cheaper materials, labor, land or other productivity inputs • Transportation: managing freight transportation and delivery systems (Examples: LTL Shipping, intermodal freight transport) • Scheduling: • Personnel staffing • Manufacturing steps • Project tasks • Network data traffic: these are known as queueing models or queueing systems. • Sports events and their television coverage • Blending of raw materials in oil refineries • Determining optimal prices, in many retail and B2B settings, within the disciplines of pricing science Operational research is also used extensively in government where evidence-based policy is used. Operations research 180 Management science In 1967 Stafford Beer characterized the field of management science as "the business use of operations research".[24] However, in modern times the term management science may also be used to refer to the separate fields of organizational studies or corporate strategy. Like operational research itself, management science (MS) is an interdisciplinary branch of applied mathematics devoted to optimal decision planning, with strong links with economics, business, engineering, and other sciences. It uses various scientific research-based principles, strategies, and analytical methods including mathematical modeling, statistics and numerical algorithms to improve an organization's ability to enact rational and meaningful management decisions by arriving at optimal or near optimal solutions to complex decision problems. In short, management sciences help businesses to achieve their goals using the scientific methods of operational research. The management scientist's mandate is to use rational, systematic, science-based techniques to inform and improve decisions of all kinds. Of course, the techniques of management science are not restricted to business applications but may be applied to military, medical, public administration, charitable groups, political groups or community groups. Management science is concerned with developing and applying models and concepts that may prove useful in helping to illuminate management issues and solve managerial problems, as well as designing and developing new and better models of organizational excellence.[25] The application of these models within the corporate sector became known as management science.[26] Related fields Some of the fields that have considerable overlap with Operations Research and Management Science include: • Business Analytics • Logistics • Data mining • Mathematical modeling • Decision analysis • Mathematical optimization • Engineering • Probability and statistics • Financial engineering • Project management • Forecasting • Policy analysis • Game theory • Simulation • Graph theory • Social network/Transportation forecasting models • Industrial engineering • Stochastic processes • Supply chain management Applications of management science Applications of management science are abundant in industry as airlines, manufacturing companies, service organizations, military branches, and in government. The range of problems and issues to which management science has contributed insights and solutions is vast. It includes:[25] • scheduling airlines, including both planes and crew, • deciding the appropriate place to site new facilities such as a warehouse, factory or fire station, • managing the flow of water from reservoirs, • identifying possible future development paths for parts of the telecommunications industry, • establishing the information needs and appropriate systems to supply them within the health service, and • identifying and understanding the strategies adopted by companies for their information systems Management science is also concerned with so-called ”soft-operational analysis”, which concerns methods for strategic planning, strategic decision support, and Problem Structuring Methods (PSM). In dealing with these sorts of challenges mathematical modeling and simulation are not appropriate or will not suffice. Therefore, during the past Operations research 181 30 years, a number of non-quantified modeling methods have been developed. These include: • stakeholder based approaches including metagame analysis and drama theory • morphological analysis and various forms of influence diagrams. • approaches using cognitive mapping • the Strategic Choice Approach • robustness analysis Societies and journals Societies The International Federation of Operational Research Societies (IFORS)[27] is an umbrella organization for operational research societies worldwide, representing approximately 50 national societies including those in the US,[28] UK,[29] France,[30] Germany, Canada,[31] Australia,[32] New Zealand,[33] Philippines,[34] India,[35] Japan and South Africa (ORSSA).[36] The constituent members of IFORS form regional groups, such as that in Europe.[37] Other important operational research organizations are Simulation Interoperability Standards Organization (SISO)[38] and Interservice/Industry Training, Simulation and Education Conference (I/ITSEC)[39] In 2004 the US-based organization INFORMS began an initiative to market the OR profession better, including a website entitled The Science of Better[40] which provides an introduction to OR and examples of successful applications of OR to industrial problems. This initiative has been adopted by the Operational Research Society in the UK, including a website entitled Learn about OR.[41] Journals INFORMS publishes thirteen scholarly journals about operations research, including the top two journals in their class, according to 2005 Journal Citation Reports.[42] They are: • Decision Analysis[43] • Information Systems Research • INFORMS Journal on Computing • INFORMS Transactions on Education[44] (an open access journal) • Interfaces: An International Journal of the Institute for Operations Research and the Management Sciences • Management Science: A Journal of the Institute for Operations Research and the Management Sciences • Manufacturing & Service Operations Management • Marketing Science • Mathematics of Operations Research • Operations Research: A Journal of the Institute for Operations Research and the Management Sciences • Organization Science • Service Science [45] • Transportation Science. Other journals • 4OR-A Quarterly Journal of Operations Research: jointly published the Belgian, French and Italian Operations Research Societies (Springer); • Decision Sciences published by Wiley-Blackwell on behalf of the Decision Sciences Institute • European Journal of Operational Research (EJOR): Founded in 1975 and is presently by far the largest operational research journal in the world, with its around 9,000 pages of published papers per year. In 2004, its total number of citations was the second largest amongst Operational Research and Management Science journals; • INFOR Journal: published and sponsored by the Canadian Operational Research Society; Operations research 182 • International Journal of Operations Research and Information Systems (IJORIS)": an official publication of the Information Resources Management Association, published quarterly by IGI Global;[46] • Journal of Defense Modeling and Simulation (JDMS): Applications, Methodology, Technology: a quarterly journal devoted to advancing the science of modeling and simulation as it relates to the military and defense.[47] • Journal of the Operational Research Society (JORS): an official journal of The OR Society; this is the oldest continuously published journal of OR in the world, published by Palgrave;[48] • Journal of Simulation (JOS): an official journal of The OR Society, published by Palgrave;[48] • Mathematical Methods of Operations Research (MMOR): the journal of the German and Dutch OR Societies, published by Springer;[49] • Military Operations Research (MOR): published by the Military Operations Research Society; • Opsearch: official journal of the Operational Research Society of India; • OR Insight: a quarterly journal of The OR Society, published by Palgrave;[48] • Production and Operations Management, the official journal of the Production and Operations Management Society • TOP: the official journal of the Spanish Society of Statistics and Operations Research.[50] Notes [1] "About Operations Research" (http:/ / www. informs. org/ About-INFORMS/ About-Operations-Research). INFORMS.org. . Retrieved 7 January 2012. [2] "Mathematics Subject Classification" (http:/ / www. mathontheweb. org/ mathweb/ mi-mathbyclass. html). American Mathematical Society. 23 May 2011. . Retrieved 7 January 2012. [3] "Definition of Operations Research" (http:/ / www. answers. com/ topic/ operations-research). Answers.com. . Retrieved 7 January 2012. [4] "What is OR" (http:/ / www. hsor. org/ what_is_or. cfm). HSOR.org. . Retrieved 13 November 2011. [5] "Operations Research Analysts" (http:/ / www. bls. gov/ oco/ ocos044. htm). Bls.gov. . Retrieved 27 January 2012. [6] "OR / Pubs / IOL Home" (http:/ / www3. informs. org/ site/ OperationsResearch/ index. php?c=10& kat=Forthcoming+ Papers). INFORMS.org. 2 January 2009. . Retrieved 13 November 2011. [7] "DecisionPro, Inc. – Makers of Marketing Engineering Software – Home Page" (http:/ / www. decisionpro. biz). Decisionpro.biz. . Retrieved 13 November 2011. [8] "Operational Research in the British Army 1939–1945, October 1947, Report C67/3/4/48, UK National Archives file WO291/1301 Quoted on the dust-jacket of: Morse, Philip M, and Kimball, George E, Methods of Operations Research, 1st Edition Revised, pub MIT Press & J Wiley, 5th printing, 1954. [9] UK National Archives Catalogue for WO291 (http:/ / www. nationalarchives. gov. uk/ catalogue/ displaycataloguedetails. asp?CATID=109& CATLN=2& Highlight=& FullDetails=True) lists a War Office organisation called Army Operational Research Group (AORG) that existed from 1946 to 1962. "In January 1962 the name was changed to Army Operational Research Establishment (AORE). Following the creation of a unified Ministry of Defence, a tri-service operational research organisation was established: the Defence Operational Research Establishment (DOAE) which was formed in 1965, and it absorbed the Army Operational Research Establishment based at West Byfleet." [10] http:/ / brochure. unisa. ac. za/ myunisa/ data/ subjects/ Quantitative%20Management. pdf [11] M.S. Sodhi, "What about the 'O' in O.R.?" OR/MS Today, December, 2007, p. 12, http:/ / www. lionhrtpub. com/ orms/ orms-12-07/ frqed. html [12] P. W. Bridgman, The Logic of Modern Physics, The MacMillan Company, New York, 1927 [13] "operations research (industrial engineering) :: History – Britannica Online Encyclopedia" (http:/ / www. britannica. com/ EBchecked/ topic/ 682073/ operations-research/ 68171/ History#ref22348). Britannica.com. . Retrieved 13 November 2011. [14] Kirby, p. 117 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA117) [15] Kirby, pp. 91–94 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA94) [16] Kirby, p. 96,109 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA109) [17] Kirby, p. 96 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA96) [18] ""Numbers are Essential": Victory in the North Atlantic Reconsidered, March–May 1943" (http:/ / www. familyheritage. ca/ Articles/ victory1943. html). Familyheritage.ca. 24 May 1943. . Retrieved 13 November 2011. [19] Kirby, p. 101 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA101) [20] (Kirby, pp. 102,103 (http:/ / books. google. co. uk/ books?id=DWITTpkFPEAC& lpg=PA141& pg=PA103)) [21] James F. Dunnigan (1999). Dirty Little Secrets of the Twentieth Century. Harper Paperbacks. pp. 215–217. [22] "RAF History – Bomber Command 60th Anniversary" (http:/ / www. raf. mod. uk/ bombercommand/ thousands. html). Raf.mod.uk. . Retrieved 13 November 2011. [23] Milkman, Raymond H. (May 1968). Operations Research in World War II. United States Naval Institute Proceedings. Operations research 183 [24] Stafford Beer (1967) Management Science: The Business Use of Operations Research [25] What is Management Science? (http:/ / www. lums. lancs. ac. uk/ departments/ ManSci/ DeptProfile/ WhatisManSci/ ) Lancaster University, 2008. Retrieved 5 June 2008. [26] What is Management Science? (http:/ / bus. utk. edu/ soms/ information/ whatis_msci. html) The University of Tennessee, 2006. Retrieved 5 June 2008. [27] "IFORS" (http:/ / www. ifors. org/ ). IFORS. . Retrieved 13 November 2011. [28] Leszczynski, Mary (8 November 2011). "Informs" (http:/ / www. informs. org/ ). Informs. . Retrieved 13 November 2011. [29] "The OR Society" (http:/ / www. orsoc. org. uk). Orsoc.org.uk. . Retrieved 13 November 2011. [30] "Société française de Recherche Opérationnelle et d'Aide à la Décision" (http:/ / www. roadef. org/ content/ index. htm). ROADEF. . Retrieved 13 November 2011. [31] www.cors.ca. "CORS" (http:/ / www. cors. ca). Cors.ca. . Retrieved 13 November 2011. [32] "ASOR" (http:/ / www. asor. org. au). ASOR. 1 January 1972. . Retrieved 13 November 2011. [33] "ORSNZ" (http:/ / www. orsnz. org. nz/ ). ORSNZ. . Retrieved 13 November 2011. [34] "ORSP" (http:/ / www. orsp. org. ph/ ). ORSP. . Retrieved 13 November 2011. [35] "ORSI" (http:/ / www. orsi. in/ ). Orsi.in. . Retrieved 13 November 2011. [36] "ORSSA" (http:/ / www. orssa. org. za/ ). ORSSA. 23 September 2011. . Retrieved 13 November 2011. [37] "EURO" (http:/ / www. euro-online. org/ ). Euro-online.org. . Retrieved 13 November 2011. [38] "SISO" (http:/ / www. sisostds. org/ ). Sisostds.org. . Retrieved 13 November 2011. [39] "I/Itsec" (http:/ / www. iitsec. org/ ). I/Itsec. . Retrieved 13 November 2011. [40] "The Science of Better" (http:/ / www. scienceofbetter. org/ ). The Science of Better. . Retrieved 13 November 2011. [41] "Learn about OR" (http:/ / www. learnaboutor. co. uk/ ). Learn about OR. . Retrieved 13 November 2011. [42] "INFORMS Journals" (http:/ / www. informs. org/ index. php?c=31& kat=-+ INFORMS+ Journals). Informs.org. . Retrieved 13 November 2011. [43] "''Decision Analysis''" (http:/ / www. informs. org/ site/ DA/ ). Informs.org. . Retrieved 13 November 2011. [44] "INFORMS Transactions on Education" (http:/ / www. informs. org/ site/ ITE/ ). Informs.org. . Retrieved 13 November 2011. [45] http:/ / servsci. journal. informs. org/ [46] "International Journal of Operations Research and Information Systems (IJORIS) (1947–9328)(1947–9336): John Wang: Journals" (http:/ / www. igi-global. com/ Bookstore/ TitleDetails. aspx?TitleId=1141). IGI Global. . Retrieved 13 November 2011. [47] The Society for Modeling & Simulation International. "JDMS" (http:/ / www. scs. org/ pubs/ jdms/ jdms. html). Scs.org. . Retrieved 13 November 2011. [48] The OR Society (http:/ / www. orsoc. org. uk); [49] "Mathematical Methods of Operations Research website" (http:/ / www. springer. com/ mathematics/ journal/ 186). Springer.com. . Retrieved 13 November 2011. [50] "TOP" (http:/ / www. springer. com/ east/ home/ business/ operations+ research?SGWID=5-40521-70-173677307-detailsPage=journal|description). Springer.com. . Retrieved 13 November 2011. References • Kirby, M. W. (Operational Research Society (Great Britain)). Operational Research in War and Peace: The British Experience from the 1930s to 1970, Imperial College Press, 2003. ISBN 1-86094-366-7, ISBN 978-1-86094-366-9 Further reading • C. West Churchman, Russell L. Ackoff & E. L. Arnoff, Introduction to Operations Research, New York: J. Wiley and Sons, 1957 • Joseph G. Ecker & Michael Kupferschmid, Introduction to Operations Research, Krieger Publishing Co. • Frederick S. Hillier & Gerald J. Lieberman, Introduction to Operations Research, McGraw-Hill: Boston MA; 8th. (International) Edition, 2005 • Michael Pidd, Tools for Thinking: Modelling in Management Science, J. Wiley & Sons Ltd., Chichester; 2nd. Edition, 2003 • Hamdy A. Taha, Operations Research: An Introduction, Prentice Hall; 9th. Edition, 2011 • Wayne Winston, Operations Research: Applications and Algorithms, Duxbury Press; 4th. Edition, 2003 • Kenneth R. Baker, Dean H. Kropp (1985). Management Science: An Introduction to the Use of Decision Models • David Charles Heinze (1982). Management Science: Introductory Concepts and Applications Operations research 184 • Lee J. Krajewski, Howard E. Thompson (1981). "Management Science: Quantitative Methods in Context" • Thomas W. Knowles (1989). Management science: Building and Using Models • Kamlesh Mathur, Daniel Solow (1994). Management Science: The Art of Decision Making • Laurence J. Moore, Sang M. Lee, Bernard W. Taylor (1993). Management Science • William Thomas Morris (1968). Management Science: A Bayesian Introduction. • William E. Pinney, Donald B. McWilliams (1987). Management Science: An Introduction to Quantitative Analysis for Management • Shrader, Charles R. (2006). History of Operations Research in the United States Army, Volume 1:1942–1962 (http://www.history.army.mil/html/books/hist_op_research/index.html). Washington, D.C.: United States Army Center of Military History. CMH Pub 70-102-1. • Gerald E. Thompson (1982). Management Science: An Introduction to Modern Quantitative Analysis and Decision Making. New York : McGraw-Hill Publishing Co. • Saul I. Gass & Arjang A. Assad (2005). An Annotated Timeline of Operations Research: An Informal History. New York : Kluwer Academic Publishers. External links • INFORMS OR/MS Resource Collection (http://www.informs.org/Resources/): a comprehensive set of OR links. • International Federation of Operational Research Societies (http://www.ifors.org/) • Occupational Outlook Handbook, U.S. Department of Labor Bureau of Labor Statistics (http://stats.bls.gov/ oco/ocos044.htm/) • "Operation Everything: It Stocks Your Grocery Store, Schedules Your Favorite Team's Games, and Helps Plan Your Vacation. The Most Influential Academic Discipline You've Never Heard Of." Boston Globe, 27 June 2004 (http://www.boston.com/news/globe/reprints/062704_postrel/) • "Optimal Results: IT-powered advances in operations research can enhance business processes and boost the corporate bottom line." Computerworld, 20 November 2000 (http://www.computerworld.com/s/article/ 54157/Optimal_Results?taxonomyId=063/) Decision theory 185 Decision theory Decision theory in economics, psychology, philosophy, mathematics, and statistics is concerned with identifying the values, uncertainties and other issues relevant in a given decision, its rationality, and the resulting optimal decision. It is closely related to the field of game theory as to interactions of agents with at least partially conflicting interests whose decisions affect each other. Normative and descriptive decision theory Most of decision theory is normative or prescriptive, i.e., it is concerned with identifying the best decision to take (in practice, there are situations in which "best" is not necessarily the maximal, optimum may also include values in addition to maximum, but within a specific or approximative range), assuming an ideal decision maker who is fully informed, able to compute with perfect accuracy, and fully rational. The practical application of this prescriptive approach (how people ought to make decisions) is called decision analysis, and aimed at finding tools, methodologies and software to help people make better decisions. The most systematic and comprehensive software tools developed in this way are called decision support systems. Since people usually do not behave in ways consistent with axiomatic rules, often their own, leading to violations of optimality, there is a related area of study, called a positive or descriptive discipline, attempting to describe what people will actually do. Since the normative, optimal decision often creates hypotheses for testing against actual behaviour, the two fields are closely linked. Furthermore it is possible to relax the assumptions of perfect information, rationality and so forth in various ways, and produce a series of different prescriptions or predictions about behaviour, allowing for further tests of the kind of decision-making that occurs in practice. In recent decades, there has been increasing interest in what is sometimes called 'behavioral decision theory' and this has contributed to a re-evaluation of what rational decision-making requires (see for instance Anand, 1993). What kinds of decisions need a theory? Choice under uncertainty This area represents the heart of decision theory. The procedure now referred to as expected value was known from the 17th century. Blaise Pascal invoked it in his famous wager (see below), which is contained in his Pensées, published in 1670. The idea of expected value is that, when faced with a number of actions, each of which could give rise to more than one possible outcome with different probabilities, the rational procedure is to identify all possible outcomes, determine their values (positive or negative) and the probabilities that will result from each course of action, and multiply the two to give an expected value. The action to be chosen should be the one that gives rise to the highest total expected value. In 1738, Daniel Bernoulli published an influential paper entitled Exposition of a New Theory on the Measurement of Risk, in which he uses the St. Petersburg paradox to show that expected value theory must be normatively wrong. He also gives an example in which a Dutch merchant is trying to decide whether to insure a cargo being sent from Amsterdam to St Petersburg in winter, when it is known that there is a 5% chance that the ship and cargo will be lost. In his solution, he defines a utility function and computes expected utility rather than expected financial value (see[1] for a review). In the 20th century, interest was reignited by Abraham Wald's 1939 paper[2] pointing out that the two central procedures of sampling–distribution based statistical-theory, namely hypothesis testing and parameter estimation, are special cases of the general decision problem. Wald's paper renewed and synthesized many concepts of statistical theory, including loss functions, risk functions, admissible decision rules, antecedent distributions, Bayesian procedures, and minimax procedures. The phrase "decision theory" itself was used in 1950 by E. L. Lehmann.[3] Decision theory 186 The revival of subjective probability theory, from the work of Frank Ramsey, Bruno de Finetti, Leonard Savage and others, extended the scope of expected utility theory to situations where subjective probabilities can be used. At this time, von Neumann's theory of expected utility proved that expected utility maximization followed from basic postulates about rational behavior. The work of Maurice Allais and Daniel Ellsberg showed that human behavior has systematic and sometimes important departures from expected-utility maximization. The prospect theory of Daniel Kahneman and Amos Tversky renewed the empirical study of economic behavior with less emphasis on rationality presuppositions. Kahneman and Tversky found three regularities — in actual human decision-making, "losses loom larger than gains"; persons focus more on changes in their utility–states than they focus on absolute utilities; and the estimation of subjective probabilities is severely biased by anchoring. Castagnoli and LiCalzi (1996), Bordley and LiCalzi (2000) recently showed that maximizing expected utility is mathematically equivalent to maximizing the probability that the uncertain consequences of a decision Daniel Kahneman are preferable to an uncertain benchmark (e.g., the probability that a mutual fund strategy outperforms the S&P 500 or that a firm outperforms the uncertain future performance of a major competitor.). This reinterpretation relates to psychological work suggesting that individuals have fuzzy aspiration levels (Lopes & Oden), which may vary from choice context to choice context. Hence it shifts the focus from utility to the individual's uncertain reference point. Pascal's Wager is a classic example of a choice under uncertainty. It is possible that the reward for belief is infinite (i.e. if God exists and is the sort of God worshiped by evangelical Christians). However, it is also possible that the reward for non-belief is infinite (i.e. if a capricious God exists that rewards us for not believing in God). Therefore, either believing in God or not believing in God, when you include these results, lead to infinite rewards and so we have no decision-theoretic reason to prefer one to the other. (There are several criticisms of the argument.) Intertemporal choice This area is concerned with the kind of choice where different actions lead to outcomes that are realised at different points in time. If someone received a windfall of several thousand dollars, they could spend it on an expensive holiday, giving them immediate pleasure, or they could invest it in a pension scheme, giving them an income at some time in the future. What is the optimal thing to do? The answer depends partly on factors such as the expected rates of interest and inflation, the person's life expectancy, and their confidence in the pensions industry. However even with all those factors taken into account, human behavior again deviates greatly from the predictions of prescriptive decision theory, leading to alternative models in which, for example, objective interest rates are replaced by subjective discount rates. Competing decision makers Some decisions are difficult because of the need to take into account how other people in the situation will respond to the decision that is taken. The analysis of such social decisions is more often treated under the label of game theory, rather than decision theory, though it involves the same mathematical methods. From the standpoint of game theory most of the problems treated in decision theory are one-player games (or the one player is viewed as playing against an impersonal background situation). In the emerging socio-cognitive engineering, the research is especially focused on the different types of distributed decision-making in human organizations, in normal and abnormal/emergency/crisis situations. Signal detection theory is based on decision theory. Decision theory 187 Complex decisions Other areas of decision theory are concerned with decisions that are difficult simply because of their complexity, or the complexity of the organization that has to make them. In such cases the issue is not the deviation between real and optimal behaviour, but the difficulty of determining the optimal behaviour in the first place. The Club of Rome, for example, developed a model of economic growth and resource usage that helps politicians make real-life decisions in complex situations. Alternatives to decision theory A highly controversial issue is whether one can replace the use of probability in decision theory by other alternatives. Probability theory The Advocates of probability theory point to: • the work of Richard Threlkeld Cox for justification of the probability axioms, • the Dutch book paradoxes of Bruno de Finetti as illustrative of the theoretical difficulties that can arise from departures from the probability axioms, and • the complete class theorems, which show that all admissible decision rules are equivalent to the Bayesian decision rule for some utility function and some prior distribution (or for the limit of a sequence of prior distributions). Thus, for every decision rule, either the rule may be reformulated as a Bayesian procedure, or there is a (perhaps limiting) Bayesian rule that is sometimes better and never worse. Alternatives to probability theory The proponents of fuzzy logic, possibility theory, Dempster–Shafer theory, and info-gap decision theory maintain that probability is only one of many alternatives and point to many examples where non-standard alternatives have been implemented with apparent success; notably, probabilistic decision theory is sensitive to assumptions about the probabilities of various events, while non-probabilistic rules such as minimax are robust, in that they do not make such assumptions. General criticism A general criticism of decision theory based on a fixed universe of possibilities is that it considers the "known unknowns", not the "unknown unknowns": it focuses on expected variations, not on unforeseen events, which some argue (as in black swan theory) have outsized impact and must be considered – significant events may be "outside model". This line of argument, called the ludic fallacy, is that there are inevitable imperfections in modeling the real world by particular models, and that unquestioning reliance on models blinds one to their limits. Decision theory 188 References [1] Schoemaker, P. J. H. (1982). "The Expected Utility Model: Its Variants, Purposes, Evidence and Limitations". Journal of Economic Literature 20: 529–563. [2] Wald, Abraham (1939). "Contributions to the Theory of Statistical Estimation and Testing Hypotheses". Annals of Mathematical Statistics 10 (4): 299–326. doi:10.1214/aoms/1177732144. MR932. [3] Lehmann, E. L. (1950). "Some Principles of the Theory of Testing Hypotheses". Annals of Mathematical Statistics 21 (1): 1–26. doi:10.1214/aoms/1177729884. JSTOR 2236552. Further reading • Akerlof, George A., Yellen, Janet L. (May 1987). Rational Models of Irrational Behavior. 77. pp. 137–142. • Anand, Paul (1993). Foundations of Rational Choice Under Risk. Oxford: Oxford University Press. ISBN 0-19-823303-5. (an overview of the philosophical foundations of key mathematical axioms in subjective expected utility theory – mainly normative) • Arthur, W. Brian (May 1991). "Designing Economic Agents that Act like Human Agents: A Behavioral Approach to Bounded Rationality". The American Economic Review 81 (2): 353–9. • Berger, James O. (1985). Statistical decision theory and Bayesian Analysis (2nd ed.). New York: Springer-Verlag. ISBN 0-387-96098-8. MR0804611. • Bernardo, José M.; Smith, Adrian F. M. (1994). Bayesian Theory. Wiley. ISBN 0-471-92416-4. MR1274699. • Clemen, Robert (1996). Making Hard Decisions: An Introduction to Decision Analysis (2nd ed.). Belmont CA: Duxbury Press. ISBN 0-534-26035-7. (covers normative decision theory) • De Groot, Morris, Optimal Statistical Decisions. Wiley Classics Library. 2004. (Originally published 1970.) ISBN 0-471-68029-X. • Goodwin, Paul and Wright, George (2004). Decision Analysis for Management Judgment (3rd ed.). Chichester: Wiley. ISBN 0-470-86108-8. (covers both normative and descriptive theory) • Hansson, Sven Ove. "Decision Theory: A Brief Introduction" (http://www.infra.kth.se/~soh/decisiontheory. pdf) (PDF). • Khemani, Karan, Ignorance is Bliss: A study on how and why humans depend on recognition heuristics in social relationships, the equity markets and the brand market-place, thereby making successful decisions, 2005. • Miller L (1985). "Cognitive risk-taking after frontal or temporal lobectomy—I. The synthesis of fragmented visual information". Neuropsychologia 23 (3): 359–69. doi:10.1016/0028-3932(85)90022-3. PMID 4022303. • Miller L, Milner B (1985). "Cognitive risk-taking after frontal or temporal lobectomy—II. The synthesis of phonemic and semantic information". Neuropsychologia 23 (3): 371–9. doi:10.1016/0028-3932(85)90023-5. PMID 4022304. • North, D.W. (1968). "A tutorial introduction to decision theory". IEEE Transactions on Systems Science and Cybernetics 4 (3): 200–210. doi:10.1109/TSSC.1968.300114. Reprinted in Shafer & Pearl. (also about normative decision theory) • Peterson, Martin (2009). An Introduction to Decision Theory. Cambridge University Press. ISBN 978-0-521-71654-3. • Raiffa, Howard (1997). Decision Analysis: Introductory Readings on Choices Under Uncertainty. McGraw Hill. ISBN 0-07-052579-X. • Robert, Christian (2007). The Bayesian Choice (2nd ed.). New York: Springer. doi:10.1007/0-387-71599-1. ISBN 0-387-95231-4. MR1835885. • Shafer, Glenn and Pearl, Judea, ed. (1990). Readings in uncertain reasoning. San Mateo, CA: Morgan Kaufmann. • Smith, J.Q. (1988). Decision Analysis: A Bayesian Approach. Chapman and Hall. ISBN 0-412-27520-1. • Charles Sanders Peirce and Joseph Jastrow (1885). "On Small Differences in Sensation" (http://psychclassics. yorku.ca/Peirce/small-diffs.htm). Memoirs of the National Academy of Sciences 3: 73–83. http://psychclassics. yorku.ca/Peirce/small-diffs.htm Decision theory 189 • Ramsey, Frank Plumpton; “Truth and Probability” ( PDF (http://cepa.newschool.edu/het//texts/ramsey/ ramsess.pdf)), Chapter VII in The Foundations of Mathematics and other Logical Essays (1931). • de Finetti, Bruno (September 1989). "Probabilism: A Critical Essay on the Theory of Probability and on the Value of Science". Erkenntnis 31. (translation of 1931 article) • de Finetti, Bruno (1937). "La Prévision: ses lois logiques, ses sources subjectives". Annales de l'Institut Henri Poincaré. de Finetti, Bruno. "Foresight: its Logical Laws, Its Subjective Sources," (translation of the 1937 article (http:/ / www. numdam. org/ item?id=AIHP_1937__7_1_1_0) in French) in H. E. Kyburg and H. E. Smokler (eds), Studies in Subjective Probability, New York: Wiley, 1964. • de Finetti, Bruno. Theory of Probability, (translation by AFM Smith of 1970 book) 2 volumes, New York: Wiley, 1974-5. • Donald Davidson, Patrick Suppes and Sidney Siegel (1957). Decision-Making: An Experimental Approach. Stanford University Press. • Pfanzagl, J (1967). "Subjective Probability Derived from the Morgenstern-von Neumann Utility Theory". In Martin Shubik. Essays in Mathematical Economics In Honor of Oskar Morgenstern. Princeton University Press. pp. 237–251. • Pfanzagl, J. in cooperation with V. Baumann and H. Huber (1968). "Events, Utility and Subjective Probability". Theory of Measurement. Wiley. pp. 195–220. • Morgenstern, Oskar (1976). "Some Reflections on Utility". In Andrew Schotter. Selected Economic Writings of Oskar Morgenstern. New York University Press. pp. 65–70. ISBN 0-8147-7771-6. Neural network The term neural network was traditionally used to refer to a network or circuit of biological neurons.[1] The modern usage of the term often refers to artificial neural networks, which are composed of artificial neurons or nodes. Thus the term has two distinct usages: 1. Biological neural networks are made up of real biological neurons that are connected or functionally related in a nervous system. In the field of neuroscience, they are often identified as groups of neurons that perform a specific physiological function in laboratory analysis. 2. Artificial neural networks are composed of interconnecting artificial neurons (programming constructs that mimic the properties of biological neurons) for solving artificial intelligence problems without creating a model of a real system. Neural network algorithms abstract away the biological complexity by focusing on the most important information. The goal of artificial neural networks is good, or human-like, predictive ability. Simplified view of a feedforward artificial neural network Neural network 190 Overview A biological neural network is composed of a group or groups of chemically connected or functionally associated neurons. A single neuron may be connected to many other neurons and the total number of neurons and connections in a network may be extensive. Connections, called synapses, are usually formed from axons to dendrites, though dendrodendritic microcircuits[2] and other connections are possible. Apart from the electrical signaling, there are other forms of signaling that arise from neurotransmitter diffusion. Artificial intelligence and cognitive modeling try to simulate some properties of biological neural networks. While similar in their techniques, the former has the aim of solving particular tasks, while the latter aims to build mathematical models of biological neural systems. In the artificial intelligence field, artificial neural networks have been applied successfully to speech recognition, image analysis and adaptive control, in order to construct software agents (in computer and video games) or autonomous robots. Most of the currently employed artificial neural networks for artificial intelligence are based on statistical estimations, classification optimization and control theory. The cognitive modelling field involves the physical or mathematical modeling of the behavior of neural systems; ranging from the individual neural level (e.g. modeling the spike response curves of neurons to a stimulus), through the neural cluster level (e.g. modelling the release and effects of dopamine in the basal ganglia) to the complete organism (e.g. behavioral modelling of the organism's response to stimuli). Artificial intelligence, cognitive modelling, and neural networks are information processing paradigms inspired by the way biological neural systems process data. History of the neural network analogy In the brain, spontaneous order appears to arise out of decentralized networks of simple units (neurons). Neural network theory has served both to better identify how the neurons in the brain function and to provide the basis for efforts to create artificial intelligence. The preliminary theoretical base for contemporary neural networks was independently proposed by Alexander Bain[3] (1873) and William James[4] (1890). In their work, both thoughts and body activity resulted from interactions among neurons within the brain. For Bain,[3] every activity led to the firing of a certain set of neurons. When activities were repeated, the connections between those neurons strengthened. According to his theory, this repetition was what led to the formation of memory. The general scientific community at the time was skeptical of Bain’s[3] theory because it required what appeared to be an inordinate number of neural connections within the brain. It is now apparent that the brain is exceedingly complex and that the same brain “wiring” can handle multiple problems and inputs. James’s[4] theory was similar to Bain’s,[3] however, he suggested that memories and actions resulted from electrical currents flowing among the neurons in the brain. His model, by focusing on the flow of electrical currents, did not require individual neural connections for each memory or action. C. S. Sherrington[5] (1898) conducted experiments to test James’s theory. He ran electrical currents down the spinal cords of rats. However, instead of demonstrating an increase in electrical current as projected by James, Sherrington found that the electrical current strength decreased as the testing continued over time. Importantly, this work led to the discovery of the concept of habituation. McCulloch and Pitts[6] (1943) created a computational model for neural networks based on mathematics and algorithms. They called this model threshold logic. The model paved the way for neural network research to split into two distinct approaches. One approach focused on biological processes in the brain and the other focused on the application of neural networks to artificial intelligence. In the late 1940s psychologist Donald Hebb[7] created a hypothesis of learning based on the mechanism of neural plasticity that is now known as Hebbian learning. Hebbian learning is considered to be a 'typical' unsupervised Neural network 191 learning rule and its later variants were early models for long term potentiation. These ideas started being applied to computational models in 1948 with Turing's B-type machines. Farley and Clark[8] (1954) first used computational machines, then called calculators, to simulate a Hebbian network at MIT. Other neural network computational machines were created by Rochester, Holland, Habit, and Duda[9] (1956). Rosenblatt[10] (1958) created the perceptron, an algorithm for pattern recognition based on a two-layer learning computer network using simple addition and subtraction. With mathematical notation, Rosenblatt also described circuitry not in the basic perceptron, such as the exclusive-or circuit, a circuit whose mathematical computation could not be processed until after the backpropagation algorithm was created by Werbos[11] (1975). The perceptron is essentially a linear classifier for classifying data specified by parameters and an output function . Its parameters are adapted with an ad-hoc rule similar to stochastic steepest gradient descent. Because the inner product is a linear operator in the input space, the perceptron can only perfectly classify a set of data for which different classes are linearly separable in the input space, while it often fails completely for non-separable data. While the development of the algorithm initially generated some enthusiasm, partly because of its apparent relation to biological mechanisms, the later discovery of this inadequacy caused such models to be abandoned until the introduction of non-linear models into the field. Neural network research stagnated after the publication of machine learning research by Minsky and Papert[12] (1969). They discovered two key issues with the computational machines that processed neural networks. The first issue was that single-layer neural networks were incapable of processing the exclusive-or circuit. The second significant issue was that computers were not sophisticated enough to effectively handle the long run time required by large neural networks. Neural network research slowed until computers achieved greater processing power. Also key in later advances was the backpropogation algorithm which effectively solved the exclusive-or problem (Werbos 1975).[11] The cognitron (1975) designed by Kunihiko Fukushima[13] was an early multilayered neural network with a training algorithm. The actual structure of the network and the methods used to set the interconnection weights change from one neural strategy to another, each with its advantages and disadvantages. Networks can propagate information in one direction only, or they can bounce back and forth until self-activation at a node occurs and the network settles on a final state. The ability for bi-directional flow of inputs between neurons/nodes was produced with adaptive resonance theory, the neocognitron and the Hopfield net, and specialization of these node layers for specific purposes was introduced through the first hybrid network. The parallel distributed processing of the mid-1980s became popular under the name connectionism. The text by Rumelhart and McClelland[14] (1986) provided a full exposition on the use of connectionism in computers to simulate neural processes. The rediscovery of the backpropagation algorithm was probably the main reason behind the repopularisation of neural networks after the publication of "Learning Internal Representations by Error Propagation" in 1986 (Though backpropagation itself dates from 1969). The original network utilized multiple layers of weight-sum units of the type , where was a sigmoid function or logistic function such as used in logistic regression. Training was done by a form of stochastic gradient descent. The employment of the chain rule of differentiation in deriving the appropriate parameter updates results in an algorithm that seems to 'backpropagate errors', hence the nomenclature. However, it is essentially a form of gradient descent. Determining the optimal parameters in a model of this type is not trivial, and local numerical optimization methods such as gradient descent can be sensitive to initialization because of the presence of local minima of the training criterion. In recent times, networks with the same architecture as the backpropagation network are referred to as multilayer perceptrons. This name does not impose any limitations on the type of algorithm used for learning. The backpropagation network generated much enthusiasm at the time and there was much controversy about whether such learning could be implemented in the brain or not, partly because a mechanism for reverse signaling was not Neural network 192 obvious at the time, but most importantly because there was no plausible source for the 'teaching' or 'target' signal. However, since 2006, several unsupervised learning procedures have been proposed for neural networks with one or more layers, using so-called deep learning algorithms. These algorithms can be used to learn intermediate representations, with or without a target signal, that capture the salient features of the distribution of sensory signals arriving at each layer of the neural network. The brain, neural networks and computers Neural networks, as used in artificial intelligence, have traditionally been viewed as simplified models of neural processing in the brain, even though the relation between this model and brain biological architecture is debated, as it is not clear to what degree artificial neural networks mirror brain function.[16] A subject of current research in computational neuroscience is the question surrounding the degree of complexity and the properties that individual neural elements should have to reproduce something resembling animal cognition. Historically, computers evolved from the von Neumann model, which is based on sequential processing and execution of explicit instructions. Computer simulation of the branching On the other hand, the origins of neural networks are based on efforts architecture of the dendrites of pyramidal to model information processing in biological systems, which may rely neurons. [15] largely on parallel processing as well as implicit instructions based on recognition of patterns of 'sensory' input from external sources. In other words, at its very heart a neural network is a complex statistical processor (as opposed to being tasked to sequentially process and execute). Neural coding is concerned with how sensory and other information is represented in the brain by neurons. The main goal of studying neural coding is to characterize the relationship between the stimulus and the individual or ensemble neuronal responses and the relationship among electrical activity of the neurons in the ensemble.[17] It is thought that neurons can encode both digital and analog information.[18] Neural networks and artificial intelligence A neural network (NN), in the case of artificial neurons called artificial neural network (ANN) or simulated neural network (SNN), is an interconnected group of natural or artificial neurons that uses a mathematical or computational model for information processing based on a connectionistic approach to computation. In most cases an ANN is an adaptive system that changes its structure based on external or internal information that flows through the network. In more practical terms neural networks are non-linear statistical data modeling or decision making tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data. However, the paradigm of neural networks - i.e., implicit, not explicit , learning is stressed - seems more to correspond to some kind of natural intelligence than to the traditional symbol-based Artificial Intelligence, which would stress, instead, rule-based learning. Neural network 193 Background An artificial neural network involves a network of simple processing elements (artificial neurons) which can exhibit complex global behavior, determined by the connections between the processing elements and element parameters. Artificial neurons were first proposed in 1943 by Warren McCulloch, a neurophysiologist, and Walter Pitts, a logician, who first collaborated at the University of Chicago.[19] One classical type of artificial neural network is the recurrent Hopfield net. In a neural network model simple nodes (which can be called by a number of names, including "neurons", "neurodes", "Processing Elements" (PE) and "units"), are connected together to form a network of nodes — hence the term "neural network". While a neural network does not have to be adaptive per se, its practical use comes with algorithms designed to alter the strength (weights) of the connections in the network to produce a desired signal flow. In modern software implementations of artificial neural networks the approach inspired by biology has more or less been abandoned for a more practical approach based on statistics and signal processing. In some of these systems, neural networks, or parts of neural networks (such as artificial neurons), are used as components in larger systems that combine both adaptive and non-adaptive elements. The concept of a neural network appears to have first been proposed by Alan Turing in his 1948 paper "Intelligent Machinery". Applications of natural and of artificial neural networks The utility of artificial neural network models lies in the fact that they can be used to infer a function from observations and also to use it. Unsupervised neural networks can also be used to learn representations of the input that capture the salient characteristics of the input distribution, e.g., see the Boltzmann machine (1983), and more recently, deep learning algorithms, which can implicitly learn the distribution function of the observed data. Learning in neural networks is particularly useful in applications where the complexity of the data or task makes the design of such functions by hand impractical. The tasks to which artificial neural networks are applied tend to fall within the following broad categories: • Function approximation, or regression analysis, including time series prediction and modeling. • Classification, including pattern and sequence recognition, novelty detection and sequential decision making. • Data processing, including filtering, clustering, blind signal separation and compression. Application areas of ANNs include system identification and control (vehicle control, process control), game-playing and decision making (backgammon, chess, racing), pattern recognition (radar systems, face identification, object recognition), sequence recognition (gesture, speech, handwritten text recognition), medical diagnosis, financial applications, data mining (or knowledge discovery in databases, "KDD"), visualization and e-mail spam filtering. Neural networks and neuroscience Theoretical and computational neuroscience is the field concerned with the theoretical analysis and computational modeling of biological neural systems. Since neural systems are intimately related to cognitive processes and behaviour, the field is closely related to cognitive and behavioural modeling. The aim of the field is to create models of biological neural systems in order to understand how biological systems work. To gain this understanding, neuroscientists strive to make a link between observed biological processes (data), biologically plausible mechanisms for neural processing and learning (biological neural network models) and theory (statistical learning theory and information theory). Neural network 194 Types of models Many models are used; defined at a different levels of abstraction, and modeling different aspects of neural systems. They range from models of the short-term behaviour of individual neurons, through models of the dynamics of neural circuitry arising from interactions between individual neurons, to models of behaviour arising from abstract neural modules that represent complete subsystems. These include models of the long-term and short-term plasticity of neural systems and its relation to learning and memory, from the individual neuron to the system level. Current research While initially research had been concerned mostly with the electrical characteristics of neurons, a particularly important part of the investigation in recent years has been the exploration of the role of neuromodulators such as dopamine, acetylcholine, and serotonin on behaviour and learning. Biophysical models, such as BCM theory, have been important in understanding mechanisms for synaptic plasticity, and have had applications in both computer science and neuroscience. Research is ongoing in understanding the computational algorithms used in the brain, with some recent biological evidence for radial basis networks and neural backpropagation as mechanisms for processing data. Computational devices have been created in CMOS for both biophysical simulation and neuromorphic computing. More recent efforts show promise for creating nanodevices[20] for very large scale principal components analyses and convolution. If successful, these efforts could usher in a new era of neural computing[21] that is a step beyond digital computing, because it depends on learning rather than programming and because it is fundamentally analog rather than digital even though the first instantiations may in fact be with CMOS digital devices. Architecture The basic architecture consists of three types of neuron layers: input, hidden, and output. In feed-forward networks, the signal flow is from input to output units, strictly in a feed-forward direction. The data processing can extend over multiple layers of units, but no feedback connections are present. Recurrent networks contain feedback connections. Contrary to feed-forward networks, the dynamical properties of the network are important. In some cases, the activation values of the units undergo a relaxation process such that the network will evolve to a stable state in which these activations do not change anymore. In other applications, the changes of the activation values of the output neurons are significant, such that the dynamical behavior constitutes the output of the network. Other neural network architectures include adaptive resonance theory maps and competitive networks. Criticism A common criticism of neural networks, particularly in robotics, is that they require a large diversity of training for real-world operation. This is not surprising, since any learning machine needs sufficient representative examples in order to capture the underlying structure that allows it to generalize to new cases. Dean Pomerleau, in his research presented in the paper "Knowledge-based Training of Artificial Neural Networks for Autonomous Robot Driving," uses a neural network to train a robotic vehicle to drive on multiple types of roads (single lane, multi-lane, dirt, etc.). A large amount of his research is devoted to (1) extrapolating multiple training scenarios from a single training experience, and (2) preserving past training diversity so that the system does not become overtrained (if, for example, it is presented with a series of right turns – it should not learn to always turn right). These issues are common in neural networks that must decide from amongst a wide variety of responses, but can be dealt with in several ways, for example by randomly shuffling the training examples, by using a numerical optimization algorithm that does not take too large steps when changing the network connections following an example, or by grouping examples in so-called mini-batches. Neural network 195 A. K. Dewdney, a former Scientific American columnist, wrote in 1997, "Although neural nets do solve a few toy problems, their powers of computation are so limited that I am surprised anyone takes them seriously as a general problem-solving tool." (Dewdney, p. 82) Arguments for Dewdney's position are that to implement large and effective software neural networks, much processing and storage resources need to be committed. While the brain has hardware tailored to the task of processing signals through a graph of neurons, simulating even a most simplified form on Von Neumann technology may compel a NN designer to fill many millions of database rows for its connections - which can consume vast amounts of computer memory and hard disk space. Furthermore, the designer of NN systems will often need to simulate the transmission of signals through many of these connections and their associated neurons - which must often be matched with incredible amounts of CPU processing power and time. While neural networks often yield effective programs, they too often do so at the cost of efficiency (they tend to consume considerable amounts of time and money). Arguments against Dewdney's position are that neural nets have been successfully used to solve many complex and diverse tasks, ranging from autonomously flying aircraft [22] to detecting credit card fraud . Technology writer Roger Bridgman commented on Dewdney's statements about neural nets: Neural networks, for instance, are in the dock not only because they have been hyped to high heaven, (what hasn't?) but also because you could create a successful net without understanding how it worked: the bunch of numbers that captures its behaviour would in all probability be "an opaque, unreadable table...valueless as a scientific resource". In spite of his emphatic declaration that science is not technology, Dewdney seems here to pillory neural nets as bad science when most of those devising them are just trying to be good engineers. An unreadable table that a useful machine could read would still be well worth having.[23] In response to this kind of criticism, one should note that although it is true that analyzing what has been learned by an artificial neural network is difficult, it is much easier to do so than to analyze what has been learned by a biological neural network. Furthermore, researchers involved in exploring learning algorithms for neural networks are gradually uncovering generic principles which allow a learning machine to be successful. For example, Bengio and LeCun (2007) wrote an article regarding local vs non-local learning, as well as shallow vs deep architecture [24]. Some other criticisms came from believers of hybrid models (combining neural networks and symbolic approaches). They advocate the intermix of these two approaches and believe that hybrid models can better capture the mechanisms of the human mind (Sun and Bookman, 1990). Recent improvements Between 2009 and 2012, the recurrent neural networks and deep feedforward neural networks developed in the research group of Jürgen Schmidhuber at the Swiss AI Lab IDSIA have won eight international competitions in pattern recognition and machine learning.[25] For example, multi-dimensional long short term memory (LSTM)[26][27] won three competitions in connected handwriting recognition at the 2009 International Conference on Document Analysis and Recognition (ICDAR), without any prior knowledge about the three different languages to be learned. Variants of the back-propagation algorithm as well as unsupervised methods by Geoff Hinton and colleagues at the University of Toronto[28][29] can be used to train deep, highly nonlinear neural architectures similar to the 1980 Neocognitron by Kunihiko Fukushima,[30] and the "standard architecture of vision",[31] inspired by the simple and complex cells identified by David H. Hubel and Torsten Wiesel in the primary visual cortex. Deep learning feedforward networks alternate convolutional layers and max-pooling layers, topped by several pure classification layers. Fast GPU-based implementations of this approach have won several pattern recognition contests, including the IJCNN 2011 Traffic Sign Recognition Competition[32] and the ISBI 2012 Segmentation of Neural network 196 Neuronal Structures in Electron Microscopy Stacks challenge.[33] Such neural networks also were the first artificial pattern recognizers to achieve human-competitive or even superhuman performance[34] on benchmarks such as traffic sign recognition (IJCNN 2012), or the MNIST handwritten digits problem of Yann LeCun and colleagues at NYU. References [1] J. J. HOPFIELD Neural networks and physical systems with emergent collective computational abilities. Proc. NatL Acad. Sci. USA Vol. 79, pp. 2554-2558, April 1982 Biophysics (http:/ / www. pnas. org/ content/ 79/ 8/ 2554. full. pdf) [2] Arbib, p.666 [3] Bain (1873). Mind and Body: The Theories of Their Relation. New York: D. Appleton and Company. [4] James (1890). The Principles of Psychology. New York: H. Holt and Company. [5] Sherrington, C.S.. "Experiments in Examination of the Peripheral Distribution of the Fibers of the Posterior Roots of Some Spinal Nerves". Proceedings of the Royal Society of London 190: 45–186. [6] McCulloch, Warren; Walter Pitts (1943). "A Logical Calculus of Ideas Immanent in Nervous Activity". Bulletin of Mathematical Biophysics 5 (4): 115–133. doi:10.1007/BF02478259. [7] Hebb, Donald (1949). The Organization of Behavior. New York: Wiley. [8] Farley, B; W.A. Clark (1954). "Simulation of Self-Organizing Systems by Digital Computer". IRE Transactions on Information Theory 4 (4): 76–84. doi:10.1109/TIT.1954.1057468. [9] Rochester, N.; J.H. Holland, L.H. Habit, and W.L. Duda (1956). "Tests on a cell assembly theory of the action of the brain, using a large digital computer". IRE Transactions on Information Theory 2 (3): 80–93. doi:10.1109/TIT.1956.1056810. [10] Rosenblatt, F. (1958). "The Perceptron: A Probalistic Model For Information Storage And Organization In The Brain". Psychological Review 65 (6): 386–408. doi:10.1037/h0042519. PMID 13602029. [11] Werbos, P.J. (1975). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. [12] Minsky, M.; S. Papert (1969). An Introduction to Computational Geometry. MIT Press. ISBN 0-262-63022-2. [13] Fukushima, Kunihiko (1975). "Cognitron: A self-organizing multilayered neural network". Biological Cybernetics 20 (3–4): 121–136. doi:10.1007/BF00342633. PMID 1203338. [14] Rumelhart, D.E; James McClelland (1986). Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Cambridge: MIT Press. [15] "PLoS Computational Biology Issue Image". PLoS Computational Biology 6 (8): ev06.ei08. 2010. doi:10.1371/image.pcbi.v06.i08. [16] Russell, Ingrid. "Neural Networks Module" (http:/ / uhaweb. hartford. edu/ compsci/ neural-networks-definition. html). . Retrieved 2012. [17] Brown EN, Kass RE, Mitra PP. (2004). "Multiple neural spike train data analysis: state-of-the-art and future challenges". Nature Neuroscience 7 (5): 456–61. doi:10.1038/nn1228. PMID 15114358. [18] Spike arrival times: A highly efficient coding scheme for neural networks (http:/ / pop. cerco. ups-tlse. fr/ fr_vers/ documents/ thorpe_sj_90_91. pdf), SJ Thorpe - Parallel processing in neural systems, 1990 [19] McCulloch, Warren; Pitts, Walter, "A Logical Calculus of Ideas Immanent in Nervous Activity", 1943, Bulletin of Mathematical Biophysics 5:115-133. [20] Yang, J. J.; Pickett, M. D.; Li, X. M.; Ohlberg, D. A. A.; Stewart, D. R.; Williams, R. S. Nat. Nanotechnol. 2008, 3, 429–433. [21] Strukov, D. B.; Snider, G. S.; Stewart, D. R.; Williams, R. S. Nature 2008, 453, 80–83. [22] http:/ / www. nasa. gov/ centers/ dryden/ news/ NewsReleases/ 2003/ 03-49. html [23] Roger Bridgman's defence of neural networks (http:/ / members. fortunecity. com/ templarseries/ popper. html) [24] http:/ / www. iro. umontreal. ca/ ~lisa/ publications2/ index. php/ publications/ show/ 4 [25] http:/ / www. kurzweilai. net/ how-bio-inspired-deep-learning-keeps-winning-competitions 2012 Kurzweil AI Interview with Jürgen Schmidhuber on the eight competitions won by his Deep Learning team 2009-2012 [26] Graves, Alex; and Schmidhuber, Jürgen; Offline Handwriting Recognition with Multidimensional Recurrent Neural Networks, in Bengio, Yoshua; Schuurmans, Dale; Lafferty, John; Williams, Chris K. I.; and Culotta, Aron (eds.), Advances in Neural Information Processing Systems 22 (NIPS'22), December 7th–10th, 2009, Vancouver, BC, Neural Information Processing Systems (NIPS) Foundation, 2009, pp. 545–552 [27] A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J. Schmidhuber. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009. [28] http:/ / www. scholarpedia. org/ article/ Deep_belief_networks / [29] Hinton, G. E.; Osindero, S.; Teh, Y. (2006). "A fast learning algorithm for deep belief nets" (http:/ / www. cs. toronto. edu/ ~hinton/ absps/ fastnc. pdf). Neural Computation 18 (7): 1527–1554. doi:10.1162/neco.2006.18.7.1527. PMID 16764513. . [30] K. Fukushima. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4): 93-202, 1980. [31] M Riesenhuber, T Poggio. Hierarchical models of object recognition in cortex. Nature neuroscience, 1999. [32] D. C. Ciresan, U. Meier, J. Masci, J. Schmidhuber. Multi-Column Deep Neural Network for Traffic Sign Classification. Neural Networks, 2012. Neural network 197 [33] D. Ciresan, A. Giusti, L. Gambardella, J. Schmidhuber. Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images. In Advances in Neural Information Processing Systems (NIPS 2012), Lake Tahoe, 2012. [34] D. C. Ciresan, U. Meier, J. Schmidhuber. Multi-column Deep Neural Networks for Image Classification. IEEE Conf. on Computer Vision and Pattern Recognition CVPR 2012. External links • A Brief Introduction to Neural Networks (D. Kriesel) (http://www.dkriesel.com/en/science/neural_networks) - Illustrated, bilingual manuscript about artificial neural networks; Topics so far: Perceptrons, Backpropagation, Radial Basis Functions, Recurrent Neural Networks, Self Organizing Maps, Hopfield Networks. • Review of Neural Networks in Materials Science (http://www.msm.cam.ac.uk/phase-trans/abstracts/neural. review.html) • Artificial Neural Networks Tutorial in three languages (Univ. Politécnica de Madrid) (http://www.gc.ssr.upm. es/inves/neural/ann1/anntutorial.html) • Another introduction to ANN (http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html) • Next Generation of Neural Networks (http://youtube.com/watch?v=AyzOUbkUf3M) - Google Tech Talks • Performance of Neural Networks (http://www.msm.cam.ac.uk/phase-trans/2009/performance.html) • Neural Networks and Information (http://www.msm.cam.ac.uk/phase-trans/2009/ review_Bhadeshia_SADM.pdf) Econometric model Econometric models are statistical models used in econometrics. An econometric model specifies the statistical relationship that is believed to hold between the various economic quantities pertaining to a particular economic phenomenon under study. An econometric model can be derived from a deterministic economic model by allowing for uncertainty, or from an economic model which itself is stochastic. However, it is also possible to use econometric models that are not tied to any specific economic theory.[1] A simple example of an econometric model is one that assumes that monthly spending by consumers is linearly dependent on consumers' income in the previous month. Then the model will consist of the equation where Ct is consumer spending in month t, Yt-1 is income during the previous month, and et is an error term measuring the extent to which the model cannot fully explain consumption. Then one objective of the econometrician is to obtain estimates of the parameters a and b; these estimated parameter values, when used in the model's equation, enable predictions for future values of consumption to be made contingent on the prior month's income. Formal definition In econometrics, as in statistics in general, it is presupposed that the quantities being analyzed can be treated as random variables. An econometric model then is a set of joint probability distributions to which the true joint probability distribution of the variables under study is supposed to belong. In the case in which the elements of this set can be indexed by a finite number of real-valued parameters, the model is called a parametric model; otherwise it is a nonparametric or semiparametric model. A large part of econometrics is the study of methods for selecting models, estimating them, and carrying out inference on them. The most common econometric models are structural, in that they convey causal and counterfactual information,[2] and are used for policy evaluation. For example, an equation modeling consumption spending based on income could be used to see what consumption would be contingent on any of various hypothetical levels of income, only Econometric model 198 one of which (depending on the choice of a fiscal policy) will end up actually occurring. Basic models Some of the common econometric models are: • Linear regression • Generalized linear models • Tobit • ARIMA • Vector Autoregression • Cointegration Use in policy-making Comprehensive models of macroeconomic relationships are used by central banks and governments to evaluate and guide economic policy. One famous econometric models of this nature are the Federal Reserve Bank econometric model. References [1] Sims, Christopher A. (Jan., 1980). "Macroeconomics and Reality". Econometrica 48 (1): 1–48. doi:10.2307/1912017. JSTOR 1912017. [2] Pearl, J. Causality: Models, Reasoning, and Inference, Cambridge University Press, 2000. • Granger, Clive (1991). Modelling Economic Series: Readings in Econometric Methodology. Oxford University Press. ISBN 978-0-19-828736-0. • Davidson, Russell; James G. MacKinnon (1993). Estimation and Inference in Econometrics. Oxford University Press. ISBN 978-0-19-506011-9. • Adrian, Pagan; Aman Ullah (1999). Nonparametric Econometrics. Cambridge University Press. ISBN 978-0-521-58611-5. External links • Manuscript of Bruce Hansen's book on Econometrics (http://www.ssc.wisc.edu/~bhansen/econometrics/) Statistical model 199 Statistical model A statistical model is a formalization of relationships between variables in the form of mathematical equations. A statistical model describes how one or more random variables are related to one or more other variables. The model is statistical as the variables are not deterministically but stochastically related. In mathematical terms, a statistical model is frequently thought of as a pair where is the set of possible observations and the set of possible probability distributions on . It is assumed that there is a distinct element of which generates the observed data. Statistical inference enables us to make statements about which element(s) of this set are likely to be the true one. Most statistical tests can be described in the form of a statistical model. For example, the Student's t-test for comparing the means of two groups can be formulated as seeing if an estimated parameter in the model is different from 0. Another similarity between tests and models is that there are assumptions involved. Error is assumed to be normally distributed in most models.[1] Formal definition A statistical model is a collection of probability distribution functions or probability density functions (collectively referred to as distributions for brevity). A parametric model is a collection of distributions, each of which is indexed by a unique finite-dimensional parameter: , where is a parameter and is the feasible region of parameters, which is a subset of d-dimensional Euclidean space. A statistical model may be used to describe the set of distributions from which one assumes that a particular data set is sampled. For example, if one assumes that data arise from a univariate Gaussian distribution, then one has assumed a Gaussian model: . A non-parametric model is a set of probability distributions with infinite dimensional parameters, and might be written as . A semi-parametric model also has infinite dimensional parameters, but is not dense in the space of distributions. For example, a mixture of Gaussians with one Gaussian at each data point is dense in the space of distributions. Formally, if d is the dimension of the parameter, and n is the number of samples, if as and as , then the model is semi-parametric. Model comparison Models can be compared to each other. This can either be done when you have done an exploratory data analysis or a confirmatory data analysis. In an exploratory analysis, you formulate all models you can think of, and see which describes your data best. In a confirmatory analysis you test which of your models you have described before the data was collected fits the data best, or test if your only model fits the data. In linear regression analysis you can compare the amount of variance explained by the independent variables, R2, across the different models. In general, you can compare models that are nested by using a Likelihood-ratio test. Nested models are models that can be obtained by restricting a parameter in a more complex model to be zero. An example Length and age are probabilistically distributed over humans. They are stochastically related, when you know that a person is of age 7, this influences the chance of this person being 6 feet tall. You could formalize this relationship in a linear regression model of the following form: lengthi = b0 + b1agei + εi, where b0 is the intercept, b1 is a parameter that age is multiplied by to get a prediction of length, ε is the error term, and i is the subject. This means that length starts at some value, there is a minimum length when someone is born, and it is predicted by age to some amount. This prediction is not perfect as error is included in the model. This error contains variance that stems from sex and Statistical model 200 other variables. When sex is included in the model, the error term will become smaller, as you will have a better idea of the chance that a particular 16-year-old is 6 feet tall when you know this 16-year-old is a girl. The model would become lengthi = b0 + b1agei + b2sexi + εi, where the variable sex is dichotomous. This model would presumably have a higher R2. The first model is nested in the second model: the first model is obtained from the second when b2 is restricted to zero. Classification According to the number of the endogenous variables and the number of equations, models can be classified as complete models (the number of equations equals to the number of endogenous variables) and incomplete models. Some other statistical models are the general linear model (restricted to continuous dependent variables), the generalized linear model (for example, logistic regression), the multilevel model, and the structural equation model.[2] References [1] Field, A. (2005). Discovering statistics using SPSS. Sage, London. [2] Adèr, H.J. (2008). Chapter 12: Modelling. In H.J. Adèr & G.J. Mellenbergh (Eds.) (with contributions by D.J. Hand), Advising on Research Methods: A consultant's companion (pp. 271-304). Huizen, The Netherlands: Johannes van Kessel Publishing. Model selection Model selection is the task of selecting a statistical model from a set of candidate models, given data. In the simplest cases, a pre-existing set of data is considered. However, the task can also involve the design of experiments such that the data collected is well-suited to the problem of model selection. Given candidate models of similar predictive or explanatory power, the simplest model is most likely to be correct.[1] Introduction In its most basic forms, model selection is one of the fundamental tasks of scientific inquiry. Determining the principle that explains a series of observations is often linked directly to a mathematical model predicting those observations. For example, when Galileo performed his inclined plane experiments, he demonstrated that the motion of the balls fitted the parabola predicted by his model. Of the countless number of possible mechanisms and processes that could have produced the data, how can one even begin to choose the best model? The mathematical approach commonly taken decides among a set of candidate models; this set must be chosen by the researcher. Often simple models such as polynomials are used, at least initially. Burnham and Anderson (2002) emphasize the importance of The scientific observation cycle. choosing models based on sound scientific principles, modeling the underlying data throughout their book. Once the set of candidate models has been chosen, the mathematical analysis allows us to select the best of these models. What is meant by best is controversial. A good model selection technique will balance goodness of fit with simplicity. More complex models will be better able to adapt their shape to fit the data (for example, a fifth-order polynomial can exactly fit six points), but the additional parameters may not represent anything useful. (Perhaps Model selection 201 those six points are really just randomly distributed about a straight line.) Goodness of fit is generally determined using a likelihood ratio approach, or an approximation of this, leading to a chi-squared test. The complexity is generally measured by counting the number of parameters in the model. Model selection techniques can be considered as estimators of some physical quantity, such as the probability of the model producing the given data. The bias and variance are both important measures of the quality of this estimator. Asymptotic efficiency is also often considered. A standard example of model selection is that of curve fitting, where, given a set of points and other background knowledge (e.g. points are a result of i.i.d. samples), we must select a curve that describes the function that generated the points. Methods for choosing the set of candidate models • Scientific method • Statistical hypothesis testing Experiments for choosing the set of candidate models • Design of experiments • Optimal design • Fractional factorial design (Screening experiments) Criteria for model selection • Akaike information criterion • Bayes factor • Bayesian information criterion • Deviance information criterion • False discovery rate • Focused information criterion • Mallows' Cp • Minimum description length (Algorithmic information theory) • Minimum message length (Algorithmic information theory) • Structural Risk Minimization • Stepwise regression • Cross-validation References [1] This follows directly from formal expressions of Occam's Razor such as Minimum Message Length and others. • Anderson, D.R. (2008), Model Based Inference in the Life Sciences, Springer. • Aznar Grasa, A. (1989). Econometric Model Selection: A New Approach, Springer. ISBN 978-0-7923-0321-3 • Burnham, K.P., and Anderson, D.R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed. Springer-Verlag. ISBN 0-387-95364-7 [This has over 16000 citations on Google Scholar.] • Chamberlin, T.C. (1890) "The method of multiple working hypotheses", Science 15: 93. (Reprinted 1965, Science 148: 754–759 (http://www.d.umn.edu/~mille066/Teaching/3000/Chamberlin-MWH.pdf).) • Claeskens, G., and Hjort, N.L. (2008). Model Selection and Model Averaging, Cambridge University Press. Model selection 202 • Phil C. Gregory (http://www.physics.ubc.ca/~gregory/gregory.html) (2005) Bayesian logical data analysis for the physical sciences: A comparative approach with Mathematica support (Cambridge U. Press, Cambridge UK) preview (http://books.google.com/books?id=yJ_5VFo0zGMC). • Lahiri, P. (2001). Model Selection, Institute of Mathematical Statistics. • Massart, P. (2007). Concentration Inequalities and Model Selection, Springer. Economic forecasting Economic forecasting is the process of making predictions about the economy. Forecasts can be carried out at a high level of aggregation - for example for GDP, inflation, unemployment or the fiscal deficit - or at a more disaggregated level, for specific sectors of the economy or even specific firms. Many institutions engage in forecasting, including international organisations such as the IMF [1], World Bank [2] and the OECD [3], national governments and central banks, and private sector entities, be they think tanks, banks or others. Some forecasts are produced annually, but many are updated more frequently. The World Bank model is available for individuals and organizations to run their own simulations and forecasts using its iSimulate platform [4]. Consensus Economics Inc [5]., among others, compiles the macroeconomic forecasts prepared by a variety of forecasters, and publishes them every month. The Economist magazine [6] regularly provides such a snapshot as well, for a broad range of countries (a recent example is provided here [7]). The financial and economic crisis that erupted in 2007 - arguably the worst since the Great Depression of the 1930s - was not foreseen by most of the forecasters, even if a few lone analysts had been crying wolf for some time (for example, Nouriel Roubini and Robert Shiller). The failure to forecast the "Great Recession" has caused a lot of soul searching in the profession. The Queen of England herself asked why had nobody noticed that the credit crunch was on its way, and a group of economists - experts from business, the City, its regulators, academia, and government - tried to explain in a letter.[8] Methods of forecasting include Econometric models, Economic base analysis, Shift-share analysis, Input-output model and the Grinold and Kroner Model. See also Land use forecasting, Reference class forecasting, Transportation planning, Calculating Demand Forecast Accuracy and Consensus forecasts References • IMF forecasts can be found here: IMF World Economic Outlook [9], World Bank forecasts are here World Bank Global Economic Prospects [10], and OECD forecasts here: Economic Outlook [11] • Two of the leading journals in the field of economic forecasting are the Journal of Forecasting [12] and the International Journal of Forecasting [13] • For a comprehensive but quite technical compendium, see Handbook of Economic Forecasting [14], North-Holland: Elsevier, 2006 • A more compact and more accessible, but pre-crisis overview is provided in Elements of Forecasting [15], 4th edn, Cincinnati, OH: South-Western College Publishing, 2007 • A recent, comprehensive and accessible guide to forecasting is Economic Forecasting and Policy [16], 2d edn, Palgrave, 2011 Economic forecasting 203 External links • International Institute of Forecasters [17] • How to predict the economy? [18] Note [1] http:/ / www. imf. org/ external/ index. htm [2] http:/ / www. worldbank. org [3] http:/ / www. oecd. org [4] http:/ / isimulate. worldbank. org [5] http:/ / www. consensuseconomics. com/ [6] http:/ / www. economist. com [7] http:/ / www. economist. com/ node/ 18959289?story_id=18959289 [8] http:/ / www. ft. com/ intl/ cms/ 3e3b6ca8-7a08-11de-b86f-00144feabdc0. pdf [9] http:/ / www. imf. org/ external/ ns/ cs. aspx?id=29 [10] http:/ / www. worldbank. org/ globaloutlook [11] http:/ / www. oecd. org/ eco/ economicoutlook. htm''OECD [12] http:/ / onlinelibrary. wiley. com/ journal/ 10. 1002/ (ISSN)1099-131X [13] http:/ / www. elsevier. com/ wps/ find/ journaldescription. cws_home/ 505555/ description [14] http:/ / www. elsevier. com/ wps/ find/ bookdescription. cws_home/ 672846/ description#description [15] http:/ / www. amazon. com/ Elements-Forecasting-Francis-X-Diebold/ dp/ 0538862440 [16] http:/ / us. macmillan. com/ economicforecastingandpolicy [17] http:/ / www. forecasters. org/ [18] http:/ / www. simonspicks. com/ economics/ howtopredicttheeconomy Mathematical optimization In mathematics, statistics, empirical sciences, computer science, or management science, mathematical optimization (alternatively, optimization or mathematical programming) is the selection of a best element (with regard to some criteria) from some set of available alternatives.[1] In the simplest case, an optimization problem consists of maximizing or minimizing a real function by systematically choosing input values from within an allowed set and computing the value of the function. The generalization of optimization theory and techniques to other Graph of a paraboloid given by f(x,y) = formulations comprises a large area of applied mathematics. More -(x²+y²)+4. The global maximum at (0,0,4) is generally, optimization includes finding "best available" values of indicated by a red dot. some objective function given a defined domain, including a variety of different types of objective functions and different types of domains. Mathematical optimization 204 Optimization problems An optimization problem can be represented in the following way Given: a function f : A R from some set A to the real numbers Sought: an element x0 in A such that f(x0) ≤ f(x) for all x in A ("minimization") or such that f(x0) ≥ f(x) for all x in A ("maximization"). Such a formulation is called an optimization problem or a mathematical programming problem (a term not directly related to computer programming, but still in use for example in linear programming - see History below). Many real-world and theoretical problems may be modeled in this general framework. Problems formulated using this technique in the fields of physics and computer vision may refer to the technique as energy minimization, speaking of the value of the function f as representing the energy of the system being modeled. Typically, A is some subset of the Euclidean space Rn, often specified by a set of constraints, equalities or inequalities that the members of A have to satisfy. The domain A of f is called the search space or the choice set, while the elements of A are called candidate solutions or feasible solutions. The function f is called, variously, an objective function, cost function (minimization),[2] indirect utility function (minimization),[3] utility function (maximization), or, in certain fields, energy function, or energy functional. A feasible solution that minimizes (or maximizes, if that is the goal) the objective function is called an optimal solution. By convention, the standard form of an optimization problem is stated in terms of minimization. Generally, unless both the objective function and the feasible region are convex in a minimization problem, there may be several local minima, where a local minimum x* is defined as a point for which there exists some δ > 0 so that for all x such that the expression holds; that is to say, on some region around x* all of the function values are greater than or equal to the value at that point. Local maxima are defined similarly. A large number of algorithms proposed for solving non-convex problems – including the majority of commercially available solvers – are not capable of making a distinction between local optimal solutions and rigorous optimal solutions, and will treat the former as actual solutions to the original problem. The branch of applied mathematics and numerical analysis that is concerned with the development of deterministic algorithms that are capable of guaranteeing convergence in finite time to the actual optimal solution of a non-convex problem is called global optimization. Notation Optimization problems are often expressed with special notation. Here are some examples. Minimum and maximum value of a function Consider the following notation: This denotes the minimum value of the objective function , when choosing x from the set of real numbers . The minimum value in this case is , occurring at . Similarly, the notation Mathematical optimization 205 asks for the maximum value of the objective function 2x, where x may be any real number. In this case, there is no such maximum as the objective function is unbounded, so the answer is "infinity" or "undefined". Optimal input arguments Consider the following notation: or equivalently This represents the value (or values) of the argument x in the interval that minimizes (or minimize) the objective function x2 + 1 (the actual minimum value of that function is not what the problem asks for). In this case, the answer is x = -1, since x = 0 is infeasible, i.e. does not belong to the feasible set. Similarly, or equivalently represents the pair (or pairs) that maximizes (or maximize) the value of the objective function , with the added constraint that x lie in the interval (again, the actual maximum value of the expression does not matter). In this case, the solutions are the pairs of the form (5, 2kπ) and (−5,(2k+1)π), where k ranges over all integers. Arg min and arg max are sometimes also written argmin and argmax, and stand for argument of the minimum and argument of the maximum. History Fermat and Lagrange found calculus-based formulas for identifying optima, while Newton and Gauss proposed iterative methods for moving towards an optimum. Historically, the first term for optimization was "linear programming", which was due to George B. Dantzig, although much of the theory had been introduced by Leonid Kantorovich in 1939. Dantzig published the Simplex algorithm in 1947, and John von Neumann developed the theory of duality in the same year. The term programming in this context does not refer to computer programming. Rather, the term comes from the use of program by the United States military to refer to proposed training and logistics schedules, which were the problems Dantzig studied at that time. Later important researchers in mathematical optimization include the following: Mathematical optimization 206 • Richard Bellman • Arkadi Nemirovski • Ronald A. Howard • Yurii Nesterov • Narendra Karmarkar • Boris Polyak • William Karush • Lev Pontryagin • Leonid Khachiyan • James Renegar • Bernard Koopman • R. Tyrrell Rockafellar • Harold Kuhn • Cornelis Roos • Joseph Louis Lagrange • Naum Z. Shor • László Lovász • Michael J. Todd • Albert Tucker Major subfields • Convex programming studies the case when the objective function is convex (minimization) or concave (maximization) and the constraint set is convex. This can be viewed as a particular case of nonlinear programming or as generalization of linear or convex quadratic programming. • Linear programming (LP), a type of convex programming, studies the case in which the objective function f is linear and the set of constraints is specified using only linear equalities and inequalities. Such a set is called a polyhedron or a polytope if it is bounded. • Second order cone programming (SOCP) is a convex program, and includes certain types of quadratic programs. • Semidefinite programming (SDP) is a subfield of convex optimization where the underlying variables are semidefinite matrices. It is generalization of linear and convex quadratic programming. • Conic programming is a general form of convex programming. LP, SOCP and SDP can all be viewed as conic programs with the appropriate type of cone. • Geometric programming is a technique whereby objective and inequality constraints expressed as posynomials and equality constraints as monomials can be transformed into a convex program. • Integer programming studies linear programs in which some or all variables are constrained to take on integer values. This is not convex, and in general much more difficult than regular linear programming. • Quadratic programming allows the objective function to have quadratic terms, while the feasible set must be specified with linear equalities and inequalities. For specific forms of the quadratic term, this is a type of convex programming. • Fractional programming studies optimization of ratios of two nonlinear functions. The special class of concave fractional programs can be transformed to a convex optimization problem. • Nonlinear programming studies the general case in which the objective function or the constraints or both contain nonlinear parts. This may or may not be a convex program. In general, whether the program is convex affects the difficulty of solving it. • Stochastic programming studies the case in which some of the constraints or parameters depend on random variables. • Robust programming is, like stochastic programming, an attempt to capture uncertainty in the data underlying the optimization problem. This is not done through the use of random variables, but instead, the problem is solved taking into account inaccuracies in the input data. • Combinatorial optimization is concerned with problems where the set of feasible solutions is discrete or can be reduced to a discrete one. • Stochastic optimization for use with random (noisy) function measurements or random inputs in the search process. Mathematical optimization 207 • Infinite-dimensional optimization studies the case when the set of feasible solutions is a subset of an infinite-dimensional space, such as a space of functions. • Heuristics and metaheuristics make few or no assumptions about the problem being optimized. Usually, heuristics do not guarantee that any optimal solution need be found. On the other hand, heuristics are used to find approximate solutions for many complicated optimization problems. • Constraint satisfaction studies the case in which the objective function f is constant (this is used in artificial intelligence, particularly in automated reasoning). • Constraint programming. • Disjunctive programming is used where at least one constraint must be satisfied but not all. It is of particular use in scheduling. In a number of subfields, the techniques are designed primarily for optimization in dynamic contexts (that is, decision making over time): • Calculus of variations seeks to optimize an objective defined over many points in time, by considering how the objective function changes if there is a small change in the choice path. • Optimal control theory is a generalization of the calculus of variations. • Dynamic programming studies the case in which the optimization strategy is based on splitting the problem into smaller subproblems. The equation that describes the relationship between these subproblems is called the Bellman equation. • Mathematical programming with equilibrium constraints is where the constraints include variational inequalities or complementarities. Multi-objective optimization Adding more than one objective to an optimization problem adds complexity. For example, to optimize a structural design, one would want a design that is both light and rigid. Because these two objectives conflict, a trade-off exists. There will be one lightest design, one stiffest design, and an infinite number of designs that are some compromise of weight and stiffness. The set of trade-off designs that cannot be improved upon according to one criterion without hurting another criterion is known as the Pareto set. The curve created plotting weight against stiffness of the best designs is known as the Pareto frontier. A design is judged to be "Pareto optimal" (equivalently, "Pareto efficient" or in the Pareto set) if it is not dominated by any other design: If it is worse than another design in some respects and no better in any respect, then it is dominated and is not Pareto optimal. The choice among "Pareto optimal" solutions to determine the "favorite solution" is delegated to the decision maker. In other words, defining the problem as multiobjective optimization signals that some information is missing: desirable objectives are given but not their detailed combination. In some cases, the missing information can be derived by interactive sessions with the decision maker. Multi-objective optimization problems have been generalized further to vector optimization problems where the (partial) ordering is no longer given by the Pareto ordering. Mathematical optimization 208 Multi-modal optimization Optimization problems are often multi-modal; that is, they possess multiple good solutions. They could all be globally good (same cost function value) or there could be a mix of globally good and locally good solutions. Obtaining all (or at least some of) the multiple solutions is the goal of a multi-modal optimizer. Classical optimization techniques due to their iterative approach do not perform satisfactorily when they are used to obtain multiple solutions, since it is not guaranteed that different solutions will be obtained even with different starting points in multiple runs of the algorithm. Evolutionary Algorithms are however a very popular approach to obtain multiple solutions in a multi-modal optimization task. See Evolutionary multi-modal optimization. Classification of critical points and extrema Feasibility problem The satisfiability problem, also called the feasibility problem, is just the problem of finding any feasible solution at all without regard to objective value. This can be regarded as the special case of mathematical optimization where the objective value is the same for every solution, and thus any solution is optimal. Many optimization algorithms need to start from a feasible point. One way to obtain such a point is to relax the feasibility conditions using a slack variable; with enough slack, any starting point is feasible. Then, minimize that slack variable until slack is null or negative. Existence The extreme value theorem of Karl Weierstrass states that a continuous real-valued function on a compact set attains its maximum and minimum value. More generally, a lower semi-continuous function on a compact set attains its minimum; an upper semi-continuous function on a compact set attains its maximum. Necessary conditions for optimality One of Fermat's theorems states that optima of unconstrained problems are found at stationary points, where the first derivative or the gradient of the objective function is zero (see first derivative test). More generally, they may be found at critical points, where the first derivative or gradient of the objective function is zero or is undefined, or on the boundary of the choice set. An equation (or set of equations) stating that the first derivative(s) equal(s) zero at an interior optimum is called a 'first-order condition' or a set of first-order conditions. Optima of inequality-constrained problems are instead found by the Lagrange multiplier method. This method calculates a system of inequalities called the 'Karush–Kuhn–Tucker conditions' or 'complementary slackness conditions', which may then be used to calculate the optimum. Sufficient conditions for optimality While the first derivative test identifies points that might be extrema, this test does not distinguish a point that is a minimum from one that is a maximum or one that is neither. When the objective function is twice differentiable, these cases can be distinguished by checking the second derivative or the matrix of second derivatives (called the Hessian matrix) in unconstrained problems, or the matrix of second derivatives of the objective function and the constraints called the bordered Hessian in constrained problems. The conditions that distinguish maxima, or minima, from other stationary points are called 'second-order conditions' (see 'Second derivative test'). If a candidate solution satisfies the first-order conditions, then satisfaction of the second-order conditions as well is sufficient to establish at least local optimality. Mathematical optimization 209 Sensitivity and continuity of optima The envelope theorem describes how the value of an optimal solution changes when an underlying parameter changes. The process of computing this change is called comparative statics. The maximum theorem of Claude Berge (1963) describes the continuity of an optimal solution as a function of underlying parameters. Calculus of optimization For unconstrained problems with twice-differentiable functions, some critical points can be found by finding the points where the gradient of the objective function is zero (that is, the stationary points). More generally, a zero subgradient certifies that a local minimum has been found for minimization problems with convex functions and other locally Lipschitz functions. Further, critical points can be classified using the definiteness of the Hessian matrix: If the Hessian is positive definite at a critical point, then the point is a local minimum; if the Hessian matrix is negative definite, then the point is a local maximum; finally, if indefinite, then the point is some kind of saddle point. Constrained problems can often be transformed into unconstrained problems with the help of Lagrange multipliers. Lagrangian relaxation can also provide approximate solutions to difficult constrained problems. When the objective function is convex, then any local minimum will also be a global minimum. There exist efficient numerical techniques for minimizing convex functions, such as interior-point methods. Computational optimization techniques To solve problems, researchers may use algorithms that terminate in a finite number of steps, or iterative methods that converge to a solution (on some specified class of problems), or heuristics that may provide approximate solutions to some problems (although their iterates need not converge). Optimization algorithms • Simplex algorithm of George Dantzig, designed for linear programming. • Extensions of the simplex algorithm, designed for quadratic programming and for linear-fractional programming. • Variants of the simplex algorithm that are especially suited for network optimization. • Combinatorial algorithms Iterative methods The iterative methods used to solve problems of nonlinear programming differ according to whether they evaluate Hessians, gradients, or only function values. While evaluating Hessians (H) and gradients (G) improves the rate of convergence, for functions for which these quantities exist and vary sufficiently smoothly, such evaluations increase the computational complexity (or computational cost) of each iteration. In some cases, the computational complexity may be excessively high. One major criterion for optimizers is just the number of required function evaluations as this often is already a large computational effort, usually much more effort than within the optimizer itself, which mainly has to operate over the N variables. The derivatives provide detailed information for such optimizers, but are even harder to calculate, e.g. approximating the gradient takes at least N+1 function evaluations. For approximations of the 2nd derivatives (collected in the Hessian matrix) the number of function evaluations is in the order of N². Newton's method requires the 2nd order derivates, so for each iteration the number of function calls is in the order of N², but for a simpler pure gradient optimizer it is only N. However, gradient optimizers need usually more iterations than Newton's algorithm. Which one is best with respect to the number of function calls depends on the problem itself. Mathematical optimization 210 • Methods that evaluate Hessians (or approximate Hessians, using finite differences): • Newton's method • Sequential quadratic programming: A Newton-based method for small-medium scale constrained problems. Some versions can handle large-dimensional problems. • Methods that evaluate gradients or approximate gradients using finite differences (or even subgradients): • Quasi-Newton methods: Iterative methods for medium-large problems (e.g. N<1000). • Conjugate gradient methods: Iterative methods for large problems. (In theory, these methods terminate in a finite number of steps with quadratic objective functions, but this finite termination is not observed in practice on finite–precision computers.) • Interior point methods: This is a large class of methods for constrained optimization. Some interior-point methods use only (sub)gradient information, and others of which require the evaluation of Hessians. • Gradient descent (alternatively, "steepest descent" or "steepest ascent"): A (slow) method of historical and theoretical interest, which has had renewed interest for finding approximate solutions of enormous problems. • Subgradient methods - An iterative method for large locally Lipschitz functions using generalized gradients. Following Boris T. Polyak, subgradient–projection methods are similar to conjugate–gradient methods. • Bundle method of descent: An iterative method for small–medium sized problems with locally Lipschitz functions, particularly for convex minimization problems. (Similar to conjugate gradient methods) • Ellipsoid method: An iterative method for small problems with quasiconvex objective functions and of great theoretical interest, particularly in establishing the polynomial time complexity of some combinatorial optimization problems. It has similarities with Quasi-Newton methods. • Reduced gradient method (Frank–Wolfe) for approximate minimization of specially structured problems with linear constraints, especially with traffic networks. For general unconstrained problems, this method reduces to the gradient method, which is regarded as obsolete (for almost all problems). • Simultaneous perturbation stochastic approximation (SPSA) method for stochastic optimization; uses random (efficient) gradient approximation. • Methods that evaluate only function values: If a problem is continuously differentiable, then gradients can be approximated using finite differences, in which case a gradient-based method can be used. • Interpolation methods • Pattern search methods, which have better convergence properties than the Nelder–Mead heuristic (with simplices), which is listed below. Global convergence More generally, if the objective function is not a quadratic function, then many optimization methods use other methods to ensure that some subsequence of iterations converges to an optimal solution. The first and still popular method for ensuring convergence relies on line searches, which optimize a function along one dimension. A second and increasingly popular method for ensuring convergence uses trust regions. Both line searches and trust regions are used in modern methods of non-differentiable optimization. Usually a global optimizer is much slower than advanced local optimizers (such as BFGS), so often an efficient global optimizer can be constructed by starting the local optimizer from different starting points. Mathematical optimization 211 Heuristics Besides (finitely terminating) algorithms and (convergent) iterative methods, there are heuristics that can provide approximate solutions to some optimization problems: • Memetic algorithm • Differential evolution • Differential Search algorithm [4] Matlab code-link has been provided in Çivicioglu, P.,(2012). • Dynamic relaxation • Genetic algorithms • Hill climbing • Nelder-Mead simplicial heuristic: A popular heuristic for approximate minimization (without calling gradients) • Particle swarm optimization • Artificial bee colony optimization • Simulated annealing • Tabu search • Reactive Search Optimization (RSO)[5] implemented in LIONsolver Applications Mechanics and engineering Problems in rigid body dynamics (in particular articulated rigid body dynamics) often require mathematical programming techniques, since you can view rigid body dynamics as attempting to solve an ordinary differential equation on a constraint manifold; the constraints are various nonlinear geometric constraints such as "these two points must always coincide", "this surface must not penetrate any other", or "this point must always lie somewhere on this curve". Also, the problem of computing contact forces can be done by solving a linear complementarity problem, which can also be viewed as a QP (quadratic programming) problem. Many design problems can also be expressed as optimization programs. This application is called design optimization. One subset is the engineering optimization, and another recent and growing subset of this field is multidisciplinary design optimization, which, while useful in many problems, has in particular been applied to aerospace engineering problems. Economics Economics is closely enough linked to optimization of agents that an influential definition relatedly describes economics qua science as the "study of human behavior as a relationship between ends and scarce means" with alternative uses.[6] Modern optimization theory includes traditional optimization theory but also overlaps with game theory and the study of economic equilibria. The Journal of Economic Literature codes classify mathematical programming, optimization techniques, and related topics under JEL:C61-C63. In microeconomics, the utility maximization problem and its dual problem, the expenditure minimization problem, are economic optimization problems. Insofar as they behave consistently, consumers are assumed to maximize their utility, while firms are usually assumed to maximize their profit. Also, agents are often modeled as being risk-averse, thereby preferring to avoid risk. Asset prices are also modeled using optimization theory, though the underlying mathematics relies on optimizing stochastic processes rather than on static optimization. Trade theory also uses optimization to explain trade patterns between nations. The optimization of market portfolios is an example of multi-objective optimization in economics. Since the 1970s, economists have modeled dynamic decisions over time using control theory. For example, microeconomists use dynamic search models to study labor-market behavior.[7] A crucial distinction is between Mathematical optimization 212 deterministic and stochastic models.[8] Macroeconomists build dynamic stochastic general equilibrium (DSGE) models that describe the dynamics of the whole economy as the result of the interdependent optimizing decisions of workers, consumers, investors, and governments.[9][10] Operations research Another field that uses optimization techniques extensively is operations research. Operations research also uses stochastic modeling and simulation to support improved decision-making. Increasingly, operations research uses stochastic programming to model dynamic decisions that adapt to events; such problems can be solved with large-scale optimization and stochastic optimization methods. Control engineering Mathematical optimization is used in much modern controller design. High-level controllers such as Model predictive control (MPC) or Real-Time Optimization (RTO) employ mathematical optimization. These algorithms run online and repeatedly determine values for decision variables, such as choke openings in a process plant, by iteratively solving a mathematical optimization problem including constraints and a model of the system to be controlled. Notes [1] " The Nature of Mathematical Programming (http:/ / glossary. computing. society. informs. org/ index. php?page=nature. html)," Mathematical Programming Glossary, INFORMS Computing Society. [2] W. Erwin Diewert (2008). "cost functions," The New Palgrave Dictionary of Economics, 2nd Edition Contents (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000390& edition=current& q=). [3] Peter Newman (2008). "indirect utility function," The New Palgrave Dictionary of Economics, 2nd Edition. Contents. (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_I000066& edition=) [4] Civicioglu, P. (2012). "Transforming geocentric cartesian coordinates to geodetic coordinates by using differential search algorithm". Computers & Geosciences 46: 229–247. doi:10.1016/j.cageo.2011.12.011. [5] Battiti, Roberto; Mauro Brunato; Franco Mascia (2008). Reactive Search and Intelligent Optimization (http:/ / reactive-search. org/ thebook). Springer Verlag. ISBN 978-0-387-09623-0. . [6] Lionel Robbins (1935, 2nd ed.) An Essay on the Nature and Significance of Economic Science, Macmillan, p. 16. [7] A. K. Dixit ([1976] 1990). Optimization in Economic Theory, 2nd ed., Oxford. Description (http:/ / books. google. com/ books?id=dHrsHz0VocUC& pg=find& pg=PA194=false#v=onepage& q& f=false) and contents preview (http:/ / books. google. com/ books?id=dHrsHz0VocUC& pg=PR7& lpg=PR6& dq=false& lr=#v=onepage& q=false& f=false). [8] A.G. Malliaris (2008). "stochastic optimal control," The New Palgrave Dictionary of Economics, 2nd Edition. Abstract (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_S000269& edition=& field=keyword& q=Taylor's th& topicid=& result_number=1). [9] Julio Rotemberg and Michael Woodford (1997), "An Optimization-based Econometric Framework for the Evaluation of Monetary Policy.NBER Macroeconomics Annual, 12, pp. 297-346. (http:/ / people. hbs. edu/ jrotemberg/ PublishedArticles/ OptimizBasedEconometric_97. pdf) [10] From The New Palgrave Dictionary of Economics (2008), 2nd Edition with Abstract links: • " numerical optimization methods in economics (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_N000148& edition=current& q=optimization& topicid=& result_number=1)" by Karl Schmedders • " convex programming (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_C000348& edition=current& q=optimization& topicid=& result_number=4)" by Lawrence E. Blume • " Arrow–Debreu model of general equilibrium (http:/ / www. dictionaryofeconomics. com/ article?id=pde2008_A000133& edition=current& q=optimization& topicid=& result_number=20)" by John Geanakoplos. Mathematical optimization 213 Further reading Comprehensive Undergraduate level • Bradley, S.; Hax, A.; Magnanti, T. (1977). Applied mathematical programming. Addison Wesley. • Rardin, Ronald L. (1997). Optimization in operations research. Prentice Hall. pp. 919. ISBN 0-02-398415-5. • Strang, Gilbert (1986). Introduction to applied mathematics (http://www.wellesleycambridge.com/tocs/ toc-appl). Wellesley, MA: Wellesley-Cambridge Press (Strang's publishing company). pp. xii+758. ISBN 0-9614088-0-4. MR870634. Graduate level • Magnanti, Thomas L. (1989). "Twenty years of mathematical programming". In Cornet, Bernard; Tulkens, Henry. Contributions to Operations Research and Economics: The twentieth anniversary of CORE (Papers from the symposium held in Louvain-la-Neuve, January 1987). Cambridge, MA: MIT Press. pp. 163–227. ISBN 0-262-03149-3. MR1104662. • Minoux, M. (1986). Mathematical programming: Theory and algorithms (Translated by Steven Vajda from the (1983 Paris: Dunod) French ed.). Chichester: A Wiley-Interscience Publication. John Wiley & Sons, Ltd.. pp. xxviii+489. ISBN 0-471-90170-9. MR2571910. (2008 Second ed., in French: Programmation mathématique: Théorie et algorithmes. Editions Tec & Doc, Paris, 2008. xxx+711 pp. ISBN 978-2-7430-1000-3.. • Nemhauser, G. L.; Rinnooy Kan, A. H. G.; Todd, M. J., eds. (1989). Optimization. Handbooks in Operations Research and Management Science. 1. Amsterdam: North-Holland Publishing Co.. pp. xiv+709. ISBN 0-444-87284-1. MR1105099. • J. E. Dennis, Jr. and Robert B. Schnabel, A view of unconstrained optimization (pp. 1–72); • Donald Goldfarb and Michael J. Todd, Linear programming (pp. 73–170); • Philip E. Gill, Walter Murray, Michael A. Saunders, and Margaret H. Wright, Constrained nonlinear programming (pp. 171–210); • Ravindra K. Ahuja, Thomas L. Magnanti, and James B. Orlin, Network flows (pp. 211–369); • W. R. Pulleyblank, Polyhedral combinatorics (pp. 371–446); • George L. Nemhauser and Laurence A. Wolsey, Integer programming (pp. 447–527); • Claude Lemaréchal, Nondifferentiable optimization (pp. 529–572); • Roger J-B Wets, Stochastic programming (pp. 573–629); • A. H. G. Rinnooy Kan and G. T. Timmer, Global optimization (pp. 631–662); • P. L. Yu, Multiple criteria decision making: five basic concepts (pp. 663–699). • Shapiro, Jeremy F. (1979). Mathematical programming: Structures and algorithms. New York: Wiley-Interscience [John Wiley & Sons]. pp. xvi+388. ISBN 0-471-77886-9. MR544669. • Spall, J. C. (2003), Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control, Wiley, Hoboken, NJ. Mathematical optimization 214 Continuous optimization • Mordecai Avriel (2003). Nonlinear Programming: Analysis and Methods. Dover Publishing. ISBN 0-486-43227-0. • Bonnans, J. Frédéric; Gilbert, J. Charles; Lemaréchal, Claude; Sagastizábal, Claudia A. (2006). Numerical optimization: Theoretical and practical aspects (http://www.springer.com/mathematics/applications/book/ 978-3-540-35445-1). Universitext (Second revised ed. of translation of 1997 French ed.). Berlin: Springer-Verlag. pp. xiv+490. doi:10.1007/978-3-540-35447-5. ISBN 3-540-35445-X. MR2265882. • Bonnans, J. Frédéric; Shapiro, Alexander (2000). Perturbation analysis of optimization problems. Springer Series in Operations Research. New York: Springer-Verlag. pp. xviii+601. ISBN 0-387-98705-3. MR1756264. • Boyd, Stephen P.; Vandenberghe, Lieven (2004) (pdf). Convex Optimization (http://www.stanford.edu/~boyd/ cvxbook/bv_cvxbook.pdf). Cambridge University Press. ISBN 978-0-521-83378-3. Retrieved October 15, 2011. • Jorge Nocedal and Stephen J. Wright (2006). Numerical Optimization (http://www.ece.northwestern.edu/ ~nocedal/book/num-opt.html). Springer. ISBN 0-387-30303-0. • Ruszczyński, Andrzej (2006). Nonlinear Optimization. Princeton, NJ: Princeton University Press. pp. xii+454. ISBN 978-0691119151. MR2199043. Combinatorial optimization • R. K. Ahuja, Thomas L. Magnanti, and James B. Orlin (1993). Network Flows: Theory, Algorithms, and Applications. Prentice-Hall, Inc. ISBN 0-13-617549-X. • William J. Cook, William H. Cunningham, William R. Pulleyblank, Alexander Schrijver; Combinatorial Optimization; John Wiley & Sons; 1 edition (November 12, 1997); ISBN 0-471-55894-X. • Gondran, Michel; Minoux, Michel (1984). Graphs and algorithms. Wiley-Interscience Series in Discrete Mathematics (Translated by Steven Vajda from the second (Collection de la Direction des Études et Recherches d'Électricité de France [Collection of the Department of Studies and Research of Électricité de France], v. 37. Paris: Éditions Eyrolles 1985. xxviii+545 pp. MR868083) French ed.). Chichester: John Wiley & Sons, Ltd.. pp. xix+650. ISBN 0-471-10374-8. MR2552933. (Fourth ed. Collection EDF R&D. Paris: Editions Tec & Doc 2009. xxxii+784 pp.. • Eugene Lawler (2001). Combinatorial Optimization: Networks and Matroids. Dover. ISBN 0-486-41453-1. • Lawler, E. L.; Lenstra, J. K.; Rinnooy Kan, A. H. G.; Shmoys, D. B. (1985), The traveling salesman problem: A guided tour of combinatorial optimization, John Wiley & Sons, ISBN 0-471-90413-9. • Jon Lee; A First Course in Combinatorial Optimization (http://books.google.com/ books?id=3pL1B7WVYnAC&printsec=frontcover&source=gbs_ge_summary_r&cad=0#v=onepage&q& f=false); Cambridge University Press; 2004; ISBN 0-521-01012-8. • Christos H. Papadimitriou and Kenneth Steiglitz Combinatorial Optimization : Algorithms and Complexity; Dover Pubns; (paperback, Unabridged edition, July 1998) ISBN 0-486-40258-4. Journals • Computational Optimization and Applications (http://www.springer.com/mathematics/journal/10589) • Journal of Computational Optimization in Economics and Finance (https://www.novapublishers.com/catalog/ product_info.php?products_id=6353) • Journal of Economic Dynamics and Control (http://www.journals.elsevier.com/ journal-of-economic-dynamics-and-control/) • SIAM Journal on Optimization (SIOPT) (http://www.siam.org/journals/siopt.php) and Editorial Policy (http:/ /www.siam.org/journals/siopt/policy.php) • SIAM Journal on Control and Optimization (SICON) (http://www.siam.org/journals/sicon.php) and Editorial Policy (http://www.siam.org/journals/sicon/policy.php) Mathematical optimization 215 External links • COIN-OR (http://www.coin-or.org/)—Computational Infrastructure for Operations Research • Decision Tree for Optimization Software (http://plato.asu.edu/guide.html) Links to optimization source codes • Global optimization (http://www.mat.univie.ac.at/~neum/glopt.html) • Mathematical Programming Glossary (http://glossary.computing.society.informs.org/) • Mathematical Programming Society (http://www.mathprog.org/) • NEOS Guide (http://www-fp.mcs.anl.gov/otc/Guide/index.html) currently being replaced by the NEOS Wiki (http://wiki.mcs.anl.gov/neos) • Optimization Online (http://www.optimization-online.org) A repository for optimization e-prints • Optimization Related Links (http://www2.arnes.si/~ljc3m2/igor/links.html) • Convex Optimization I (http://see.stanford.edu/see/courseinfo. aspx?coll=2db7ced4-39d1-4fdb-90e8-364129597c87) EE364a: Course from Stanford University • Convex Optimization – Boyd and Vandenberghe (http://www.stanford.edu/~boyd/cvxbook) Book on Convex Optimization • Book and Course (http://apmonitor.com/me575/index.php/Main/BookChapters) on Optimization Methods for Engineering Design • Optimization Online (http://www.optimization-online.org) A repository for optimization e-prints Programming paradigm A programming paradigm is a fundamental style of computer programming. There are four main paradigms: object-oriented, imperative, functional and declarative.[1] Their foundations are distinct models of computation: Turing machine for object-oriented and imperative programming, lambda calculus for functional programming, and first order logic for logic programming. Overview A programming model is an abstraction of a computer system. For example, the "von Neumann model" is a model used in traditional sequential computers. For parallel computing, there are many possible models typically reflecting different ways processors can be interconnected. The most common are based on shared memory, distributed memory with message passing, or a hybrid of the two. A programming language can support multiple paradigms. For example, programs written in C++ or Object Pascal can be purely procedural, or purely object-oriented, or contain elements of both paradigms. Software designers and programmers decide how to use those paradigm elements. In object-oriented programming, programmers can think of a program as a collection of interacting objects, while in functional programming a program can be thought of as a sequence of stateless function evaluations. When programming computers or systems with many processors, process-oriented programming allows programmers to think about applications as sets of concurrent processes acting upon logically shared data structures. Just as different groups in software engineering advocate different methodologies, different programming languages advocate different programming paradigms. Some languages are designed to support one particular paradigm (Smalltalk supports object-oriented programming, Haskell supports functional programming), while other programming languages support multiple paradigms (such as Object Pascal, C++, Java, C#, Scala, Visual Basic, Common Lisp, Scheme, Perl, Python, Ruby, Oz and F#). Many programming paradigms are as well known for what techniques they forbid as for what they enable. For instance, pure functional programming disallows the use of side-effects, while structured programming disallows the use of the goto statement. Partly for this reason, new paradigms are often regarded as doctrinaire or overly rigid by Programming paradigm 216 those accustomed to earlier styles.[2] Avoiding certain techniques can make it easier to prove theorems about a program's correctness—or simply to understand its behavior. Multi-paradigm programming language A multi-paradigm programming language is a programming language that supports more than one programming paradigm. As Leda designer Timothy Budd puts it: "The idea of a multiparadigm language is to provide a framework in which programmers can work in a variety of styles, freely intermixing constructs from different paradigms." The design goal of such languages is to allow programmers to use the best tool for a job, admitting that no one paradigm solves all problems in the easiest or most efficient way. One example is C#, which includes imperative and object-oriented paradigms as well as some support for functional programming through type inference, anonymous functions and Language Integrated Query. Some other ones are F# and Scala, which provides similar functionality to C# but also includes full support for functional programming (including currying, pattern matching, algebraic data types, lazy evaluation, tail recursion, immutability, etc.). Perhaps the most extreme example is Oz, which has subsets that are logic (Oz descends from logic programming), a functional, an object-oriented, a dataflow concurrent, and other language paradigms. Oz was designed over a ten-year period to combine in a harmonious way concepts that are traditionally associated with different programming paradigms. Lisp, while often taught as a functional language, is known for its malleability and thus its ability to engulf many paradigms. History The lowest level programming paradigms are machine code, which directly represents the instructions (the contents of program memory) as a sequence of numbers, and assembly language where the machine instructions are represented by mnemonics and memory addresses can be given symbolic labels. These are sometimes called first- and second-generation languages. In the 1960s assembly languages were developed to support library COPY and quite sophisticated conditional macro generation and pre-processing capabilities, CALL to (subroutines), external variables and common sections (globals), enabling significant code re-use and isolation from hardware specifics via use of logical operators such as READ/WRITE/GET/PUT. Assembly was, and still is, used for time critical systems and frequently in embedded systems as it gives the most direct control of what the machine actually does. The next advance was the development of procedural languages. These third-generation languages (the first described as high-level languages) use vocabulary related to the problem being solved. For example, • C - developed c. 1970 at Bell Labs • COBOL (Common Business Oriented Language) - uses terms like file, move and copy. • FORTRAN (FORmula TRANslation) - using mathematical language terminology, it was developed mainly for scientific and engineering problems. • ALGOL (ALGOrithmic Language) - focused on being an appropriate language to define algorithms, while using mathematical language terminology and targeting scientific and engineering problems just like FORTRAN. • PL/I (Programming Language One) - a hybrid commercial/scientific general purpose language supporting pointers. • BASIC (Beginners All purpose Symbolic Instruction Code) - was developed to enable more people to write programs. All these languages follow the procedural paradigm. That is, they describe, step by step, exactly the procedure that should, according to the particular programmer at least, be followed to solve a specific problem. The efficacy and efficiency of any such solution are both therefore entirely subjective and highly dependent on that programmer's experience, inventiveness and ability. Programming paradigm 217 Later, object-oriented languages (like Simula, Smalltalk, C++, Eiffel and Java) were created. In these languages, data, and methods of manipulating the data, are kept as a single unit called an object. The only way that a user can access the data is via the object's 'methods' (subroutines). Because of this, the internal workings of an object may be changed without affecting any code that uses the object. There is still some controversy by notable programmers such as Alexander Stepanov, Richard Stallman[3] and others, concerning the efficacy of the OOP paradigm versus the procedural paradigm. The necessity of every object to have associative methods leads some skeptics to associate OOP with software bloat. Polymorphism was developed as one attempt to resolve this dilemma. Since object-oriented programming is considered a paradigm, not a language, it is possible to create even an object-oriented assembler language. High Level Assembly (HLA) is an example of this that fully supports advanced data types and object-oriented assembly language programming - despite its early origins. Thus, differing programming paradigms can be thought of as more like 'motivational memes' of their advocates - rather than necessarily representing progress from one level to the next. Precise comparisons of the efficacy of competing paradigms are frequently made more difficult because of new and differing terminology applied to similar (but not identical) entities and processes together with numerous implementation distinctions across languages. Within imperative programming, which is based on procedural languages, an alternative to the computer-centered hierarchy of structured programming is literate programming, which structures programs instead as a human-centered web, as in a hypertext essay – documentation is integral to the program, and the program is structured following the logic of prose exposition, rather than compiler convenience. Independent of the imperative branch, declarative programming paradigms were developed. In these languages the computer is told what the problem is, not how to solve the problem - the program is structured as a collection of properties to find in the expected result, not as a procedure to follow. Given a database or a set of rules, the computer tries to find a solution matching all the desired properties. The archetypical example of a declarative language is the fourth generation language SQL, as well as the family of functional languages and logic programming. Functional programming is a subset of declarative programming. Programs written using this paradigm use functions, blocks of code intended to behave like mathematical functions. Functional languages discourage changes in the value of variables through assignment, making a great deal of use of recursion instead. The logic programming paradigm views computation as automated reasoning over a corpus of knowledge. Facts about the problem domain are expressed as logic formulae, and programs are executed by applying inference rules over them until an answer to the problem is found, or the collection of formulae is proved inconsistent. References [1] Nørmark, Kurt. Overview of the four main programming paradigms (http:/ / people. cs. aau. dk/ ~normark/ prog3-03/ html/ notes/ paradigms_themes-paradigm-overview-section. html). Aalborg University, 9 May 2011. Retrieved 22 September 2012. [2] Frank Rubin published a criticism of Dijkstra's letter in the March 1987 CACM where it appeared under the title 'GOTO Considered Harmful' Considered Harmful. Frank Rubin (March 1987). "'GOTO Considered Harmful' Considered Harmful" (http:/ / www. ecn. purdue. edu/ ParaMount/ papers/ rubin87goto. pdf) (PDF). Communications of the ACM 30 (3): 195–196. doi:10.1145/214748.315722. . [3] "Mode inheritance, cloning, hooks & OOP (Google Groups Discussion)" (http:/ / groups. google. com/ group/ comp. emacs. xemacs/ browse_thread/ thread/ d0af257a2837640c/ 37f251537fafbb03?lnk=st& q="Richard+ Stallman"+ oop& rnum=5& hl=en#37f251537fafbb03). . External links • Classification of the principal programming paradigms (http://www.info.ucl.ac.be/~pvr/paradigms.html) Mathematical model 218 Mathematical model A mathematical model is a description of a system using mathematical concepts and language. The process of developing a mathematical model is termed mathematical modelling. Mathematical models are used not only in the natural sciences (such as physics, biology, earth science, meteorology) and engineering disciplines (e.g. computer science, artificial intelligence), but also in the social sciences (such as economics, psychology, sociology and political science); physicists, engineers, statisticians, operations research analysts and economists use mathematical models most extensively. A model may help to explain a system and to study the effects of different components, and to make predictions about behaviour. Mathematical models can take many forms, including but not limited to dynamical systems, statistical models, differential equations, or game theoretic models. These and other types of models can overlap, with a given model involving a variety of abstract structures. In general, mathematical models may include logical models, as far as logic is taken as a part of mathematics. In many cases, the quality of a scientific field depends on how well the mathematical models developed on the theoretical side agree with results of repeatable experiments. Lack of agreement between theoretical mathematical models and experimental measurements often leads to important advances as better theories are developed. Examples of mathematical models • Many everyday activities carried out without a thought are uses of mathematical models. A geographical map projection of a region of the earth onto a small, plane surface is a model[1] which can be used for many purposes such as planning travel. • Another simple activity is predicting the position of a vehicle from its initial position, direction and speed of travel, using the equation that distance travelled is the product of time and speed. This is known as dead reckoning when used more formally. Mathematical modelling in this way does not necessarily require formal mathematics; animals have been shown to use dead reckoning.[2][3] • Population Growth. A simple (though approximate) model of population growth is the Malthusian growth model. A slightly more realistic and largely used population growth model is the logistic function, and its extensions. • Model of a particle in a potential-field. In this model we consider a particle as being a point of mass which describes a trajectory in space which is modeled by a function giving its coordinates in space as a function of time. The potential field is given by a function V : R3 → R and the trajectory is a solution of the differential equation Note this model assumes the particle is a point mass, which is certainly known to be false in many cases in which we use this model; for example, as a model of planetary motion. • Model of rational behavior for a consumer. In this model we assume a consumer faces a choice of n commodities labeled 1,2,...,n each with a market price p1, p2,..., pn. The consumer is assumed to have a cardinal utility function U (cardinal in the sense that it assigns numerical values to utilities), depending on the amounts of commodities x1, x2,..., xn consumed. The model further assumes that the consumer has a budget M which is used to purchase a vector x1, x2,..., xn in such a way as to maximize U(x1, x2,..., xn). The problem of rational behavior in this model then becomes an optimization problem, that is: subject to: Mathematical model 219 This model has been used in general equilibrium theory, particularly to show existence and Pareto efficiency of economic equilibria. However, the fact that this particular formulation assigns numerical values to levels of satisfaction is the source of criticism (and even ridicule). However, it is not an essential ingredient of the theory and again this is an idealization. • Neighbour-sensing model explains the mushroom formation from the initially chaotic fungal network. • Computer Science: models in Computer Networks, data models, surface model,... • Mechanics: movement of rocket model,... Modeling requires selecting and identifying relevant aspects of a situation in the real world. Significance in the natural sciences Mathematical models are of great importance in the natural sciences, particularly in physics. Physical theories are almost invariably expressed using mathematical models. Throughout history, more and more accurate mathematical models have been developed. Newton's laws accurately describe many everyday phenomena, but at certain limits relativity theory and quantum mechanics must be used, even these do not apply to all situations and need further refinement. It is possible to obtain the less accurate models in appropriate limits, for example relativistic mechanics reduces to Newtonian mechanics at speeds much less than the speed of light. Quantum mechanics reduces to classical physics when the quantum numbers are high. For example the de Broglie wavelength of a tennis ball is insignificantly small, so classical physics is a good approximation to use in this case. It is common to use idealized models in physics to simplify things. Massless ropes, point particles, ideal gases and the particle in a box are among the many simplified models used in physics. The laws of physics are represented with simple equations such as Newton's laws, Maxwell's equations and the Schrödinger equation. These laws are such as a basis for making mathematical models of real situations. Many real situations are very complex and thus modeled approximate on a computer, a model that is computationally feasible to compute is made from the basic laws or from approximate models made from the basic laws. For example, molecules can be modeled by molecular orbital models that are approximate solutions to the Schrödinger equation. In engineering, physics models are often made by mathematical methods such as finite element analysis. Different mathematical models use different geometries that are not necessarily accurate descriptions of the geometry of the universe. Euclidean geometry is much used in classical physics, while special relativity and general relativity are examples of theories that use geometries which are not Euclidean. Some applications Since prehistorical times simple models such as maps and pre-designed diagrams have been used. Often when engineers analyze a system to be controlled or optimized, they use a mathematical model. In analysis, engineers can build a descriptive model of the system as a hypothesis of how the system could work, or try to estimate how an unforeseeable event could affect the system. Similarly, in control of a system, engineers can try out different control approaches in simulations. A mathematical model usually describes a system by a set of variables and a set of equations that establish relationships between the variables. Variables may be of many types; real or integer numbers, boolean values or strings, for example. The variables represent some properties of the system, for example, measured system outputs often in the form of signals, timing data, counters, and event occurrence (yes/no). The actual model is the set of functions that describe the relations between the different variables. Mathematical model 220 Building blocks In business and engineering, mathematical models may be used to maximize a certain output. The system under consideration will require certain inputs. The system relating inputs to outputs depends on other variables too: decision variables, state variables, exogenous variables, and random variables. Decision variables are sometimes known as independent variables. Exogenous variables are sometimes known as parameters or constants. The variables are not independent of each other as the state variables are dependent on the decision, input, random, and exogenous variables. Furthermore, the output variables are dependent on the state of the system (represented by the state variables). Objectives and constraints of the system and its users can be represented as functions of the output variables or state variables. The objective functions will depend on the perspective of the model's user. Depending on the context, an objective function is also known as an index of performance, as it is some measure of interest to the user. Although there is no limit to the number of objective functions and constraints a model can have, using or optimizing the model becomes more involved (computationally) as the number increases. For example, in economics students often apply linear algebra when using input-output models. Complicated mathematical models that have many variables may be consolidated by use of vectors where one symbol represents several variables. Classifying mathematical models Many mathematical models can be classified in some of the following ways: 1. Linear vs. nonlinear: Mathematical models are usually composed by variables, which are abstractions of quantities of interest in the described systems, and operators that act on these variables, which can be algebraic operators, functions, differential operators, etc. If all the operators in a mathematical model exhibit linearity, the resulting mathematical model is defined as linear. A model is considered to be nonlinear otherwise. The question of linearity and nonlinearity is dependent on context, and linear models may have nonlinear expressions in them. For example, in a statistical linear model, it is assumed that a relationship is linear in the parameters, but it may be nonlinear in the predictor variables. Similarly, a differential equation is said to be linear if it can be written with linear differential operators, but it can still have nonlinear expressions in it. In a mathematical programming model, if the objective functions and constraints are represented entirely by linear equations, then the model is regarded as a linear model. If one or more of the objective functions or constraints are represented with a nonlinear equation, then the model is known as a nonlinear model. Nonlinearity, even in fairly simple systems, is often associated with phenomena such as chaos and irreversibility. Although there are exceptions, nonlinear systems and models tend to be more difficult to study than linear ones. A common approach to nonlinear problems is linearization, but this can be problematic if one is trying to study aspects such as irreversibility, which are strongly tied to nonlinearity. 2. Deterministic vs. probabilistic (stochastic): A deterministic model is one in which every set of variable states is uniquely determined by parameters in the model and by sets of previous states of these variables. Therefore, deterministic models perform the same way for a given set of initial conditions. Conversely, in a stochastic model, randomness is present, and variable states are not described by unique values, but rather by probability distributions. 3. Static vs. dynamic: A static model does not account for the element of time, while a dynamic model does. Dynamic models typically are represented with difference equations or differential equations. 4. Discrete vs. Continuous: A discrete model does not take into account the function of time and usually uses time-advance methods, while a Continuous model does. Continuous models typically are represented with f(t) and the changes are reflected over continuous time intervals. Mathematical model 221 5. Deductive, inductive, or floating: A deductive model is a logical structure based on a theory. An inductive model arises from empirical findings and generalization from them. The floating model rests on neither theory nor observation, but is merely the invocation of expected structure. Application of mathematics in social sciences outside of economics has been criticized for unfounded models.[4] Application of catastrophe theory in science has been characterized as a floating model.[5] A priori information Mathematical modeling problems are used often classified into black box or white box models, according to how much a priori information is available of the system. A black-box model is a system of which there is no a priori information available. A white-box model (also called glass box or clear box) is a system where all necessary information is available. Practically all systems are somewhere between the black-box and white-box models, so this concept is useful only as an intuitive guide for deciding which approach to take. Usually it is preferable to use as much a priori information as possible to make the model more accurate. Therefore the white-box models are usually considered easier, because if you have used the information correctly, then the model will behave correctly. Often the a priori information comes in forms of knowing the type of functions relating different variables. For example, if we make a model of how a medicine works in a human system, we know that usually the amount of medicine in the blood is an exponentially decaying function. But we are still left with several unknown parameters; how rapidly does the medicine amount decay, and what is the initial amount of medicine in blood? This example is therefore not a completely white-box model. These parameters have to be estimated through some means before one can use the model. In black-box models one tries to estimate both the functional form of relations between variables and the numerical parameters in those functions. Using a priori information we could end up, for example, with a set of functions that probably could describe the system adequately. If there is no a priori information we would try to use functions as general as possible to cover all different models. An often used approach for black-box models are neural networks which usually do not make assumptions about incoming data. The problem with using a large set of functions to describe a system is that estimating the parameters becomes increasingly difficult when the amount of parameters (and different types of functions) increases. Subjective information Sometimes it is useful to incorporate subjective information into a mathematical model. This can be done based on intuition, experience, or expert opinion, or based on convenience of mathematical form. Bayesian statistics provides a theoretical framework for incorporating such subjectivity into a rigorous analysis: one specifies a prior probability distribution (which can be subjective) and then updates this distribution based on empirical data. An example of when such approach would be necessary is a situation in which an experimenter bends a coin slightly and tosses it once, recording whether it comes up heads, and is then given the task of predicting the probability that the next flip comes up heads. After bending the coin, the true probability that the coin will come up heads is unknown, so the experimenter would need to make an arbitrary decision (perhaps by looking at the shape of the coin) about what prior distribution to use. Incorporation of the subjective information is necessary in this case to get an accurate prediction of the probability, since otherwise one would guess 1 or 0 as the probability of the next flip being heads, which would be almost certainly wrong.[6] Mathematical model 222 Complexity In general, model complexity involves a trade-off between simplicity and accuracy of the model. Occam's razor is a principle particularly relevant to modeling; the essential idea being that among models with roughly equal predictive power, the simplest one is the most desirable. While added complexity usually improves the realism of a model, it can make the model difficult to understand and analyze, and can also pose computational problems, including numerical instability. Thomas Kuhn argues that as science progresses, explanations tend to become more complex before a Paradigm shift offers radical simplification. For example, when modeling the flight of an aircraft, we could embed each mechanical part of the aircraft into our model and would thus acquire an almost white-box model of the system. However, the computational cost of adding such a huge amount of detail would effectively inhibit the usage of such a model. Additionally, the uncertainty would increase due to an overly complex system, because each separate part induces some amount of variance into the model. It is therefore usually appropriate to make some approximations to reduce the model to a sensible size. Engineers often can accept some approximations in order to get a more robust and simple model. For example Newton's classical mechanics is an approximated model of the real world. Still, Newton's model is quite sufficient for most ordinary-life situations, that is, as long as particle speeds are well below the speed of light, and we study macro-particles only. Training Any model which is not pure white-box contains some parameters that can be used to fit the model to the system it is intended to describe. If the modeling is done by a neural network, the optimization of parameters is called training. In more conventional modeling through explicitly given mathematical functions, parameters are determined by curve fitting. Model evaluation A crucial part of the modeling process is the evaluation of whether or not a given mathematical model describes a system accurately. This question can be difficult to answer as it involves several different types of evaluation. Fit to empirical data Usually the easiest part of model evaluation is checking whether a model fits experimental measurements or other empirical data. In models with parameters, a common approach to test this fit is to split the data into two disjoint subsets: training data and verification data. The training data are used to estimate the model parameters. An accurate model will closely match the verification data even though these data were not used to set the model's parameters. This practice is referred to as cross-validation in statistics. Defining a metric to measure distances between observed and predicted data is a useful tool of assessing model fit. In statistics, decision theory, and some economic models, a loss function plays a similar role. While it is rather straightforward to test the appropriateness of parameters, it can be more difficult to test the validity of the general mathematical form of a model. In general, more mathematical tools have been developed to test the fit of statistical models than models involving differential equations. Tools from non-parametric statistics can sometimes be used to evaluate how well the data fit a known distribution or to come up with a general model that makes only minimal assumptions about the model's mathematical form. Mathematical model 223 Scope of the model Assessing the scope of a model, that is, determining what situations the model is applicable to, can be less straightforward. If the model was constructed based on a set of data, one must determine for which systems or situations the known data is a "typical" set of data. The question of whether the model describes well the properties of the system between data points is called interpolation, and the same question for events or data points outside the observed data is called extrapolation. As an example of the typical limitations of the scope of a model, in evaluating Newtonian classical mechanics, we can note that Newton made his measurements without advanced equipment, so he could not measure properties of particles travelling at speeds close to the speed of light. Likewise, he did not measure the movements of molecules and other small particles, but macro particles only. It is then not surprising that his model does not extrapolate well into these domains, even though his model is quite sufficient for ordinary life physics. Philosophical considerations Many types of modeling implicitly involve claims about causality. This is usually (but not always) true of models involving differential equations. As the purpose of modeling is to increase our understanding of the world, the validity of a model rests not only on its fit to empirical observations, but also on its ability to extrapolate to situations or data beyond those originally described in the model. One can argue that a model is worthless unless it provides some insight which goes beyond what is already known from direct investigation of the phenomenon being studied. An example of such criticism is the argument that the mathematical models of Optimal foraging theory do not offer insight that goes beyond the common-sense conclusions of evolution and other basic principles of ecology.[7] References [1] landinfo.com, definition of map projection (http:/ / www. landinfo. com/ resources_dictionaryMP. htm) [2] Gallistel. The Organization of Learning. 1990. [3] Dead reckoning (path integration) requires the hippocampal formation: evidence from spontaneous exploration and spatial learning tasks in light (allothetic) and dark (idiothetic) tests, IQ Whishaw, DJ Hines, DG Wallace, Behavioural Brain Research 127 (2001) 49 – 69 (http:/ / hsinnamon. web. wesleyan. edu/ wescourses/ NSB-Psyc255/ Readings/ 17. Spatial Limbic System/ Whishaw. pdf) [4] Stanislav Andreski (1972) Social Sciences as Sorcery, St. Martin’s Press [5] Clifford Truesdell (1984) An Idiot’s Fugitive Essays on Science, 121–7, Springer ISBN 3-540-90703-3 [6] MacKay, D.J. Information Theory, Inference, and Learning Algorithms, Cambridge, (2003-2004). ISBN 0-521-64298-1 [7] "Optimal Foraging Theory: A Critical Review - Annual Review of Ecology and Systematics, 15(1):523 - First Page Image" (http:/ / arjournals. annualreviews. org/ doi/ abs/ 10. 1146/ annurev. es. 15. 110184. 002515?journalCode=ecolsys). Arjournals.annualreviews.org. 2003-11-28. . Retrieved 2011-03-27. Further reading Books • Aris, Rutherford [ 1978 ] ( 1994 ). Mathematical Modelling Techniques, New York : Dover. ISBN 0-486-68131-9 • Bender, E.A. [ 1978 ] ( 2000 ). An Introduction to Mathematical Modeling, New York : Dover. ISBN 0-486-41180-X • Lin, C.C. & Segel, L.A. ( 1988 ). Mathematics Applied to Deterministic Problems in the Natural Sciences, Philadelphia : SIAM. ISBN 0-89871-229-7 • Gershenfeld, N. (1998) The Nature of Mathematical Modeling, Cambridge University Press ISBN 0-521-57095-6 . • Yang, X.-S. (2008) Mathematical Modelling for Earth Sciences, Dunedin Academic, ISBN 1-903765-92-7 . Specific applications • Peierls, Rudolf (January 1980) Model-making in physics (http://www.informaworld.com/smpp/ content~content=a752582770~db=all~order=page), Contemporary Physics Volume 21(1): 3–17. Mathematical model 224 • Korotayev A., Malkov A., Khaltourina D. (2006). Introduction to Social Macrodynamics: Compact Macromodels of the World System Growth (http://cliodynamics.ru/index.php?option=com_content&task=view&id=124& Itemid=70). Moscow : Editorial URSS (http://urss.ru/cgi-bin/db.pl?cp=&lang=en&blang=en&list=14& page=Book&id=34250) ISBN 5-484-00414-4 . External links General reference material • McLaughlin, Michael P. ( 1999 ) 'A Tutorial on Mathematical Modeling' (http://www.causascientia.org/ math_stat/Tutorial.pdf) PDF (264 KiB) • Patrone, F. Introduction to modeling via differential equations (http://www.fioravante.patrone.name/mat/u-u/ en/differential_equations_intro.htm), with critical remarks. • Plus teacher and student package: Mathematical Modelling. (http://plus.maths.org/issue44/package/index. html) Brings together all articles on mathematical modeling from Plus, the online mathematics magazine produced by the Millennium Mathematics Project at the University of Cambridge. Philosophical background • Frigg, R. and S. Hartmann, Models in Science (http://plato.stanford.edu/entries/models-science/), in: The Stanford Encyclopedia of Philosophy, (Spring 2006 Edition) • Griffiths, E. C. (2010) What is a model? (http://www.emily-griffiths.postgrad.shef.ac.uk/models.pdf) Simulation modeling Simulation modeling is the process of creating and analyzing a digital prototype of a physical model to predict its performance in the real world. Simulation modeling is used to help designers and engineers understand whether, under what conditions, and in which ways a part could fail and what loads it can withstand. Simulation modeling can also help predict fluid flow and heat transfer patterns. Uses of Simulation Modeling Simulation modeling allows designers and engineers to avoid repeated building of multiple physical prototypes to analyze designs for new or existing parts. Before creating the physical prototype, users can virtually investigate many digital prototypes. Using the technique, they can: • Optimize geometry for weight and strength • Select materials that meet weight, strength, and budget requirements • Simulate part failure and identify the loading conditions that cause them • Assess extreme environmental conditions or loads not easily tested on physical prototypes, such as earthquake shock load • Verify hand calculations • Validate the likely safety and survival of a physical prototype before testing Simulation modeling 225 Typical Simulation Modeling Workflow Simulation modeling follows a process much like this: 1. Use a 2D or 3D CAD tool to develop a virtual model, also known as a digital prototype, to represent a design. 2. Generate a 2D or 3D mesh for analysis calculations. Automatic algorithms can create finite element meshes, or users can create structured meshes to maintain control over element quality. 3. Define finite element analysis data (loads, constraints, or materials) based on analysis type (thermal, structural, or fluid). Apply boundary conditions to the model to represent how the part will be restrained during use. 4. Perform finite element analysis, review results, and make engineering judgments based on results. Simulation Modeling Software Programs • Abaqus • ANSYS • Autodesk Algor Simulation • Autodesk Inventor Professional • Nastran • Solidworks Simulation References • The CBS Interactive Business Network [1] • University of Central Florida, Institute for Simulation and Training [2] • Winsberg, Eric (2003), Simulated Experiments: Methodology for a Virtual World [3] • Roger D. Smith: "Simulation: The Engine Behind the Virtual World" [4], eMatter, December, 1999 External links • Institute for Simulation and Training, University of Central Florida [5] • National Center for Simulation [6] • Simulation Interoperability Standards Organization [7] • The Society for Modeling and Simulation International (Formerly the Society of Computer Simulation) [8] References [1] http:/ / www. bnet. com/ topics/ simulation+ model [2] http:/ / www. ist. ucf. edu/ background. htm [3] http:/ / www. cas. usf. edu/ ~ewinsb/ methodology. pdf [4] http:/ / www. modelbenders. com/ Bookshop/ techpapers. html [5] http:/ / www. ist. ucf. edu [6] http:/ / www. simulationinformation. com [7] http:/ / www. sisostds. org [8] http:/ / www. scs. org Dynamical system 226 Dynamical system A dynamical system is a concept in mathematics where a fixed rule describes the time dependence of a point in a geometrical space. Examples include the mathematical models that describe the swinging of a clock pendulum, the flow of water in a pipe, and the number of fish each springtime in a lake. At any given time a dynamical system has a state given by a set of real numbers (a vector) that can be represented by a point in an appropriate state space (a geometrical manifold). Small changes in the state of the system create small changes in the numbers. The evolution rule of the dynamical system is a fixed rule that describes what future states follow from the current state. The rule is deterministic; in other words, for a given time interval only one future state follows from the current state. The Lorenz attractor arises in the study of the Lorenz Oscillator, a dynamical system. Overview The concept of a dynamical system has its origins in Newtonian mechanics. There, as in other natural sciences and engineering disciplines, the evolution rule of dynamical systems is given implicitly by a relation that gives the state of the system only a short time into the future. (The relation is either a differential equation, difference equation or other time scale.) To determine the state for all future times requires iterating the relation many times—each advancing time a small step. The iteration procedure is referred to as solving the system or integrating the system. Once the system can be solved, given an initial point it is possible to determine all its future positions, a collection of points known as a trajectory or orbit. Before the advent of fast computing machines, solving a dynamical system required sophisticated mathematical techniques and could be accomplished only for a small class of dynamical systems. Numerical methods implemented on electronic computing machines have simplified the task of determining the orbits of a dynamical system. For simple dynamical systems, knowing the trajectory is often sufficient, but most dynamical systems are too complicated to be understood in terms of individual trajectories. The difficulties arise because: • The systems studied may only be known approximately—the parameters of the system may not be known precisely or terms may be missing from the equations. The approximations used bring into question the validity or relevance of numerical solutions. To address these questions several notions of stability have been introduced in the study of dynamical systems, such as Lyapunov stability or structural stability. The stability of the dynamical system implies that there is a class of models or initial conditions for which the trajectories would be equivalent. The operation for comparing orbits to establish their equivalence changes with the different notions of stability. • The type of trajectory may be more important than one particular trajectory. Some trajectories may be periodic, whereas others may wander through many different states of the system. Applications often require enumerating these classes or maintaining the system within one class. Classifying all possible trajectories has led to the qualitative study of dynamical systems, that is, properties that do not change under coordinate changes. Linear dynamical systems and systems that have two numbers describing a state are examples of dynamical systems where the possible classes of orbits are understood. • The behavior of trajectories as a function of a parameter may be what is needed for an application. As a parameter is varied, the dynamical systems may have bifurcation points where the qualitative behavior of the dynamical system changes. For example, it may go from having only periodic motions to apparently erratic behavior, as in Dynamical system 227 the transition to turbulence of a fluid. • The trajectories of the system may appear erratic, as if random. In these cases it may be necessary to compute averages using one very long trajectory or many different trajectories. The averages ar