"Experimental Design and Scientific Method"
Experimental Design and Scientific Method Dr Andy Harvey School of Engineering and Physical Sciences Heriot Watt University, Edinburgh, EH14 4AS Tel. 0131 451 3356 firstname.lastname@example.org Aims and Objectives • Aim • To improve the scientific usefulness of your project • Objectives – To be able to apply the scientific method to your project – To be able to understand how to construct hypotheses • and/or objectives – Understand how to test the hypothesis • or objective Overview • Hypothesis and objectives • Hypothesis testing • Objectivity • Experimental design • An exercise – Critique an experimental design • There are many types of projects – a universal and all-pervasive recipe for success is not possible – you are responsible for applying the principles of scientific method to your project • However all experiments are (or should be) typified by • Hypotheses and/or objectives • testing of hypothesis and objectives • objectivity • sound and logical conclusions based on the experimental evidence Types of project • Observation – Examples • characterisation of spatial frequencies in an image • characterisation of delays in a computer network – but observation is not science; science involves some degree of increased understanding, from, for example, generalisation. – For example • image spatial frequencies obey a 1/f law • delays in a computer network increase with data traffic • Design and demonstration of a new technique or system – A new measurement or imaging technique – A new software computer interface • Development of a computer model of phenomena, technique or system. This is often a risk reduction exercise prior to experimentation – a computer model of an imaging system – a computer model of data communication within a network What is the Scientific Method ? • The repeated testing of a hypothesis against empirical evidence • In the face of contradictory evidence the hypothesis is either – Refined or – Rejected • The hypothesis is generally only accepted when – (a) evidence generally supports it or preferably – (b) repeated attempts to disprove it have failed • For example – Around 1900 there was a hypothesis that the Earth moved through a stationary ether. – Michelson and Morley conducted an experiment with the objective of measuring the „drag‟ of the ether on light. They failed to measure an effect and the hypothesis of the ether had to be rejected. – This led Einstein to propose special relativity as an improved hypothesis. • This hypothesis has been tested many times and its predictions have been found to be consistent with observation. • Special relativity has therefore been accepted as a sound scientific theory. – This is an example of how the scientific method leads to an improved understanding of the world What is a hypothesis/objective? • Oxford Dictionary: – Hypothesis: “An idea or suggestion that is based on known facts and is used as a basis for reasoning or further investigation.” • Statement of explanation or prediction • Examples – All swans are white – Women are more careful drivers than men • In science and technology it is common to use objectives in place of hypotheses – The objective of my computer code is to calculate the point spread function (impulse response) of an imaging system that is within 1% of the measured value – The objective of my spectrometer will enable measurement of spectral lines with an amplitude accuracy of 1% and a frequency resolution of 1 GHz. – Like a hypothesis, an objective should be measurable - was it achieved or not ? What is a good hypothesis ? • The hypothesis should be well-founded and informative – „Santa Claus lives on the dark side of the moon‟ • is not well-founded since there is no evidence to support it – „a computer is faster than an abacus‟ • is uninformative – „computer analysis of cervical smears is more efficient at detection of abnormalities than analysis by a person‟ • is a good hypothesis • A hypothesis/objective should be testable, preferably by objective criteria – Popper tells us that it is more useful to attempt to disprove a hypothesis. If we are unable to disprove the hypothesis, then this is the best ground for accepting its truth. • As humans, objective and rigorous testing of hypothesis is often weak or missing…. Hypothesis Joke • The hypothesis should be well-founded and informative – A Philosopher, an Engineer, a Physicist and a Mathematician travel by train to Scotland. On seeing a field of sheep though the window of the train: • Philosopher on seeing a single sheep: “Sheep are white” • Engineer on seeing field of sheep: “Sheep in Scotland are white” • Physicist: “Sheep in that field are white” • Mathematician: “Sheep in that field are white and at least one side of each sheep is white” – Which is the most useful hypothesis • It needn‟t be the only correct one Hypothesis:“men are better drivers than women” – Arguments commonly used to for/against this hypothesis ? • “women don‟t understand cars and therefore can‟t drive as well” • “men are more aggressive and are therefore more likely to have accidents” – These are simply unproven hypotheses. The scientific method requires that we test the original hypothesis with an objective measure • an objective measure is a fundamental feature of the scientific method – Testable hypothesis: • “men have fewer driving accidents than women” – Observation: • men have three times as many accidents as women – but, on average, men drive more miles than women – Refine hypothesis to • “Men have fewer accidents per thousand miles” – The observation • Women: 0.027 accidents per 1000 miles . • Men: 0.039 accidents per 1000 miles – What is the conclusion ? • If we have controlled all other variables, this evidence refutes the hypothesis that men are safer drivers than women. Your project should have a well- founded hypotheses or objective and it should be testable • For example – To develop a computer model that will predict the imaging performance of a lens with an accuracy of 1% Testing a hypothesis/objective • Use objective criteria – measurements – comparisons • quartz is harder than slate • 8 out of 10 people preferred ... – It is often possible to devise objective criteria in very difficult circumstances • Measurements of happiness • Sound quality of a hi-fi • Handling of a car • Beauty • A common error is to try to prove the hypothesis by (subconsciously ?) selecting favourable data set – the focus of scientific method is testing not proving – it is more useful to attempt to disprove the hypothesis and fail than to attempt to prove the hypothesis and succeed (Popper) A simplified example of testing (the hypothesis) that the code sample yields correct results for all arguments) • Consider that you would like to use the following simple function (sinc): • How do I test that the output is as expected ? – Plot it – Checking that it gives zeros when x=1,2,3.. • Or, preferably I could search for examples where the function fails ... • Which approach is most informative ? Control variables and modify in a controlled manner • change as few variables at a time as possible – use controls • test/validate the effect of each item as it is introduced – computer code and equipment is assembled and tested incrementally – experimental equipment is validated/calibrated • Example errors – Challenger space shuttle disaster – Cold fusion – Arianne space shuttle Use objective recording • Where feasible automate data collection – human recording tends to make mistakes and statistically these mistakes tend to favour the preferred hypothesis – use a digital camera to record experimental setups – write detailed, precise and unequivocal notes • Record data in a log book as it is recorded – recording at a later time introduces increased subjectivity • Think about the results as you progress – write up your project as you progress Invite constructive criticism • All humans – make mistakes • they are often only recognised in retrospect – are not adept at criticising their own ideas • An error at the beginning of an experiment can render the experiment useless • Invite constructive criticism of your proposed experiment and analysis from your supervisor/colleagues – at an early stage – throughout the project An example exercise • A new spectral imager was invented that would record multiple spectral images on a single detector where each image is a „pure colour‟. • This involved the use of waveplates that should be oriented at exactly 45° sandwiched between polarisers. • If the angle of the waveplate is not almost exactly 45° the colour would not be pure and the device would be useless. • The researcher realised that when the waveplates were aligned at 45° the efficiency of light transmission into two spots would be equal • One of these spots was red, the other blue • S/he rotated the waveplate until the red and blue spots appeared to be of identical brightness to the eye and assumed that this corresponded to 45 ° • S/he proceeded to assemble all 5 sets of waveplate/polariser assemblies and then test the complete system. This took one month. • The researcher conducted above work without consultation • Exhaustive testing showed that – the colours were not at all pure – the image quality was very poor due to poor quality polarisers • One month of effort was wasted and the experiment had to be repeated properly • what mistakes in scientific methodology did the researcher make and how would you design the experiment • Conceptual error – Assumed equal light intensities is equivalent to equal transmission • how much red/blue light is emitted by the source • what is the relative sensitivity to red/ blue light • The method errors – insufficient self criticism of proposal • did s/he ask under what circumstances might it not work • what would be the cost if it does not work – (wasted month of effort) – no discussion (and constructive criticism) with others – No objective measurements of light intensity • use a camera interfaced to a computer to measure intensities – No prediction and measurement of effect for each assembled component • All variables introduced simultaneously • How accurately must wave plates be aligned Conclusions • Define an objective/hypothesis – predict what you expect • Designing an objective testing mechanism • Self criticise and invite constructive criticism • Conduct some trial investigative experiments • Test whether the objectives have been met • Draw conclusions based on the evidence obtained