"Excavator Model Uncertainty Analysis"
Excavator Model Uncertainty Analysis ME6105: Modeling and Simulation in Design Submitted to Dr. Chris Paredis March 29th, 2007 Group Members: Rashid Enahora, Thomas Johnson, Ivan Lee, Dan Sankar Table of Contents Task 1: Become Familiar with ModelCenter.................................................................................. 3 Task 2: Explore Monte-Carlo Simulation....................................................................................... 3 Task 3: Determine Main Effects ..................................................................................................... 5 Task 4: Compare the Convergence of MC and LHS ...................................................................... 7 Task 5: Identify and Model the Uncertainty in the Design Analysis Model .................................. 9 Task 6: Determine the Expectation and Standard Deviation of the Model Output ...................... 21 Task 7: Perform a Sensitivity Analysis......................................................................................... 24 Task 8: Lessons Learned............................................................................................................... 26 Rashid Enahora: .................................................................................................................... 26 Tommy Johnson:................................................................................................................... 27 Ivan Lee: ............................................................................................................................... 27 Dan Sankar:........................................................................................................................... 28 2 Task 1: Become Familiar with ModelCenter To begin this assignment, our group installed the Arena and ModelCenter software packages and familiarized ourselves with the software by completing the “Getting Started Guide”. We then installed and tested the LHS driver. Afterwards, we wrapped our Modelica excavator model by placing the dymosim.exe and dsin.txt.template files in a new folder named Excavator. We finished by creating the Excavator.fileWrapper in order to analyze the uncertain variables in our excavator Modelica model. Task 2: Explore Monte-Carlo Simulation The following equation was used to generate a simple model using the Monte Carlo method: a3( x 2 + 2)3 ( x 2 / x1 + a 2) y = a1( x1 − a 2) 2 + + a4 (1) a1 The following are the probability distributions for the inputs: x1 is distributed as Normal(5,1) x2 is distributed as Normal(-3,0.5) a1 is distributed as Triangular(11,12,15) a2 is distributed as Uniform(4,6) a3 is distributed as Weibull(1,2) a4 is distributed as Weibull(5,0.5) The histogram shown in Figure 1 is the result of this Monte Carlo probabilistic analysis for 10,000 trials. 3 Figure 1. Histogram and statistics of the Monte Carlo model with 10,000 trials. As can be seen in the figure above, the mean of the simulation was 25.9235 and the standard deviation was 31.2628. The confidence interval expression ⎛ σ ⎞ μ ± zα / 2 ⎜ ⎟ (2) ⎝ n⎠ where μ was the sample mean, σ was the sample standard deviation, n was the number of required samples, and zα/2 =1.645 was a value used for a confidence 95%. Knowing the standard deviation and zα/2, the equation ⎛ σ ⎞ zα / 2 ⎜ ⎟ = ±1 (3) ⎝ n⎠ was solved for n. The required number of trials was found to be 2645. 4 Task 3: Determine Main Effects Using the DOE tool, a level 3 Full Factorial experiment was conducted as shown in Figure 2. The upper and lower bounds were taken from the 10th and 90th percentiles for each design variable of the experiment conducted in Task 2. Figure 2. Level 3 Full Factorial experiment The main effects graph is shown in Figure 3. The variable a4 holds the most effect at 76%. This makes sense because there is a wider range between the 10th and 90th percentiles for a4. 5 Figure 3. Main effects graph for the level 3 Full Factorial experiment A Central Composite experiment was also conducted using the same design variables and their 10th and 90th percentile bounds. The main effects graph is shown in Figure 4. 6 Figure 4. Main effects graph for Central Composite experiment. Both experiments supplied very similar results. The main differences between the two experiments exist in the form of a discrepancy between the effects of the variables a4 and a1. The full factorial experiment suggested that a4 had a greater effect on the response of the equation. The central composite experiment suggested that variable a1 had an increased impact on the response, lessening the impact of a4. Task 4: Compare the Convergence of MC and LHS The convergences of the Latin Hypercube Sampling (LHS) and Monte Carlo (MC) methods were investigated over the range from 1 to 100 trial runs in increments of 1 trial run. Figure 5 illustrates the convergence of the MC simulation. 7 Figure 5. MC simulation convergene upon the expected values of the response variables. Figure 6 illustrates the convergence of the LHS simulation. 8 Figure 6. LHS simulation convergence upon the model’s expectation and the standard deviation. The convergences of the LHS and MC simulations are very close. Both simulations appear to converge upon the expectations after about 10 to 20 trials. The LHS converges slightly quicker than the MC, but seems to behave in a more erratic manner. We expected the LHS to perform better since it is supposed to be a more behaved simulation Task 5: Identify and Model the Uncertainty in the Design Analysis Model The purpose of the Modelica model is to allow us to determine under uncertainty the values for two measures of effectiveness: the energy output from the variable displacement pump and the time required to complete a dig cycle. The uncertainty in the model is contributed by every 9 variable included in the model. However, accounting for the uncertainty supplied by each variable would be a grueling endeavor. The large number of variables would require the elicitation of thousands of uncertainty distributions. Alternatively, we approached the problem by identifying eleven variables which we believed to contribute most significantly to the model’s uncertainty. In a first attempt at eliciting beliefs about the uncertain variables, each variable was assigned an initial, triangular distribution. The variables and their corresponding distributions can be viewed in Table 1. Table 1. Initial uncertain variables and their corresponding, triangular uncertainty distributions. Triangular Distributions Uncertain Variables Lower Bound Center Upper Bound Terrain Damping 1000 10000 40000 (N/(m/s)) Terrain Stiffness 1000 10000 40000 (N/m) Swing Motor 0.005 0.03 0.06 Displacement (m3) Bucket Cylinder 0.0013 0.005 0.0154 Piston Area A(m2) Bucket Cylinder 0.0012 0.0043 0.0134 Piston Area B (m2) Arm Cylinder 0.0013 0.005 0.0154 Piston Area A (m2) Arm Cylinder 0.0012 0.0043 0.0134 Piston Area B (m2) Boom Cylinder 0.002 0.0079 0.0177 Piston Area A (m2) Boom Cylinder 0.0016 0.0071 0.0157 Piston Area B (m2) Pump Maximum 0.02 0.04 0.07 Displacement (m3) Maintained Pressure 1e6 4e6 8e6 Difference (N/m2) In order to understand which variables in Table 1 most contributed to the uncertainty of the model, we performed a main effects study using a central composite experiment in ModelCenter. However, using all of the identified variables in a central composite experiment required far too many simulation runs. In order to overcome this problem, we cropped out four variables from the 10 study, namely: the bucket cylinder piston area B (the piston area minus the rod area), arm cylinder piston area B, boom cylinder piston area B, and maintained pressure difference. In order to account for the missing information about the area of the piston minus the area of the rod, we set the rod diameter to a constant value in the Modelica model. Hence, the experiment’s design variables were displacement of the swing motor, the maximum displacement of the pump, the damping and stiffness of the terrain, and the piston areas of the four hydraulic cylinders. The ouput variables were the cycle time and the output energy. Figure 7 below shows the main effects on the cycle time. Figure 7. The main effects on the cycle time of the excavator Modelica model. Figure 8 below shows the main effects on the output energy. 11 Figure8. The main effects on the output energy of the excavator Modelica model. We interpreted from Figures7 and 8 that the four most important effects were the three piston areas and the maximum displacement of the variable displacement pump. These, interestingly enough, are four of the five design variables (the fifth being the motor displacement) which we will be optimizing in the next homework assignment. The piston area of the arm had the largest effect on both measures of effectiveness. This was likely due to the large movement created by the arm’s cylinder while the bucket was in the terrain. The other actuators made little or small movements while the bucket was in the terrain. Hence, they had a lesser influence on the measures of effectiveness. In contrast to our initial beliefs, the terrain parameters had little influence on the cycle time and output energy. This may be a product of overly conservative upper and lower bounds for the parameters. Little data was available concerning common values of soil stiffness and damping, so the upper and lower bounds may need to be reevaluated. 12 With the four most important effects identified, we elicited more detailed cumulative uncertainty distributions for the variables. When eliciting the distributions, we recorded our beliefs for parameter values at cumulative percentages that were broken up into five percent intervals. In order to determine a value, we asked ourselves the following question: “What value of this parameter contains this percentage of all of the possible values?” For example, when computing the distribution for the pump displacement, Tommy believed that the range of 0 to 0.064 m3 contained 52.5% of the possible displacement values. Tommy’s elicitation can be viewed Figure 9 below. 100 90 80 70 Cumulative Percentage 60 50 40 30 20 10 0 0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 3 Maximum Pump Displacement (m ) Figure 9. Cumulative distribution of the possible values of maximum pump displacement. Using the Input Analyzer from Arena 7.0, this cumulative distribution was fit to a probability function represented by a normal distribution with a mean of 0.0628 m3 and a standard deviation of 0.0214 m3. The results from the Input Analyzer can be seen below in Figure 10. 13 Figure 10. Input Analyzer results for the maximum pump displacement. Rashid elicited his beliefs about the bucket cylinder’s piston area in a similar manner. His cumulative distribution can be seen below in Figure 11. 14 100 90 80 70 Cumulative Percentage 60 50 40 30 20 10 0 0 0.005 0.01 0.015 0.02 0.025 0.03 2 Piston Area of the Bucket Cylinder (m ) Figure 11. Cumulative distribution of the possible values of bucket cylinder’s piston area. Using the Input Analyzer from Arena 7.0, this cumulative distribution was fit to a probability function represented by a Weibull distribution with a scale of 0.0628 m2 and a shape of 2.3. The results from the Input Analyzer can be seen below in Figure 12. 15 Figure 12. Input Analyzer results for the bucket cylinder’s piston area. Ivan elicited his beliefs about the distribution of the piston area of the boom cylinder. His cumulative distribution can be viewed below in Figure 13. 16 100 90 80 70 Cumulative Percentage 60 50 40 30 20 10 0 0.007 0.009 0.011 0.013 0.015 2 Piston Area of the Boom Cylinders (m ) Figure 13. Cumulative distribution of the possible values of bucket cylinder’s piston area. Using the Input Analyzer from Arena 7.0, this cumulative distribution was fit to a probability function represented by a normal distribution with a mean of 0.012 m2 and a standard deviation of 0.00209 m2. The results from the Input Analyzer can be seen below in Figure 14. 17 Figure 14. Input Analyzer results for the boom cylinder’s piston area. Dan elicited his beliefs about the distribution of the piston area of the arm cylinder. His cumulative distribution can be viewed below in Figure 15. 18 100 90 80 70 Cumulative Percentage 60 50 40 30 20 10 0 0 0.005 0.01 0.015 0.02 0.025 0.03 2 Piston Area of the Arm Cylinder (m ) Figure 15. Cumulative distribution of the possible values of arm cylinder’s piston area. Using the Input Analyzer from Arena 7.0, this cumulative distribution was fit to a probability function represented by a Weibull distribution with a scale of 0.0145 m2 and a shape of 1.56. The results from the Input Analyzer can be seen below in Figure x. 19 Figure 15. Input Analyzer results for the arm cylinder’s piston area. 20 Task 6: Determine the Expectation and Standard Deviation of the Model Output In order to determine the expectation and standard deviation of the model, we first performed a MC simulation with 100 trial runs using the four important design variables and elicited distributions discussed in task 4. However, the simulation times were very long, requiring hour’s long expectation and standard deviation calculations. Knowing that the swing motor displacement did not greatly affect the model outputs, the swing portion of the dig cycle was eliminated such that simulation times could be drastically shortened. Figure 16 below shows a schematic of the uncertainty model developed in ModelCenter. Figure 16. ModelCenter uncertainty model. Figures 17 and 18 below show the results of the MC. 21 Figure 17. Final cycle time MC simulation results. Figure 18. Pump output energy MC simulation results. Next, an LHS simulation was performed using the same variables and distributions and 50 trial runs. The number of trial runs was decreased since it was assumed that the LHS should converge 22 faster than MC upon the expectation and standard deviation. The results of this simulation are displayed in Figures 19 and 20. Figure 19. Final cycle time LHS simulation results. Figure 20. Pump output energy LHS simulation results. 23 We expected the output time to be roughly between 8 and 16 seconds. Both of the analyses displayed a time that satisfied our expectation. LHS displayed a mean time of about 12.23 seconds and MC displayed a mean time of 12.04 seconds. The output energy met our expectations as well. We expected the output energy to fall between 5e6 J and 2e7 J. The output energy for LHS is 1.23e7 J and the output energy for MC is 9.6e6 J. The output time distribution histograms for both types of sampling analyses are shaped like lognormal curves. The output energy histograms appear to take on a decaying exponential shape. These shapes are probably due to the fact that we had to establish lower bounds for the variables such that unrealistic values were not selected from the distributions. The results from the MC are the most desirable results since they display a smaller time as well as a smaller value for energy output. However, we believe that the results from the LHS are a better estimation of the expected value and its standard deviation since LHS takes a wider and more evenly distributed range of input parameters. Task 7: Perform a Sensitivity Analysis The input distributions were elicited based on rough beliefs, so they could have been inaccurate. Thus, a sensitivity analysis was done to determine whether small inaccuracies in our beliefs would cause a big change in the output time and total energy used. To perform this sensitivity analysis, four parameters of the input distributions were chosen: the mean of the two normal distributions, and the scale factor of the two Weibull distributions. The mean of the two normal distributions were adjusted by ±0.1σ, while the two Weibull distributions were adjusted by ±1%. These adjustments were tested with the help of a DOE experiment, where the adjustments were entered for each variable. A Central Composite experiment was chosen and the results are shown in Table 2. 24 Table 2: Results from Sensitivity Analysis Boom Arm Bucket Pump Area Scale Scale Displacement Average Average Mean Factor Factor Mean Time Std. Dev. Energy Std. Dev. 0.011791 0.017028 0.014355 0.06058 12.00952 3.186496 10217778 14057063 0.012209 0.017028 0.014355 0.06058 11.81915 2.487053 10203084 12669999 0.011791 0.017372 0.014355 0.06058 12.02465 3.506986 11691123 16649408 0.012209 0.017372 0.014355 0.06058 12.42864 3.955454 8093446 10094232 0.011791 0.017028 0.014645 0.06058 12.03208 2.797822 9304826 10995671 0.012209 0.017028 0.014645 0.06058 12.65921 5.764271 9726265 11633463 0.011791 0.017372 0.014645 0.06058 12.0397 3.12823 10434986 13974590 0.012209 0.017372 0.014645 0.06058 12.0943 3.232565 10032249 12761436 0.011791 0.017028 0.014355 0.06482 11.67709 2.960821 12425373 16277279 0.012209 0.017028 0.014355 0.06482 12.10877 4.372992 11626621 14459251 0.011791 0.017372 0.014355 0.06482 11.82409 3.288829 9602237 11716721 0.012209 0.017372 0.014355 0.06482 11.99078 3.946417 10658383 14667178 0.011791 0.017028 0.014645 0.06482 11.79132 3.055239 11934282 16409350 0.012209 0.017028 0.014645 0.06482 11.93658 3.169696 12194356 15172689 0.011791 0.017372 0.014645 0.06482 11.81458 3.277025 10876492 13770575 0.012209 0.017372 0.014645 0.06482 12.37524 4.155494 10400933 13551756 0.012 0.0172 0.0145 0.0627 12.07461 3.332705 10557600 13558622 0.011791 0.0172 0.0145 0.0627 12.1077 2.887275 10166443 11586231 0.012209 0.0172 0.0145 0.0627 12.07668 3.292481 9789752 11265178 0.012 0.017028 0.0145 0.0627 11.88737 2.702545 11152230 13851426 0.012 0.017372 0.0145 0.0627 11.83617 3.3373 11275777 14785797 0.012 0.0172 0.014355 0.0627 11.8849 2.778604 9554100 11693550 0.012 0.0172 0.014645 0.0627 11.73289 2.414716 10089503 12183399 0.012 0.0172 0.0145 0.06058 11.9811 2.765455 10233221 13619721 0.012 0.0172 0.0145 0.06482 12.08491 4.27891 12034055 18072542 From Figure 21, it can be shown that the boom area input distribution affected the output the most. However, even for these input distributions, the effect it has on the results were minimal, with the current discrepancies possibly due to the imprecise nature of the LHS. The standard deviation for the output energy was prohibitively high, with the standard deviation being even greater than the mean value. Since the result cannot be negative, the only reason would be that some runs took exceptionally high energy, where their effects were magnified due to the squaring of error. 25 Figure 21: Sensitivity of input distribution The sensitivity analysis showed that the system was not sensitive to minor adjustments to the elicited beliefs. This verified that the distributions used were of the valid type, and that minor discrepancies in the beliefs would not cause drastic changes to the expectation calculations. Task 8: Lessons Learned Rashid Enahora: This assignment taught me various new things. Me senior year in undergraduate school, I took a class called Statistics in Mechanical Engineering. We learned how to use a program called Minitab to analyze data. One type of analysis that we did was a Design of Experiments. After doing this assignment, I’ve learned how much easier it is to do a DOE using Model Center. I also learned about Latin Hypercube Sampling and Monte Carlo Sampling. We never cover this information in my other statistics class. I also learned how Latin Hypercube Sampling reflects reality a little better that Monte Carlo since it uses a wider range of input variables. 26 Tommy Johnson: This homework assignment improved my understanding of the importance of considering uncertainty in model-based design. It has reinforced the idea that an engineer must consider the fact that a model is never correct. If one neglects model uncertainty when making design decisions, he is assuming that the model is perfect. This is a clear violation of the definition of a model. I now know how to analyze model uncertainty using a variety of simulations and experiments. I feel very comfortable using ModelCenter and Arena's Input Analyzer for studying the effects of uncertainty. However, I did initially struggle with belief elicitations. I found that it was hard to relax conservatism while eliciting my true beliefs about model uncertainty Based on what I have learned, I would not approach the uncertainty problem again in drastically different fashion. I felt that the tasks were placed in an appropriate order. Eliciting your beliefs about model uncertainty and determining the main effects on the model are essential starting tasks. Only once these beliefs have been used to determine variable distributions can one determine the expectation of the model and the sensitivities of the belief distributions. The only thing that I would address differently would be my belief elicitations. I feel that I could have obtained more information about appropriate variable distributions had I increased my knowledge about the different system components. When studying the main effects on our model, I learned that our excavator design variables play a major role in the expectation of the model. This reassures our selection of the design variables. I was surprised to learn that the terrain played a smaller role than expected. I would be interested in modifying the terrain's parameter distributions such that they would employ larger ranges of parameter values. Ivan Lee: From this assignment, I found out that the ModelCenter tool is very useful as a test bed for any general purpose numerical processes, with only a wrapper needed to allow ModelCenter to interact with the inputs and outputs freely. One thing I struggled in was eliciting beliefs of the input distributions. It is very difficult to quantize such beliefs, and even harder to correctly mark them down in a CDF. Based on the experiences of running the sensitivity analysis for hours, I 27 would have only changed one input distribution at a time and examine the difference individually (sort of like the Central Composite Experiment, but without the corners). This would greatly decrease the number of runs, which will reduce the time needed to solve the solution. One thing that I learned was that the Excavator dig cycle we used were not very stable. Most runs ran in around 1 minute, but some particular inputs caused it to run for hours. One reason for this was that the 4 parameters were not independent of each other, as larger cylinders require more displacement in the pump. Dan Sankar: For this assignment, I learned how useful simulations of models can be, when run properly. It was surprising to see how long some of these simulations took to run, especially when multiple trials were used. I learned that time management is critical when simulations are required in a certain timeframe. It was difficult to determine how long the simulations would run for, therefore it was good that Tommy ran some over spring break – thus giving us a head start on the assignment, and providing a timeframe for the simulations. It was also interesting to see which parameters of the system had the most effect on the performance, especially when compared to initial hypotheses of the main contributors. 28