Report on the Croatia trial of Multi-Sensor Systems (October 2007) – Y.Das As an ITEP observer at the trials of multi-sensor (MS) mine detector systems held in Benkovac Croatia during 22-26 October, 2007, I understood my responsibility to be to report on three related issues. These were: 1. To observe and comment on how well the trial plan was followed. 2. To comment on the possibility of changing the controversial trial plan to make it acceptable to the non-participating manufacturers. 3. To assess the role of the ITEP Multi-Sensor Working Group (MSWG) in coming up with guidelines for test and evaluation of MS systems, which would be acceptable to most of the stakeholders. I will discuss each of the above issues. But first let me list a number of things that have happened in the meantime that I came to know from Mr. Dieter Guelle, details of which should be confirmed at a higher level by the ITEP ExCom before acting on them. I have added some personal comments to some items. a. The US Department of State has revoked the license that allowed BAM/Germany to transport the HSTAMIDS to Croatia for testing. b. The German Army is in the process of getting a clarification from the US authorities as to what restrictions may apply to any testing of the HSTAMIDS that the German military may like to do for their own use. c. It seems the HSTAMIDS may not be made available for any independent multi- national comparative testing for humanitarian demining in the immediate future. (My assessment: This is an understandable position, noting that the HSTAMIDS is currently deployed with the US Forces. However, this only leaves two systems, Japanese ALIS and Vallon MineHound, available for humanitarian demining testing. Also note that the ALIS is still a prototype and not a COTS system. ITEP should keep this lack of systems to test in mind in assessing the effectiveness of any resources spent on this topic) d. Although ITEP had two observers (Dr.Y. Das of Canada and Dr.Y.Yvinec of Belgium) the subject trial at Benkovac was sponsored by the Japanese and conducted in co-operation with the Croatian Mine Action Centre- Centre for Testing, Development and Training (HCR-CTRO) with some assistance from BAM. As a result, this is not an ITEP trial and ITEP should not expect to have automatic access to the results and reports coming out of it. (I agree with this view. On the bright side, I have been informed verbally by HCR-CTRO that the reports will be circulated for comments from the ITEP observers and as such ITEP will have access to the results). I will now discuss the three original mandates as an ITEP observer at the trial. 1. To observe and comment on how well the trial plan was followed. The trial was well run and the plan followed as far as possible within the limits imposed by poor weather and the non-participation of two of the three original manufacturers. Some minor modifications and additions to the original plans were done. Examples are: i. In addition to the hand held ALIS system, the vehicle-mounted robotic arm-based MS system called the Gryphon was also tested during the trial. Only three lanes were used for the ALIS trial, while the Gryphon was used on all six. ii. One additional type of clutter had been added to bring the mine to clutter ratio to 1:1. iii. The originally planned soil measurement for GPR performance that were to be done by Germany were not conducted. However, the Japanese team measured soil moisture content and electrical conductivity at several points along the lanes and at various times during the day. Although this was not as extensive a soil characterization as was planned, these measurements should yield valuable insight into the dependence of GPR performance on soil properties. As well, it should be noted that the magnetic susceptibility of these lanes had been previously characterized rather thoroughly. iv. The planned scanning of the lanes by a non-integrated standalone metal detector was somewhat curtailed and modified. A metal detector mounted on the Gryphon and not deminer-operated one was used for this. The MineLab F3, which is mounted on the first of the two vehicles in the Gryphon system, was used to scan all the six lanes while the CEIA detector was used only for a limited number of lanes. v. In addition to the six test lanes, the Gryphon was used to scan a so-called “blind” lane outside the normal test area. While we wait for the results and conclusions, these tests should be viewed as what they are. They are scenario based trials limited to partially answering very specific questions, for example, how well does the detector detect typical difficult-to-detect AP mines found in Croatia (PMA-2 and PMA-3 in this case) and discriminate them from a limited number and types of typical clutter found there. Clutter items used were not all ferrous and they were of different sizes. The soils used are also typical of Croatia and the skill level (including physical strength, endurance and technical competence) of employed deminers are assumed to be typical of Croatian deminers. One needs to bear these constraints in mind when evaluating the results. These trials are by no means all encompassing general trials, the results of which can be used to unequivocally compare the performance of detectors under all circumstances – a goal that some think is not only achievable but is a must have before they would subject their equipment to testing. In my humble opinion, such “silver-bullet” trials , even if they can be technically designed and executed (which I doubt), would be well beyond available resources for such a purpose. That is why I firmly believe that a trial such as the one just conducted in Benkovac should not be used to choose a “winner” or “loser” (as was indicated in the original trial plan), but should be used to just gather and report information on the performance of various MS detectors under stated test conditions and noting how changes in certain test parameters may change the conclusions. However, if a particular entity, such as a nation’s armed forces or a demining organization, wishes to select a MS detector they should and will run their own comparative tests based on their particular requirements and scenario, without much regard for what a particular manufacturer may think the tests should involve. However, these organizations can profitably use the results of tests such as the Benkovac one as a starting point in designing their tests. As well, although limited in scope, these Benkovac trials will produce a lot of valuable information about the ALIS and Gryphon systems, information that can be used in the further development of the systems. I feel that the non-participants missed an opportunity to get additional performance information on their systems. At the request of the Japanese team, both Dr.Yvinec and I provided our comments and impressions of the two systems tested and suggested some possible improvements. Dr.Yvinec will likely provide his own detailed comments to ITEP on this trial. 2. To comment on the possibility of changing the controversial trial plan to make it acceptable to the non-participating manufacturers. To the best of my knowledge, the trial plan was put together primarily by BAM/Germany. It seems there was considerable lack of communication among parties interested in the test plan and some felt their views were not heard. Particularly, there appears to have been a break in communication between the ITEP MSWG and the personnel developing the plan. As I have already said there is no all-encompassing “silver-bullet” test plan for MS detectors because of the large number of variables and scenarios that are involved and the process of developing a workable test plan is primarily one of building a consensus among stake holders about the question(s) a specific test plan should be designed to answer. The importance of doing this is further illustrated by the fact that even the various scientists differ in their view of how to best design and analyze the statistical aspects involved in a test plan. In my experience, no matter what test plan one comes up with, it will surely lack in some respects and will be criticized by those not involved in designing the plan. Against this background, it would seem desirable to modify the BAM/German plan or come up with a new one that is acceptable to most stake holders including manufacturers. (Although some would, no doubt, argue that including manufacturers in designing test plans for their equipment would have the appearance of conflict of interest. It should also be kept in mind that it took years to develop reasonable test plans for metal detectors; MS detectors are too new and more complex and hence one should expect more difficulties in coming up with acceptable plans). However there are some practical issues that ITEP ExComm needs to consider before deciding if such an effort would be worthwhile. These include: i. The current BAM/German test plan was designed around the six mine lanes in Benkovac, Croatia, where a number of mines and clutter items have been buried for a while. I have been informed that drastically altering these lanes is out of the question for various reasons: introducing additional target and clutter items will make these lanes too congested; disturbing these lanes would make comparing results of previous tests with new ones difficult; and so on. ii. So one needs to find a new area (in Croatia or elsewhere) to prepare test lanes for implementing any new test plan that gets agreed upon. This obviously would require a sponsor and resources. iii. It would only make sense to spend resources to “fix” the plan if ITEP plans to run another test in the near future. This too would require a sponsor and resources. iv. Assuming a sponsor and necessary resources can be found for such a plan and testing, ITEP should first ascertain how many of the three manufacturers would participate in a new test and evaluation effort. Obviously it would make little sense to carry out testing again with only one supplier. 3. To assess the role of the ITEP Multi-Sensor Working Group (MSWG) in coming up with guidelines for test and evaluation of MS systems, which would be acceptable to most of the stakeholder The ITEP MSWG is probably the body that should be charged with coming up with short-term “fix” and/or expansion of the current BAM/German test plan, if the ITEP ExComm decides to follow through with further test and evaluation of MS detectors in the immediate future. However, I see a bigger role for the MSWG. This Group should resume regular meetings and activities to consider the scientific details and intricacies of MS testing with a view to coming up with a general set of guidelines for testing handheld MS systems. In this regard, I note that a draft document titled “Proposal for a Standard for Reliability Tests of Dual Sensors in Humanitarian Demining” have been circulated by BAM for comments. While MSWG may choose to use this document as a starting point, the group should freely discuss all issues involved in testing MS detectors. I would also recommend that the resulting document not be referred to as a “Standard”, but “Guidelines”. “Standard” has a sense of rigidity and often gets peoples’ backs up. Further, testing of MS systems is too new and there are too many unknowns for one to be able to prescribe a “Standard”. “Guidelines” are general directions from which an user can chose. Since the process of developing such guidelines need not involve coming up with a specific user test plans, all interested parties including manufacturers can be invited to freely participate and contribute. It may even be worthwhile to invite various organizations and entities to submit their proposed test plans and include a collection of these in the guideline document with comments from the MSWG. As well, such guidelines would be of value even if there are no immediate plans to implement them in a test. One of the issues that I would like the group to seriously consider is the dependence of the performance of MS systems on operators, how to quantify and/or eliminate this dependence in tests, and the associated training burden that such dependence puts on any potential deployment of such systems. From my observation of how the current systems work, I am convinced that the dependence of performance on operator training (and maintenance of skills through retraining) is the most important factor that will decide if MS detectors will be widely employed.
Pages to are hidden for
"Report on the Croatia trial of Multi-Sensor Systems (October 2007) - Y"Please download to view full document