Report on the Croatia trial of Multi-Sensor Systems (October 2007) - Y by ito20106


									Report on the Croatia trial of Multi-Sensor Systems (October 2007) – Y.Das

As an ITEP observer at the trials of multi-sensor (MS) mine detector systems held in
Benkovac Croatia during 22-26 October, 2007, I understood my responsibility to be to
report on three related issues. These were:

1. To observe and comment on how well the trial plan was followed.
2. To comment on the possibility of changing the controversial trial plan to make it
   acceptable to the non-participating manufacturers.
3. To assess the role of the ITEP Multi-Sensor Working Group (MSWG) in coming up
   with guidelines for test and evaluation of MS systems, which would be acceptable to
   most of the stakeholders.

I will discuss each of the above issues. But first let me list a number of things that have
happened in the meantime that I came to know from Mr. Dieter Guelle, details of which
should be confirmed at a higher level by the ITEP ExCom before acting on them. I have
added some personal comments to some items.

   a. The US Department of State has revoked the license that allowed BAM/Germany
      to transport the HSTAMIDS to Croatia for testing.
   b. The German Army is in the process of getting a clarification from the US
      authorities as to what restrictions may apply to any testing of the HSTAMIDS that
      the German military may like to do for their own use.
   c. It seems the HSTAMIDS may not be made available for any independent multi-
      national comparative testing for humanitarian demining in the immediate future.
      (My assessment: This is an understandable position, noting that the HSTAMIDS
      is currently deployed with the US Forces. However, this only leaves two systems,
      Japanese ALIS and Vallon MineHound, available for humanitarian demining
      testing. Also note that the ALIS is still a prototype and not a COTS system. ITEP
      should keep this lack of systems to test in mind in assessing the effectiveness of
      any resources spent on this topic)
   d. Although ITEP had two observers (Dr.Y. Das of Canada and Dr.Y.Yvinec of
      Belgium) the subject trial at Benkovac was sponsored by the Japanese and
      conducted in co-operation with the Croatian Mine Action Centre- Centre for
      Testing, Development and Training (HCR-CTRO) with some assistance from
      BAM. As a result, this is not an ITEP trial and ITEP should not expect to have
      automatic access to the results and reports coming out of it. (I agree with this
      view. On the bright side, I have been informed verbally by HCR-CTRO that the
      reports will be circulated for comments from the ITEP observers and as such
      ITEP will have access to the results).

I will now discuss the three original mandates as an ITEP observer at the trial.
   1. To observe and comment on how well the trial plan was
The trial was well run and the plan followed as far as possible within the limits imposed
by poor weather and the non-participation of two of the three original manufacturers.
Some minor modifications and additions to the original plans were done. Examples are:

   i. In addition to the hand held ALIS system, the vehicle-mounted robotic arm-based
      MS system called the Gryphon was also tested during the trial. Only three lanes
      were used for the ALIS trial, while the Gryphon was used on all six.
  ii. One additional type of clutter had been added to bring the mine to clutter ratio to
 iii. The originally planned soil measurement for GPR performance that were to be done
      by Germany were not conducted. However, the Japanese team measured soil
      moisture content and electrical conductivity at several points along the lanes and at
      various times during the day. Although this was not as extensive a soil
      characterization as was planned, these measurements should yield valuable insight
      into the dependence of GPR performance on soil properties. As well, it should be
      noted that the magnetic susceptibility of these lanes had been previously
      characterized rather thoroughly.
 iv. The planned scanning of the lanes by a non-integrated standalone metal detector
      was somewhat curtailed and modified. A metal detector mounted on the Gryphon
      and not deminer-operated one was used for this. The MineLab F3, which is
      mounted on the first of the two vehicles in the Gryphon system, was used to scan all
      the six lanes while the CEIA detector was used only for a limited number of lanes.
  v. In addition to the six test lanes, the Gryphon was used to scan a so-called “blind”
      lane outside the normal test area.

While we wait for the results and conclusions, these tests should be viewed as what they
are. They are scenario based trials limited to partially answering very specific questions,
for example, how well does the detector detect typical difficult-to-detect AP mines found
in Croatia (PMA-2 and PMA-3 in this case) and discriminate them from a limited number
and types of typical clutter found there. Clutter items used were not all ferrous and they
were of different sizes. The soils used are also typical of Croatia and the skill level
(including physical strength, endurance and technical competence) of employed deminers
are assumed to be typical of Croatian deminers. One needs to bear these constraints in
mind when evaluating the results. These trials are by no means all encompassing general
trials, the results of which can be used to unequivocally compare the performance of
detectors under all circumstances – a goal that some think is not only achievable but is a
must have before they would subject their equipment to testing. In my humble opinion,
such “silver-bullet” trials , even if they can be technically designed and executed (which I
doubt), would be well beyond available resources for such a purpose. That is why I
firmly believe that a trial such as the one just conducted in Benkovac should not be used
to choose a “winner” or “loser” (as was indicated in the original trial plan), but should be
used to just gather and report information on the performance of various MS detectors
under stated test conditions and noting how changes in certain test parameters may
change the conclusions. However, if a particular entity, such as a nation’s armed forces or
a demining organization, wishes to select a MS detector they should and will run their
own comparative tests based on their particular requirements and scenario, without much
regard for what a particular manufacturer may think the tests should involve. However,
these organizations can profitably use the results of tests such as the Benkovac one as a
starting point in designing their tests. As well, although limited in scope, these Benkovac
trials will produce a lot of valuable information about the ALIS and Gryphon systems,
information that can be used in the further development of the systems. I feel that the
non-participants missed an opportunity to get additional performance information on their

At the request of the Japanese team, both Dr.Yvinec and I provided our comments and
impressions of the two systems tested and suggested some possible improvements.
Dr.Yvinec will likely provide his own detailed comments to ITEP on this trial.

   2. To comment on the possibility of changing the controversial
      trial plan to make it acceptable to the non-participating

To the best of my knowledge, the trial plan was put together primarily by
BAM/Germany. It seems there was considerable lack of communication among parties
interested in the test plan and some felt their views were not heard. Particularly, there
appears to have been a break in communication between the ITEP MSWG and the
personnel developing the plan. As I have already said there is no all-encompassing
“silver-bullet” test plan for MS detectors because of the large number of variables and
scenarios that are involved and the process of developing a workable test plan is
primarily one of building a consensus among stake holders about the question(s) a
specific test plan should be designed to answer. The importance of doing this is further
illustrated by the fact that even the various scientists differ in their view of how to best
design and analyze the statistical aspects involved in a test plan. In my experience, no
matter what test plan one comes up with, it will surely lack in some respects and will be
criticized by those not involved in designing the plan. Against this background, it would
seem desirable to modify the BAM/German plan or come up with a new one that is
acceptable to most stake holders including manufacturers. (Although some would, no
doubt, argue that including manufacturers in designing test plans for their equipment
would have the appearance of conflict of interest. It should also be kept in mind that it
took years to develop reasonable test plans for metal detectors; MS detectors are too new
and more complex and hence one should expect more difficulties in coming up with
acceptable plans). However there are some practical issues that ITEP ExComm needs to
consider before deciding if such an effort would be worthwhile. These include:

  i. The current BAM/German test plan was designed around the six mine lanes in
     Benkovac, Croatia, where a number of mines and clutter items have been buried for
     a while. I have been informed that drastically altering these lanes is out of the
      question for various reasons: introducing additional target and clutter items will
      make these lanes too congested; disturbing these lanes would make comparing
      results of previous tests with new ones difficult; and so on.
  ii. So one needs to find a new area (in Croatia or elsewhere) to prepare test lanes for
      implementing any new test plan that gets agreed upon. This obviously would
      require a sponsor and resources.
 iii. It would only make sense to spend resources to “fix” the plan if ITEP plans to run
      another test in the near future. This too would require a sponsor and resources.
 iv. Assuming a sponsor and necessary resources can be found for such a plan and
      testing, ITEP should first ascertain how many of the three manufacturers would
      participate in a new test and evaluation effort. Obviously it would make little sense
      to carry out testing again with only one supplier.

   3. To assess the role of the ITEP Multi-Sensor Working Group
      (MSWG) in coming up with guidelines for test and evaluation of
      MS systems, which would be acceptable to most of the

The ITEP MSWG is probably the body that should be charged with coming up with
short-term “fix” and/or expansion of the current BAM/German test plan, if the ITEP
ExComm decides to follow through with further test and evaluation of MS detectors in
the immediate future. However, I see a bigger role for the MSWG. This Group should
resume regular meetings and activities to consider the scientific details and intricacies of
MS testing with a view to coming up with a general set of guidelines for testing handheld
MS systems. In this regard, I note that a draft document titled “Proposal for a Standard
for Reliability Tests of Dual Sensors in Humanitarian Demining” have been circulated by
BAM for comments. While MSWG may choose to use this document as a starting point,
the group should freely discuss all issues involved in testing MS detectors. I would also
recommend that the resulting document not be referred to as a “Standard”, but
“Guidelines”. “Standard” has a sense of rigidity and often gets peoples’ backs up.
Further, testing of MS systems is too new and there are too many unknowns for one to be
able to prescribe a “Standard”. “Guidelines” are general directions from which an user
can chose. Since the process of developing such guidelines need not involve coming up
with a specific user test plans, all interested parties including manufacturers can be
invited to freely participate and contribute. It may even be worthwhile to invite various
organizations and entities to submit their proposed test plans and include a collection of
these in the guideline document with comments from the MSWG. As well, such
guidelines would be of value even if there are no immediate plans to implement them in a

One of the issues that I would like the group to seriously consider is the dependence of
the performance of MS systems on operators, how to quantify and/or eliminate this
dependence in tests, and the associated training burden that such dependence puts on any
potential deployment of such systems. From my observation of how the current systems
work, I am convinced that the dependence of performance on operator training (and
maintenance of skills through retraining) is the most important factor that will decide if
MS detectors will be widely employed.

To top