Docstoc

adequacy - NII-Research

Document Sample
adequacy - NII-Research Powered By Docstoc
					'ID          DIRECTION   SYSTEM-ID   PRIORITY       TYPE     RESOURCE_BILINGUAL
             EJ
BASELINE1-ej-1           BASELINE1              1   SMT      YES
             EJ
BASELINE2-ej-1           BASELINE2              1   SMT      YES
BJTUX-ej-1 EJ            BJTUX                  1   SMT      YES
BJTUX-ej-2 EJ            BJTUX                  2   SMT      YES
FRDC-ej-1.g JE           FRDC                   1   SMT      YES
ICT-ej-1     EJ          ICT                    1   SMT      YES
ICT-ej-2     EJ          ICT                    2   SMT      YES
ICT-ej-3     EJ          ICT                    3   SMT      YES
ICT-ej-4     EJ          ICT                    4   SMT      YES
ICT-ej-5     EJ          ICT                    5   SMT      YES
ICT-ej-6     EJ          ICT                    6   SMT      YES
JAPIO-ej-1   EJ          JAPIO                  1   RBMT     NO
KLE-ej-1     EJ          KLE                    1   SMT      YES
KLE-ej-2     EJ          KLE                    2   SMT      YES
KLE-ej-3     EJ          KLE                    3   SMT      YES
KLE-ej-4     EJ          KLE                    4   SMT      YES
KLE-ej-5     EJ          KLE                    5   SMT      YES
KYOTO-ej-1 EJ            KYOTO                  1   EBMT     YES
KYOTO-ej-2 EJ            KYOTO                  2   SMT      YES
NTT-UT-ej1 EJ            NTT-UT                 1   SMT      YES
NTT-UT-ej2 EJ            NTT-UT                 2   SMT      YES
NTT-UT-ej3 EJ            NTT-UT                 3   SMT      YES
ONLINE1-ej-1 EJ          ONLINE1                1   SMT      NO
RBMT4-ej-1 EJ            RBMT4                  1   RBMT     NO
RBMT5-ej-1 EJ            RBMT5                  1   RBMT     NO
RBMT6-ej-1 EJ            RBMT6                  1   RBMT     NO
TORI-ej1     EJ          TORI                   1   HYBRID   YES
TORI-ej2     EJ          TORI                   2   HYBRID   YES
TORI-ej3     EJ          TORI                   3   HYBRID   YES
TORI-ej4     EJ          TORI                   4   HYBRID   YES
TORI-ej5     EJ          TORI                   5   HYBRID   YES
UOTTS-ej-1 EJ            UOTTS                  1   SMT      yes
UOTTS-ej-2 EJ            UOTTS                  2   SMT      yes
           RESOURCE_EXTERNAL
RESOURCE_MONOLINGUAL RESOURCE   EXTERNAL   CONTEXT   OFFLINE-TIME
NO         NO         YES       NO         NO        1 week
NO         NO         YES       NO         NO        2 days
NO         NO         YES       NO         NO        2 weeks
NO         NO         YES       NO         NO        2 weeks
YES        NO         YES       YES        NO        10 days
YES        NO         YES       NO         NO        3 weeks
YES        NO         YES       NO         NO        3 weeks
YES        NO         YES       NO         NO        3 weeks
YES        NO         YES       NO         NO        3 weeks
YES        NO         YES       NO         NO        3 weeks
YES        NO         YES       NO         NO        3 weeks
NO         YES        NO        YES        YES                  0
NO         YES        YES       YES        NO        2 weeks
NO         YES        YES       YES        NO        2 weeks
NO         YES        YES       YES        NO        2 weeks
NO         YES        YES       YES        NO        2 weeks
NO         NO         YES       NO         NO        2 weeks
NO         NO         YES       NO         NO        3 days
NO         NO         YES       NO         NO        3 days
YES        NO         YES       YES        NO        3 days
NO         NO         YES       NO         NO        3 days
NO         NO         YES       NO         NO        3 days
NO         YES        NO        YES        NO                   0
NO         YES        NO        YES        NO                   0
NO         YES        NO        YES        NO                   0
NO         YES        NO        YES        NO                   0
YES        YES        YES       YES        NO        5 weeks
YES        YES        YES       YES        NO        5 weeks
YES        YES        YES       YES        NO        5 weeks
YES        YES        YES       YES        NO        4 weeks
YES        NO         YES       YES        NO        5 weeks
no         no         YES       NO         NO        3 weeks
no         no         YES       NO         NO        1 week
ONLINE-TIME                  S
              MACHINE-SPECYSTEM-DESCRIPTION  BLEU             NIST          RIBES
3 days                        The 8 CPU, 24GB memory   phrase-based MT system. 0.720015
              Xeon X5550 2.67GHzMoses hierarchical0.3166            7.7954
4 hours                       The 8 CPU, 24GB memory    0.319
              Xeon X5550 2.67GHzMoses phrase-based MT system. 7.8811             0.706785
2 Hours                      Modified Verison of the 0.2705
                                                      Moses         7.5395       0.655925
              Core2 E8400 3GHz Duo CPU, 3GB memory phrase-based MT system. The language model uses the tr
2 Hours                      Modified Verison of the 0.2584
                                                      Moses          7.497       0.649227
              Core2 E8400 3GHz Duo CPU, 3GB memory hierarchical phrase-based system. We train the model with
1 day                         Hierarchical phrase-based MT           The language0.680992 trained using srilm.
              Xeon 2.5GHz(8 cores) , 40GB memory 0.2781 system. 7.4941            model is
2 days                        System combination four single systems. Four single systems consists of outputs of h
              AMD 1GHz *16 CPU, 64GB memory of 0.3291               8.1443       0.690293
2 days                        In house version of       0.321       8.0788        0.68608
              AMD 1GHz *16 CPU, 64GB memoryhierachical phrase-based MT system setting rule filter threshold to 0
2 days                        In house version of     0.3172        8.1695       0.688819
              AMD 1GHz *16 CPU, 64GB memoryhierachical phrase-based MT system setting rule filter threshold to 0
2 days                        In house version of     0.3206        8.0883       0.688771
              AMD 1GHz *16 CPU, 64GB memoryhierachical phrase-based MT system setting rule filter threshold to 1
2 days                        In house version of     0.3217        8.1202       0.689267
              AMD 1GHz *16 CPU, 64GB memoryhierachical phrase-based MT system setting rule filter threshold to 1
2 days                        Open Source decoder: 0.3017
              AMD 1GHz *16 CPU, 64GB memory Moses                   7.8334        0.66149
10 minutes                    RBMT CPU E8500 @3.16GHz 0.2272         memory
              Intel(R) Core(TM)2 Duowith original dictionary 3.21 GB6.2892       0.708765
1 days                         CPU, 8GB memory        0.3403        8.2467       0.690476
              Xeon 3GHz dual We concatenate two SMT systems: E-E' and E'-J. E' is a modified English corpus that Ja
1 days                         CPU, 8GB memory        0.2982        7.8441       0.645376
              Xeon 3GHz dual We concatenate two SMT systems: E-E' and E'-J. E' is a modified English corpus that Ja
1 days                         CPU, 8GB memory        0.2851        7.6125       0.640937
              Xeon 3GHz dual We concatenate two SMT systems: E-E' and E'-J. E' is a modified English corpus that Ja
1 days                         CPU, 8GB memory        0.2839        7.6761       0.641663
              Xeon 3GHz dual We concatenate two SMT systems: E-E' and E'-J. E' is a modified English corpus that th
1 days                         CPU, 8GB memory          0.351
              Xeon 3GHz dual A hierarchical phrase-based SMT system.8.2846       0.742908
3 hours                       EBMT system with non-bayesian alignment.
              Xeon 2.4GHz quad CPU * 2, 24GB memory   0.2457        6.9313       0.661025
3 hours                       Moses * 2, 24GB our 0.3001 alignment results. 0.681232
              Xeon 2.4GHz quad CPUoutput with memory  modified      7.7135
1 hour                        Combination of three SMT systems: two systems are0.78129
                                                       1024GB       8.7134         phrase-based ones with pre-order
              Xeon 2.26GHz (6-cores) quad-CPU w/ 0.3948 memory for tuning/decoding, Xeon 2.93GHz (4-cores) qu
1 hour                        Combination of three SMT systems: two systems are phrase-based ones with pre-order
                                                       1024GB       8.5444       0.777737
              Xeon 2.26GHz (6-cores) quad-CPU w/ 0.3784 memory for tuning/decoding, Xeon 2.93GHz (4-cores) qu
1 hour                        A monotone translation 1024GB memory for tuning/decoding, Xeon 2.93GHz (4-cores) qu
                                                        system      8.3854       0.772869
              Xeon 2.26GHz (6-cores) quad-CPU w/ 0.3683 by WFSTs with English pre-ordering.
                              An online SMT system. 0.2546          6.8303       0.699055
                                                      0.1688
                              A commercial rule-based MT system. 5.3165          0.684957
                                                      0.1644
                              A commercial rule-based MT system. 5.2993          0.666868
                                                      0.2066
                              A commercial rule-based MT system. 5.9181          0.707608
3   days                      2 Single CPU, 16GB 0.2775
              Phenom-II 3.0GHzStage MT, Modify I. memory            7.3284       0.747874
3   days                      Two stage machine translation
                                                      0.2475         First       0.678157
              Phenom-II 3.0GHz Single CPU, 16GB memory system.7.1413stage is rule based MT. Second stage is nor
3   days                      2 Single CPU, 16GB 0.161
              Phenom-II 3.0GHzStage MT. Modify T.memory             5.1694       0.681843
3   days                      Normal SMT. No parameter tuning.
              Phenom-II 3.0GHz Single CPU, 16GB memory0.2203        6.8573       0.650888
3   days                      2 Single CPU, 16GB 0.0831
              Phenom-II 3.0GHzStage MT, Modify I. memory            3.7711       0.590203
1   day                        CPU, 64GB memory 0.2799              7.2575       0.686112
              Xeon 3GHz dual Our implementation of an HPSG-forest-to-string SMT system.
1   day                        CPU, 64GB memory 0.2781              7.2363       0.689864
              Xeon 3GHz dual Our implementation of a hierarchical phrase based SMT system.
adequacy
               SYSTEM-ID-PRIORITY
average adequacy                     5     4    3     2
          3.67 NTT-UT-1             97    87   43    66
  3.563333333 NTT-UT-3              84    83   57    70
  3.506666667 RBMT6-1               68    96   64    64
  3.463333333 JAPIO-1               63   101   58    68
  3.253333333 RBMT4-1               43   104   56    80
          2.84 RBMT5-1              37    61   69    83
  2.666666667 ONLINE1-1             35    51   38   131
  2.603333333 BASELINE1-1           35    40   39   143
           2.6 TORI-1               15    63   63   105
  2.476666667 BASELINE2-1           32    29   37   154
  2.353333333 KLE-1                 23    28   38   154
  2.346666667 FRDC-1                21    31   39   149
          2.32 ICT-1                16    29   41   163
  2.193333333 UOTTS-1               15    26   28   164
          2.18 KYOTO-2              23    18   29   150
  2.046666667 KYOTO-1               15    25   38   103
  1.793333333 BJTUX-1                6    13   18   139




           100%

            90%

            80%

            70%

            60%
                                                          1
            50%
                                                          2
            40%                                           3
            30%                                           4
            20%                                           5

            10%

             0%
      1 total number
      7           300
      6           300
      8           300
     10           300
     17           300
     50           300
     45           300
     43           300
     54           300
     48           300
     57           300
     60           300
     51           300
     67           300
     80           300
    119           300
    124           300




1
2
3
4
5
acceptability
                (tie)
pairwise comparison score                   AA
                                SYSTEM-ID-PRIORITY              A              B
  0.694666667 0.197666667       NTT-UT-1                   93             46                26
  0.655666667           0.198   RBMT6-1                    44             79                27
        0.6515 0.195166667      JAPIO-1                    41             87                26
          0.479 0.290666667     ONLINE1-1                  18             40                17
  0.471666667           0.304   BASELINE1-1                31             20                10
  0.456333333           0.308   BASELINE2-1                31             15                11
  0.434166667          0.3095   KLE-1                      19             19                 9
  0.431833333 0.299166667       TORI-1                     14             21                12
          0.411         0.315   UOTTS-1                    17             10                 7
        0.4105 0.311166667      ICT-1                      10             18                11
  0.403666667           0.309   KYOTO-1                    12             18                 6

pairwise comparison score = ((number of win) + 0.5 * (number of tie)) / (number of pairs)
(tie) = 0.5 * (number of tie) / (number of pairs)


          100%

            90%

            80%

            70%

            60%
                                                                                     F
            50%
                                                                                     C
            40%                                                                      B
            30%                                                                      A
            20%                                                                      AA

            10%

             0%
                                               Pairwise
C        F         total number                     score
    16       119             300   NTT-UT-1        0.6947
    30       120             300   RBMT6-1         0.6557
    26       120             300   JAPIO-1         0.6515
    13       212             300   ONLINE1-1       0.4790
    18       221             300   BASELINE1-1     0.4717
    11       232             300   BASELINE2-1     0.4563
    11       242             300   KLE-1           0.4342
    14       239             300   TORI-1          0.4318
    13       253             300   UOTTS-1         0.4110
    14       247             300   ICT-1           0.4105
    12       252             300   KYOTO-1         0.4037
Rate
     AA     A or higher   B or higher   C or higher   F or higher
   0.3100     0.4633        0.5500       0.6033         1.0000
   0.1467     0.4100        0.5000       0.6000         1.0000
   0.1367     0.4267        0.5133       0.6000         1.0000
   0.0600     0.1933        0.2500       0.2933         1.0000
   0.1033     0.1700        0.2033       0.2633         1.0000
   0.1033     0.1533        0.1900       0.2267         1.0000
   0.0633     0.1267        0.1567       0.1933         1.0000
   0.0467     0.1167        0.1567       0.2033         1.0000
   0.0567     0.0900        0.1133       0.1567         1.0000
   0.0333     0.0933        0.1300       0.1767         1.0000
   0.0400     0.1000        0.1200       0.1600         1.0000

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:2
posted:2/25/2012
language:
pages:8