Docstoc

Binary Alternatives to XML Encoding

Document Sample
Binary Alternatives to XML Encoding Powered By Docstoc
					   BINARY ALTERNATIVES TO XML ENCODING




                           By

                    Nigel Thompson

MS Computer Science, Colorado Technical University, 2005

 BSc (Hons) Electronics, University of Southampton, 1977




                A Dissertation Submitted to
       the Faculty of Colorado Technical University
         in Partial Fulfillment of the Requirements
       for the Degree of Doctor of Computer Science




               Colorado Springs, Colorado

                       June 2008
 Copyright by

Nigel Thompson

    2008
              BINARY ALTERNATIVES TO XML ENCODING




                                    By

                              Nigel Thompson




                                Approved:



_________________________________    _________________________________
Cynthia M. Calongne, DCS             Bo I. Sandén, PhD
Professor of Computer Science        Professor of Computer Science
(Director of Dissertation)           (Committee Member)




_________________________________
Elaine Waybright, PhD
Professor of Computer Science
(Committee Member)




                      Date Approved: _June 19, 2008__
                                        ABSTRACT



       XML has become widely adopted as a means of encoding hierarchical self-

describing data both for storage and also for communication between systems. As the

application of XML has broadened it has become apparent that it has some serious

limitation especially when used to encode binary data. Various industry groups and

researchers have examined ways to improve XML both generally and when used to

encode binary information. This research examines binary encodings as alternatives to

XML for both data storage and communications and shows that improvements in both

encoded data size and CPU time are possible with no loss of information or ease of use.

       In every case tested, the binary protocol encodings performed much better than

the XML protocol encoding. All of the binary implementations performed better than

XML and they all performed better for every data type tested. The improvement in

individual performance measurements varied from about four times to more than a

thousand times in one specific case. Of the three binary encodings tested, ASN.1 DER

shows clear advantages over the other encodings and clearly offers an alternative to XML

where encoded data size and CPU overhead are important especially when the data

content is partially or exclusively binary.
                       DEDICATION


This work is dedicated to my wife Tammy for her consistent support and

  encouragement without which I could not have completed it.




                              ii
                                ACKNOWLEDGMENTS


       Many thanks to: Dr. Cynthia Calongne for her great enthusiasm, constant

encouragement and a continuous email stream of *grins* and *waves*. To Dr. Bo

Sanden and Dr. Elaine Waybright for reading my work and offering many suggestions for

improving it. Thanks are also due to Kristal Ray, Dave Fleck and Jason Andress for

support, opinion, advice and the injection of some good laughs when the going got rough.

       Very special thanks to my daughter Elizabeth for her rendition of “Indian Dance”

recorded in the short audio clip that was used in the experiments.




                                            iii
                                                                                                                                 4

                                              TABLE OF CONTENTS


                                                                                                                            Page

DEDICATION ............................................................................................................. ii

ACKNOWLEDGMENTS ........................................................................................... iii

LIST OF TABLES ....................................................................................................... 9

LIST OF FIGURES ..................................................................................................... 12

CHAPTER I ................................................................................................................. 18
     INTRODUCTION ................................................................................................. 18
        1.1     Background .......................................................................................... 18
            1.1.1 Data formatting requirements ........................................................ 19
            1.1.2 Loose coupling between applications ............................................ 21
            1.1.3 Parsing data in a loosely-coupled environment ............................. 22
                1.1.3.1 Scenario 1................................................................................. 22
                1.1.3.2 Scenario 2................................................................................. 22
                1.1.3.3 Scenario 3................................................................................. 23
            1.1.4 How XML is used .......................................................................... 24
            1.1.5 Problems with XML ...................................................................... 26
                1.1.5.1 Latency times ........................................................................... 27
                1.1.5.2 CPU costs ................................................................................. 28
            1.1.6 Possible approaches ....................................................................... 33
        1.2     Problem Definition............................................................................... 34
            1.2.1 Improving XML performance........................................................ 35
            1.2.2 Alternatives to XML ...................................................................... 35
                1.2.2.1 Encoding a data tree ................................................................. 36
                1.2.2.2 Encoding different data types .................................................. 36
                1.2.2.3 Determining performance ........................................................ 37
            1.2.3 Meaningful Measurements............................................................. 38
        1.3     Hypothesis............................................................................................ 40
        1.4     Research Questions .............................................................................. 40
            1.4.1 Measuring protocol performance ................................................... 41
            1.4.2 Fairness .......................................................................................... 41
            1.4.3 Determining appropriate data sets ................................................. 42
            1.4.4 Determine alternative encodings to test ......................................... 43
        1.5     Proof Criteria ....................................................................................... 44
            1.5.1 Null hypothesis .............................................................................. 44
            1.5.2 Completion criteria ........................................................................ 44
                                                                                                                                 5
              1.5.3 Success criteria............................................................................... 46
                  1.5.3.1 Repeatability ............................................................................ 46
          1.6     Assumptions......................................................................................... 47
          1.7     Dependencies ....................................................................................... 49
          1.8     Limitations ........................................................................................... 50
          1.9     Constraints ........................................................................................... 51

CHAPTER II ................................................................................................................ 52
     LITERATURE REVIEW ...................................................................................... 52
        2.1     Introduction .......................................................................................... 52
            2.1.1 Early Performance Work ............................................................... 53
        2.2     The Use of XML and Other Text Encoding Schemes ......................... 55
        2.3     Comparing the performance of XML to other protocols ..................... 56
        2.4     Optimizing XML ................................................................................. 58
            2.4.1 Optimizing SOAP .......................................................................... 60
        2.5     General optimization............................................................................ 61
        2.6     Remote Procedure Call Mechanisms ................................................... 63
            2.6.1 XML-RPC ...................................................................................... 64
            2.6.2 SOAP ............................................................................................. 65
            2.6.3 Call optimization............................................................................ 65
            2.6.4 RPC for high-performance computing and real-time applications 66
        2.7     Network Protocols ............................................................................... 67
            2.7.1 Standard protocols ......................................................................... 67
            2.7.2 Other protocol work ....................................................................... 71
        2.8     Data formatting .................................................................................... 72
        2.9     Tree Walking and Searching ................................................................ 75
        2.10 Papers about ASN.1 ............................................................................. 76
        2.11 Binary Protocols in Commercial and Industrial Applications ............. 78

CHAPTER III .............................................................................................................. 81
     METHODOLOGY ................................................................................................ 81
       3.1     New contribution ................................................................................. 81
       3.2     Prior work ............................................................................................ 82
       3.3     Selection of protocol encodings ........................................................... 82
       3.4     Determination of generic forms ........................................................... 84
       3.5     Selection of data structures .................................................................. 85
       3.6     Determination of experiment vs. analysis ............................................ 85
       3.7     The Approach Taken to Writing the Code ........................................... 87
           3.7.1 Object Oriented Design.................................................................. 87
               3.7.1.1 Protocol Verification Test Code .............................................. 89
               3.7.1.2 Data Generation ....................................................................... 90
               3.7.1.3 Structure of the experimental code .......................................... 91

CHAPTER IV .............................................................................................................. 93
                                                                                                                                6
     EXPERIMENTAL DESIGN ................................................................................. 93
       4.1     Introduction .......................................................................................... 93
           4.1.1 Protocol requirements .................................................................... 93
       4.2     Evaluating specific encodings.............................................................. 94
       4.3     The common protocol driver ............................................................... 95
       4.4     Expressing the hierarchy ...................................................................... 96
       4.5     Parsing the encoded form..................................................................... 97
       4.6     The performance measurements .......................................................... 98
       4.7     The data sets used ................................................................................ 100
       4.8     Data generation .................................................................................... 101
       4.9     Test runs and data recording ................................................................ 102
           4.9.1 Use of derived values ..................................................................... 105

CHAPTER V ............................................................................................................... 107
     ANALYSIS AND RESULTS ................................................................................ 107
       5.1     Validation ............................................................................................. 107
       5.2     Results .................................................................................................. 108
           5.2.1 Table row and column labels ......................................................... 109
       5.3     Table set 1 – Measured values for each data type ............................... 110
       5.4     Table set 2 – Data type variation for each measured value ................. 116
       5.5     Histogram set 1 – Measured values for each data type ........................ 124
           5.5.1 Document data type ....................................................................... 124
           5.5.2 Email data type .............................................................................. 130
           5.5.3 3D-mesh data type ......................................................................... 136
           5.5.4 Audio data type .............................................................................. 142
           5.5.5 Web query data type ...................................................................... 147
           5.5.6 Web query response data type ....................................................... 153
           5.5.7 Spreadsheet data type ..................................................................... 159

CHAPTER VI .............................................................................................................. 165
     CONCLUSIONS.................................................................................................... 165
       6.1     Overall analysis .................................................................................... 165
       6.2     Analysis by data type ........................................................................... 166
           6.2.1 Audio.............................................................................................. 166
           6.2.2 Document ....................................................................................... 167
           6.2.3 Email .............................................................................................. 168
           6.2.4 3D Mesh ......................................................................................... 169
           6.2.5 Web Query ..................................................................................... 170
           6.2.6 Web Query Response ..................................................................... 171
           6.2.7 Spreadsheet .................................................................................... 171
           6.2.8 Text vs. binary content ................................................................... 172
       6.3     Analysis by measured value................................................................. 172
           6.3.1 Create time ..................................................................................... 173
           6.3.2 Render time .................................................................................... 173
                                                                                                                                7
              6.3.3 Write time ...................................................................................... 175
              6.3.4 Parse time ....................................................................................... 176
              6.3.5 Extract time .................................................................................... 177
              6.3.6 Read time ....................................................................................... 177
              6.3.7 Total time ....................................................................................... 178
              6.3.8 Natural size .................................................................................... 178
              6.3.9 Packet size...................................................................................... 178
              6.3.10 Ratio of packet size to natural size ................................................ 179
              6.3.11 RT cost figure ................................................................................ 179
          6.4     Analysis by protocol ............................................................................ 179
              6.4.1 ASN.1............................................................................................. 180
              6.4.2 TVL1 .............................................................................................. 181
              6.4.3 TVM1............................................................................................. 182
          6.5     Analysis of completion criteria ............................................................ 183
              6.5.1 Protocol encodings ......................................................................... 183
              6.5.2 Data types....................................................................................... 183
              6.5.3 Framework ..................................................................................... 183
              6.5.4 Data-generation code ..................................................................... 183
              6.5.5 Protocol testing .............................................................................. 183
              6.5.6 Initial testing .................................................................................. 184
          6.6     Analysis of success criteria .................................................................. 184
              6.6.1 Comparison of overall costs........................................................... 184
              6.6.2 Comparison of costs for binary data .............................................. 185
          6.7     Rejection of null hypothesis................................................................. 185
          6.8     Acceptance of alternative hypothesis................................................... 185
          6.9     Other observations ............................................................................... 186
              6.9.1 Variable-length fields..................................................................... 186
              6.9.2 Dependence of create and extract times on protocol choice .......... 186
              6.9.3 The use of C-style structures.......................................................... 188
              6.9.4 Data sizes ....................................................................................... 188
          6.10 Summary .............................................................................................. 189

CHAPTER VII ............................................................................................................. 190
     AREAS FOR FUTURE RESEARCH ................................................................... 190
       7.1   Alternative TLV Form ......................................................................... 190
       7.2   Other binary encodings ........................................................................ 190
       7.3   Expanded data set types ....................................................................... 191
       7.4   Improved node implementation ........................................................... 191
     REFERENCES ...................................................................................................... 192

APPENDICES ............................................................................................................. 200
     TLV1 ENCODING ................................................................................................ 201
     TVM1 ENCODING ............................................................................................... 210
                                                                                                             8
THE EXPERIMENTAL DATA SETS .................................................................. 219
SYSTEM INFORMATION ................................................................................... 232
EXPERIMENTAL REPEATABILITY TESTING ............................................... 234
                                                                                                                             9
                                                  LIST OF TABLES


TABLE                                                                                                                   Page

Table 1: Various encodings of the integer value 140 ....................................................... 30

Table 2: Log file columns ............................................................................................... 105

Table 3: Protocol labels .................................................................................................. 109

Table 4: Data type labels................................................................................................. 109

Table 5: Measurement labels .......................................................................................... 109

Table 6: Mean values for the Audio type........................................................................ 110

Table 7: Ratio to Xnode for Audio type ......................................................................... 110

Table 8: Mean values for the Document type ................................................................. 111

Table 9: Ratio to Xnode for Document type ................................................................... 111

Table 10: Mean values for the Email type ...................................................................... 111

Table 11: Ratio to Xnode for Email type ........................................................................ 112

Table 12: mean values for the 3D Mesh type ................................................................. 112

Table 13: Ratio to Xnode for 3D Mesh type................................................................... 113

Table 14: Mean values for the Spreadsheet type ............................................................ 113

Table 15: Ratio to Xnode for Spreadsheet type .............................................................. 113

Table 16: Mean values for the Web Query type ............................................................. 114

Table 17: Ratio to Xnode for Web Query Type ............................................................. 114

Table 18: Mean values for the Web Query Response type ............................................. 115

Table 19: Ratio to Xnode for Web Query Response type .............................................. 115

Table 20: Mean values for Create time in microseconds ................................................ 116
                                                                                                                         10
Table 21: Ratios to Xnode for Create time ..................................................................... 116

Table 22: Mean values for Render time in microseconds............................................... 116

Table 23: Ratios to Xnode for Render time .................................................................... 117

Table 24: Mean values for Parse time in microseconds ................................................. 117

Table 25: Ratios to Xnode for Parse time ....................................................................... 117

Table 26: Mean values for Extract time in microseconds............................................... 118

Table 27: Ratios to Xnode for Extract time .................................................................... 118

Table 28: Mean values for Write time in microseconds ................................................. 118

Table 29: Ratios to Xnode for Write time ...................................................................... 119

Table 30: Mean values for Read time in microseconds .................................................. 119

Table 31: Ratios to Xnode for Read time ....................................................................... 119

Table 32: Mean values for Total time in microseconds .................................................. 120

Table 33: Ratios to Xnode for Total time ....................................................................... 120

Table 34: Mean values for Natural data size in bytes ..................................................... 120

Table 35: Ratios to Xnode for Natural size .................................................................... 121

Table 36: Mean values for Packet data size in bytes ...................................................... 121

Table 37: Ratios to Xnode for Packet size ...................................................................... 121

Table 38: Mean values for Ratio ..................................................................................... 121

Table 39: Ratios to Xnode for Ratio of sizes .................................................................. 123

Table 40: Mean values for RT cost figure ...................................................................... 123

Table 41: Ratios to Xnode for RT cost figure................................................................. 123

Table 42: The fixed types ............................................................................................... 206
                                                                                                                         11
Table 43: The fixed types ............................................................................................... 215

Table 44: The data structures .......................................................................................... 219

Table 45: Run-repeat data ............................................................................................... 236
                                                                                12
                                      LIST OF FIGURES


FIGURE                                                                        Page

Figure 1 An XML example using child nodes                                      25

Figure 2: An XML example using attributes                                      25

Figure 3: An XML example showing the use of a namespace                        26

Figure 4: An example of a C structure                                          31

Figure 5: A packet encoded with fixed length fields                            31

Figure 6: The IP packet header                                                 68

Figure 7: Tree node class structure                                            88

Figure 8: Serializer and parser classes                                        89

Figure 9: Experiment class structure                                           92

Figure 10: Pseudo-code for serializing engine data.                            96

Figure 11: An example of pseudo code to extract data from the encoded form.    98

Figure 12: The email data set                                                 101

Figure 13 Pseudo-code for generating test runs                                103

Figure 14: Document create times                                              124

Figure 15: Document render times                                              125

Figure 16: Document parse times                                               125

Figure 17: Document extract times                                             126

Figure 18: Document write times                                               126

Figure 19: Document read times                                                127

Figure 20: Document total times                                               127
                                       13
Figure 21: Document natural sizes     128

Figure 22: Document packet sizes      128

Figure 23: Document size ratios       129

Figure 24: Document RT cost figures   129

Figure 25: Email create times         130

Figure 26: Email render times         130

Figure 27: Email parse times          131

Figure 28: Email extract times        131

Figure 29: Email write times          132

Figure 30: Email read times           132

Figure 31: Email total times          133

Figure 32: Email natural sizes        133

Figure 33: Email packet sizes         134

Figure 34: Email size ratios          134

Figure 35: Email RT cost figures      135

Figure 36: 3D mesh create times       136

Figure 37: 3D mesh render times       136

Figure 38: 3D mesh parse times        137

Figure 39: 3D mesh extract times      137

Figure 40: 3D mesh write times        138

Figure 41: 3D mesh read times         138

Figure 42: 3D mesh total times        139
                                           14
Figure 43:3D mesh natural data sizes      139

Figure 44: 3D mesh packet sizes           140

Figure 45: 3D mesh size ratios            140

Figure 46: 3D mesh RT cost figures        141

Figure 47: Audio create times             142

Figure 48: Audio render times             142

Figure 49: Audio parse times              143

Figure 50: Audio extract times            143

Figure 51: Audio write times              144

Figure 52: Audio read times               144

Figure 53: Audio total times              145

Figure 54: Audio packet sizes             145

Figure 55: Audio size ratios              146

Figure 56: Audio RT cost figures          146

Figure 57: Web query create times         147

Figure 58: Web query render times         148

Figure 59: Web query parse times          148

Figure 60: Web query extract times        148

Figure 61: Web query write times          149

Figure 62: Web query read times           149

Figure 63: Web query total times          150

Figure 64: Web query natural data sizes   150
                                                    15
Figure 65: Web query packet sizes                  151

Figure 66: Web query size ratios                   151

Figure 67: Web query RT cost figures               152

Figure 68: Web query response create times         153

Figure 69: Web query response render times         153

Figure 70: Web query response parse times          154

Figure 71: Web query response extract times        154

Figure 72: Web query response write times          155

Figure 73: Web query response read times           155

Figure 74: Web query response total times          156

Figure 75: Web query response natural data sizes   156

Figure 76: Web query response packet sizes         157

Figure 77: web query response size ratios          157

Figure 78: web query response RT cost figures      158

Figure 79: Spreadsheet create times                159

Figure 80: Spreadsheet render times                159

Figure 81: Spreadsheet parse times                 160

Figure 82: Spreadsheet extract times               160

Figure 83: Spreadsheet write times                 161

Figure 84: Spreadsheet read times                  161

Figure 85: Spreadsheet total times                 162

Figure 86: Spreadsheet natural data sizes          162
                                                           16
Figure 87: Spreadsheet packet sizes                       163

Figure 88: Spreadsheet size ratios                        163

Figure 89: Spreadsheet RT cost figures                    164

Figure 90: TLV1 packet structure                          202

Figure 91: Container node tag structure                   204

Figure 92: Leaf node tag structure                        205

Figure 93: The TVM1 general form                          211

Figure 94: A packet node containing fixed-length fields   212

Figure 95: Container node tag structure                   213

Figure 96: Leaf node tag structure                        214

Figure 97: A node hierarchy                               221

Figure 98: The document data structure                    222

Figure 99: The email data structure                       223

Figure 100: The 3D mesh data structure                    225

Figure 101: The audio data structure                      226

Figure 102: The web search query data structure           227

Figure 103: The web query response data structure         229

Figure 104: The spreadsheet data structure                230

Figure 105: System information                            232

Figure 106: Statistical data                              237

Figure 107: Distribution of total time for ten runs       237

Figure 108: Statistics with outliers excluded             238
                                                      17
Figure 109: Run data with outlier excluded           238

Figure 110: Descriptive statistics for 20ms loops    239

Figure 111: Histogram of loop times for 20ms loop    240

Figure 112: Time series plot for 20ms loops          240

Figure 113: Probability plot for 20ms loops          241

Figure 114: Descriptive statistics for 2ms loops     242

Figure 115: Histogram of 2ms loops                   242

Figure 116: Time series plot for 2ms loops           243

Figure 117: Probability plot for 2ms loops           243

Figure 118: Descriptive statistics for 200ms loops   244

Figure 119: Histogram for 200ms loops                245

Figure 120: Time series plot for 200ms loops         245

Figure 121: Probability plot for 200ms loops         246

Figure 122: Descriptive statistics for 2s loops      247

Figure 123: Histogram for 2s loops                   247

Figure 124: Time series plot for 2s loops            248

Figure125: Probability plot for 2s loops             248
                                                                                         18
                                       CHAPTER I

                                    INTRODUCTION


1.1    Background

       Many applications have a need to store or communicate information that has

many parts, often organized in some sort of hierarchical tree. Users of these applications

would like to be able to read data written by an older or newer version and to exchange

data between different versions. The application might be a word processor, a web

interface or a satellite data collection system. For all of these, upgrading the application

should not make existing data inaccessible.

       XML has emerged as the common way to encode hierarchical, self-describing

data. XML is attractive because its text-based form is easy to generate, it can be directly

decoded by a human reader and it has multiple ways of encoding data so that application

creators can choose a form that most simply supports their needs.

       As the use of XML has expanded from its roots in web interfaces to many other

applications some limitations have come to light. XML encodings can be very verbose

when compared to the data being encoded. This has two direct effects: the size of the

encoded data is larger which requires more storage space or network bandwidth to

transport it and the amount of CPU time required to create the encoded structure and

recover data from such a structure are also increased.

       Recent work by organizations such as the W3C and various researchers has

identified a need for a binary alternative to XML for use in some applications [1].
                                                                                       19
1.1.1   Data formatting requirements

        In order to determine the optimal way to encode data used by a particular

application we must first examine some basic requirements. Applications need data

structures which are well defined so that when data is written by one instance of the

application it can be read correctly by another. The concepts of read and write here apply

equally to disk files or to a network communications link.

        At the most basic level, scalar types such as integers and floating-point numbers

need to be encoded in well-defined way. Some standards such as ASN.1 have multiple

ways to encode simple data types such as floating-point values [2]. Allowing multiple

formats provides choices for the writer of the data but imposes extra cost on the reader

which must be prepared to decode all the possible forms.

        Data that has structure to it such as a document, spreadsheet or a list of sampled

sensor values must include a way to define the structure. In some systems, the definition

of the data structure is built into the application when it is created. Systems using

CORBA use an Interface Definition Language (IDL) to specify the exact nature of the

structure [3]. The writer and reader of the data must both have been constructed using the

same IDL information. This very tight coupling between applications is not desirable. It

forces the reader of the data to have a complete understanding of the data structure and

constrains the writer of the data only to write in that exact format.

        If each data item is given a unique tag of some kind such as a numeric value or

string of characters, that data becomes self-describing to some extent. A simple example

can be found in the automotive industry. The LIN bus is used in cars and trucks to
                                                                                        20
provide a way to monitor values such as engine RPM and to allow the control of simple

devices such as seat heaters and positioning motors [4]. The LIN bus protocol allows a

device such as an engine RPM sensor to send a data packet that contains a numeric

identifier signifying that the packet contains engine RPM data and an encoded value for

the actual RPM. The value may be encoded in any format so long as it does not exceed

eight bytes in length. A receiver of this data such as a dashboard tachometer or an engine

management computer monitors packets on the bus looking for the engine RPM tag.

When a packet with the matching tag is found, the RPM value is decoded from the body

of the packet. This allows the writer (the sensor) and the reader (the tachometer) to be

decoupled to some degree. Neither application is affected by other traffic on the bus. The

LIN specification does not define application level identifiers or the format of any

application-specific data such as engine RPM. Manufacturers are free to choose

identifiers that suit their own needs and encode data in any format they desire. This

allows a car manufacturer great freedom in how data is exchanged on the bus in the car it

makes, but makes it very difficult for any other application to take advantage of that data

since the application must know exactly how that particular manufacturer encodes each

piece of information. A vendor wishing to make a data telemetry system for race track

use would need to know how every car manufacturer tagged and encoded the engine

RPM data in order to sell a product that could be used with any manufacturer‟s car.

       Tagging data has the advantage that an application which can recognize the tag

can extract the data it needs and ignore any other tagged items which might be present. If

we apply the idea of tags to data structures then we can construct nested data structures
                                                                                          21
(trees) in which each node has a unique tag. This is the mechanism used by XML. XML

consists of a hierarchy of nodes in which each node has an identifier. In the case of XML

nodes, the identifier is one or more characters in a string. Extending the engine

monitoring example, we might expect an XML node with the name “Engine RPM”.


1.1.2   Loose coupling between applications

        As a general principle, we would like to be able to write data with version N of

some application and be able to read it with version N+1, N+2 and so on. We might also

like to be able to write the data with version N and read it with version N-1, N-2 and so

on. This idea of forward and backward compatibility of data between versions of an

application is particularly important when the application may have thousands or millions

of instances that are distributed to a large number of users. In such a situation, it is not

generally possible to upgrade all users at once to a new version. Even if such an upgrade

were possible, we would not want to have to upgrade all data files created by those

applications.

        The same issue applies to networked applications which send data to each other.

If we create a service for use on the Internet we need to be sure that we can upgrade the

client application employed by end users asynchronously from any upgrades we might

want to do to the servers providing the service. When we upgrade the server side, the

server may send out data that the client application has never seen before and could not

have been tested with. In this case, the client must be able to ignore the new data and still

function correctly. We assume that the server is still sending the old data as well as the

new. If a new version of the client software is made available, we might want to
                                                                                           22
distribute that before all the server upgrades are in place. The new client would be

receiving old data for some period of time and we would like it to behave correctly even

in the absence of the new information it was designed to receive and use.


1.1.3     Parsing data in a loosely-coupled environment

          To understand the issues involved in parsing data in a loosely-coupled

environment it is helpful to examine a scenario in which all participating applications use

a single data format and some scenarios in which we might want to alter the data being

exchanged between the applications without modifying all the participants


1.1.3.1      Scenario 1

          Consider an application which is based on CORBA and has been constructed

using an IDL specification. The application is sent out to multiple organizations (such as

banks) which use it for internal transactions and also for business between organizations.

Once the system is deployed and working, any change to the data structures exchanged in

a particular transaction typically requires a new version of the application to be

distributed to all users. It is possible to add new functions and new data structures that are

associated with the new functions but this does not allow an organization with the new

functionality to use it in a transaction with an organization using the older product.


1.1.3.2      Scenario 2

          An application has been developed to show information such as engine

operational data. The first version displays engine RPM and temperature because that is

all the data available from the engine sensors. New sensors are added to the engine to
                                                                                          23
measure oil temperature and pressure. We would like the application to be able to show

the new data without upgrading it. The application was shipped before the new sensors

were developed so we cannot have tested it with them.

          This forward compatibility feature is possible if the application has some way to

look for a collection of data and for each item in the collection to be able to decode it and

display it. A unique tag can identify the collection (engine parameters) and separate tags

identify the individual items of data. Each individual item might be tagged with

additional information about the type of data and with hints on how to display it. In our

example we might present temperatures with values that represent tenths of a degree

Centigrade. The temperature items include a scale factor value and a string that indicates

the units of measurement. Using such a system for encoding the data, the application can

display data without knowing anything about the individual items of data.


1.1.3.3      Scenario 3

          We have sold a system that consists of a server and a number of client

applications to a customer. The client application and the server are revised in a new

version of the product. The customer needs to evaluate the new client before committing

to it and wants to do this with the existing server. The new client must be created to work

in such a way that it is not dependent on the data coming from the new server.

          If the server is a new engine data telemetry system and the client is the display

and recording application, the client may be capable of displaying the oil temperature and

pressure but the server in use does not send this data. The client must either show the new

data fields marked as „unknown‟, leave them blank or perhaps not display them at all.
                                                                                        24
1.1.4    How XML is used

         XML is based on a human-readable text format that allows the definition of a

hierarchy of nodes or elements. Each node has a textual name associated with it. Nodes

may have attributes and may also have child nodes. The start and end of node are marked

by the use of special characters (angle-brackets). A few other characters are treated in

special ways but for the most-part XML is a fairly free-form textual encoding.

         XML has proven useful in applications which use a lot of text but less so in cases

where there is either a lot of numeric data such as scientific computing applications or

where the majority of the data is binary. Binary data includes photographs, video, audio,

radar images and the like. This leads to more specific questions such as: Is XML the best

format for audio data? Is XML the best format for scientific computing?

         Numeric data is encoded as text in the same way we would expect to see it

printed. So the number twenty-five is encoded as the character string “25”. Strings of text

appear nominally as they would if printed with the exception that some characters are

encoded to avoid conflicts with the characters used to define nodes. The string sequence:

“ab<cd” must be encoded as “ab&lt;cd” so that the „<‟ character in the string is not

interpreted as the start of a new node.

         The nodes in an XML encoding have textual names. We might, for example, use

common words to describe the car engine data node such as “RPM”, “Temperature” and

so on.

         Nodes may have named attributes associated with them so that we can have a

“Car” node which has a “Model” attribute. We could equally well create a child node
                                                                                           25
called “Model” in which we keep the model description. XML provides a lot of

flexibility in how structured data is encoded. Figure 1 shows an example of how XML

might be used to encode engine data using child nodes for each item.



<EngineData>
      <RPM>2500</RPM>
      <Temperature>140</Temperature>
</EngineData>

                        Figure 1 An XML example using child nodes

       Figure 2 shows the same data but encoded using attributes in the XML node

rather than as the content of a set of child nodes.



<EngineData RPM=”2500” Temperature=”140 />

                        Figure 2: An XML example using attributes



       An examination of the data in Figure 1 shows that the “RPM” tag is unique within

the scope of the “EngineData” node but it is obvious that it might not be unique in a

larger context. If a tree of nodes is to be self-describing to the extent that a sub-tree can

be extracted from it and still be interpreted correctly even if inserted into a different tree

at some point, then tags must be globally unique. XML uses namespaces as a way to

solve this problem [5]. Figure 3 shows an example of how a namespace can be used to

fully qualify node names.
                                                                                      26
<bc:bc xmlns:bc=’http://www.bobs.com/carstuff’>
<bc:EngineData>
      < bc:RPM>2500</RPM>
      < bc:Temperature>140</Temperature>
</EngineData>

                 Figure 3: An XML example showing the use of a namespace



        The use of the namespace prefix to the tags provides unambiguous scoping for

those tags.

        XML provides a means to encode self-describing hierarchical data. It is easy to

generate and understand. A parser can skip over nodes it does not recognize and the use

of namespaces allows unambiguous tagging of individual items of data. When

namespaces are used, sub-trees of XML nodes may be taken from one source, combined

with other sub-trees into a new data tree with no loss of meaning of any item in the

combined tree.


1.1.5   Problems with XML

        Data encoded with XML can be very verbose. If we look at the engine data

example shown in Figure 1, ignoring whitespace there are 65 characters in the encoding

that are not part of the data. Only seven characters represent the numeric values. The 65

other characters are used to identify nodes (tags) and to represent the hierarchy. So for

this example only about 11% of the encoded data is the information. We can take this a

bit further and look at the actual data values. The RPM value of 2500 can be represented

in a 16-bit unsigned integer field and the temperature value of 140 could be represented

in an unsigned 8-bit field. So we need only 3 bytes to represent the data values. The
                                                                                          27
complete packet in Figure 1 uses 72 bytes. Since XML is a human readable form, it is

common to add whitespace characters such as spaces, tabs and new-lines to make it look

like it does in Figure 1. The indentation aids with understanding the hierarchy and using

one line per node aids with clarity. The example in Figure 1 would thus include two tab

characters and four new-line characters or possibly six or eight spaces instead of the tabs

making it even less efficient.


1.1.5.1      Latency times

          The larger the encoded packet of data, the more storage required for it, the longer

it takes to send over a network link and the larger the latency between sender and

receiver. Latency is important to consider because it affects how quickly a receiver can

respond to a command sent to it. If we imagine a microcomputer module monitoring

engine RPM and feeding that data to an engine management system over a 100 kpbs LIN

bus, the entire packet shown in Figure 1 would take about 6 milliseconds to send which is

about 90 degrees of rotation of the crankshaft at 2500 RPM. In a car engine management

system, such a latency is not important since engine control does not depend on

instantaneous RPM measurement but if the system were used to manage something more

critical such as a fly-by-wire supersonic aircraft, the time taken for measurements to get

to the control computer and out again to the aircraft‟s control surfaces is part of the

feedback system that keeps the aircraft stable. Large latency times make the aircraft un-

flyable. Increasing the network speed is an obvious answer but the some improvement

can also be obtained by reducing data packet size.
                                                                                                    28
           If we used a single byte to identify each node in Figure 1, two bytes for the RPM

and one byte for the temperature, we would need only six bytes. In practice we also need

to either include a length value for each node or and end marker for the data for each

node. Let‟s say we include a one-byte length value in each case. The total packet size is

now 11 bytes which is about eight times smaller than our XML example. Increasing a

network‟s speed by eight times would be considered a significant improvement. If we can

improve network performance by the same amount by simply changing the way we

encode the data we can improve the system with no change to the physical

implementation.


1.1.5.2        CPU costs

           To convert the internal representation of some data type to its encoded form takes

some number of CPU cycles as does locating the data we need in an encoded packet and

converting it back to the native form needed by the application. We can improve the

performance of an application by reducing these conversion costs. Data encodings used

to exchange information between heterogeneous systems cannot simply send native

format data because all recipients would need to know all possible native formats that

could be sent to them1. So data which is exchanged between heterogeneous systems

either in the form of files or network packets must be encoded according to rules which

allow the recipient to correctly decode the data.

           Textual data is probably the simplest to encode. We ignore here the issue of

language. It is not expected that a sender writing French content convert it to German for

1
    CORBA does do this for floating point values, leaving the conversion problem to the receiver.
                                                                                           29
transmission. We assume that the encoded character set (US-ASCII, UNICODE, UTF-8)

is understood by all senders and receivers. It is becoming common to encode text in UTF-

8 because it supports all the US-ASCII character set and all the 16-bit UNICODE set. It

has the advantage that because it uses variable length encodings for individual characters,

it is very efficient for US-ASCII, using only one byte per character as opposed to

UNICODE which uses two bytes per character. UTF-8 is defined in RFC 3629 [6].

           When text encoded in UTF-8 is used there are two common methods for encoding

an item. The first approach is to include a prefix that contains the length of the string. The

second approach uses a null value (0x00) to terminate the string. This latter method is

used in the C language for strings and although defined originally for ASCII characters

only, also works with UTF-8 which accounts at least partly for the popularity of UTF-8.

In either case, the number of bytes required to encode the data is only one or two more

than the length of the original string. Since text tends to consist of sequences of ten to

thousands of characters, the encoding efficiency is very good, tending to 100% for long

strings.

           Encoding integer data introduces a number of issues. Computer systems tend to

use the native CPU word size for integers. This has varied from 8 to 128 bits over time

with many variations in between. Computer manufacturers are free to store data in any

physical format they choose but two forms have become popular. Integers are stored as

2‟s-complement values with either the most significant byte first in memory or the least

significant byte first. Most significant byte first is sometimes referred to as the big-endian
                                                                                       30
or Motorola format and the least significant byte first form as the little-endian or Intel

format. These formats are not limited to computers made by Intel and Motorola.

       It has been conventional in network data structures to use the big-endian format

since this was the common format on most Unix platforms when Internet protocols were

first developed. We can avoid the endian problem completely by encoding integers as

strings. So the value one hundred and forty is encoded as “140”. This is how integer

values are encoded in XML. If we choose to represent the value as a binary number, we

have many choices, some of which are shown in Table 1.




                     Table 1: Various encodings of the integer value 140

           Bits                    Big-endian              Little-endian

           8 bits                  8C                      8C

           16 bits                 008C                    8C00

           32 bits                 0000008C                8C000000



       The use of fixed-length fields for the tags and length values allows for a very

simple implementation when a device needs to send only small packets of data. As an

example, consider the engine parameter data in Figure 1 encoded using a TLV scheme in

which the tag and length fields are one byte long and the data values are two bytes. We

can describe this structure in the C language as shown in Figure 4.
                                                                                      31
typedef struct {
      BYTE engParamTag;                         // packet tag
      BYTE engParamLen;                         // packet length
      // Temperature
      BYTE engParamTempTag;                     // Temp tag
      BYTE engParamTempLen;                     // Temp length
      UINT16 engParamTemp;                      // Temp value
      // RPM
      BYTE engParamRpmTag;                      // RPM tag
      BYTE engParamRpmLen;                      // RPM length
      UINT16 engParamRpm;                       // RPM value
}; ENG_PARAM_STRUCT

                          Figure 4: An example of a C structure



       Using a fixed structure can be of considerable advantage to microcontroller

applications as they do not need to compute packet node lengths in order to transmit the

data correctly formatted. If we encode the data using the structure in Figure 4 with big-

endian values for the RPM and temperature fields, the entire encoded packet would be as

shown in Figure 5, where PP is the tag for the parameter node, TT is the tag for the

temperature node and RR is the tag for the RPM node.




                  Tag   Len   Tag    Len    Value    Tag    Len    Value

                 PP     08    TT     02    00   8C   RR     02    02   09


                                           Temp                    RPM

                                    Temp Node              RPM Node

                                     Complete packet

                    Figure 5: A packet encoded with fixed length fields
                                                                                        32
       It is also possible to encode integers in variable-length fields as is done in ASN.1

DER [2]. The format uses the least number of bytes required to encode the value and

effectively encodes seven bits of data into each encoded byte. So values up to 127 occupy

only one byte, up to 16,384 occupy two bytes and so on.

       The choice of encoding affects how much CPU time is required to generate and

parse it. Converting native integer formats to and from text is a common operation in

computer languages and the conversion routines have been made as efficient as possible.

The textual format of integer values has an advantage that it uses only as many characters

as necessary. The value 140 needs three characters. This is not as efficient as using one

byte in ASN.1 DER but is better than a scheme requiring all integers use a 32-bit field (4

bytes). If the majority of computers on a network use a common format such as the 32-bit

little endian form found in many computers running 32-bit Windows, then it could be

advantageous to choose that format since it requires no conversion from native to

encoded form. Computers not using this format (the Mac for instance) would need to

convert to and from this format. This is the same penalty Intel CPU-based machine pay

when using common network protocols since those use the big-endian format. The

CORBA IIOP protocol uses the idea that the sender picks the format of data and the

receiver must convert it. This works very well in homogeneous networks because no

conversion is required by either the sender or the receiver [7].

       The encoding of floating-point values is of particular importance in scientific

computing applications because those applications typically deal with very large amounts

of data and the encoding and decoding costs can be the majority of the cost of reading
                                                                                          33
and writing the data. There are a large number of floating points formats but two have

become popular: The IEEE single precision and double precision formats [8]. The single

precision format uses four bytes and the double precision format uses eight bytes.

Unfortunately these are not the format that some languages choose to store floating point

values in although language compilers now commonly include options to use a particular

floating point format as part of their library packages.

        As with integer values, floating point values may be converted to text for storage

or transmission. XML uses a text format and ASN.1 DER includes an option to use text

for floating point values as an alternative to its own internal form [2]. The text format can

be more or less efficient than a binary encoding depending on the numeric value. For

example, zero can be represented as a single character “0”. Very large or very small

values encoded using scientific notation in text can be longer. “-1.23E-8” is eight

characters but might have been a single-precision value occupying only four bytes on the

machine it came from. Converting floating point values to and from text is another

common operation that has been well optimized. Converting from one binary format to

another such as from IEEE double-precision to ASN.1 DER binary might require a

custom conversion routine which is less efficient.


1.1.6   Possible approaches

        What constitutes the best format is a combination of measurable values like the

size of the encoded format and the amount of time it takes to create it, as well as how

easy it is for an application developer to make use of it. XML owes at least part of its

popularity to the fact that the encoded form is human readable. This drives the perception
                                                                                        34
that XML is simple to generate (since we have many trivial ways to programmatically

generate text) and that applications using it are simple to debug since the encoded form

can be directly examined by the developer.

         Reducing the size of the encoded form improves performance because less disk

space is required to store the encoded data, less network bandwidth is required to transmit

it and for a constant network speed, latency is reduced between sender and receiver.

Protocol encodings which require less meta-data than XML result in smaller encoded

forms.

         Reducing the amount of time required to create the encoded form from the native

data and to recover the native data from the encoded form have direct effects on an

application‟s performance. Replacing XML with a binary encoding format can help

improve performance because binary encodings are typically less verbose than XML and

so create smaller encoded forms. Binary encoding formats can also be simpler to

implement. XML has many options for how data may be encoded which makes a parser

more complex. This complexity can lead to reduced performance when compared to an

encoding with a smaller and simpler set of rules.


1.2      Problem Definition

         The performance of applications that currently use XML-like protocols can be

improved either by improving the way XML is used or by replacing the XML encoding

with something more efficient.
                                                                                       35
1.2.1   Improving XML performance

        The performance of XML can be improved in several ways. When XML is

created, it can be done so almost without any white space. Tags can be contracted from

long English words to abbreviations. Sensible use of attributes instead of child nodes can

also help. The whole data structure can possibly be compressed, and parsers can be

improved by using better techniques to reduce parsing costs. Added all together, this

makes XML somewhat more efficient but reduces its ease of use. This area of research

has been covered extensively by others [9-14].


1.2.2   Alternatives to XML

        Instead of trying to modify XML to improve its packing efficiency and reduce

parsing costs we can consider the use of other encoding methods. The requirements are

that we can create a hierarchy of data in which nodes are self-describing. The encoding

must support the data types an application needs such as text, numeric data and blocks of

binary (for images and sounds).

        An additional requirement worth some consideration is the ease with which the

encoding can be implemented and used. Part of XML‟s popularity lies with the fact that it

is very easy to generate [15]. Protocol encodings like ASN.1 DER are complex and non-

trivial to verify [16-18]. ASN.1 BER (Basic Encoding Rules) was superseded by the DER

(Distinguished Encoding Rules) largely because BER offered too many ways to encode

the same thing and this made verification of parsers to complex or expensive. DER limits

the way each data type can be encoded and thus makes it easier and cheaper to implement

and verify a parser [17].
                                                                                         36
1.2.2.1      Encoding a data tree

          Encoding a tree of data in a serial form can be done in many ways. Two common

approaches are Tag-Length-Variable (TLV) and Tag-Variable-Marker (TVM). In a TLV

scheme, each node consists of a tag followed by a length value which describes the entire

body of the node (the variable part). The body may or may not have child nodes. In a

TVM scheme each node starts with a tag which is followed by the variable part and

terminated with another tag. The variable part (the body) may contain data or child nodes.

XML is an example of a TVM scheme. ASN.1 is an example of a TLV scheme.

          TLV is attractive because parsing it is simple. If the parser does not recognize a

tag it uses the length data to skip over that node to the next tag. Parsing a TVM scheme

means that for an unknown node, the parser must still be able to parse all contained child

nodes in order to determine where the end of the unknown node occurs. Constructing

TLV data requires the length of each node to be known before the outer node can be

generated so typically the tree of data must be traversed once to determine the length of

every node and again to serialize it. A TVM scheme can serialize the data with only one

pass of the data tree because no length values are required.

          The body of this research work examines several binary protocols based on TLV

and TVM schemes and compares them to XML in terms of their CPU cost and the size of

the encoded form.


1.2.2.2      Encoding different data types

          Any encoding which is to be generally useful must support the encoding of text,

numeric values in both integer and floating-point form and blocks of binary data. The
                                                                                          37
binary form is required to support sounds, images and video segments. These could all

have specific encodings and there are many examples such as the JPEG specification for

images [19] but for the purpose of comparing efficiency it is sufficient to consider all of

these formats as opaque blocks of binary data.

          Data items can be encoded into fixed-length fields or variable-length fields. Data

such as text is naturally encoded as a variable length field but the choice is less clear for

numeric data. Integers may be encoded in variable length fields from one to N bytes but

floating-point formats are most commonly viewed as fixed-length objects (four and eight

bytes being common sizes). Binary data is again naturally a variable length item. Since

all data in an XML packet is text (including binary data represented typically in the

Base64 character set), all XML fields are variable length. ASN.1 DER uses variable

length fields for tags and many data items but some are of fixed length.


1.2.2.3      Determining performance

          For any given encoding and given piece of data, we can determine exactly how

much space the encoded form will take based on the encoding rules. For data items such

as integers, the encoded form may be a fixed size such as four bytes or it may be variable

from one to eight bytes (for example) depending on the encoding rules and the range of

values supported. Where the encoded size depends on the value encoded we can model

the minimum space required and the maximum space required.

          Determining the amount of CPU time required to encode a data tree or to decode

the tree from its encoded form is possible by modeling the protocol, a compiler and the
                                                                                         38
runtime environment or by conducting and experiment. The approach taken here is to

conduct experiments which are cheaper and less prone to errors in the modeling.


1.2.3   Meaningful Measurements

        Part of the problem in comparing protocols is to determine what measurements

have useful meaning. For example we can measure the time to encode a data set

according to the protocol's encoding rules but that absolute value has no meaning. We

need to compare that value to some other time such as the time taken to encode the same

data using a defined set of XML encoding rules. The ratio of the two times gives us a

meaningful measure of the time cost of the encoding with respect to XML.

        When the encoded packet size is measured, the value is also meaningless without

knowledge of the data that was encoded. We can obtain a more meaningful measurement

by comparing the size of the encoded form to the size of the data in its natural form in the

computer's memory. The ratio of the encoded size to the natural size gives us a measure

of how much meta-data is required to describe the packet structure. We can compare the

ratio of one protocol with another to see how well the protocols pack data into the

encoded form.

        In order to obtain a single measure for the performance of a protocol, we must

combine both the time aspect and the space aspect into a single figure. For this work we

derive a total cost value as follows. For any given protocol and data set we measure four

times: The time taken to convert the raw data into its hierarchical tree form - the create

time; the time taken to render the node tree into the protocol's encoded form – the

rendering time; the time taken to extract the node tree from the encoded form – the parse
                                                                                        39
time and the time taken to extract the raw data from the node tree – the extract time. The

create time, render time, parse time and extract time are summed to provide a total

execution time. This gives us a measure of the time cost. For each protocol and data set

combination we also need a measure of the size of the encoded form. For this work we

measure the native size of the data in bytes and the encoded size in bytes and compute the

ratio of encoded size to native size. This gives us a figure for space cost. Most protocol

encodings will have space costs of greater than one since the meta data used to describe

the data hierarchy takes space in the resulting encoding. If compression is used in the

encoding then the space cost may be less than unity. Compression is not considered in

this work.

       The overall cost figure is obtained by multiplying the total time cost by the space

cost. This gives us a figure which increases with time and encoded packet size. We do not

apply weighting to time or space costs. There is some degree of interaction between the

space and time cost figures since larger packets tend to take longer to construct and parse

so a protocol that requires a lot of space for the encoded form is also likely to show

increased time costs. Reducing encoded size by using compression generally increases

the CPU time cost since more work need to be done to generate the encoded form and to

parse it from the encoded form so compression may yield smaller packets at the cost of

some CPU time. If compression can be done externally to the main CPU such as in

special hardware then the overall cost including the compression costs would be lower

since there would be less data in the encoded form and the CPU overhead required for

compression and decompression would be limited to that required to interface with the
                                                                                          40
special hardware. In this work, all protocols are encoded without any compression and

the benefits of compression are not considered.

       The single cost value based on the product of total time cost and space cost is the

basis for evaluating alternatives to XML in the following hypothesis.


1.3    Hypothesis

       The hypothesis for this work is:

       H: Applications which currently use XML as a data encoding can lower costs by

using an alternative binary encoding form.

       The associated Null-Hypothesis is:

       H0: Applications which currently use XML as a data encoding would see the

same or higher costs by using an alternative binary encoding form.

       It should be clarified here that the term cost is not monetary cost but rather the

cost to the application in terms of CPU usage, storage space and network bandwidth.

These things do, of course, also affect the financial costs incurred in running applications.


1.4    Research Questions

       The essential question to be examined is: Is XML the best way to encode

hierarchical, self-describing data? This question can be broken into a number of questions

related to methods of measuring the cost of encoding and decoding data, appropriate data

sets to be tested and alternative protocol encodings to be evaluated.
                                                                                        41
1.4.1    Measuring protocol performance

         Can we find a way to measure protocol performance in such a way that we can

determine if one encoding is better than another for a given set of data? There are two

values that we can measure simply that relate to protocol performance: the time it takes to

perform the translation of the raw data to or from the protocol's encoded form, and the

size of the encoded data structure. Longer times indicate lower performance. Larger

encoded packets indicate lower performance too because larger packets take longer to

transmit over a network or to write to storage and in the later case occupy more storage

space.


1.4.2    Fairness

         How can we ensure that we are measuring the effect of the protocol encoding and

not the effect of the code that implements it? Measurement of the size of the encoded

form is trivial and should be completely determined by the protocol's encoding rules. If

we take any known set of data and encode it according to the rules, we always get the

same result. We get the same byte sequence and it is always the same length. Some

protocols (notably XML) have optional ways of encoding the same data so the results can

vary according to the choices made within the protocol's set of encoding rules. Two

applications using XML may produce different encoded forms for the same data by

making different choices about how to encode that data. If we are to compare protocols

by examining the size of the encoded data, we need to be sure to note any choices that are

made about how the encoding is used.
                                                                                         42
        Measuring the execution time required to encode or decode a data block with the

protocol's encoding rules is highly dependent on how the implementation is done. In an

extreme case we might compare one protocol written by a novice programmer in a non-

optimal language such as Visual Basic with another protocol written by an expert team

that makes use of special hardware to accelerate generation and parsing of the protocol

packets. To make a fair assessment of the CPU cost of one protocol's encoding versus

another we need to implement each protocol using the same language and methodology

and run the tests on the same environment under the same operating conditions. This

perhaps seems like an obvious point but many of the early critics of XML compared

dissimilar implementations operating on the same data and drew general conclusions

about XML performance from those studies [20].


1.4.3   Determining appropriate data sets

        Can we define data sets that represent the data we are interested in applying the

protocol encoding to? The W3C conducted an extensive study to examine the types of

data that are in common use by applications currently using XML as the encoding [1, 21].

The study identified a number of application use cases ranging from supercomputing to

military to general web services. For each use case a set of must have, should have and

nice to have properties were identified. For this work a very limited set of use cases were

considered which in each case focus on a single type of data. The basic data types being:

numeric, text and binary.

        Since the performance of a protocol encoding can be affected by the type of data,

it is important to identify data sets that accurately represent those that would be employed
                                                                                       43
in real-world applications. This is especially important with hierarchical data since the

total content is not one simple type but consists of some mixture of the primary type

(perhaps numeric data for a scientific computing application) and also perhaps some text

in the form of captions, comments and so on. Even when the data is binary as it is for an

audio segment, for example, there is still a need to identify the composer, the time the

recording was made and other similar data.

        The performance of a protocol can also be affected by the actual data values in

some cases. Consider the case where numeric values are encoded using XML. If we have

a lot of numbers and all the values are zero, then the minimum encoding is the single

character: '0'. If the numbers are large or need to be represented in scientific notation

because they are either very large or very small, then the character sequence length is

greatly increased. So in measuring protocol performance with numeric data, it is

important to generate random values within the range expected by the application.


1.4.4   Determine alternative encodings to test

        Can we identify protocol encodings that have better performance than XML for

the same data sets? Several researchers have compared ASN.1 with XML and shown that

ASN.1 has better performance both in terms of encoded size and CPU cost [11, 20, 22].

These studies were often for a particular type of data but there is enough data to suggest

that ASN.1 is worth further investigation. ASN.1 uses a tree of nested tag-length-variable

(TLV) structures and this leads to the question of whether a generic TLV solution might

be better than XML. ASn.1 has many encoding options and options tend to slow down

the parser. If we select an optimal set of ASN.1 choices and write a generic TLV
                                                                                         44
encoding based on those, the result might be better than a general ASN.1 solution. XML

uses a tag-variable-marker structure with each node having well-defined start and end

tags. We might take that approach for a binary protocol and design a generic TVM

encoding that uses binary start and end tags with variable-length data between them.

        For this work four encodings were evaluated: XML, ASN.1 DER, a generic TLV

implementation and a generic TVM implementation. These are described in detail below.


1.5     Proof Criteria

        In this section we look at the null hypothesis, a set of proof criteria and measures

of repeatability. The proof criteria are broken into two sections: completion criteria that

establish an end point for the research work and success criteria that establish that the

null hypothesis is rejected.


1.5.1   Null hypothesis

        As stated above, the null hypothesis is:

        H0: Applications which currently use XML as a data encoding would see the

same or higher costs by using an alternative binary encoding form.

        In other words: changing from an XML-based encoding to a binary form will not

improve application performance and may even degrade it.


1.5.2   Completion criteria

        The following criteria were used to establish completion of the work.
                                                                                 45
   The set of protocol encodings to be tested has been identified and described in

    sufficient detail that the protocol encoding and decoding code can be

    implemented correctly.

   A set of data types has been identified that all tests will be applied to. These

    must be described in sufficient detail that the data generation code can be

    written from the specifications.

   A framework for the tests has been established and a methodology defined so

    that for each data type and each protocol encoding, the code written will take

    as similar an approach as possible.

   All the data generation code has been written and tested to ensure that the

    generated test sets conform to the specifications for the data sets and that the

    content is generated randomly.

   All the protocols to be tested have been implemented in the program

    framework and have been tested against the protocol specification so that we

    can be confident that the code produces correct encodings and parses the

    encoded form correctly. The testing can be partly automated in code but will

    require some inspection of the encoded forms.

   An initial test run is completed to show that the implementation works

    correctly and that the data logged is reasonable. The test program shows no

    obvious bad behavior such as memory leaks. No obvious anomalies are

    observed that might indicate that the implementation of any protocol or data

    set generator is incorrect.
                                                                                         46
1.5.3     Success criteria

          The following criteria are used to establish that the null hypothesis has been

rejected.

             For the majority of the data types tested, the binary encodings show reduced

              costs compared to the costs measured for XML.

             For the data types that have binary data as the primary type, all the binary

              protocol encodings show reduced costs when compared to the XML encoding.


          As described above, the measurement to be tested will be the overall cost

computed from the total of the create, render, parse and extract times, multiplied by the

space ratio obtained by dividing the encoded packet size by the natural size of the raw

data. The expectation is that the mean cost figure will be reduced significantly when

tested using a large number of repeats for each combination of data type and protocol.


1.5.3.1       Repeatability

          We would like to obtain a confidence interval of at least 99% for the sample data

obtained in the experiments. Specifically, since we are looking for shifts in the mean cost

value we want a confidence interval of at least 99% that the sample data actually contains

the mean of the population data. For this type of work it is not possible to obtain the

entire population data since the population is almost an infinite combination of runtime

parameters including those being varied by the experiment. We aim to produce a

representative sample of the true population distribution by using a large number of

repeated runs of the experiment with the experimental parameters fixed. So for example,
                                                                                        47
if we are looking at the performance of the document data set and the XML encoding, we

can run this combination a large number of times (say 100) to attempt to remove the

effects of other sources of variation such as the operating system switching tasks.

       For an α value of 0.01, given that we are making a one-tail test, we require a shift

in the mean of 2.33 times the population mean divided by the square root of the number

of samples. Since we are using a large number of samples we look for this shift in the

sample means. For 100 samples, that requires a shift in the sample mean of more than

0.233 times the sample mean to produce a significant result. I.e. a shift of more than

2.33% in the sample mean will be considered a significant effect.

       The number of repeats for each combination of generated data set and protocol

can easily be set to at least 100 samples. The limit for the number of repeats is the total

experiment time. If the time becomes very long (more than a few hours) then it is

possible that other exogenous variables might come into play such as a daily virus scan if

this were not disabled. In general, increasing the number of repeats should increase the

confidence intervals.


1.6    Assumptions

       Several assumptions made in this work have already been stated. They are

repeated here in no specific order for clarity.

       The protocol implementations are assumed to be representative of the

performance of actual operational implementations. It is noted that the implementations

here are not optimized in any way so their performance is expected to be lower than a
                                                                                           48
commercial implementation but the relative performance of each protocol tested is

assumed to be accurately represented by the code used here.

       Protocols based on XML have a lot of freedom in how data is encoded. In these

tests we use one set of choices and assume that the results are still valid for other choices.

For example XML allows attributes or child nodes to be used to represent parameters of a

single node. The choice may affect performance. The choices made for testing XML in

this work were done based on observations of common practice and from research into

books and papers that discuss XML encoding methods [11, 15, 23, 24].

       It was assumed that a simpler implementation in terms of program code results in

lower CPU times and possibly smaller packet sizes so the form to be tested here would be

representative of a best case for XML in general. This may not be true since a complex

implementation may embody an optimization technique such as compression which

overall produces a lower net cost. The code for this work was deliberately kept simple to

help avoid coding errors and to help keep the implementations of the various protocols

fair and consistent.

       The ASN.1 DER encoding used here is a subset of the complete specification.

Only enough of the encoding was implemented to support the data types chosen for

testing. It was assumed that results obtained for this limited implementation would be

applicable to the ASN.1 DER encoding in general.

       Another assumption that was made was that experimental results obtained on the

test system would be reproducible on other systems; even those not running the same

operating system. This assumption seems common in other research into this topic area
                                                                                         49
and is also common in the programming domain where best practices for a given

language are assumed to apply to all implementations of that language across all

operating environments that support it.

       It was assumed that other implementations of the same protocol specifications

would produce similar results to those obtained here – assuming that no particular

optimization was applied unfairly. It was also assumed that the language chosen for the

implementation (C++) did not affect the results. I.e. if the experiment had been coded in

C, C#, Visual Basic, etc, the relative results would be the same.

       It was assumed that one block of binary data is equally representative as any other

and because of this no attempt was made to randomize the data used in the audio test

cases. There are some cases where this is not always true and these are discussed

elsewhere.


1.7    Dependencies

       Most of the risks and dependencies are addressed above in the proof criteria. A set

of binary protocol encodings to be tested must be identified and that set of encodings are

all expected to produce valid test results.

       There is some risk associated with creating the protocol encodings in that the

resulting code may not work correctly or may have hidden bugs which significantly

affect the results. Separate validation code was used to mitigate the risk to some degree as

was manual inspection of encoded data to assure conformance with protocol

specifications.
                                                                                         50
        The data sets used may not show significant differences in costs when applied to

the set of test protocols. As an example, consider a simple text string of ASCII characters.

Most protocols will encode this either as a length value followed by the character array or

as the character array followed by a null character value. In either case, for strings more

than a few characters long, the efficiency in terms of encoded space is very good and one

would not expect to see much difference if any in the protocol performance. To mitigate

this risk to some degree, the data sets used here have some structure to them so that the

protocol encodings must emit meta-data that describes the structure.

        Another risk with this work is that the results are not easily reproducible in a

different environment. Great care has been taken to ensure that the test code has no

dependencies on the operating environment used to test it.


1.8     Limitations

        This work is limited primarily by the amount of effort required to implement each

protocol and data set that is part of the experiment set. It takes considerable time to

review a protocol encoding specification, write an implementation and verify that that

implementation is correct. For this reason, only a few protocol implementations were

selected for testing.

        The number of possible data sets that could be evaluated is very large and again

time is required to write the code to generate random data that represents the data set and

verify that it is correct. An initial list based on work by W3C was reduced to seven data

sets. These were chosen to represent the most common applications and to provide a fair
                                                                                        51
mix of data types. Future work could look at expanding the test data sets to verify the

results obtained here.


1.9      Constraints

         To demonstrate that binary protocols in general have lower costs than XML it was

important to evaluate more than one potential binary encoding. For this work ASN.1

DER was chosen as the primary protocol to evaluate because it is in common use and has

the right set of features. Two additional protocols were designed that are generic forms of

TLV and TVM encodings to see if a generic binary encoding is better than XML.

         It was also important to provide a baseline for a current XML implementation so a

version of Xerces was included in the protocol set. The actual version of Xerces used was

2.7.0.

         It is already known that XML has poor performance when used with binary data

so at least some binary data sets needed to be included in the tests. XML is commonly

used with textual data so it was also important to include data sets which primarily

contain text. In addition, XML is increasingly used to transport or store numeric data so

some form of numeric data set needed to be included.
                                                                                     52
                                     CHAPTER II

                               LITERATURE REVIEW

       This chapter begins with a review of performance work in various areas of the

communications and computing fields. It examines how XML came to be popular and

some of the work done by researchers to improve its performance in a variety of

applications. Binary protocols such as ASN.1 are examined and work done with those

encodings is discussed especially where a comparison of the performance of a binary

encoding is compared to XML.


2.1    Introduction

       Since the very beginnings of electronic data transmission, there has been a

continuous effort to improve the performance of transmission systems. These efforts

cover techniques to improve the speed of the data carrier medium and to improve the

efficiency with which data is carried over a given medium. The earliest data carrier

medium was a single wire which transported telegraph signals. The signal consisted of

the presence or absence of a voltage on the wire. The Earth was used as the return path.

Since that time we have seen the development of analog voice circuits (the telephone),

analog digital transmission (modems over voice circuits) and a variety of dedicated

digital systems running over copper wires, radio waves, laser beams and optical fibers.

For any one of these carrier technologies, considerable effort was given to making the

most of the available bandwidth and in developing techniques to improve that bandwidth

by encoding the data in more sophisticated ways.
                                                                                       53
        In more recent times, data transmission has become the backbone of modern

business and is becoming a backbone for modern society. As digital data transmission has

been adopted for everything from bank transitions to the entertainment industry, a need

has evolved not only for faster pipes to push the data through but more flexible and

robust means to encode the data. With client-server systems being deployed in which

there are literally millions of clients, it is not possible to maintain such a system with

each client running exactly the same version of the application. It takes time to upgrade

each system connected to the network. This leads to a requirement for network protocols

which are not only efficient and flexible but also robust and adaptive.

        The need for both hierarchical data encoding and a flexible and robust way to

implement network protocols has led to the use of protocols like ASN.1 [25] and recently

many based on XML, including SOAP [26]. The adoption of XML as a data encoding

technique has been widespread. It is popular with developers and users alike because it is

a text-based protocol and thus, human readable. Now that we are seeing widespread

adoption of complex web-based client-server systems using protocols like SOAP, the

community at large is starting to question whether XML (and text-based protocols in

general) are the best choice.


2.1.1   Early Performance Work

        In 1928 Harry Nyquist wrote his seminal paper "Certain Topics in Telegraph

Transmission Theory" [27]. Nyquist was trying to understand how the speed of

transmission of telegraph signals was constrained by the transmission system. The paper

is probably best known for his development of what we know today as the Nyquist
                                                                                         54
sampling criterion which states that for a signal at a given frequency to be faithfully

transmitted it must be sampled at a frequency of at least twice the signal frequency. This

provided fundamental limits for the speed of transmission given the bandwidth of the

transmission medium. As Bealieu observes in his 2002 paper on Nyquist's work [28],

Nyquist spent his entire career working on the mathematical principles behind data

transmission theory and set the stage for a lot of subsequent work on transmission theory.

       Given the fixed upper limits of transmission speed for a given telegraph circuit

and the need to transmit a lot of data it became important to design transmission

equipment so as to take advantage of the available bandwidth. By the 1950's telegraph

circuitry was becoming quite sophisticated and as Boughtwood explains in his paper on

telegraph circuit design [29], there were numerous factors to consider if the most was to

be made of a given transmission carrier frequency or range of frequencies.

       Boughtwood's paper focuses on how to take best advantage of the carrier medium

while others became interested in ways of conserving bandwidth by employing different

encoding schemes. In 1961 Oberman presented a paper in which he described a means to

improve the rate of telegraph data transmission by omitting the 'stop' bit [30]. Telegraph

characters are encoded using one start bit, five data bits and one stop bit. The stop bit is

1.5 times as long as the data element bits and thus represents 20% of the character

transmission time. By changing the encoding scheme to eliminate the stop bit, Oberman

achieved a 20% reduction in the size of the data to be transmitted which effectively

increases the bandwidth of the data channel by the same amount. This change in the

encoding scheme became possible because the decoding machinery was no longer
                                                                                            55
mechanical and the electronic version could interpret the transmitted data accurately

without the need for the stop bit interval. It is important to note here, that the entire focus

of this work is towards an improvement in transmission speed and at the time, a 20%

increase was considered valuable.

       By the 1960's computers were being considered for the encoding and transmission

of data. In 1961 Kagan proposed a scheme which used an IBM 1620 computer to encode

the serial signals required to drive multiple teletype circuits in such as way that they

could be output simultaneously from one parallel port on the computer [31]. He noted

that this was significantly more efficient than having the computer drive the printers via

multiple, separate serial ports. This is interesting because as computing power increased,

it became possible for the computer to improve the effectiveness of low speed

transmission circuits. More CPU time was cheaper to employ than implementing a faster

circuit between the machines.


2.2    The Use of XML and Other Text Encoding Schemes

       There has always been considerable support for text-based protocols, largely

because developers find it easier to see the protocol in operation and hence to analyze and

debug it. Some text-based protocols such as SMTP which is used for email [32] offer an

efficient solution because the bulk of the data transmitted is text and the headers required

to construct a packet contain largely textual data. In the case of an email message, the

headers describe email addresses which are text items. Another natural text encoding

occurs when we wish to describe a data structure that is to be rendered into a form of

some kind for presentation to a user and again, we assume that the bulk (if not all) of the
                                                                                       56
data being encoded in the structure is text. This is the origin of the eXtended markup

Language (XML) and its predecessor SGML. Several protocols have been developed

based on XML as the means of encoding the data and also for describing the overall

packet structure.

       Usdin and Graham offer a general perspective on XML as a means of encoding

information [33]. They conclude that XML is a useful tool but not perfect for all

applications. They also note that unlike HTML which specifies tags and defines how they

should be used, XML is a general approach and has no specific tag definitions. This

allows users of XML to customize it to describe data in any way they choose but means

that for data exchange, both parties must understand the tagging scheme.

       Hanslo & Macgregor examined the viability of using XML in wireless

middleware applications and conclude that although it is suitable in many respect its main

drawback is it verbosity [34].

       Hardy et al describe how documents encoded in XML can be converted to PDF

format files. The paper is interesting only in that it shows an example of document

encoding in XML and because the sample of the document shown in Figure 6 of the

paper nicely shows how verbose XML can be [23].


2.3    Comparing the performance of XML to other protocols

       Several researchers have done studies that examine the relative performance of

XML to other protocols. Some studies focus on the packet size, while others are more

concerned with the processing costs. Some general papers look at both aspects of

performance in comparing XML with other protocols or encoding methods.
                                                                                       57
       De Sutter et al. propose that for applications involving continuous data feeds such

as Really Simple Syndication (RSS), XML can be replaced by MPEG-7 encoding [35].

MPEG-7 is a binary encoding which produces smaller packets than XML for the same

data content. The authors also note that for the RSS application, sending deltas of data

content instead of the entire content considerably reduces packet size. They note that

XML is not well suited to encoding such deltas.

       The De Sutter et al. paper builds upon several works which examine aspects of

the MPEG-7 protocol. Niedermeier et al. describe how MPEG-7 can be used to

efficiently encode XML data which may be used as part of a search query or for

delivering delta updates from source to sink in a continuous data stream [36].

       There are also other text-based protocols. A recent example is JSON [37] which is

a textual means of encoding JavaScript data structures. Crockford describes JSON as a

general text format for the serialization of structured data. He offers no reasoning as to

why this would be better than using XML for the same purpose or indeed simply sending

the data in its native, binary form. JSON certainly uses less text than a typical XML

encoding.

       Mundy and Chadwick [20] examined an XML-based messaging system used to

pass medical data and concluded that by replacing the XML with an equivalent ASN.1

protocol, they could achieve at least an order of magnitude improvement in performance.

The XML-based implementation used the Document Object Model (DOM) to parse the

data. ASN.1 parsing does not use a similar 'template' approach but parses the data into a

hierarchical tree directly. Although their work shows significant differences between the
                                                                                        58
XML and ASN.1 implementations, they do not explore the reason for this in depth but

hint that the DOM approach to parsing the XML could be a factor.

       Bayardo et al look at binary adaptations to XML to enable faster parsing,

especially of stream-based applications [38]. They mention a lot of other binary XML

formats and optimizations. One conclusion is that a trivial binary encoding should offer

performance benefits for most applications despite losing the view-source principle (not

human-readable). Their work was based on modifications to the SAX parser.


2.4    Optimizing XML

       Since the adoption of XML as a data encoding in various protocols (notably

SOAP), there has been a developing interest in improving the performance of these

protocols. The W3C organization formed a working group to conduct some research to

find typical use cases for XML and to determine if a binary form of XML would improve

performance in those applications [1]. Cokus and Pericas-Geertsen published a paper in

2004 in which they examine the potential benefits to different serialization techniques for

XML. Whilst their work does cover a wide range of use cases it provides neither

conclusions nor recommendations.

       In an examination of the applications of XML from a datacenter point of view,

Derrick [39] notes that there can be considerable resources required to provide XML-

based web services and employing hardware acceleration devices to offload the XML

parsing work can produce more than 50X improvement in performance. This does not

address the issue of whether XML is inefficient but rather accepts that a large body of
                                                                                      59
applications use XML, and that there are ways to mitigate against having to purchase

more servers for the data center.

       In his very extensive paper entitled "Pros & Cons of XML", Schmelzer [15]

concludes that "XML is a space, processor and bandwidth hog". This statement reflects

two issues with XML: that is produces larger packets than an equivalent binary protocol

and that it takes significant CPU resources to parse. Lawrence [11] examines the space

efficiency of XML in detail. In his conclusions he notes: "Data-centric XML documents

on average consist of approximately 25% data, 50% schema, 20% overhead and 5%

whitespace". This represents a poor efficiency from the data point of view. The tradeoff,

as he observes in his paper, is that XML can be self-describing through extensive use of

tags and this tagging is what constitutes the data inefficiency.

       Imamura and Maruyama view XML in a different light [22]. They view XML and

the available tools for manipulating XML as a great benefit when working with older

protocols based on ASN.1. They developed conversion routines to translate ASN.1 to and

from XML specifically for the purpose of using the XML tools to edit the ASN.1 source

data. This is a different perspective on the use of XML. In most cases XML is used

directly to implement the network protocol. ASN.1 is a far more compact protocol but is

generally considered harder to parse. In practice, ASN.1 simply represents hierarchical

data as does XML and mapping between the two, while not trivial is certainly possible

and practical as Imamura observes.
                                                                                        60
        Sundaresan and Moussa examine the data patterns in XML structures and derive a

series of algorithms for efficient use of XML [13]. Using their algorithms, they were able

to achieve compression ratios of up to 25% on some XML documents.

        Toshiro et al, consider better ways to efficiently parse XML packets [14]. In their

work they observe that XML contains highly repetitive sequences and those sequences

can be parsed once and recognized in future data streams. Using this technique they

achieved 6% to 26% faster parsing of XML packets.


2.4.1   Optimizing SOAP

        The Simple Object Access Protocol (SOAP) is based on XML encoding [26].

Several researchers have investigated possible ways to improve the efficiency of systems

which employ SOAP. Abu-Ghazaleh et al note SOAP's relatively poor performance and

propose a method they call differential serialization to improve the generation of SOAP

packets [9]. This work is in some ways similar to that of Toshiro [14] in that it exploits

the repetitive nature of the XML packets.

        Davis and Parashar performed experimental analysis to determine the relative

efficiency of SOAP when compared to Java RMI, CORBA and HTTP [40]. They observe

that systems based on SOAP may have 10 or 20 times the latency of systems based on

Java RMI or CORBA. In their experiments they observed that the specific

implementation employed multiple system calls to construct the XML packet and

conclude that significant gains in performance could be obtained by having a more direct

means to build the packets.
                                                                                      61
        Matsa et al show that a byte-code based interpretive parser can be up to 80% of

the performance of a fully-compiled XML parser. This is another example of trying to

make XML faster without a lot of extra work.

        There has also been a great deal of work recently on efficient searching of XML

data using one or more indices. Wang et al. show one method of indexing the data to

make searching faster [41]. Kostoulas et al. produced a very fast XML parser based on a

strategy of removing layers from the implementation, not copying data and not

converting data formats.

        The work by Bayado et al. noted above also applies to the optimization of SOAP

[38].


2.5     General optimization


        As the Internet has developed and expanded, several researchers have considered

ways to improve the overall performance of the network. Clark and Tennenhouse

observed that although the current TCP/IP protocols seem adequate for the majority of

today's applications, they may not be adequate for the future where we see a diversity of

network technologies and applications. Their 1990 paper [42] discusses many aspects of

protocol architectures and considers how these should influence the next generation of

networks. They pay particular attention to the issues of byte ordering and alignment and

how these affect overall performance in distributed systems. One of their conclusions is

that "Data manipulation costs more than transfer control operations" implying the need to
                                                                                       62
avoid manipulation of the data as it is passed over the network. Manipulation in this

context means formatting the data according to the requirements of the protocol.

       Markatos published a paper in 2002 [43] which examines the cost of the TCP/IP

protocol stack from the point of view of the computer systems that must implement it in

order to communicate. He observed that the nature of the protocol requires considerable

computing time and that the overall performance of a remote procedure call (RPC)

mechanism based on TCP/IP could be considerably improved if the performance of the

TCP/IP suite itself was improved. Markatos does not offer a solution to the problem but

simply notes that it will become increasingly important to optimize TCP/IP stack

performance as CPU power increases. As we get more CPU power, applications can

perform more work per second and this may require more network packets to be

generated. The network link quickly becomes a bottleneck in the process. So although the

increase in CPU performance will also increase the TCP stack performance, the

relationship is not linear and to get the most from the platform, we need better TCP

stacks. Since the time his paper was written, several manufacturers (such as Intel) have

started shipping intelligent network interface cards (NIC) which include an on-board

processor. The on-board processor implements the TCP/IP stack and frees the main CPU

from this task. This is an interesting approach to optimizing the protocol by simply doing

the work somewhere else.

       An interesting approach to protocol performance enhancement is suggested by

Adler and Maggs [44]. Their work examines the issues involved in asymmetric

communication channels in which one channel direction (download) has much greater
                                                                                      63
band width than the other direction (upload). They optimize the upload channel

performance by using an encoding scheme in which the upload data sends codes based on

data obtained from the download channel. This system is in effect a compression scheme

where the compression keys are known by both the client and server (the server being in

the upload direction form the client).

       Mishra and Shi [45] performed a series of experiments to determine how well

CORBA performs over various network configurations and concluded that the

marshaling of data parameters into and out of network packets could be a significant

overhead. There experiments show that as much as 38% of the overhead in a local

network case was due to parameter marshalling.


       Kostoulas et al created a very fast XML parser by observing the work of Watson

and Mamrak [10, 46]. Watson and Mamrak‟s work was focused on improving network

protocol stack performance. They noted that implementations which were created using

layers that mimicked the layers of the network stack model were less efficient than those

which removed the layering from the implementation.



2.6    Remote Procedure Call Mechanisms

       There are many ways to implement remote procedure call (RPC) mechanisms. In

1990, Tay and Ananda produced a survey of the available techniques [47]. At that time,

no protocols were based on XML. Their work focused on the relative efficiency of

various hardware and operating system platforms. They note in particular that

implementations which perform a lot of data copying were significantly worse in
                                                                                       64
performance. It is commonly understood that performance of computer programs can

often be improved by reducing the amount of data copying that occurs.


2.6.1   XML-RPC

        An RPC mechanism based on XML encoding was developed soon after XML was

first documented. In 2003 Allman produced an analysis of XML-RPC [48]. In his paper

he notes that XML-RPC offers much looser coupling between the applications than other

RPC mechanisms which typically employ tools to create code stubs. He also notes that

XML-RPC is easily layered on top of other protocols such as HTTP. In his testing he

compared various implementations of XML-RPC and observed significant differences in

performance. He does not compare the performance of XML-RPC with other RPC

mechanisms.

        Dissanaike et al. examine the use of XML-RPC in embedded systems [49]. In

particular they compare XML-RPC to its successor (or derivative) SOAP. They note that

it is simple to implement XML-RPC on embedded systems and the simplicity of the

protocol aids in this (as compared to SOAP which is more complex). They conclude that

XML-RPC is indeed a suitable protocol for use in embedded systems.

        Koudas et al. examine issues related to the need to route XML-based database

queries. Their rather short paper identifies that there may be some issue with routing but

do not offer a solution or make any recommendations [50].

        Luo et al. examine the use of XML-based protocols in the WebGIS environment.

They propose that XML is a suitable encoding for large and complex data structures.

Furthermore, they examine the various transactions required to implement WebGIS and
                                                                                      65
show that all forms may be implemented in XML. They offer no reason for choosing

XML or show if the resulting system efficiency is affected by this choice.


2.6.2   SOAP

        The Simple Object Access Protocol (SOAP) [26] is a development from XML-

RPC. Where XML-RPC was very simple and almost free-form, SOAP has a more formal

structure and was designed to run over HTTP. In practice, it can be run layered on top of

other protocols. SOAP has gained popularity as an RPC mechanism because it is flexible

and easy to implement in an environment that already supports an HTTP web server.

Several toolkits are available to create SOAP-based applications. Like XML-RPC, the

efficiency of SOAP is governed by how well it serializes the XML used to create packets.

Govindaraju et al examined various toolkits [51] to see how well each one supported the

needs of an RPC mechanism for scientific computing. They observed considerable

differences in performance which were related to the conversion of binary data such as

floating point numerical values to the text form required for XML.



2.6.3   Call optimization

        Several researchers have examined how remote procedure call mechanisms may

be optimized. In 1986, Almes published a paper [52] in which he examined in detail an

RPC mechanism written in the Modula-2 language and run on the V operating system on

Sun workstations. He experimented with different data structures to see what factors

affected the RPC call execution time and noted that although there was some increase in
                                                                                       66
call time for larger data structures, the increase was acceptable. He concludes that the

Modula-2 language is quite suitable for development of RPC mechanisms.

        Eisenhauer and Daley examined the messaging systems required for a

heterogeneous distributed computing environment [53]. Their paper focuses on their own

Portable Binary Input/Output (PBIO) library and examines in detail how each message

packet is constructed and parsed. They discuss issues such as byte-ordering and in

particular the costs associated with any form of data copy (as required to construct

packets). To address the byte-ordering issue in heterogeneous environments, they opted

to always have the sender construct packets in its native byte order. The receiver must

reverse the byte order if its own byte ordering is different from the sender's. The packet

contains a flag to indicate byte order. This is the same approach taken by CORBA. They

conclude that the flexibility of their PBIO package does not influence its performance

and suggest that the package is suitable for general use. They further observe that their

PBIO package performs significantly better than commercial middleware packages for

message passing.



2.6.4   RPC for high-performance computing and real-time applications

        Bustamante et al. compare XML-based RPC with a custom-designed protocol

which employs native data formats in a high-performance computing environment [54].

They discuss the flexibility of an XML-based approach and examine the costs which they

conclude are very high for binary data. They propose their own wire format which they

call Natural Data Representation (NDR) and show how this has far higher efficiency than
                                                                                     67
the XML form. They compare their approach to PBIO [53] and conclude that their own

approach is superior.

        He et al. examine the requirements for a protocol to transport data in real-time

sensor networks [55]. They offer a protocol called SPEED which claims to highly

efficient and scalable. They note that sensor networks have a specific need for location

data rather than the more common logical node address used in most computer networks.

Routing of packets that contain location data rather than logical addresses make demands

on the protocol. The paper discusses the routing requirements in detail but does not

address the specifics of the protocol used in the simulations.



2.7     Network Protocols

        In this section, several network protocol papers, books and specification

documents are reviewed. There are very many network protocols in use today. Those

examined here are significant either because they are fundamental to modern

communications or because they represent either a means to encode complex data types

or are an attempt to improve performance through special encoding techniques.



2.7.1   Standard protocols

        In the late 70's and early 80's Jon Postel and others developed the Internet

protocols which today carry our information around the globe. The Internet Protocol (IP)

[56] and the Transmission Control Protocol (TCP) [57] together (TCP/IP) provide the

basic error-correcting delivery mechanism upon which many other protocols are layered.
                                                                                         68
Both of these protocols are of interest because they employ simple, small data headers

which describe the protocol packets. Over time, these header designs have been shown to

be somewhat limiting but for the past 20 years, these designs have provided an excellent

digital communications system. Postel's papers are clear and concise in their description

of the motivation for the design and how these protocols work. It is notable that

considerable effort went into minimizing the size of the header required for each type of

protocol packet so that the maximum amount of available bandwidth was dedicated to the

transmission of the packet payload. This was achieved by packing short bit sequences

together to create the packet header. Figure 6 shows the IP packet header exactly as it is

printed in the original RFC 760 document [56]


     0                   1                   2                    3
     0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |Version| IHL |Type of Service|              Total Length        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          Identification        |Flags|       Fragment Offset    |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   | Time to Live |      Protocol   |          Header Checksum      |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                        Source Address                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     Destination Address                        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                     Options                     |    Padding   |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

                              Figure 6: The IP packet header


       One of Postel's other major developments was the Simple Mail Transfer Protocol

(SMTP) otherwise known as RFC 788 [32]. This protocol is a simple text-based design

used to encode and transmit electronic mail. This simple protocol is still used to today for

the majority of email systems. SMTP was developed at a time when computers used
                                                                                     69
teletypes for input and output and computers communicated with each other by linking

together teletype ports using modems and telephone lines. The text-based protocols are a

natural extension of the textual nature of the Teletype communications channels used to

implement the link. Text-based protocols are also convenient for the developer since the

packets can be printed directly for analysis. Note that although SMTP is transported

today over TCP/IP it was originally used for direct computer to computer

communications where there was no underlying transport protocol - just a teletype

circuit.

           In 1990, Case, Fedor, Schoffstall and Davin published the Simple Network

Management Protocol (SNMP) also known as RFC 1157 [58]. This protocol is very

interesting because it focuses heavily on creating very compact network packets and is

capable of transporting both textual and binary data. The protocol was developed as a

means to manage to increasing number of routers and switches which formed the

Internet. SNMP is based upon the Abstract Syntax Notation [number] 1 (ASN.1) [25].

ASN.1 is a general syntax for describing hierarchical data structures. It supports a

number of encodings including the Basic Encoding Rules (BER) [59] and the

Distinguished Encoding Rules (DER). These rules are quite complex and this has caused

some difficulty in understanding the SNMP protocol definition. Kaliski published a paper

providing a "Layman's guide to a Subset of ASN.1, BER and DER" [60]. Kaliski's paper

is essential reading for those interested in adopting ASN.1 for protocol work. The paper

describes a fairly complete subset of ASN.1 and provides useful examples of data types

encoded using both the basic and distinguished rules. More detailed works on ASN.1
                                                                                      70
include the books by Dubuisson [17] and Larmouth [18]. Both books provide extensive

descriptions of ASN.1. Larmouth's work covers BER and PER but not DER. Dubuison's

book covers all common encoding rules including BER, CER, DER, PER and others.

       As computers became used more for data communications and in particular a

need developed for programs to operate in distributed environments, several protocols

were developed to enable data (and data structures) to be sent from one computer to

another. One of the more popular schemes was the Common Object Request Broker

Architecture (CORBA) [7]. CORBA provides an architecture and underlying protocol

(IIOP) to enable computers to exchange data structures. The protocol considers issues

such as the byte-ordering requirements of different computer hardware platforms and the

representation of floating point numbers. CORBA is owned and evangelized by the

Object Management Group (OMG). There have been some attempts to create an Open

Source version of CORBA including the offering by MICO [61] but despite this, CORBA

has not seen widespread adoption. One of CORBA's limitations is that is does not deal

well with security or the need for client and server applications to be running different

versions of the application software. Applications employing CORBA tend to be

distributed within closed environments such as a corporation and require deployment of

all client and server components as a single package. CORBA is less suited to large-scale

Internet-type applications where the client and server components cannot all be required

to run a single version of the application.
                                                                                       71
2.7.2   Other protocol work

        Other work that was discovered when researching protocol performance issues

includes a packet classification taxonomy survey by Taylor [62]. Taylor observed that

packet classification is fundamental to optimizing data flow paths for various

applications. Packets must be associated with particular flow paths in order to process

them efficiently. To classify a packet, the router (or other network device) must examine

various fields in one or more packet headers and then apply an algorithm to determine

which data flow the packet belongs to. This analysis is greatly complicated by the

diversity of packets (from multiple protocol types) and the complexity of packet

structure. In particular, packets with variable length headers are problematic because

extracting the relevant filed data can be time consuming. This has implications for future

protocol design. Protocols with fixed header structures are likely to be more easily

routed. This is the approach taken by the fibre-channel designers. They created a simple

but fixed packet header with the view that such a header could be more trivially parsed by

hardware in the network switches. As implemented, a 1Gbit fibre-channel link has

significantly faster throughput than an equivalent 1Gbit IP link.

        In 1991 Ulhorn published a paper describing a new high-speed data bus for

military aircraft [63]. The paper focuses mostly on the mechanical and electrical

requirements but also includes considerable detail of the proposed protocol. The protocol

is based on a token-ring concept and uses simple, fixed length headers. Messages in this

system are assigned priority rankings and the protocol assures that high priority messages

have low latency across the system. The paper proposes only one protocol as well as only
                                                                                         72
one transmission design and one hardware interconnect method. There is little evidence

that other protocols etc have been considered.



2.8    Data formatting

       Work done by Clark and Tennenhouse to define the needs of post-1990 network

protocols identified the need for new protocol structures [42]. One of the needs identified

is that of “presentation formatting” – the need to convert data from some native format to

a format which the application uses (at the presentation layer). They note both that this

takes time and that the size of the converted data may be different than the original

requiring the application to allocate new space for the converted form. During

experimentation they observed that the time difference between copying items of data in

native format and converting them to an ASN.1 encoded form was 4-5 times and this was

for a hand-coded conversion routine. Gokhale and Schmidt found similar issues with

presentation layer conversion costs in applications which used CORBA [64]. In their

work they compared CORBA field marshalling times to hand-crafted C and C++ code

and concluded that presentation layer conversion costs were one of the most important

areas to be considered for optimization in order to achieve higher network throughput.

       In their work on wire formats for high-performance computing applications,

Bustamante, Eisenhauer Schwan and Widner noted that traditional systems used in high-

performance computing offered good performance but at the expense of tightly coupling

applications [54]. They were attracted to XML because of its flexibility and “plug-and-

play” nature. They took as a baseline the performance of a system using binary data
                                                                                      73
exchange in a homogeneous environment noting that this represents the optimal case. The

data is never required to be converted from one format to another. Their PBIO format

uses native data representations combined with meta-data which is used to describe the

format of the data. This essentially pushes the decoding responsibility and cost to the

recipient of the data. As with CORBA, if the system is homogeneous then no data

conversion is required by sender or receiver.

       Stanberry, Branstetter and Nessett published work in ACM SIGGCOMM in

which they too commented that distributed scientific applications benefit greatly from

homogeneous networks of computers but heterogeneous networks offer promise if the

time taken to exchange data between machines can be minimized [65]. The time is

broken into two parts: latency caused by the transfer over the network and data format

conversion time. They examined the use of ASN.1 (The BER form) as a means of

exchanging data and concluded that a better encoding form which used fixed size fields

allowed for simpler optimization of the conversion functions. Another optimization they

suggest is to encode arrays of data as a single ASN.1 node which avoids duplicating tag

and length values. This does require fixed-length fields but improves performance.

       Data format conversion costs show up in Digital Signal Processing (DSP) work as

well as in general computer networking and data storage. In systems examined by Ito,

Lucke and Parhi, a cost was associated with DSP nodes including those required to

perform data format conversions [66]. Cost in a DSP application consist of two parts: the

time it takes to do the conversion and the number of DSP gates required to implement the

conversion algorithm. The conversion time is expressed as conversion latency because
                                                                                         74
latency is a key part of building the scheduling algorithm in a parallel or net-based

solution where multiple processors contribute to the overall system. The latency may be

reducible by implementing a more complex system of gates but this may not reduce cost

since cost consists of both latency and the gate usage.

       Gavrilovska and Schwan present a novel idea for data format conversion over a

network. Instead of requiring the sender or the receiver to convert the data, they provide a

number of data format conversion services so that data can be converted in transit

between heterogeneous systems [67]. Their approach used specific network processors

but the idea is generally applicable. Providing a pipelined data format conversion

offloads the source and destination servers and introduces only a small additional latency.

This approach requires a conversion service for every source and destination format

combination. This work is similar in concept to the DSP work of Ito, Lucke and Parhi

above [66].

       Haines, Mehrota and Rosendale also comment on the common approach taken by

scientific applications of chaining programs together to form a pipeline. In such an

arrangement, they note, applications are often required to perform data format

conversions. Their solution is to provide a self-describing file format which can be used

to exchange data between applications. The concept uses a header at the start of the file

which describes the entire contents. This is a very coarse-grained approach but supports

the idea of self-describing data [68].

       3D graphics systems require extremely high performance and rely on

combinations of hardware and software to convert tens of thousands of floating point
                                                                                        75
values into images. In a paper written for SIGGRAPH 2005, Kuehne, True, Commike

and Shreiner describe some common ways to improve the performance of OpenGL

programs [69]. The paper is very general and covers a lot of topics but some themes run

through many topics. In particular, since there are so many data items, any reduction of

the amount of data in a single item can have significant performance impact. They cite an

example of drawing many cubes and show that by processing the vertex data in a

different way, less data is transferred to the renderer and the application is faster. They

also note that the encapsulation offered by OO languages can hinder efforts to optimize

the code and suggest that the convenience of the OO approach is less applicable since

performance is of key importance in many 3D applications. This comment applies

equally well to network stacks as noted elsewhere in this chapter.



2.9    Tree Walking and Searching

       Walking data trees to find data is a common algorithm in many areas of

computing. As Carlson notes, these algorithms can be very space consuming [70].

Improvements in space usage can be obtained at the expense of speed by changing the

algorithm so that it does not require the entire tree to be present at any one time.

Carlson‟s work showed alternative algorithms for depth-first, breadth-first and heuristic

searching. Gasieniec et al, discuss the minimum memory requirements for walking a tree

under conditions that the tree may not be marked and may have anonymous nodes. They

note a need for O(log2 N)-bit memory to perform such an exploration.
                                                                                           76
       A lot of recent work focuses on not just walking XML trees but extracting data

from them. There are many works such as that by Wang et al which construct one or

more indices of the data in order to search it more efficiently [41].


2.10    Papers about ASN.1

       ASN.1 is a protocol specification that was designed specifically to support

hierarchical, self-describing data where the data is in a binary format. It uses a tag-length-

variable (TLV) format and was standardized in 1984. It is in current use in a number of

applications including the Simple Network Management Protocol (SNMP) and the X509

digital certificate specification [58, 71]. ASN.1 is covered in detail in the books by

Larmouth and Dubuisson [17, 18]. The basic specification can be found in X.680 and the

encoding rules for Specification of Basic Encoding Rules (BER), Canonical Encoding

Rules (CER) and Distinguished Encoding Rules (DER) can be found in X.690 [2, 25].

ASN.1 is simple in principle but the various encodings combine to provide both a rich

protocol and a complex one from a testing point of view.

       Trivunović et al describe an implementation of H.323 which includes a complete

ASN.1 parser. Their work required the implementation of 730 C++ classes which gives

an indication of the complexity of ASN.1 when used in full [16].

       Early adopters of ASN.1 found that converting data into and out of the ASN.1

encoding forms could be costly. Huitema and Doghri proposed an alternative to simply

using ASN.1 BER which was to negotiate the format to be used between systems at the

start of a session and thus avoid needless conversion into and out of the ASN.1 form [72].

They observe several issues with ASN.1 encodings which affect performance including
                                                                                         77
the need to produce the variable-length fields for integers and a different variable-length

field for tags and length fields. The point is illustrated with some C code. They conclude

that using fixed data formats can reduce computation time and that negotiation of the

format to use at the start of the session retains some level of abstraction so that a sender

can choose a format to send providing the recipient agrees.

       Lin also noted that data conversion costs in ASN.1 were a limiting factor in

adopting the BER form [73]. His work showed that with appropriate engineering effort,

conversion rates compatible with TCP/IP and ATM could be obtained. For his work he

constructed a number of hand-coded conversion routines in pairs. Each pair used a

different approach or a different encoding option in ASN.1 BER. Lin focused on integers,

strings and the ASN.1 CHOICE item but did not include floating-point values. He

concludes that prior work which focused on compilers which generated ASN.1 but which

did not take performance into account may have presented a poor picture of ASN.1 in

regard to its inherent performance. This argument can also be applied to XML where

early implementations focused on simply getting the syntax right and did not consider

performance at all. Hoschka and others have developed better ways to create the format

conversion stub code required in implementing protocols such as CORBA, Java-RMI and

ASN.1 [74].

       In recent work on sensor networks which make use of very small computer

systems, Chang and Gay examined the feasibility of using ASN.1 and similar protocols

but found that their complex structure is not easily implemented with the resources

typically available on small, sensor-net platforms. They adopt an approach which is based
                                                                                       78
on fixed C-style structures but which uses the network htons and ntohs runtime functions

to populate the data fields in a way that removes any endian bias introduced by the host

platform[75]. With this approach they achieve interoperability between heterogeneous

nodes but introduce a much tighter coupling than is desirable in large networks because

each node must understand the data structures used by every other node it needs to

communicate with.


2.11   Binary Protocols in Commercial and Industrial Applications

       There is a family of protocols which are used in commercial and industrial

monitoring and control applications that include automotive applications and even

spacecraft. The driving force in the Automotive industry was reduction in wiring costs.

This led to the development of several interconnect busses that are all based on some

form of serial data encoding. They use a variety of physical interconnects including

optical fibre and span data rates from less than 10 kbps to over 25 Mbps. The term X-by-

wire is now used in the automotive industry to describe anything controlled by some sort

of signaling bus. A good overview of the application of communication busses in

automotive applications is provided by Leen and Heffernan who describe the evolution of

automotive busses from very simple monitoring systems to complex audio-visual systems

[76]. The simplest systems such as seat motor and heating controls use the LIN bus

which is simple and operates up to about 20 kbps. An overview of the LIN bus can be

found on the consortium‟s web site [77] as can the full LIN 2.1 specification [4]. The LIN

protocol can be used to transport simple signal values or diagnostic messages. The signals
                                                                                          79
are either scalar values or byte arrays. There is no hierarchy possible in the data structure

and a single message frame carries only one piece of information.

       The CAN bus operates between 1 Kbps and 1 Mbps [78]. Like the LIN bus, it is a

serial bus and may be implemented on various physical connections. CAN uses messages

which may be unicast or multicast to the bus. Applications are free to create and transmit

messages of any construction that fit within the frames carried by the CAN protocol. So

applications may create arbitrary data structures and use CAN as a transport layer.

       FlexRay improves on the CAN bus speed for high-performance automotive

applications and operates from 100 Kbps to 10 Mbps [79]. It can be used in single-bus or

dual-bus topologies if physical redundancy is required. FlexRay is a transport-level

protocol which carries payload data in frames. As an optional feature, the first two bytes

of the payload may be used as a message identifier. Like the Can bus, FlexRay supports

the concept of a message ID in the first two bytes of each message frame. FlexRay

increases the 8 byte payload size in LIN and Can up to 254 bytes.

       The European Space Agency has defined SpaceWire as a successor to FlexRay

[80]. SpaceWire operates at speeds from 1 Mbps to over 100 Mbps and offers a

redundant, double-bus mode for reliability [81]. Because of the need for high-speed

transfer rates for video data in in-car entertainment systems, SpaceWire has been

proposed as an automotive bus [82].

       All of the LIN, CAN, FlexRay and SpaceWire protocols (and others in this

family) provide a transport layer type of service for the exchange of message packets

between nodes on a local network. This collection of protocols are often referred to as X-
                                                                                       80
by-wire since they offer monitoring and control of a large range of devices by wire (or

optical fiber) [76].

        Vehicle manufacturers (notably in Europe) which have adopted the CAN bus

have defined their own application layer protocols. The SAE has defined J1939 in an

attempt to provide a common application layer for trucks and agricultural machinery. The

application layer protocol deals only with very simple scalar values like engine RPM. A

typical message contains an identifier for the message type and a few bytes of data that

contain the value. Message identifiers are either 11 or 29 bits and the message payload

may be from zero to eight bytes long. Data in CAN bus messages is thus neither

hierarchical nor self-describing in a global sense since a receiver must know the ID codes

used by each manufacturer and the format of the associated data packet.
                                                                                      81
                                     CHAPTER III

                                   METHODOLOGY

       This chapter describes the plan that was adopted for this research in terms of the

work that needed to be done and the approach taken to doing it. Details of the design of

the experiments that were conducted are provided in the following chapter.


3.1    New contribution

       This research provides a new approach to measuring protocol performance with a

view to offering a fairer way to compare protocol encodings based on a common

implementation approach and the use of six performance measurements. The most

significant part of the work is comparing performance using similar methods of

implementation. A lot of prior work compared very dissimilar implementations which led

to some misleading conclusions. Another aspect of this work is that it focuses on

randomly generated data samples and is thus unbiased by the specific data. A third novel

aspect is the detailed performance measurements taken which consist of the following six

measurements:

          Time taken to convert the raw data to a tree format

          Time taken to render the tree into the protocol's encoded form

          Time taken to parse the encoded form back into a tree

          Time taken to extract the raw data from the recovered tree

          Natural size of the raw data

          Encoded size of the protocol packet that contains the data
                                                                                      82
       These detailed measurements allow for a much deeper analysis of the causes of

poor protocol performance and go some way towards understanding what optimizations

can be used to improve protocol performance.


3.2    Prior work

       The literature search above was conducted for work related to protocol

performance in general and also for the use of XML in applications, how XML could be

optimized, non-XML protocols used in similar applications, and protocols suggested as

alternatives to XML in a general sense. In addition, a search was conducted for work

describing algorithms for tree searching and parsing since this was considered a possible

cause of poor performance in parsing hierarchical data.


3.3    Selection of protocol encodings

       There are a very large number of different protocol encodings in use today. For

this work a small selection of potential alternatives to XML had to be made. The

selection needed to be small because each candidate protocol would have to be

implemented in code as part of the experimental procedure. Each implementation was

expected to take considerable time to code and test so the selection needed to be kept

reasonably small in order to complete the work in the time available.

       XML has a large amount of variation in how it can be used to encode the same

tree of data. A decision was made to use the simplest constructs possible without

knowingly optimizing the structure in some way. This was considered to be a fair use of

XML. In practice, XML is used in many different ways which are in some cases more
                                                                                        83
complex than the approach taken here. It was assumed that a simpler implementation

results in lower CPU times and possibly smaller packet sizes so the form to be tested here

would be representative of a best case for XML in general.

       Xerces is an open-source XML generator and parser. It received some bad

reviews with regards to its performance when it first came out but has been improved

considerably and is now in common use. Xerces was included in the test protocol set to

get a baseline for an optimized XML parser. If one or more of the binary protocols tested

were to show better performance than the test XML implementation then that protocol

might display even better performance if optimized. So if the experimental XML

implementation was N times as fast as the binary form tested but only half the speed of

the Xerces implementation we might be able to speed up the binary protocol by a factor

of two times if we could apply the same optimization approach to it that was used in

Xerces. This is not a very strong argument but it was considered important to use Xerces

as a baseline for evaluating the XML implementation just because it is in common use.

       ASN.1 cropped up in many of the papers that offered alternatives to XML and in

its own right as the basis for the SNMP protocol and X.509 digital certificate encoding.

ASN.1 has been around for a long time and is considered to be quite efficient in terms of

encoded data size if not in terms of CPU usage. Some researchers and users of ASN.1

noted that in the case of the BER form, there were far too many variations in how data

could be encoded and this led directly to increased parser complexity and possibly to

reduced performance. These issues with BER led to the development of the Distinguished

Encoding Rules (DER) form and later to other forms. ASN.1 is well understood and in
                                                                                        84
use in standard protocols (in the DER form) so it was an ideal candidate for testing. The

implementation was chosen to be limited to a subset of the DER form. Only enough of

the encoding was implemented as required to support the data types chosen for testing. It

was assumed that results obtained for this limited implementation would be applicable to

the ASN.1 DER encoding in general.


3.4    Determination of generic forms

       Both XML and ASN.1 DER offer many options in how data is encoded. One

avenue of thought was that the simplest possible implementation of any encoding might

be faster than any general form and this led to the idea that we could create a very simple

derivative of ASN.1. ASN.1 uses a Tag-Length-Variable (TLV) format so a very simple

TLV protocol was designed (See Appendix A). The design used fixed length fields for

the tag and length and was intended to support a tagged hierarchy of data as simply as

possible.

       XML is an example of a protocol that uses a Tag-Value-Marker form. Nodes in

the tree have start and end markers with data or child nodes between them. This TVM

form can be implemented using binary values for the tags at the start and end of the

blocks. A generic design using simple, fixed-length tags was created to also be evaluated.

This is documented in Appendix B. The reasoning behind this effort was that parsing the

TVM form might turn out to be simpler (and thus faster) than parsing the TLV form.

       For both the TLV1 and TVM1 protocols some choices were made about the

implementation that could affect performance. The tag length fields in TLV1 were set at

a fixed length of 16 bits. This was somewhat arbitrary. Eight bits was not considered
                                                                                        85
enough because it limited the packet size too severely. Sixteen bits gives a better packet

size without using too much space but does not allow for data over 64K bytes in size

which is too small for commercial use in applications such as numeric computing, word

processing, etc. It would be easy to modify both TLV1 and TVM1 to use 32 bit tags and

length values and this could be studied in future work along with making these fields

variable length.


3.5    Selection of data structures

       A sample set of representative data structures were needed that could be generated

on demand. The basis for the selection was work done by W3C committee in evaluating

applications and data that might benefit from a binary version of XML. The final set was

reduced to seven which were considered representative of very common applications.

The set chosen was designed to include data sets for applications which primarily use

text, those that primarily use numeric values and those that primarily use binary data

(such as audio, images or video). Details of the actual data sets used are described in

Appendix C.


3.6    Determination of experiment vs. analysis

       Having selected a set of protocol encodings to test and a set of representative data

sets, some consideration was given to doing a theoretical analysis of the performance vs.

doing an experimental evaluation. Creating an accurate simulation of each protocol that

could be analyzed accurately was estimated to be considerably more difficult than

conducting a controlled experiment. The majority of prior work found on the topic was
                                                                                      86
also based on experimental evidence. Studies of protocol performance across networks

are commonly simulated because the nature of the networks and the size and content of

network packets can be modeled without much difficulty. Application performance is

dependent on so many factors that modeling produces only limited information.

       The tests were conducted on a personal computer running Windows XP because

that was available at the time the work was done. Nothing about the approach to the

implementation of the protocols should prevent the results from being duplicated in other

environments so long as reasonable care is taken to ensure that the test machine is not

running other tasks that could adversely affect the timing measurements.

       The timing of code execution can be difficult to do accurately because the

measurements are invasive. Windows provides timing accurate to a few nanoseconds

based on CPU hardware counters. The API is simple to use and the cost of the calls is

low. Using this technique it is easy to get accurate measurements of one or two

microseconds. Early testing showed execution times of the order of milliseconds or even

tens or hundreds of milliseconds in some cases so microsecond accuracy was considered

adequate and consequently doing the timing in the software itself was adopted as the

measurement method.
                                                                                        87

3.7     The Approach Taken to Writing the Code

3.7.1   Object Oriented Design

        In order to keep the implementation of each protocol encoding as fair as possible

a simple object-oriented framework was designed so that implementations could share as

much common code as possible. The same approach was used with the test data sets.

Interfaces were designed to support generic use of the data sets and the protocol

encodings. In this way a generic test application could be written that made use of any

data set with any protocol in a consistent way. This also greatly simplified recording the

measured values because recording needed to be done in only one place. It is worth

noting again that no attempt was made to optimize the design in any way. The goal was

to keep the implementations fair and consistent. It was assumed that these non-optimized

forms would perform relatively the same way if each one were optimized using the same

techniques. This may, of course, not be reasonable or possible. One protocol encoding

might lend itself naturally to some form of optimization that is not possible (or produces

insignificant gain) in a different protocol.

        Data for each protocol was constructed in a tree form using a base class to provide

the tree parent-child semantics and classes for each protocol node type derived from the

base class as shown in Figure 7. Each derived node class implements a render function

which converts the tree structure to the protocol's encoded form.
                                                                                         88



                                 ListNode




       1                   1                     1                   1
            Asn1Node               Tlv1Node           Tvm1Node           XmlNode



       *                   *                     *                   *


                               Figure 7: Tree node class structure


       For each protocol in addition to the node class, a parser and a serializer class were

created. The role of the parser is to convert encoded form of the data into a tree of nodes.

The serializer class provides a consistent interface to all the parsers. This two-part

approach which is shown in

       Figure 8, simplified the integration of all the test protocols into the experiment

along with the Xerces parser which has its own node structures and parser.
                                                                                                               89
                                         «interface»Serializer




  XercesSerializer       XmlSerializer        Asn1Serializer         Tlv1Serializer       Tvm1Serializer


                     1                    1                      1                    1                    1




                           XmlParser      1     Asn1Parser       1     Tlv1Parser     1    Tvm1Parser      1
XercesDOMParser      1




                              Figure 8: Serializer and parser classes



          Each protocol (including Xerces) thus has a common serialization interface which

is used to translate data to and from the encoded form. This serialization is decoupled

from the actual parser approach used in each case.

          All the protocol code described above was implanted in a static-link library. This

allowed the library to be used to implement test cases for the protocol encodings as well

as the experiment application.


3.7.1.1      Protocol Verification Test Code

          Since each protocol was being implemented from scratch considerable effort was

made to ensure that the implementations conformed to the protocol specifications. This

was done in two ways: by writing automated test code and by manual inspection of the

encoded form of known data structures.
                                                                                           90
          The automated tests were used regularly as the test framework was developed to

ensure that changes to the overall framework did not break the protocol code in some

way. When manual inspection found errors in the encoding or the encoded from was not

correctly decoded, then addition test cases were added to the test suite.

          The final form of the code was not extensively tested against the specifications for

each protocol but a lot of effort was put into attempting to keep the implementations as

accurate as possible. It was also assumed that other implementations of the same protocol

specifications would produce similar results to those obtained here – assuming that no

particular optimization was applied unfairly. It was also assumed that the language

chosen for the implementation (C++) did not affect the results.


3.7.1.2      Data Generation

          It was anticipated that all data sets used in the experiments would need to be

generated randomly and that the order of experiments would need to be randomized. In

addition, it was anticipated that a single combination of data and protocol would need to

be run many times to remove timing anomalies caused by process switching in the

operating system. The test framework took these requirements into consideration. The

resulting design allowed for a data set to be generated on demand and them used multiple

times as required. Since the order of running the individual test cycles needed to be

random, some consideration was given to storing the test parameters in such a way as to

make it simple to generate them and randomize them later before execution.

          The data in each data set needed to be randomized so a set of data generation

functions were created to provide randomized primitives that could be combined
                                                                                          91
(randomly) into more complex structures. Examples of primitives include an integer

numeric value, a floating-point value, a textual word, sentence or paragraph. Additional

functions were created to generate random titles, subject lines, email addresses and so on.

These functions were then used in combination to construct the full data sets such as

documents, emails, spreadsheets and so on.

          For the binary data blocks, a random collection of byte values was generated in all

cases except for the audio clip tests. For these, the same audio clip was used from an

actual recording. Since none of the protocols parse the binary blocks in any way, it was

assumed that one block of binary data is equally representative as any other. There is one

case where this was known not to be true. The TVM1 protocol uses a two-byte end

marker and this marker was deliberately selected to avoid data value found in common

files. The TVM1 protocol is described in more detail in Appendix B. The specific two-

byte sequence used for the end markers in TVM1 occurs only 0.002% in randomly

generated data (one in 65536).


3.7.1.3      Structure of the experimental code

          Figure 9 shows the relationship between the experiment, the data for a single run,

the protocol and the test data. The experiment contained many experimentData objects.

One object was used for each combination of generated test data and a protocol. The

DataDescriptor class was used to provide base functionality such as random text

generation and also to provide a consistent interface to each data type.
                                                                                              92
                                                              ProtocolDescriptor


                                                         1

        Experiment           ExperimentData

                                              1
                                                              DataDescriptor
                     1   *
                                              1
                                                          1




                                                  EmailData        WebQueryData    SpreadsheetData




                              Figure 9: Experiment class structure

       Note that not all of the data types are shown in Figure 9. The final experiment

used seven different data types which are described in the following chapter.
                                                                                            93
                                       CHAPTER IV

                                EXPERIMENTAL DESIGN


4.1     Introduction

        The difference in performance between two protocols which both support

hierarchical, self-describing data can only be measured with any degree of accuracy if the

two encoding methods are implemented using the same programming language and

methodology. Comparisons of performance with dissimilar implementations such as that

reported by Mundy and Chadwick [20] do not indicate if the protocol encoding is causal

or the implementation. For this research work, each encoding was implemented in C++

using a consistent and unbiased methodology. Each encoding was implemented

according to the specification although not all features or data types supported by the

specific protocol were implemented. In order to understand the approach taken, it is

useful to first review the protocols in terms of their attributes and also in terms of the data

types they support.


4.1.1   Protocol requirements

        The protocol must be able to describe a hierarchy (tree) of data. There must be

some way to identify nodes in the hierarchy at least within the scope of a parent node and

preferably globally. Data should be encoded in a compact way so as not to waste space.

Data should be encoded in such a way that recovering it from the encoded form does not

result in incorrect data.
                                                                                        94
       The protocol must be able to encode textual data, numeric data such as integers

and floating-point values and it must also be able to encode blocks of binary data. The

binary data could be an audio segment, an image or a video clip.


4.2    Evaluating specific encodings

       In order to evaluate a particular encoding with respect to another, both must be

implemented the same way and the method of measurement must be consistent. For

example, we cannot compare an XML implementation written in Java with an ASN.1

implementation written in C++ and infer anything about the performance of the encoding.

We can certainly measure the performance of the implementations and in some cases this

is enough. For a particular application various protocol implementations may be available

as perhaps open-source work. Some might be in C, some in C++ and some in Java but it

is unlikely that all are available in every language. A project may select a protocol

because it is implemented in a specific language and that implementation is measured as

having adequate performance.

       The approach taken here is to implement each protocol the same way and measure

differences in encoded size and CPU cost caused by the protocol itself. The encodings are

all written in C++ using the same class structure. A lot of effort was put into keeping the

implementations fair and not biasing one against another. The general nature of the code

means that a bit more abstraction is used than might be required or than might occur in a

practical implementation. The additional abstraction probably reduces overall

performance slightly but this is applied to all the encodings so it should not alter the

relative performance measurements.
                                                                                         95
4.3    The common protocol driver

       In order to compare protocols fairly it was necessary to develop a way to write

one piece of code that could use any encoding to encode and subsequently decode a given

set of data. It would have been possible to write this separately for each encoding but this

layer might introduce more variation in performance than was due strictly to the

difference in the encodings. The idea was to have a sample „program‟ into which the

various protocol encodings could be plugged. The program itself which consists of the

serialization logic and all the measurement and logging more closely represents an

application which uses a third-party library as a protocol implementation. One obstacle to

this approach was that the tags in nodes for some protocols like ASN.1 are numeric

whereas for XML they are text. This was overcome by using a class to hold the tag

information. The tag objects were constructed with both a numeric identifier and a textual

identifier. At runtime, the protocol plug-in extracted the tag data from the tag object as

required. Abstracting the tag data this way allowed the code to be common across all the

protocols but again introduces another level of abstraction that most likely would not be

present in a normal implementation. (One would not abstract an integer or a string).

       A set of tags were built for use by all the protocols. Numeric tag values were

assigned from one, incrementing by one for each new tag added as the test program

evolved. Overall there were less than 256 tags so no tag value required more than one

byte. So any protocol that used variable length tags was not biased because some tags

were shorter than others. For the textual part of each tag, a short English word was

chosen to reflect common practice in the XML world where quite verbose tag names are
                                                                                        96
common. So, for example, words like “Speed” and “Address” were used rather than “v”

or “addr” or “a”. It is understood that reducing the length of XML tags directly reduces

the encoded packet size and may improve parser performance.

       The common driver layer also made it simpler to measure the performance

parameters more consistently since the measurements occur in the same piece of code no

matter which protocol is being evaluated. The actual measurements recorded are

described below.


4.4    Expressing the hierarchy

       A way was developed to express the construction of the data trees and subsequent

parsing of the encoded form back into a tree that was independent of the protocol

encoding scheme. This serialization code abstracts the node layers and provides the

concept of a current node or section of the hierarchy. Operations are performed relative to

the current section. A section is begun, operated on and subsequently ended. A new child

section can be created in the current section. The new child becomes the current section.

An „exit‟ operation allows the operations to leave the child section and pop up to the

parent section. Figure 10 shows a pseudo-code example for the engine parameters.


Start engine parameter section
Add temperature value
Add RPM value
Exit section

                    Figure 10: Pseudo-code for serializing engine data.
                                                                                          97
4.5      Parsing the encoded form

         The serialization abstraction also extends to parsing the encoded data back into its

original tree form and then extracting the data items from that tree. Not all parsers work

this way and this is not necessarily the most efficient way to parse large data streams

since it requires all the data to be in memory but this approach was taken because it could

be implemented for each encoding in the same way as could have been a streaming

model.

         Parsing of the encoded data was done in such a way as to always recover all of the

encoded data. Data was extracted in the order it was encoded. The parsers were all

written to allow random access to the encoded data since this is a requirement if the

receiver is not to be tied to the encoding order. Knowledge of the encoded order is an

advantage since it does not require the parser to move backwards through the encoded

data stream. Since the approach used here constructs the entire tree before doing

extraction, random access would be possible. Since the tree structures were deliberately

kept very similar between the encodings, it is expected that random access to any one of

them would perform similarly although this was not tested. The extracted data items were

not verified to be the correct values. Testing of the serializer code before the performance

tests were run verified that the correct nodes were extracted from the encoded form for

each protocol. This verification testing was done in a separate test suite which was

developed in parallel with the protocol encodings and was used throughout development

to prove correctness of the encodings and the supporting serialization code.
                                                                                          98
        The parsing and extraction code was implemented to represent how a typical

application might want to extract data from the encoded form. Figure 11shows a pseudo

code example which extracts the engine performance data.



Find   engine parameters section
Find   temperature value
Find   RPM value
Exit   section

       Figure 11: An example of pseudo code to extract data from the encoded form.



4.6     The performance measurements

        There are a number of aspects to how a protocol encoding performs. Of most

interest is the size of the encoded data, the time it takes to create the encoded form and

the time it takes to recover the data from the encoded form. The encoded size is

independent of platform but the timings are not. So timings measured on one platform are

only of value when compared to other timings on the same platform. In practice,

encoding the data consists of two parts: constructing the tree of nodes from the raw data

and converting the node tree to the encoded form. We refer to the tree construction as the

creation time and the time to convert that tree to the encoded form as the rendering time.

Extracting the data from the encoded form also has two parts: the time taken to recover

the tree structure from the encoded data and the time taken to extract the data values from

the tree. It should be noted that a parser does not need to construct the whole tree in order

to extract data from it. A parser may watch the encoded data as a stream and use a state

machine or other approach to extract data on the fly.
                                                                                          99
       The size of the encoded packet is dependent on the encoding scheme but also on

the data values encoded. Integer values encoded into a variable length field will differ in

length depending on the values encoded. Strings of text of different sizes obviously create

different packet sizes. So the absolute size of the encoded data is not a complete measure

of the protocols space efficiency. We can however compare the encoded size to the size

of the raw data. For a string, this is the number of bytes required to hold the string in the

computer memory. Note that for a UTF-8 string this might be more than the simple

character count because some characters require more than one byte. Integer values are

commonly stored in the machine‟s native word size or perhaps half that value or twice

that value for „short‟ and „long‟ integer types respectively. Floating point values are

commonly stored as either four bytes or as eight bytes for the double-precision case. If

we measure both the raw data size and the encoded data size we can establish a value for

the efficiency of the protocol in terms of how well it packs data into the encoded packet.

       The experiments used to measure protocol efficiency recorded six values:

          Time taken to create the data tree from the data (create time)

          Time taken to convert the tree to the encoded form (render time)

          Time taken to recover the tree from the encoded form (parse time)

          Time taken to extract the data from the recovered tree (extract time)

          Size of the raw data

          Size of the encoded data

       The times were measured in microseconds and the sizes were measured in bytes.

After the six basic measurements were taken some derived values were also recorded.
                                                                                      100
The create and render times were added to give a time to write the packet. The parse and

extract times were added to give the time taken to receive a packet. The write and read

times were added to give an overall time for the encoding. The ratio of encoded size to

raw size was calculated to give a figure for the packing efficiency of the encoding. An

overall figure of cost for the protocol was obtained by multiplying the total time by the

packing ratio value. Shorter times and smaller packets produce a smaller cost figure. So a

more efficient protocol has a lower cost figure. The primary goal was to see if the binary

encodings have lower costs than the XML encoding. The finer grained measurements

were used to help understand the overall results.


4.7    The data sets used

       The idea was to evaluate protocol encodings when applied to the kinds of data

sets which might be used by typical applications. The W3C identified a large number of

application types which it considered might benefit from a binary alternative to XML [1].

       For the experiments conducted here, several data sets were designed to represent a

small collection of application types but which each focus on one primary data type such

as text, numeric or binary data. One data set used was an email which is shown in Figure

12: The email data set. The other data sets used are documented in Appendix C.
                                                                                       101
Email
        From
        To list
              To
              To
        CC list
              CC
              CC
        Subject
        Text content
        Attachment
              Display title
              Binary data

                               Figure 12: The email data set

        Each data set has a primary type. The email data set is primarily text but includes

a small binary attachment to represent and image or perhaps a digital certificate that

might be used in practice.


4.8     Data generation

        Data for each data type was generated randomly using program code for most

cases. The only exception was for the audio data set where a sample recording was used.

A data generator was written for each data type and some convenience functions were

layered on top of the basic data generators to make creation of titles, sentences,

paragraphs, email addresses and so on easy.

        The program code used the current time to seed the random number generator at

the start of each experiment set. The rand C-runtime function was used to obtain random

integer values. Floating point values were generated by dividing two randomly generated

integer values (with code added to prevent a divide by zero exception). Dividing two

integer values generates both large floating point values and small values.
                                                                                        102
       Binary data blocks were generated with random lengths (between fixed bounds)

and with byte values created by truncating random integer values. For most of the

encodings, the values in the binary blocks have no effect on the encoding. This is not true

for binary TVM encodings because some values cannot be permitted in the encoded

stream because they are used to indicate the ends of node variables. If the end marker

value occurs in the data stream being encoded it has to be replaced by a different byte

sequence which is not the end marker sequence and which can be recognized and

decoded when the data is recovered from the encoded form. If the source data stream

contains many byte sequences that are the same as the end marker sequence for the

encoding then the encoder will likely take longer to create the packets, the packets will be

larger by some amount and the decoder will incur a penalty in decoding them. The byte

sequence used in the TVM1 encoding described in Appendix B was designed to avoid

values found commonly in audio and video files. It is assumed therefore that the audio

clip used in some of the experimental data blocks has no effect on the validity of the

experiment.


4.9    Test runs and data recording

       Each experiment run was done on a personal computer running Windows XP.

Before any test runs were conducted, all non-essential applications and services were

stopped so that the experiment was the only application running. Test runs typically took

about eight hours to complete so runs were initiated in the morning and the data

examined in the evening or the next day.
                                                                                           103
        During early testing of the experiment code, several short runs were done with

only one data generation cycle and one repeat per combination to examine how

repeatable the collected data were. It was expected that some variation in timing would

be observed due to process switching in the operating system and these initial runs were

designed to make sure that despite occasional task switches, the application would run

well enough to create useful data. The number of run repeats was then adjusted to give

sufficient averaging of the results for one protocol/data set combination without making

the overall run time too excessive. Details of the machine used to run the experiments can

be found in Appendix D. The details of the tests for repeatability are shown in Appendix

E. The steps shown in Figure 13 were taken by the experiment code to create the suite of

tests to be run:

For every protocol:
      For every data set:
            For the number of data generation runs:
                  For the number of run repeats:
                        Create one set of run data


                       Figure 13 Pseudo-code for generating test runs



        The run data thus consists of and identifier for the protocol to use, an identifier for

the data set type, and an identifier for the data generation run to use. The resulting set of

combinations was then randomized before the experiment was run. The number of

protocols was five (including Xerces). The number of data sets was seven. The number of

data generation cycles and the number of run repeats was configurable at runtime.
                                                                                        104
        The data generation cycles were necessary to address variations in data encoding

where numeric values occurred in the data since encoded numeric data can vary greatly in

size depending on the actual values.

        The timing measurements, native data size and encoded packet size information

were measured by the experiment program code and recorded in a file specific to each

experiment run. The data in the log files was recorded in tab-separated-variable format to

make it easy to import into Microsoft Excel and from there into MINITAB and

MATLAB. The names of the data files were created by the experiment program based on

the date and time to ensure that data from one experiment could be tracked accurately and

also so that consecutive experiment runs could not overwrite any existing data from

earlier runs.

        Final runs were done using ten data generation cycles from each protocol/data set

combination. Each individual combination was run 100 times. With the Xerces protocol

included in the test runs, a total of five protocols were tested. There were seven data sets

used giving 35 combinations. For each of these, ten random data sets were generated

giving 350 experiments to run. Each of the 350 experiments was run 100 times for a total

of 35,000 test cycles. So the resulting log file for such an experiment run contained

35,000 lines of measured data.

        To make importing the data into Excel and other applications easier, some derived

values were also recorded from the basic measurements. Table 2: Log file columns shows

the columns recorded for each row of the result log.
                                                                                       105
                                 Table 2: Log file columns

Column Identifier Direct/ Description
                  Derived
1      Proto      Direct  The name of the protocol. One of: ASN1, Xnode,
                          TLV1, TVM1 or Xerces.
2      Struct     Direct  The name of the data structure being evaluated. One of:
                          Document, Email, 3D_Mesh, Audio, WebQuery,
                          WebQueryresponse or Spreadsheet.
3      Create     Direct  The time to create the data node tree from the raw data
                          in microseconds.
4      Render     Direct  The time taken to convert the data node tree to the
                          protocol's encoded form in microseconds.
5      Write      Derived The sum of the create and render times in
                          microseconds.
6      Natural    Direct  The natural size of the raw data in bytes.
7      Packet     Direct  The size of the encoded packet in bytes.
8      Parse      Direct  The time taken to parse the encoded form back into a
                          data node tree in microseconds.
9      Extract    Direct  The time taken to extract the raw data from the data
                          node tree in microseconds.
10     Read       Derived The sum of the parse and extract times in
                          microseconds.
11     Ratio      Derived The ratio of encoded size to natural size (Packet /
                          Natural).
12     Total      Derived The sum of the create, render, parse and extract times
                          in microseconds. This is the measure of total time for
                          the protocol/data set combination.
13     SR         Derived The product of the total time and the ratio of encoded
                          to natural size. The is the overall cost figure.



4.9.1   Use of derived values

        During initial development of the experimental code only the direct measured

values were recorded. In order to validate the data, some initial analysis was required and

this was originally done using the following manual steps:

           Load the log file into Excel

           Sort the rows by Struct and Proto columns
                                                                                      106
          Create separate tables for each Struct type

          Import each Struct table into MINITAB

          Examine the recorded data by generating simple statistics (mean, standard

           deviation, etc)

          Plotting histograms of a parameter such as create time vs. protocol

          Making judgment calls about the validity of the data

       It was soon apparent that looking at the individual time measurements and the

individual size measurements was only useful in terms of validating the basic data

recording mechanism. Although this was useful in detecting some bugs in the code, it did

not lend itself to any validation of the hypothesis.

       In order to examine the overall cost results it was necessary to compute the total

time values and the ratio of packet to natural data size. This was initially done in Excel

but the task quickly became too tedious to repeat.

       The experiment code was altered to compute the derived values and record those

with the direct measurements. This greatly reduced the amount of manual work required

to validate the test run data and also revealed more bugs in the code. The derived values

were randomly sampled tested for correctness by using a calculator to obtain them from

the direct measurements. Once the correctness of the derived values was confirmed, the

majority of the data validation was then done using only the derived values.
                                                                                     107


                                     CHAPTER V

                             ANALYSIS AND RESULTS



5.1    Validation

       Several trial runs were performed to evaluate the experimental code. This testing

revealed a few coding errors and some problems with logging the test results. Once these

were fixed, more test runs were conducted in order to verify that the data being recorded

was reasonable. The MINITAB program was used to produce some time series plots as

well as to perform basic statistical analysis of the result data. This initial analysis

confirmed that timing measurements are affected by process switching in the test

computer and that these values could be removed from the data sets by removing

measurements that exceeded three times the mean measured value. Several manual tests

were conducted to verify that this approach was reasonable. Once that was done,

MATLAB code was used to process the data values and remove outliers.

       The results of the MATLAB processing were again examined manually for

anything that seems unreasonable in order to verify that the MATLAB code was acting

correctly. Samples of the measured data values were inspected to make sure that they

were correctly included or excluded from the data set produced by the MATLAB code.

       Having verified that the experimental code was correct, the recording mechanism

was correct and the MATLAB processing code was correct, several experiment runs were
                                                                                       108
conducted with the full set of protocols, data types, data generation runs and with 100

repeats of each case. These runs took about six hours to execute.

       The MATLAB code was used to generate the data tables shown below as well as

the 3D histograms. These were inspected to verify that each protocol had been tested with

each data type according to the design of the experiment detailed above.


5.2    Results

       The following sections show tables and histograms of mean measured values as

they vary by protocol for each of the data types tested. Because there are a large number

of measured values which vary both by data type and by protocol they are presented in

two groups. The first groups of tables and histograms show one table for each data type

that has the individual measurements vs. protocol. The following section shows one table

per measured value showing how that value varies with data type vs. protocol. The tables

are presented in pairs. The first table in the pair shows the actual measured values. Times

are in microseconds and sizes are in bytes. The second table in each pair shows the ratio

of the measured value to the value for the Xnode protocol type. These ratio tables give a

clearer idea of the difference between the binary encodings and the XML encoding. Note

that measured values for Xerces are included in the first of each table pair but not in the

second since ratios between the protocol coded for the experiments and the open-source

Xerces code are not valid in an apples-to-apples sense. Following the sections containing

the tables are sections containing histograms of the same data.
                                                                                      109
5.2.1     Table row and column labels

          The row and column labels for each of the following tables were generated by the

MATLAB code that also generated the table contents. Some of the labels are rather terse

so a description of the meaning of each one is provided here for clarity.

                                   Table 3: Protocol labels

                        Tag      Meaning
                        ASN1     Abstract Syntax Notation 1 (ASN.1)
                        TLV1     Tag Length Variable 1
                        TVM1     Tag Variable Marker 1
                        Xnode    XML nodes
                        Xerces   Xerces XML




                                  Table 4: Data type labels

Tag                Meaning
Document           A simple document consisting of several paragraphs of text with some
                   meta-data
Email              An email with a small binary attachment
3D Mesh            A set of vertex and face descriptions for a 3D mesh
Audio              A sample of audio data encoded using the MP3 format
Web Query          A web query requests for a search for papers by author
Web                The response to the web request containing a list of papers
Response
Spreadsheet        A spreadsheet containing decimal values and meta-data for rows and
                   columns.



                                 Table 5: Measurement labels

Tag                        Meaning
Create                     Time to create node tree from raw data in microseconds
Render                     Time to render node tree to protocol format in microseconds
Parse                      Time to parse node tree from encoded form in microseconds
Extract                    Time to extract raw data from node tree in microseconds
Write                      Sum of Create and Render times
Read                       Sum of Parse and Extract times
                                                                                                    110
Total                            Sum of Create, Render, Parse and Extract times
Natural                          Natural size of the raw data in bytes
Packet                           Encoded size of the packet in bytes
Ratio                            Ratio of encoded size to raw size
RT                               Cost figure – Product of Ratio and Total time




5.3        Table set 1 – Measured values for each data type

                                Table 6: Mean values for the Audio type

                             ASN1        TLV1         TVM1          Xnode           Xerces
                 Create      600.7       598.7        599.1         17794.5         21223.6
                 Render      927.5       952.6        118074.2      2778.5          13899.3
                 Parse       1129.8      1137.7       225670.3      2843950.2       12270.5
                 Extract     454.0       452.2        461.2         170390.6        177754.0
                 Write       1530.1      1554.3       118679.2      20581.1         35129.7
                 Read        1585.6      1591.9       226138.9      3014327.4       190109.2
                 Total       3121.7      3154.6       344862.1      3035035.3       225484.1
                 Natural     54341.0     54341.0      54341.0       54341.0         54341.0
                 Packet      54354.0     54365.0      54517.0       74450.0         74504.0
                 Ratio       1.0         1.0          1.0           1.4             1.4
                 RT          3121.7      3155.6       345978.6      4158155.9       309148.6



                                 Table 7: Ratio to Xnode for Audio type2

                                           ASN1            TLV1     TVM1
                                   Create 0.034            0.034    0.034
                                   Render 0.334            0.343    42.496
                                   Parse   0.000           0.000    0.079
                                   Extract 0.003           0.003    0.003
                                   Write   0.074           0.076    5.766
                                   Read    0.001           0.001    0.075
                                   Total   0.001           0.001    0.114
                                   Natural 1.000           1.000    1.000
                                   Packet 0.730            0.730    0.732
                                   Ratio   0.730           0.730    0.732
                                   RT      0.001           0.001    0.083




2
    Note that the some ratios were less than 0.1% and consequently show up as zero in this table.
                                                                    111
           Table 8: Mean values for the Document type

           ASN1      TLV1         TVM1        Xnode       Xerces
Create     10781.2   10753.2      10694.2     3181.3      3676.0
Render     8381.7    8554.9       21772.2     7640.2      1945.8
Parse      14695.2   14729.0      54534.8     362288.9    3345.1
Extract    7509.9    7498.5       7513.0      12431.7     2626.0
Write      19200.1   19336.2      32476.9     10837.7     5618.5
Read       22245.2   22261.3      62062.1     374739.0    5969.0
Total      41465.2   41597.5      94539.0     385587.8    11610.6
Natural    6670.0    6670.0       6670.0      6670.0      6670.0
Packet     7030.9    7178.0       7129.6      8948.9      8953.5
Ratio      1.1       1.1          1.1         1.3         1.3
RT         43711.3   44770.3      101059.0    517407.5    15611.3

           Table 9: Ratio to Xnode for Document type

                          ASN1      TLV1     TVM1
                Create    3.389     3.380    3.362
                Render    1.097     1.120    2.850
                Parse     0.041     0.041    0.151
                Extract   0.604     0.603    0.604
                Write     1.772     1.784    2.997
                Read      0.059     0.059    0.166
                Total     0.108     0.108    0.245
                Natural   1.000     1.000    1.000
                Packet    0.786     0.802    0.797
                Ratio     0.786     0.802    0.797
                RT        0.084     0.087    0.195



            Table 10: Mean values for the Email type

            ASN1     TLV1      TVM1          Xnode       Xerces
 Create     1618.2   1619.4    1609.4        926.8       1331.7
 Render     1165.7   1216.0    9181.5        1142.1      1083.6
 Parse      2273.9   2293.6    19733.3       172792.8    2304.0
 Extract    1180.2   1180.4    1187.4        7017.3      9979.9
 Write      2789.0   2835.4    10791.0       2069.1      2417.3
 Read       3452.8   3473.3    20926.2       179864.3    12287.6
 Total      6238.0   6311.9    31796.4       181956.2    14726.5
 Natural    3736.4   3736.4    3736.4        3736.4      3736.4
 Packet     3777.8   3803.2    3807.7        4511.6      4562.6
 Ratio      1.0      1.0       1.0           1.2         1.2
 RT         6308.4   6427.4    32407.6       219864.1    18026.9
                                                                     112


            Table 11: Ratio to Xnode for Email type

                         ASN1      TLV1      TVM1
               Create    1.746     1.747     1.736
               Render    1.021     1.065     8.039
               Parse     0.013     0.013     0.114
               Extract   0.168     0.168     0.169
               Write     1.348     1.370     5.215
               Read      0.019     0.019     0.116
               Total     0.034     0.035     0.175
               Natural   1.000     1.000     1.000
               Packet    0.837     0.843     0.844
               Ratio     0.837     0.843     0.844
               RT        0.029     0.029     0.147




           Table 12: mean values for the 3D Mesh type

        ASN1       TLV1          TVM1         Xnode       Xerces
Create 208744.3    208214.4      206867.1     107604.7    112133.7
Render 147898.0    155005.5      154123.1     119354.7    12039.4
Parse   290622.6   292675.5      492572.4     1714020.8   26165.4
Extract 130770.7   130485.1      130645.6     332552.9    3190.4
Write   356778.6   363219.8      360990.3     226959.5    124173.1
Read    421393.3   423160.6      623218.0     2046074.8   29413.0
Total   778171.9   786380.5      984208.2     2272971.1   153586.2
Natural 9895.2     9895.2        9895.2       9895.2      9895.2
Packet 11715.3     17170.4       15215.3      35027.9     34674.4
Ratio   1.2        1.7           1.5          3.5         3.5
RT      922064.5   1367618.3     1514495.8    8107432.2   540687.3
                                                                    113
            Table 13: Ratio to Xnode for 3D Mesh type

                          ASN1      TLV1     TVM1
                Create    1.940     1.935    1.922
                Render    1.239     1.299    1.291
                Parse     0.170     0.171    0.287
                Extract   0.393     0.392    0.393
                Write     1.572     1.600    1.591
                Read      0.206     0.207    0.305
                Total     0.342     0.346    0.433
                Natural   1.000     1.000    1.000
                Packet    0.334     0.490    0.434
                Ratio     0.334     0.490    0.434
                RT        0.114     0.169    0.187



          Table 14: Mean values for the Spreadsheet type

           ASN1       TLV1        TVM1       Xnode        Xerces
Create     12245.4    12250.5     12163.5    6114.3       6828.1
Render     8507.9     9012.4      10040.7    7001.2       1040.1
Parse      16914.4    17072.1     36955.4    118901.7     3003.0
Extract    7237.0     7206.3      7226.4     4789.9       4209.0
Write      20761.1    21291.5     22177.3    13182.6      7870.3
Read       24186.7    24286.0     44237.5    123713.9     7219.5
Total      44928.9    45633.7     66665.8    136933.2     15156.0
Natural    721.9      721.9       721.9      721.9        721.9
Packet     970.3      1214.7      1217.2     2522.2       2549.2
Ratio      1.3        1.7         1.7        3.5          3.5
RT         60403.2    76807.6     112438.5   478866.4     53529.6

          Table 15: Ratio to Xnode for Spreadsheet type

                          ASN1      TLV1     TVM1
                Create    2.003     2.004    1.989
                Render    1.215     1.287    1.434
                Parse     0.142     0.144    0.311
                Extract   1.511     1.504    1.509
                Write     1.575     1.615    1.682
                Read      0.196     0.196    0.358
                Total     0.328     0.333    0.487
                Natural   1.000     1.000    1.000
                Packet    0.385     0.482    0.483
                Ratio     0.385     0.482    0.483
                RT        0.126     0.160    0.235
                                                          114



    Table 16: Mean values for the Web Query type

        ASN1     TLV1       TVM1      Xnode     Xerces
Create 1383.9    1386.3     1374.0    276.0     523.6
Render 1028.4    1052.6     1175.4    868.5     481.0
Parse   1906.4   1910.5     4178.8    15265.7   1770.3
Extract 932.7    936.8      935.7     1218.9    3485.9
Write   2412.0   2438.1     2552.3    1145.5    1008.6
Read    2840.1   2846.5     5127.9    16497.5   5255.2
Total   5256.6   5284.1     7697.4    17664.9   6264.1
Natural 86.5     86.5       86.5      86.5      86.5
Packet 130.4     144.5      144.5     330.6     381.6
Ratio   1.5      1.7        1.7       3.9       4.5
RT      7922.9   8832.9     12867.0   67574.5   27764.5

     Table 17: Ratio to Xnode for Web Query Type

                    ASN1      TLV1    TVM1
          Create    5.015     5.023   4.979
          Render    1.184     1.212   1.353
          Parse     0.125     0.125   0.274
          Extract   0.765     0.769   0.768
          Write     2.106     2.128   2.228
          Read      0.172     0.173   0.311
          Total     0.298     0.299   0.436
          Natural   1.000     1.000   1.000
          Packet    0.394     0.437   0.437
          Ratio     0.394     0.437   0.437
          RT        0.117     0.131   0.190
                                                             115
  Table 18: Mean values for the Web Query Response type

          ASN1      TLV1      TVM1      Xnode      Xerces
Create    4693.5    4688.9    4652.8    900.7      1259.7
Render    3371.5    3444.4    6472.5    2987.8     840.3
Parse     6436.9    6489.9    19184.1   101459.2   2280.2
Extract   3268.2    3260.9    3267.0    7046.3     3913.8
Write     8127.7    8170.4    11181.0   3886.4     2103.2
Read      9749.1    9776.5    22466.9   108543.9   6196.5
Total     17921.6   17979.9   33684.6   112459.2   8300.7
Natural   1539.1    1539.1    1539.1    1539.1     1539.1
Packet    1692.4    1727.9    1727.9    2431.5     2475.3
Ratio     1.1       1.1       1.1       1.6        1.6
RT        19709.4   20189.1   37821.7   177728.0   13445.3




   Table 19: Ratio to Xnode for Web Query Response type

                     ASN1       TLV1    TVM1
             Create 5.211       5.206   5.166
             Render 1.128       1.153   2.166
             Parse   0.063      0.064   0.189
             Extract 0.464      0.463   0.464
             Write   2.091      2.102   2.877
             Read    0.090      0.090   0.207
             Total   0.159      0.160   0.300
             Natural 1.000      1.000   1.000
             Packet 0.696       0.711   0.711
             Ratio   0.696      0.711   0.711
             RT      0.111      0.114   0.213
                                                                                  116
5.4   Table set 2 – Data type variation for each measured value

               Table 20: Mean values for Create time in microseconds

                      ASN1        TLV1       TVM1         Xnode        Xerces
      Document        10781.2     10753.2    10694.2      3181.3       3676.0
      Email           1618.2      1619.4     1609.4       926.8        1331.7
      3D Mesh         208744.3    208214.4   206867.1     107604.7     112133.7
      Audio           600.7       598.7      599.1        17794.5      21223.6
      Web Query       1383.9      1386.3     1374.0       276.0        523.6
      Web Response    4693.5      4688.9     4652.8       900.7        1259.7
      Spreadsheet     12245.4     12250.5    12163.5      6114.3       6828.1



                     Table 21: Ratios to Xnode for Create time

                                  ASN1          TLV1     TVM1
                     Document     3.389         3.380    3.362
                     Email        1.746         1.747    1.736
                     3D Mesh      1.940         1.935    1.922
                     Audio        0.034         0.034    0.034
                     Web Query    5.015         5.023    4.979
                     Web Response 5.211         5.206    5.166
                     Spreadsheet  2.003         2.004    1.989



               Table 22: Mean values for Render time in microseconds

                       ASN1       TLV1        TVM1        Xnode        Xerces
      Document         8381.7     8554.9      21772.2     7640.2       1945.8
      Email            1165.7     1216.0      9181.5      1142.1       1083.6
      3D Mesh          147898.0   155005.5    154123.1    119354.7     12039.4
      Audio            927.5      952.6       118074.2    2778.5       13899.3
      Web Query        1028.4     1052.6      1175.4      868.5        481.0
      Web Response     3371.5     3444.4      6472.5      2987.8       840.3
      Spreadsheet      8507.9     9012.4      10040.7     7001.2       1040.1
                                                                          117
               Table 23: Ratios to Xnode for Render time

                            ASN1         TLV1      TVM1
               Document     1.097        1.120     2.850
               Email        1.021        1.065     8.039
               3D Mesh      1.239        1.299     1.291
               Audio        0.334        0.343     42.496
               Web Query    1.184        1.212     1.353
               Web Response 1.128        1.153     2.166
               Spreadsheet  1.215        1.287     1.434



        Table 24: Mean values for Parse time in microseconds

                ASN1        TLV1        TVM1        Xnode       Xerces
Document        14695.2     14729.0     54534.8     362288.9    3345.1
Email           2273.9      2293.6      19733.3     172792.8    2304.0
3D Mesh         290622.6    292675.5    492572.4    1714020.8   26165.4
Audio           1129.8      1137.7      225670.3    2843950.2   12270.5
Web Query       1906.4      1910.5      4178.8      15265.7     1770.3
Web Response    6436.9      6489.9      19184.1     101459.2    2280.2
Spreadsheet     16914.4     17072.1     36955.4     118901.7    3003.0



               Table 25: Ratios to Xnode for Parse time

                                ASN1      TLV1     TVM1
               Document         0.041     0.041    0.151
               Email            0.013     0.013    0.114
               3D Mesh          0.170     0.171    0.287
               Audio            0.000     0.000    0.079
               Web Query        0.125     0.125    0.274
               Web Response     0.063     0.064    0.189
               Spreadsheet      0.142     0.144    0.311
                                                                           118
       Table 26: Mean values for Extract time in microseconds

             ASN1           TLV1         TVM1        Xnode      Xerces
Document     7509.9         7498.5       7513.0      12431.7    2626.0
Email        1180.2         1180.4       1187.4      7017.3     9979.9
3D Mesh      130770.7       130485.1     130645.6    332552.9   3190.4
Audio        454.0          452.2        461.2       170390.6   177754.0
Web Query    932.7          936.8        935.7       1218.9     3485.9
Web Response 3268.2         3260.9       3267.0      7046.3     3913.8
Spreadsheet  7237.0         7206.3       7226.4      4789.9     4209.0



               Table 27: Ratios to Xnode for Extract time

                                 ASN1      TLV1     TVM1
               Document          0.604     0.603    0.604
               Email             0.168     0.168    0.169
               3D Mesh           0.393     0.392    0.393
               Audio             0.003     0.003    0.003
               Web Query         0.765     0.769    0.768
               Web Response      0.464     0.463    0.464
               Spreadsheet       1.511     1.504    1.509



        Table 28: Mean values for Write time in microseconds

                ASN1        TLV1         TVM1        Xnode      Xerces
Document        19200.1     19336.2      32476.9     10837.7    5618.5
Email           2789.0      2835.4       10791.0     2069.1     2417.3
3D Mesh         356778.6    363219.8     360990.3    226959.5   124173.1
Audio           1530.1      1554.3       118679.2    20581.1    35129.7
Web Query       2412.0      2438.1       2552.3      1145.5     1008.6
Web Response    8127.7      8170.4       11181.0     3886.4     2103.2
Spreadsheet     20761.1     21291.5      22177.3     13182.6    7870.3
                                                                          119
               Table 29: Ratios to Xnode for Write time

                            ASN1          TLV1     TVM1
               Document     1.772         1.784    2.997
               Email        1.348         1.370    5.215
               3D Mesh      1.572         1.600    1.591
               Audio        0.074         0.076    5.766
               Web Query    2.106         2.128    2.228
               Web Response 2.091         2.102    2.877
               Spreadsheet  1.575         1.615    1.682



        Table 30: Mean values for Read time in microseconds

               ASN1       TLV1          TVM1       Xnode       Xerces
Document       22245.2    22261.3       62062.1    374739.0    5969.0
Email          3452.8     3473.3        20926.2    179864.3    12287.6
3D Mesh        421393.3   423160.6      623218.0   2046074.8   29413.0
Audio          1585.6     1591.9        226138.9   3014327.4   190109.2
Web Query      2840.1     2846.5        5127.9     16497.5     5255.2
Web Response   9749.1     9776.5        22466.9    108543.9    6196.5
Spreadsheet    24186.7    24286.0       44237.5    123713.9    7219.5



               Table 31: Ratios to Xnode for Read time

                                ASN1      TLV1     TVM1
               Document         0.059     0.059    0.166
               Email            0.019     0.019    0.116
               3D Mesh          0.206     0.207    0.305
               Audio            0.001     0.001    0.075
               Web Query        0.172     0.173    0.311
               Web Response     0.090     0.090    0.207
               Spreadsheet      0.196     0.196    0.358
                                                                             120
        Table 32: Mean values for Total time in microseconds

             ASN1         TLV1          TVM1       Xnode          Xerces
Document     41465.2      41597.5       94539.0    385587.8       11610.6
Email        6238.0       6311.9        31796.4    181956.2       14726.5
3D Mesh      778171.9     786380.5      984208.2   2272971.1      153586.2
Audio        3121.7       3154.6        344862.1   3035035.3      225484.1
Web Query    5256.6       5284.1        7697.4     17664.9        6264.1
Web Response 17921.6      17979.9       33684.6    112459.2       8300.7
Spreadsheet  44928.9      45633.7       66665.8    136933.2       15156.0



              Table 33: Ratios to Xnode for Total time

                                ASN1      TLV1     TVM1
             Document           0.108     0.108    0.245
             Email              0.034     0.035    0.175
             3D Mesh            0.342     0.346    0.433
             Audio              0.001     0.001    0.114
             Web Query          0.298     0.299    0.436
             Web Response       0.159     0.160    0.300
             Spreadsheet        0.328     0.333    0.487



         Table 34: Mean values for Natural data size in bytes

                  ASN1       TLV1        TVM1      Xnode        Xerces
  Document        6670.0     6670.0      6670.0    6670.0       6670.0
  Email           3736.4     3736.4      3736.4    3736.4       3736.4
  3D Mesh         9895.2     9895.2      9895.2    9895.2       9895.2
  Audio           54341.0    54341.0     54341.0   54341.0      54341.0
  Web Query       86.5       86.5        86.5      86.5         86.5
  Web Response    1539.1     1539.1      1539.1    1539.1       1539.1
  Spreadsheet     721.9      721.9       721.9     721.9        721.9
                                                                      121
          Table 35: Ratios to Xnode for Natural size

                       ASN1           TLV1     TVM1
          Document     1.000          1.000    1.000
          Email        1.000          1.000    1.000
          3D Mesh      1.000          1.000    1.000
          Audio        1.000          1.000    1.000
          Web Query    1.000          1.000    1.000
          Web Response 1.000          1.000    1.000
          Spreadsheet  1.000          1.000    1.000



      Table 36: Mean values for Packet data size in bytes

               ASN1      TLV1       TVM1       Xnode        Xerces
Document       7030.9    7178.0     7129.6     8948.9       8953.5
Email          3777.8    3803.2     3807.7     4511.6       4562.6
3D Mesh        11715.3   17170.4    15215.3    35027.9      34674.4
Audio          54354.0   54365.0    54517.0    74450.0      74504.0
Web Query      130.4     144.5      144.5      330.6        381.6
Web Response   1692.4    1727.9     1727.9     2431.5       2475.3
Spreadsheet    970.3     1214.7     1217.2     2522.2       2549.2


          Table 37: Ratios to Xnode for Packet size

                           ASN1      TLV1     TVM1
         Document          0.786     0.802    0.797
         Email             0.837     0.843    0.844
         3D Mesh           0.334     0.490    0.434
         Audio             0.730     0.730    0.732
         Web Query         0.394     0.437    0.437
         Web Response      0.696     0.711    0.711
         Spreadsheet       0.385     0.482    0.483


               Table 38: Mean values for Ratio

                  ASN1      TLV1    TVM1      Xnode    Xerces
  Document        1.1       1.1     1.1       1.3      1.3
  Email           1.0       1.0     1.0       1.2      1.2
  3D Mesh         1.2       1.7     1.5       3.5      3.5
  Audio           1.0       1.0     1.0       1.4      1.4
  Web Query       1.5       1.7     1.7       3.9      4.5
                                           122
Web Response 1.1   1.1   1.1   1.6   1.6
Spreadsheet  1.3   1.7   1.7   3.5   3.5
                                                                             123
               Table 39: Ratios to Xnode for Ratio of sizes

                            ASN1           TLV1      TVM1
               Document     0.784          0.801     0.795
               Email        0.836          0.842     0.843
               3D Mesh      0.334          0.490     0.434
               Audio        0.730          0.730     0.732
               Web Query    0.391          0.434     0.434
               Web Response 0.693          0.708     0.708
               Spreadsheet  0.384          0.481     0.482



                Table 40: Mean values for RT cost figure

               ASN1        TLV1          TVM1         Xnode       Xerces
Document       43711.3     44770.3       101059.0     517407.5    15611.3
Email          6308.4      6427.4        32407.6      219864.1    18026.9
3D Mesh        922064.5    1367618.3     1514495.8    8107432.2   540687.3
Audio          3121.7      3155.6        345978.6     4158155.9   309148.6
Web Query      7922.9      8832.9        12867.0      67574.5     27764.5
Web Response   19709.4     20189.1       37821.7      177728.0    13445.3
Spreadsheet    60403.2     76807.6       112438.5     478866.4    53529.6



               Table 41: Ratios to Xnode for RT cost figure

                                 ASN1      TLV1      TVM1
               Document          0.084     0.087     0.195
               Email             0.029     0.029     0.147
               3D Mesh           0.114     0.169     0.187
               Audio             0.001     0.001     0.083
               Web Query         0.117     0.131     0.190
               Web Response      0.111     0.114     0.213
               Spreadsheet       0.126     0.160     0.235
                                                                                                                        124

5.5     Histogram set 1 – Measured values for each data type

        The following sets of histograms show the distributions of experimental

measurements by protocol encoding type. The protocol types are colored and the key in

the upper-right corner shows the order. For readers with black and white copies of the

document the order of encodings is as follows: ASN1 (blue) is at the front, followed by

TVM1 (cyan), TLV1 (yellow) and Xnode (red) and the back. The vertical axis (Y) is the

number of samples in each range box. The front horizontal axis (X) is the set of range

bins for the measured value and the right hand side horizontal axis (Z) is the protocol

encoding.


5.5.1   Document data type

                                           Data type:doc, Value:Create
                                                                                                                ASN1
                  500
                                                                                                                TVM1
                                                                                                                TLV1
                  400                                                                                           Xnode



                  300


                  200


                  100


                   0
                   0
                        0.2
                              0.4
                                    0.6
                   x 10
                        4                 0.8
                                                1
                                                    1.2
                                                          1.4
                                                                1.6
                                                                      1.8               3 4
                                                                                    2
                                                                            2   1
                                                                                          Time (microseconds)



                                          Figure 14: Document create times
                                                                                                               125
                         Data type:doc, Value:Render
                                                                                               ASN1
600
                                                                                               TVM1
                                                                                               TLV1
500                                                                                            Xnode

400


300


200


100


 0
 0

          0.5

                    1
      4
 x 10                    1.5

                                 2

                                      2.5

                                               3
                                                                     3 4
                                                                 2
                                                       3.5   1
                                                                         Time (microseconds)



                    Figure 15: Document render times



                                Data type:doc, Value:Parse
                                                                                                       ASN1
      1000
                                                                                                       TVM1
                                                                                                       TLV1
          800                                                                                          Xnode



          600


          400


          200


            0
           -1

                    0

                           1
                5
           x 10                  2

                                        3

                                               4

                                                         5
                                                                               3 4
                                                                           2
                                                             6       1
                                                                                 Time (microseconds)



                        Figure 16: Document parse times
                                                                                                            126
                           Data type:doc, Value:Extract
                                                                                                    ASN1
350
                                                                                                    TVM1
300                                                                                                 TLV1
                                                                                                    Xnode
250

200

150

100

50

 0
0.2
        0.4
               0.6

      4               0.8
 x 10
                                 1
                                       1.2
                                                 1.4
                                                        1.6
                                                                            3 4
                                                                        2
                                                              1.8   1
                                                                              Time (microseconds)



              Figure 17: Document extract times



                            Data type:doc, Value:Write
                                                                                                    ASN1
400
                                                                                                    TVM1
350                                                                                                 TLV1
                                                                                                    Xnode
300

250

200

150

100

50

 0
 0
      0.5
               1
                     1.5
      4
 x 10                        2
                                 2.5
                                             3
                                                  3.5
                                                          4
                                                                            3 4
                                                                        2
                                                              4.5   1
                                                                              Time (microseconds)



              Figure 18: Document write times
                                                                                                 127
                        Data type:doc, Value:Read
                                                                                         ASN1
1000
                                                                                         TVM1
                                                                                         TLV1
800                                                                                      Xnode



600


400


200


  0
  -1
           0
                1

       5            2
  x 10
                            3
                                  4
                                        5
                                              6
                                                                 3 4
                                                             2
                                                     7   1
                                                                   Time (microseconds)



               Figure 19: Document read times



                        Data type:doc, Value:Total
                                                                                         ASN1
1000
                                                                                         TVM1
                                                                                         TLV1
800                                                                                      Xnode



600


400


200


  0
  -1
           0
                1

       5            2
  x 10
                            3
                                  4
                                        5
                                              6
                                                                 3 4
                                                             2
                                                     7   1
                                                                   Time (microseconds)



               Figure 20: Document total times
                                                                                                 128
                        Data type:doc, Value:Natural
                                                                                         ASN1
 200
                                                                                         TVM1
                                                                                         TLV1
                                                                                         Xnode
 150



 100



  50



   0
2000
       3000
              4000
                     5000
                            6000
                                   7000
                                          8000
                                                 9000
                                                                        3 4
                                                                    2
                                                        10000   1
                                                                          Size (bytes)



              Figure 21: Document natural sizes



                        Data type:doc, Value:Packet
                                                                                         ASN1
 300
                                                                                         TVM1
                                                                                         TLV1
 250                                                                                     Xnode

 200


 150


 100


  50


   0
2000

        4000

                 6000

                            8000

                                   10000

                                            12000
                                                                        3 4
                                                                    2
                                                        14000   1
                                                                          Size (bytes)



              Figure 22: Document packet sizes
                                                                                              129
                             Data type:doc, Value:Ratio
                                                                                      ASN1
1000
                                                                                      TVM1
                                                                                      TLV1
800                                                                                   Xnode



600


400


200


  0
  1
       1.05
                1.1
                      1.15
                               1.2
                                     1.25
                                            1.3
                                                  1.35
                                                                        3 4
                                                                    2
                                                          1.4   1
                                                                              Ratio



                Figure 23: Document size ratios



                              Data type:doc, Value:RT
                                                                                      ASN1
 900
                                                                                      TVM1
 800                                                                                  TLV1
                                                                                      Xnode
 700

 600

 500

 400

 300

 200

 100

   0
   0
            1
                  2

        5                3
   x 10
                                 4
                                        5
                                             6
                                                    7
                                                                        3 4
                                                                    2
                                                           8    1
                                                                          Cost



            Figure 24: Document RT cost figures
                                                                                                                         130


5.5.2   Email data type

                                            Data type:email, Value:Create
                                                                                                                 ASN1
                      350
                                                                                                                 TVM1
                      300                                                                                        TLV1
                                                                                                                 Xnode
                      250

                      200

                      150

                      100

                       50

                          0
                      500

                              1000

                                       1500

                                                   2000

                                                             2500
                                                                                     3 4
                                                                                 2
                                                                      3000   1
                                                                                           Time (microseconds)



                                     Figure 25: Email create times



                                            Data type:email, Value:Render
                                                                                                             ASN1
                     1000
                                                                                                             TVM1
                                                                                                             TLV1
                      800                                                                                    Xnode



                      600


                      400


                      200


                          0
                          0

                              2000

                                     4000

                                              6000

                                                      8000

                                                             10000
                                                                                     3 4
                                                                                 2
                                                                     12000   1
                                                                                       Time (microseconds)



                                     Figure 26: Email render times
                                                                                                         131


                           Data type:email, Value:Parse
                                                                                                 ASN1
1000
                                                                                                 TVM1
                                                                                                 TLV1
800                                                                                              Xnode



600


400


200


  0
-0.5

              0

                     0.5
        5
  x 10
                                 1

                                        1.5

                                                    2
                                                                         3 4
                                                                     2
                                                           2.5   1
                                                                           Time (microseconds)



                  Figure 27: Email parse times



                           Data type:email, Value:Extract
                                                                                                 ASN1
600
                                                                                                 TVM1
                                                                                                 TLV1
500                                                                                              Xnode

400


300


200


100


  0
500
       1000
              1500
                     2000
                             2500
                                     3000
                                            3500
                                                   4000
                                                                         3 4
                                                                     2
                                                          4500   1
                                                                           Time (microseconds)



                  Figure 28: Email extract times
                                                                                                    132
                       Data type:email, Value:Write
                                                                                            ASN1
800
                                                                                            TVM1
700                                                                                         TLV1
                                                                                            Xnode
600

500

400

300

200

100

  0
  0

         2000

                4000

                        6000

                                 8000

                                        10000
                                                                    3 4
                                                                2
                                                12000       1
                                                                      Time (microseconds)



                Figure 29: Email write times



                       Data type:email, Value:Read
                                                                                            ASN1
1000
                                                                                            TVM1
                                                                                            TLV1
800                                                                                         Xnode



600


400


200


  0
-0.5

           0

                 0.5
       5
  x 10
                            1

                                  1.5

                                            2
                                                                    3 4
                                                                2
                                                      2.5   1
                                                                      Time (microseconds)



                 Figure 30: Email read times
                                                                                                          133
                             Data type:email, Value:Total
                                                                                                  ASN1
1000
                                                                                                  TVM1
                                                                                                  TLV1
 800                                                                                              Xnode



 600


 400


 200


   0
 -0.5

               0

                      0.5
         5
   x 10
                                  1

                                         1.5

                                                     2
                                                                          3 4
                                                                      2
                                                            2.5   1
                                                                            Time (microseconds)



                      Figure 31: Email total times



                            Data type:email, Value:Natural
                                                                                                  ASN1
 200
                                                                                                  TVM1
                                                                                                  TLV1
                                                                                                  Xnode
 150



 100



  50



   0
3000
        3200
               3400
                      3600
                              3800
                                      4000
                                             4200
                                                    4400
                                                                          3 4
                                                                      2
                                                           4600   1
                                                                                Size (bytes)



                   Figure 32: Email natural sizes
                                                                                              134
                      Data type:email, Value:Packet
                                                                                      ASN1
 200
                                                                                      TVM1
                                                                                      TLV1
                                                                                      Xnode
 150



 100



  50



   0
3000

        3500

                     4000

                             4500

                                       5000
                                                                 3 4
                                                             2
                                                  5500   1
                                                                       Size (bytes)



               Figure 33: Email packet sizes



                       Data type:email, Value:Ratio
                                                                                      ASN1
1000
                                                                                      TVM1
                                                                                      TLV1
 800                                                                                  Xnode



 600


 400


 200


   0
   1

       1.05

               1.1

                      1.15

                              1.2

                                    1.25

                                            1.3
                                                                 3 4
                                                             2
                                                  1.35   1
                                                                   Ratio



                 Figure 34: Email size ratios
                                                                               135
                      Data type:email, Value:RT
                                                                       ASN1
1000
                                                                       TVM1
                                                                       TLV1
800                                                                    Xnode



600


400


200


  0
-0.5

           0

                0.5
       5
  x 10                1

                           1.5

                                    2

                                         2.5
                                                              3 4
                                                          2
                                                  3   1
                                                                Cost



               Figure 35: Email RT cost figures
                                                                                                                      136


5.5.3   3D-mesh data type

                                           Data type:mesh, Value:Create
                                                                                                              ASN1
                     250
                                                                                                              TVM1
                                                                                                              TLV1
                     200                                                                                      Xnode



                     150


                     100


                      50


                      0
                     0.4
                           0.6
                                 0.8
                                       1
                           5
                      x 10                  1.2
                                                  1.4
                                                        1.6
                                                              1.8
                                                                    2
                                                                                      3 4
                                                                                  2
                                                                        2.2   1
                                                                                        Time (microseconds)



                                  Figure 36: 3D mesh create times



                                           Data type:mesh, Value:Render
                                                                                                              ASN1
                     250
                                                                                                              TVM1
                                                                                                              TLV1
                     200                                                                                      Xnode



                     150


                     100


                      50


                      0
                     0.4
                           0.6
                                 0.8
                                       1
                           5
                      x 10                  1.2
                                                  1.4
                                                        1.6
                                                              1.8
                                                                    2
                                                                                      3 4
                                                                                  2
                                                                        2.2   1
                                                                                        Time (microseconds)



                                 Figure 37: 3D mesh render times
                                                                                                    137


                        Data type:mesh, Value:Parse
                                                                                            ASN1
700
                                                                                            TVM1
600                                                                                         TLV1
                                                                                            Xnode
500

400

300

200

100

 0
 0

          0.5

                    1
      6
 x 10
                            1.5

                                        2

                                                2.5
                                                                    3 4
                                                                2
                                                        3   1
                                                                      Time (microseconds)



                Figure 38: 3D mesh parse times



                        Data type:mesh, Value:Extract
                                                                                            ASN1
300
                                                                                            TVM1
                                                                                            TLV1
250                                                                                         Xnode

200


150


100


50


 0
 0
        0.5
                1

      5             1.5
 x 10
                              2
                                  2.5
                                            3
                                                 3.5
                                                                    3 4
                                                                2
                                                        4   1
                                                                      Time (microseconds)



              Figure 39: 3D mesh extract times
                                                                                                      138
                          Data type:mesh, Value:Write
                                                                                              ASN1
200
                                                                                              TVM1
                                                                                              TLV1
                                                                                              Xnode
150



100



50



 0
 1

           1.5

                      2
      5
 x 10
                                2.5

                                          3

                                                3.5
                                                                      3 4
                                                                  2
                                                         4    1
                                                                        Time (microseconds)



                 Figure 40: 3D mesh write times



                          Data type:mesh, Value:Read
                                                                                              ASN1
700
                                                                                              TVM1
600                                                                                           TLV1
                                                                                              Xnode
500

400

300

200

100

 0
 0

          0.5

                  1
      6
 x 10                     1.5

                                      2

                                          2.5

                                                  3
                                                                      3 4
                                                                  2
                                                        3.5   1
                                                                        Time (microseconds)



                 Figure 41: 3D mesh read times
                                                                                                         139
                              Data type:mesh, Value:Total
                                                                                                 ASN1
 500
                                                                                                 TVM1
                                                                                                 TLV1
 400                                                                                             Xnode



 300


 200


 100


   0
  0
         0.5
                     1

         6                  1.5
  x 10
                                     2
                                         2.5
                                               3
                                                   3.5
                                                                         3 4
                                                                     2
                                                             4   1
                                                                           Time (microseconds)



                     Figure 42: 3D mesh total times



                             Data type:mesh, Value:Natural
                                                                                                 ASN1
 200
                                                                                                 TVM1
                                                                                                 TLV1
                                                                                                 Xnode
 150



 100



  50



   0
4000
       5000
              6000
                     7000
                            8000
                                   9000
                                      10000
                                          11000
                                              12000
                                                  13000                  3 4
                                                                     2
                                                      14000      1
                                                                           Size (bytes)



             Figure 43:3D mesh natural data sizes
                                                                                               140
                           Data type:mesh, Value:Packet
                                                                                       ASN1
 400
                                                                                       TVM1
 350                                                                                   TLV1
                                                                                       Xnode
 300

 250

 200

 150

 100

  50

   0
 0.5
           1
                 1.5
                       2
       4
   x 10                     2.5
                                    3
                                        3.5
                                              4
                                                  4.5
                                                                      3 4
                                                                  2
                                                          5   1
                                                                        Size (bytes)



                 Figure 44: 3D mesh packet sizes



                           Data type:mesh, Value:Ratio
                                                                                       ASN1
1000
                                                                                       TVM1
                                                                                       TLV1
800                                                                                    Xnode



600


400


200


  0
  1

           1.5

                       2

                              2.5

                                         3

                                              3.5
                                                                      3 4
                                                                  2
                                                          4   1
                                                                            Ratio



                   Figure 45: 3D mesh size ratios
                                                                               141
                      Data type:mesh, Value:RT
                                                                       ASN1
900
                                                                       TVM1
800                                                                    TLV1
                                                                       Xnode
700

600

500

400

300

200

100

 0
 0
          2
              4
                  6
      6
 x 10                  8
                           10
                                12
                                     14
                                          16
                                                              3 4
                                                          2
                                                 18   1
                                                                Cost



          Figure 46: 3D mesh RT cost figures
                                                                                                                                           142


5.5.4   Audio data type

                                                     Data type:audio, Value:Create
                                                                                                                               ASN1
                     1000
                                                                                                                               TVM1
                                                                                                                               TLV1
                      800                                                                                                      Xnode



                      600


                      400


                      200


                          0
                          -2
                                   0
                                         2
                                                 4
                          x 10
                                 4                       6
                                                             8
                                                                 10
                                                                      12
                                                                                14
                                                                                     16                3 4
                                                                                                   2
                                                                                          18   1
                                                                                                         Time (microseconds)



                                             Figure 47: Audio create times



                                                     Data type:audio, Value:Render
                                                                                                                                   ASN1
                     1000
                                                                                                                                   TVM1
                                                                                                                                   TLV1
                      800                                                                                                          Xnode



                      600


                      400


                      200


                          0
                       -2
                                     0
                                             2

                               4                     4
                          x 10
                                                             6
                                                                  8
                                                                           10
                                                                                     12
                                                                                                       3 4
                                                                                                   2
                                                                                          14   1
                                                                                                             Time (microseconds)



                                             Figure 48: Audio render times
                                                                                                      143


                         Data type:audio, Value:Parse
                                                                                              ASN1
1000
                                                                                              TVM1
                                                                                              TLV1
800                                                                                           Xnode



600


400


200


  0
-0.5

           0

                   0.5
       6
  x 10                        1

                                  1.5

                                         2

                                               2.5
                                                                      3 4
                                                                  2
                                                          3   1
                                                                        Time (microseconds)



                    Figure 49: Audio parse times



                         Data type:audio, Value:Extract
                                                                                              ASN1
1000
                                                                                              TVM1
                                                                                              TLV1
800                                                                                           Xnode



600


400


200


  0
  -5

               0


  x 10
       4                  5

                                   10

                                             15
                                                                      3 4
                                                                  2
                                                        20    1
                                                                        Time (microseconds)



                   Figure 50: Audio extract times
                                                                                                                  144


                               Data type:audio, Value:Write
                                                                                                          ASN1
1000
                                                                                                          TVM1
                                                                                                          TLV1
800                                                                                                       Xnode



600


400


200


   0
  -2
               0
                       2

           4                  4
  x 10
                                      6
                                            8
                                                 10
                                                       12
                                                                              3 4
                                                                          2
                                                                 14   1
                                                                                    Time (microseconds)



                           Figure 51: Audio write times



                                  Data type:audio, Value:Read
                                                                                                      ASN1
 1000
                                                                                                      TVM1
                                                                                                      TLV1
  800                                                                                                 Xnode



  600


  400


  200


       0
  -0.5
                   0
                       0.5

               6               1
       x 10
                                     1.5
                                             2
                                                 2.5
                                                        3
                                                                              3 4
                                                                          2
                                                                3.5   1
                                                                                Time (microseconds)



                             Figure 52: Audio read times
                                                                                                                    145
                                Data type:audio, Value:Total
                                                                                                            ASN1
1000
                                                                                                            TVM1
                                                                                                            TLV1
800                                                                                                         Xnode



600


400


200


  0
-0.5
                   0
                        0.5

           6                    1
  x 10
                                        1.5
                                                2
                                                    2.5
                                                              3
                                                                                3 4
                                                                            2
                                                                  3.5   1
                                                                                      Time (microseconds)



                              Figure 53: Audio total times




                                Data type:audio, Value:Packet
                                                                                                       ASN1
 1000
                                                                                                       TVM1
                                                                                                       TLV1
  800                                                                                                  Xnode



  600


  400


  200


       0
       5

                       5.5


   x 10
               4                    6

                                              6.5

                                                          7
                                                                                3 4
                                                                            2
                                                                  7.5   1
                                                                                  Size (bytes)



                              Figure 54: Audio packet sizes
                                                                                                  146
                            Data type:audio, Value:Ratio
                                                                                          ASN1
1000
                                                                                          TVM1
                                                                                          TLV1
800                                                                                       Xnode



600


400


200


  0
 0.9
       0.95
               1
                    1.05
                           1.1
                                 1.15
                                        1.2
                                              1.25
                                                     1.3
                                                           1.35                 3 4
                                                                            2
                                                                  1.4   1
                                                                                  Ratio



                      Figure 55: Audio size ratios



                             Data type:audio, Value:RT
                                                                                          ASN1
1000
                                                                                          TVM1
                                                                                          TLV1
800                                                                                       Xnode



600


400


200


  0
-0.5
         0
              0.5
                      1
  x 10
         6                 1.5
                                   2
                                        2.5
                                                3
                                                     3.5
                                                             4                  3 4
                                                                            2
                                                                  4.5   1
                                                                                  Cost



               Figure 56: Audio RT cost figures
                                                                                                                                     147


5.5.5   Web query data type



                                               Data type:webq, Value:Create
                                                                                                                             ASN1
                      450
                                                                                                                             TVM1
                      400                                                                                                    TLV1
                                                                                                                             Xnode
                      350

                      300

                      250

                      200

                      150

                      100

                       50

                        0
                       0
                            200
                                    400
                                              600
                                                    800
                                                           1000
                                                                   1200
                                                                           1400
                                                                                                     3 4
                                                                                                 2
                                                                                      1600   1
                                                                                                       Time (microseconds)



                                  Figure 57: Web query create times



                                               Data type:webq, Value:Render
                                                                                                                             ASN1
                      200
                                                                                                                             TVM1
                                                                                                                             TLV1
                                                                                                                             Xnode
                      150



                      100



                       50



                        0
                      400
                            500
                                  600
                                        700
                                              800
                                                    900
                                                          1000
                                                                 1100
                                                                        1200
                                                                               1300                  3 4
                                                                                                 2
                                                                                      1400   1
                                                                                                       Time (microseconds)
                                                                                                  148
            Figure 58: Web query render times



                        Data type:webq, Value:Parse
                                                                                          ASN1
1000
                                                                                          TVM1
                                                                                          TLV1
800                                                                                       Xnode



600


400


200


  0
  0

            0.5


  x 10
       4                 1

                                    1.5

                                             2
                                                                  3 4
                                                              2
                                                    2.5   1
                                                                    Time (microseconds)



             Figure 59: Web query parse times



                        Data type:webq, Value:Extract
                                                                                          ASN1
250
                                                                                          TVM1
                                                                                          TLV1
200                                                                                       Xnode



150


100


 50


  0
400

           600

                  800

                             1000

                                     1200

                                            1400
                                                                  3 4
                                                              2
                                                   1600   1
                                                                    Time (microseconds)



            Figure 60: Web query extract times
                                                                                                149


                     Data type:webq, Value:Write
                                                                                        ASN1
300
                                                                                        TVM1
                                                                                        TLV1
250                                                                                     Xnode

200


150


100


 50


  0
500

          1000

                 1500

                              2000

                                         2500
                                                                3 4
                                                            2
                                                 3000   1
                                                                  Time (microseconds)



            Figure 61: Web query write times



                     Data type:webq, Value:Read
                                                                                        ASN1
900
                                                                                        TVM1
800                                                                                     TLV1
                                                                                        Xnode
700

600

500

400

300

200

100

  0
 0

          0.5

                 1
      4
 x 10
                        1.5

                                     2

                                           2.5
                                                                3 4
                                                            2
                                                   3    1
                                                                  Time (microseconds)



            Figure 62: Web query read times
                                                                                                        150
                         Data type:webq, Value:Total
                                                                                                ASN1
500
                                                                                                TVM1
                                                                                                TLV1
400                                                                                             Xnode



300


200


100


 0
 0

          0.5

                     1
      4
 x 10
                             1.5

                                         2

                                                   2.5
                                                                        3 4
                                                                    2
                                                            3   1
                                                                          Time (microseconds)



             Figure 63: Web query total times



                         Data type:webq, Value:Natural
                                                                                                ASN1
300
                                                                                                TVM1
                                                                                                TLV1
250                                                                                             Xnode

200


150


100


50


 0
50
        60
                70
                     80
                             90
                                   100
                                             110
                                                    120
                                                                        3 4
                                                                    2
                                                          130   1
                                                                          Size (bytes)



      Figure 64: Web query natural data sizes
                                                                                                   151
                         Data type:webq, Value:Packet
                                                                                           ASN1
400
                                                                                           TVM1
350                                                                                        TLV1
                                                                                           Xnode
300

250

200

150

100

 50

  0
 50
       100
              150
                        200
                              250
                                    300
                                           350
                                                 400
                                                                      3 4
                                                                  2
                                                       450    1
                                                                            Size (bytes)



             Figure 65: Web query packet sizes



                          Data type:webq, Value:Ratio
                                                                                           ASN1
1000
                                                                                           TVM1
                                                                                           TLV1
800                                                                                        Xnode



600


400


200


  0
  1

        1.5

                    2

                          2.5

                                    3

                                          3.5

                                                  4
                                                                      3 4
                                                                  2
                                                        4.5   1
                                                                        Ratio



              Figure 66: Web query size ratios
                                                                             152
                    Data type:webq, Value:RT
                                                                     ASN1
700
                                                                     TVM1
600                                                                  TLV1
                                                                     Xnode
500

400

300

200

100

 0
 0
        1
            2
                3
 x 10
      4             4
                        5
                            6
                                 7
                                     8
                                          9                 3 4
                                                        2
                                               10   1
                                                              Cost



        Figure 67: Web query RT cost figures
                                                                                                                                     153

5.5.6   Web query response data type



                                                   Data type:webr, Value:Create
                                                                                                                             ASN1
                      400
                                                                                                                             TVM1
                      350                                                                                                    TLV1
                                                                                                                             Xnode
                      300

                      250

                      200

                      150

                      100

                       50

                        0
                       0

                             1000

                                      2000

                                                  3000

                                                          4000

                                                                      5000

                                                                               6000
                                                                                                     3 4
                                                                                                 2
                                                                                      7000   1
                                                                                                       Time (microseconds)



                       Figure 68: Web query response create times



                                                  Data type:webr, Value:Render
                                                                                                                             ASN1
                      300
                                                                                                                             TVM1
                                                                                                                             TLV1
                      250                                                                                                    Xnode

                      200


                      150


                      100


                       50


                        0
                     1000
                            2000
                                   3000
                                          4000
                                                 5000
                                                        6000
                                                               7000
                                                                      8000
                                                                             9000
                                                                                10000                3 4
                                                                                                 2
                                                                                    11000    1
                                                                                                       Time (microseconds)



                      Figure 69: Web query response render times
                                                                                                                154


                              Data type:webr, Value:Parse
                                                                                                        ASN1
1000
                                                                                                        TVM1
                                                                                                        TLV1
 800                                                                                                    Xnode



 600


 400


 200


   0
  -2
         0
                2
                       4
  x 10
         4                    6
                                     8
                                           10
                                                  12
                                                         14
                                                              16                3 4
                                                                            2
                                                                   18   1
                                                                                  Time (microseconds)



  Figure 70: Web query response parse times



                             Data type:webr, Value:Extract
                                                                                                        ASN1
 300
                                                                                                        TVM1
                                                                                                        TLV1
 250                                                                                                    Xnode

 200


 150


 100


  50


   0
1000
       2000
              3000
                     4000
                            5000
                                   6000
                                          7000
                                                 8000
                                                        9000
                                                           10000                3 4
                                                                            2
                                                               11000    1
                                                                                  Time (microseconds)



 Figure 71: Web query response extract times
                                                                                                    155
                      Data type:webr, Value:Write
                                                                                            ASN1
 300
                                                                                            TVM1
                                                                                            TLV1
 250                                                                                        Xnode

 200


 150


 100


  50


   0
2000

       4000

              6000

                     8000

                            10000

                                    12000

                                            14000
                                                                    3 4
                                                                2
                                                    16000   1
                                                                      Time (microseconds)



  Figure 72: Web query response write times



                      Data type:webr, Value:Read
                                                                                            ASN1
1000
                                                                                            TVM1
                                                                                            TLV1
 800                                                                                        Xnode



 600


 400


 200


   0
  -5

              0


  x 10
       4              5

                                10

                                            15
                                                                    3 4
                                                                2
                                                      20    1
                                                                      Time (microseconds)



   Figure 73: Web query response read times
                                                                                                                   156
                             Data type:webr, Value:Total
                                                                                                           ASN1
  800
                                                                                                           TVM1
  700                                                                                                      TLV1
                                                                                                           Xnode
  600

  500

  400

  300

  200

  100

   0
   0
        0.2
               0.4
                      0.6
   x 10
         5                  0.8
                                    1
                                        1.2
                                              1.4
                                                    1.6
                                                          1.8                  3 4
                                                                           2
                                                                   2   1
                                                                                     Time (microseconds)



    Figure 74: Web query response total times



                             Data type:webr, Value:Natural
                                                                                                       ASN1
  300
                                                                                                       TVM1
                                                                                                       TLV1
  250                                                                                                  Xnode

  200


  150


  100


   50


    0
  800
        1000
               1200
                      1400
                             1600
                                    1800
                                           2000
                                                  2200
                                                         2400
                                                                               3 4
                                                                           2
                                                                2600   1
                                                                                 Size (bytes)



Figure 75: Web query response natural data sizes
                                                                                                157
                        Data type:webr, Value:Packet
                                                                                        ASN1
 300
                                                                                        TVM1
                                                                                        TLV1
 250                                                                                    Xnode

 200


 150


 100


  50


   0
 500

        1000

               1500

                       2000

                                2500

                                       3000

                                                3500
                                                                       3 4
                                                                   2
                                                        4000   1
                                                                         Size (bytes)



  Figure 76: Web query response packet sizes



                        Data type:webr, Value:Ratio
                                                                                        ASN1
1000
                                                                                        TVM1
                                                                                        TLV1
800                                                                                     Xnode



600


400


200


  0
  1
        1.1
               1.2
                      1.3
                              1.4
                                    1.5
                                          1.6
                                                  1.7
                                                                       3 4
                                                                   2
                                                         1.8   1
                                                                             Ratio



       Figure 77: web query response size ratios
                                                                            158
                    Data type:webr, Value:RT
                                                                    ASN1
800
                                                                    TVM1
700                                                                 TLV1
                                                                    Xnode
600

500

400

300

200

100

 0
 0

          0.5

                1
      5
 x 10
                      1.5

                               2

                                     2.5
                                                           3 4
                                                       2
                                               3   1
                                                             Cost



Figure 78: web query response RT cost figures
                                                                                                                                159

5.5.7   Spreadsheet data type



                                             Data type:spread, Value:Create
                                                                                                                        ASN1
                      300
                                                                                                                        TVM1
                                                                                                                        TLV1
                      250                                                                                               Xnode

                      200


                      150


                      100


                       50


                        0
                      0.4
                              0.6
                                     0.8

                             4               1
                       x 10
                                                   1.2
                                                          1.4
                                                                  1.6
                                                                          1.8
                                                                                                3 4
                                                                                            2
                                                                                   2    1
                                                                                                  Time (microseconds)



                                 Figure 79: Spreadsheet create times



                                             Data type:spread, Value:Render
                                                                                                                        ASN1
                      300
                                                                                                                        TVM1
                                                                                                                        TLV1
                      250                                                                                               Xnode

                      200


                      150


                      100


                       50


                        0
                     4000
                            5000
                                    6000
                                           7000
                                                  8000
                                                         9000
                                                                10000
                                                                        11000
                                                                                                3 4
                                                                                            2
                                                                                12000   1
                                                                                                  Time (microseconds)



                                 Figure 80: Spreadsheet render times
                                                                                                                  160


                              Data type:spread, Value:Parse
                                                                                                          ASN1
1000
                                                                                                          TVM1
                                                                                                          TLV1
800                                                                                                       Xnode



600


400


200


  0
  0
          0.2
                 0.4
                        0.6
  x 10
           5                  0.8
                                      1
                                          1.2
                                                1.4
                                                      1.6
                                                            1.8               3 4
                                                                          2
                                                                  2   1
                                                                                    Time (microseconds)



                Figure 81: Spreadsheet parse times



                              Data type:spread, Value:Extract
                                                                                                      ASN1
 300
                                                                                                      TVM1
                                                                                                      TLV1
 250                                                                                                  Xnode

 200


 150


 100


  50


      0
3000
          4000
                 5000
                        6000
                               7000
                                      8000
                                             9000
                                                 10000
                                                       11000
                                                                              3 4
                                                                          2
                                                              12000   1
                                                                                Time (microseconds)



               Figure 82: Spreadsheet extract times
                                                                                                          161
                          Data type:spread, Value:Write
                                                                                                  ASN1
300
                                                                                                  TVM1
                                                                                                  TLV1
250                                                                                               Xnode

200


150


100


50


 0
0.5

              1

                    1.5
      4
 x 10
                                2

                                          2.5

                                                    3
                                                                      3 4
                                                                  2
                                                        3.5   1
                                                                            Time (microseconds)



              Figure 83: Spreadsheet write times



                          Data type:spread, Value:Read
                                                                                              ASN1
900
                                                                                              TVM1
800                                                                                           TLV1
                                                                                              Xnode
700

600

500

400

300

200

100

  0
  0

              0.5


  x 10
          5                1

                                    1.5

                                                2
                                                                      3 4
                                                                  2
                                                        2.5   1
                                                                        Time (microseconds)



              Figure 84: Spreadsheet read times
                                                                                                    162
                        Data type:spread, Value:Total
                                                                                            ASN1
800
                                                                                            TVM1
700                                                                                         TLV1
                                                                                            Xnode
600

500

400

300

200

100

  0
 0

           0.5


 x 10
       5                 1

                                 1.5

                                              2
                                                                    3 4
                                                                2
                                                      2.5   1
                                                                      Time (microseconds)



              Figure 85: Spreadsheet total times



                       Data type:spread, Value:Natural
                                                                                            ASN1
200
                                                                                            TVM1
                                                                                            TLV1
                                                                                            Xnode
150



100



 50



  0
500

        600

                 700

                        800

                               900

                                       1000

                                              1100
                                                                    3 4
                                                                2
                                                     1200   1
                                                                      Size (bytes)



      Figure 86: Spreadsheet natural data sizes
                                                                                               163
                         Data type:spread, Value:Packet
                                                                                       ASN1
 300
                                                                                       TVM1
                                                                                       TLV1
 250                                                                                   Xnode

 200


 150


 100


  50


   0
 500

       1000

              1500

                         2000

                                  2500

                                         3000

                                                3500
                                                                      3 4
                                                                  2
                                                       4000   1
                                                                        Size (bytes)



          Figure 87: Spreadsheet packet sizes



                         Data type:spread, Value:Ratio
                                                                                       ASN1
1000
                                                                                       TVM1
                                                                                       TLV1
800                                                                                    Xnode



600


400


200


  0
  1

        1.5

                     2

                                2.5

                                          3

                                                3.5
                                                                      3 4
                                                                  2
                                                          4   1
                                                                            Ratio



              Figure 88: Spreadsheet size ratios
                                                                            164
                  Data type:spread, Value:RT
                                                                    ASN1
900
                                                                    TVM1
800                                                                 TLV1
                                                                    Xnode
700

600

500

400

300

200

100

 0
 0
          1
              2

      5           3
 x 10
                      4
                            5
                                  6
                                        7
                                                           3 4
                                                       2
                                               8   1
                                                             Cost



      Figure 89: Spreadsheet RT cost figures
                                                                                          165


                                       CHAPTER VI

                                     CONCLUSIONS


         In this chapter we examine the results and draw conclusions about the

performance of the tested protocols for the set of tested data types. We begin with a brief

overall analysis of the results and follow that with detailed analysis for each data type and

for each measurement value. The chapter concludes with a review of the completion and

success criteria and some notes on other observations made from the results.


6.1      Overall analysis

         Recall that the derived RT cost figure is the product of the total time to create,

render, parse and extract data, and the ratio of packet size to natural size. This value gives

an overall cost associated with using a given protocol and a given data type. The

computed values for RT are shown in Table 40 which includes the RT values for Xerces

as a comparison. A more useful view of the results is show in Table 41 which shows the

ratio of the RT cost figure for the ASN.1, TLV1 and TVM1 protocols to the Xnode

(XML) protocol implementation. This table shows how each of the binary protocols

compares to the XML implementation for each data type. We can see that in every case,

the binary protocols performed much better than the XML protocol. All of the binary

implementations performed better than XML and they all performed better for every data

type tested. The improvement varies from about four times to more than a thousand

times.
                                                                                      166
        We will now break down the results by data type since some data types showed

much more improvement than others when encoded with a binary protocol. We will

follow that with some analysis of the individual measurements and attempt to determine

what factors are in play that affect protocol performance.


6.2     Analysis by data type

        In this section we look at the results for each data type and draw some

conclusions about over protocol performance for each type.


6.2.1   Audio

        The Audio data type consisted of a block of binary data with a small amount of

textual meta-data. The binary data provided the bulk of each packet and so this data type

can be considered as representative of any data type in which binary content is the

majority of the payload. This includes audio, video and images.

        Table 6 shows the measured and derived values for the Audio data type as

measured for each protocol encoding. This table also includes the Xerces values for

reference, but recall that the implementation of Xerces has been optimized so direct

comparison of the Xerces figures with the other protocol encodings does not provide

much insight.

        One obvious factor is the ratio of encoded packet size to raw data size which is

show pictorially in Figure 55. In the binary encodings, the raw binary data is transferred

to the packet either intact or modified slightly (in the TVM1 case). For the XML

encoding, the binary data needs to be converted to text which takes up more space. The
                                                                                       167
Base64 encoding uses four characters to encode three bytes of data so the resulting

encoded image is 33% bigger. It also takes time to convert the binary data to text as part

of the rendering process and to convert the text back to binary as part of the parsing

process. Figure 49 and Figure 50 show the parse and extract times graphically. Having to

convert from text to binary increases the overall time and compounds the effect of the

increased packet size giving a much worse overall RT cost figure as depicted in Figure 56

which also shows the slightly worse performance of TVM1 when compared to the other

two binary encodings.


6.2.2   Document

        The Document data type consists entirely of blocks of text. The bulk of the

content is a few paragraphs of text. The remainder is a collection of meta data such as the

author's name. Table 8 shows the mean values for the Document data type. All the

protocols are good at encoding text since the text itself can be copied directly to the

output stream. The differences lie in how the meta data is used to describe the text blocks

and what costs are associated with that meta-data. We see that Xnode and Xerces show

slightly larger resulting packets than ASN.1, TLV1 and TVM1. Recall that since XML

protocols use textual tags for the nodes, longer tags make for a larger packet. In the

experimental code, short tags were used which provided enough context to make the

XML human-readable but these tags like "Author" are none the less larger than the one or

two bytes used in the binary protocols to identify the same node.

        The major difference in performance shows in the parse time (and the derived

read and total times) which are shown graphically in Figure 16 and Figure 17. The
                                                                                        168
parsing of XML is less efficient than for the binary protocols. This can be explained as

being largely due to the length of the start and end node tags. Any test involving a tag

name requires some sort of string compare operation and this will take more time if the

tags are longer. The problem occurs mostly at the end of a node when it is necessary to

compare the end tag with the start tag to confirm that the end tag is a match for the start

tag of that node. In a binary protocol using a TLV structure this is not required because

there is no end tag to search for. In a TVM structure, the end tag is a short sequence (2

bytes in the case of TVM1) and this test is generally faster than the string compare

required in the XML protocol.

        The overall results show that the Document type benefits from the binary

protocols because parsing the encoded form is faster. It should be noted that The Xerces

parses does a much better job than the Xnode parser used here and is in fact faster than all

of the binary protocols. Xerces uses a much more sophisticated parsing scheme than the

one used in the experimental protocols. It might be expected that optimizing the binary

and Xnode protocol parsers to use the same approach used by Xerces (and other modern

parsers) would result in improved performance but the ratio of performance between the

Xnode parser and the binary parsers would stay constant since parsing is a function of the

complexity of the encoded data and that is not changed by optimizing the parser.


6.2.3   Email

        The email type has similar characteristics to the Document type except that it also

includes a small binary element to represent an attachment such as a small image or

digital certificate. The presence of the binary element increases the packet size for the
                                                                                                          169
XML encodings and also contributes to an increased rendering and parsing time (See the

commentary above about the Audio type). Table 10 shows the mean values for the email

type and Table 11 shows ratios of the binary encodings compared to the Xnode encoding.

           Packet size ratios for the XML encodings are not much worse than for the binary

protocols because the binary element in the data is not large compared to the textual

content. As with the document type, the most significant factor is the parsing time. The

Xnode parser performs much worse than the binary parsers and this greatly affects the

overall result as can be seen in Figure 30 which shows the email read times (sum of parse

and extract times).


6.2.4      3D Mesh

           The 3D Mesh data uses a large number of floating point values to represent the

vertices and faces of a 3D object. The data structure contains a lot of meta-data compared

to the size of the information since there are a large number of small items. Table 12

shows the mean values for the 3D Mesh data type. Looking at the ratio of encoded packet

size to raw data size we can see that the XML encodings generally use more space than

the binary encodings. It is worth noting that a textual representation of a floating point

value varies greatly in length according to the actual value encoded whereas a binary

encoding is always of a fixed size.3 Probably the most significant contribution to the

increased packet sizes in the XML encodings is due to the large amount of meta data. For

each number encoded, XML must include a start marker and an end marker. The size of

these tags can easily be greater than the size of the encoded data. For example:

3
    Actually, a binary encoding need not be a fixed size but all of the encodings in common use are fixed size.
                                                                                        170
<SomeNumber>0</SomeNumber> is a possible encoding for a zero value which is

obviously very space-inefficient. The binary encodings need to include tags with each

value too but the tags are typically shorter. For ASN.1 the tag and length values can be

only one byte each which compares favorably to an 8-byte double-precision encoded

value. The TLV1 and TVM1 protocols use 2-byte tags. TLV1 uses a 2-byte length field

and TVM1 uses a 2-byte end marker so in both of these protocols, four bytes of meta data

accompany the encoded value. This is still better than the XML case since the smallest

possible tag is one character giving a node like: "<a>123.56</a>" where the tags require

seven characters. In terms of overall results, Table 13 shows the mean ratios of the binary

protocol measurements compared to the Xnode protocol. Note how much better the

overall RT figure is for the binary encodings. This is shown graphically in Figure 46

which also shows how much the cost figures can vary. This variation is caused by the

randomizing of the data. Different data values result in quite different encoded node sizes

as mentioned above.


6.2.5   Web Query

        The Web Query type contains several small textural data items and consequently

has a high ratio of meta-data to raw data. Table 16 shows the mean values for the Web

Query type. The large amount of meta data makes the XML encoded packets much larger

than the binary encoded forms. It is interesting to note that the Xerces encodings are quite

a bit larger than the Xnode encodings. This is because the packets are small and Xerces

includes a lot of XML header information that Xnode does not. This header data is large

compared to the rest of the packet when the overall amount of data in the packet is small.
                                                                                          171
The binary encodings also show fairly poor encoded packet size ratios because of the

amount of meta-data required compared to the raw data volume.

        As with the other data types, parsing plays a big part in the overall performance

and again we see that the Xnode parser contributes largely to the overall cost. The

combined RT cost figures are shown graphically in Figure 67.


6.2.6   Web Query Response

        The Web Query Response data shown in Table 18 show similar results to the Web

Query values except that packet ratios improve because there is more raw data compared

to the amount of meta-data. This affects the binary and XML protocols in the same way.

Lots of small items in the packet always increase the relative encoded packet size because

of the need for more meta-data to describe the objects. The overall RT cost results in

Figure 78 show significant spread in the Xnode results and some spread in the TVM1

results. A possible factor here is that larger bodies of data take more time to search for the

end markers used in Xnode and TVM1 encodings and since we get some time variation

caused by other processes running on the test host, larger packets involve more variation

in measured time.


6.2.7   Spreadsheet

        The Spreadsheet type is very similar to the 3D Mesh type except that there is

more textual data present in the form of the row and column descriptions so overall there

is a slightly higher ratio of text to numeric data than for the 3D Mesh type. However the
                                                                                       172
results for packet size shown in Table 14 are very similar to those of the 3D Mesh type as

are the overall RT cost figures depicted in Figure 89.


6.2.8   Text vs. binary content

        From the sections above which describe the results for each data type we can see

that each of the protocols does a good job with textual data and also for the most part

with numeric data providing that there are not a large number of either small text items or

numeric values. With large numbers of numeric data or small text items, the amount of

meta-data increases and this makes the encoded packets larger. This affects XML and

binary encodings in the same way but the XML packets typically get bigger faster

because of the size of the text required to create the node tags.

        Where binary content is present in the data, XML packets grow significantly

because of the need to convert the binary data to a textual form. This increases packet

sizes and also increases rendering and parsing times. We can summarize by saying that

the XML protocols are very poor at handing binary data compared to any of the binary

encodings.


6.3     Analysis by measured value

        The following sections examine the individual measurements and derived values

and attempt to provide some insight into how each was affected by the protocol encoding

and data type.
                                                                                         173
6.3.1   Create time

        Create time is the time required to convert the raw data into a tree of nodes. This

process would seem to be largely independent of the protocol encoding, and it is except

for two factors. First, node objects for XML trees can contain attributes whereas the node

objects for the ASN.1, TLV1 and TVM1 protocols do not, so the later are slightly simpler

to construct. Second, the XML nodes can only contain text, so binary and numeric data

items must be converted into text when the node is constructed. Data types with more

objects and thus more nodes can take longer to construct than data types with fewer

objects. In the implementations used in this work, Xnode objects use C++ string objects

as the basic container. The binary protocols used a smart pointer to a dynamically

resizable buffer object which is slightly more complex to construct.

        Table 20 shows the mean times for node construction and Table 21 shows the

ratios for the binary protocols to the Xnode protocol. We can see that the construction

time ratios in Table 21vary quite a lot and that the binary protocol implementations cost

more in construction time than the Xnode objects which is consistent with the more

complex way that the data is stored in these nodes. A simpler approach using fixed-size

allocated blocks might lead to faster times.

        The audio data fared the worst in the Xnode case because of the need to convert

the large amount of binary content in it to text for insertion in the XML data object.


6.3.2   Render time

        Render time is the time taken to convert the data in the node tree format into the

protocol packet encoded format. Each protocol has its own rules that govern encoding of
                                                                                         174
various data types and in addition, protocols that use a tag-length-variable format such as

ASN.1 and TLV1 require that each node's size is computed before any data can be

rendered. This is required because the first item to be rendered to the output stream will

be the root node and the length of this node can only be determined from the lengths of

all its child nodes. So rendering any TLV format tree essentially involves walking the

tree twice: once to compute all the node lengths and once to create the output stream.

        The mean rendering times are shown in Table 22. Looking at the ratios of render

times                                                                                     in
                                                                                       175
Table 23 we can see one notable fact that the TVM1 render times are much longer than

for the other binary protocols for most of the data types. This is caused by the end marker

in the TVM1 protocol which consists of a sequence of two bytes. This sequence must not

be allowed to occur in the data body of any node so the whole body must be scanned as it

is rendered and any occurrence of the two-byte marker sequence escaped so that it will

not look like an end-marker when the packet is parsed later. This scanning of the data is

very costly.


6.3.3   Write time

        Write time is the sum of the create and render time. The ratio values shown in
                                                                                                        176
Table 29 show that for the most part, creating the binary encodings is actually more

costly than the XML case except when there is a large amount of binary data present as in

the Audio data type. When a lot of binary content is present, the cost of doing the binary

to text conversion outweighs other factors and the write times for ASN.1 and TLV1

become much small than for the Xnode protocol. We should note here again that TVM1

incurs additional costs when writing binary data because it must parse the data body and

escape any byte sequences that match the end-marker sequence and this takes

considerable CPU time.4


6.3.4    Parse time

         Parse time is the time taken to convert the encoded form into a tree of nodes.

Parse time variation was probably the most significant result of the experiments. Parse

times varied far more than other times for most combinations of data type and protocol

indicating that work done to improve parser performance would have significant impact

on overall performance. The parse time ratios shown in Table 25 show that all of the

binary encodings performed better than the XNode encoding for all of the data types

tested. The ratios for audio performance were more than 1000 times better for ASN.1 and

TLV1 when compared to Xnode. If we examine the Xerces parser data in Table 24, we

see that Xerces does a far better job than the Xnode parser but is still more than ten times

worse than ASN.1 and TLV1. This is a significant factor considering that the ASN.1 and

TLV1 implementations are not optimized in any way.



4
 It is worth noting that this operation is well suited to hardware acceleration which would make this
encoding much more efficient.
                                                                                       177
6.3.5   Extract time

        Extract time is the time taken to extract the raw data from the node tree created

during the parse phase. Extracting the raw data from the node tree has more to do with

the way data is stored in the node trees than the protocol encodings and thus represents an

artifact of the implementation rather than the protocol. In most of the cases shown in

Table 27 extraction times for the binary encodings are not that different from the Xnode

case. The exception is for the Audio data type where extracting large amounts of binary

data was very costly. This indicates an area of the implementation which could probably

be improved by altering the way the binary data is stored in the tree nodes. See the

section below on areas for future work for more details.


6.3.6   Read time

        Read time is the sum of the parse and extract time values and represents the

practical cost of extracting the raw data from the protocol's encoded form. The ratio

values in Table 31 show again that the binary protocol encodings perform much better

than the Xnode encoding for all data types and especially for the Audio data type.

Looking at the mean values in Table 30 we can see that the Xerces parser is a lot slower

than ASN.1 and TLV1 but better than TVM1 and Xnode. So again, we see that the un-

optimized ASN.1 and TLV1 encodings do much better than event the optimized Xerces

parser when the data is primarily binary.
                                                                                       178
6.3.7   Total time

        Total time is the sum of the read and write times which is also the same as the

sum of the create, render, parse and extract times. The ratios in Table 33 show that all of

the binary encodings have better performance than the Xnode encoding for every data

type and especially for the audio type. The total time value gives us a useful time

measurement for systems which both send and receive binary data since a host at either

end of a communications link will need to both read and write the data in its encoded

form. The data show quite clearly that from a CPU cost point of view there are significant

advantages to using a binary encoding to transport binary data.


6.3.8   Natural size

        The natural size data in Table 34 is included for completeness since it documents

the sizes of the data used in the experiment but since the data was the same for each

protocol tested there is no variation by protocol to be analyzed.


6.3.9   Packet size

        The ratios of packet sizes for the binary encodings compared to the Xnode

encoding shown in Table 37 provide insight into the amount of meta data required for

each encoding and we can clearly see that in all cases the meta data requirements for the

binary protocols is less than for the Xnode protocol. As was discussed above this is due

to the rather verbose nature of the textual tags used in the XML encoding scheme.
                                                                                       179
6.3.10 Ratio of packet size to natural size

       A clearer expression of the meta-data costs is shown in Table 38 which shows the

mean packet size ratios. These figures show how large numbers of small data items

reduce the overall packing efficiency because of the need for increased amounts of meta-

data. For very large numbers of small items the meta-data can easily equal or exceed the

size of the encoded raw data. The worse case in Table 38 is for the Xerces parser with the

Web Query Data. In this particular case, the amount of data in the packet is very small

and is swamped by the XML header that Xerces generates. The Xnode protocol which

does not include the header does a little better but it is clear that the binary encodings

perform the best.


6.3.11 RT cost figure

       The RT cost figures in Table 40 and the ratios in Table 41 show the overall cost

results for each data type and protocol. The ratio values in Table 41 summarize the results

of the experiments well. We can see that all three binary encodings tested out-perform the

Xnode XML implementation for all the data types and their performance for the Audio

data type is outstanding.


6.4    Analysis by protocol

       The following sections provide a brief summary of the results for each of the

binary encodings tested as compared to the Xnode XML encoding.
                                                                                       180
6.4.1    ASN.1

         ASN.1 has been in commercial use in various encoding forms for many years

now. The Distinguished Encoding Rules (DER) variant which the code for this work was

modeled on is not the most efficient in terms of encoded space (CER and others are

better) but its rules are simple to implement and its variable length tags and node-length

fields mean that it has a very efficient encoded form for almost any data type.

         The ASN.1 implementation used in this work out-performed the custom TLV1

and TVM1 protocols in overall RT cost figure for every data type. It was the best

performer in terms of total CPU time as shown in Table 33. This was not expected. The

rules for ASN.1 are more complex than those for TLV1 and the use of variable-length

fields was expected to be more costly in CPU time than the fixed-length fields in TLV1

and TVM1.

         ASN.1 was also the winner for encoded packet size as shown by the ratios in

Table 37 although the difference between ASN.1 and the other two binary protocols is

small.

         ASN.1 compares very favorably in terms of CPU time, encoded size and overall

cost to XML. The code used to implement the ASN.1 protocol and the Xnode protocol

for these experiments is very similar with the exception of how data is stored in the nodes

which was shown to be more efficient in the Xnode case. So some work to improve data

storage in the binary protocol nodes could further improve ASN.1 over XML for the

same set of tests.
                                                                                         181
6.4.2   TVL1

        The TLV1 protocol performed better in terms of CPU time, packet size and

overall cost than the Xnode protocol. It did not perform quite as well as ASN.1 and this

was something of a disappointment. The TLV1 design was meant to be a simplified

version of ASN.1 with fixed-length fields because it was expected that the CPU time

needed to deal with the fixed-length fields would be less than the variable-length fields

used in ASN.1. Any reduction in time cost was expected to show up significantly in the

data structures which used a lot of elements. Since the data used in these experiments was

all relatively small (less than 128 bytes) in most cases, the ASN.1 protocol is able to use a

single byte for the node field length and the code required to generate that single-byte

field and parse it from its encoded form turns out to be faster than the code to generate

and parse the two-byte field used in TLV1. So we might conclude that simplicity of

design does not necessarily lead to better performance either in terms of encoded size or

in terms of CPU cost.

        It should also be noted that the TLV1 design used a 2-byte length field which

limits it to packets of just less than 64K bytes in size. This is adequate for many protocol

needs but is none the less limiting. A 4-byte field would be more practical but this would

further reduce the performance with respect to the ASN.1 encoding. Converting TLV1 to

use variable-length fields for both node tags and node length fields might make it faster

than ASN.1. Such a change in design essentially turns TLV1 into ASN.1 and it is more

obvious to simply choose to use a good implementation of ASN.1 than a custom protocol

which is very similar.
                                                                                     182
6.4.3   TVM1

        The performance of the TVM1 protocol was better than the Xnode XML

encoding for all data types. It produced small packets than the Xnode encoding and used

less CPU time. It was, however, the worst of the three binary encodings. It total time

figures were nearly twice those of ASN.1 for the Web Query data and more than twice

for the Document type. In terms of encoded packet size, it performed almost as well as

the other binary protocols and far better than the Xnode XML encoding.

        The TVM1 protocol was intended as a binary form of XML. It uses start and end

tags as XML does but could carry a binary payload in each node which XML cannot

without translating the payload to a textual form.

        The poor performance of TVM1 compared to ASN.1 and TLV1 can be attributed

to the problem of ensuring that the two-byte end-marker sequence does not occur in the

node body. Instead of being able to copy the binary node body to the output stream as is

done in the ASN.1 and TLV1 protocols, the stream had to be parsed as it was written and

if the end-marker sequence was detected it had to be replaced by an escape sequence so

that the parser would not see the end of the node too early. The parser was also burdened

with having to watch for the escape sequence and translating that back to the proper data

sequence when encountered.

        So we can conclude that the simple idea of turning the string tags in XML to

binary tags and replacing the string content of XML nodes with binary is not a good way

to improve XML. It does improve performance but the gains are not as good as those

obtained by either the TLV1 or ASN.1 encodings.
                                                                                        183
6.5     Analysis of completion criteria

6.5.1   Protocol encodings

        The set of protocol encodings to be tested was identified and described in

sufficient detail to allow the protocol encoding and decoding code can be implemented

correctly.


6.5.2   Data types

        A set of data types was identified that all tests could be applied to. These were

described in sufficient detail that the data generation code could be written from the

specifications.


6.5.3   Framework

        A code framework for the tests was designed and written with a common

methodology so that for each data type and each protocol encoding, the code written took

as similar an approach as possible.


6.5.4   Data-generation code

        All the data generation code was written and tested to ensure that the generated

test sets conform to the specifications for the data sets and that the content was generated

randomly.


6.5.5   Protocol testing

        All the protocols to be tested were implemented using the program framework

and were tested against the protocol specification and verified as producing correct
                                                                                     184
encodings. The code was verified that it parsed the encoded form correctly. The testing

was completed largely with the aid of some automated testing that executed a set of test

cases. The final encoded forms were manually inspected to ensure that they correctly

conformed to the protocol specifications.


6.5.6   Initial testing

        Several initial test runs were done to find any bugs in the code and to show that

the implementation worked correctly and that the data logged was correct. After the

initial bugs were removed, the test programs showed no obvious bad behavior such as

memory leaks. No obvious anomalies were observed that might indicate that the

implementation of any protocol or data set generator was incorrect. Test runs with many

data generation cycles and many run repeats were conducted before the final set of test

data was obtained. Test cycles took from a few minutes to about six hours depending on

the number of repeats used.




6.6     Analysis of success criteria

6.6.1   Comparison of overall costs


        For all of the data types tested, the binary encodings show reduced costs

compared to the costs measured for XML. Costs for all three binary encodings were

lower than for the Xnode XML encoding in terms of packet size, CPU time and overall

cost.
                                                                                    185
6.6.2   Comparison of costs for binary data

        For the data types that have binary data as the primary type, all the binary

protocol encodings greatly reduced overall cost figures when compared to the XML

encoding. Improvement in the mean value of the overall RT cost figure varied from about

4X for the TVM1 protocol and Spreadsheet data to about 1000X for the ASN.1 and

TLV1 encodings with the Audio data type.



6.7     Rejection of null hypothesis

        The null hypothesis for this work was stated above as:

        H0: Applications which currently use XML as a data encoding would see the

same or higher costs by using an alternative binary encoding form.

        The results of the experimental work clearly show that the null hypothesis is

rejected for all combinations of the data types and the binary protocol encodings that

were tested.


6.8     Acceptance of alternative hypothesis

        Based on the clear rejection of the null hypothesis for all combinations of data

type and binary encodings, the following alternative hypothesis is accepted:

        Ha: Applications which currently use XML as a data encoding would see

significantly lower costs in terms of both packet size and CPU time by using an

alternative binary encoding form.
                                                                                          186
6.9     Other observations

        The following observations have been noted in the text above but are included

again here for clarity.


6.9.1   Variable-length fields

        The variable-length fields used in ASn.1 were expected to cause increased CPU

costs simply because there is more complexity involved in determining the encoded form

of a tag or length field and also in parsing those structures. In practice the code to support

the variable-length fields in ASN.1 was simple to implement and the extra CPU costs

were not observed. In fact, as noted above, TLV1 incurred additional cost in parsing 2-

byte fields that were only one byte in the ASN.1 implementation. The cost of having to

manipulate the extra byte exceeded the cost of dealing with the variable-length field

rules. This clearly indicates that simplicity is not always going to provide the best

performance.


6.9.2   Dependence of create and extract times on protocol choice

        Because of the common framework and methodology used to create the code, the

time to create a node tree from the raw data (the Create time in the measurements) and

the time to extract the raw data from the tree (the Extract time in the measurements) were

expected to be independent of the protocol. These two operations have nothing to do with

how the data is represented in its encoded form (The encoding and decoding times were

measured as the Render and Parse times respectively). In practice, the implementation of

the ASN.1, TLV1 and TVM1 binary protocols was different enough from the
                                                                                      187
implementation of the Xnode XML protocol that differences in the Create and Extract

times were observed. The data in Table 20 shows that for the Document type, the ASN.1,

TLV1 and TVM1 protocols took about three times as long to create the data tree as the

Xnode protocol. As discussed above, this is due to the way that data is stored in the tree

nodes. The mechanism for the binary protocols uses a C++ smart-pointer to a re-sizable

binary memory block object. The mechanism in the Xnode protocol uses an STL string

object. The complexity of the binary storage mechanism degrades the Create time

significantly.

        The                     Extract                      times                      in
                                                                                      188
Table 26 show the opposite effect but to a lesser degree. Extracting Document data from

the Xnode tree was about 60% more expensive than extracting it from the binary protocol

trees. The binary storage mechanism allows direct access to the underlying memory

block whereas the Xnode trees are bound by the behavior of the STL string object –

which is apparently more complex in its operations.



6.9.3   The use of C-style structures

        It was noted above that one of the advantages of the TLV1 design was that for

some very simple implementations the encoded form of an entire packet could be created

with a C-style structure. This could be an advantage in microcontroller-based applications

but it should be noted that this optimization cannot be applied to parsing the same

packets. The parser must always expect the packet to be variable in size and in terms of

its structure if the applications using it are not to be bound to one exact variant of the

protocol forever.


6.9.4   Data sizes

        The TLV1 and TVM1 protocols were both designed with 16-bit length fields

limiting the maximum data size to about 64K bytes. This is adequate for some

applications but obviously too small for others. These length fields could be extended to

32-bit values or made variable length as they are in ASN.1. This limit caused a minor

problem during testing as the initial audio sample used was much bigger than 64K

causing a numeric overflow in the computation of the length value and subsequent

corruption of the encoded packet. This was found during analysis of the results which
                                                                                      189
showed the ratio of encoded packet size to natural size for the audio packets as less than

one. Given the excellent performance of ASN.1, which uses variable-length fields for

both the node tags and the node-length, extending the field lengths of TLV1 to 4 bytes

has no value. TLV1 was intended as a simplified version of ASN.1 but clearly ASN.1

performs better in all cases and increasing the field size in TLV1 will only worsen its

performance.


6.10   Summary

       In every case tested, the binary protocol encodings performed much better than

the XML protocol encoding. All of the binary implementations performed better than

XML and they all performed better for every data type tested. The improvement in

individual performance measurements varied from about four times to more than a

thousand times in one specific case. Of the three binary encodings tested, ASN.1 DER

shows clear advantages over the other encodings and clearly offers an alternative to XML

where encoded data size and CPU overhead are important especially when the data

content is partially or exclusively binary.
                                                                                        190



                                      CHAPTER VII

                          AREAS FOR FUTURE RESEARCH


7.1    Alternative TLV Form

       The TLV1 protocol used here employed fixed-length tag and length fields. It was

assumed that fixed-length fields would be faster to generate and parse than variable

length fields (as used in ASN.1). Some of the results indicate that this assumption is false

and that variable length fields which yield shorted encoded packets may be a net gain in

overall performance since the code required to generate and parse these fields is not much

more complex than that required to generate and parse the fixed-length forms.


7.2    Other binary encodings

       The three binary encodings used here represent only a small portion of those

available. A lot of work is going on in the MPEG committees to find binary extensions or

alternatives to XML for streaming data such as video and for data that needs to be sent as

incremental updates. The TVM1 and TLV1 protocols tested here are very simple and do

not address the MPEG requirements for compression or for the representation of deltas.

More work could be done using the experimental method used here to evaluate the

suitability of protocols for MPEG applications.
                                                                                       191
7.3    Expanded data set types

       Work by the W3C identified a large number of application types which might

benefit from a binary version of XML. The work done here limited experimentation to

just seven data structures because of the amount of time required to encode, verify and

test each one. The approach used here to measure protocol performance could be applied

to any candidate protocols considered as a replacement for XML across the entire set of

use cases identified by the W3C work.


7.4    Improved node implementation

       Analysis of the performance of the ASN.1, TLV1 and TVM1 encodings showed

that for simple string data there was quite a lot more time spent constructing the node

trees than for the Xnode (XML) type. This is likely due to the method of storing the data

in the nodes. The Xnode object uses an STL std::string for storage since all data in XML

nodes are strings. The binary protocols used a C++ smart-pointer template class to

address a re-sizable binary store object. This is a very general design but is probably

more costly than simple calls to malloc and free (or new and delete in C++). There is no

need in the implementation for the buffers to be resizable so a simple allocation just once

is probably all that is needed and this is close to what happens when an STL string is

allocated. This change could further improve the performance of all three binary

encodings with respect to the Xnode encoding and perhaps make them more competitive

with Xerces.
                                     REFERENCES



[1]    W3C, "XML Binary Characterization Working Group," R. Berjon, Ed.: W3C,
       2005.

[2]    ISO/IEC, "X.690 - ASN.1 encoding rules: Specification of Basic Encoding Rules
       (BER), Canonical Encoding Rules (CER) and Distinguished Encoding Rules
       (DER)," 2002.

[3]    OMG, "Common Object Request Broker Architecture: Core Specification
       v3.0.3," Object Management Group, 2004.

[4]    L. Consortium, "LIN Specification Package Revision 2.1," LIN Consortium,
       2006.

[5]    W3C, "Namespaces in XML 1.0 (Second edition)," 2006.

[6]    IETF, "UTF-8, a transformation               format   of   ISO    10646."     vol.
       http://www.ietf.org/rfc/rfc3629.txt, 2003.

[7]    OMG, "Common Object Request Broker Architecture: Core Specification,"
       Object Management Group, 2004.

[8]    IEEE, "IEEE Standard for Binary Floating-Point Arithmetic (IEEE Std 754-
       1985)," IEEE, 1985.

[9]    N. Abu-Ghazaleh, M. J. Lewis, and M. Govindaraju, "Differential serialization for
       optimized SOAP performance," in High performance Distributed Computing,
       2004. Proceedings. 13th IEEE International Symposium on, 2004, pp. 55-64.

[10]   M. G. Kostoulas, M. Matsa, N. Mendelsohn, E. Perkins, A. Heifets, and M.
       Mercaldi, "XML Screamer: An Integrated Approach to High Performance XML
       Parsing, Validation and Deserialization," in Proceedings of the 15th international
       conference on World Wide Web WWW '06, 2006.


                                           192
[11]   R. Lawrence, "The space efficiency of XML," in Information and Software
       Technology 46 (2004) 753–759: Elsevier, 2004, pp. 753 - 759.

[12]   M. Matsa, E. Perkins, A. Heifets, M. G. Kostoulas, D. Silva, N. Mendelsohn, and
       M. Leger, "A High Performance Interpretive Approach to Schema Directed
       Parsing," in WWW 2007 / Track: XML and Web Data, 2007.

[13]   N. Sundaresan and R. Moussa, "Algorithms and programming models for
       efficient representation of XML for Internet applications," in Proceedings of the
       10th international conference on World Wide Web Hong Kong, Hong Kong:
       ACM Press, 2001.

[14]   T. Toshiro, M. Hisashi, S. Toyotaro, and T. Michiaki, "An adaptive, fast, and safe
       XML parser based on byte sequences memorization," in Proceedings of the 14th
       international conference on World Wide Web Chiba, Japan: ACM Press, 2005.

[15]   R. Schmelzer, "The Pros and Cons of XML," Zapthink, 2001.

[16]   B. D. Trivunović, M. V. Popović, I. V. Bašičević, and I. S. Velikić, "A Design
       and Implementation of ASN.1 Parser for H.323 Protocol Stack," in TELSIKS
       2007
Serbia: IEEE, 2007.

[17]   O. Dubuisson, ASN.1 Communication Between Heterogeneous Systems: Morgan
       Kaufmann, 2001.

[18]   J. Larmouth, ASN.1 Complete: Morgan Kaufmann, 2000.

[19]   CCITT, "T.81 JPEG Standard," ITU, 1992.

[20]   D. Mundy and D. W. Chadwick, "An XML alternative for performance and
       security: ASN.1," IT Professional, vol. 6, pp. 30-36, 2004.

[21]   M. Cokus and S. Pericas-Geertsen, "XML Binary Characterization Use Cases,"
       W3C 2005.



                                          193
[22]   T. Imamura and H. Maruyama, "Mapping between ASN.1 and XML," Symposium
       on Applications and the Internet (SAINT'01) pp. 57 - 64, 2001.

[23]   M. R. B. Hardy, D. F. Brailsford, and P. L. Thomas, "Creating Structured PDF
       Files
Using XML Templates," in DocEng'04 Milwaukee, Wisconsin, USA.: ACM, 2004.

[24]   M. Cokus and S. Pericas-Geertsen, "XML Binary Characterization Use Cases,"
       M. C. Mike Cokus and S. M. Santiago Pericas-Geertsen, Eds.: W3C, 2004.

[25]   ISO/IEC and ITU, "X.680 - Abstract Syntax Notation One (ASN.1) Specification
       of Basic Notation," in ITU-T Rec. X.680 (2002) | ISO/IEC 8824-1:2002, 2002.

[26]   M. Gudgin, M. Hadley, N. Mendelsohn, J. Moreau, and H. Nielsen, "SOAP
       Version 1.2 Part 1: Messaging Framework," M. H. Martin Gudgin, Noah
       Mendelsohn, Jean-Jacques Moreau, Henrik Frystyk Nielsen, , Ed.: W3C, 2003.

[27]   H. Nyquist, "Certain topics in telegraph transmission theory," Proceedings of the
       IEEE, vol. 90, pp. 280 - 305, Feb. 2002 1928.

[28]   N. Beaulieu, "Introduction to “Certain Topics in Telegraph Transmission
       Theory”," PROCEEDINGS OF THE IEEE, vol. 90, pp. 276 - 279, 2002.

[29]   J. Boughtwood, "Telegraph Terminal AN/FGC-29 Circuit Design Aspects,"
       Communications Systems, Transactions of the IRE Professional Group on, vol. 2,
       pp. 152 - 157, 1954.

[30]   R. M. M. Oberman, "Start-Stop Data Transmission Without the Stop Element,"
       Communications, IEEE Transactions on, vol. 9, pp. 252 - 258, Sep. 1961 1961.

[31]   C. Kagan, "Computer Coding, Multiplexing, and Distribution of Telegraph
       Signals," Communications, IEEE Transactions on [legacy, pre - 1988], vol. 14,
       pp. 31 - 39, 1966.

[32]   J. B. Postel, "Simple Mail Transfer Protocol," IETF, 1981.



                                          194
[33]   T. Usdin and T. Graham, "XML: not a silver bullet, but a great pipe wrench,"
       StandardView, vol. 6, pp. 125-132, 1998.

[34]   W. HANSLO and K. MACGREGOR, "The Efficiency of XML as an
       Intermediate Data Representation for Wireless Middleware Communication," in
       SAICSIT 2004, 2004, pp. 279 - 283.

[35]   R. D. Sutter, S. Lerouge, D. D. Schrijver, and R. V. d. Walle, "Enhancing RSS
       Feeds: Eliminating Overhead through Binary Encoding," in Proceedings of the
       Third International Conference on Information Technology and Applications
       (ICITA’05), 2005.

[36]   U. Niedermeier, J. Heuer, A. Hutter, W. Stechele, and A. Kaup, "An MPEG-7
       Tool for Compression and Streamin of XML Data," in IEEE International
       Conference on Multimedia and Expo (ICME), Lausanne, Switzerland, 2002, pp.
       521-524.

[37]   D. Crockford, "JavaScript Object Notation (JSON) (Internet Draft)," IETF, 2006.

[38]   R. J. Bayardo, D. Gruhl, V. Josifovski, and J. Myllymaki, "An Evaluation of
       Binary XML Encoding Optimizations for Fast Stream Based XML Processing,"
       in WWW2004, New York, 2004, pp. 345-354.

[39]   J. Derrick, "Cost Effective XML Processing in the Datacenter," deepX Ltd., 2005.

[40]   D. Davis and M. Parashar, "Latency performance of SOAP implementations," in
       Proceedings of the 2nd IEEE/ACM International Symposium on Cluster
       Computing and the Grid (CCGRID'02), 2002, pp. 377-382.

[41]   W. Wang, H. Jiang, H. Wang, X. Lin, H. Lu, and J. Li, "Efficient Processing of
       XML Path Queries Using the
Disk-based F&B Index," in Proceedings of the 31st international conference on Very
       large data bases VLDB '05 2005.

[42]   D. D. Clark and D. L. Tennenhouse, "Architectural considerations for a new
       generation of protocols," in Proceedings of the ACM symposium on


                                         195
       Communications architectures \& protocols Philadelphia, Pennsylvania, United
       States: ACM Press, 1990.

[43]   E. P. Markatos, "Speeding up TCP/IP: faster processors are not enough," in
       Performance, Computing, and Communications Conference, 2002. 21st IEEE
       International, 2002, pp. 341 - 345.

[44]   M. Adler and B. M. Maggs, "Protocols for Asymmetric Communication
       Channels," Journal of Computer and System Sciences, vol. 63, pp. 573-596, 2001.

[45]   S. Mishra and N. Shi, "Improving the Performance of Distributed CORBA
       Applications," in Proceedings of the International Parallel and Distributed
       Processing Symposium (IPDPS'02), 2002.

[46]   R. W. WATSON and S. A. MAMRAK, "Gaining Efficiency in Transport
       Services by Appropriate Design and Implementation Choices," ACM
       Transactions on Computer Systems (TOCS), vol. 5, 1987.

[47]   B. H. Tay and A. L. Ananda, "A Survey of Remote Procedure Calls," ACM
       SIGOPS Operating Systems Review vol. 24, pp. 68 - 79, 1990.

[48]   M. Allman, "An evaluation of XML-RPC," SIGMETRICS Perform. Eval. Rev.,
       vol. 30, pp. 2-11, 2003.

[49]   S. Dissanaike, P. Wijkman, and M. Wijkman, "Utilizing XML-RPC or SOAP on
       an embedded system," in Proceedings of the 24th International Conference on
       Distributed Computing Systems Workshops (ICDCSW’04), 2004, pp. 438-440.

[50]   N. Koudas, M. Rabinovich, D. Srivastava, and T. Yu, "Routing XML queries," in
       Proceedings of the 20th International Conference on Data Engineering
       (ICDE’04), 2004, p. 844.

[51]   M. Govindaraju, A. Slominski, K. Chiu, P. Liu, R. van Engelen, and M. J. Lewis,
       "Toward characterizing the performance of SOAP toolkits," in Proceedings of the
       Fifth IEEE/ACM International Workshop on Grid Computing (GRID’04), 2004,
       pp. 365-372.


                                         196
[52]   G. T. Almes, "The Impact of Language and System on Remote Procedure Call
       Design," IEEE Computer Society Press, 1986.

[53]   G. Eisenhauer and L. K. Daley, "Fast heterogeneous binary data interchange,"
       Heterogeneous Computing Workshop, 2000. (HCW 2000) Proceedings. 9th, 2000.

[54]   F. Bustamante, G. Eisenhauer, K. Schwan, and P. Widener, "Efficient wire
       formats for high performance computing," in Proceedings of the 2000 ACM/IEEE
       conference on Supercomputing (CDROM) Dallas, Texas, United States: IEEE
       Computer Society, 2000.

[55]   T. He, J. A. Stankovic, C. Lu, and T. Abdelzaher, "SPEED: a stateless protocol
       for real-time communication in sensor networks," Distributed Computing
       Systems, 2003. Proceedings. 23rd International Conference on, pp. 46 - 55, 19-22
       May 2003 2003.

[56]   J. Postel, "Internet Protocol," IETF, 1980.

[57]   J. Postel, "TRANSMISSION CONTROL PROTOCOL," J. Postel, Ed.: IETF,
       1981.

[58]   J. Case, M. Fedor, M. Schoffstall, and J. Davin, "A Simple Network Management
       Protocol (SNMP)," IETF, 1990.

[59]   ISO/IEC, "Information technology - Open Systems Interconnection - Basic
       Reference Model: The Basic Model," ISO/IEC, 1994.

[60]   B. S. Kaliski, "A Layman's Guide to a Subset of ASN.1, BER, and DER," RSA
       Laboratories, 1993.

[61]   A. Puder and K. Romer, Mico. An Open Source CORBA Implementation: Morgan
       Kaufmann, 2000.

[62]   D. E. Taylor, "Survey and Taxonomy of Packet Classification Techniques," ACM
       Computing Surveys, vol. 37, pp. 238-275, September 2005 2005.



                                           197
[63]   R. W. Uhlhorn, "The fiber-optic high-speed data bus for a new generation of
       military aircraft," LCS, IEEE [see also IEEE LTS], vol. 2, pp. 36 - 45, Feb. 1991
       1991.

[64]   A. Gokhale and D. C. Schmidt, "Measuring the performance of communication
       middleware on high-speed networks," in ACM SIGCOMM Computer
       Communication Review , Conference proceedings on Applications, technologies,
       architectures, and protocols for computer communications, 1996.

[65]   L. C. Stanberry, M. L. Branstetter, and D. M. Nessett, "Vectorized presentation-
       level services for scientific distributed applications," ACM SIGCOMM Computer
       Communication Review, vol. 24, 1994.

[66]   K. Ito, L. E. Lucke, and K. K. Parhi, "Module selection and data format
       conversion for cost-optimal DSP synthesis," in Proceedings of the 1994
       IEEE/ACM international conference on Computer-aided design ICCAD '94 1994.

[67]   A. Gavrilovska and K. Schwan, "Programmable/extensible networks: Addressing
       data compatibility on programmable network platforms," in Proceedings of the
       2005 ACM symposium on Architecture for networking and communications
       systems ANCS '05, 2005.

[68]   M. Haines, P. Mehrotra, and J. V. Rosendale, "SmartFiles: an OO approach to
       data file interoperability," in ACM SIGPLAN Notices , Proceedings of the tenth
       annual conference on Object-oriented programming systems, languages, and
       applications OOPSLA '95, 1995.

[69]   B. Kuehne, T. True, A. Commike, and D. Shreiner, "Performance OpenGL:
       platform independent techniques: Performance OpenGL: platform independent
       techniques," in ACM SIGGRAPH 2005 Courses SIGGRAPH '05, 2005.

[70]   D. A. Carlson, "TIME-SPACE TRADEOFFS FOR TREE SEARCH AND
       TRAVERSAL," in Proceedings of 1986 ACM Fall joint computer conference
       1986, pp. 585-594.

[71]   R. Housley, W. Polk, W. Ford, and D. Solo, "Internet X.509 Public Key
       Infrastructure Certificate and Certificate Revocation List (CRL) Profile," IETF,
       2002.
                                           198
[72]   C. Huitema and A. Doghri, "Defining faster transfer syntaxes for the OSI
       presentation protocol," ACM SIGCOMM Computer Communication Review, vol.
       19, 1989.

[73]   H.-A. P. Lin, "Estimation of the optimal performance of ASN.1/BER transfer
       syntax," ACM SIGCOMM Computer Communication Review, vol. 23, 1993.

[74]   P. Hoschka, "Compact and efficient presentation conversion code," IEEE/ACM
       Transactions on Networking (TON), vol. 6, 1998.

[75]   K. K. Chang and D. Gay, "Language support for interoperable messaging in
       sensor networks," in Proceedings of the 2005 workshop on Software and
       compilers for embedded systems SCOPES '05, 2005.

[76]   G. Leen and D. Heffernan, "Expanding automotive Electronic Systems," in IEEE
       Computer: IEEE, 2002.

[77]   L. Consortium, "Local Interconnect Network Technical Overview." vol. 2007,
       2007.

[78]   R. Bosch, "CAN Specification Version 2.0," Robert Bosch GmbH, 1991.

[79]   FlexRay, "FlexRay protocol Specification," FlexRay, 2004.

[80]   ECSS, "SpaceWire - Links, nodes, routers and networks," ECSS Secretariat,
       2003.

[81]   G. P. Rakow, C. Dailey, L. Haynes, and M. Pagen, "SpaceWire Physical Level
       Redundancy Mechanism," in 2nd IEEE International Conference on Space
       Mission Challenges for Information Technology (SMC-IT'06): IEEE Computer
       Society, 2006.

[82]   S. Saponara, E. Petri, M. Tonarelli, I. D. Corona, and L. Fanucci, "FPGA-based
       Networking Systems for High Data-rate and
Reliable In-vehicle Communications," in EDAA, 2007.



                                         199
APPENDICES




   200
                                      APPENDIX A


                                   TLV1 ENCODING



A.1Introduction

       TLV1 is a Tag-Length-Variable protocol which uses fixed length fields for the tag

and length components. It also uses 32-bit aligned structures for the Variable field. It is

designed to be efficient on 32-bit architectures by avoiding data items that cross the 32-

bit memory boundaries. The overall concept is similar to ASN.1 DER (Distinguished

Encoding Rules). The main variations are the use of fixed length fields instead of the

variable length fields specified in ASN.1 DER and the 32-bit aligned fields for the data

areas instead of the variable length field used in DER. Using fixed length fields tends to

increase the size of the packet in some circumstances but enables packets to be defined

using C-style structures of a fixed size. This makes it easy to implement in firmware

because packets can be created by filling out a fixed structure rather than by being

dynamically created. This helps to overcome one of the limitations of TLV designs in

general, that the length values of each node must be computed before the packet can be

sent. By using fixed length fields, all node sizes can be pre-determined so that they are

already known at runtime. The idea for this approach came from working with an

existing firmware system which used an interface based on C-style structures. For each


                                           201
revision of the firmware, new functions were added and the existing structures were often

changed. This meant that external devices talking to the firmware needed to be

conversant with multiple versions of the interface so that the correct structure definitions

were used for a given version of the firmware. The primary purpose of the firmware

interface was to send data rather than receive it so a protocol based on fixed structure

definitions meant simpler firmware.


A.2Protocol packet structure

        TLV1 uses nested node tuples of a tag value (T), a length value (L) and a data or

value field (V). The value part of a node may contain either data or zero or more child

nodes. If it contains data, it is called a leaf node. If it contains zero or more child nodes, it

is called a container node. Note that both leaf nodes and container nodes may have a zero

size variable field.

    Tag       Length                                Variable




    Tag       Length           Variable             Tag        Length           Variable




    Tag       Length           Variable             Tag        Length           Variable


                              Figure 90: TLV1 packet structure


                                              202
         Figure 90shows an example of a TLV1 packet. The outer node shown in the upper

line contains two child nodes show in the middle line. The fist child node contains two

child nodes shown in the lower line and the second child node is a leaf node containing a

data item. The child nodes shown on the lower line are both leaf nodes containing data

items.

         For TLV1, the size of the tag and length fields was chosen to be 16-bits. This

gives a limit to the overall packet size of about 64K bytes which is adequate for small

applications. Larger applications might use 32-bit values for the tag and length fields to

provide for larger packets while retaining the 32-bit alignment. The smallest data item is

a 32-bit value. Larger items are integral multiples of 32-bits. This means that some types

such as strings will include padding. This is described in more detail below.


A.3Byte order

         All fields of the TLV1 protocol which represent elements larger than one byte are

sent using the big-endian convention. That is, the most significant byte of a multi-byte

value is sent first. So the 16-bit value 1024 (0x0400) is sent as the byte 0x04 followed by

the byte 0x00.


A.4Structure of the tag field

         The tag fields primarily provide a way to uniquely identify container nodes and

individual leaf nodes. A secondary purpose is to indicate if the node is a container node


                                            203
or a leaf node. If the set of tag values in a given protocol using TLV1 are unique then

knowledge of the container/leaf status of an individual node is not strictly required but

being able to identify container vs. leaf nodes in a debugger that does not understand all

elements of a protocol can be very helpful. Furthermore, knowing the container/leaf

status of a node allows the recipient of a packet to parse the node tree without knowledge

of the purpose of each node. This is fundamental to being able to extend a protocol by

inserting new data into an existing node tree without breaking a recipient which is

unaware of the change.

        Tag values are 16 bits. The most significant bit (b15) is used to indicate the

container/leaf status of a node. For container nodes, b15 is set to 1. For leaf nodes it is set

to 0.



                 1                    Container tag value

                          Figure 91: Container node tag structure


        Container nodes have 15 bits to define a unique value (32,768 values). Tag values

are only required to be unique within the scope of a single node in the packet. As an

example, a „user‟ record might have an „address‟ structure and within that „address‟

structure would be the ZIP code. The tag for the ZIP code field only needs to be unique

within the address structure.



                                             204
        Leaf node tags are constructed with three fields. Bit 15 is set to zero to indicate

that the node is a leaf node. The remaining 15 bits are divided into a 5-bit type field and a

10-bit tag value field.


                 0        type                          ID

                             Figure 92: Leaf node tag structure

        The purpose of the type field is to define a few fixed types so that a debugger can

print out their values in a node tree. All parsers must understand the pre-defined types.

For other types a debugger may emit the value as a block of binary data if it cannot

determine the type from the context and the ID field.




                                            205
                                 Table 42: The fixed types

Type      Name                 Description
number
0         byte                 An unsigned 8-bit value (padded)
1         char                 A signed 8-bit character value (like a C char)
2         uint16               Unsigned 16-bit integer
3         int16                Signed 16-bit integer
4         uint32               Unsigned 32-bit integer
5         int32                Signed 32-bit integer
6         uint64               Unsigned 64-bit integer
7         int64                Signed 64-bit integer
8         uint128              Unsigned 128-bit integer
9         int128               Signed 128-bit integer
10        enum32               32-bit enumerated type
11        enum64               64-bit enumerated type
12        flags32              32-bit bit vector
13        flags64              64-bit bit vector
14        string               UTF-8 string (padded)
15        binary               Binary data (padded)
16-31     Application types    These can be used by applications. Parsers treat these types
                               As binary if not otherwise recognized




       Items shown as (padded) are stored in a 32-bit aligned field. This means that some

bytes are not used and are set to zero. For padded string types, the padding is appended to

the string as one or more zero bytes. For binary data, padding is appended as one or more

zero bytes at the end of the block. All multi-byte items are stored in big-endian format.

       Applications are not required to use the fixed types. They are a convenience only.

An application may use any of the application-specific type values (16 – 31) to define a



                                             206
type. The ID part of the tag is used to specify a particular item within the context of the

parent node.


A.5Tag scope

       Tag values for container and leaf nodes are only required to be unique within the

scope of the parent node. As an example, consider an address node which contains items

for street address, city, state and zip code. All of these may use the pre-defined string

type. The ID field values might be 1, 2, 3 and 4 or any other values as defined by the

application. An instance of the application parsing the address node would look for the

street address child node, the zip code node and so on. So each child node only needs to

be uniquely identifiable within the address structure.


A.6Implementation with C-style structures

       The TLV1 protocol is designed so that applications which need to send data

efficiently may define a structure that conforms to the overall protocol but which contains

fixed length fields. For most of the pre-defined types the length of the item is fixed. For

strings however, additional padding of zero value bytes may be appended to a string to

make it up to the length of the pre-allocated field.

       Using this approach we can pre-define a structure for the entire packet and

compute all the tag and length values at compilation time so that when the application




                                             207
needs to send a data packet it can copy a template structure containing the pre-computed

length and tag values and simply fill in the data areas.


A.7Comparison with ASN.1 DER

           TLV1 is simpler than ASN.1 DER in that it does not allow for multiple

representations of a single type. For example, the Distinguished Encoding Rules allow

floating point values to be encoded in a binary form or a textual (string) form. Allowing

multiple encoding formats adds a burden on the decoder. The decoder must be able to

parse all forms even if only one is used. This means additional code and additional

testing.


A.8Comparison with XML

           TLV1 is not as flexible as XML in several ways. It does not allow for node

attributes as well as child nodes or content. It can be used to form similar structures by

making each node into a compound node that has one branch for attributes and one for

child nodes or content.

           TLV1 is very efficient at transferring binary data whereas XML must encode the

data into a textual form. The encoding and decoding takes time and the resulting packet

can be up to 50% bigger than the source data for a typical base64 encoding.

           TLV1 uses tags which have local uniqueness only. This prohibits parts of packets

form different applications being mixed into a single packet. XML uses namespaces to


                                             208
allow this behavior. It is possible to deal with this problem by wrapping TLV1 nodes into

a parent node that also contains application version information. This is a variation on the

idea of having node attributes. In this case the „attributes‟ would be an identifier

describing the application that encoded the node tree.




                                            209
                                        APPENDIX B


                                     TVM1 ENCODING



B.1 Introduction

        TVM1 is a protocol which uses variable-length blocks delimited by markers. It is

designed to avoid the problem with the TLV1 protocol which requires the length of each

node to be computed before the packet can be sent. Instead of using a length value, TVM

packets are sent as a hierarchical set of nodes where each node consists of a tag, the

variable (data) field and a marker. Nodes may be either container nodes or leaf nodes.

Container nodes have zero or more child nodes. Leaf nodes contain some element of data

such as an integer value or a string.

        Some simple leaf node types are defined as fixed length. This slightly increases

the parser complexity since it must know which leaf node types are fixed length and

which are variable but it improves packet size by omitting the markers from nodes that do

not require them.


B.2 Protocol packet structure

        TVM1 uses nested tuples of a tag value (T), a value field (V) and a marker (M).

The value part of a node may contain either data or zero or more child nodes. If it

contains data, it is called a leaf node. If it contains zero or more child nodes, it is called a

                                              210
container node. Note that both leaf nodes and container nodes may have a zero size

variable field.



       Tag                           Variable                          Marker




       Tag          Variable        Marker      Tag         Variable        Marker




       Tag          Variable        Marker      Tag         Variable        Marker



                           Figure 93: The TVM1 general form




        Figure 93 shows an example of a TVM1 packet. The outer node shown in the

upper line contains two child nodes show in the middle line. The fist child node contains

two child nodes shown in the lower line and the second child node is a leaf node

containing a data item. The child nodes shown on the lower line are both leaf nodes

containing data items.

        For TVM1, the size of the tag field was chosen to be 16-bits. This gives the same

range of tag options as TLV1. Since there are no length fields, the size of the tag field

does not affect the maximum packet size which is a limitation of TLV1.

                                          211
B.3 Packets containing fixed-length nodes

        TVM1 contains 14 fixed-length types varying in length from one to 16 bytes.

Applications are not required to use these types but because nodes of these types do not

include a marker, they help to reduce packet size if there are large numbers of these

primitive types in the packet.

        As an example consider a node that contains a variable length string followed by

two 16-bit numeric fields.



    Tag                                                               Marker




    T            String          M    T        int16       M      T          Int16   M


                   Figure 94: A packet node containing fixed-length fields




B.4 Byte order

        All fields of the TVM1 protocol which represent elements larger than one byte are

sent using the big-endian convention. That is, the most significant byte of a multi-byte

value is sent first. So the 16-bit value 1024 (0x0400) is sent as the byte 0x04 followed by

the byte 0x00.


B.5 Structure of the tag field

                                            212
        The tag fields primarily provide a way to uniquely identify container nodes and

individual leaf nodes. A secondary purpose is to indicate if the node is a container node

or a leaf node. If the set of tag values in a given protocol using TVM1 are unique then

knowledge of the container/leaf status of an individual node is not strictly required but

being able to identify container vs. leaf nodes in a debugger that does not understand all

elements of a protocol can be very helpful. Furthermore, knowing the container/leaf

status of a node allows the recipient of a packet to parse the node tree without knowledge

of the purpose of each node. This is fundamental to being able to extend a protocol by

inserting new data into an existing node tree without breaking a recipient which is

unaware of the change.

        Tag values are 16 bits. The most significant bit (b15) is used to indicate the

container/leaf status of a node. For container nodes, b15 is set to 1. For leaf nodes it is set

to 0.



                 1                    Container tag value

                          Figure 95: Container node tag structure


        Container nodes have 15 bits to define a unique value (32,768 values). Tag values

are only required to be unique within the scope of a single node in the packet. As an

example, a „user‟ record might have an „address‟ structure and within that „address‟



                                             213
structure would be the ZIP code. The tag for the ZIP code field only needs to be unique

within the address structure.

        Leaf node tags are constructed with three fields. Bit 15 is set to zero to indicate

that the node is a leaf node. The remaining 15 bits are divided into a 5-bit type field and a

10-bit tag value field.


                 0        type                          ID

                             Figure 96: Leaf node tag structure


        The purpose of the type field is to define a few fixed types so that a debugger can

print out their values in a node tree. All parsers must understand the pre-defined types.

For other types a debugger may emit the value as a block of binary data if it cannot

determine the type from the context and the ID field.




                                            214
                                Table 43: The fixed types

Type      Name               Description                                          Length
number                                                                            (Bytes)
0         byte               An unsigned 8-bit value                              1
1         char               A signed 8-bit character value (like a C char)       1
2         uint16             Unsigned 16-bit integer                              2
3         int16              Signed 16-bit integer                                2
4         uint32             Unsigned 32-bit integer                              4
5         int32              Signed 32-bit integer                                4
6         uint64             Unsigned 64-bit integer                              8
7         int64              Signed 64-bit integer                                8
8         uint128            Unsigned 128-bit integer                             16
9         int128             Signed 128-bit integer                               16
10        enum32             32-bit enumerated type                               4
11        enum64             64-bit enumerated type                               8
12        flags32            32-bit bit vector                                    4
13        flags64            64-bit bit vector                                    8
14        string             UTF-8 string                                         Var.
15        binary             Binary data                                          Var.
16-30     Application        These can be used by applications. Parsers treat     Var.
          types              these types
                             As binary if not otherwise recognized
31        Prohibited         (Conflicts with marker byte)                         n/a

       Types 0 through 13 are fixed length fields. All other types are considered variable

length and require a marker to terminate them as discussed below.


B.6 Structure of the marker field

       The marker is a two-byte value inserted into the data stream. The marker field

consists of a sequence of two bytes such that the first byte is the marker value and the

second byte is not the marker value. The marker byte value is chosen so as not to be a

common data value and not to conflict with the most significant byte of a tag field.

                                           215
Analysis of common files such as text, audio and video revealed certain byte values as

common. Common values include 0x00, 0xFF, the letters „a‟ and „e‟ and 0x7F (127). A

value of 0x7C (124) was chosen for the marker. This seems no more common that other

values in randomly sampled files. The byte following the marker can be any value other

that the marker byte because data blocks will encode the marker value as a sequence of

two marker bytes and this must not be able to be confused with the end of a section. In

practice, the byte following the marker is chosen to be 0x00.

       The marker value 0x7F if used as the most significant byte of a tag field would

yield a tag value of 0x7Cxx. This would indicate a leaf node (bit15 is zero) and a type

value of all ones which is 31. This value is prohibited as a type field.

       The marker sequence is written at the end of every variable length value field and

at the end of a section of child nodes. So, for example, if the field has a tag that indicates

it is a 32-bit integer value, we know the length is four bytes and we do not need a marker

to find the next tag. If the field is binary data, then we have no idea how long it is and the

marker is required to indicate the end of the block.


B.7 Structure of the value field

       Value fields are structured in two ways. Fixed length items are written as a fixed

length byte sequence in big-endian order. Variable length items are encoded so that any

occurrence of the marker byte value in the data is written as two bytes, each containing

the marker value. The parser will decode variable length blocks up to a marker byte. If

                                             216
the byte value following the marker is also the marker byte value then the parser inserts a

single byte into the output stream of the marker value and continues. If the byte following

the marker is not the marker byte value then the parser has found the end of the variable

length block. Note that this prohibits the marker value being the most significant byte of a

tag field as discussed above.

       If we chose to use a leaf node with a type field of 31, the marker could have been

any value in the range 0x7C to 0x7F. 0x7C was chosen since 0x7F occurs commonly in

some uncompressed wave files.


B.8 Comparison with TLV1

       TVM1 is more compact than TLV1 because it does not use fixed alignment of the

fields and does not require the length fields at all. The 16-bit size of the tag has no

bearing on packet size so there is no packet size limit as there is with TLV1. TVM1 does

require the encoding of variable length blocks such as strings and binary data. The 0x7C

marker byte value does not conflict with any ASCII character but may cause some UTF-8

strings to be slightly longer if the value occurs in the string. Binary will be encoded as a

slightly longer field than the natural length of the data because any marker byte values

must be replaced by pairs of bytes with that value. For random data, the encoded form

would be on average 257/256 times as large (~ 0.4%).




                                            217
218
                                      APPENDIX C


                          THE EXPERIMENTAL DATA SETS


        This appendix describes the structure of each of the data sets used in the

experiments. Table 44 lists the identifiers of the structures as used in the experimental

code and a brief description of each structure. Each structure is described in detail below.

                               Table 44: The data structures

Structure Identifier        Description

Document                    A document containing some meta-data such as title and

                            author and several paragraphs of text.

Email                       An email containing to, from and subject information with a

                            few paragraphs of text and a small binary attachment.

3D_Mesh                     A set of vertex and face descriptors for a 3D object.

Audio                       A small audio clip which also contains a small amount of

                            meta-data such as the artist's name.

WebQuery                    A structure that might be sent to a web search engine to find

                            documents matching author and tile information.

WebQResp                    A response from a web search query that contains descriptions

                            of several documents that matched the search criteria.



                                            219
Spreadsheet                 A spreadsheet containing several rows and columns of figures

                            as well as row and column captions.



       Each structure is described in terms of its hierarchy and data types. A section is

also included that describes how the data was generated at runtime for the experiments.


C.1Structure description format


       All the data structures are show here in a hierarchical representation such as might

be used by a typical application. Each structure was serialized according to the rules for

each protocol. This generally resulted in an encoded structure similar to the hierarchy

shown here but note that a protocol may encode some hierarchical representation in

whatever manner it chooses so although parent-child relationships shown here were

typically encoded as similar sets of parent-child nodes, that need not be the case in

practice. For example, the author's name for a document could be encoded as a child node

of the document node. If we were using XML, it could also be encoded as an attribute of

the document node rather than in a child node per se.

       The data structures are described using indentation to represent parent-child

relationships in the hierarchy. Figure 97 shows a data structure with two nodes that are

peers at the root level. Each root-level node has a number of child nodes and on of the

child nodes has a child of its own.


                                           220
Root_A
         Child_1 (text)
         Child_2 (text)
Root_B
         Child_3
               Sub-child_4 (integer)




                               Figure 97: A node hierarchy

         For each leaf node, the type of the data is also shown. Note that this is the raw

data type and is not necessarily the type when the structure is encoded. For example, an

integer or floating point value may be encoded as a string in XML but as a binary image

in ASN.1.


C.2The document structure


         Figure 98shows a representation of document with some introductory meta-data

and a few paragraphs such as might be used by a word processing program. The intent is

to provide a structure containing a large amount of textual data but also with some

additional meta-data that the application might use for formatting and so on.




                                            221
Document
      Properties
            Author (text)
            Date last edited (text)
            Title (text)
            File location (text)
      Section
            Font
                  Face name (text)
                  Point size (integer)
            Text (text)
      Section
            Font
                  Face name (text)
                  Point size (integer)
            Text (text)
      Section
            Font
                  Face name (text)
                  Point size (integer)
            Text (text)



                          Figure 98: The document data structure

       The structure used here is considerably simpler than that used in a real word

processor. For the purpose of testing a protocol encoding it provides a large amount of

textual data and enough meta-data that the protocol must use some additional space to

describe the hierarchy of the sections.


C.2.1 Data generation
      The author, edit date, file location and title were generated as short random

strings. The number of sections was randomly chosen between 10 and 50. For each

section the font information was generated randomly and a random paragraph of text was



                                           222
also randomly generated. Each paragraph consisted of five to 10 sentences. Each sentence

consisted of five to ten words where each word was 2 to 7 characters long.


C.2.2 The email structure
      Figure 99 shows a structure that contains the essential parts of an email with a

single, small binary attachment. The attachment could be an image, a digital certificate in

X.509 format or perhaps a segment of audio.


Email
        From (text)
        To list
              To (text)
              To (text)
        CC list
              CC (text)
              CC (text)
        Subject (text)
        Text content (text)
        Attachment
              Display title (text)
              Binary data (binary)


                           Figure 99: The email data structure

        The structure is simplified from most modern emails but contains enough

hierarchy to force the protocol encoding to generate meta-data to describe that hierarchy.

The binary attachment serves to prevent all the data from being text. Although emails are

traditionally textual in nature, most emails now contain attachments such as images or

digital certificates.


C.2.3 Data generation

                                           223
       Each email contained 2 to 7 entries in the to list and the cc list. Each entry was a

randomly generated email address. The subject was a randomly generated sentence and

the body was five to 10 paragraphs. Each paragraph consisted of five to 10 sentences.

Each sentence consisted of five to ten words where each word was 2 to 7 characters long.



C.3The 3D mesh structure

       Figure 100 describes part of a 3D mesh structure which might be sent from server

to client where it is rendered as a solid object which is part of a game or other simulation.




                                            224
Mesh
       Object ID
       Vertices
             Vertex
                        ID (integer)
                        X (floating-point)
                        Y (floating-point)
                        Z (floating-point)
               Vertex
                        ID (integer)
                        X (floating-point)
                        Y (floating-point)
                        Z (floating-point)
               Vertex
                        ID (integer)
                        X (floating-point)
                        Y (floating-point)
                        Z (floating-point)
               Vertex
                        ID (integer)
                        X (floating-point)
                        Y (floating-point)
                        Z (floating-point)
               Vertex
                        ID (integer)
                        X (floating-point)
                        Y (floating-point)
                        Z (floating-point)
       Faces
               Face
                        Face ID (integer)
                        Vertex ID (integer)
                        Vertex ID (integer)
                        Vertex ID (integer)
               Face
                        Face ID (integer)
                        Vertex ID (integer)
                        Vertex ID (integer)
                        Vertex ID (integer)


                          Figure 100: The 3D mesh data structure

       A single object in a simulation might contain hundreds of vertices and faces and a

scene might contain hundreds or thousands of objects.

                                           225
C.3.1 Data generation

        The number of vertices was from 100 to 300 and the number of faces was from

100 to 300. For each vertex, the ID and coordinate values were generated randomly. The

floating-point values were generated by dividing two randomly generated integer values.

The face data consisted of all randomly generated integers.


C.4The audio structure


        Figure 101 shows the audio data structure. This was primarily a large section of

binary data which consisted of an MP3 encoding of a live recording. The title and other

meta-data was included to force some structure to the data set. Most audio files contain at

least some meta-data that describes the format and possibly the origin, copyright and so

on of the sound track.




Audio
        Title (text)
        Artist (text)
        Duration (integer)
        DataType (text)
        BinaryData (binary)


                           Figure 101: The audio data structure




                                           226
           Note that in this case, the binary data was not generated randomly. A real MP3

encoding was used that was made from a live recording5.


C.4.1 Data generation

           The title, artist and duration information were hard-coded with the actual

information about the recording. The binary data body was inserted directly from the

body of an MP3 file. Note that the MP3 file also contains its own meta-data but this was

ignored and considered part of the binary blob in the test structure.


C.5The web search query structure


           Figure 102 is an example of the kind of data structure that might be required to

describe a web search query for articles by one or more authors for a paper containing

some specific words in the title and some optional words in the title.


Query
           Authors
                 Author (text)
                 Author (text)
           Required title words
                 Word (text)
                 Word (text)
           Optional title words
                 Word (text)
                 Word (text)


                            Figure 102: The web search query data structure



5
    The recording is used with permission of the artist.
                                                       227
C.5.1 Data generation

       Each query contained 2 to 4 authors, 2 to 7 required words and 2 to 7 optional

words. Author names were generated using short randomized word to form a structure

like: "Joe. P. Someone". Each 'word' was 5 to 8 characters long.


C.6The web query response structure

       Figure 103 is an example of a data packet that might be returned in response to a

web search for papers by a given set of authors with particular words in the title. The

response contains a number of descriptions of the papers found. Each description

contains author, title, publication date and so on.




                                             228
Search response
      Paper
            Authors
                  Author (text)
                  Author (text)
            Title (text)
            Publication date (text)
            Publisher (text)
            Abstract (text)
            Location reference (text)
            Citations
                  Citation (text)
                  Citation (text)
      Paper
            Authors
                  Author (text)
                  Author (text)
            Title (text)
            Publication date (text)
            Publisher (text)
            Abstract (text)
            Location reference (text)
            Citations
                  Citation (text)
                  Citation (text)



                   Figure 103: The web query response data structure



C.6.1 Data generation
      Each response structure contained 2 to 8 responses. The author, title and so on

were generated randomly as was the content for abstract paragraph and the citations.


C.7The spreadsheet structure

       Figure 104 represents numeric data such as might be contained in a spreadsheet.

The example shown has two rows with four columns of data in each row. The structure


                                          229
also contains the textual labels for the rows and columns. The cell data was chosen to be

floating-point values.

Spreadsheet
      Column Labels
            Label (text)
            Label (text)
            Label (text)
            Label (text)
      Row
            Row label (text)
            Column data
                  Cell value         (floating-point)
                  Cell value         (floating-point)
                  Cell value         (floating-point)
                  Cell value         (floating-point)
      Row
            Row label (text)
            Column data
                  Cell value         (floating-point)
                  Cell value         (floating-point)
                  Cell value         (floating-point)
                  Cell value         (floating-point)


                         Figure 104: The spreadsheet data structure




C.7.1 Data generation
      Each structure was created as five to 10 columns and 10 to 20 rows. Each row and

column label was generated as a random word. The cell data were all generated as

random floating-point values.




                                            230
231
                                    APPENDIX D


                             SYSTEM INFORMATION


       The following data was obtained by running the Windows msinfo32 tool on the

computer used to run the experiments.



OS Name      Microsoft Windows XP Professional
Version      5.1.2600 Service Pack 2 Build 2600
OS Manufacturer    Microsoft Corporation
System Name SP750A
System Manufacturer      Compaq
System Model       Professional Workstation SP750
System Type X86-based PC
Processor    x86 Family 6 Model 8 Stepping 6 GenuineIntel ~996 Mhz
BIOS Version/Date Compaq 686J4 v3.05, 4/10/2001
SMBIOS Version     2.3
Windows Directory C:\WINDOWS
System Directory C:\WINDOWS\system32
Boot Device \Device\HarddiskVolume1
Locale       United States
Hardware Abstraction Layer      Version         =          "5.1.2600.2180
(xpsp_sp2_rtm.040803-2158)"
User Name    THOMPSON\nigelt
Time Zone    Mountain Standard Time
Total Physical Memory    1,152.50 MB
Available Physical Memory       707.15 MB
Total Virtual Memory     2.00 GB
Available Virtual Memory        1.96 GB
Page File Space    2.13 GB
Page File    D:\pagefile.sys




                            Figure 105: System information




                                         232
233
                                        APPENDIX E


                    EXPERIMENTAL REPEATABILITY TESTING


       Two separate experiments were conducted to assess the repeatability of the test

method. The first of these examined how well the results would be repeated if the whole

experiment was run multiple times. The second looked at how timing measurements are

affected by process switching in the operating system.


E.1 Repeatability between experiments

       Several separate experiment runs were conducted with one data generation cycle

and only one repeat of each protocol/data set combination to see how repeatable the

results were likely to be from run to run.

       In one case, ten runs were done and the data from each run for one protocol/data-

set combination was extracted to look at between-run variations. Data from each run was

imported into Excel and sorted by data set then protocol and the data for the

ASN1/Document combination was extracted (one row of data). These rows were inserted

into a new Excel spreadsheet indexed by run number. Table 45: Run-repeat data shows

the data obtained from the test runs.




                                             234
235
                                                                                      236
                                    Table 45: Run-repeat data

Run   Proto   Struct     Create   Render   Write    Natural     Packet   Parse   Extract    Read    Ratio     Total   RT
1     ASN1    Document   10431    8925     19356    9663        10187    13090   8391       21481   1.05423   40837   43051
2     ASN1    Document   10188    8876     19064    9663        10187    13569   8064       21633   1.05423   40697   42903
3     ASN1    Document   10297    8836     19133    9663        10187    13372   8037       21409   1.05423   40542   42740
4     ASN1    Document   11276    8995     20271    9663        10187    13099   7686       20785   1.05423   41056   43282
5     ASN1    Document   10345    9596     19941    9663        10187    13379   7962       21341   1.05423   41282   43520
6     ASN1    Document   10183    8812     18995    9663        10187    13735   8056       21791   1.05423   40786   42997
7     ASN1    Document   10729    9118     19847    9663        10187    37708   7764       45472   1.05423   65319   68861
8     ASN1    Document   10777    8923     19700    9663        10187    12964   7847       20811   1.05423   40511   42707
9     ASN1    Document   10591    8812     19403    9663        10187    13969   7960       21929   1.05423   41332   43573
10    ASN1    Document   10828    9110     19938    9663        10187    12995   7695       20690   1.05423   40628   42831




                                  236
       Analysis of the data was performed with MINITAB. The following data show the

statistics for the total time measured:

Descriptive Statistics: RT

Variable              N   N*    Mean   SE Mean     StDev     Minimum           Q1     Median      Q3   Maximum
RT                   10    0   45647      2581      8162       42707        42808      43024   43533     68861

                                           Figure 106: Statistical data



The same data is plotted as a histogram in Figure 107.



                                                 Histogram of Total

                     9

                     8

                     7

                     6
         Frequency




                     5

                     4

                     3

                     2

                     1

                     0
                               40000     45000      50000           55000     60000       65000
                                                            Total



                                Figure 107: Distribution of total time for ten runs

If run 7 is considered as an outlier caused (perhaps) by timing data being corrupted by a
process switch and we re-generate the basic statistics and histogram excluding that run
we get the results shown in Figure 108 and Figure 109.


                                                        237
Descriptive Statistics: RT

Variable            N   N*    Mean   SE Mean   StDev    Minimum       Q1   Median      Q3   Maximum
RT                  9    0   43067       107     322      42707    42786    42997   43401     43573

                                 Figure 108: Statistics with outliers excluded



                                            Histogram of Total

              2.0




              1.5
  Frequency




              1.0




              0.5




              0.0
                               40600           40800              41000          41200
                                                        Total


                                 Figure 109: Run data with outlier excluded


               As can be seen from the standard deviation value, run repeatability is very good if

we are willing to exclude outliers. In the case shown here, the one run excluded had a

value of about 3σ greater than the sample mean.


E.2 Timing accuracy


                                                       238
       For this experiment a short piece of code was created that performed a CPU-

bound task many times in a loop. The time taken for the loop to complete was measured.

This as repeated a number of times and the total loop execution time was recorded in

each case. It was expected that the loop would be interrupted periodically by process

switches in the operation system. The purpose of the experiment was to measure how

significant those switches would be and to determine if a large number of run repeats

would be able to mitigate the effect.

       For each of the experiment runs the number of cycles of the tight loop was

adjusted to give some required execution time (say 2 ms). The timing was repeated

enough times to give sufficient samples without the experiment taking too long to run

(less than a minute). Data for the loop times was written to a log file which was imported

into MINITAB for analysis. In each case, basic statistics and a histogram plot were

generated along with a time series and a normality prediction plot.


E.2.1 Run one
      Cycle time: ~20 ms. Number of repeats: 1000.



Variable      N   N*    Mean    SE Mean   StDev    Minimum      Q1    Median      Q3
C1         1000    0   16563       20.0     632      16128   16380     16447   16585

Variable   Maximum
C1           31237

                       Figure 110: Descriptive statistics for 20ms loops




                                             239
                                            Histogram of C1

            900

            800

            700

            600
Frequency




            500

            400

            300

            200

            100

             0
                  16000    18000    20000       22000   24000     26000     28000   30000
                                                        C1




                     Figure 111: Histogram of loop times for 20ms loop




                                         Time Series Plot of C1
            32500


            30000


            27500


            25000
  C1




            22500


            20000

            17500


            15000
                      1    100     200    300     400    500    600   700    800    900   1000
                                                        Index




                          Figure 112: Time series plot for 20ms loops

                                                    240
       The histogram in Figure 111 does not show the outliers very well at the maximum

value of 31237 but there are several in that area. The following two figures show the

times series and probability plots.

The time series plot in Figure 112 clearly shows a number of measurements which are well
                  separated from the mean value. The probability plot in

       Figure 113 shows the deviations from the normal distribution.




                                           Probability Plot of C1
                                                  Normal
                     99.99
                                                                               Mean        16563
                                                                               StDev       632.3
                                                                               N            1000
                       99                                                      AD        198.007
                                                                               P-Value    <0.005
                       95

                       80
           Percent




                       50

                       20

                        5
                        1


                      0.01
                             15000 17500 20000 22500 25000 27500 30000 32500
                                                  C1




                               Figure 113: Probability plot for 20ms loops




                                                    241
E.2.2 Run 2 – Shorter loops
      For the second run, the loop time was reduced by a factor of ten to produce loops

of about 2ms.



Variable      N           N*       Mean      SE Mean   StDev        Minimum        Q1   Median       Q3
C1         1000            0     1691.4         7.79   246.3         1630.0    1630.0   1631.0   1680.8

Variable   Maximum
C1          7062.0

                                 Figure 114: Descriptive statistics for 2ms loops




                                                   Histogram of C1
                        800

                        700

                        600

                        500
            Frequency




                        400

                        300

                        200

                        100

                         0
                              1600    2400      3200    4000         4800     5600   6400
                                                               C1




                                      Figure 115: Histogram of 2ms loops




                                                        242
                                    Time Series Plot of C1

          7000


          6000


          5000
 C1




          4000


          3000


          2000


          1000
                  1    100   200    300      400    500    600     700     800   900       1000
                                                   Index




                      Figure 116: Time series plot for 2ms loops



                                    Probability Plot of C1
                                              Normal
          99.99
                                                                                 Mean         1691
                                                                                 StDev       246.3
                                                                                 N            1000
            99                                                                   AD        215.139
                                                                                 P-Value    <0.005
            95

            80
Percent




            50

            20

             5
             1


           0.01
                  0   1000   2000     3000    4000     5000      6000    7000
                                             C1




                      Figure 117: Probability plot for 2ms loops



                                               243
       For the shorter loop time, there were less interruptions but the interruptions cause

a larger effect as can be see clearly in the time series plot in Figure 116. The probability

plot in Figure 117 shows how the interruptions cause significant variation from the

normal distribution.




E.2.3 Run 3 – longer loop times
      For this run, the loop time was made about 10 times longer than for the first

experiment so that it was of the order of 200 ms. The number of repeats was reduced to

100 to keep the experiment time short.




Variable     N   N*      Mean   SE Mean    StDev    Minimum       Q1   Median       Q3
C1         100    0    170676       413     4133     167482   168188   168694   170598

Variable   Maximum
C1          184045



                       Figure 118: Descriptive statistics for 200ms loops




                                              244
                                            Histogram of C1

            40




            30
Frequency




            20




            10




            0
                     168000        171000        174000        177000   180000    183000
                                                          C1


                          Figure 119: Histogram for 200ms loops




                                         Time Series Plot of C1
            185000




            180000




            175000
C1




            170000




            165000
                      1       10    20      30      40      50     60   70   80   90       100
                                                          Index




                       Figure 120: Time series plot for 200ms loops

                                                    245
        The histogram in Figure 119clearly shows two separate timing groups and this is

illustrated well in the time series plot in Figure 120.




                                                  Probability Plot of C1
                                                         Normal
                         99.9
                                                                                         Mean      170676
                                                                                         StDev       4133
                          99
                                                                                         N            100
                                                                                         AD        13.328
                          95                                                             P-Value   <0.005
                          90
                          80
                          70
               Percent




                          60
                          50
                          40
                          30
                          20
                          10
                           5

                           1


                          0.1
                                160000   165000     170000    175000   180000   185000
                                                        C1




                                Figure 121: Probability plot for 200ms loops



        The probability plot in Figure 121 shows the two groupings well. From the time

series plot and the probability plot we can see that about every third or fourth loop is

being interrupted and the interruptions are about the same order of time as the loop time.


E.2.4   Run 4 – very long times
        For the final run, the loop time was extended by another factor of ten to give

times of the order of two seconds. Only ten repeats were done.


                                                             246
Variable    N            N*       Mean     SE Mean     StDev          Minimum         Q1      Median        Q3
C1         10             0    1708332        2468      7803          1695198    1702070     1708209   1714748

Variable   Maximum
C1         1719470


                                 Figure 122: Descriptive statistics for 2s loops




                                                     Histogram of C1

                         3.0


                         2.5


                         2.0
             Frequency




                         1.5


                         1.0


                         0.5


                         0.0
                                 1695000   1700000     1705000        1710000   1715000   1720000
                                                                 C1




                                       Figure 123: Histogram for 2s loops



       We can see from the statistical data and the histogram in Figure 123 that the

interruptions are now occurring frequently enough to affect all the loop times.




                                                          247
                                        Time Series Plot of C1
               1720000



               1715000



               1710000
     C1




               1705000



               1700000



               1695000

                         1      2       3     4       5           6   7     8       9         10
                                                          Index




                             Figure 124: Time series plot for 2s loops



                                        Probability Plot of C1
                                                  Normal
          99
                                                                                    Mean      1708332
                                                                                    StDev        7803
          95                                                                        N              10
                                                                                    AD          0.157
          90
                                                                                    P-Value     0.929
          80
          70
Percent




          60
          50
          40
          30
          20

          10

          5


          1
           1690000            1700000       1710000         1720000       1730000
                                             C1




                             Figure125: Probability plot for 2s loops


                                                   248
        We can clearly see in Figure that the distribution is now very close to normal and

we can infer from this and the other data that interruptions occur in every cycle to about

the same degree.


E.2.5   Conclusions
        From these tests, it was concluded that the operating system was interrupting the

process somewhere in the region of every 20ms. With a large number of repeats, it could

clearly been seen which data were affected by the interruptions and that by excluding

outliers above about 3σ from the mean, the remaining data represent the code time which

is being attempted to be measured.




                                           249

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:18
posted:10/13/2011
language:English
pages:252