LOVE by liwenting

VIEWS: 4 PAGES: 8

									                         Alert Based Monitoring of Stock Trading Systems
        Edward W.Y. Ho 1, Dickson K.W. Chiu2, Senior Member, IEEE, and Patrick C. K. Hung3
               1
              Department of Computer Science, Hong Kong University of Science and Technology
                        2
                          Department of Computing, Hong Kong Polytechnic University
    3
     Faculty of Business and Information Technology, University of Ontario Institute of Technology, Canada
                        email: hoed@ust.hk, dicksonchiu@ieee.org, patrick.hung@uoit.ca


                    Abstract                                    HKEx’s servers [2].
                                                                     To allow routing of stock orders to the stock ex-
     Nowadays, stocks are traded electronically instead of      changes electronically, investment banking organizations
manually with open outcry approach. As a result, business       have to develop new trading systems to meet the specifica-
activities of investment banking organizations rely heavily     tions of the message standard of the stock exchanges.
on the availability of their trading systems. Any system        These new trading systems have to be connected to other
failure will directly affect their business and in turn dam-    internal systems to complete the transactions. The life
age their reputation. Due to the complexity of the business,    cycle of an order is illustrated on Figure 1.
trading of stock requires services provided by many sys-
tems even within the same company. Any failure of a sin-                      Order Capture       Validation of order
                                                                                                 And client information
gle system may stop the business. However, the monitor-
ing of many systems simultaneously is not an easy task.                     Order Management    Store order Information
                                                                                                and keep track of status
This paper proposes a Web service approach to monitor
all the systems related to stock trading within an invest-                                        Sending orders to
                                                                                Execution          stock exchanges
ment banking organization. We develop a model for speci-                                          or external brokers

fying how to detect potential system problems quickly, how
to escalate the issues to relevant parities on time with an                    Settlement        Payment by cash and
                                                                                                  Transfer of stocks

alert mechanism, and how to manage system outages
properly.
                                                                                 Figure 1. Order Life Cycle
1. Introduction
                                                                     As the trading systems are so critical to the business
     Recent advances in telecommunication systems and           of the investment banking organizations, they must be
Internet technologies have made the flow of news and            monitored closely. Any potential issues must be first noti-
information among investors and investment banking or-          fied to the technology staff. Technology staff must then
ganizations more rapidly than ever. Investors are now           determine how to resolve the issues and escalate to the all
much more knowledgeable. They feel quite comfortable            the affected parties.
to place stock orders without consulting a financial advisor.        This paper proposes an alert based approach to moni-
Furthermore, Internet technologies allow investors to send      tor different types of trading systems (order management,
orders easily with mobile phones or Internet browsers to        execution, settlement, etc.), based on our case study in a
all markets in the world. As a result, the demand of get-       large global investment bank. The new monitoring system,
ting more services from the financial industry has been         known as Stock Trading Monitoring System (STMS), de-
growing quickly for the recent ten years. To cope with          tects system issues quickly and informs relevant parties to
such demand, the stock exchanges of many countries have         take action. When there is an outage, appropriate actions
implemented electronic trading systems, which allow bro-        can be taken as soon as possible to minimize the impacts.
kers and financial institutions to hook up their trading sys-   After the issues have been resolved, all the detailed infor-
tems to the stock exchanges’ trading platform [1]. For          mation such as symptoms, root cause, resolution, and pre-
instance, the Hong Kong Exchange (HKEx) has estab-              ventive actions are recorded. All relevant staff can also
lished the Open Gateway (OG) for brokers and financial          learn from experience how to prevent similar problems. If
institutions to connect their own trading systems to the        the same problem occurs, the remedial actions can then be
                                                                carried out quickly.
     The subsequent sections will firstly discuss the back-         Market Data Systems – They distribute the market in-
ground information which includes motivation and related       formation such as bid and ask prices of stocks, which bro-
work. Then the detailed description of the new monitoring      kers are trading on the market and financial news to other
system is proposed. Finally, there are discussion, conclu-     systems. The data integrity and latency have a large im-
sion and future work.                                          pact on the execution systems.
                                                                    Reference Data Systems – They store the static infor-
2. Background and Related Work                                 mation of the clients such as credit profile as well as data
                                                               of the stocks such as the symbols (e.g., 0005.HK refers to
     Owing to the recent development of information tech-      HSBC in Hong Kong market). Missing such data will stop
nology and globalization of economy, large and global          the client or stock from trading.
investment banks have been developing many new sys-                 Settlement Systems – These refer to the settlement sys-
tems in the recent 10 years. As a result, there are many       tem of the investment banks. Their function is similar to
new systems running on the platform with cutting edge          the clients’. The difference is that the investment banks’
technologies, but many old systems are still running on old    settlement systems handle the trades of many clients while
technologies in the enterprise. The new systems usually        the clients’ settlement systems handle only their own
provide Web interface for administration and monitoring.       trades
Monitoring their status of is not difficult. However, the           External Reference Data Systems – They provide ref-
old systems usually do not have sophisticated interface for    erence data to investment banks’ reference data system,
monitoring. In order to improve the efficiency and effec-      such as Bloomberg, Standard & Poors, etc.
tiveness in such a complex environment, a common way                External Market Data Systems – They provide market
to monitor all the systems becomes vital. We study the         data to the market data system of the investment banks,
trading system of an investment bank because of its im-        such as Reuters and stock exchanges.
portance to both the business and also the clients. The typ-        External Stock & Cash Depository Systems – They are
ical main components include:                                  systems of external parties who keep track of the payment
     Client Trading Systems – The clients (e.g. investment     and the stocks clearance. Examples include commercial
management companies such as Fidelity) have their own          banks and central clearing companies such as HKEx in
systems to store the stock orders. They send the orders to     Hong Kong [6].
the investment banks with brokerage services using the
FIX protocol [3] via leased lines or the Internet. The ex-                                           External
                                                                                                      External
                                                                                                  Reference Data
                                                                                                                         External
                                                                                                                          External
                                                                                                                       Market Data
                                                                                                   Reference Data       Market Data
                                                                                                  Vendor System       Vendor System
ecutions and the order status can be updated in real-time.                                         Vendor System       Vendor System

                                                                     Client
     Client Settlement Systems – After trades are done,               Client
                                                                    Trading
                                                                     Trading
                                                                    System
                                                                                                    Reference
                                                                                                       Data
                                                                                                                         Market
                                                                                                                          Data
                                                                                                                                           Stock
                                                                                                                                            Stock
                                                                                                                                         Exchanges
                                                                     System                          System              System           Exchanges
payment instructions and how the stocks should be deli-
                                                                                                                         Manual
vered are sent from the investment banks through the ex-                         Order Routing
                                                                                   System
                                                                                                   Order Mgmt
                                                                                                     System
                                                                                                                        Execution
                                                                                                                         System
                                                                                                                                          External
                                                                                                                                           External
                                                                                                                                       Broker Systems
                                                                                                                                        Broker Systems
ternal stock and cash depository system. These systems
                                                                     Client                                            Automated
                                                                      Client                        Settlement
monitor such transaction status.                                   Settlement
                                                                    Settlement
                                                                     System
                                                                      System
                                                                                                      System           Execution
                                                                                                                        System
     Order Routing Systems – These systems capture the
                                                                                                     External
orders from the client and validate them. If an order is                                              External
                                                                                                    Stock Cash
                                                                                                   Stock && Cash
                                                                                                 Depository System
                                                                                                  Depository System
valid, it will be routed to the stock market of the target
country. For example, if the order is to buy a Japanese                 Figure 2. Trading platform of an investment bank
stock, it will be routed to the order management system
which handles the Japan stocks.                                     Figure 2 illustrates the interconnections of all the
     Order Management Systems – They keep track of the         components. It can be easily seen that all these compo-
order status and record the quantities of orders have been     nents are highly inter-dependent. Therefore, a failure in
executed and their transaction prices.                         any one of them will affect the whole operations of trading.
     Execution Systems (Manual & Automatic) – They in-         Also, both old and new systems co-exist. Old systems typ-
terface between the investment banks and the stock ex-         ically include order routing systems, manual execution
changes or external brokers, typically with the FIX proto-     systems, settlement systems, and the reference data sys-
col (e.g., the Singapore stock exchange [4]). Manual ex-       tems. New systems typically include order management
ecution systems require brokering staff to manually click      systems, automated execution systems, and market data
buttons to send orders to the stock exchanges or outside       systems. It makes support difficult. Furthermore, there are
brokers. In contrast, automatic execution systems send         external dependencies. For instance, if there are problems
orders to the stock exchanges or brokers automatically         with any of the external market data vendor systems, ex-
according to a pre-defined algorithm [5].                      ternal reference data systems, stock exchanges, external
broker systems, or stock and cash depository systems, the       new approach to monitor mission-critical enterprise sys-
whole trading platform will be affected. The diagnosis          tems.
processes are often time-consuming and further compli-
cated with user’s desktop problems or physical network          3. System Overview
problems.
      Recently, due to cost reduction, many investment               The new stock trading monitoring system (STMS) is
banks have outsourced system development to external            designed to match the needs of all stakeholders which in-
contractors and even to overseas (such as TATA in India         clude external clients, internal clients, the technology ap-
[7]). As a result, instead of having many project teams to      plications project team, the technology applications sup-
monitor their own systems, a single support team has to         port team, the technology infrastructure team, external
monitor all the trading systems. Since resources are li-        service providers, stock exchanges and external broker
mited, the support team cannot manually login to all sys-       firms (as illustrated on Figure 3).
tems to check. This organizational change drives the need
for a common and easy monitor system. When there is a
                                                                                                         External Brokers
problem, the support team can then escalate the problem                                                                        Stock Exchanges
                                                                              External Clients
details to the relevant parties quickly.
      On the other hand, the business environment of the fi-
nancial industry becomes more competitive than ever.
Many clients (especially institutional ones) can directly                                            Securities Trading              External
                                                                                                                                 Service Providers
connect to the investment banks’ servers for services, such
                                                                      Internal Clients               Monitoring System
                                                                                                          (STMS)
as FIX protocol trading, online research reports. Any sys-
tem problems can be easily revealed to clients and possi-
bly spread to competitors. As electronic trading services                Technology Applications
                                                                                                                                 Technology
                                                                                                                             Infrastructure Team
                                                                             Project Team
are not very differentiable, clients can easily switch to                                          Technology Applications
                                                                                                       Support Team
competitors to get similar services upon repeated problems.
      There are three types of common complaints from the                            Figure 3. Stakeholders of STMS
clients in this trading environment. Firstly, it is unable to
detect problems quickly. The monitoring of all systems is            External clients include both institutional clients such
not centralized. When one system fails, only a small group      as Fidelity and retail customers. Internal clients are the
of (technology) people are aware of it. Secondly, it is not     people who use the systems for their daily business opera-
quick enough to take remedial actions to minimize the           tions. They are sales, traders, dealers, financial controllers,
impacts. After a system is recovered, other systems are         and settlement officers. Applications project teams usually
still disconnected from it because different systems are        refer to the team responsible for system development. For
being taken care by different teams. Clients are also not       example, they design the Index Arbitrage systems, FIX
notified of the progress of system recovery. Thirdly, simi-     engines, and the gateways to the stock exchanges. The
lar problems are not prevented. There are no centralized        technology applications support team is responsible to
knowledgebase storing the details of the problems and           provide daily maintenance of all the applications. They are
solutions or preventive measures to avoid similar prob-         the people who operate the STMS. The Infrastructure
lems. So, when similar problems happen again, it still          teams are teams maintaining the hardware of the servers,
takes similar amount of time to fix the problems.               the desktop workstations, the database, and the network.
      There have been technical papers on how to stream-        External service providers include market data vendors
line the flow of alerts. For example, Chiu et al. [8] de-       such as Reuters, reference data vendors such as Bloom-
scribe how to use Web services to implement alerts in           berg, wide area network (WAN) network providers such as
healthcare processes. However, it does not directly fit into    AT&T, telephone companies such as PCCW, and also
the financial service industry, which has a radically differ-   some external technology solutions provider that provide
ent business environment. Ng and Chiu [9] also propose          maintenance of some services such as email. Stock ex-
an emergency route advisory system but the context is           changes are the official exchanges where stocks are traded
similarly inapplicable to the financial industry. Chiu et al.   such as Hong Kong Exchange (http://www.hkex.com),
[10] also describe about the distributed e-Monitoring Sys-      Singapore Exchange (http://www.sgx.com) and Australian
tem and how to use the Web services to monitor but it does      Stock Exchange (http://www.asx.com.au). External bro-
not address the problems found in the investment banking        kers are the broker firms which can execute the orders on
organizations, such as existence of old and new systems,        behalf of other investment banks. They are usually used in
etc. Therefore, we have a strong motivation to propose a        the locations where infrastructure is relatively poor, finan-
cial regulations are strict, or the business size is not large,                       Central Outage Knowledge Management System.                                                            It is
such as the Philippines, China, and Malaysia.                                         illustrated on Figure 5.
                            Infrastructure                                                                                           Central
                             Infrastructure
                              Monitoring
                               Monitoring                                    mobile                                             Monitoring Server
                             Web Server
                              Web Server
                                                                                                Checking Daemons
                                                                                                 Checking Daemons                 Web Services                  New Servers
                                                                                                                                                                  New Servers
                                                                                                   Running on
                                                                                                    Running on                      Adaptor                    With web servers
                                                                                                                                                               With web servers
       New Systems              Central          Alert System                                      old servers
                                                                                                    old servers
        New Systems
          with             System Monitoring     (To send out
           with
       Web services           Web Server       Warning messages)              email
        Web services                                                                                                              Infrastructure
                                                                                                                                    Monitoring
                                                                                                                                   Web Server
                                Web
                                 Web                                                      Central Outage Knowledge                                                   Alert
         Central               Service
                                Service                                                     Management System                                                       System
     Outage Knowledge          Adaptor                                                                                               Central
      Mgmt System               Adaptor                            Client
                                                                    Client                                                         Transaction




                                                                                                                                                                    Logger
                                                                   System




                                                                                                                                                                     Alert
                                                                    System                                                           Handler




                                                                                                      Incident
                                                                                                       Logger
                             Old Systems
                              Old Systems
                               without
                                without         Console




                                                                                                                 Web Server




                                                                                                                                                       Web Server
                                                                                                                                     Operator




                                                                                                                                                                    Transmitter
                             Web services
                             Web services                                                                                       Controlling Handler




                                                                                                                                                                       Alert
                                                                                           Incident                                                                                 Alert
                                                                                          Database                                                                                Database
                        Figure 4. Components of STMS




                                                                                                      Incident
                                                                                                      Matcher




                                                                                                                                                                    Tracker
                                                                                                                                                                     Alert
     As shown in Figure 4, the STMS can be logically di-
vided into several components:                                                                                                Support Team’s console

     Central System Monitoring Web Server – The applica-                                                         Figure 5. Architecture of STMS
tion support team can use a single console to see the status
of all servers. The technologies to be used between all web                                The Central Monitoring Server consists of the Web
components are WSDL, UDDI, and SOAP for better shar-                                  services adaptor, the infrastructure monitoring web server,
ing of data and services.                                                             the central transaction handler, and the web server. The
     Alert System – When there is a system problem or po-                             Web services adaptor calls a number of remote checking
tential issue, the alert system sends messages to all inter-                          daemons running on the remote old servers to get the sta-
ested parties by email, mobile phone, and/or external                                 tus. Such collected data include the status of whether any
client systems. Some designated staff will be the primary                             specified processes are running, any application errors in
receivers of the message. If the message is not acknowl-                              the log file that cannot be obtained by any Web services.
edged within a certain time, it will be resent to a more                              An example of error in log file is shown below:
                                                                                          10:56:06,027:: ERROR [AffinityThreadPool Dispat-
senior staff.                                                                             chingThreadPool#1] -- Child order [ED1-3-3] has
                                                                                          been rejected: Cannot create an external reference
     Web Service Adaptor – If the old systems do not pro-                                 for
vide Web services for monitoring, a Web service adaptor is
used to capture their status and pass the results to the cen-                              The infrastructure monitoring web server use Web
tral system monitoring web server. Status of the old sys-                             services to check the health of hardware of various servers,
tems may be obtained by executing some system-specific                                including the internal and external network. The central
commands or just examining the log file if there are no                               transaction handler is the central commander, which gets
better alternatives.                                                                  the results from other components and sends the com-
     Infrastructure Monitoring Web Server – It regularly                              mands to appropriate components to take actions. The
checks the health of the network as well as the hardware                              operator controlling handler is the interface between the
of servers such as CPU usage and disk space. It reports                               support team’s console and the central transaction handler.
the results back to the central system monitoring web                                 It can control the parameters such as the severity of an
server.                                                                               alert, user roles and permission, etc.
     Central Outage Knowledge Management System – It                                       The Central Outage Knowledge Management System
stores the details of the problems and their resolutions.                             stores the details of the outage and employs case-based
Preventive maintenance can then be created from the                                   reasoning system to give suggestions to the support team
knowledge to avoid similar problems. When a similar                                   how to escalate and fix the problems quickly. The incident
problem occurs, relevant information can be retrieved as                              logger validates the incident and writes it into the database.
reference. Statistics can be generated to illustrate the fre-                         The incident matcher creates indices of the incidents ac-
quencies of different types of problems. As a result, the                             cording to certain attributes. When a new alert is encoun-
performance of relevant technology staff can be measured                              tered, it will search the database to find the relevant cases
against the statistics.                                                               for reference. Users may login to the knowledge manage-
                                                                                      ment system to use the search engine to find the relevant
4. System Design                                                                      cases to plan for project roll-out and preventive mainten-
                                                                                      ance.
    The SMTS can be divided into 3 major subsystems,
                                                                                           The Alert System is responsible to send alerts to all
which are Central Monitoring Server, Alert System and the
                                                                                      stakeholders. The alert logger records the details of the
alert into the database. The alert transmitter determines         tics to retrieve the relevant cases and send short message
who should get the alerts according to the details of the         to the group of people same as those receiving the alerts.
alerts. For example, if there is hardware issue, such as disk     The relevant staff can then use a web browser to retrieve
90% full, the infrastructure team, the application project        the details. The staff should update the incident and close
team, and the support team will be notified. The clients          it after fixing the problem. If the incident is not closed
need not know about this. In contrast, if a stock order is        after a specified time, a new alert will be generated again
rejected because of missing client details, the external and      Of course, this alert will not create any new incident; oth-
the internal clients should also be informed. The alert           erwise there will be duplicated incidents and loops. Figure
tracker will keep track of who has acknowledged an alert          6 illustrates the relation between incident, alert, owner,
within a specified time. If there is no reply, the alert will     system, individual and group while the high level data
be escalated to a larger group or to a more senior level          structure is illustrated in the tables below.
person.
     All the Web services provided by different sub-
                                                                      System
systems employ SOAP for better compatibility and expan-
                                                                           1
sion. For example, if a new client joins and they request
                                                                           1..*
for monitoring their system, the development time will be                          1    1..*
much shorter.                                                        Incident                     Alert

     Regarding the flow, an alert is the starting point.                    1..*                     1..*
When the central transaction handler receives a message
                                                                                                    1
from other web servers, it considers that there is a real or
                                                                                         1
potential problem. It will then send a request to the alert                                      Owner
system, which will generate an alert. Simultaneously, the
central transaction handler will also send a request to the
central outage knowledge management system, which                    Individual    *                            *     Group
                                                                      Person
creates a new incident and store it into the database.
     The alert tracker uses a timestamp field in an alert to
store the time when it is sent. If there is no reply after a                           Figure 6. Relationship of Data
specified time, the urgency will be increased. Depending
on the severity of the alert, the urgency does not necessari-     ALERT
ly start from the lowest one, say “Normal”. Let t be              Fields                         Description
elapsed time after the first alert is sent. Various actions       Source                         Where problem is detected
will be taken according to the example urgency policies as        Details                        Brief description
illustrated in Table 1. The criticality is illustrated as fol-    Urgency                        (Explained in Table 1)
lows.
                                                                  Entered Time                   Time when alert is produced
    0≤t<T                               Normal
                                                                  Resolved                       Yes or No
    T ≤ t < T +t1                       Urgent
                                                                  Owner                          Person who acknowledge alert
    T + t 1 ≤ t < T + t1 + t2           Very Urgent
                                                                  Timestamp                      Time where it was last touch
    T + t1 + t2 ≤ t < T + t1 + t2+ t3   Critical
                                                                                               Table 2: Alert Record
    T + t1 + t2+ t3 ≤ t                 Very Critical
Urgency            Action                                         INCIDENT
Normal             Submit the first alert to a group              Fields                         Description
Urgent             Resubmit to the same group, reminding them     Type                           Hardware, software, etc
                   that deadline is close                         Alert                          Related to alert record
Very Urgent        Resubmit to a more senior staff of the same    System                         Related to system record
                   group                                          Symptom                        Detailed description of the problem
Critical           Resubmit to all technology staff               Resolution                     Time when alert is produced
Very Critical      Broadcast the message the all relevant staff   Root Cause                     Cause of problem
                                                                  Severity                       High, Medium, Low, etc.
                       Table 1: Urgency Table
                                                                  Dollar Impact                  Real or potential loss in dollar value
    When an alert is created, an incident is also created.                                   Table 3:Incident Record
According to the details given by the alert, the central out-
                                                                  SYSTEM
age knowledge management system uses various heuris-
                                                                  Fields                         Description
Name                  Name of System                            the incident with details. Once the application team re-
Owner                 Project manager                           ceived the alert with CLOSE state, they knew that problem
Description           Brief description of system               was fixed. They restarted some processes in the order
Nature                Order routing, Order management, Ex-      management system and informed the internal clients, who
                      ecution, Settlement, etc.                 then resumed their normal operations.
Uptime                24 x 7 or Mon – Fri, etc                        Before the STMS was launched, the outage typically
                   Table 4: System Record                       lasted for 2 hours. The reason was that even after the serv-
                                                                er team knew and fixed the problem, the application sup-
OWNER
                                                                port team and the internal clients were not aware of the
Fields                Description                               updated situation. The sales and traders still kept putting
Name                  Name                                      order into the order management system and using the
Nature                Group (G) or Individual (I)               execution systems. However, all the orders were stalled.
Rights                Some can close incident, some can have    The sales and trades kept complaining that there was no
                      read-only rights                          response but the application support team found that all
Contact Details       mobile number, email address, etc         the processes were running although the CPU usage was
Uptime                24 x 7 or Mon – Fri, etc                  high. Finally, the support team called the server team to
                   Table 5: Owner Record                        find out the cause. After the processes were restarted in the
    In order to have better communications with Web ser-        server, the problem was fixed. However, owing to 2 hours
vices, the messages will be in XML format. An example           outage, some external clients switched to another invest-
of a new alert found in the automatic execution system          ment bank. Now with the help of the STMS, the outage
(AES)                                                           was reduced to 20 minutes.
                                                                      Here is another scenario. Another system failure
    <Alert>                                                     started from a network problem. There was a few minutes’
         <Source>AES</Source>                                   disconnection between Hong Kong and Japan at 3:00am.
         <Details>Order Rejected</Details>                      Alert was sent to the infrastructure team (including the
         <Urgency>Very Urgent</Urgency>                         network team, the market data team and the server team),
         <EnteredTime>11:30:33</EnteredTime>                    the application support team and the internal clients. The
         <Resolved>N</Resolved>                                 network team acknowledged the alert and called the exter-
         <Owner></Owner>                                        nal network provider to investigate. It was decided to
         <TimeStamp>11:30:33</TimeStamp>                        switch over the WAN from the primary link to the backup
    <Alert>                                                     link. The problem was fixed at 5:00am. Once the market
                                                                data team knew that the incident was closed, they re-
5. System Walkthrough Scenarios                                 bounced the market data server. When the application sup-
                                                                port team arrived at the office at 7:00am, they checked that
     Here is an illustrative scenario showing how the prob-     all processes ran normally. When the sales and trades
lems are detected, escalated, and resolved. The disk of the     started using the trading systems at 7:30am, there was no
order management system failed at 2:00pm. An alert was          problem. Before the STMS was implemented, the network
sent to the infrastructure team, the application support        problem made the market data system fail, which in turn
team, and the internal clients (sales and traders). After the   made the automated execution system operation go wrong,
server team of the infrastructure team received the alert,      i.e., orders with wrong prices were sent to the market.
they acknowledged it. The acknowledgement was for-              There could be significant loss in that case. Now the
warded to the application support team and the internal         STMS significantly made the problem transparent to the
client to inform them that someone was investigating the        clients.
problem. Simultaneously, the application support team got             Some common issues that the STMS monitors are il-
the relevant cases from knowledge management system.            lustrated in the following table
They discovered that other systems including a manual
execution system and an automated execution system were         Issues                                    Type
affected. Therefore, they inform the affected internal          Missing processes in server               Applications
clients that other alternatives such as stock exchange trad-    Disk response time, percentage of disk    Hardware
ing terminals should be used. After the internal clients        usage
were informed, they redirected their orders to exchange         Percentage of CPU usage                   Hardware
terminals. The disk problem was fixed at 2:30pm. The            Packet loss in network                    Network
server team marked the alert to a CLOSE state and closed
Latency of getting market prices              Market Data       6. Discussion and Summary
Checking data content of market price         Market Data
                                                                     The STMS has shortened the outage time and mini-
Transaction rejected by processes             Applications
                                                                mized the business impact. As shown in the cases in the
Orders rejected because of missing stock Applications
                                                                previous section, the infrastructure team and the applica-
or counterparty
                                                                tion support team can now work closely to resolve issues.
Response of an transaction longer than Applications
                                                                The internal clients will be kept posted of the progress of
expected time (e.g. after an order is sent to
                                                                the problems. They can make better decisions during the
the stock exchange, there is no acknowl-
                                                                outage. It is even better that some problems are fixed be-
edgement after 2 seconds)
                                                                fore the clients can detect them. For example, if there is
Inconsistency of static data (stocks, cli- Applications
                                                                inconsistency of static data among different systems, such
ents, users) between different applications
                                                                as a new stock, the operations department can rectify them
Missing data file (e.g. A list of restricted Applications
                                                                before the data is used. As a result, when the external
stocks is downloaded from the compliance
                                                                clients send an order of the new stock, there will be no
database to the trading database to prevent
                                                                problem. In contrast, orders will be rejected. Nowadays,
trading for some stocks)
                                                                hardware cost is much cheaper that human cost. After the
Checking the status of overnight jobs         Applications      STMS is launched, new systems can be easily monitored.
Heartbeat missing (heartbeat is used to Applications            No additional support staff is normally required. It is
make that the connection between process                        planned to use STMS in all offices.
in servers and clients are active)                                   However, there may be problems in some countries
                Table 6: Issues monitored by STMS               where the language and infrastructure have limitations.
     The STMS is designed to handle most problems               People in some countries may not comprehend English
found in the trading environment. However, there will be        very well. Having multi-lingual support of the STMS is a
some exceptions and limitations. Some typical problems          big effort. Having local language interface for some im-
are highlighted as follows, in which human attention could      portant modules is possible but it will take extra time and
be drawn for solving the problems manually.                     cost to build it. Also, initially, false alarms may occur of-
     During summer in Hong Kong, typhoons may lead to           ten. It takes some time to filter out the wrong one. In long
half-day trading. For example, the stock exchange may           term, it is hoped that the investment banking organizations,
announce that trading to be halted at lunch time. As a re-      the stock exchanges and the external vendors can develop
sult, there will be many alerts reporting that there is no      more Web services to make the monitoring process more
market data update in the afternoon. In this case, the sup-     efficient.
port team may have to login to the system to manually                The main future work is to evaluate the effectiveness
override some rules.                                            of this approach. In order to get more comprehensive
     Furthermore, when the STMS cannot get the status of        feedback, after the STMS has been launched for several
some servers being monitored for a certain period of time,      months, feedbacks have to be obtained from all level of
it cannot distinguish whether it is a network problem or it     staff from various departments. For example, question-
is a server problem (e.g., a system hangs). Manual inter-       naires should be sent out by email. Representatives from
vention is needed to check what has happened.                   each group of internal and external clients should be inter-
     The Web service adaptor is used to check the status of     viewed. On the other hand, we can evaluate whether the
the old systems. Some old systems actually run on non-          same system problems can be avoided. The number can
English platform such as Korean, Japanese, or Chinese.          easily be obtained from the incident records of the STMS.
Finding the errors from the log file may be very difficult.     If a particular team can reduce the outage significantly, the
     As investment banks are global organizations, “fol-        results will be present to the management for appreciation
low-the-sun” support model is usually adopted. For in-          and rewards.
stance, when there is a system failure in a server in Hong           The Internet technology is evolving rapidly. Web ser-
Kong at midnight, the New York team should attend.              vice is the trend of communications between various sys-
However, if the day is a holiday in New York, the alterna-      tems. This Web service approach of monitoring the critical
tive is London team. The holiday table is maintained in         trading systems should significantly reduce the response
STMS. If the holiday table has some wrong or missing            time of rectifying system problems. Consequently, the
data, alerts may be sent to an incorrect destination. It will   competitiveness of the company can be increased to a
make the response time of acknowledging an alert much           higher level. We expect our alert based approach is general
longer.                                                         enough and can be apply to most other organizations, in
                                                                     Order Received



                      Check                                  Req
            Enquiry                 Prepare       Send                  Send          Prepare   Deliver &   Payment
  Begin    Received
                      System                                 Extra                                                           End
                                   Quotation     Quotation   Info     Extra Info      Service    Install    Received
                      Config

                                                                        Request
Sell Integrated System                                                   Extra
                                                                         Info




              which the criticality of application systems are general less Install
                                                 Order
                                                            Assemble                             Test
                                 Begin          Missing                                                                End
              than the case that we have studied in this paper.
                                                 Parts
                                                             System        Software             System


                               Prepare Service
     System
     Integrator
              References           Begin
                                                 Update
                                                                         End
                                                 Catalog

              [1] Hong Kong Monetary Authority, “Impact of Electronic
                              Receive Part Info Updates
                   Trading”, www.info.gov.hk/hkma/eng/research/RM16-
                   2001.pdf
              [2] Hong Kong Exchanges and Clearing Limited, “AMS/3 sys-
                   tem http://www.hkex.com.hk/infra/ams3/ams3.htm
              [3] FIX Protocol Limited, “What is FIX ?”,
                   http://www.fixprotocol.org/what-is-fix.shtml
              [4] Singapore Stock Exchange, “Trading System”,
                   http://www.sgx.com
              [5] Investopedia”, “What is algorithmic trading”,
                   http://www.investopedia.com/terms/a/algorithmictrading.as
                   p
              [6] Hong Kong Exchange and Clearing Limited, “CCASS/3”,
                   http://www.hkex.com.hk/infra/ccass3/ccass3.htm
              [7] TATA group, “IT Services”, http://www.tata.com/
              [8] Dickson K.W. Chiu, Benny W. C. Kwok, Ray L. S. Wong,
                   S.C. Cheung, Alerts for Healthcare, 2005.
              [9] Cherrie W.W. Ng and D.K.W. Chiu. “e-Government Integra-
                   tion with Web services and Alerts: A Case Study on an
                   Emergency Route Advisory System in Hong Kong,” 39th
                   Hawaii International Conference on System Sciences, Jan
                   2006.
              [10] Frank K.W. Cheong, Dickson K.W. Chiu, and S.C. Cheung.
                   “Developing a Distributed e-Monitoring System for Website
                   and Web services: An Experience Report with Free Libra-
                   ries and Tools.”, 2005

								
To top