Information Technology for Assisted Living at Home - Working at by sa30230


									EECS, University of California, Berkeley and Aarhus University                 August 2005

   Information Technology for
   Assisted Living at Home
   - Working at Berkeley
   Progress report by

   Thomas Riisgaard Hansen (,
   University of California at Berkeley, University of Aarhus 2005


   Shankar Sastry                      Ruzena Bajcsy             Mike Eklund

   Morten Kyng                          Jakob Bardram
          EECS, University of California, Berkeley and Aarhus University                                                                         August 2005

                                                                                6     TALKS, CLASSES AND SUPERVISION                                           30
                                                                                6.1    TALKS                                                                   30
1.1    INTRODUCTION                                                         3
                                                                                6.2    CLASSES                                                                 30

2     RELATED WORK                                                          4   6.3    SUPERVISION                                                             31

2.1    HOME MONITORING                                                      5
                                                                                7     OTHER COLLABORATIVE PROJECTS                                             32
2.2    PERSONAL MONITORING SYSTEMS                                          6
                                                                                7.1 MOBILE VIDEO CONFERENCING BETWEEN MULTIPLE PARTICIPANTS (WITH MARCI
2.3    OTHER RELATED RESEARCH PROJECTS                                      6   MEINGAST)                                                                      32

                                                                                7.2    COMPUTER VISION ON MOBILE DEVICES (WITH PARVEZ AHAMMAD)                 33
3     CONCEPTS                                                              7
                                                                                7.3 CONTROLLING UAVS (UNMANNED AIR VEHICLES) WITH MOBILE PHONES (WITH
3.1    CATEGORIES OF RESEARCH PROJECTS                                      7   DAVID HYUNCHUL SHIM)                                                           34

                                                                                BAYEN)                                                                         34
3.3    PRIVACY                                                             12
                                                                                7.5    USB / MOBILE PHONE (MIKE MANZO, MIKE EKLUND)                            35
3.4    ENVISIONED SYSTEM ARCHITECTURE VERSION 1.0                          13
                                                                                8     ACTIVITIES RELATED TO PREVIOUS WORK AND PUBLICATIONS                     36
4     PROTOTYPES, EXPERIMENTS AND RESULTS                                  15
                                                                                8.1    PRESENTATIONS AT CHI 2005                                               36
                                                                                8.2    MIXIS FACE TRACKING                                                     36
4.2    PROTOTYPE TWO: CONNECT FALL SENSOR TO PC                            17
                                                                                8.3    PUBLICATIONS WRITTEN DURING MY STAY                                     37
                                                                                9     REFERENCES                                                               38
                                                                                9.1    PAPERS                                                                  38
                                                                                9.2    COLLECTION OF ELDERLY RELATED RESEARCH PROJECTS                         39
4.6    DATA COLLECTION: SONOMA                                             23
                                                                                9.3    UNIVERSITY RESEARCH PROJECTS                                            39
                                                                                9.4    RESEARCH PROJECT WITHIN LARGE COMPANIES                                 39
5     FUTURE WORK                                                          27
                                                                                9.5    OTHER REFERENCES                                                        40
5.1    DESIGNING THE HUMANS INTO THE LOOP                                  27

5.2    FALL SENSOR VERSION 2.0                                             27

5.3    IT SUPPORTED HEALTHNET                                              28

5.4    CONCEPTS, THEORIES AND FRAMEWORKS                                   29

                                               1                                                                             2
   EECS, University of California, Berkeley and Aarhus University                                                             August 2005

1 Information Technology for Assisted Living at                           have formed a strong group who is going to make a
                                                                          significant contribution to how IT can be used to provide
  Home (ITALH)                                                            better healthcare for the elderly in the future.
1.1 Introduction                                                          During the last five month I have been working with
                                                                          researchers at University of California at Berkeley on
     The demand for healthcare is growing together with a                 surveying the research area, discussing novel ideas and
     population that is growing older. To meet the demand for             central concepts, building exploratory prototypes and
     more healthcare people are starting to look for new                  collecting data.
     healthcare constructions that provide better healthcare to
     more people without a huge increase in cost.                         Section 2 briefly describes related research projects
                                                                          addressing healthcare for the elderly and assisted living.
     The foundations of these new constructions are going to be           Section 3 reports on some of our initial concepts and ideas
     information technology and especially the newest version of          that we have found interesting, but that still need further
     information technology labelled pervasive or ubiquitous              work. Section 4 lists the set of prototypes we have built and
     computing. That is information technology that is not                data collection sessions we have performed and the section
     trapped in a desktop computer, but is distributed in the             reports on some of the results. Section 5 concludes the
     environment, in our clothes and maybe even inside our                ITALH part by listing future research challenges for the
     bodies.                                                              ITALH project at the current point in time.
     One way this new technology will restructure the health              Section 6, 7 and 8 lists other activities I have performed
     industry is by keeping people out of residential homes and           during my visited that is not directly related to the ITALH
     hospitals and in their own home continuing their daily               project. Section 6 lists talks given and classes taken.
     routines. The promise is that IT will help detect signs of           Section 7 lists other projects I have been discussing with
     illnesses before it gets serious, IT will support everyday           researchers at UC Berkeley and Section 8 lists papers
     tasks, IT will allow medical consultation between the home           written while at Berkeley and research activities related to
     and the hospital, and IT will help in continuous monitoring          previous projects.
     of a patient’s condition by medical professionals while the
     patient stays in their own home.
                                                                      2 Related Work
     Another promise of this new technology is that it will enable
     elderly people to grow old in their own home without having          Healthcare for elderly people is a huge area with lots of
     to move to a residential home for elderly people.                    potentials and many of the larger universities and
                                                                          companies are doing some kind of research within the area.
     However, how these future healthcare constructions is going          The following list is far from a complete list of related
     to be realised or how the information technology is going to         research projects, but more samples of research projects
     be constructed is far from obvious and there is a large              within some research themes within health care and
     unmapped research area to be explored.                               assistive living at home. CAST is a dynamic web site for
                                                                          keeping track of research project related to aging and
     The promises of a huge market together with the                      [Ross] gives another good overview of the area.
     challenging nature of the research area have drawn the
     attention of many large research companies and
     universities. The ITALH project is a coalition put together by
     researchers and companies from the Bay Area in California,
     Finland and Denmark. By combining research resources and
     experiences from different regions of the world we hope to

                                        3                                                              4
    EECS, University of California, Berkeley and Aarhus University                                                               August 2005

2.1 Home Monitoring                                                     2.2 Personal monitoring systems
2.1.1 Sensor Monitoring                                                       Personal monitoring systems are systems that are worn
                                                                              either attached to a person, part of a person’s cloth or
      A large cluster of research projects focus on detecting the             implanted. The advantages of personal monitoring systems
      activities and well being of elderly persons by fusing sensor           are that they are designed to always follow the user and
      data. The common setup is to distribute a large number of               they can sense bio-signals not accessible to remote sensors.
      sensors throughout the home (e.g. detecting open and                    The disadvantages are that these devices have to be worn
      closed doors, the number of times the stove or the                      and therefore might be more intrusive that devices mounted
      refrigerator is used, movements in the different rooms, the             in the home. Supplying the devices with the required power
      amount of water used …). Based on a fusion of these sensor              is another issue with personal monitoring systems.
      data the researchers tries to extract the current activity of
      the person being monitored and try to locate deviations in              UbiMon by Imperial College in London [UbiMon] [Lo], Mobile
      daily routines that can be due to a medical condition.                  Health Toolkit by IBM [IBM], CodeBlue from Harvard
                                                                              [CodeBlue] are examples of monitoring system. Lo and Yang
      Placelab is a MIT project that works with sensor network in             [Lo] list some challenges for personal monitoring systems:
      the home [Placelab], Intel has a similar project called                 Designing novel Biosensors and MEMS integration, provide
      ProHealth [ProHealth], Georgia Tech has a project called                novel power sources and ways of doing power scavenging,
      AwareHome [AwareHome], Rochester has their center for                   explore novel radio techniques and wireless data paths, use
      future health [Rochester], British Telecom’s HouseCare                  context awareness and multi-sensory data fusion, and find
      [HouseCare], and General Electric has a project Called                  reliable and secure light-weight protocols.
      Home Assurance System [GE HAS]. The projects are pretty
      similar and pursue the same goals and many of them have a         2.3 Other Related Research projects
      test house wired up with sensors where preliminary
      experiments have been conducted.                                  2.3.1 Fall Detection

      However, there are still a number of tough research                     The main technology that has been used to detect elderly
      questions. The challenge of accurately determine a person’s             people falling is cameras. SIMBAD [SimBad] and UbiSence
      current activity is still not solved. And even with a correct           [UbiSence] are two British research projects that both use
      classification of activities it is still difficult to pick out          cameras to track the movement of an elderly person and try
      activities that might be due to a potential illness. Multi-user         to detect falls. However, only preliminary results have been
      monitoring in the home seems to be a largely unexplored                 collected with the technology at the current point in time.
                                                                        2.3.2 Telemedicine and homecare
2.1.2 Video Monitoring
                                                                              Another huge related research area is telemedicine applied
      A special type of monitoring sensor in the home is the                  to homecare. The basic scenario is that medical
      camera. While a camera can provide a rich set of data it is             professionals at a hospital are able to remotely monitor and
      still not trivial to analyze the video stream and a whole set           check a patient’s health condition through tele-sensing and
      of privacy issues are related to camera monitoring.                     tele-conferencing technology. The Peach project at Aarhus
      UbiSense [Yang], SimBad [Sixsmith] and CareMedia                        looks at how medical professionals are able to treat foot
      [CareMedia] [Hauptman] are examples of systems that use                 ulcers remotely over a video link [Clemensen] and
      video tracking to locate people, detect falls and determine             Honeywell has an entire platform for monitoring a person’s
      current activity. The challenges with video monitoring have             health remotely [HomMed].
      some of the same problems and issues as mentioned above
      in the sensor monitoring section.

                                         5                                                                 6
   EECS, University of California, Berkeley and Aarhus University                                                               August 2005

3 Concepts
3.1 Categories of research projects                                              Assistive technologies

                                                                                 Assistive technologies are aimed at helping people with
     As pointed out by the list of related projects the area is
                                                                                 temporal or chronicle dieses.
     huge and it can be helpful to subdivide the projects
     depending on the type of healthcare service they aim to                        Assistive services
     provide. We have looked at a categorisation of the different
     research projects into three basic categorises depending on                   •   Help to take the right medicine
     the main focus: Preventive, Intervention, or Assistive. Is                    •   Hearing aid
     the service going to be preventive? Is it going to be an                      •   Helping device for blind people
     intervention? Or is it an assistive service?                                  •   Wheel chairs
                                                                                   •   Accessible homes
      Preventive services

      A preventive service aims at detecting or removing a
      possible medical condition before it happens and can be
      subdivided into:
         Healthy Living                Predicting health problems

         •   Encourage exercise                •   Look for change in daily
         •   Keeping social                        behaviour
             relationships                     •   Look for change in physical
         •   Eat healthy                           abilities
         •   Be mentally active                •   Regular visits by district


      Projects that focus on intervention aims at helping the
      elderly when medical conditions arise. They can also be
      divided into:
          Emergency                    Scheduled intervention
                                      • Surgery
         • Send alert in case of a    • Removing of stitches
             fall or a heart attack
         • Transmit vital signs in
             case of an emergency

                                        7                                                                     8
    EECS, University of California, Berkeley and Aarhus University                                                          August 2005

3.2 IT Supported HealthNet

      Within the ITALH project we have worked on a sub project
      called IT Supported HealthNet (prior know as SensorNet).
      The focus of ITS HealthNet is in its current phase on
      interventions in emergency situations. Being able to get
      help in an emergency situation seems to be one of the
      fundamental challenges of elderly living independently in
      their own homes. Will someone be able to intervene and
      provide the required help in case of an emergency?
      Providing the elderly person with the comfort of knowing
      that someone will be there in case of an emergency is the
      focus of the first phase of ITS HealthNet.

      The first emergency condition we have focused on has been
      falling. Researchers estimate that 30 percent of people aged
      75 will fall at least once a year. The focus has therefore
      been on developing IT support for detecting serious falls,
      but at the same time we have tried to come up with a
      general architecture that is able to expand and support
      other kinds of applications and use scenarios. Several other
      research projects are pursuing the same goal, but in the
      following we will try to highlight our angle on some of the
      research problems.

3.2.1 Component based architecture

      We have worked with and envisioned a component based
                                                                          Table 1: Component based architecture
      architecture. However, instead of just focusing on sensor
      components or software components a service in our
      architecture is not only made out of technical components,
                                                                       User components
      but the role of the user, the role of the organization and the
      business models for the service are also treated as              We have explored the idea of user components and the
      replaceable components in the service (Table 1 illustrate the    basic idea is specify in which situations the user is going to
      concept).                                                        interact with the service and the type or purpose of the
                                                                       interaction. E.g. the user being responsible for turning of
      Technical components
                                                                       false alarms could be a user component (or role), the user
                                                                       regularly using the service to check and self reflect on the
      A technical component can be purely hardware (e.g. a
                                                                       user’s own health condition is another example.
      camera), it can be purely software (e.g. a software module
      or an algorithm) or it can be a mixture (a smart sensor).
                                                                       Organizational support components

                                                                       This component addresses what kind of support the service
                                                                       requires from an organization or from other people
                                                                       connected to the service. Are the relatives going to check

                                         9                                                          10
EECS, University of California, Berkeley and Aarhus University                                                          August 2005

  the service once in a while? Do a team need to be working      3.3 Privacy
  24/7 to response to emergency calls or do a doctor have to
  spend time on looking on medical data once in a while?              A central concept dealing with assisted living at home is
                                                                      privacy. Some research projects aim at turning the elderly
  Business model components                                           person’s home into a glass house where relatives and
                                                                      medical professionals are able to monitor the elderly
  The final layer deals with how the service is going to be           person’s every movement.
  financed. Is the service offered as a subscription service,
  does the user have to pay each time the service is used, is         In an emergency situation most people will want to reveal
  the equipment rented or do you by cheap disposable                  as much information about their everyday life as possible,
  components?                                                         but that doesn’t mean that everything should be revealed in
                                                                      non-emergency situations.
  Table 1 shows an example of how services can be
  constructed with the above mentioned components.                    To deal with privacy we have emphasised the following
                                                                      design goals:

                                                                      • Decentralise computation: By letting the sensors handle
                                                                         most of the computation and only transmitting data when
                                                                         some interesting behaviour is discovered, we protect the
                                                                         users’ privacy, reduces bandwidth and enhance security,
                                                                         because data is not continuously transmitted.
                                                                      • Putting the user in control: In many proposed
                                                                         monitoring systems the elderly, the main user of the
                                                                         system, is often designed out of the loop. We want to
                                                                         make the user in charge of what information is
                                                                         transmitted, who is able to see these information and in
                                                                         what situations information is transmitted. We do not
                                                                         necessarily want the user to configure the system by
                                                                         them self, but it should be visible to the user how the
                                                                         system is configured.

                                     11                                                           12
      EECS, University of California, Berkeley and Aarhus University                                                                                                                                          August 2005

3.4 Envisioned System Architecture version 1.0                                                                                                             refrigerator, water, and bed sensors (for detecting
                                                                                                                                                           activities). Also user interface components for e.g.
            The following figure shows an overview of the envisioned                                                                                       helping the elderly in communicating with relatives or
            system architecture in the current phase of the project. The                                                                                   friends or for monitoring the person’s health condition is
            architecture is not novel in its current stage, but provide a                                                                                  also seen as part of the home area network.
            good starting point for building reusable components and in                                                                                 • Personal Home Health System: The home health
            exploring interactive services. The architecture consists of                                                                                   system is a central hop and a gateway. Health
            four main components.                                                                                                                          information is gathered in the home health system and
                                                                                                                                                           kept here. No information is transmitted from the health
Emergency Service
                                                                                                                                                           system unless the user has agreed to share some health
                                                                                                                                                           information with people or services outside the user’s
                                                                                                   Personal Area Network
                                                                                                                                                           home. And sharing of health information is seen as highly
                                                                                         Mobile Phone
                                                                                                                Bluetooth or
                                                                                                                                                           context dependent. E.g. sharing live video from the
     Hospital                                                 GSM/UMTS/3G                          Integrated
                                                                                                                  Zigbee       Small wireless sensors
                                                                                                                                e.g. Berkeley Motes        user’s home with relatives or emergency services might
                                              Personal Home
                                                                                                                                                           only be allowed if a serious condition e.g. a fall is
     Terminal, WLAN
                                              Health System
                                                                                                        Home Area Network                               • External Social and Service Network: The last group of
                                                                                         Novel User Terminals
                                                                                           designed for the              Wireless Cameras
                                                                                                                                                           components in the architecture is components used by
                                                                                                elderly                                                    people and services that get access to part of the user’s
Relatives                                                                                                                                                  health information in different situations. It can be
                                                                                                                                                           relatives that are notified if the medicine is not taken
                                                                               / WLAN
                                                                                                           Small wireless sensors                          regularly, a network of elderly that share health
                        Peer to Peer
                                                                                                                                                           information about a common illness, emergency services
                                                                                                            e.g. Berkeley Motes

                                                    District Nurses
                                                                                                                                                           that are notified in the case of a serious fall or a medical
 Other Elderly People                                                                                                                                      worker that regularly checks the user’s medical data to
                                                                                                                                                           look for abnormalities. However, the design space for
                                                    Health Information
                                                                                                                                                           connecting external networks is large and still mainly
                                                 consulting with the elderly

            Figure 1: Component based architecture

            • Personal Area Network (PAN): PAN describes a set of
               components that are closely connected to the user and
               follows the user wherever the user goes. PAN component
               will normally consists of at least one component that is
               able to communicate with the rest of the system and a
               larger set of smaller components that can communicate
               with this super node through some local connectivity
               channel. Examples of PAN component can be mobile
               phones (for communication), fall sensor (for detecting
               falls), EKG-sensors, implantable sensors etc.
            • Home Area Network (HAN): The home area network
               describes components that are connected to the home of
               the elderly person and will normally be fixed installations.
               E.g. Video components (to detect falls) and door,

                                                 13                                                                                                                                   14
    EECS, University of California, Berkeley and Aarhus University                                                              August 2005

4 Prototypes, Experiments and Results                                        nearby PC. The prototype consists of two components. The
                                                                             component on the phone is written in Symbian C++ and is
                                                                             able to control the integrated camera, establish a Bluetooth
      The following section will describe a set of prototypes
                                                                             connection to a nearby PC, compress the image data and
      developed and experiments carried out during my stay to
                                                                             transfer them over the Bluetooth Connection.
      explore or realise part of the ITS HealthNet described
                                                                             The other component runs on a nearby PC and is called
                                                                             BlueServer. It is a previously developed component for
4.1 Prototype one: Streaming video over BT from mobile
                                                                             connecting a PC to one or more Bluetooth devices. The
    phone to PC.                                                             BlueServer is written in C++ and uses the Microsoft
                                                                             Bluetooth Stack. That means that the PC running the server
4.1.1 Introduction
                                                                             needs a Bluetooth chip that is supported by the Microsoft
                                                                             Bluetooth Stack [PervasiveInteraction].
      The purpose of the first prototype was to transmit live video
      data from a user’s mobile phone to an external computer
                                                                       4.1.3 Results
      system. One scenario could be that a fall is detected and to
      help asses the situation live video images are transmitted
                                                                             We were able to build the system and it is able to stream
      from the mobile phone. Another scenario could be a
                                                                             video in a resolution of 320x240 or 160x120 without any
      teleconferencing situation where the mobile phones camera
                                                                             considerable lag. Sometimes the connection is lost due to an
      is used to record and transmit images of e.g. pictures of a
                                                                             error and we believe it is not connected to the program, but
      wound to a remote medical worker.
                                                                             to an error in the implementation of the Bluetooth stack on
                                                                             the phone for transmitting these large amounts of data.
4.1.2 The prototype
                                                                             Figure 2 shows a picture of the system in use.

      Figure 2: Stream video from mobile phone to PC over Bluetooth.

      Since no cellular network at the current point in time in the
      US is able to transmit live video we simulated the situation
      by using Bluetooth and connected the mobile phone to a

                                         15                                                              16
    EECS, University of California, Berkeley and Aarhus University                                                                      August 2005

4.2 Prototype two: Connect Fall Sensor to PC                                      BlueGiga module. The second program, FallGraph, was a C#
                                                                                  program that just displayed a graph of the accelerometer
4.2.1 Introduction                                                                data transmitted from the fall sensor.

      A fall sensor consisting of a board with a processor, a set of              The data path was that the fall sensor transmitted sensor
      accelerometers and a GPS unit was developed at Berkeley                     data through a RS-232 connection to the BlueGiga module
      (Figure 3 right). However, the previous way of getting data                 that again transmitted the data over Bluetooth to a nearby
      from the device was to use the device for a period of time                  PC running the BlueSerial. BlueSerial then again send the
      and then connect a cable to a nearby PC to download the                     data over a socket connection to the C# program that
      data. This process was both time consuming and there was                    displayed the data. Later on a MatLab program was
      a lack of correspondence between the data and the actual                    developed, that streamed the data directly into Matlab from
      event since the data was not accessible directly.                           BlueSerial and was especially used by some of the students
                                                                                  (see Section 4.7) for data analysis.
      In this project we want to stream the data from the fall
      sensor to a nearby PC directly to be able to better                   4.2.3 Results
      understand the data produced and to ease and speed up the
      data collection process.                                                    The program worked perfectly. It worked robust and was
                                                                                  able to display the fall sensor data directly on the PC as long
                                                                                  as the PC was close to the sensor. The range differed, but
                                                                                  normally it could at least keep the connection over 30 feet.
                                                                                  The tool was extended into the fall sensor recording tool
                                                                                  described in the next section.

      Figure 3: The Fall sensor (left) and the setup with the fall sensor
      connected to a Giga Bluetooth Cable Replacer

4.2.2 The prototype

      To build this prototype we bought a Bluetooth Cable
      Replacer Module from the Company BlueGiga [BlueGiga].
      The module is able to translate serial data into a Bluetooth
      connection. This module was attached to the fall sensor and
      the program on the fall sensor board was change to stream
      the data instead of caching it (Figure 3 left).

      Two programs were developed on the PC. The first program
      was a modified version of BlueServer called BlueSerial that
      instead of acting as a server listen for incoming data on a
      special virtual Bluetooth serial port connecting to the

                                         17                                                                     18
    EECS, University of California, Berkeley and Aarhus University                                                                August 2005

4.3 Prototype three: Fall sensor recording and analyzing
4.3.1 Introduction

      This prototype is mainly an extension of the previous
      prototype for connection the fall sensor to a nearby PC over
      Bluetooth. What we wanted to do was collect data, but also
      be able to have images connected to the data. This would
      allow us to look at the data and localize the reason for a
      specific data set.

4.3.2 The prototype

                                                                              Figure 5: The recorded data from Sonoma is analyzed and
                                                                              replayed with this MatLab program. The graph shows an elderly
                                                                              woman walking with a walker.

                                                                        4.3.3 Results

                                                                              The program was developed and worked perfectly. It was
      Figure 4: The data record prototype. The prototype recorded             used for over an hour in the data collection session in
      synchronized images and fall sensor data and stored the data on
      the hard drive.                                                         Sonoma without any problems and used to collect data
                                                                              locally by students.
      The prototype was an extension of the previous FallGraph
      prototype. The new RecordTool program was connected to a
      webcam, and with the webcam we were able to store
      synchronized images together with the fall sensor data on
      the desk (Figure 4).

      Together with the recording tool a graphical Matlab program
      was developed (by Mike Eklund) for analyzing and replaying
      the recorded data (Figure 5).

                                         19                                                                20
    EECS, University of California, Berkeley and Aarhus University                                                              August 2005

4.4 Prototype four: Streaming Fall Sensor Data to Phone               4.5 Prototype five: Fall Detection Algorithm and System
4.4.1 Introduction                                                    4.5.1 Introduction

      The setup where the fall sensor streams data to a PC is               In this prototype we wanted to build a prototype that could
      good for testing purposes, but in a real life situation the           work as a first proof of concept of the entire system. We
      elderly person is not always within range of a computer.              wanted to implement a fall detection algorithm running on
      Therefore we wanted to connect the fall sensor to a type of           the fall sensor and then letting the fall sensor actively
      device we expect most elderly people will own in the future,          initiate a Bluetooth Connection if and only if a fall is
      a mobile phone.                                                       detected. The PC system will in the case of a fall send out
                                                                            an alarm and start transmitting live video from the scene.
4.4.2 Prototype
                                                                      4.5.2 Prototype
      The prototype consists of a program running on mobile
      phones written in Symbian C++ that is able to connect to              The prototype system used a specialized program developed
      the fall sensor and draw a graph of the data stream                   for the fall sensor that was able to communicate with the
      received.                                                             BlueGiga Bluetooth module through simple commands. The
                                                                            PC was running the BlueServer and a C# program that
4.4.3 Results                                                               started an alarm and video stream if it received an alarm
                                                                            from the sensor.
      The described prototype is developed and tested and works.
      There is still more work to be done with this prototype. E.g.   4.5.3 Results
      if a fall is detected we want to ask the user if s/he is okay
      and we also want to be able to send alarm signals over the            The prototype was developed and seems to work. However,
      cellular network to emergency personals.                              there are still some open issues. At the moment the fall
                                                                            sensor is only transmitting data and does not receive any
                                                                            data back. That can be a problem since the fall sensor is not
                                                                            informed about the status of the Bluetooth connection. Was
                                                                            it successfully established or should it retry sending the
                                                                            alarm signal? In a further version it could be beneficial to
                                                                            put the Bluetooth radio directly on or closer to the fall
                                                                            sensor board.

                                         21                                                              22
    EECS, University of California, Berkeley and Aarhus University                                                            August 2005

4.6 Data collection: Sonoma                                               To do the data collection the recording tool was used and we
                                                                          were able to capture over an hour of picture and fall sensor
4.6.1 Introduction                                                        data with three elderly persons doing daily activities e.g.
                                                                          walking with walker, sitting in soft chair, showing pictures,
      The purpose of this data collection session was to get some         running to get a phone, stretching, making coffee, walking
      data from a set of different elderly persons doing normal           outside, and telling stories (See Figure 6 and Figure 7).
      activities. The data collect session was done at a facility in
      Sonoma, where a group of elder people live in their own or
      shared homes.

                                                                          Figure 7: The recorded data from Sonoma is analyzed and
                                                                          replayed with this MatLab program. The graph shows an elderly
      Figure 6: Examples of pictures recorded of a person sitting down.
                                                                          woman walking with a walker (top right), a woman sitting down
      The first number is the number of the test person, the second
                                                                          (top left), a woman standing up (bottom right) and a man
      number is the current data series and the last number is for
                                                                          stretching his arm (bottom left)
      synchronizing with the sensor data.

                                         23                                                            24
    EECS, University of California, Berkeley and Aarhus University                                                                  August 2005

4.6.2 Results                                                           4.7 Data collection: Systematic fall data gather by students
      We collected a total amount of 1.5 gb of image and sensor         4.7.1 Introduction
      data from Sonoma and the data was analyzed using MatLab.
      Figure 7 shows some selected situations with some of the                One of the lessons learned from the Sonoma data collection
      participants. Since the purpose of the data collection mainly           trip was that to be able to test and validate an algorithm for
      was to get an understanding of how the sensor behaved in                a fall sensor we needed to come up with a protocol for doing
      different situations we could not directly use the data to              systematic testing. First, we needed to build a calibration
      draw any conclusions. The main purpose of the data                      routine since the sensor data drifted a bit as the battery
      collection was more to guide the future development of a                power decreased. Second, we needed to have a test set of
      fall detection algorithm.                                               different falls and non fall scenarios we could test the
                                                                              algorithm against.
      Based on the Sonoma data set we have developed an
      algorithm for detection fall (mainly the work of the              4.7.2 Results
      undergraduate and superb students). We have also worked
      on specifying a set of positive and negative examples of falls          A simple algorithm was developed that ran on the fall sensor
      future version of the fall sensor is going to be tested against         and search for high acceleration peaks. If a high
      (see the following section).                                            acceleration peak occurred the position of the device was
                                                                              queried to see if the person was lying down and not moving.

                                                                              Falls                                    Non Falls
                                                                              1 Fall forward from standing             86 Stair descending walking
                                                                              2 Fall backward from standing            87 Stair climbing jogging
                                                                              3 Fall leftward from standing            88 Stair descending jogging
                                                                              35 Fall leftward onto object             89 Sitting down slow onto a
                                                                              36 Fall rightward onto object            chair
                                                                              37 Fall forward after bouncing off a     90 Sitting down hard onto a
                                                                              small object                             chair
                                                                              55 Fall leftward going upstairs
                                                                              56 Fall rightward going upstairs
                                                                              57 Fall forward going downstairs
                                                                              Table 2: Types of falls

                                                                              Two students came up with a list of 99 fall types and some
                                                                              activities related to falls. Table 1 shows some of the fall and
                                                                              non-fall types. The two students did also perform all 99 falls
                                                                              and activities with the fall sensor and recorded the data and
                                                                              noticed the performance of the algorithm developed.

                                                                              The conclusion from this study was that the simple
                                                                              algorithm worked pretty well in many of the fall performed.
                                                                              One of the biggest problems with the algorithm was
                                                                              problems with detection falls where the user is not ending
                                                                              up lying flat down (e.g. when falling and landing on stairs).

                                         25                                                                 26
    EECS, University of California, Berkeley and Aarhus University                                                                August 2005

5 Future Work                                                                To do this extensive data collection a new version of the fall
                                                                             sensor might need to be developed to facilitate the
5.1 Designing the Humans into the loop                                       experiments. Experiments need to be conducted both with
                                                                             elderly test subjects, but also performing similar experiment
      Many of the proposed research projects addressing the                  with students should generate useful data.
      aging population views the elderly person as an unreliable
      and “mentally weak” person that need to be designed out of        5.3 IT Supported HealthNet
      the loop. It is the intelligent house that needs to figure out
      what the person is doing. Maybe it is even going to be a               In the project we have currently focused on the fall sensor
      sport amongst elderly people in the future to actually try to          however, the goal of ITS HealthNet is to build an
      fool your own house – e.g. what happens if you turn on the             infrastructure and some reusable components that fits many
      shower, open the front door and go into the kitchen and                different settings and applications. However, to truly be able
      turn on the oven – what will the sensors or the neural                 to build flexible components we need to work with several
      network behind the sensors think you are doing?                        types of sensors, applications and settings.

      It is true that some elderly people might lack the focus or            From a technical perspective one consideration is the type of
      the technical skills of younger people, but that is not an             sensor network deployed. Should it be a decentralised
      argument for designing them out of the loop. Instead they              architecture where a large part of the computation is
      should be viewed as the most valuable resource and the                 distributed to the sensors? The advantage of this approach
      technology should be designed to fit their skills and needs. A         is that the “smart” sensors are loosely coupled from the rest
      starting point can be to study the use and skills of IT                of the system, the disadvantage is that the sensors are
      amongst elderly people and use their stories and skills as             going to be more power consuming, more expensive and
      starting point for new health services instead of purely               that a single sensor might not by itself have enough
      addressing the research area from a technical perspective.             information to do the reasoning.

5.2 Fall Sensor version 2.0                                                  The advantage of a central sensor architecture is that the
                                                                             sensor can be cheap and simple and that input from several
      As described in the above section we have for some time                sensors can be used in the reasoning process. The
      been working with the fall sensor in its current version,              disadvantage is a tight coupling between the sensors and
      collected data and explored different algorithms. However, a           the central server.
      fall sensor is going to run every day, analyzing the huge
                                                                             To determine the best suited architecture we need a better
      variety of daily activities to look for sensor data that might
                                                                             understanding of the healthcare needed by elderly people.
      indicate a fall. A fall might only last for a few seconds and a
                                                                             Preventing and detecting falls is a good research goal, but
      fall can happens in a number of ways. A basic research
                                                                             we need to find more similar healthcare goal to drive the
      question is if a fall sensor based on accelerometers is able
                                                                             technical development. Especially a continues dialog with
      to accurately distinguish daily activities from falls.
                                                                             care givers and healthcare professionals can be beneficial in
      A method for addressing this question is to collect data               further identifying both healthcare challenges and assistive
      about daily activities and search for examples of sensor data          needs within the elderly population.
      that might resemble a fall, but that is not a fall. This could
      be done by running a simple algorithm on the sensor that is
      able to identify fall like data sequences and store or transmit
      these sensor data for further analysis.

                                         27                                                               28
   EECS, University of California, Berkeley and Aarhus University                                                           August 2005

5.4 Concepts, Theories and Frameworks                               6 Talks, Classes and Supervision
     Even though a lot of research interest is focused on the       6.1 Talks
     growing elderly population few concepts, theories and
     frameworks exists for understanding how to build                    I have given two talks at the Berkeley campus.
     technology for elderly users. And that applies on all levels
     from developing technical algorithms and components to         6.1.1 Programming mobile phones, CHESS seminar talk, May 18,
     understanding how the new IT-enabled group of elderly                2005.
     users are going to use technology. The coming generation of
     elderly people will have used a computer most of their              This talk was about how to program mobile phones and
     working life, will be checking email regularly, have a              current trends in the field of programming mobile devices.
     personal web page and send SMS messages from their                  The slides are available at:
     phones. However, how to design IT technology for this IT- 
     enabled elderly generation is largely unexplored.
                                                                    6.1.2 Getting started with programming for Symbian OS in C++,
                                                                          full day seminar May 31, 2005

                                                                         The purpose of the seminar was to introduce the
                                                                         participants to programming for mobile phones in Symbian
                                                                         OS C++ and give the participants an introduction to the
                                                                         basic concepts and hands on experience with developing
                                                                         code for this platform. The course material is available at:

                                                                    6.2 Classes

                                                                         During my stay I have participated in the following classes
                                                                         or seminar. I have however, not taken the exams in any of
                                                                         the classes.

                                                                         Hybrid        System         by       Shankar      Sastry

                                                                         Information Technology in Medical Research and Health
                                                                         Care by Ruzena Bajcsy and Mike Eklund

                                                                         BioHybridControl Group – Hybrid Systems Theory for
                                                                         Biology by Alessandro Abate

                                                                         Intel Research, Berkeley Seminars

                                        29                                                           30
   EECS, University of California, Berkeley and Aarhus University                                                               August 2005

6.3 Supervision                                                     7 Other Collaborative Projects
     During my stay I have had the opportunity (together with            Besides working on the ITALH project I have started smaller
     Mike Eklund and Ruzena Bajcsy) to supervise two                     projects or discussed potential projects with people from the
     undergraduate (Rustom Jamsheed Dessai and Albert Chang)             Robotics Research Lab at Berkeley.
     and one Superb student in their work with the fall sensor
     system during the summer 2005 (Garrett Bruhn).                 7.1 Mobile video conferencing between multiple participants
     Beside the three summer student I have also been working
                                                                         (With Marci Meingast)
     with a Berkeley undergraduate student (Adeeti Ullal) who is
     working on developing a J2ME interface running on mobile            The challenge in this project is to work with video
     devices to remote medical systems.                                  conferencing technologies on mobile devices between
                                                                         multiple participants. The challenges in the project is both to
                                                                         come up with compression schemes for transmitting video
                                                                         of a person’s face on limited bandwidth and to design a
                                                                         human computer interaction scheme for these kinds of

                                                                         Figure 8: A prototype running on the phone for viewing multiple
                                                                         participants. The interaction is purely done with face tracking.

                                                                         We have currently implemented a simple prototype where
                                                                         face tracking is used to navigate the view of a group of
                                                                         participants (Figure 7Figure 8). We have also worked on
                                                                         improving the face tracking algorithm by e.g. locating the
                                                                         position of the eyes of a user and other features. A
                                                                         prototype is currently running on the PC for the eye
                                                                         detection, but more work on the prototype needs to be done
                                                                         before the algorithm is ported to the phone.

                                        31                                                             32
   EECS, University of California, Berkeley and Aarhus University                                                           August 2005

7.2 Computer vision on mobile devices                               7.3 Controlling UAVs (unmanned air vehicles) with mobile
     (With Parvez Ahammad)
                                                                         (With David Hyunchul Shim)
     In this project we have worked with cameras on mobile
     devices. We have looked at how mobile cameras can be                In this project a mobile phone is used to control already
     used to develop novel application e.g. for handicapped              existing UAVs and possibly display images from the plane on
     people.                                                             the screen of the mobile phone.

                                                                         We have discussed the project and it seems pretty straight
                                                                         forward to build an initial prototype, but no system has yet
                                                                         been build.

                                                                    7.4 Using Mobile Phones to estimate traffic on highways

                                                                         (With Alexandre M. Bayen)

                                                                         The idea about the project was to use peoples mobile
                                                                         phones to either actively report their position on the
                                                                         highway or inactively scan for the presence of mobile
                                                                         phones on highways to estimate the traffic.

                                                                         We have discussed the project and some of its implications,
                                                                         but have not started the project.

     Figure 9: Corner detection algorithm running on a PC with a

     We have currently implemented a corner detection
     algorithm in C# using a webcam and are discussing how to
     improve the algorithm before porting the project to the
     mobile phone (Figure 9 shows a screen shot of the

                                        33                                                            34
   EECS, University of California, Berkeley and Aarhus University                                                                August 2005

7.5 USB / Mobile Phone                                                8 Activities related           to    previous       work       and
     (with Mike Manzo, Mike Eklund)

     We have been working with researchers at Nokia / Intel on        8.1 Presentations at CHI 2005
     connecting Berkeley Motes to mobile phones. Currently we
     are working with two parallel solutions.                              Participated in the CHI 2005 (Conference On Human Factors
                                                                           In Computing Systems) in Portland where we presented our
     The first solution is directly to attach an USB device (e.g.          short paper on the MIXIS system and presented a workshop
     the Berkeley mote) to the mobile phone. The problem with              paper focusing on awareness [PervasiveInteraction].
     attaching USB devices directly is that mobile phones (at the
     current point in time) only act as USB slaves and not as USB     8.2 Mixis Face tracking
     masters. We have received a device from Nokia research
     that allows us to connect USB devices directly to the phone.
     We are however still working on getting it to work in our lab
     (Figure 10, left show the Nokia USB box).

                                                                           Figure 11: A pong demonstration application that uses face
                                                                           tracking (right) and the Nokia 6680 phone running a map
                                                                           navigation application that uses face tracking as input (left).

                                                                           I have developed the project Mixis Face Tracking that
                                                                           extends previous work focusing on interaction on mobile
                                                                           devices. It runs an algorithm for doing face tracking on a
                                                                           mobile phone with dual cameras and uses the location of the
     Figure 10: The Nokia USB to phone box (left) and Intel’s IMote
     that supports Bluetooth (right).
                                                                           face as input vector to a set of applications (See Figure 11).
     The second solution is to use a hub mote that supports both
     Bluetooth and Zigbee. We have talked with researchers at
     Intel and apparently their new iMote 2.0 should support
     both standards however, we have not been able to receive
     this device yet (Figure 10 right is iMote one, Bluetooth
     enabled, but not Zigbee enabled).

                                        35                                                                36
   EECS, University of California, Berkeley and Aarhus University                                                              August 2005

8.3 Publications written during my stay                             9 References
     Eklund, J. Mikael, Hansen, Thomas Riisgaard, Sprinkle,              Most of the reference is cited in the report, but some extra
     Jonathan and Sastry, Shankar, Information Technology for            references to related research projects might be present.
     Assisted Living at Home: building a wireless infrastructure
     for assisted living, paper accepted at 27th Annual             9.1 Papers
     International Conference of the IEEE Enginerring In Medicine
     and Biology Society (EMBS).                                         [Clemensen] Clemensen, J., S. B. Larsen and J. Bardram:
                                                                         Developing Pervasive e-Health for Moving Experts from
     (… Hansen, Thomas Riisgaard …) Using smart sensors and a
                                                                         Hospital to Home. Proceedings of the IADIS e-Society
     camera phone to detect and verify the fall of elderly
                                                                         Conference, pp.441-448, Avilla, Spain, 2004.
     persons, accepted based on abstract to EMBEC’05 3rd
     European Medical & Biological Engineering Conference.               [Consolvo] Consolvo, S., Roessler, P., Shelton, B., LaMarca,
                                                                         A., Schilit, B., Bly, S., Technology for Care Networks of
     (… Hansen, Thomas Riisgaard …) A decentralised
                                                                         Elders, Pervasive Computing vol. 3 nr. 2 , IEEE , pages 22-
     information and communication technology architecture,
                                                                         29, 2004.
     accepted based on abstract as poster to EMBEC’05 3rd
     European Medical & Biological Engineering Conference.               [Hauptmann] Hauptmann, A., Gao, J., Yan, R., Qi, Y., Yang,
                                                                         J., Wactlar, H., Automated Analysis of Nursing Home
     Hansen, Thomas Riisgaard, Eriksson Eva, Lykke-Olesen,
                                                                         Observations, Pervasive Computing vol. 3 nr. 2 , IEEE ,
     Andreas, Mixed Interaction Spaces – a new interaction
                                                                         pages 22-29, 2004.
     technique for mobile devices, demonstration accepted at the
     UbiComp conference 2005.                                            [Lo] Lo, B., Yang, G., Key Technical Challenges and Current
                                                                         Implementations of Body Sensor Networks, The 2nd
     Hansen,     Thomas   Riisgaard,   Bardram,    Jakob   E.,
                                                                         International Workshop on Body Sensor Networks (BSN
     ActiveTheatre – a Collaborative, Event-based Capture and
                                                                         2005) , pages xx-xx, 2005.
     Access System for the Operating Theatre, accepted as full
     paper at UbiComp 2005 (camera ready paper prepared                  [Ross] Ross, Philip E., Managing Care Through the Air,
     while in Berkeley).                                       
     Hansen, Thomas Riisgaard, Interaction with multiple devices
     in hospitals, Workshop paper accepted to the Spaces in-             [Rowan] Rowan, Jim, Mynatt, Elizabeth D., Digital Family
     between: Seamful vs. Seamless Interactions workshop at              Portrait Field Trial: Support for Aging in Place, Proceedings
     UbiComp 2005.                                                       at CHI, ACM Press, 2005.
     Hansen, Thomas Riisgaard, Eriksson Eva, Lykke-Olesen,               [Sixsmith] Sixsmith, A., Johnson, N., A Smart Sensor to
     Andreas, Movement and Space – Exploring the Space in                Detect the Falls of the Elderly, Pervasive Computing vol. 3
     Movement based Interaction, Workshop Paper accepted at              nr. 2 , IEEE, pages 42-47, 2004.
     Approaches to Movement-based Interaction at the Aarhus
     2005 conference, Critical Computing.                                [Yang] Yang, G., Lo, B., Wang, J., Rans, M., Thiemjarus, S.,
                                                                         Ng, J., Garner, P., Brown, S., Majeed, B., Neild, I., From
                                                                         Sensor Networks to Behaviour Profiling: A Homecare
                                                                         Perspective of Intelligent Building, The IEE Seminar for
                                                                         Intelligent Buildings , pages xx-xx, 2004.

                                        37                                                             38
    EECS, University of California, Berkeley and Aarhus University                                                           August 2005

9.2 Collection of elderly related research projects                       [GE HAS] General Electric Global Research: Home Assurance
      [Cast] CAST - Center for aging services technologies      
                                                                          [ProHealth] Intel Research, ProHelath Research Project
      [PervasiveComputing] IEEE Pervasive Computing Vol 3, Nr.  
      2 - April-June 2004. Special Issue: Successful Aging:            [IBMZurich] IBM Research, Mobile Health Toolkit

                                                                          [Phillips] Philips Research, Personal Healthcare Systems
9.3 University Research Projects                                          ml

      [AwareHome] Georgia Tech: AwareHome                                 [HomMed] Honeywell HomMed                          

      [CareMedia] Carnegie Mellon University, CareMedia              9.5 Other References
                                                                          [PervasiveHealthcare] Pervasive Healthcare
      [CodeBlue] Harvard University: Sensor Network for         
      Emergency                     [PervasiveInteraction] Pervasive Interaction
      [Intille] MIT, Stephen S. Intille                                  [BlueGiga]
      [Nighingale] University of Sidney, Project Nightingale

      [PlaceLab] MIT, Placelab

      [Rochester] University of Rochester

      [UbiMon] Imperial College London, Ubiquitous Monitoring
      Environment for Wearable and Implantable Sensors

9.4 Research Project within large companies

      [HouseCare] British Telecom, House Care

                                         39                                                            40

To top