Defesa Proposta tese ffc CIn Centro de Informtica da UFPE

Document Sample
Defesa Proposta tese ffc CIn Centro de Informtica da UFPE Powered By Docstoc
					A Embedded software
 component quality
    framework

     Fernando Ferreira de Carvalho
      Advisor: Silvio Romero de Lemos Meira

    Informatics Center - Federal University of Pernambuco
 C.E.S.A.R. - Recife Center for Advanced Studies and Systems
                        ffc@cin.ufpe.br


                12-Junho-2008
                                                           A Embedded software
                                                           component quality framework

Introduction / Motivation

Embedded system is at the heart of many systems
  So, embedded system industry needs,
  • Low production cost
  • Short time to market
  • High Quality
                      to be more efficient and competitive (Brown, 2000)

  The CBD with reuse technique had been a nice direction to reach this
    objectives…

  But, Component reuse without quality assurance give catastrophic
    results [ariane].




                             http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                      A Embedded software
                                                      component quality framework

Introduction / Motivation


  The CBD technologies do not take into
  account the specifics needs of embedded-
  systems development: Timing, Memory,
  power, hardware constraints and others.




     SAAP 2008.1        http://www.rise.com.br
                   Fernando Carvalho, PhD Candidate
                                                        A Embedded software
                                                        component quality framework

Introduction /


• Solution
  o   Certification is the future of software
      components
      [Wallnau, Heineman, Councill, Shaw]

  o   According to Weber et al. (Weber et al., 2002),
      the need for quality assurance in software
      development has exponentially increased in the
      past few years


       SAAP 2008.1        http://www.rise.com.br
                     Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework

Introduction / Motivation
• Benefits
  o   More reliability
  o   Short time-to-market;
  o   More product quality;
  o   higher quality levels,
  o   reduced maintenance time,
  o   investment return,
  o   reduced time-to-market, among others
  o   Wide used, where certification is mandatory (aircraft)




       SAAP 2008.1           http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                     A Embedded software
                                                     component quality framework

Rise Framework




    SAAP 2008.1        http://www.rise.com.br
                  Fernando Carvalho, PhD Candidate
                                                      A Embedded software
                                                      component quality framework

Component Certification Process




     SAAP 2008.1        http://www.rise.com.br
                   Fernando Carvalho, PhD Candidate
                                                       A Embedded software
                                                       component quality framework

Component Certification Process
focused in embedded systems


  An Embedded Computer System: A computer system that
  is part of a larger system and performs some of the
  requirements of that system; for example, a computer
  system used in an aircraft or rapid transit system.
  (IEEE,1992).




      SAAP 2008.1        http://www.rise.com.br
                    Fernando Carvalho, PhD Candidate
                                                               A Embedded software
                                                               component quality framework

Component Certification Process
focused in embedded systems


    Embedded systems :
     •     Used for specifics propose
     •     Used to control systems (ex: mechanical machines)
     •     Ultra small devices with simple specific functionality
     •     Small systems with sophisticated functions
     •     Produced in large scale




         SAAP 2008.1             http://www.rise.com.br
                            Fernando Carvalho, PhD Candidate
                                                          A Embedded software
                                                          component quality framework

Component Certification Process
focused in embedded systems


    Embedded systems has a specific requirements:
    - real-time
    - hi reliability
    - low power consumption
    - low data and code memory
    - low resources
    - low CPU capabilities
    - others




     SAAP 2008.1            http://www.rise.com.br
                       Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


Problem Formulation
    The CBD is an efficient and effective way for design of simple
    and complex embedded systems.

However, quality assurance of components is must be done to
take advantage of CBD.
        The Software industry still far to reach the maturity level the
        hardware industry which it have catalogues and datasheets
        available for its components.

For this reason, a well-defined and consistent embedded software
component quality assurance is essential for CBD and reuse adoption.


        SAAP 2008.1             http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


Proposed solution
 An Embedded Software Component Quality Verification Framework

It is composed of four inter-relation module:, based on a set of
activities, metrics and guidelines.
        • Embedded software component Quality Model (EQM)

        • Maturity Level evaluation Techniques

        • Metrics Approach

        • Component Certification

 based on a set of activities, metrics and guidelines.

         SAAP 2008.1            http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                                A Embedded software
                                                                component quality framework


Proposed solution
 This Framework is based in the standards

 • ISO/IEC 9126, 2001 - Quality Model for Software Product

 • ISO/IEC 14598, 1998 - Software Product Evaluation Process

                      This two standards converged to:

        •     ISO/IEC 25010, 2005 - Software product quality -
            requirements                              and evaluation

        the Framework adapted the quality model and evaluation to
          component context and embedded domain.
        SAAP 2008.1               http://www.rise.com.br
                             Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework


Out of scope
  This Framework is part of broad context, some aspects were
  expected since initial definition. Nevertheless, other process
  can be added in the future.



      •Cost Model

      •Formal Proof

      •Prediction of the component assembly




      SAAP 2008.1            http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


2 – Embedded System Design
Embedded system design comprise:



• Ultra-small device x simple functionality

• Small system x sophisticated functions

• Large systems and distributed systems

• Systems produced in large quantities x low production cost

• Systems produced in low volume x important features


         SAAP 2008.1            http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


2 – Embedded System Design
 The different requirements of embedded systems have a impact on
feasibility, on use of CBD for it.

 A common characteristic in different area of embedded domain
 is increasing importance of software [Crnkovic, 2003].


Example, the software cost in embedded systems:
     • in industrial robots constitute about 75% of total cots
     • in car industry it is about 30%
Fifteen year ago:
    • 25% of total cots in industrial robots
    • Negligible for cars


        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


2 – Embedded System Design
Properties that involves embedded software component is divided in:

• Functional property (component interface)

• Non-functional or Extra-functional property, so called Quality
  attributes, fox example:
       • Timing
       • Performance
       • Consumption
       • Resource Behavior, and others.


     This properties can be classified in run-time and life-cycle.


       SAAP 2008.1             http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


2 – Specific Requirements for
Embedded System
In the most of case, embedded system is real-time with limited
resource. So, it has specifics characteristics which depends on domain
application, but it have strong implication on requirements.
       The REQUIREMENTS are related Extra-functional
       property or Quality attributes, and its priority depends on
       the domain application.




        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


2 – Specific Requirements for
Embedded System
 There has been developed a research in order to find the most
 important characteristics in different areas in embedded
 domain.

       • Industrial Automation
       • Automotive
       • Medical
       • Electronic consumer
       • Other domain



        SAAP 2008.1           http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


2 – Specific Requirements for
Embedded System – Industrial Automation
 Industrial Automation was classified by research’s Larsson,
 [Larsson, 2002]

       The most important characteristics, following the research:

   At low level:          At high level:
        a. Availability       a. Performance
        b. Timeliness         b. Usability
        c. Reliability        c. Integrability



        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                                  A Embedded software
                                                                  component quality framework


2 – Specific Requirements for
Embedded System - Automotive
 Akerholm [Akerholm, 2005] executed a research in vehicle industry.
 The resulting list of characteristics
    is presented below
             1.    Safety
             2.    Reliability
             3.    Predictability
             4.    Usability
             5.    Extendibility
             6.    Maintainability
             7.    Efficiency
             8.    Testability
             9.    Security
             10.   Flexibility
        SAAP 2008.1                 http://www.rise.com.br
                               Fernando Carvalho, PhD Candidate
                                                                 A Embedded software
                                                                 component quality framework


2 – Specific Requirements for
Embedded System - Medical
 Wijnstra [Wijnstra, 2001] describe their experience with
 characteristics in the development of medical imaging family.
  The resulting list of characteristics is presented below
                       1.   Reliability
                       2.   Safety
                       3.   Functionality
                       4.   Portability
                       5.   Modifiability
                            a. Configurability
                            b. Extensibility and Evolvability
                       6.   Security
                       7.   Serviceability

         SAAP 2008.1               http://www.rise.com.br
                              Fernando Carvalho, PhD Candidate
                                                                A Embedded software
                                                                component quality framework


2 – Specific Requirements for
Embedded System – Others Domain
                                                                 Characteristics         Sub-characteristics
 Crnkovic [Crnkovic, 2003] summarized
                                                              Real-time properties    Response time or latency
 the main characteristics and sub-                                                    execution time
 characteristics in the CBD approach apply                                            worst case execution time
 to embedded system in his research.
                                                                                      Deadline
 The table show the results.                                  Dependability           Reliability
                                                                                      Availability
                                                                                      integrity
                                                                                      confidentiality
                                                                                      safety
                                                              Resource consumption    Power consumption
                                                                                      computation (CPU) power
                                                                                      memory Consumption
                                                                                      execution (CPU) time,
                                                              Life cycle properties   maintainability
        SAAP 2008.1             http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


2 – Embedded System Design –
    Software component quality
So, embedded software component quality verification must be
different that general propose component, because the component
evaluation is realized focused in specifics requirements
We divided the quality verification in two groups:
 • General propose software component quality process
     o desktops, servers, x86 architecture

 • Specific propose software component quality process
    o   embedded systems




         SAAP 2008.1            http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 The relevant research explore the theory of component quality and
 certification in academic scenarios, but not rich in reports in practical
 experience.


  The pioneering works focus in mathematical and test model, while
  recent researchers have focused in techniques and model based on
  predicting quality requirements.




         SAAP 2008.1            http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                                  A Embedded software
                                                                  component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Timeline of research in the embedded software component quality and
 certification area




   proposed standard
 X fail
 → a work was extended by another

          SAAP 2008.1               http://www.rise.com.br
                               Fernando Carvalho, PhD Candidate
                                                               A Embedded software
                                                               component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 In 1993, Poore [Poore et al., 1993] develop an approach based on
 three mathematical model (sampling, component and certification
 models), using test cases to report the failures to achieve a reliability
 index


  Poore estimated the reliability of a complete system, and not of
  individual software units, although, they did consider how each
  component affected the system’s reliability.



         SAAP 2008.1             http://www.rise.com.br
                            Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Wohlin [Wohlin et al., 1994] presented the first method of component
 certification using modeling techniques, making it possible not only to
 certify components but to certify the system.
       •It is composed of the usage model and the usage profile.
       •The failure statistics from the usage test form the input of a
       certification model.
       •An interesting point of this approach is that the usage and profile
       models can be reused in subsequent certifications


 However, even reusing those models, the considerable amount of effort
 and time that is needed makes the certification process a hard task.
         SAAP 2008.1            http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                                A Embedded software
                                                                component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 In 1994, Merrit (Merrit, 1994) presented an interesting suggestion: the
 use of components certification levels. These levels depend on the
 nature, frequency, reuse and importance, as follows:
  • Level 1: No tests are performed; the degree of completeness is unknown;
  • Level 2: A source code component must be compiled and metrics are determined;
  • Level 3: Testing, test data, and test results are added; and
  • Level 4: A reuse manual is added.

      These levels represent an initial component maturity model.
 However, this is just a suggestion of certification levels and no practical
 work was actually done to evaluate it.
         SAAP 2008.1              http://www.rise.com.br
                             Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
In 1996, Rohde (Rohde et al., 1996) provided a reuse and certification of
embedded software components at Rome Laboratory of the US Air
Force, a Certification Framework (CF), that included:

 • To define the elements of the reuse context that to certification;
 • To define the underlying models and methods of certification; and,
 • To define a decision-support technique to construct a context-sensitive
 process for selecting and applying the techniques and tools to certify
 components.



         SAAP 2008.1            http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 • A Cost/Benefit plan that describes a systematic approach of evaluating
 the costs and benefits.


 Rohde et al. considered only the test techniques to obtain the defects
 result in order to certify software components. This is only one of the
 important techniques that should be applied to component certification.




        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                               A Embedded software
                                                               component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Voas [Voas, 1998] defined a certification methodology using automated
 technologies, such as black-box testing and fault injection to determine
 if a component fits into a specific scenario.
         This methodology uses three quality assessment techniques:
 (i) Black-box component testing determine if the component quality is
 high enough;
 (ii) System-level fault injection determine how well a system will tolerate a
 faulty component;
 (iii) Operational system testing determine how well the system will
 tolerate a properly functioning component


         SAAP 2008.1             http://www.rise.com.br
                            Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 According to Voas, this approach is not foolproof and perhaps not well
 suited to all situations. The methodology does not certify that a
 component can be used in all systems. This approach certify a
 component within a specific system and environment.




        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                               A Embedded software
                                                               component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Wohlin and Regnell [Wohlin and Regnell, 1998] extended their previous
 research (Wohlin et al., 1994), now, focusing on techniques for certifying
 both components and systems.
 Thus, the certification process includes :

 (i) usage specification (consisting of a usage model and profiles), and

 (ii) certification procedure, using a reliability model.




         SAAP 2008.1             http://www.rise.com.br
                            Fernando Carvalho, PhD Candidate
                                                               A Embedded software
                                                               component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 The main contribution of that work is the division of components into
 classes for certification and the identification of three different ways of
 certifying software systems:

            i. Certification process, the functional requirements are
                  validated    during usage-based testing;
            ii. Reliability certification of component and systems, the
                  component             models that were built are revised
                  and integrated to certify the       system that they form;
                  and,
            iii. Certify or derive system reliability, where the focus is on
                  reusing the models that were built to certify new
                                   systems.
                  components orhttp://www.rise.com.br
         SAAP 2008.1
                            Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
         However, the proposed methods are theoretical without
           experimental study. According to Wohlin et al., “both
           experiments in a laboratory environment and industrial
           case studies are needed to facilitate the understanding of
           component reliability, its relationship to system reliability
           and to validate the methods that were used only in
           laboratory case studies” (pp. 09).

         Until now, no progress in those directions was achieved.


     SAAP 2008.1             http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
             In 2000, Jahnke, Niere and Wadsack [Jahnke, Niere and
             Wadsack, 2000] developed a methodology for semi-
             automatic analysis of embedded software component
             quality.
                     This approach evaluates data memory (RAM)
             utilization in Java technology by the component.
                     The work is restricted because:
                     - Verifies the component quality from only one point
             of view, use of data memory in a specific language,
                     - Java is widely used for the development of
             desktops systems not          useful for embedded
             development.
     SAAP 2008.1              http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
  Stafford (Stafford et al., 2001) developed a model for the component
  marketplaces that supports prediction of system properties prior to
  component selection.
       The model use functional verification and quality-related
       values associated with a component, called credentials.
          This work introduced notable changes in this area.

   It use a specific notation such as <property,value,credibility>.
   Through credentials, the developer chooses the best components to
   use in the application development based on the “credibility” level.

      SAAP 2008.1            http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Stafford introduced the notion of active component dossier, its is an
     abstract component that defines credentials.
 Stafford et al. finalized their work with some open questions, such as:

    •   how to certify measurement techniques?
    •   What level of trust is required under different circumstances?
    •   Are there other mechanisms that might be used to support
        trust?




        SAAP 2008.1           http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                                  A Embedded software
                                                                  component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
     In 2002, Comella-Dorda et al. (Comella-Dorda et al., 2002)
     proposed a COTS software product evaluation process. The process
     contains four activities, as follows:
i.   Planning the evaluation -> evaluation team, stakeholders, required resources,
     basic characteristics of the evaluation
ii. Establishing the criteria -> evaluation requirements , evaluation criteria;
iii. Collecting the data -> component data are collected, the evaluations plan is done
     and the evaluation is executed; and
iv. Analyzing the data -> the results of the evaluation are analyzed and some
     recommendations are given.

     The proposed process is an ongoing work and, no real case study was
     accomplished, becoming unknown the real efficiency.
          SAAP 2008.1               http://www.rise.com.br
                               Fernando Carvalho, PhD Candidate
                                                                  A Embedded software
                                                                  component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
    In 2003, Beus-Dukic (Beus-Dukic et al., 2003) proposed a method to
    measure quality characteristics of COTS components, based on the
    international standards for software product quality (ISO/IEC
    9126, ISO/IEC 12119 and ISO/IEC 14598). The method is composed
    of four steps:
 i. Establish evaluation requirements, specifying the purpose and scope of the
      evaluation, specifying evaluation requirements;
 ii. Specify the evaluation, selecting the metrics and the evaluation methods;
 iii. Design the evaluation, considers the component documentation, development
      tools, evaluation costs and expertise required in order to make the evaluation plan;
 iv. Execute the evaluation, the execution of the evaluation methods and the analysis
      of the results.
   However, the method proposed was not evaluated in a real case study
          SAAP 2008.1               http://www.rise.com.br
                               Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
  In 2003, Hissam (Hissam et al., 2003) introduced Prediction-
  Enabled Component Technology (PECT) as a means of packaging
  predictable assembly.

  This work, which is an evolution of Stafford et al.’s work (Stafford
  et al., 2001), attempts to validate the PECT and its components,
  giving credibility to the model




      SAAP 2008.1            http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                          A Embedded software
                                                          component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
   During 2003, a CMU/SEI’s report, Wallnau extended Hissam work
   (Hissam et al., 2003), in order to achieve Predictable Assembly
   from Certifiable Components (PACC).

   This novel model requires a better maturation by the software
   engineering community in order to achieve trust in it




      SAAP 2008.1           http://www.rise.com.br
                       Fernando Carvalho, PhD Candidate
                                                                 A Embedded software
                                                                 component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
     Magnus Larsson, in 2004 (Larsson, 2004), define A predictability
     approach of the quality attributes, where one of the main objectives
     is to enable integration of components as black boxes.
 According to composition principles, results types of attributes:
 • Directly compassable attributes. is a function of only the same attribute.
 • Architecture-related attributes. is a function of the same attribute and of the
     software architecture.
 •   Derived attributes. depends on several different attributes
 •   Usage-depended attributes. is determined by its usage profile.
This work is very useful, but before the component quality must be known.

          SAAP 2008.1              http://www.rise.com.br
                              Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
  Finally, in 2006 Daniel Karlson (Karlson et al., 2006) presented the
  verification of component-based embedded system designs. These
  techniques is Formal Methods based modeling approach(Petri net),
  called PRES+.
          Two problems are addressed:
          • component verification and
          • Integration verification.

  This approach verifies the component from only one perspective:
  functionality.
  Formal verification, it is used only in few cases when it is mandatory,
  because much time and financial effort are employed.
      SAAP 2008.1            http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Failures in Software Component Certification

            Two failure cases that can be found in the literature .

     First failure occurred in the US government, when trying to
     establish criteria for certificating components (NIAP). Thus, from
     1993 until 1996, NSA and the NIST used the Trusted Computer
     Security Evaluation Criteria (TCSEC), “Orange Book”.

     It had defined no means of features across classes of components,
     but only for a restricted set of behavioral assembly properties
     (Hissam et al., 2003).
        SAAP 2008.1             http://www.rise.com.br
 •                         Fernando Carvalho, PhD Candidate
     The second failure happened with an IEEE committee, in an
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: A Survey
 Failures in Software Component Certification

    The second failure happened with an IEEE committee, in an
    attempt to obtain a component certification standard.

The initiative was suspended, in this same year.
The committee came to a consensus that they were still far from getting to
the point where the document would be a strong candidate for a
standard. (Goulao et al., 2002a).


        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                                A Embedded software
                                                                component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
      One of the main objectives of software engineering

      •  Improve the quality of software products,
      •  Establishing methods and technologies to build software
         products.
      The quality area could be basically divided into two main topics
         (Pressman, 2005):
  •       Software Product Quality: aiming to assure the quality of the
            generated product; and
  •       Software Processes Quality: looking for the definition, evaluation
          and improvement of software development processes.
           SAAP 2008.1            http://www.rise.com.br
                             Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
  Software Product Quality:
   •    ISO/IEC 9126 (ISO/IEC 9126, 2001),
   •    ISO/IEC 12119 (ISO/IEC 12119, 1994),
   •    ISO/IEC 14598 (ISO/IEC 14598, 1998),
   •    SQuaRE project (ISO/IEC 25000, 2005) (McCall et al., 1977), (Boehm et al.,
           1978), among others

  Software Processes Quality:
   •   Capability Maturity Model (CMM) (Paulk et al., 1993),
   •   Capability Maturity Model Integrated (CMMI) (CMMI, 2000),
   •   Software Process Improvement and Capability dEtermination (SPICE)
       (Drouin, 1995), among others


       SAAP 2008.1              http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                                  A Embedded software
                                                                  component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization           Standards                                 Overview

   Many institutions are creating                ISO/IEC 61131      component-based approach for industrial systems
                                                 RTCA DO 178B       guidelines for development of aviation software
   standards to properly evaluate                ISO/IEC 61508      Security Life cycle for industrial software

   the quality and development                   ISO/IEC 9126       Software Products Quality Characteristics
                                                 ISO/IEC 14598      Guides to evaluate software product, based on practical usage
   processes of the software product,                               of the ISO 9156 standard

   in different domain.                          ISO/IEC 12119
                                                 SQuaRE project
                                                                    Quality Requirements and Testing for Software Packages
                                                                    Software Product Quality Requirements and Evaluation
                                                 (ISO/IEC 25000)
                                                 IEEE P1061         Standard for Software Quality Metrics Methodology
The Table shows a set of national and            ISO/IEC 12207      Software Life Cycle Process.

   international standards.                      NBR ISO 8402       Quality Management and Assurance.
                                                 NBR ISO 9000-1-2   Model for quality assurance in Design, Development, Test,
                                                                    Installation and Servicing
                                                 NBR ISO 9000-3     Quality Management and Assurance. Application of the ISO
                                                                    9000 standard to the software development process (evolution
                                                                    of the NBR ISO 8402).
                                                 CMMI               SEI’s model for judging the maturity of the software
                                                 (Capability        processes of an organization and for identifying the key
                                                 Maturity Model     practices that are required to increase the maturity of these
                                                 Integration)       processes.
                                                 ISO 15504          It is a framework for the assessment of software processes.
        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
   ISO/IEC 25000, 2005 / SQuaRE project - Software Product
      Quality Requirements and Evaluation has been created
      specifically to make two standards converge:

   •   ISO/IEC 14598, 1998 - define a software product evaluation
       process, based on the ISO/IEC 9126.

   •   ISO/IEC 9126, 2001 - define a quality model for software
       product

   Trying to eliminate the gaps, conflicts, and ambiguities that they
      present.
       SAAP 2008.1            http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
 ISO/IEC 25000, 2005 / SQuaRE project
 The objective is :
 • To respond to the evolving needs of users through an improved, and
 •   Unified set of normative documents covering three complementary
    quality processes:
       • Requirements specification,
       • Measurement and
       • Evaluation.

  The motivation is to supply for developing and acquiring software
  products with quality engineering instruments supporting both the
  specification and evaluation of quality requirements.
        SAAP 2008.1           http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                                 A Embedded software
                                                                 component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
     SQuaRE include:
 •   Criteria for the specification of
     quality requirements
 •   Evaluation of quality requirements,
 •   Recommended measures of software
     product quality attributes.

 which can be used by:
 • Developers,
 • Acquirers, and
 • Evaluators.


         SAAP 2008.1               http://www.rise.com.br
                              Fernando Carvalho, PhD Candidate
                                                          A Embedded software
                                                          component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
          Quality Requirements Division (ISO/IEC 2503n)

   ISO/IEC25030 - 2007, standard for supporting the specification of
   quality requirements, either during software product quality
   requirement elicitation or as an input for an evaluation process:

     Quality requirements and guide: to enable software
    product quality to be specified in terms of quality
    requirements;



      SAAP 2008.1           http://www.rise.com.br
                       Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
                Quality Model Division (ISO/IEC 2501n)

   ISO/IEC 25010 – 2005, contains the detailed quality model and its
   specific characteristics and sub-characteristics for internal quality,
   external quality and quality in use. This division includes:

    Quality model and guide: to describe the model for
    software product internal and external quality, and quality
    in use. The document present the characteristics and sub-
    characteristics for internal and external quality and
    characteristics for quality in use.
      SAAP 2008.1             http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
      Product Quality General Division (ISO/IEC 2500n)
 ISO/IEC 25000 – 2005 contains the unit standards defining all
 common models, terms and definitions referred to by all other
 standards in the SQuaRE series.
 This division includes two unit standards:

  •Guide to SQuaRE: to provide the SQuaRE structure, terminology,
  document overview, intended users and associated parts of the series,
  as well as reference models;
  •Planning and management: to provide the requirements and guidance
  for planning and management support functions for software product
  evaluation.
       SAAP 2008.1            http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
           Quality Measures Division (ISO/IEC 2502n)
 ISO/IEC 25020 - 2007 were derived from ISO/IEC 9126 and ISO/IEC
 14598.

 This division covers the mathematical definitions and guidance for
 practical measurements of internal quality, external quality and
 quality in use.

 It will include the definitions for the measurement primitives and the
 Evaluation Module to support the documentation of measurements.


       SAAP 2008.1            http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                          A Embedded software
                                                          component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
     Quality Measures Division (ISO/IEC 2502n)

           Measurement reference model and guide
           Measurement primitives
           Measures for internal quality
           Measures for external quality
           Measures for quality in use


     SAAP 2008.1            http://www.rise.com.br
                       Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
     Quality Evaluation Division (ISO/IEC 2504n)
 ISO/IEC - 25040 contains the standards for providing requirements,
 recommendations and guidelines for software product evaluation,
 whether performed by evaluators, acquirers or developers:
               Quality evaluation overview and guide
               Process for developers
               Process for acquirers
               Process for evaluators
               Documentation for the evaluation module


       SAAP 2008.1              http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
                ISO/IEC 2501n (Quality Model Division)
 ISO/IEC 2501n is composed of the ISO/IEC 9126 -1 standard, which
 provides a Quality Model for software product.

 At the present time, this division contains only one standard: 25010 –
 Quality Model and guide. This is an ongoing standard in development.
   Quality Model Division does not prescribe specific quality
   requirements for software, but rather defines a generic quality
   model, which can be applied to every kind of software.


       SAAP 2008.1            http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
                                                              Characteristics       Sub-Characteristics
ISO/IEC 2501n (Quality Model Division)                       Functionality      Suitability
                                                                                Accuracy

 Characteristics and Sub-Characteristics                                        Interoperability
                                                                                Security
                                                                                Functionality Compliance
             in SQuaRE project                               Reliability        Maturity
                                                                                Fault Tolerance
                                                                                Recoverability
                                                                                Reliability Compliance
                                                             Usability          Understandability
   The ISO/IEC 25010 defines a quality                                          Learnability
                                                                                Operability
   model that comprises six characteristics                                     Attractiveness
                                                                                Usability Compliance

   and 27 sub-characteristics:                               Efficiency         Time Behavior
                                                                                Resource Utilization
                                                                                Efficiency Compliance
                                                             Maintainability    Analyzability
                                                                                Changeability
                                                                                Stability
                                                                                Testability
                                                                                Maintainability Compliance
                                                             Portability        Adaptability
                                                                                Installability
                                                                                Replaceability
                                                                                Coexistence
       SAAP 2008.1            http://www.rise.com.br                            Portability Compliance
                         Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
  Quality in Use characteristics and are modeled with four characteristics:
     • effectiveness,
     • productivity,
     • security and
     • satisfaction


  The main drawback of the ISO/IEC 25010, is that they provide very
  generic quality models and guidelines, which are very difficult to
  apply to specific domains such as embedded components and CBSD.



       SAAP 2008.1             http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
          ISO/IEC 2504n (Quality Evaluation Division)
The ISO/IEC 2502n is composed
of the ISO/IEC 14598 standard,
which provides a generic model
of an evaluation process,
supported by the quality
measurements from ISO/IEC
9126. This process is specified in
four major sets of activities for
an evaluation:


        SAAP 2008.1             http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
         ISO/IEC 2504n (Quality Evaluation Division)
The ISO/IEC 2504n is divided in five standards:
•ISO/IEC 25040 – Evaluation reference model and guide;
•ISO/IEC 25041 – Evaluation modules;
•ISO/IEC 25042 – Evaluation process for developers;
•ISO/IEC 25043 – Evaluation process for acquirers; and
•ISO/IEC 25044 – Evaluation process for evaluators.




       SAAP 2008.1           http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
        ISO/IEC 2502n (Quality Measurement Division)
The ISO/IEC 2502n - 2007 improve the quality measurements provided
by ISO/IEC 9126-2/3/4 (external metrics), (internal metrics) and (quality
in use metrics)
The most significantly is the adoption of the Goal-Question- Metrics
(GQM) paradigm (Basili et al., 1994), thus, the metrics definition
becomes more flexible and adaptable to the software product evaluation
context.




        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: Standardization
        ISO/IEC 2502n (Quality Measurement Division)
The ISO/IEC 2502n is divided in five standards:
      •ISO/IEC 25020 - Measurement reference model and guide;
      •ISO/IEC 25021 – Measurement primitives;
      •ISO/IEC 25022 – Measurement of internal quality;
      •ISO/IEC 25023 – Measurement of external quality; and
      •ISO/IEC 25024 – Measurement of quality in use.

These standards contain some examples in how to define metrics for
different kinds of perspectives, such as internal, external and quality in
use.
        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                              A Embedded software
                                                              component quality framework


3 – Embedded Software Component
Quality and Certification: Certification

“certification, in general, is the process of verifying a property value
associated with something, and providing a certificate to be used as proof of
validity”. (Stafford et al., 2001)

“Third-party certification is a method to ensure that software components
conform to well-defined standards; based on this certification, trusted
assemblies of components can be constructed.” (Councill, 2001)




        SAAP 2008.1             http://www.rise.com.br
                           Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3 – Embedded Software Component
Quality and Certification: Certification
        Third party certification is often viewed as a good way of bringing
trust in software components.
        Components can be obtained from existing systems through
 reengineering, designed and built from scratch, or purchased.
         After that, the components are certified, in order to achieve some
 trust level, and stored into a repository system




        SAAP 2008.1            http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                            A Embedded software
                                                            component quality framework


3 – Embedded Software Component
Quality and Certification: Certification
The CBSE community is still far from reaching a consensus:

•how it should be carried out,
•what are its requirements and
•who should perform it.

Some difficulties, was found due
to the relative novelty of this
area (Goulao et al., 2002a).



        SAAP 2008.1           http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                              A Embedded software component
                                                              quality verification framework



4 - Embedded software Component
    Quality Verification Framework
In a survey of the state-of-the-art was noted that there is a lack of
processes, methods, techniques and tools available for evaluating
component quality, specifically for embedded is much more scarce.

This necessity is pointed out by different researchers (Voas, 1998),
(Morris et al., 2001), (Wallnau, 2003), (Alvaro et al., 2005), (Bass et al.,
2003), (Softex, 2007) and (Lucrédio et al., 2007).

Most researchers agree that component quality is an essential aspect of
the CBSE adoption and software reuse success.


                           Fernando Carvalho, PhD Candidate
                                                              A Embedded software component
                                                              quality verification framework



4 - Embedded software Component
    Quality Verification Framework
Its idea is to improve the lack of consistency between the available
standards for software product quality (ISO/IEC 9126), (ISO/IEC
14598), (ISO/IEC 25000), also including the software component
quality context and extend it to the embedded domain.

These standards provide a high-level definition of characteristics and
metrics for software products but do not provide ways to be used in an
effective way, becoming very difficult to apply them without acquiring
more knowledge from supplementary sources.



                           Fernando Carvalho, PhD Candidate
                                                       A Embedded software component
                                                       quality verification framework



4 - Embedded software Component
    Quality Verification Framework
             Overview of the Framework
       in robust framework for software reuse context




                    Fernando Carvalho, PhD Candidate
                                                            A Embedded software component
                                                            quality verification framework



4 - Embedded software Component
    Quality Verification Framework
                   Overview of the Framework
       The framework will allow that the embedded components
    produced in a Software Reuse Environment are certified before
    being stored in a Repository System.
 The Embedded Software Component Quality Verification Framework is
    composed of four modules:
    •   an Embedded software component Quality Model,
    •   a Maturity Level Evaluation Techniques,
    •   a Metrics Approach, and
    •   a Component Certification Process.
                         Fernando Carvalho, PhD Candidate
                                                              A Embedded software component
                                                              quality verification framework



4 - Embedded software Component
    Quality Verification Framework
                     Overview of the Framework
          The framework cover two perspectives of the three considered in
     SQuaRE project : acquirers and evaluators.
 •         acquirer’s perspectives is used to define which component best
     fits the customer’s needs and application/domain context.
 •         evaluator’s perspectives should be considered for evaluation
     required by companies in order to achieve trust in its components.
 •        developer’s perspectives is not contemplate, because it very hard
     for only one developer to execute all activities, independent of his
     knowledge


                           Fernando Carvalho, PhD Candidate
                                                            A Embedded software component
                                                            quality verification framework



4 - Embedded software Component
    Quality Verification Framework
                         Details of framework
    Embedded software component Quality Model (EQM)

 The evaluation occurs through models that measure quality
 These models describe and organize the quality characteristics that
will be considered during the evaluation
 To measure the quality it is necessary to develop a Quality Model
 The EQM proposed is based on SQuaRE project (ISO/IEC 25000,
2005), with adaptations for components and in embedded domain

                         Fernando Carvalho, PhD Candidate
                                                              A Embedded software component
                                                              quality verification framework



4 - Embedded software Component
    Quality Verification Framework
                          Details of framework
    Embedded software component Quality Model (EQM)
 Some definitions:
 Quality characteristic is a set of properties by which its quality can be
        described and evaluated, and refined into sub-characteristics.
 Attribute is a quality property to which a metric can be assigned, where a
 Metric is a procedure for examining a component.
 Quality model is the set of characteristics and sub-characteristics, that
        provide the basis for specifying quality requirements and for
        evaluating quality (Bertoa et al., 2002).

                           Fernando Carvalho, PhD Candidate
                                                                A Embedded software component
                                                                quality verification framework



4 - Embedded software Component
    Quality Verification Framework
      Embedded software component Quality Model (EQM)
Identifying important quality characteristics, classified in different criteria:
   i. Local or Global characteristics
        a. individual components (local characteristics )
        b. software architecture level (global characteristics).
   ii. Moment in which it can be measured (Preiss et al.,2001):
        a. characteristics at runtime (e. g. Performance)
        b. characteristics at cycle-life (e. g. Maintainability).
   iii. Application Metrics
        a. internal metrics (white-box)
        b. external metrics (black-box)
   iv. Marketing characteristics
                             Fernando Carvalho, PhD Candidate
                                                       A Embedded software component
                                                       quality verification framework



4 - Embedded software Component
    Quality Verification Framework
                                       Characteristics     Sub-Characteristics       Sub-Characteristics
                                                               Run-time                  Life cycle

The EQM follow the ISO/IEC 25010,      Functionality      Real-time
                                                          Accuracy
                                                                                    Suitability
                                                                                    Interoperability
some changes were made to adequate                        Security                  Compliance
                                                                                    Self-contained

for software components in embedded    Reliability        Recoverability
                                                          Fault Tolerance
                                                                                    Maturity

                                                          Safety
context.                               Usability          Configurability           Understandability
                                                                                    Learnability
                                                                                    Operability

The characteristics :                  Efficiency         Time Behavior
                                                          Resource behavior

•Relevant were maintained;                                Scalability
                                                          Energy consumption
                                                          Memory utilization
•Not interesting was eliminated;       Maintainability    Analyzability             Changeability

•The name was changed to adequate it   Portability
                                                          Stability
                                                          Deployability
                                                                                    Testability
                                                                                    Replaceability
                                                                                    Flexibility
to new context;                                                                     Reusability

•New important characteristics was     Marketability                     Development time
                                                                      Compatibles architectures
                                                                                Cost
added                                                                      Time to market
                                                                          Targeted market
                                                                            Affordability
                                                                             Licensing
                                                                 A Embedded software component
                                                                 quality verification framework



4 - Embedded software Component
    Quality Verification Framework
 The use of attributes and metrics is used to determine whether a
    component fulfills in the characteristics and sub-characteristics .
 The EQM consists of four elements:

    •   Characteristics,
    •   Sub-characteristics,
    •   Attributes and
    •   Metrics.

                       A quality characteristic is a set of properties of a software product through which its
                       quality can be described and evaluated
                       An attribute is a measurable physical or abstract property of an entity.

                       A metric defines the measurement method and the measurement scale.
                                                                   A Embedded software component
                                                                   quality verification framework



4 - Embedded software Component
    Quality Verification Framework
 Embedded software Component Quality Attributes that are observable
   at runtime and life-cycle.
                                                           Sub-               Sub-                  Attributes
The table groups the            Characteristics        Characteristics
                                                         (Runtime)
                                                                         Characteristics
                                                                          (Life-cycle)

attributes by characteristics                                                              1.Response time (Latency)
                                                                                           a.Throughput (“out”)
                                                                                           b.Processing Capacity (“in”)
and sub-characteristics, and                      Real-time                                1.Execution time
indicates the metrics used                                                                 1.Worst case execution time
                                                                                           1.Dead line
for evaluating each                               Accuracy                                 1.Correctness
                                Functionality
attribute.                                                                                 1.Data Encryption
                                                  Security                                 1.Controllability
                                                                                           1.Auditability
                                                                                           1.Standardization
                                                                         Compliance
                                                                                           1.Certification
                                                                         Self-contained    1.Dependability
                                                                   A Embedded software component
                                                                   quality verification framework



4 - Embedded software Component
    Quality Verification Framework
 Embedded software Component Quality Attributes
                                                               Sub-                Attributes
                                      Sub-
                                                          Characteristic
           Characteristics        Characteristics
                                                                 s
                                    (Runtime)
                                                           (Life-cycle)
                             Recoverability                                1.Error Handling
                                                                           1.Mechanism availability
                             Fault Tolerance
           Reliability                                                     1.Mechanism efficiency
                                                                           1.Environment analyze
                             Safety
                                                                           1.Integrity
                                                                           1.Effort to configure
           Usability         Configurability
                                                                           1.Understandability
                             Resource behavior                             1.peripheral utilization
                             Energy consumption                            1.Mechanism availability
           Efficiency        Data Memory utilization                       1.Mechanism availability
                             Program             Memory                    1.Mechanism availability
                             utilization
                                                                   A Embedded software component
                                                                   quality verification framework



4 - Embedded software Component
    Quality Verification Framework                   Sub-                    Sub-                      Attributes
 Embedded Software      Characteristics          Characteristics
                                                   (Runtime)
                                                                        Characteristics
                                                                         (Life-cycle)

     Component                            Stability                                       1.Modifiability
                                                                                          1.Extensibility
   Quality Attributes                                                  Changeability      1.Customizability
                                                                                          1.Modularity

                        Maintainability                                                   1.Test suite provided
                                                                                          1.Extensive component test
                                                                                          cases
                                                                       Testability
                                                                                          1.Component tests in a specific
                                                                                          environment
                                                                                          1.Proofs the components tests
                                          Deployability                                   1.Complexity level
                                                                       Replaceability     1.Backward Compatibility
                                                                                          1.Mobility
                                                                       Flexibility
                                                                                          1.Configuration capacity
                                                                                          1.Domain abstraction level
                        Portability
                                                                                          1.Architecture compatibility
                                                                                          1.Modularity
                                                                       Reusability
                                                                                          1.Cohesion
                                                                                          1.Coupling
                                                                                          1.Simplicity
                                                   A Embedded software component
                                                   quality verification framework



4 - Embedded software Component
    Quality Verification Framework
 The model is complemented with Quality in Use characteristics
    (ISO/IEC 25000, 2005) are composed of:

    •   Productivity,    • Bring relevant information for new customers,
    •   Satisfaction,    • This is the user’s view of the component,
                         • Obtained when the component in an execution environment, and
    •   Security, and
                         • Analyze the results according to their expectations.
    •   Effectiveness.

 Quality in Use characteristics are useful to show the component’s
    behavior in different environments.
 These characteristics are measured through the customer’s feedback
                                                  A Embedded software component
                                                  quality verification framework



4 - Embedded software Component
    Quality Verification Framework
                       Relevant Component Information
 The Additional Information characteristics complement the model and
    are composed of:
 Technical Information is important for developers to analyze the actual
    state of the component ,
 Organization Information is important to know who is the responsible
    for that component.
                           Additional   Technical Information
                          Information       Component Version
                                            Programming Language
                                            Patterns Usage
                                            Architecture compatible
                                            Program Memory used
                                            Technical Support
                                        Organization Information
                                            CMMi Level
                                            Organization’s Reputation
                                             A Embedded software component
                                             quality verification framework



4 -1 Maturity Level Evaluation Techniques
 The quality characteristics proposed not need to be evaluated with the
     same degree of details and depth for all types of application.
 (E. g. evaluation of a component used in railway system and game).

 Different evaluation levels must be used in order to provide degree of
     confidence for different domains and risk-levels.
                                            A Embedded software component
                                            quality verification framework



4 -1 Maturity Level Evaluation Techniques
    Embedded software component Maturity Model (EMM)
 The Details of an evaluation is a reflex of the evaluation techniques
     used.
 So, an Embedded software component Maturity Model (EMM)
     was defined. It is based on CMMI (CMMI, 2000) and model for
     general propose component (Alvaro et al., 2007a).
                                             A Embedded software component
                                             quality verification framework



4 -1 Maturity Level Evaluation Techniques
 The EMM is constituted of five hierarchical levels of quality
    characteristics where the components can be evaluated in different
    the depth of the evaluation gives different degrees of confidence.

 Each company/customer decides which level is better for evaluating its
    components, analyzing the cost/benefits of each level.

 The evaluation levels can be chosen independently for each
    characteristic (e.g. functionality → EMM I, reliability → EMM III,
    usability → EMM IV).
                                                           A Embedded software component
                                                           quality verification framework



4 -1 Maturity Level Evaluation Techniques
                Guidelines for selecting evaluation level
       Level      Environment      Safety/Security        Economic           Domain
      EMM I        No damage          Few material         Negligible     Entertainment,
                                   damage; No specific   economic loss
                                          risk
      EMM II      Small/Medium         Few people        Few economic       household
                 damage properly        disabled              loss
      EMM III    Damage properly    Large number of       Significant        Security,
                                    people disabled      economic loss    Control systems
      EMM IV       Recoverable      Threat to human          Large           Medical,
                   environment           lives           economic gross      Financial
                     damage
      EMM V       Unrecoverable    Many people killed      Financial      Transportation,
                  environmental                             disaster      Nuclear systems
                     damage
                                                  A Embedded software component
                                                  quality verification framework



4 -1 Maturity Level Evaluation Techniques
 One of the main concerns during EMM definition is that the levels and the
        evaluation techniques selection must be appropriated to completely
         evaluate the quality attributes proposed on the EQM, presented in
    session 4.2. This is achieved through a mapping of the Quality Attributes
        X Evaluation Technique. For each quality attribute proposed on the
     EQM, it is interesting that at least one technique is proposed in order to
      cover it completely, also being capable of measuring it properly. Table
      4.3.3 shows this matching between the EQM quality attributes and the
                       proposed EMM evaluation techniques.
 Table 4.3.3 shows that the main concern is not to propose a large amount of
     isolated techniques, but to propose a set of techniques that are essential
       for measuring each quality attribute, complementing each other and,
         thus, becoming useful to compose the Maturity Level Evaluation
                                     Techniques.
                                                                                                                A Embedded software component
                                                                                                                quality verification framework



4 -1 Maturity Level Evaluation Techniques
Characteristi                   EMM I                                    EMM II                                EMM III                          EMM IV                  EMM V
     cs
Functionality     Time constraint analysis             Evaluation measurement (Time              System Test                         Functional Tests (white-   Formal Proof
                  Requirements and Documentation       analysis)                                  Documents Inspection (check list)   box) with coverage
                  Analysis                              Functional Testing (black box), Unit      Code Inspection                     criteria and code
                  Accuracy analysis                    Test, Regression Test (if possible)                                             inspection
  Reliability     Dependability analysis               Programming Language Facilities (Best     Error Manipulation analysis         Error recover              Formal Proof
                  Suitability analysis                 Practices)                                 Fault tolerance analysis            Reliability growth
                                                                                                   Error Injection analysis            model
   Usability      Effort to Configure analysis         Interfaces inspection provided and        Code and component’s interface      Analysis of the pre and    User mental
                  Documentation analysis (Use          required                                   inspection correctness and           post-conditions of the      model
                  Guide, architectural analysis, etc)                                              completeness)                        component
  Efficiency      Constraint analyses                  Evaluation measurement (memory,           Tests of performance(memory,        Algorithmic complexity     Performance
                  Accuracy analysis                    power and resource)                        power and resource)                  Performance                profiling analysis
                                                        Memory Analysis                                                                optimization (memory,       Formal Proof
                                                        Power consumption Analysis                                                     power and resource)
                                                        Resource Analysis
Maintainability   Customizability analysis             Inspection of Documents                   Code metrics and programming        Analysis of the            Traceability
                  Extensibility analysis               Analysis of the provided test suite (if   rules                                component development       evaluation
                                                        exists)                                    Static Analysis                     process                     Component Test
                                                                                                                                                                    Formal Proof
  Portability     Component execution in specific      Deployment analysis                       Conformity to programming rules     Environment and            Analysis of the
                  environment and architectural         Backward compatibility                                                         architectural constraints   component’s
                  analysis                              Mobility analysis                                                              evaluation                  architecture
                  Cohesion, Coupling, Modularity       Configurable analysis                                                          Domain abstraction
                  and Simplicity analyses               Hardware/Software analysis                                                     analysis
                  Cohesion of the documentation
                  with the source code analysis
                                                                        A Embedded software component
                                                                        quality verification framework



4 -1 Maturity Level Evaluation Techniques
     Charac-             Sub-                             Quality             Evaluation Techniques
     teristic        Characteristics                     Attributes
    Functional   Real-Time             Response time (Latency)        • Evaluation measurement (Time analysis)
       ity                             a.Throughput (“out”)           • Time constraint analysis
                                       b.Processing Capacity (“in”)   • Formal Proof
                                       Execution time                 • Evaluation measurement
                                       Worst case execution time      • Evaluation measurement
                                                                      • System Test
                                       Dead line                      • Evaluation measurement
                                                                      • System Test
                 Accuracy              Correctness                    • Requirements and Documentation Analysis
                                                                      • Accuracy analysis
                                                                      • Functional Testing (black box),Unit Test,
                                                                      Regression Test (if possible)
                                                                      • Functional Tests (white-box) with
                                                                      coverage criteria
                 Security              Data Encryption                • System Test
                                                                      • Code Inspection
                                       Controllability                • System Test
                                                                      • Code Inspection
                                       Auditability                   • System Test
                                                                      • Code Inspection
                 Compliance            Standardization                • Inspection of Documents
                                       Certification                  • Inspection of Documents
                 Self-contained        Dependability                  • Documents Inspection
                                                                      • Code Inspection
                                                                A Embedded software component
                                                                quality verification framework



4 -1 Maturity Level Evaluation Techniques
   Charac-             Sub-                         Quality              Evaluation Techniques
   teristic        Characteristics                 Attributes
  Reliability   Recoverability       Error Handling               • Programming Language Facilities (Best
                                                                  Practices)
                                                                  • Error Manipulation analysis
                                                                  • Error Injection analysis
                                                                  • Error recover
                                                                  • Reliability growth model
                                                                  • Formal Proof
                Fault Tolerance      Mechanism available          • Suitability analysis
                                                                  • Dependability analysis
                                     Mechanism efficiency         • Error injection analysis
                                                                  • Programming Language Facilities (Best
                                                                  Practices)
                                                                  • Fault tolerance analysis
                                                                  • Reliability growth model
                                                                  • Formal Proof
                Safety               Environment analyze          • Dependability analysis
                                                                  • Environment analyses
                                                                  • System analyses
                                     Integrity                    • System analyses
                                                            A Embedded software
                                                            component quality framework


Embedded Software Component Quality Process
• Certification is the future of software components
  [Wallnau, Heineman, Councill, Shaw]
• RiSE Approach
   1.Embedded Component Quality Model (ECQM)
       Embedded Software Component Maturity Model (ESCMM)
   2.Certification Techniques Framework
       Defining techniques for evaluate quality attributes of ECQM
   3.Metrics Framework
       Track the properties of the ECQM, certification techniques and
        process.
   4.Embedded Software Component Certification Process
       Defining the steps for certify a component




       SAAP 2008.1            http://www.rise.com.br
                         Fernando Carvalho, PhD Candidate
                                                                        A Embedded software
                                                                        component quality framework

Embedded Component Certification Process
                                                                       Changes in the Proposed Component Quality
                                                                           Model, in relation to ISO/IEC 25010
 1. Embedded Component Quality                                          Characteristics   Sub-Characteristics
    Model (ECQM)                                                        Functionality     Suitability
     o Embedded Software Component                                                        Accuracy
                                                                                          Interoperability
       Maturity Model (ESCMM)                                                             Security
                                                                                          Compliance
     o Based ISO/IEC 25010 (*)                                                            Self-contained

       standard, with some news and                                     Reliability       Maturity

       extended characteristics.                                                          Recoverability
                                                                                          Fault Tolerance

                                                                        Usability         Understandability
                                                                                          Configurability
                                                                                          Learnability
                                                                                          Operability

                                                                        Efficiency        Time Behavior
                                                                                          Resource behavior
                                                                                          Scalability

                                                                        Maintainability   Analyzability
                                                                                          Stability
                                                                                          Changeability
                                                                                          Testability

                                                                        Portability       Deployability
*ISO/IEC 9126 - Software Products Quality                                                 Replaceability
                                                                                          Adaptability
ISO/IEC 14598 - Evaluation software product                                               Reusability

                                                                        Marketability     Development time
                                                                                          Cost
                                                                                          Time to market
                                                                                          Targeted market
           SAAP 2008.1                   http://www.rise.com.br                           Affordability
                                    Fernando Carvalho, PhD Candidate                      Licensing
                                                                      A Embedded software
                                                                      component quality framework

 2. Certification Techniques Framework
  o Defining techniques for evaluate quality attributes of ECQM

                            Guidelines for selecting evaluation level.

                        Environment                      Safety/Security              Economic
  Level
ESCMM I                   No damage                 Few material damage; No       Negligible economic
                                                         specific risk                    loss
ESCMM II         Small/Medium damage                    Few people disabled        Few economic loss
                       properly
ESCMM III               Damage properly              Large number of people       Significant economic
                                                            disabled                       loss
ESCMM IV        Recoverable environment                Threat to human lives        Large economic
                        damage                                                           gross
ESCMM V               Unrecoverable                      Many people killed        Financial disaster
                  environmental damage

          SAAP 2008.1                   http://www.rise.com.br
                                   Fernando Carvalho, PhD Candidate
                                                                               A Embedded software
                                                                               component quality framework

   2. Certification Techniques Framework
    o Defining techniques for evaluate quality attributes of ECQM

                                  Guidelines for selecting evaluation level.



 Level         ESCMM I              ESCMM II                  ESCMM III             ESCMM IV         ESCMM V


Reliability
              Suitability • Programming                 • Fault tolerance       Reliability growth   Formal
               analysis Language                        analysis                model                 Proof
                             Facilities (Best           • Error Manipulation
                             Practices)                 analysis
                             • Maturity analysis




               SAAP 2008.1                       http://www.rise.com.br
                                            Fernando Carvalho, PhD Candidate
                                                             A Embedded software
                                                             component quality framework


3. Metrics Framework
 o Track the properties of the ECQM, certification techniques and
   process.

                                     Functionality

      Sub-Characteristic Accuracy

      Quality Attribute   Correctness

      Goal                Evaluates the percentage of the results that were obtained
                          with precision

      Question            Based on the amount of tests executed, how much test
                          results return with precision?

      Metric              Precision on results / Amount of tests

      Interpretation      0 <= x <= 1; which closer to 1 is better




    SAAP 2008.1                http://www.rise.com.br
                          Fernando Carvalho, PhD Candidate
                                                         A Embedded software
                                                         component quality framework




4.   Embedded Software Component Certification Process
 o   Defining the steps for certify a component
       4.1 Establish Evaluation Requirements activity
       4.2 Specify the Evaluation activity
       4.3 Design the Evaluation activity
       4.4 Execute the Evaluation activity




       SAAP 2008.1         http://www.rise.com.br
                      Fernando Carvalho, PhD Candidate
                                                    A Embedded software
                                                    component quality framework


4. Embedded Software Component
Certification Process




   SAAP 2008.1        http://www.rise.com.br
                 Fernando Carvalho, PhD Candidate
                                                     A Embedded software
                                                     component quality framework



4.1 Establish Evaluation Requirements activity




    SAAP 2008.1        http://www.rise.com.br
                  Fernando Carvalho, PhD Candidate
                                                 A Embedded software
                                                 component quality framework



4.2 Specify the Evaluation activity




SAAP 2008.1        http://www.rise.com.br
              Fernando Carvalho, PhD Candidate
                                                    A Embedded software
                                                    component quality framework


4.3 Design the Evaluation activity




   SAAP 2008.1        http://www.rise.com.br
                 Fernando Carvalho, PhD Candidate
                                                   A Embedded software
                                                   component quality framework



4.4 Execute the Evaluation activity




  SAAP 2008.1        http://www.rise.com.br
                Fernando Carvalho, PhD Candidate
                                                        A Embedded software
                                                        component quality framework

Submissão de artigo dia 05/06/2008 - SBCARS




  SAAP 2008.1             http://www.rise.com.br
                     Fernando Carvalho, PhD Candidate
                                                 A Embedded software
                                                 component quality framework




SAAP 2008.1        http://www.rise.com.br
              Fernando Carvalho, PhD Candidate
                                                           A Embedded software
                                                           component quality framework




Thank you !

Questions ?
• Fernando F. de Carvalho
  o   ffc@cin.ufpe.br




       SAAP 2008.1           http://www.rise.com.br
                        Fernando Carvalho, PhD Candidate