Evolutionary Computation by gyvwpsjkko


									Evolutionary Computation
Artificial Neural Networks

                                                            Out put Signals
    Input Signals

                              First   Second
                    Input    hidden   hidden       Output
                    layer     layer    layer        layer

                            Fixed Architecture
                            Require training set

                        Genetic algorithm

Fixed size chromosome
                            Neuro-Genetic Hybrid
                                                          x1              0.9
                                                               1                4
From neuron:     1      2      3   4    5   6    7    8
                                                                          0.1       -0.6
To neuron: 1      0      0      0   0   0    0   0    0            0.4
            2     0      0      0   0   0    0   0    0                         5
                                                                 -0.3               0.1
            3     0      0      0   0   0    0   0    0                                        y
                                                          x2         0.6
            4    0.9   -0.3   -0.7 0    0    0   0    0        2                           8
            5   -0.8    0.6    0.3 0    0    0   0    0             -0.2
            6    0.1   -0.2    0.2 0    0    0   0    0                         6
            7    0.4    0.5    0.8 0    0    0   0    0            -0.7
                                                                          0.3       0.9
            8     0      0      0 -0.6 0.1 -0.2 0.9   0
                                                               3          0.8   7

Chromosome: 0.9 -0.3 -0.7 -0.8 0.6 0.3 0.1 -0.2 0.2 0.4 0.5 0.8 -0.6 0.1 -0.2 0.9

                  GA replaces Back-propagation to train the NN
                  Fixed NN architecture, fixed size chromosome
  Evolutionary Complexification
• Two major goals in intelligent systems are the
  discovery and improvement of solutions to
  complex problems.
• Complexification, i.e. the incremental elaboration
  of solutions through adding new structure,
  achieves both these goals.
• To discover and improve complex solutions,
  evolution, and search in general, should be
  allowed to complexify as well as optimize.
    Evolutionary Computation
• Class of algorithms that can be applied to
  open-ended learning problems in AI
• Traditionally such algorithms evolve fixed
  length genomes assuming the space of
  the genome is sufficient to encode the
• In many cases a solution may be known to
  exist in that space
 Indefinite numbers of parameters
• Many common structures are defined by
  an indefinite number of parameters
• E.g., the number of neurons in an ANN
• So it is often not clear what number of
  genes is appropriate to solve a problem
• Researchers must use heuristics to
  determine a priori the appropriate number
  of genes
       Fixed Length Encoding
• Pre-determination of the appropriate number of
  genes is difficult
• Larger the genome the larger the search space
• Sometimes the solutions should evolve in an
  open-ended way (games) with no final solution
• Fixing the maximum size of the genome also
  fixes the maximum complexity of the evolved
• Ping-pong playing robot - solution is to
  make the genome very large
• Open-ended problems when no final
  solution can be accepted, improving after
  a certain point not possible with a fixed
  length genome
          Continual Evolution
• Such continual evolution is difficult with a fixed
  genome for two reasons:
• When a good strategy is found in a fixed-length
  genome, the entire representational space of the
  genome is used to encode it. Thus, the only way
  to improve it is to alter the strategy, thereby
  sacrificing some of the functionality learned over
  previous generations.
• Fixing the size of the genome in such domains
  arbitrarily fixes the maximum complexity of
  evolved creatures, defeating the purpose of the
• Extending the length and size of the genome
• Adds new genes that lead to increased
  phenotypic complexity
• Called complexification
• Specifically with evolving neural nets it means
  adding nodes and connections to an already
  functional ANN
• Allow more complex strategies to elaborate on
  simpler strategies.
   Evolving neuro-architecture
• Over many generations, new hidden
  nodes and connections are added,
  complexifying the space of potential
• In this way, more complex strategies
  elaborate on simpler strategies, focusing
  search on solutions that are likely to
  maintain existing capabilities.
Alteration vs. Elaboration
          Alteration vs. Elaboration
•   The dark robot must evolve to avoid the lighter robot, which attempts to
    cause a collision.
•   In the alteration scenario (top), the dark robot first evolves a strategy to go
    around the left side of the opponent. However, the strategy fails in a future
    generation when the opponent begins moving to the left.
•   The dark robot alters its strategy by evolving the tendency to move right
    instead of left. However, when the light robot later moves right, the new,
    altered, strategy fails because the dark robot did not retain its old ability to
    move left.

•   In the elaboration scenario (bottom), the original strategy of moving left also
    fails. However, instead of altering the strategy, it is elaborated by adding a
    new ability to move right as well. Thus, when the opponent later moves
    right, the dark robot still has the ability to avoid it by using its original
•   Elaboration is necessary for a coevolutionary arms race to emerge and it
    can be achieved through complexification.
               Key ideas
• Keeping track of which genes match with
  differently sized genes throughout
• Speciation, so that solutions of differing
  complexity can exist independently
• Beginning with a uniform population of
  small networks
• Open-ended problems with no explicit
  fitness function
• Fitness depends on comparisons with
  other agents performing the same task
  (uses coevolution)
• Robot duel domain. No known best
  strategy for a robot.
    Complexification in Nature
• In nature optimization does not occur with
  fixed size genes
• New genes are occasionally added to the
• Speciation protects newly formed more
  complex genes
          Gene duplication
• Gene duplication is a kind of mutation in
  which multiple copies of parental genes
  are copied into offspring genome
• The offspring has redundant genes
  expressing the same proteins
• Gene duplication is a possible explanation
  how natural evolution expanded the size of
  genomes throughout evolution
 Evidence for Gene Duplication
• Gene duplication has been responsible for key
  innovations in overall body morphology over the course
  of natural evolution
• A major gene duplication event occurred around the time
  that vertebrates separatedfrom invertebrates.
• Invertebrates have a single HOX cluster (of genes) while
  vertebrates have four, suggesting that cluster duplication
  significantly contributed to elaborations in vertebrate
• Researchers agree that gene duplication in some form
  contributed significantly to body-plan elaboration.
         Gene Duplication and
         Genetic Programming
• Gene duplication is a possible explanation how
  natural evolution indeed expanded the size of
  genomes throughout evolution, and provides
  inspiration for adding new genes to artificial
  genomes as well.
• Gene duplication motivated Koza (1995) to allow
  entire functions in genetic programs to be
  duplicated through a single mutation, and later
  differentiated through further mutations.
• When evolving neural networks, this process
  means adding new neurons and connections to
  the networks.
• Such systems evolve different sized and shaped
  network topologies which are difficult to
  crossover without losing information
• Artificial crossover may disrupt evolved
• Optimizing variable length genomes may take
  longer and more complex networks be
  eliminated before they have had a chance to be
       Variable Length Genes

• Crossover causes problems through
• Optimization takes longer causing early
  elimination of possible innovations
• Depending on when new structure was
  added, the same gene may exist at
  different positions, or conversely, different
  genes may exist at the same position.
• Thus, artificial crossover may disrupt
  evolved topologies through misalignment.
• Alignment processes have been observed
  in nature – synapsis
• Second, innovations in nature are protected
  through speciation. Organisms with significantly
  divergent genomes never mate because they
  are in different species.
• If any organism could mate with any other,
  organisms with initially larger, less-fit genomes
  would be forced to compete for mates with their
  simpler, more fit counterparts.
• As a result, the larger, more innovative genomes
  would fail to produce offspring and disappear
  from the population.
• NeuroEvolution of Augmenting Topologies
  (NEAT) improved genetic algorithms by
  making including complexification and
  speciation in the algorithm
• Alignment during crossover through
• Speciation protects complexification
     Competitive Coevolution
• Fitness signifies only the relative strength
  of solutions
• Ideally solutions evolve in an “arms race”
  towards better performance
• Interesting strategies only evolve if the
  arms race continues for a large number of
        Progress in Evolution
• Evolution finds simplest strategy that can win
• Strategies switch back and forth opportunistically
  between variations, losing some abilities and
  attaining others
• Techniques: “Hall of Fame”, Fitness Sharing,
  Pareto Coevolution – finding the best learners
  and best teachers in a population
• These techniques allow sustaining the arms
  race longer but do not encourage continual
  evolution – creating new solutions that maintain
  existing capabilities.
• Complexification elaborates strategies by
  adding new dimensions, enabling
  indefinite progress
         NeuroEvolution of
       Augmenting Topologies
• Using historical markings to line up genes
  for crossover
• Protecting topological evolution through
• Minimization of topologies throughout
         Genetic Encoding
• A genome includes a list of connecting
  genes, an in-node, an out-node, weight,
  expression enable bit and an innovation
Genetic Encoding
• Mutation in NEAT can change both connection
  weights and network structures.
• Connection weights mutate (usual NE algorithm)
• Structural mutation operates in two ways - add
  connection and add node – connection split,
  new in-weight of 1 out-weight same as old
  weight so functionality does not change initially
    Structural Mutation in NEAT

The connection between the first node and the old node is given the weight 1
and the connection between the new node and the second is given the same
weight of the connection being split.
         Historical Markings
• If the two above mutations occur
  consecutively the innovation numbers
  associated with the new genes allow the
  system to keep track of the histories of
  every gene in the system
Crossover using innovation numbers

                    Historical markings are lined up
                    and randomly chosen for the
                    Genes that do not match are
                    inherited from the more fit parent
                    or randomly.
                    Disabled genes are inherited at
• It turns out that a population of varying complexities
  cannot maintain topological innovations on its own.
• Because smaller structures optimize faster than larger
  structures, and adding nodes and connections usually
  initially decreases the fitness of the network, recently
  augmented structures have little hope of surviving more
  than one generation even though the innovations they
  represent might be crucial towards solving the task in the
  long run.
• The solution is to protect innovation by speciating the
• NEAT speciates the population so that individuals
  compete primarily within their own niches instead of with
  the population at large. This way, topological innovations
  are protectedand have time to optimize their structure
  before they have to compete with other niches in the
• Speciation prevents bloating of genomes: Species with
  smaller genomes survive as long as their fitness is
  competitive, ensuring that small networks are not
  replaced by larger ones unnecessarily.
• Protecting innovation through speciation follows the
  philosophy that new ideas must be given time to reach
  their potential before they are eliminated.
   Distance between networks

    E is the number of excess genes
    D is the number of disjoint genes
    W is the average weight difference of matching genes
    N is the number of genes in the larger genome

If the distance of from a test gene to a randomly chosen member
of a species is less than the current compatibility threshold the
test gene is placed in the species
                    Fitness Sharing
Organisms in the same species must share the fitness of their niche. The
adjusted fitness f’ for organism i is calculated according to its distance from
every other organism j in the population where sh is set to 0 when the distance
is above the threshold and 1 otherwise.

  The factor                      reduces to the number of organisms in the
  same species as organism i

Every species is assigned a potentially different number of offspring in
proportion to the sum of adjusted fitnesses f’i of its member organisms.
Species reproduce by first eliminating the lowest performing members from the
population. The entire population is then replaced by the offspring of the
remaining organisms in each species.
            A Run models
        increasing complexity
• Run begins with a uniform population with
  no hidden nodes that differ in the random
  assignments of weights
• The gradual production of increasingly
  complex structures constitutes the model
  of complexification
        Coevolution Domain
• Domain where it is possible to develop a
  wide range increasingly sophisticated
• Sophistication can be readily measured.
• A coevolution domain is particularly
  appropriate because a sustained arms
  race should lead to increasing
   Duel Robot Domain

                             Food is represented by sandwiches
                             and robots by the circles representing
                             sensors and arrows representing
                             directions. The objective is to forage to
                             obtain a higher level of energy than the
                             opponent and then collide with it

                             The duel domain supports
                             sophisticated strategies that are

             The Robot ANN

Each has five robot finder sensors and five to sense food. Each
has two wheels controlled by separate motors and can read the
opponents energy level and has a wall sensor. Energy is consumed
in proportion to the amount applied to the motors.
       About the duel domain
• The observed state taken by the sensors
  does not include the internal state of the
• The next observed state depends on the
  decision of the opponent
• It is necessary for the robots to learn to
  predict what the opponent is likely to do.
        Opponent Sampling
• Evolve two separate populations.
• In each generation, each population is
  evaluated against an intelligently chosen
  sample of networks from the other
• The population currently being evaluated
  is called the host population, and the
  population from which opponents are
  chosen is called the parasite population
• Each host was evaluated against the four highest
  species’ champions. They are good opponents because
  they are the best of the best species, and they are
  guaranteed to be diverse because their distance must
  exceed the species threshold
• Another eight opponents were chosen randomly from a
  Hall of Fame composed of all generation The Hall of
  Fame ensures that existing abilities need to be
  maintained to obtain a high fitness.
• Together speciation, fitness sharing, opponent sampling
  and Hall of Fame comprise an effective competitive
  coevolution methodology.
    Population and Competition

• Each population had 256 networks
• Host networks received 1 point for each
  win and 0 for losing
• Each host was evaluated in 24 games (12
  opponents x 2 games each)
• Of the 12, 4 were species champions and
  8 were Hall of Famers.
      Difficulty of tournaments
• For example, if strategy A defeats 499 out of 500
  opponents, and B defeats 498, counting will
  designate A as superior to B even if B defeats A
  in a direct comparison.
• In order to decisively track strategic innovation,
  we need to identify dominant strategies - those
  that defeat all previous dominant strategies.
• This way, we can make sure that evolution
  proceeds by developing a progression of strictly
  more powerful strategies, instead of e.g.
  switching between alternative ones.
       Dominance Tournament
• A run returns record of every generation champion from
  both populations
• A network a is superior to a network b if a wins more
  games than b out of 288 total games with different food
• A generational champion is the winner of a 288 game
  comparison between the host and parasite champions of
  a single generation
• The first dominant strategy d1 is the first generation
• The dominant strategy dj, j> 1 is a generation champion
  such that for all i < j dj is superior to di
• Process is called a dominance tournament
      Features of a dominance
• Fewer games than other tournaments
• Allows identification of a sequence of
  increasingly sophisticated strategies
  (dominant individuals)
     Results – 33 Evolutions
• Each of the 33 evolution runs took
  between 5 and 10 days on a 1GHz
  Pentium III processor, depending on the
  progress of evolution and sizes of the
  networks involved.
         Measuring Complexity
• Define complexity as the number of nodes and
  connections in a network: The more nodes and
  connections there are in the network, the more complex
  behavior it can potentially implement.
• The results were analyzed to answer three questions:
• (1) As evolution progresses does it also continually
• (2) Does such complexification lead to more
  sophisticated strategies?
• (3) Does complexification allow better strategies to be
  discovered than does evolving fixed-topology networks?
    Emergence of Complexity

The hashed lines represent the average over 13 runs of the structure of
the highest dominant network in each generation. A hash mark appears
each time a new dominant network emerged. The two other lines
represent the average over five runs of the most and least complex
networks without fitness selection (random assignment of fitness). This
shows that without fitness a wide range of complexity is evolved.
        Emergence of Strategies
           Dn – network with dominance level n
           Sk – best network in species S at generation k
           hl – l’th hidden node to arise from a structural mutation

Begin with S100: Mature no hidden node strategy, followed even when the
opponent had more energy leaving it vulnerable to attack
S200: Evolved a resting strategy. Not a complexification
S267: h22 appeared. Switched between resting and all out attack
S315: improved ability to attack at appropriate times.
Best Complexifying Network

     11 hidden nodes and 202 connections
Fixed-Topology vs Complexification

Complexifying Evolution only searches higher-dimensional structures
that are elaborations of known good lower-dimensional structures. The
values of the existing genes have already been optimized over
preceding generations. This may mean that the search in the higher-
dimensional space is starting in a position of some advantage
compared to a purely random position in that space. This may explain
why this method is able to find solutions that fixed topology coevolution

To top