# System Modeling and Optimization - Chapter 10 Dynamic Optimization

Document Sample

```					  System Modeling and Optimization
Chapter 10: Dynamic Optimization

Dr. Marco Laumanns
Institute for Operations Research, ETH Zurich

11 Nov 2009

Program Remaining

Date         Topic                                          Who           Script
11 Nov       Dynamic Optimization                           Laumanns      Chapter 10
18 Nov       Optimization at Schindler                      Finschi
25 Nov       Stochastic Optimization                        Laumanns      Chapter 11
02 Dec       Markov Chains                                  Laumanns      Chapter 12
09 Dec       Decision Support Models and Optimi-            Costa
e
zation Tools in Supply Chain at Nestl´
16 Dec       Stochastic Dynamic Optimization                Laumanns      Chapter 13
Case Study: Resource Management in             Guarisco      Appendix A3
Electricity Networks

System Modeling and Optimization    10 – Dynamic Optimization             11 Nov 2009       2 / 42
Inventory Management                     General Dynamic Optimization Problem             Dynamic Programming

Chapter 10: Dynamic Optimization

1   Example: Inventory Management

2   General Dynamic Optimization Problem

3   Dynamic Programming

System Modeling and Optimization             10 – Dynamic Optimization              11 Nov 2009        3 / 42

Inventory Management                     General Dynamic Optimization Problem             Dynamic Programming

What is Dynamic Optimization?

So far: static optimizaton problems
Goal: Optimal one-time decision

Now: dynamic optimization problems
Decision to be made in time steps (stages) t = 1, 2, . . .
Earlier decisions condition later choices (dependencies)
Goal: determine optimal decision sequence (policy) π = (x1 , x2 , . . .)
Assumption: future entirely predictable
(If not, then stochastic dynamic optimization, see Chapter 13.)

x1                         x2
s1                            s2                             s3        ···
t=1                           t=2

System Modeling and Optimization             10 – Dynamic Optimization              11 Nov 2009        4 / 42
Inventory Management                General Dynamic Optimization Problem        Dynamic Programming

Learning Objectives

Chapter 10: Dynamic Optimization
to understand the general setting and structure of dynamic
optimization problems
to learn diﬀerent modeling (and solution) approaches
1   formulation as simultaneous problem
2   dynamic programming
to get to know some fundamental models in inventory
management and to understand how they relate to dynamic
optimization

System Modeling and Optimization        10 – Dynamic Optimization         11 Nov 2009        5 / 42

Inventory Management                General Dynamic Optimization Problem        Dynamic Programming

1   Example: Inventory Management
Purpose, Cost Elements, and Trade-oﬀs
The EOQ Model

2   General Dynamic Optimization Problem
A Dynamic Lot Sizing Model
General Formulation
Formulation as a Simultaneous Problem
Lot Sizing as Mixed-Integer Linear Program (MILP)
Multi-Period Planning with Linear Programming

3   Dynamic Programming
Transition Graph
Dynamic Programming
Wagner-Whitin Model

System Modeling and Optimization        10 – Dynamic Optimization         11 Nov 2009        6 / 42
Inventory Management                General Dynamic Optimization Problem        Dynamic Programming

Purpose of Inventories

Inventory Costs
Storage costs (labor, physical space)
Invested capital (opportunity costs)
Depreciation (loss of value, shrinkage)
Insurance and taxes

So why keeping inventories?
1 demand variations
or other uncertainties (e.g. lead times, production quantities)
2   non-linearity of sourcing costs
ﬁxed order costs, wholesale rebates, economies of scale in transport
increase in ﬂexibility of production planning

System Modeling and Optimization        10 – Dynamic Optimization         11 Nov 2009        8 / 42

Inventory Management                General Dynamic Optimization Problem        Dynamic Programming

Inventory Management: Cost Elements

Goal: to determine optimal inventory policy
(= how many products to order/produce xt at each time step t)
Cost elements
1 Inventory costs
Storage costs (labor, physical space)
Invested capital (opportunity costs)
Depreciation (loss of value, shrinkage)
2   Purchase or production costs
Price or production costs per item (quantity dependent)
Order or setup costs (independent of the quantity)
3   Back order costs (relevant mostly under stochastic demand)
Lost sales
Penalties
Main trade-oﬀ in the deterministic case:
bigger lots → lower setupDynamic Optimization inventory costs 2009
System Modeling and Optimization10 – costs, higher                 11 Nov                    9 / 42

smaller lots → higher setup costs, lower inventory costs
Inventory Management                General Dynamic Optimization Problem          Dynamic Programming

Economic Order Quantity (EOQ Model)

Assumption
Fixed demand rate D [items/time]

c     item cost                [\$/item]
S      ﬁxed order costs         [\$/order]
i    holding cost factor      [1/time]

Total cost / time unit:

TC (q) = item cost / time
+ order cost / time                   Goal
+ holding cost / time                 Find optimal order quantity q ∗ that
minimizes total average cost per time

System Modeling and Optimization        10 – Dynamic Optimization          11 Nov 2009        11 / 42

Inventory Management                General Dynamic Optimization Problem          Dynamic Programming

Economic Order Quantity (EOQ Model)

System Modeling and Optimization        10 – Dynamic Optimization          11 Nov 2009        12 / 42
Inventory Management                General Dynamic Optimization Problem              Dynamic Programming

1    Example: Inventory Management
Purpose, Cost Elements, and Trade-oﬀs
The EOQ Model

2    General Dynamic Optimization Problem
A Dynamic Lot Sizing Model
General Formulation
Formulation as a Simultaneous Problem
Lot Sizing as Mixed-Integer Linear Program (MILP)
Multi-Period Planning with Linear Programming

3    Dynamic Programming
Transition Graph
Dynamic Programming
Wagner-Whitin Model

System Modeling and Optimization        10 – Dynamic Optimization              11 Nov 2009        13 / 42

Inventory Management                General Dynamic Optimization Problem              Dynamic Programming

Dynamic Lot Sizing Problem

Example 10.1
A company needs 300 items of raw material for its production over the
next T = 6 time steps as follows:
Time step t            1       2        3       4    5    6
Net demand dt         10      20      110      50   70   40

Holding costs 0.5 per item                                  Costs per item constant
Fixed costs 100 per order                                   Lead time zero

Determine an optimal order policy, i.e., the amounts xt to order at
each time t ∈ {1, . . . , 6} that minimizes total costs.1

1
o
The data of this example is taken from P. Sch¨nsleben (2000): Integral
Logistics Management.       Boca Raton, St. Lucie Press, p. 466f.
System Modeling and Optimization       10 – Dynamic Optimization      11 Nov 2009                 15 / 42
Inventory Management                      General Dynamic Optimization Problem              Dynamic Programming

The Dynamics of the Inventory System

The state of the system at time t is the inventory level st
The decision at time t is the order quantity xt
The transition to the next state st+1 is given by

st+1 = ft (st , xt ) = st + xt − dt

and incurs the stage cost

100 falls xt > 0
gt (st , xt ) = 0.5st +
0   falls xt = 0

x1                          x2
s1                             s2                             s3        ···
g1 (s1 , x1 )                  g2 (s2 , x2 )

t=1                            t=2

System Modeling and Optimization              10 – Dynamic Optimization              11 Nov 2009        16 / 42

Inventory Management                      General Dynamic Optimization Problem              Dynamic Programming

Dynamic Optimization: Ingredients

State st at the beginning of every time step t
+ Set of feasible states St .
Decision xt (action) at every time step t
+ Set of feasible decisions Xt (st ).
Cost gt (st , xt ) in state st when taking decision xt
Transition function ft , which determines next state as

st+1 = ft (st , xt )

x1                          x2
s1                             s2                             s3        ···
g1 (s1 , x1 )                  g2 (s2 , x2 )

t=1                            t=2

System Modeling and Optimization              10 – Dynamic Optimization              11 Nov 2009        18 / 42
Inventory Management                     General Dynamic Optimization Problem                 Dynamic Programming

General Dynamic Optimization Problem

The sequence of decisions π = {x1 , x2 , . . . , xT } is called a policy.
Each policy gives rise to a discrete-time dynamic process via the state
transition function ft as

st+1 = ft (st , xt ),

and incurs the total cost
T
Jπ (s1 ) =           gt (st , xt ).
i=1

The goal is to ﬁnd

J ∗ (s1 ) := min Jπ (s1 )
π

System Modeling and Optimization              10 – Dynamic Optimization                11 Nov 2009         19 / 42

Inventory Management                     General Dynamic Optimization Problem                 Dynamic Programming

General Dynamic Optimization Problem

T
∗
J (s1 ) =              min                     gt (st , xt )
(xt ,st+1 )T
t=1       t=1
subject to                         st+1 = ft (st , xt )      (t = 1, . . . , T )
st+1 ∈ St+1               (t = 1, . . . , T )
xt ∈ Xt (st )         (t = 1, . . . , T )

(Minimization jointly and simultaneously with respect to the actual
decision variables xt and the (auxiliary) state variables st .)

x1                             x2
s1                            s2                              s3          ···
g1 (s1 , x1 )                   g2 (s2 , x2 )

t=1                              t=2

System Modeling and Optimization              10 – Dynamic Optimization                11 Nov 2009         21 / 42
Inventory Management                       General Dynamic Optimization Problem                 Dynamic Programming

Formulation as Mixed-Integer Linear Program
(MILP)

T
minimize(xt ,st+1 )T                        0.5st + 100zt                                         (1)
t=1
t=1
subject to                            st+1 = st + xt − dt               (t = 1, . . . , T )       (2)
st+1 ≥ 0                          (t = 1, . . . , T )       (3)
0 ≤ xt ≤ M · zt                   (t = 1, . . . , T )       (4)
zt ∈ {0, 1}                       (t = 1, . . . , T )       (5)

Remark: Inequality (4) called Big M constraint.

System Modeling and Optimization                10 – Dynamic Optimization                11 Nov 2009            23 / 42

Inventory Management                       General Dynamic Optimization Problem                 Dynamic Programming

Multi-Period Planning with Linear Programming

Example 5.3 (Investo Real Estates Inc.)
Cash-ﬂow of 3 possible investments (to be chosen as fractions 0 ≤ xi ≤ 1)
0                6         12          18         24       30         36 Mt.
1 (South)                 -3’000           -1’000     -1’800       400       1’800    1’800         5’500
2 (Old Complex)           -2’000            -500       1’500      1’500      1’500     200         -1’000
3 (Hotel)                 -2’000           -2’000     -1’800      1’000      1’000    1’000         6’000

Available cash-ﬂow from previous investments:
0            6     12          18      24       30       36 Mt.
2’000         500   400          380     360      340       300

Interest rate savings: 3.0 % per 6 months
Interest rate loan: 3.5 % per 6 months

Further time-dependent auxiliary decision variables
yt : loan, zt : savings, st : remaining cash
System Modeling and Optimization                10 – Dynamic Optimization                11 Nov 2009            25 / 42
Inventory Management                 General Dynamic Optimization Problem                  Dynamic Programming

Multi-Period Investment Problem

Example 5.3 (Investo Real Estates Inc.)
Maximize z7

subject to cash-ﬂow balance equations

3000x1    + 2000x2   + 2000x3                               − y1   + z1   =    2000
1000x1    + 500x2    + 2000x3    + 1.035y1     − 1.03z1     − y2   + z2   =     500
1800x1    − 1500x2   + 1800x3    + 1.035y2     − 1.03z2     − y3   + z3   =     400
−    400x1    − 1500x2   − 1000x3    + 1.035y3     − 1.03z3     − y4   + z4   =     380
−   1800x1    − 1500x2   − 1000x3    + 1.035y4     − 1.03z4     − y5   + z5   =     360
−   1800x1    − 200x2    − 1000x3    + 1.035y5     − 1.03z5     − y6   + z6   =     340
−   5500x1    + 1000x2   − 6000x3    + 1.035y6     − 1.03z6            + z7   =     300

Exercise 10.1
Write this problem as a dynamic optimization problem by specifying the initial state
st , the transition functions ft , the cost functions gt , the feasible state sets St , and
the feasible decision sets Xt .

System Modeling and Optimization         10 – Dynamic Optimization                  11 Nov 2009        26 / 42

Inventory Management                 General Dynamic Optimization Problem                  Dynamic Programming

1   Example: Inventory Management
Purpose, Cost Elements, and Trade-oﬀs
The EOQ Model

2   General Dynamic Optimization Problem
A Dynamic Lot Sizing Model
General Formulation
Formulation as a Simultaneous Problem
Lot Sizing as Mixed-Integer Linear Program (MILP)
Multi-Period Planning with Linear Programming

3   Dynamic Programming
Transition Graph
Dynamic Programming
Wagner-Whitin Model

System Modeling and Optimization         10 – Dynamic Optimization                  11 Nov 2009        27 / 42
Inventory Management                    General Dynamic Optimization Problem                Dynamic Programming

Transition Graph Representation
Nodes: states st
Arcs: decisions xt (arc weight = cost g (st , xt ) of decision)
s1               s2                                          sT           sT +1
C                C                                           C

C −1              C −1                                       C −1

.                .                ···                       .
.                .                                          .
0                0                                           0             0

time t

System Modeling and Optimization             10 – Dynamic Optimization               11 Nov 2009        29 / 42

Inventory Management                    General Dynamic Optimization Problem                Dynamic Programming

Principle of Optimality
Observation: A state decouples past and future
Theorem (Bellman 1957)
∗            ∗
Let π ∗ = {x1 , . . . , xT } be an optimal policy.
For each state st that is reached under π ∗ , the

tail policy       ∗                   ∗
πt = {xt∗ , . . . , xT }

is optimal for the tail problem. (Like in the shortest path problem!)

Proof idea:
If a diﬀerent tail policy πt was better, the combined policy
∗            ∗
{x1 , . . . , xt−1 , xt , . . . , xT }

would be better than the π ∗ . Contradiction.
System Modeling and Optimization             10 – Dynamic Optimization               11 Nov 2009        31 / 42
Inventory Management                    General Dynamic Optimization Problem                   Dynamic Programming

Dynamic Programming Algorithm

Idea
Compute the optimal tail cost Jt (st ) “backwards”.

Initialization Set JT +1 (sT +1 ) := 0
Iteration For each t from T to 1 set for each state st
immediate cost             optimal tail cost

Jt (st ) :=      min            g (st , xt )       + Jt+1 (ft (st , xt ))
xt ∈Xt (st )
next state

Theorem (Proof by induction)
The DP algorithm solves the dynamic optimization problem, and the
minimal cost is J ∗ (s1 ) = J1 (s1 ).
System Modeling and Optimization             10 – Dynamic Optimization                  11 Nov 2009        32 / 42

Inventory Management                    General Dynamic Optimization Problem                   Dynamic Programming

Shortest Path in the Transition Network

The dynamic optimization problem with ﬁnite state space can be
modeled as a shortest path problem in the transition graph:
Nodes: states st
Arcs: decisions xt (arc weight = cost of decision)
s1          s2                         sT         sT +1
C                 C                                          C

C −1              C −1                                       C −1

.                 .                ···                      .
.                 .                                         .
0                 0                                            0              0

System Modeling and Optimization             10 – Dynamic Optimization                    time t
11 Nov 2009        33 / 42
Inventory Management                            General Dynamic Optimization Problem                          Dynamic Programming

Solving the Lot Sizing Problem by DP
gt = 100 + 0.5st
xt > 0 (order)

gt = 0.5st
xt = 0 (no order)

st   inventory at t

75
110

135          55
70
0                             135
10                                                 20
0                              35           40
10
100                             20
175                        120                            100
100                             100                            100
0                             0                               0                                0

d4 =50                          d5 =70                        d6 =40
t =4                       t =5                               t =6                            t =7

System Modeling and Optimization                    10 – Dynamic Optimization                         11 Nov 2009         35 / 42

Inventory Management                            General Dynamic Optimization Problem                          Dynamic Programming

Wagner-Whitin Model

Lot sizing model studied in by Wagner and Whitin (1958)2
Theorem
An optimal policy must be a zero inventory ordering policy, i.e,

st · xt = 0 ∀t ∈ {1, . . . , T }

Proof idea
Investigate ILP formulation for any ﬁxed choice of the binary “order
indicator” variables zt .

2
H. M. Wagner and T. M. Whitin (1958): Dynamic Version of the Economic
Lot Size Model. Management Science 5, 89–96.
System Modeling and Optimization                    10 – Dynamic Optimization                         11 Nov 2009         36 / 42
Inventory Management                            General Dynamic Optimization Problem                                  Dynamic Programming

Mixed-Integer Linear Program
T
Minimize                          0.5st + 100z1
t=1
subject to                  st+1 = st + xt − dt                             (t = 1, . . . , T )
xt ≥ 0                                          (t = 1, . . . , T )
st ≥ 0                                          (t = 1, . . . , T )
s1 = 0
xt ≤ M · zt                                     (t = 1, . . . , T )
zt ∈ {0, 1}                                     (t = 1, . . . , T )

non-negative system with 2T variables and T equations
→ T basis variables and T non-basis variables
xt and st cannot be both zero
→ Either xt = 0 or st = 0 in any basis
System Modeling and Optimization                       10 – Dynamic Optimization                              11 Nov 2009               37 / 42

Inventory Management                            General Dynamic Optimization Problem                                  Dynamic Programming

Wagner-Whitin Model: All ZIO Policies
290 435 145                 290                                     gt = 100
xt > 0 (order)
435             270
gt = 0.5st
250          125            310                                                      xt = 0 (no order)
230
optimal policy (path)
13
5
5

315                                                                  Jt (st ) optimal value at state st
11

180           90            225               155
5
5

160               160                                     st   state (inventory) at time t
100

295                                             80
130                                           195
65            230                                    75
120
110                                    110
8
80

60
135     55
145             70
5
55

50                                                20
35         40
255                                             25                                           20
355             20 325       10            245               175                  120                     100
0                    0                    0                 0                    0                       0                    0
100                  100                  100                100                  100                     100
d1 =10               d2 =20            d3 =110               d4 =50               d5 =70                  d6 =40
t =1                 t =2                 t =3               t =4                 t =5                    t =6                 t =7

System Modeling and Optimization                       10 – Dynamic Optimization                              11 Nov 2009               38 / 42
Inventory Management                             General Dynamic Optimization Problem                                                Dynamic Programming

Optimality of ZIO Policies
290 435 145                 290                                               gt = 100
xt > 0 (order)
435             270                                        Proof Idea (using ILP formulation).
gt = 0.5st
250           125                                                                            x = 0 (no order)
310                                  For any non-ZIO policy πt
230                                        construct a ZIO-policy π
with same order times zt optimal policy (path)

13
= zt

5
and lower inventory costs
315                                                                           Jt (st ) optimal value at state st
gt (st , 0) < gt (st , 0)

11
180           90            225                    155           while order cost do not increase.

5
160                    160                                         st state (inventory) at time t
100

295                                                  80
130                                                195
65            230                                              75
120
110                                              110

80
60
86           43                                           140               135      55
133        80
66                          145                  70

55
0                                                   50                                                       20
10                                                                  0                         35          40
255                                                 10 5
2                                                   20
355 100         20 325       10            245                    175                       120                       100
100
0                    0                     0                      0                         0                         0                     0
100                  100                   100                    100                       100                       100
d1 =10                d2 =20                d3 =110                d4 =50                    d5 =70                    d6 =40
=1
zt1 = 1                  =0
zt2 = 2                   =1
zt3 = 3                zt4 = 4
=1                       =0
zt5 = 5                        =1
zt6 = 6               t =7

System Modeling and Optimization                         10 – Dynamic Optimization                                        11 Nov 2009                 39 / 42

Inventory Management                             General Dynamic Optimization Problem                                                Dynamic Programming

Inﬂuence of Inventory Restrictions
290 435 145                 290                                              gt = 100
xt > 0 (order)
435             270
gt = 0.5st
250          125            310                                                                 xt = 0 (no order)
230
optimal policy (path)
13
5

315                                                                            Jt (st ) optimal value at state st
11

180           90            225                    155
5

160                    160                                           st   state (inventory) at time t
100

295                                                  80
130                                                195
65            230                                              75
120
110                                             110
80

60
135     55
145                 70
55

50                                                        20
35          40
255                                                  25                                                  20
355             20 325        10            245                    175                       120                       100
0                    0                     0                      0                         0                         0                    0
100                   100                   100                    100                       100                       100
d1 =10                d2 =20             d3 =110                   d4 =50                    d5 =70                    d6 =40
t =1                 t =2                  t =3                   t =4                     t =5                       t =6                  t =7

Exercise!

System Modeling and Optimization                         10 – Dynamic Optimization                                        11 Nov 2009                 40 / 42
Inventory Management                General Dynamic Optimization Problem         Dynamic Programming

Further Aspects

Forward calculation
Instead of computing cost-to-go (tail costs) backwards, compute the
cost-to-arrive forwards!

The Planning Horizon Theorem
If there is some xt ∗ > 0 in the optimal policy, then the ﬁrst part
x1 , . . . , xt−1 will remain optimal even if horizon T is increased (e.g.
when more data become available).

System Modeling and Optimization        10 – Dynamic Optimization         11 Nov 2009        41 / 42

Inventory Management                General Dynamic Optimization Problem         Dynamic Programming

Summary Dynamic Optimization

General formulation of dynamic optimization problem
Model as LP or ILP sometimes possible
Solution via dynamic programming can be much more eﬃcient,
depending on how much the number of necessary states can be
reduced (decoupling future decision from past actions)

System Modeling and Optimization        10 – Dynamic Optimization         11 Nov 2009        42 / 42

```
DOCUMENT INFO
Shared By:
Categories:
Stats:
 views: 13 posted: 6/9/2010 language: English pages: 16
How are you planning on using Docstoc?