1 Initial feasible origin: 1. Set values of original variables to zero. 2. Set values of slack variables according to the dictionary. The problems we have.

Presentation on theme: "1 Initial feasible origin: 1. Set values of original variables to zero. 2. Set values of slack variables according to the dictionary. The problems we have."— Presentation transcript:

1 1 Initial feasible origin: 1. Set values of original variables to zero. 2. Set values of slack variables according to the dictionary. The problems we have solved so far always had an initial feasible origin. 1.Make up one problem which does NOT have an initial feasible origin. 2.What do you look at in the initial problem to tell if there is an initial feasible origin or not?

2 2 Office hours: MWR 4:20-5:30 inside or just outside Elliott 162- tell me in class that you would like to attend. For those of you who cannot stay: MWR: 1:30-2:30pm. But let me know 24 hours in advance you would like me to come in early. We will meet outside Elliott 162- Please estimate the length of time you require. I am also very happy to provide e-mail help: (wendym@csc.UVic.ca). Important note: to ensure your e-mail gets through the spam filter, use your UVic account. I answer all e-mails that I receive from my students.

3 3 http://wiki.hsc.com/wiki/Main/ConvexOptimization George Dantzig: Founder of the Simplex method.

4 4 In Dantzig’s own words: During my first year at Berkeley I arrived late one day to one of Neyman's classes. On the blackboard were two problems which I assumed had been assigned for homework. I copied them down. A few days later I apologized to Neyman for taking so long to do the homework - the problems seemed to be a little harder to do than usual. I asked him if he still wanted the work. He told me to throw it on his desk. I did so reluctantly because his desk was covered with such a heap of papers that I feared my homework would be lost there forever.

5 5 About six weeks later, one Sunday morning about eight o'clock, Anne and I were awakened by someone banging on our front door. It was Neyman. He rushed in with papers in hand, all excited: "I've just written an introduction to one of your papers. Read it so I can send it out right away for publication." For a minute I had no idea what he was talking about. To make a long story short, the problems on the blackboard which I had solved thinking they were homework were in fact two famous unsolved problems in statistics.

6 John von Neumannn established the theory of duality also in 1947. John von Neumann: Dec. 28, 1903 – Feb. 8, 1957. He made major contributions to a vast number of fields, including mathematics (set theory, functional analysis, ergodic theory, geometry, numerical analysis, and many other mathematical fields), physics (quantum mechanics, hydrodynamics, and fluid dynamics), economics (game theory), computer science (linear programming), and statistics. He is generally regarded as one of the greatest mathematicians in modern history. http://en.wikipedia.org/wiki/John_von_Neumann 6

7 His contributions to computer science: mergeSort. established game theory as a mathematical discipline. contributions to mathematics of economics. introduced ideas leading to Karmarker’s algorithm. developed a fast method for making pseudorandom numbers. first to describe a computer architecture where the data and the program are both stored in the computer's memory in the same address space. designed first template of a computer virus. 7

8 Other contributions: helped design nuclear bomb. member of committee responsible Hiroshima and Nagasaki as the first targets of the atomic bomb. oversaw computations related to the expected size of the bomb blasts, estimated death tolls, and the distance above the ground at which the bombs should be detonated for optimum shock wave propagation and thus maximum effect. 8

9 Popularity sky rocketed with it was realized it could be used to solve problems in production management formerly tackled by hit-or-miss or intuitive approaches. Awareness grew of advantages of stating decision problems in well-defined, clear cut terms. The Nobel prize in economics was awarded in 1975 to the mathematician Leonid Kantorovich (USSR) and the economist Tjalling Koopmans (USA) for their contributions to the theory of optimal allocation of resources. 9

10 Kantorovich: Jan. 19, 1912,- April 7, 1986. Koopmans: Aug. 28, 1910- Feb. 26, 1985. 10

11 Leonid Khachiyan: May 3, 1952- April 29, 2005. The Simplex method runs very fast in practice but has exponential worst case time. In 1979, Khachiyan presented the first polynomial time algorithm (the ellipsoid method) to solve linear programming problems (but it was not efficient in practice). Leonid Khachiyan was a Soviet mathematician of Armenian descent who taught Computer Science at Rutgers University. 11

12 Karmarker in 1984 presented the first reasonably efficient algorithm for solving linear programs in polynomial time. Our text: copyright 1983. At the time he discovered the algorithm, Narendra Karmarkar was employed by AT&T. AT&T applied for a patent on Karmarkar's algorithm. This left many mathematicians uneasy, such as Ronald Rivest (himself one of the holders of the patent on the RSA algorithm), who expressed the opinion that research proceeded on the basis that algorithms should be free. The patent was eventually granted but proved to be of limited commercial value. http://tetrahedral.blogspot.ca/2010/10/karmarkar-algorithm-algorithms-should.html Narendra Karmarkar 1957- 12

13 13 The Simplex Method (algorithm we are using) How can we solve problems which are not in standard form? How can we prove that the solution is optimal at the end? How can this be implemented in the computer? How can numerical round off errors be mitigated? How long does it take in the worst case? What can we do if we do not have an initial feasible solution?

14 Can we choose an entering variable to make the algorithm terminate faster? How should we try to do that? Can we prove that there always exists a basic feasible solution for feasible problems? If there is more than one optimal solution, what can the solution space look like? How can we analyze problems given that there could be small changes to the constraints (without starting from scratch)? 14

15 15 How long does it take in the worst case? Problem:As described, the Simplex method could end up in an infinite loop! For the following linear programming problem, the pivot variable is chosen to be the one with the largest positive coefficient in the z row. After 6 pivots, the dictionary is the same as the one we started with. This results in an infinite loop. The tightest constraint corresponding to the variable with smallest subscript is chosen to enter.

16 16 Input file: 4 3 10 -57 -9 -24 0 0.5 -5.5 -2.5 9 0 0.5 -1.5 -0.5 1 1 1 0 0 0 *************** Problem 1 *************** Phase 1: Input dictionary. X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4 X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4 --------------------------------------------------- z = 0.00+ 10.00 X1 -57.00 X2 - 9.00 X3 -24.00 X4

17 17 The initial dictionary: X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4 X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4 ------------------------------------------------------ Z = -0.00+ 10.00 X1 - 57.00 X2 - 9.00 X3 - 24.00 X4 X1 enters. X5 leaves. z = -0.000000 After 1 pivot: X1 = 0.00+ 11.00 X2 + 5.00 X3 - 18.00 X4 - 2.00 X5 X6 = 0.00- 4.00 X2 - 2.00 X3 + 8.00 X4 + 1.00 X5 X7 = 1.00- 11.00 X2 - 5.00 X3 + 18.00 X4 + 2.00 X5 ------------------------------------------------------- z = -0.00+ 53.00 X2 + 41.00 X3 - 204.00 X4 - 20.00 X5 X2 enters. X6 leaves. z = -0.000000

18 18 After 2 pivots: X1 = 0.00- 0.50 X3 + 4.00 X4 + 0.75 X5 - 2.75 X6 X2 = 0.00- 0.50 X3 + 2.00 X4 + 0.25 X5 - 0.25 X6 X7 = 1.00+ 0.50 X3 - 4.00 X4 - 0.75 X5 + 2.75 X6 ------------------------------------------------------ z = -0.00+ 14.50 X3 - 98.00 X4 - 6.75 X5 - 13.25 X6 X3 enters. X1 leaves. z = -0.000000 After 3 pivots: X3 = 0.00- 2.00 X1 + 8.00 X4 + 1.50 X5 - 5.50 X6 X2 = 0.00+ 1.00 X1 - 2.00 X4 - 0.50 X5 + 2.50 X6 X7 = 1.00- 1.00 X1 + 0.00 X4 + 0.00 X5 + 0.00 X6 ------------------------------------------------------ z = -0.00- 29.00 X1 + 18.00 X4 + 15.00 X5 - 93.00 X6 X4 enters. X2 leaves. z = -0.000000

19 19 After 4 pivots: X3 = 0.00+ 2.00 X1 - 4.00 X2 - 0.50 X5 + 4.50 X6 X4 = 0.00+ 0.50 X1 - 0.50 X2 - 0.25 X5 + 1.25 X6 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X5 + 0.00 X6 ----------------------------------------------- z = -0.00- 20.00 X1 - 9.00 X2 + 10.50 X5 - 70.50 X6 X5 enters. X3 leaves. z = -0.000000 After 5 pivots: X5 = 0.00+ 4.00 X1 - 8.00 X2 - 2.00 X3 + 9.00 X6 X4 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X6 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X6 ----------------------------------------------- z = -0.00+ 22.00 X1 - 93.00 X2 - 21.00 X3 + 24.00 X6 X6 enters. X4 leaves. z = -0.000000

20 20 After 6 pivots: X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4 X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4 ------------------------------------------------------ z = -0.00+ 10.00 X1 - 57.00 X2 - 9.00 X3 - 24.00 X4 This is the same as: The initial dictionary: X5 = 0.00- 0.50 X1 + 5.50 X2 + 2.50 X3 - 9.00 X4 X6 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X4 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X4 ------------------------------------------------------ z = -0.00+ 10.00 X1 - 57.00 X2 - 9.00 X3 - 24.00 X4

21 21 After 5 pivots: X5 = 0.00+ 4.00 X1 - 8.00 X2 - 2.00 X3 + 9.00 X6 X4 = 0.00- 0.50 X1 + 1.50 X2 + 0.50 X3 - 1.00 X6 X7 = 1.00- 1.00 X1 + 0.00 X2 + 0.00 X3 + 0.00 X6 ------------------------------------------------------- z = -0.00+ 22.00 X1 - 93.00 X2 - 21.00 X3 + 24.00 X6 Using smallest subscript rule instead for the entering variable: X1 enters. X4 leaves. z = -0.000000 After 6 pivots: X5 = 0.00+ 4.00 X2 + 2.00 X3 - 8.00 X4 + 1.00 X6 X1 = 0.00+ 3.00 X2 + 1.00 X3 - 2.00 X4 - 2.00 X6 X7 = 1.00- 3.00 X2 - 1.00 X3 + 2.00 X4 + 2.00 X6 ---------------------------------------------------- z = -0.00-27.00 X2 + 1.00 X3 - 44.00 X4 - 20.00 X6 X3 enters. X7 leaves. z = -0.000000

22 22 After 7 pivots: X5 = 2.00- 2.00 X2 - 4.00 X4 + 5.00 X6 - 2.00 X7 X1 = 1.00+ 0.00 X2 + 0.00 X4 + 0.00 X6 - 1.00 X7 X3 = 1.00- 3.00 X2 + 2.00 X4 + 2.00 X6 - 1.00 X7 ---------------------------------------------------- z = 1.00-30.00 X2 -42.00 X4 -18.00 X6 - 1.00 X7 The optimal solution: 1.000000 X1 = 1.0000 X2 = 0.0000 X3 = 1.0000 X4 = 0.0000 X5 = 2.0000 X6 = 0.0000 X7 = 0.0000

23 23 What caused the infinite loop? For each basis, there is a unique dictionary and value for z. If the value for z increases at each iteration, no dictionary can be repeated and hence, there will be no infinite loop. Degenerate solution: has at least one basic variable with value 0. Degenerate pivot: pivot that does not increase the value for z. Infinite loop: sequence of degenerate pivots.

24 24 How many different bases can there be if the original problem has n variables and m equations? Example given: n=4, m=3. How can we prove that for each choice of basis, the dictionary for that choice of basis has the same equations? Or equivalently, if the rows/columns are listed in sorted order according to the subscripts of the variables then there is a unique dictionary for each basis.

25 25 Theorem: Any two dictionaries with the same choice of basis must be the same equations. Proof (by contradiction). Assume not. Consider two dictionaries D and D* which have the same basis. B is the set of subscripts of the basic variables.

26 26

27 27 We get from one dictionary to another by: 1. Adding the same thing to both sides of an equation. 2. Multiplying both sides of an equation by a non-zero constant. 3. Adding a constant multiple of one equation to another one. If you do these operations to a set of equations, the set of solutions is preserved.

28 28 Operation 3: (1)f(x) = b 1 (2)g(x) = b 2 (3)g(x) + c * f(x) = c* b 1 + b 2 Equations (1) and (2) have exactly the same solutions as equations (1) and (3). If x is a solution to (1) and (2) then clearly it satisfies (3) and hence is a solution to (1) and(3). If x is a solution to (1) and (3), then by (1), c * f(x) = c * b 1 and hence from (3) g(x) = b 2 so it also satisfies (1) and (2). Equations (2) and (3) do not imply (1) if c=0.

29 29

30 30 Choose one non-basic variable x k and set x k =t and set other non-basic variables to zero: Dictionary D: x i = b i - a ik t for each i ∈ B. ----------------------------------------------------------- z = v + c k t Dictionary D* with the same basis B: x i = b i * - a ik * t for each i ∈ B. ----------------------------------------------------------- z = v * + c k * t

31 31 X5 = 1 - 2.5 X1 + 5.5 X2 - 2.5 X3 - 9.0 X4 X6 = 2 - 1.5 X1 - 1.5 X2 + 0.5 X3 - 1.0 X4 D X7 = 3 - 1.0 X1 + 0.0 X2 + 0.0 X3 + 0.0 X4 -------------------------------------------- Z = 5 +10.0 X1 - 7.0 X2 + 9.0 X3 - 4.0 X4 X5 = b 5 - a 5,1 X1 - a 5,2 X2 - a 5,3 X3 - a 5,4 X4 X6 = b 6 - a 6,1 X1 - a 6,2 X2 - a 6,3 X3 - a 6,4 X4 D* X7 = b 7 - a 7,1 X1 - a 7,2 X2 - a 7,3 X3 - a 7,4 X4 --------------------------------------------- Z = v + c 1 X1 + c 2 X2 + c 3 X3 + c 4 X4 If D and D* both have X1=X2=X3=X4=0, X5=1, X6=2 and X7=3 as solutions, what does this say about D*? What if they both also have X2=X3=X4=0, X1=1 and X5= -1.5, X6= 0.5, and X7= 2?

32 32 What can we conclude from setting t=0? Dictionary D: x i = b i - a ik t for each i ∈ B. ----------------------------------------------------------- z = v + c k t Dictionary D* with the same basis B: x i = b i * - a ik * t for each i ∈ B. ----------------------------------------------------------- z = v * + c k * t

33 33 What can we conclude from setting t=1? Dictionary D: x i = b i - a ik t for each i ∈ B. ----------------------------------------------------------- z = v + c k t Dictionary D* with the same basis B: x i = b i * - a ik * t for each i ∈ B. ----------------------------------------------------------- z = v * + c k * t

34 34 What can we conclude from trying all choices for a non-basic variable x k ? Dictionary D: x i = b i - a ik t for each i ∈ B. ----------------------------------------------------------- z = v + c k t Dictionary D* with the same basis B: x i = b i * - a ik * t for each i ∈ B. ----------------------------------------------------------- z = v * + c k * t

35 35 What can we do to stop our programs from going into an infinite loop?