Bellman function in stochastic control and harmonic analysis

F. Nazarov^{∗}, S. Treil^{†}and A. Volberg ^{‡}^{§}

Abstract

The stochastic optimal control uses the differential equation of Bellman and its solution—the Bellman function. We show how the homonym function in harmonic analysis is (and how it is not) the same stochastic optimal control Bellman function. Then we present several creatures from Bellman’s Zoo: a function that proves the inverse Hölder inequality, as well as several other harmonic analysis Bellman functions and their corresponding Bellman PDE’s. Finally we translate the approach of Burkholder to the language of “our” Bellman function.

The goal of this paper is almost entirely methodological: we relate the ideas to each other, rather than presenting the new ones.

Contents

1 General approach: Bellman equation for stochastic optimal control 2 1.1 Stochastic optimal control . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Harmonic analysis Bellman function 6 2.1 Harmonic analysis problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 General remarks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 An example: A_{∞} weights and their associated Carleson measures . . . . . . . 7

2.3.1 “Common sense” approach . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3.2 Connections with stochastic control . . . . . . . . . . . . . . . . . . . 9 2.3.3 From supersolution to the theorem . . . . . . . . . . . . . . . . . . . 10 2.3.4 Continuous (infinitesimal) version . . . . . . . . . . . . . . . . . . . . 11 2.3.5 Solution of the Bellman equation . . . . . . . . . . . . . . . . . . . . 12

2.4 Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 3 New creatures from Bellman’s Zoo 14

∗Department of Mathematics, Michigan State University, East Lansing, MI 48823 †Department of Mathematics, Brown University, 151 Thayer Street/Box 1917, Providence, RI 02912 ‡Department of Mathematics, Michigan State University, East Lansing, MI 48823 §All authors are supported by the NSF grant DMS 9970395

3.1 A two-weight inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 3.2 Inverse Hölder inequality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.3 Sharp constant in the (dyadic) Carleson embedding theorem . . . . . . . . . 19

3.3.1 Properties of B(F, f, M) . . . . . . . . . . . . . . . . . . . . . . . . . 21

4 Burkholder meets Bellman 24 4.1 Bellman equation for Burkholder’s theorem . . . . . . . . . . . . . . . . . . . 25 4.2 Finding B: biconcave function . . . . . . . . . . . . . . . . . . . . . . . . . . 26

5 Conclusions 29

1. General approach: Bellman equation for stochastic optimal control

1.1. Stochastic optimal control

The goal of this paper is methodological. We want to relate the equation of Bellman from optimal control of stochastic processes to the concept of Bellman function that appeared in the first preprint version of [10] in 1995, was developed extensively in [8], and proved to be very useful in [9], [5], [11], [12], [13], [17], [14]. We want also to connect the ideas behind the construction of Burkholder’s function b(x, y) = (|x|−(p−1)|y|)(|x|+|y|)^{p−1} (which gives the exact estimates in certain martingale inequalities) to the concept of the Bellman equation.

Let us start with the problem of control of rather general stochastic process. We will borrow the pieces of exposition from the book [6].

Let we have a stochastic process x^{t} in ℝ^{d}, satisfying the following integral equation

x^{t} = x + ∫_{0}^{t} σ(α^{s}, x^{s}) d w^{s} + ∫_{0}^{t} b(α^{s}, x^{s})ds  (1.1)

Here t is the time, w^{t} is a d_{1}-dimensional Wiener process (white noise), σ = σ(α, y) is a d × d_{1} matrix, and b is a d-dimensional vector. The process α^{t} is supposed to be a control that we have to chose. We denote by A ⊂ ℝ^{d_{2}} the set of admissible controls, i. e. the domain where the vector of control parameters α runs.

The choice of stochastic process α^{s} (in our cases it will be also d-dimensional, d_{2} = d, except for the case in Section 3.3) gives us different “motions”—different solutions of (1.1). Of course, the questions about existence and uniqueness of the solution immediately arise, nut in this paper we just assume that the solution exists and unique.

From the practical point of view it is reasonable to consider that the values of the control process α^{s} at the time s > 0 are chosen on the basis of observation of the process x^{t} up to the moment s, so we think that (for each individual trajectory, i. e. for each point ω of the probability space Ω) α^{s} is the function of the trajectory {(t, x^{t}) : 0 ≤ t ≤ s}, that is α^{s} = α^{s}(x^{[0,s]}).

Suppose we are given the profit function f^{α}(y); on the trajectory x^{t} for the time interval [t, t + Δt] the profit is f^{α^{t}}(x^{t})Δt + o(Δt).

Therefore, on the whole trajectory we get ∫_{0}^{∞} f^{α^{t}}(x^{t}) d t.

We want to chose the control α = {α^{s}(x^{[0,s]})} to maximize the average profit

v^{α}(x) = 𝔼 ∫_{0}^{∞} f^{α^{t}}(x^{t}) d t + lim_{t→∞} 𝔼(F (x^{t}))  (1.2)

for the process x^{t} starting at x. Here F is a bonus function—one gets it when one retires. The optimal average gain is what is called the Bellman function for stochastic control:

v(x) = sup_{α} v^{α}(x)  (1.3)

here supremum is taken over all admissible controls α; x ∈ ℝ^{d} is the starting point of the process: it is the same x as in (1.1)

It satisfies a well known Bellman differential equation, which we are going to explain now. The deduction of the Bellman equation for the stochastic optimal control is based on two things:

1. Bellman’s principle,

2. Ito’s formula.

We want to review them to show what changes have to be made to obtain our Bellman function of harmonic analysis pedigree.

Bellman’s principle states that

v(x) = sup_{α^{s}} 𝔼 [∫_{0}^{t} f^{α^{s}}(x^{s}) d s + v(x^{t})]  (1.4)

for each t ≥ 0. Here the supremum is taken over all admissible control processes α^{s}.

To explain it, let us fix some time t > 0, and let us consider an individual trajectory of the system. The profit for the interval [0, t] is given by

∫_{0}^{t} f^{α^{s}}(x^{s})ds.

Suppose that the trajectory of the process has reached the point, say y, at the moment t. Then the maximal average profit we can gain starting at the moment t from the point y is exactly v(y). Indeed, since the increments of w^{s} for s ≥ t do not depend on w^{τ}, τ < t, and they behave as corresponding increments after time 0, and the equations are time invariant, there is no difference between starting at time 0 or at time t. Applying now full probability formula to take into account all possible endpoints x^{t}, we get exactly the Bellman Principle (1.4).

Ito’s formula gives us a representation of v(x^{t}) as a stochastic integral (one can also say that it is a formula for the stochastic differential of v(x^{t})). We then going to hit this formula with the averaging 𝔼 over the probability.

For a reader who is not familiar with the stochastic differential/integral calculus, we a going to explain the version of Ito’s formula we need.

Let us fix moment of time s, and a small increment Δs. We want to estimate the difference v(x^{s+Δs}) − v(x^{s}). Recall that w^{s} = (w_{1}^{s}, w_{2}^{s}, …, w_{d_{1}}^{s})^{T} ∈ ℝ^{d_{1}}, x = (x_{1}, x_{2}, … x_{d})^{T} ∈ ℝ^{d}, and denote Δw^{s} := w^{s+Δs} − w^{s}, Δw_{k}^{s} := w_{k}^{s+Δs} − w_{k}^{s}, etc.

Using Taylor’s formula (we think that v is smooth, which might not be the case, but we do not care now—we are heading towards formal deduction of the Bellman equation), we will have to consider the term

∑_{k=1}^{d} \frac{∂v}{∂x_{k}} (x^{s}) ∑_{j=1}^{d_{1}} σ_{k,j}(α^{s}, x^{s})Δw_{j}^{s} + ∑_{k=1}^{d} \frac{∂v}{∂x_{k}} (x^{s})b_{k}(α^{s}, x^{s})Δs.

Since we are going to average over the probability, the first term in the sum will vanish—the terms Δw_{k}^{s} are independent of x^{s} and have zero averages. The second term can be rewritten as (L_{1}^{α^{s}}(x^{s})v)(x^{s})Δs, where the first order differential operator L_{1}^{α}(x) is given by

L_{1}^{α}(x) := ∑_{k=1}^{d} b_{k}(α, x)\frac{∂}{∂x_{k}}

The main difference between the stochastic calculus and the regular one that in the former we need to consider higher order terms. The next term in the Taylor formula will be

\frac{1}{2} ∑_{i,j} \frac{∂^{2}v}{∂x_{i}∂x_{j}} (Σ_{k}σ^{jk}Δw_{k}^{s} + b_{j}(α^{s}, x^{s})Δs) · (Σ_{k}σ^{ik}Δw_{k}^{s} + b_{i}(α^{s}, x^{s})Δs)  (1.5)

Let us see what happens with this term after averaging over the probability: the terms involving Δw_{k}^{s}Δs disappear again, and we can disregard the terms involving (Δs)^{2}, for they are o(Δs). Since

𝔼Δw_{k}^{s}Δw_{m}^{s} = \{ \begin{matrix}0, & k ≠ m \\ Δs & k = m , \end{matrix}

the averaging of the (1.5) gives

𝔼 [(L_{2}^{α^{s}}(x^{s}) v) (x^{s})] Δs,  (1.6)

where the second order differential operator L_{2}^{α}(x) is given by

L_{2}^{α}(x) := ∑_{i,j=1}^{d} a^{ij}(α, x)\frac{∂^{2}}{∂x_{i}∂x_{j}} ,

and

a^{ij}(α, x) := \frac{1}{2} ∑_{k=1}^{d_{1}} σ^{ik}(α, x)σ^{jk}(α, x).

So, we have got an extra term with Δs!

The higher order Taylor terms other than (1.5) will give us Δs to powers greater than 1, and, therefore, can be omitted by an obvious reason.

Gathering all together and integrating with respect to ds we get

𝔼(v(x^{t})) = v(x) + 𝔼 ∫_{0}^{t} L^{α^{s}}(x^{s})v(x^{s}) d s  (1.7)

where L^{α}(x) := L_{1}^{α}(x) + L_{2}^{α}(x). That is exactly the application of Ito’s formula we need.^{1} Putting (1.7) into the Bellman Principle (1.4) one gets:

0 = sup_{α^{s}} 𝔼 [∫_{0}^{t} f^{α^{s}}(x^{s}) d s + ∫_{0}^{t} L^{α^{s}}(x^{s})v(x^{s}) d s]

Dividing by t and letting t tend to zero, one gets Bellman’s equation on Bellman function v:

sup_{α∈A} [L^{α}(x)v(x) + f^{α}(x)] = 0  (1.8)

(Of course, to justify taking the limit one has to make some assumptions, so the above presentation is not a complete proof, but just a general scheme, explaining how the Bellman equation was derived).

Notice that in the above Bellman equation (1.8) the supremum is taken not over all control processes α^{s}, but over all admissible values α ∈ A ⊂ ℝ^{d_{2}}.

Let us also notice, that the bonus function F is not included in the equation: it usually figures in the boundary conditions and (or) inequalities that v has so satisfy. We will see examples later, see Section 3.2 below.

Note, that it is possible to reverse the above reasonings, namely to show that if (under some additional assumptions) a function v satisfies the Bellman equation (1.8) with appropriate boundary conditions, then it has to be the Bellman function (1.3).

In applications to the Harmonic Analysis we will be more interested in supersolutions of the Bellman equation, i. e. functions V satisfying

sup_{α∈A} [L^{α}(x)V (x) + f^{α}(x)] ≤ 0.  (1.9)

Suppose the bonus function F ≡ 0 and the profit density f^{α}(x) ≥ 0. Then clearly the Bellman function v from (1.3) is nonnegative. Then we claim that any V ≥ 0 satisfying the “Bellman inequality” (1.9) majorates v, V ≥ v.

1Ito’s formula gives us an integral representation of v(x^{t}) before we average it. Its proof is more complicated than the reasoning presented above.

Indeed, (1.9) implies that −L^{α}(x)V (x) ≥ f^{α}(x). Using this inequality in the Ito’s formula (1.7) for V we get

V (x) = 𝔼V (x^{t}) − 𝔼 ∫_{0}^{t}(L^{α^{s}}(x^{s})V )(x^{s})ds
≥ 𝔼V (x^{t}) + 𝔼 ∫_{0}^{t} f^{α^{s}}(x^{s})ds ≥ 𝔼 ∫_{0}^{t} f^{α^{s}}(x^{s})ds

(the last inequality holds because V ≥ 0). Taking limit as t → ∞ and then supremum over all controls α, we get the desired inequality V (x) ≥ v(x).

One more remark: if σ(α, x) ≡ 0, i. e. the stochastic integral (with respect to the Wiener process w) disappears from the equation (1.1) and we have only integral with ds, then the problem becomes essentially deterministic. Namely, to find the optimal control α one just needs to find it for each trajectory separately, i. e. to solve a deterministic problem. In the Bellman equation (1.8) the second order term L_{2} disappears, and we get as a result a first order PDE, which is exactly the classical (deterministic) Bellman–Jacobi equation from optimal control. The second order operator L_{2} reflects the specifics of the stochastic case.

2. Harmonic analysis Bellman function

2.1. Harmonic analysis problems

By “harmonic analysis problems” we always will mean here some dyadic problem, dealing with averages over dyadic intervals. It is often possible to pass from a dyadic problem to a problem with analytic or harmonic function by using some kind of Green’s formula, see [9].

By dyadic lattice D we mean the collection of all dyadic interval, i. e. the collection of all intervals of form [j · 2^{k}, (j + 1) · 2^{k}), j, k ∈ ℤ.

If I is an interval, we denote by |I| its length, and by χ_{I} its characteristic function, χ_{I} (x) = 1 if x ∈ I, and χ_{I} (x) = 0 if x ∉ I. Symbols I_{+} and I_{−} denote right and left halves of the interval I respectively.

And finally, given a function f we denote by ⟨f⟩_{I} its average over the interval I,

⟨f⟩_{I} := |I|^{−1} ∫_{I} f(x)dx

For an interval I define the Haar function h_{I} := |I|^{−1/2}(χ_{I_{+}} − χ_{I_{−}}). The Haar system {h_{I} : I ∈ D} is an orthonormal basis in L^{2}(ℝ).

2.2. General remarks.

In “our” Bellman function deduction we must have several changes in the scheme. The most important change is that our time will be discrete. Let us postpone this modification. So, our time is still continuous.

2.3 An example: A_{∞} weights and their associated Carleson measures 7

The second change is that in most of our problems the corresponding Wiener process will be one-dimensional (d_{1} = 1). So, it simplifies the things a bit.

The matrix σ(α, x) will be so just a column (σ_{1}, σ_{2}, …, σ_{d})^{T}. Very often it will be very simple, just σ_{k} = α_{k}, k = 1, 2, …, d.

The first order operator L_{1}^{α} will remain the same, but the second order part will be much simpler,

L_{2}^{α}(x) = \frac{1}{2} ∑_{i,j} σ_{i}(α, x)σ_{j}(α, x)\frac{∂^{2}}{∂x_{i}∂x_{j}} (= \frac{1}{2} ∑_{i,j} α_{i}α_{j} \frac{∂^{2}}{∂x_{i}∂x_{j}} if σ_{k} = α_{k}, k = 1, 2, …, d. )

Usually, the solutions of Bellman equation (1.8) are quite difficult to find, so in most cases we will be looking for supersolutions (1.9), when the equality is replaced by ≤ sign. The Bellman inequality (1.9) in our case usually mean some kind of convexity. For example, if there is no linear term L_{1}, and the profit density f^{α} ≡ 0, then, in the case σ_{k} = α_{k} ∀k, the Bellman inequality (1.9) becomes

\frac{1}{2} sup_{α} ∑_{i,j=1}^{d} α_{i}α_{j} \frac{∂^{2}v}{∂x_{i}∂x_{j}} ≤ 0,

i. e. exactly the concavity condition for v.

But let us consider a simple example to illustrate what we were saying.

2.3. An example: A_{∞} weights and their associated Carleson measures

We call a positive function w on the line ℝ an A_{∞} weight (dyadic A_{∞} weight, actually) if

⟨w⟩_{J} ≤ c_{1} e^{⟨log w⟩_{J}} , ∀J ∈ D  (2.1)

Here D denotes a dyadic lattice on ℝ, and ⟨·⟩_{J} denotes the averaging over J.

We are going to use our Bellman function technique to prove the following result of Buckley [1] that can be found (along with it “continuous analogs) also in the paper of Fefferman–Kenig–Pipher [4].

Theorem 2.1. Let w ∈ A_{∞}. Then

∀I ∈ D, \frac{1}{|I|} ∑_{ℓ⊂I} (\frac{⟨w⟩_{ℓ_{+}} − ⟨w⟩_{ℓ_{−}}}{⟨w⟩_{ℓ}} )^{2} |ℓ| ≤ c_{2},  (2.2)

where c_{2} depends only on c_{1} in (2.1).

The meaning of (2.2) is that a certain measure associated with w ∈ A_{∞} must be a Carleson measure.

2.3.1. “Common sense” approach

We present first a “naïve” approach, based essentially just on common sense, and then show how it correspond to the general scheme presented above.

Let us fix the averages w := ⟨w⟩_{I}, u := ⟨log w⟩_{I}.

Define a function B = B_{I} = B_{I} (u, w) of two real variables u, w as

B_{I} (u, w) := sup \frac{1}{|I|} ∑_{J⊂I} (\frac{⟨w⟩_{J_{+}} − ⟨w⟩_{J_{−}}}{⟨w⟩_{J}} )^{2} |J|,

where the supremum is taken over all weights w on I satisfying the A_{∞} condition (2.1) and such, that ⟨w⟩_{I} = w, ⟨log w⟩_{I} = u. We use the letter B here for Bellman.

Note, that the function B_{I} does not depend on the interval I: it clearly doesn’t change if one shifts the interval I, and the scaling is chosen in such a way, that B remains the same if one changes the size of I. Therefore, in what follows, we just skip the subscript I.

Clearly, the above Theorem 2.1 holds if and only if B ≤ c_{2}. Let us try to investigate the properties of the function B. Clearly, the following properties hold

1. Domain: the function B is defined on the domain D e^{u} ≤ w ≤ c_{1}e^{u}.

The first inequality here is just Jensen inequality e^{⟨log w⟩_{I}} ≤ ⟨w⟩_{I} (geometric mean is at most the arithmetic mean), and the second one is the A_{∞} condition (2.1).

It takes some effort to prove that for any pair u, w from the domain one can find an A_{∞} weight w such that ⟨log w⟩_{I} = u, ⟨w⟩_{I} = w, but we really do not need this fact to prove the theorem: we need it only to be sure that we did not miss anything in our Bellman function approach.

2. 0 ≤ B ≤ c_{2}; the first inequality just follows from the definition, and the second one is just our belief that the theorem is true;

3. The main inequality: for any three pairs (u, w), (u_{+}, w_{+}), (u_{−}, w_{−}) in the domain satisfying

u = \frac{1}{2}(u_{+} + u_{−}), w = \frac{1}{2}(w_{+} + w_{−})

the following inequality

B(u, w) ≥ (\frac{w_{+} + w_{−}}{w} )^{2} + \frac{1}{2}(B(u_{+}, w_{+}) + B(u_{−}, w_{−}))  (2.3)

holds.

The explanation for this main inequality is rather simple: consider all A_{∞} weights w such that ⟨log w⟩_{I} = u, ⟨w⟩_{I} = w, and the corresponding averages over subintervals I_{+} and I_{−} are u_{+}, w_{+}, and u_{−}, w_{−} respectively. Then

\frac{1}{|I|} ∑_{J⊂I} (\frac{⟨w⟩_{J_{+}} − ⟨w⟩_{J_{−}}}{⟨w⟩_{J}} )^{2} |J| = (\frac{w_{+} + w_{−}}{w} )^{2} + \frac{1}{|I|} ∑_{J⊂I_{+}} … + \frac{1}{|I|} ∑_{J⊂I_{−}} …

Taking the supremum of the left side over all such w we get at most B(u, w). On the other hand, since one can define the function w independently on the intervals I_{+} and I_{−}, the

supremum of the right side gives us (\frac{w_{+} + w_{−}}{w} )^{2} + \frac{1}{2}(B(u_{+}, w_{+}) + B(u_{−}, w_{−})). The main

inequality is proved.

2.3.2. Connections with stochastic control

Now, let us see how does it fit in the general scheme of stochastic control. First of all, the time here is discrete. Instead of the Wiener process, we consider its simplest discrete time analogue—the process ∑_{k=0}^{n} ξ^{k}, n = 0, 1, 2, …, where ξ^{k} are independent coin tosses, i. e. the independent random quantities taking values 1 and −1 with probabilities 1/2. Now we have to deicide what is the state space variable x and what is the control α. For x it is easy to deicide: it is two-dimensional vector of averages, x = (u, w).

The equation of the process is

x^{n+1} = x^{n} + α^{n}ξ^{n}, x^{0} = (u, w)^{T}

The interpretation is the following. We start from the interval I and going down to its subintervals, so at the moment n we are on a subinterval J of lengths 2^{−n}|I|. The position of the interval J is determined by the sequence of coin tosses ξ^{k}: we move from the interval J to J_{+} if ξ^{n} = 1, and to J_{−} if ξ^{n} = −1.

If we are given a weight w, it completely defines the process, namely, the initial state (u, w) (just the averages over the interval I) as well as the control α. Namely, the vector x^{n} = (u^{n}, w^{n}) is just the vector of averages over an appropriate interval J, and if x^{n} = (u^{n}, w^{n}) = (⟨log w⟩_{J} , ⟨w⟩_{J}), then α^{n} = (x^{n+1} − x^{n})/ξ^{n}, so

α_{1}^{n} = \frac{1}{2} (⟨log w⟩_{J_{+}} − ⟨log w⟩_{J_{−}}) , α_{2}^{n} = \frac{1}{2} (⟨w⟩_{J_{+}} − ⟨w⟩_{J_{−}}) .

On the other hand, if we have a process (i. e. the initial state (u, w) and the control process α) such, that lim_{n→∞} e^{u^{n}} = lim_{n→∞} w^{n} with probability 1, then it uniquely defines an A_{∞} weight w.

Indeed, the process defines the “averages” u_{J}, w_{J} for all dyadic subintervals J of I, and we just put

w = lim_{n→∞} ∑_{J⊂I:|J|=2^{−n}|I|} w_{J} χ_{J} .

Since we always stay in the domain, the resulting weight w clearly satisfies the A_{∞} condition (2.1).

Since we established one to one correspondence between weights and controlled processes, in what follows we will switch freely from one language to another.

The profit density in our case is 4(α_{2}/x_{2})^{2} = 4(α_{2}/w)^{2}, so we need to maximize the average gain

𝔼 ∑_{n=0}^{∞} 4 · (\frac{α_{2}^{n}}{w^{n}} )^{2} = \frac{1}{|I|} ∑_{J⊂I} (\frac{⟨w⟩_{J_{+}} − ⟨w⟩_{J_{−}}}{⟨w⟩_{J}} )^{2} |J|.

(the term |J|/|I| in the right hand sum is the probability of getting to the interval J). The Bellman principle for the discrete process (for time t = 1) reads

B(u, w) = sup 𝔼 \{4 · (\frac{α_{2}^{0}}{w} )^{2} + B(u^{1}, w^{1})\}

where the supremum is taken over all admissible control processes α. The explanation of the Bellman principle is exactly the same as in the case of continuous time.

Let us now deduce the Bellman equation. For a given control process α let us compute the expectation

𝔼 \{4 · (\frac{α_{2}^{0}}{w} )^{2} + B(u^{1}, w^{1})\} = 4 · (\frac{α_{2}^{0}}{w} )^{2} + \frac{1}{2} (B(x + α^{0}) + B(x − α^{0}))

If we now denote u_{±} := u ± α_{1}^{0}, w_{±} := w ± α_{2}^{0}, then the above identity immediately implies the main inequality (2.3) (property 3 from Section 2.3.1).

One can be tempted to take the supremum over all possible values of α^{0} to get the Bellman equation

B(u, w) = sup \{4 · (\frac{α_{2}^{0}}{w} )^{2} + \frac{1}{2} (B(u + α_{1}^{0}, w + α_{2}^{0}) + B(u − α_{1}^{0}, w − α_{2}^{0}))\} ,  (2.4)

but unfortunately this leads nowhere: the obvious choice α^{0} = 0 gives us trivial identity B(x) = B(x).

So, what we have now, is an inequality for a supersolution.

2.3.3. From supersolution to the theorem

As we already have discussed above in Section 1.1, a supersolution majorates the Bellman function, so if we find any functions satisfying the conditions 1–3 from Section 2.3.1 we prove Theorem 2.1. For the sake of completeness, let us present the reasoning in this discrete case.

Take an arbitrary A_{∞} weight w, and for a dyadic subinterval J of I let us denote u_{J} := ⟨log w⟩_{J}, w_{J} := ⟨w⟩_{J} . The main inequality 3 (Bellman inequality) implies

|I| · B(u_{I} , w_{I}) ≥ |I| · (\frac{w_{I_{+}} − w_{I_{−}}}{w_{I}} )^{2} + |I_{+}| · B(u_{I_{+}}, w_{I_{+}}) + |I_{−}| · B(u_{I_{−}}, w_{I_{−}})

Writing similar inequalities for I_{+} and I_{−}, then for the intervals of the next generation, etc, and them summing everything up to the intervals of the generation n + 1, we get

|I| · c_{2} ≥ |I| · B(u_{I} , w_{I}) ≥ ∑_{\begin{matrix}J⊂I \\ 2^{−n}|I|<|J|≤|I| \end{matrix}} |J| · (\frac{w_{J_{+}} − w_{J_{−}}}{w_{J}} )^{2} + ∑_{\begin{matrix}J⊂I \\ |J|=2^{−n}|I| \end{matrix}} |J_{−}| · B(u_{J_{−}}, w_{J_{−}})

≥ ∑_{\begin{matrix}J⊂I \\ 2^{−n}|I|<|J|≤|I| \end{matrix}} |J| · (\frac{w_{J_{+}} − w_{J_{−}}}{w_{J}} )^{2}

(the last inequality holds because B ≥ 0). Taking limit as n → ∞, we get exactly the conclusion of the theorem.

2.3.4. Continuous (infinitesimal) version

To solve the above Bellman equation (inequality), we first move to its continuous (infinitesimal) version.

Let x = (u, w) in the interior of the domain of B and α = (α_{1}, α_{2}) = (du, dw) be fixed, and let x_{±} = (u_{±}, w_{±}) = x ± α. (Here we are skipping the superscript 0 at α_{k} from (2.4)). Multiplying α by a small constant, we can always assume that the whole interval [x_{−}, x_{+}] belongs to the domain.

Consider functions φ, Φ of the scalar argument t,

φ(t) = B(x + αt), Φ(t) = B(x) − \frac{1}{2}(B(x + αt) + B(x − αt)) − (\frac{2α_{2}t}{w})^{2} , −1 ≤ t ≤ 1.

If the function B is C^{2}-smooth

lim_{t→0} \frac{φ(t) + φ(−t) − 2φ(0)}{t^{2}} = φ′′(0) = ∑_{j,k=1}^{2} \frac{∂^{2}B(x)}{∂x_{j}∂x_{k}} α_{j}α_{k}  (2.5)

The Bellman Inequality (2.3) implies that Φ(t) ≥ 0 ∀t ∈ [−1, 1]. Dividing this inequality by t^{2} and taking limit as t → 0, we get (taking into account (2.5))

−\frac{1}{2} ∑_{j,k=1}^{2} \frac{∂^{2}B(x)}{∂x_{j}∂x_{k}} α_{j}α_{k} ≥ 4 · (\frac{α_{2}}{w} )^{2} ,  (2.6)

or, equivalently −d^{2}B ≥ 8(dw/w)^{2}. We have proved the above inequality only for small α, but because of the homogeneity, it holds for all α.

Note, that if we consider a continuous time controlled process

x^{t} = ∫_{0}^{t} α^{s}dw^{s}, x^{s} = (x_{1}^{s}, x_{2}^{s}), α^{s} = (α_{1}^{s}, α_{2}^{s}),

(here w^{s} is one dimensional Wiener process, and not a weight) with profit density f^{α}(x) = 4(α_{2}/x_{2}), then the Bellman equation for this process will be

sup_{α} \{\frac{1}{2} ∑_{j,k=1}^{2} \frac{∂^{2}B(x)}{∂x_{j}∂x_{k}} α_{j}α_{k} + 4 · (\frac{α_{2}}{x_{2}} )^{2}\} = 0.  (2.7)

The inequality (2.6) (−d^{2}B ≥ 8(dw/w)^{2}), which we have just derived from the discrete Bellman inequality, is simply the inequality for a supersolution of the continuous time Bellman equation (2.7).

This is not a coincidence: we get the infinitesimal version of the discrete Bellman equation if we assume that all changes of state variables are infinitely small. Mathematically it can be expressed (at least formally) by considering the process

x^{n+1} = x^{n} + α^{n}ξ^{n},

where ξ^{n} are independent random variables taking values −1/\sqrt{N} and 1/\sqrt{N} with proba-

bilities 1/2, and maximizing the average 𝔼 ∑_{n=0}^{∞} (\frac{α_{2}}{x_{2}} )^{2} \frac{1}{N}.

Taking the limit as N → ∞ (assuming that the time elapsed from n to n + 1 is 1/N), we get (at least formally) the continuous time process.

We leave all the details as an exercise for the reader. A detailed proof of this equivalence with justification of all limiting procedures, is beyond the scope and goals of this paper.

We will call equation (2.7) the Bellman equation for Theorem 2.1. Note, that formally this equation is not well defined, namely, any function satisfying −d^{2}B ≥ 8(dw/bw)^{2} automatically satisfies (2.7) (supremum is attained at α = 0). To get the equation one has to require, for example that supremum is attained at some α ≠ 0.

It is possible to show (under some additional assumption) that if B is C^{2}-smooth, then the the Bellman function must satisfy this equation. However, in most situations it is rather hard to show that the Bellman function is smooth (moreover, it is often not the case), so it does not make a lot of sense proving that the Bellman equation is necessary.

Besides, very often we are not able to solve the equation, and guessing a supersolution is the best we can do. And in rare cases when we can solve the equation, it is usually possible to show that the solution is indeed the Bellman function.

2.3.5. Solution of the Bellman equation

In this case we are lucky—we can just guess a solution of the Bellman equation (2.7). Namely,

B(u, w) := 8 · (log w − u)

clearly satisfies the equation. What is more important, it is indeed the Bellman function! Note, that equation (2.7) does not have a unique solution: for example, one can get infinitely many solutions adding arbitrary linear functions.

Let us first show, that the function B satisfies conditions 1–3 from Section 2.3.1. Clearly, B(u, w) ≥ 0 if e^{u} ≤ w, and B(u, w) ≤ c_{2} = 8 log c_{1} if w ≤ c_{1}e^{u}. So, conditions 1 and 2 are satisfied.

Let us prove the main inequality (2.3). Denote x = (u, w), x_{±} = (u_{±}, w_{±}) = (u±α_{1}, w± α_{2}). Then

B(x) − \frac{1}{2}(B(x_{+}) + B(x_{−})) = 8 · (log w − \frac{1}{2}(log(w + α_{2}) + log(w − α_{2})))

= −4 log(1 − (\frac{α_{2}}{w} )^{2}) ≥ 4 · (\frac{α_{2}}{w} )^{2} = (\frac{w_{+} − w_{−}}{w} )^{2}.

The main inequality (2.3), and so the theorem are proved.

Let us show now, that the function B we constructed is indeed the Bellman function. Note first, that ⟨log w⟩_{I} = log(⟨w⟩_{I}) if and only if w ≡ Const, so the Bellman function is 0 if u = log w. Our function B satisfies this property, so this is a good indication.

To prove that the function B(u, w) := 8 · (log w − u) is the Bellman function let us fix a point x = (u, w) in the domain, and given ε > 0 construct a process (chose “control” α)

x^{n+1} = x^{n} + α^{n}ξ^{n}

in the domain, starting at x, such that lim_{n→∞} e^{u^{n}} = lim_{n→∞} w^{n} with probability 1. Recall, that ξ^{n}, n = 0, 1, 2, … is the sequence of independent coin tosses, each ξ^{n} takes values ±1 with probability 1/2.

As we had discussed above, such processes correspond to A_{∞} weights. So, if we are able, given ε > 0 to pick control α = {α^{n}}_{n=0}^{∞} such, that

𝔼 ∑_{n=0}^{∞} (\frac{α^{n}}{w^{n}} )^{2} ≥ (1 − ε)B(x),

then we are done (constructed an ε-optimal weight).

To construct such a process, let us fix in the domain a line segment L = [A, B] containing x and with both endpoints A and B on the lower bound w = e^{u} of the domain, see Fig. 1. Our process will be just a discrete random walk on the line L with small steps. Namely, at each step we will move by on the line L by δ, picking one of two possible directions with probability 1/2.

Formally, it will be written as follows. Let ℓℓℓ be a unit vector parallel to L. Let δ > 0 be a small number to be picked later. We put α^{n} := δℓℓℓ if δ ≤ d := dist(x, {A, B}), and put α^{n} := d · ℓℓℓ otherwise. The latter simply means that we have to decrease the step near the boundary, to avoid going out of the domain Ω. It also means, that when we hit the boundary, we just stay there forever.

It is clear, that with probability 1 all trajectories eventually hit the boundary. Applying the Taylor’s formula and using compactness of L we can find δ > 0 such, that

B(x) − \frac{1}{2}(B(x + α) + B(x − α)) ≤ (1 + ε) · 4 · (\frac{α_{2}}{w} )^{2}.

Since with probability 1 all trajectories end on the lower boundary w = e^{u} and on this boundary B(x) = 0, we can conclude that

𝔼 ∑_{n=0}^{∞} (\frac{α^{n}}{w^{n}} )^{2} ≥ \frac{1}{(1 + ε)}B(x) ≥ (1 − ε)B(x),

and that is exactly what we need.

2.4. Concluding remarks

The above example illustrates pretty much main ideas of our Bellman function method: determine variables (usually averages), find discrete Bellman inequality, then its infinitesimal version, and finally pick a supersolution. There are few points we want to make

• Domain of B is essential, to emphasize this we even included it as a property. For example, it is impossible to find a bounded solution of the Bellman inequality (2.3) (or (2.6)) on the whole plane.

• Infinitesimal Bellman equation (inequality) can be obtained (at least formally) as the Bellman equation (inequality) for the corresponding continuous time process. This formalism sometimes can be easier than investigation of the discrete case

[figure: x = (u, w); w = e^{u}; w = c_{1}e^{u}; u; w; L]

Figure 1: Constructing suboptimal random walk.

This feature of our Bellman function theory—guessing the supersolution of Bellman equation—will eventually be replaced hopefully by some intelligent method. But so far no such method is available—so the guessing of simple blocks and putting them together as in the lego game is the only approach now. The excuse is that our Bellman equation is usually very complicated.

3. New creatures from Bellman’s Zoo

There are many of them to be found in [8]. Here we have a few more.

3.1. A two-weight inequality

Theorem 3.1. Let u, v be two positive functions such that for any dyadic interval I,

⟨u⟩_{J} ⟨v⟩_{J} ≤ 1

Then for any dyadic interval I

\frac{1}{|I|} ∑_{J∈D,J⊂I} |⟨u⟩_{J_{+}} − ⟨u⟩_{J_{−}}| · |⟨v⟩_{J_{+}} − ⟨v⟩_{J_{−}}| · |J| ≤ 16 ⟨u⟩_{I}^{1/2}⟨v⟩_{I}^{1/2}  (3.1)

This inequality is a particular case of two-weight inequality for martingale transform. The general two-weight inequality for martingale transform was found in [10] (and it has far reaching consequences in [17], [14], [5], [11], [12], [13]). But this particular case (3.1) was our first Bellman function inequality, and it appeared in the first preprint version of [10].

Let us deduce the Bellman equation (inequality) for this problem. The choice of variables is obvious, we take the averages u = ⟨u⟩_{I}, v = ⟨v⟩_{I}. Let us fix a dyadic interval I and define the function B = B_{I} as follows:

B(u, v) := sup\{\frac{1}{|I|} ∑_{J∈D,J⊂I} |⟨u⟩_{J_{+}} − ⟨u⟩_{J_{−}}| · |⟨v⟩_{J_{+}} − ⟨v⟩_{J_{−}}| · |J|\}

where the supremum is taken over all non-negative functions u and v satisfying ⟨u⟩_{J} ⟨v⟩_{J} ≤ 1 for all dyadic intervals J, and such, that ⟨u⟩_{I} = u, ⟨v⟩_{I} = v. Note, that again, because of right scaling, the function B does not depend on the interval I, so we will skip it.

The Bellman function B satisfies the following properties:

1. Domain Ω: u, v ≥ 0, uv ≤ 1;

2. Range: 0 ≤ B ≤ 16\sqrt{uv};

3. Main inequality: for any three pairs x = (u, v), x_{+} = (u_{+}, v_{+}), x_{−} = (u_{−}, v_{+}) belonging to the domain Ω and satisfying x = (x_{+} + x_{−})/2, the inequality

B(x) ≥ \frac{1}{2}(B(x_{+}) + B(x_{−})) + |u_{+} − u_{−}| · |v_{+} − v_{−}|  (3.2)

holds.

The inequality B ≥ 0 follows immediately from the definition of B, and the inequality B(u, v) ≤ 32\sqrt{uv} is our belief that the theorem is true.

The explanation for the main inequality (3.2) is also simple. Let us consider all weights u, v satisfying ⟨u⟩_{J} ⟨v⟩_{J} ≤ 1 for all dyadic intervals J, and such, that the averages over the intervals I, I_{+}, I_{−} are fixed,

⟨u⟩_{I} = u, ⟨v⟩_{I} = v, ⟨u⟩_{I_{±}} = u_{±}, ⟨v⟩_{I_{±}} = v_{±}

Then, for any such u, v

B(u, v) ≥ \frac{1}{|I|} ∑_{J∈D,J⊂I} |⟨u⟩_{J_{+}} − ⟨u⟩_{J_{−}}| · |⟨v⟩_{J_{+}} − ⟨v⟩_{J_{−}}| · |J|

= |u_{+} − u_{−}| · |v_{+} − v_{−}| + \frac{1}{2}( ∑_{J⊂I_{+}} … + ∑_{J⊂I_{−}} …),

and taking the supremum over all such u, v we get the main inequality (3.2).

Again, if we find any function satisfying the conditions 1–3 above, the theorem is proved. Indeed, for any weights u, v from the theorem the main inequality (3.2) implies

|⟨u⟩_{I_{+}} − ⟨u⟩_{I_{−}}| · |⟨v⟩_{I_{+}} − ⟨v⟩_{I_{−}}| · |I|

≤ |I|B(⟨u⟩_{I} , ⟨v⟩_{I}) − |I_{+}|B(⟨u⟩_{I_{+}}, ⟨v⟩_{I_{+}}) − |I_{−}|B(⟨u⟩_{I_{−}}, ⟨v⟩_{I_{−}})

Writing similar inequalities for I_{+} and I_{−}, then for the intervals of the next generation and summing everything up to the intervals of the generation n + 1, we get

∑_{\begin{matrix}J⊂I \\ 2^{−n}|I|<|J|≤|I| \end{matrix}} |⟨u⟩_{J_{+}} − ⟨u⟩_{J_{−}}| · |⟨v⟩_{J_{+}} − ⟨v⟩_{J_{−}}| · |J| ≤ |I|B(⟨u⟩_{I} , ⟨v⟩_{I}) − ∑_{\begin{matrix}J⊂I \\ |J|=2^{−n}|I| \end{matrix}} B(⟨u⟩_{J} , ⟨v⟩_{J})

≤ |I|B(⟨u⟩_{I} , ⟨v⟩_{I}) ≤ 16|I|\sqrt{⟨u⟩_{I} ⟨v⟩_{I}} .

Taking limit as n → ∞ we get the conclusion of the theorem.

The stochastic control interpretation is as follows. We have a controlled process

x^{n+1} = x^{n} + α^{n}ξ^{n}

where ξ^{n} is the sequence of independent coin tosses, and we want to maximize the average profit 𝔼(∑ 4|α_{1}^{n}| · |α_{2}^{n}|).

Here again the weights u, v determine the process, namely, the initial state x = (u, v) (it is just the averages over the interval I) and the control α = (α_{1}, α_{2}). Namely, as it was discussed in Section 2.3.2, the sequence of coin tosses ξ^{n} determines the interval we are on: we move from an interval J to J_{+} if ξ^{n} = 1 and to J_{−} if ξ^{1} = −1. If at the time n we are on an interval J, then the state space vector x is just the vector of averages over J,

x^{n} = (u^{n}, v^{n}) = (⟨u⟩_{J} , ⟨v⟩_{J})

and the control α^{n} = (α_{1}^{n}, α_{2}^{n}) is determined by

α_{1}^{n} = \frac{1}{2}(⟨u⟩_{J_{+}} − ⟨u⟩_{J_{−}}), α_{1}^{n} = \frac{1}{2}(⟨v⟩_{J_{+}} − ⟨v⟩_{J_{−}}).

In this situation we have that the average profit 𝔼(∑ 4|α_{1}^{n}| · |α_{2}^{n}|) is exactly the sum in the left side of (3.1).

Again, as in Section 2.3.4 we can find that the main inequality (3.2) implies its infinitesimal version

−\frac{1}{2} ∑_{j,k=1}^{2} \frac{∂^{2}B(x)}{∂x_{j}∂x_{k}} α_{j}α_{k} ≥ 4|α_{1}| · |α_{2}|,  (3.3)

or equivalently, −d^{2}B ≥ 8|du| · |dv|.

This inequality also can be formally obtained by considering a continuous time analogue of our process and writing corresponding Bellman equation for it. The continuous time version of our process is

x^{t} = ∫_{0}^{t} α^{s}dw^{s}

where w^{t} is one dimensional Wiener process, x^{t} = (x_{1}^{t}, x_{2}^{t}), α^{t} = (α_{1}^{t}, α_{2}^{t}), and we want to maximize 𝔼(∫_{0}^{∞} 4|α_{1}^{t}α_{2}^{t}|dt), so the profit density f^{α} = |α_{1}α_{2}|. The corresponding Bellman equation (see Section 1.1) is

sup_{α} \{\frac{1}{2} ∑_{j,k=1}^{2} \frac{∂^{2}B(x)}{∂x_{j}∂x_{k}} α_{j}α_{k} + 4|α_{1}| · |α_{2}|\} = 0

so the partial differential inequality (3.3) is simply the inequality for supersolution.

We do not know how to solve the equation, but it is really not too hard to guess a supersolution. It is an easy exercise in multivariable calculus to compute the Hessian (the matrix of second derivatives) and to verify that for the function

B(u, v) := 4 · (4\sqrt{uv} − uv)

the partial differential inequality (3.3) holds.

We are almost done. We are saying “almost”, because to prove the theorem we need the discrete main inequality (3.2). If the domain Ω were convex, the discrete inequality (3.2) would immediately follow from the infinitesimal version (3.3). Unfortunately, the domain is not convex, so additional work is needed. Let, according to our notational convention x_{±} = (u_{±}, v_{±}) = x ± α = (u, v) ± (α_{1}, α_{2}). Then, if α_{1}α_{2} ≥ 0, and the points x, x_{+} and x_{−} are in the domain Ω, then the whole interval [x_{−}, x_{+}] is in the domain. Therefore, for such α_{1}, α_{2} we can apply Taylor’s formula to get the main inequality (3.2) from it infinitesimal version (3.3).

There is also an elementary way to get to that conclusion. Since

uv − \frac{1}{2}((u + α_{1})(v + α_{2}) + (u − α_{1})(v − α_{2})) = −α_{1}α_{2}  (3.4)

and the function \sqrt{uv} is concave, the main inequality (3.2) follows immediately if α_{1}α_{2} > 0. To treat the case α_{1}α_{2} < 0 we note, that the function t ↦ \sqrt{t} is concave, so

\sqrt{uv} − \frac{1}{2}(\sqrt{u_{+}v_{+}} + \sqrt{u_{−}v_{−}}) ≥ \sqrt{uv} − \sqrt{\frac{u_{+}v_{+} + u_{−}v_{−}}{2}}

= \sqrt{uv} − \sqrt{uv + α_{1}α_{2}} ≥ −\frac{1}{2} \frac{α_{1}α_{2}}{\sqrt{uv}} ≥ \frac{1}{2}|α_{1}α_{2}|.

The term −uv works against us in the case α_{1}α_{2} < 0, see (3.4), but it gets overpowered by 4\sqrt{uv}, so the main inequality (3.2) holds in this case as well.

Our “the first harmonic analysis Bellman function”

B(u, v) := 4 · (4\sqrt{uv} − uv)

looks rather simple. A much more complicated siblings of this Bellman function helped us to solve the two-weight Martingale transform in [10]. But, actually, even this “simple” one has a challenging minus sign. However, this minus sign is not too difficult to explain: it serves the goal to make B strictly concave (it must be quite concave—see (3.3)) exactly where \sqrt{uv} looses concavity (near the diagonal u = v).

3.2. Inverse Hölder inequality

Let us recall that a weight (a nonnegative locally L^{1} function) w is called a dyadic A_{∞} weight if

⟨w⟩_{J} ≤ c exp(⟨log w⟩_{J}),  for any dyadic interval J  (3.5)

The best constant c is called the A_{∞}-norm of w. Of course, the A_{∞} norm is not a real norm, but we will use the term anyway.

Theorem 3.2. If w is in dyadic A_{∞}, then there exists p > 1 (depending only on A_{∞}-norm of w) such that for any dyadic interval I

⟨w^{p}⟩_{I} ≤ C ⟨w⟩_{I}^{p}  (3.6)

To prove the theorem let us fix the averages

x_{1} = w := ⟨w⟩_{I} ,  x_{2} = u := ⟨log w⟩_{I} ,  (3.7)

and define the Bellman function B by

B(x) = B(w, u) := |I|^{−1} sup{∫_{I} w^{p}dx : w satisfying (3.5), (3.7) }

(clearly it does not depend on the interval I).

In terms of stochastic processes the equation can be written as

x^{n+1} = x^{n} + ( \begin{matrix}α_{1}^{n} \\ α_{2}^{n} \end{matrix} ) ξ^{n}, x^{0} = (w, u)^{T} .

and we want to maximize

lim_{n→∞} (E|x_{1}^{n}|^{p}).

The variables are easy to read: if at the moment of time n we are at an interval I, then x^{n} = (x_{1}^{n}, x_{2}^{n})^{T} = (w^{n}, u^{n})^{T} = (⟨w⟩_{I} , ⟨log w⟩_{I})^{T}, and

α_{1}^{n} = (⟨w⟩_{I_{+}} − ⟨w⟩_{I_{−}})/2, α_{2}^{n} = (⟨log w⟩_{I_{+}} − ⟨log w⟩_{I_{−}})/2,

There is no profit function to optimize—f^{α}(x) = 0, only the retirement bonus F (x) = |x_{1}|^{p}.

The properties of B:

1. Domain: Ω = Ω_{c} = {w, u : 1 ≤ w e^{−u} ≤ c}

2. Range: w^{p} ≤ B(x) ≤ Cw^{p} (the first inequality is just the Hölder inequality ⟨w⟩_{I}^{p} ≤ ⟨w^{p}⟩_{I}, and the second one should hold if the theorem is true).

3. Main inequality: for any x, x_{+}, x_{−} ∈ Ω and satisfying x = (x_{+} + x_{−})/2, the inequality B(x) ≥ (B(x_{+}) + B(x_{−}))/2 holds.

Again, if we find a function B satisfying the above properties 1–3, the theorem is proved. Indeed, fix an A_{∞} weight w, and for an interval I denote

x_{I} = (w_{I} , u_{I}) = (⟨w⟩_{I} , ⟨log w⟩_{I}).

Main inequality 3 implies

|I_{+}| · B(x_{I_{+}}) + |I_{−}| · B(x_{I_{−}}) ≤ |I| · B(x_{I}).

Applying the same inequality to I_{+} and I_{−}, and then to their subintervals, we get after going n generations down

∑_{\begin{matrix}J⊂I \\ |I|=2^{−n}|I| \end{matrix}}⟨w⟩_{J}^{p}|J| ≤ ∑_{\begin{matrix}J⊂I \\ |I|=2^{−n}|I| \end{matrix}} B(x_{J})|J| ≤ |I| · B(x_{I}) ≤ C · |I|⟨w⟩_{I}^{p}.

Taking limit as n → ∞ we get

∫_{I} w^{p} ≤ C · |I|⟨w⟩_{I}^{p},

which is exactly the conclusion of the theorem.

The infinitesimal version of the main inequality is simply the concavity condition d^{2}B ≤ 0. It can be also obtained by writing the formal Bellman equation for the continuous analogue of our process.

So, we need to find a concave function B in Ω, satisfying the inequalities 2. We actually should prove that there exists p = p(c) > 1, for which such a concave function exists in Ω_{c}.

And indeed such a function can be found in the form B(x) = x_{1}^{p}φ(x_{1} e^{−x_{2}}). The reader can find φ on [1, c] in such a way that the above aggregate is concave in Ω_{c}, g being bounded and φ ≥ 1.

Looks like we are done, but a careful reader can notice that there remains an issue to be resolved. Namely, because the domain Ω = Ω_{c} is not convex, the concavity condition d^{2}B ≤ 0 does not imply main inequality 3.

To overcome this difficulty, let us notice, that if all 3 points, x, x_{+} and x_{−} are in the domain Ω_{c}, then the whole interval [x_{−}, x_{+}] is in the domain Ω_{c^{2}}. This would immediately solve all our problems, since a concave function in the domain Ω_{c^{2}} satisfies the main inequality 3 in the smaller domain Ω_{c}.

To show that the interval [x_{−}, x_{+}] lies in Ω_{c^{2}}, define w_{τ} := w + (w_{+} − w_{−})τ/2, u_{τ} := u + (u_{+} − u_{−})τ/2, so w_{±1} = w_{±}, u_{±1} = u_{±}. Define g(τ) = log w_{τ} − u_{τ}. Note that g(τ) is concave (g′′ ≤ 0) non-negative function on [−1, 1] and g(0) ≤ log c. Therefore, the whole graph of g lies within the “butterfly” on Figure 2, and we get g(τ) ≤ 2g(0) ≤ 2 log c, which means w_{τ}e^{−u_{τ}} ≤ C^{2}.

3.3. Sharp constant in the (dyadic) Carleson embedding theorem

In this section we are going to prove the following theorem.

[figure: •; g(0); 2g(0) ≤ 2 log c; −1; 0; 1]

Figure 2: The graph of g(τ) = log w_{τ} − u_{τ} has to be inside the “butterfly”.

Theorem 3.3. If the values μ_{I}, I ∈ D satisfy the Carleson measure condition, that means

∑_{J is dyadic ; J⊂I} μ_{J} ≤ |I| for any dyadic interval I,

then for any f ∈ L^{2}

∑_{I is dyadic} μ_{I} |⟨f⟩_{I} |^{2} ≤ 4∥f∥_{L^{2}}^{2} ,

and, moreover, the constant 4 is sharp (cannot be replaced by a smaller one)

It is the well known Carleson embedding theorem (dyadic version). There exist several different proofs of it. For example, it was proved in [8] (with the same constant) by the Bellman function method. The new fact here is that the constant is sharp.

Let us prove the theorem. Let us first define the Bellman function. Fix a dyadic interval I and real numbers F, f, M. Consider all sequences {μ_{J} }_{J∈D} satisfying the Carleson condition

\frac{1}{|J|} ∑_{J′⊂J} μ_{J′} ≤ 1 ∀J ∈ D  (3.8)

and such that

\frac{1}{|I|} ∑_{J⊂I} μ_{J} = M .  (3.9)

Consider also all functions f ∈ L^{2} for which the quantities

⟨f^{2}⟩_{I} = F ,  ⟨f⟩_{I} = f  (3.10)

are fixed, and define the Bellman function B as

B(F, f, M) = \frac{1}{|I|} sup\{∑_{J⊂I} μ_{J} · ⟨f⟩_{J}^{2} : f, \{μ_{J} \}_{J∈D} satisfy (3.8), (3.9), (3.10) \}.

In terms of stochastic processes, we have a process

x^{n+1} = x^{n} + ( \begin{matrix}α_{1}^{n} \\ α_{2}^{n} \\ α_{3}^{n} \end{matrix} ) ξ^{n} + ( \begin{matrix}0 \\ 0 \\ α_{4}^{n} \end{matrix} ) , x^{0} = (F, f, M)^{T} ,

and we want to maximize 𝔼(∑_{n}(f^{n})^{2}α_{4}^{n}). Here the state space vector x is just the vector of averages (F, f, M)^{T}, that is if at the moment n we are at the interval J, then

x^{n} = (F^{n}, f^{n}, M^{n})^{T} = (⟨|f|^{2}⟩_{J} , ⟨f⟩_{J} , |J|^{−1} ∑_{J′⊂J} μ_{J} )^{T} .

The vector α of controls is determined by

α_{1}^{n} = (F_{J_{+}} − F_{J_{−}})/2, α_{2}^{n} = (f_{J_{+}} − f_{J_{−}})/2 α_{3}^{n} = (M_{J_{+}} − M_{J_{−}})/2 α_{4}^{n} = |J|^{−1}μ_{J}

Note that B(F, f, M) does not depend on the choice of an interval I.

3.3.1. Properties of B(F, f, M)

1. Domain: f^{2} ≤ F, 0 ≤ M ≤ 1 (Cauchy inequality and the Carleson condition);

2. Range: 0 ≤ B(F, f, M) ≤ CF (our belief that the theorem is true);

3. Key property: for all 0 ≤ ΔM ≤ M

B(F, f, M) ≥ ΔMf^{2} + \frac{1}{2}\{B(F_{+}, f_{+}, M_{+}) + B(F_{−}, f_{−}, M_{−})\} ,

whenever all triples (F, f, M), (F_{±}, f_{±}, M_{±}) belong to the domain and

F = \frac{1}{2}(F_{+} + F_{−}), f = \frac{1}{2}(f_{+} + f_{−}), and M = \frac{1}{2}(M_{+} + M_{−}) + ΔM .

The main property 3 is simply discrete Bellman inequality. It was explained in details how to get it in [8]; one can also just apply the technique from Sections 1, 2 to get the discrete Bellman inequality.

It is also easy to show, and was shown in [8] that existence of a function B satisfying the above properties 1–3 implies the estimate

∑_{I is dyadic} μ_{I} |⟨f⟩_{I} |^{2} ≤ C∥f∥_{L^{2}}^{2},

where C is the same as in in the property 2. So we need to show that C = 4 is the best possible constant in 2.

We need to get a continuous version of the main property 3. It again was done in [8], and here we present an alternative explanation. Namely, let us just write the Bellman equation for the continuous analogue of our process. It will be

sup_{α}\{\frac{1}{2} ∑_{j,k=1}^{3} \frac{∂^{2}B}{∂x_{j}∂x_{k}} α_{j}α_{k} + \frac{∂B}{∂M} α_{4}\} = 0.  (3.11)

This immediately implies the following 2 inequalities

3′. d^{2}B ≤ 0;

3′′. \frac{∂B}{∂M} ≥ f^{2},

which, as it was shown in [8], constitute the infinitesimal version of the main property 3 (the derivatives here can be understood in the sense of distributions). Inequalities 3’, 3” clearly follows from 3 (just consider infinitely small changes of variables). Since the domain is convex, it is not difficult to show, that the infinitesimal version implies, in turn, the discrete version 3, see again [8].

To “solve” the Bellman equation, let us eliminate one variable, using a trick of Burkholder, see [2]. We will also use this trick in the next section.

Namely, let us suppose the “real” Bellman function (i. e. obtained as sup) B satisfies the inequality B ≤ cF. Define

u(f, M) := sup_{F}{B(F, f, M) − C · F }.

We claim, that the function u is concave, and satisfies the inequalities ∂u/∂M ≥ f^{2} and −Cf^{2} ≤ u(f, M) ≤ 0.

The last inequality is trivial: we simply take supremum in the inequality

−CF ≤ B(F, f, M) − CF ≤ 0.

The concavity is not completely trivial. We know, that the infimun of concave functions is concave, but that is not true for supremum. However, we have not just a family of concave functions, but the function that is concave in all variables, and the following simple lemma takes care of everything.

Lemma 3.4. Let φ(x, y) be a concave function, and let Φ(x) := sup_{y} φ(x, y). Then Φ is concave.

The proof of this lemma is a simple exercise in convex analysis, and we omit it.

The inequality ∂u/∂M ≥ f^{2} is also not difficult, and we leave its proof to the reader. Summarizing, if the Carleson Embedding Theorem holds with constant C, there exists

a convex function u(f, M) satisfying ∂u/∂M ≥ f^{2} and such that −Cf^{2} ≤ u(f, M) ≤ 0. On the other hand, if such u exists, we can define B(F, f, M) := u(f, M) + CF, so the Carleson Embedding Theorem holds with the same constant C. Hence, the best constant in the Carleson Embedding Theorem is exactly the best constant for which the function u exists.

Now we use the homogeneity of B and u to eliminate one more variable. Namely, the “real” Bellman function B (defined as the supremum) satisfies the following property:

B(a^{2}F, af, M) = a^{2}B(F, f, M) ∀a ∈ ℝ.

Therefore

u(af, M) = a^{2}u(f, M),  (3.12)

i. e. u can be represented as

u(f, M) = f^{2}φ(M).  (3.13)

Note, that to get the homogeneity condition (3.12), one does not have to know anything about the “real” Bellman function. Namely, if we have a function ũ satisfying all the inequalities, then any its “rescaling” u_{a} := a^{−2}ũ(af, M) satisfies all the inequalities too (because all the inequalities have correct homogeneity). So, if we define u(f, M) := inf_{a≠0} a^{−2}ũ(af, M), it satisfies (3.12). It is clear that u is concave (as infimum of concave functions), and it is easy to show that ∂u/∂M ≥ f^{2} (in the sense of distributions).

For the function u defined by (3.13) the Hessian (matrix of second derivatives) equals to

( \begin{matrix}2φ(M) & 2fφ′(M) \\ 2fφ′(M) & f^{2}φ′′(M) \end{matrix} ) ,

so the concavity of u is equivalent to the following two inequalities:

φ ≤ 0, φφ′′ − 2(φ′)^{2} ≥ 0 for M ∈ [0, 1]

The inequality ∂u/∂M ≥ f^{2} means simply that φ′ ≥ 1, and φ also has to satisfy −C ≤ φ ≤ 0. Now it is an easy task to find φ. The inequality φφ′′ − 2(φ′)^{2} ≥ 0 can be rewritten as

φ^{4} (φ′/φ^{2})′ ≥ 0.

The critical case (= 0) gives rise to the differential equation

(φ′/φ^{2})′ = 0

which has general solution

φ(M) = 1/(C_{1} + C_{2}M).

Let us consider only solutions that are non-positive on [0, 1] and satisfy φ′ ≥ 1 there. Among all such solutions function

φ(M) = −4/(1 + M)

has the best lower bound on [0, 1], inf{φ(M) : M ∈ [0, 1]} = −4.

It is not difficult to show that if we consider on [0, 1] all non-negative solutions of the inequality φφ′′ − 2(φ′)^{2} ≥ 0 which satisfy φ′ ≥ 1, we do not improve the lower bound.

So, C = 4 is the best constant in the Carleson Embedding Theorem, and it is given by the function u(f, M) = −4f^{2}/(1 + M).

Note that

B(F, f, M) = u(f, M) + 4F = 4(F − f^{2}/(1 + M))

is exactly the Bellman function that was obtained in [8]. It means that the best estimate in the Carleson Embedding Theorem was already obtained in [8], although we did not realize that until now.

Note also, that the function B defined above is not the “real” Bellman function: in particular, it does not satisfy the Bellman equation (3.11) (recall, that “satisfies” means that supremum is attained at some non-zero α; it is always attained at α = 0.) Of course, this is only an indication, not a proof, for we did not prove that the Bellman Equation in this sense (supremum is attained at non-zero α) is necessary. But it is easy to see that on the boundary F = f^{2} the “real” Bellman function must satisfy the boundary condition B(F, f, M) = Mf^{2} = MF, and the function we constructed does not satisfy this condition.

So, our Bellman function only touches the real one along some set.

4. Burkholder meets Bellman

Let us recall a famous result of Burkholder [2] about basis constant for the Haar system in L^{p}. Symbol h_{I} here denotes the normalized Haar function for the interval I.

Theorem 4.1. Let 1 < p < ∞, and let f = ∑_{I∈D} c_{I} h_{I} (we can always assume that the sum is finite) and g = ∑_{I∈D} ε_{I} c_{I} h_{I}, where ε_{I} = ±1. Then for any choice of the sequence ε_{I} (= ±1)

∥g∥_{p} ≤ C_{p}∥f∥_{p};

here C_{p} = p^{∗} − 1, p^{∗} = max{p, p′}, where p′ is the conjugate exponent, 1/p + 1/p′ = 1. Moreover, the constant C_{p} here is sharp.

Let us translate Burkholder’s approach to the Bellman function language.

But first, few trivial observations. First of all, by duality, it is sufficient to prove the theorem only for p > 2 (case p = 2 is trivial because of orthogonality). So, from now on we assume that 2 < p < ∞, so p^{∗} = p. The second observation is that the theorem holds true if ε_{I} are arbitrary numbers in the interval [−1, 1], because any such sequence belongs to the convex hall of ±1 sequences (it is sufficient to consider only sequences with finitely many non-zero terms).

4.1. Bellman equation for Burkholder’s theorem

To construct the Bellman function, let us fix a dyadic interval I, and consider all functions f = ∑_{J∈D} c_{J} h_{J}, g = ∑_{J∈D} ε_{J} c_{J} h_{J}, ε_{J} = ±1, such that the averages over the interval I are fixed:

⟨f⟩_{I} = f,  ⟨g⟩_{I} = g,  ⟨|f|^{p}⟩_{I} = F.  (4.1)

Define function B(f, g, F) of 3 real variables (all functions are assumed to be real-valued as B(f, g, F) := sup{⟨|g|^{p}⟩_{I} : f, g satisfying (4.1)}. Clearly, it does not depend on the interval I, due to the right scaling.

In terms of stochastic processes, the equation of our motion is

x^{n+1} = x^{n} + α^{n}ξ^{n}, x^{0} = (f, g, F)^{T} ,

where ξ^{n} are independent (scalar) “coin tosses”, and we want to maximize lim_{n→∞} 𝔼(|x_{2}^{n}|^{p}) (only “retirement bonus” here).

All variables and controls α here are easy to understand: if at the moment n we are at an interval J, then

x^{n} = (x_{1}^{n}, x_{2}^{n}, x_{3}^{n})^{T} = (⟨f⟩_{J} , ⟨g⟩_{J} , ⟨|f|^{p}⟩_{J})^{T} ,

and

α^{n} = (α_{1}^{n}, α_{2}^{n}, α_{3}^{n})^{T} = (\frac{⟨f⟩_{J_{+}} − ⟨f⟩_{J_{−}}}{2} , \frac{⟨g⟩_{J_{+}} − ⟨g⟩_{J_{−}}}{2} , \frac{⟨|f|^{p}⟩_{J_{+}} − ⟨|f|^{p}⟩_{J_{−}}}{2} )^{T} .

Let us check the properties of the function B: it is defined on the (convex) domain |f|^{p} ≤ F (Hölder inequality), and it has to satisfy the inequalities

B(f, g, F) ≥ |g|^{p},  (4.2)
B(0, 0, F) ≤ C_{p}^{p}F  (4.3)

The last inequality (4.3) here is our belief that the theorem is true: if we consider f = ∑_{J⊂I} c_{J} h_{J}, then ⟨f⟩_{I} = ⟨g⟩_{I} = 0.

What about the main property—Bellman equation? Since there is no profit function, only the “retirement bonus”, it looks like the Bellman equation (inequality) is simply the concavity condition. However, one can easily see that it is impossible to find a concave function B in the domain |f|^{p} ≤ F, satisfying (4.2) and (4.3). So, what is wrong?

The answer is simple: we did not take into account that the controls α are not arbitrary, they satisfy some restrictions. Namely, since f = ∑_{J} c_{J} h_{J} and g = ∑_{J} ε_{J} c_{J} h_{J}, ε_{J} = ±1, we have that for any interval I,

|⟨f⟩_{I_{+}} − ⟨f⟩_{I_{−}}| = |⟨g⟩_{I_{+}} − ⟨g⟩_{I_{−}}|,

so |α_{1}| = |α_{2}|. So, the Bellman inequality is not the concavity, but restricted concavity

B(f, g, F) ≥ (B(f + α_{1}, g + α_{2}, F + α_{3}) + B(f − α_{1}, g − α_{2}, F − α_{3}))/2  (4.4)

whenever |α_{1}| = |α_{2}| (and all the arguments belong to the domain).

Note, that if we construct any function B = B(f, g, F) on the domain |f|^{p} ≤ F, satisfying the Bellman inequality (4.4) and the estimates (4.2), (4.3), then we prove the theorem.

Indeed, consider f supported by a dyadic interval I, f = ∑_{J⊂I} c_{J} h_{J}. Then the Bellman inequality (4.4) implies that

|I| · B(0, 0, ⟨|f|^{p}⟩_{I}) = |I| · B(⟨f⟩_{I} , ⟨g⟩_{I} , ⟨|f|^{p}⟩_{I})
≥ |I_{+}| · B(⟨f⟩_{I_{+}}, ⟨g⟩_{I_{+}}, ⟨|f|^{p}⟩_{I_{+}}) + |I_{−}| · B(⟨f⟩_{I_{−}}, ⟨g⟩_{I_{−}}, ⟨|f|^{p}⟩_{I_{−}})

Writing the Bellman inequalities to I_{+} and I_{−}, then to their halves, we get after going n generations down:

|I| · B(0, 0, ⟨|f|^{p}⟩_{I}) ≥ ∑_{J⊂I:|J|=2^{−n}|I|} |J| · B(⟨f⟩_{J} , ⟨g⟩_{J} , ⟨|f|^{p}⟩_{J})

≥ ∑_{J⊂I:|J|=2^{−n}|I|} |J| · ⟨|g|^{p}⟩_{J}

(the last inequality here is just (4.2)).

Taking limit as n → ∞ and using (4.3) we get

C_{p}^{p} ∫_{I} |f|^{p} = C_{p}^{p}|I|⟨|f|^{p}⟩_{I} ≥ |I| · B(0, 0, ⟨|f|^{p}⟩_{I}) ≥ ∫_{I} |g|^{p}.

To complete the proof it remains to notice that any finite sum f = ∑ c_{J} h_{J} is supported by a union of at most 2 disjoint dyadic intervals.

4.2. Finding B: biconcave function

To find B, let us first understand what does the restricted concavity (4.4) mean.

To do that let us do a simple change of variables. Let us introduce new variables, x := g + f, y := g − f, so

f = (x − y)/2, g = (x + y)/2.

Define B̃(x, y, F) := B(f, g, F) = B((x − y)/2, (x + y)/2, F). Then the restricted concavity (4.4) of B just mean that the function B̃ for any fixed y is concave in arguments x, F, and for any fixed x is concave in arguments y, F. The inequalities (4.2), (4.3) can be translated into

B̃(x, y, F) ≥ |\frac{x + y}{2}|^{p}
B̃(0, 0, F) ≤ C_{p}^{p}F.

Domain of B̃ will be |\frac{x−y}{2} |^{p} ≤ F.

Burkholder in [2, 3] did the above change of variables in the original problem, using the so-called “zigzag martingales”.

4.2 Finding B: biconcave function 27

To find B̃, one can use the homogeneity of the “real” Bellman function B to get the condition

B̃(ax, ay, |a|^{p}F) = |a|^{p}B̃(x, y, F).

This allows us to reduce the number of variable to 2, and the partial differential equations appearing from the concavity conditions can be solved (by Burkholder, not by us).

It is really amazing, that Burkholder, see [3] was able to solve these PDE’s: they are really complicated, and solving them definitely goes beyond our abilities.

However, there is an easier method, also due to Burkholder. This method gives us a supersolution with the best constant, as well as the proof that the constant is best. We essentially following Burkholder [2] here, and we refer the reader to [2, 3] for all the details.

Suppose we found a function B satisfying B(0, 0, F) ≤ C_{p}^{p}F with some C_{p}, as well as all other necessary inequalities. Changing variables to x, y we get the function B̃(x, y, F). Let us now eliminate variable F, using the same trick as in the previous section with the Carleson Embedding Theorem. Define

u(x, y) := sup_{F ≥|\frac{x−y}{2}|^{p}} \{B̃(x, y, F) − C_{p}^{p}F \} .

Using Lemma 3.4 we get that the function u is biconcave, that means the functions u(x, ·) and u( · , y) are concave. The function u also satisfies the inequalities

u(0, 0) ≤ 0,  u(x, y) ≥ |\frac{x + y}{2}|^{p} − C_{p}^{p} · |\frac{x − y}{2}|^{p} .  (4.5)

The first one is trivial, because B̃(0, 0, F) − C_{p}^{p}F ≤ 0, and the second one follows from the inequality

B̃(x, y, F) − C_{p}^{p}F ≥ |\frac{x + y}{2}|^{p} − C_{p}^{p}F

and the fact that sup \{−F : F ≥ |\frac{x−y}{2} |^{p}\} = − |\frac{x−y}{2} |^{p}.

We have proved, that if a function B exists for some C_{p}, then a function u exists for the same constant C_{p}. The converse is also true: if a function u exists for some C_{p}, then we can define B̃(x, y, F) := u(x, y) + C_{p}^{p}F, and then changing variables x, y back to f and we get B. Thus the estimate in the Burkholder’s theorem is equivalent to the existence of such biconvex u, and the best constant is the best C_{p} for which such u exists.

Note, that without loss of generality one can assume that u is symmetric and homogenous of degree p:

u(x, y) = u(y, x),  u(ax, ay) = |a|^{p}u(x, y), a ∈ ℝ.  (4.6)

For a function u obtained from the “real” Bellman function it can be deduced from the symmetry and homogeneity of B, but there is an alternative, very simple explanation. Namely, if a function u = u(x, y) satisfies inequalities (4.5), then the functions |a|^{−p}u(ax, ay) and u(y, x) satisfy them as well. So, if we have some biconcave u_{0}, then we can define a symmetric and homogenous biconcave u by

u(x, y) = inf_{a≠0} |a|^{−p}(u_{0}(ax, ay) + u_{0}(ay, ax))/2

(infimum of concave functions is concave). Note, that u(0, 0) = 0 even if u_{0}(0, 0) > 0, so we do not have to worry about first inequality in (4.5).

The fact that u is homogenous means that it is (almost) completely defined by w(x) := u(x, 1), namely u(x, y) = |y|^{p}w(x/y) if y ≠ 0.

If we denote

v(x, y) = |\frac{x + y}{2}|^{p} − C_{p}^{p} · |\frac{x − y}{2}|^{p}

then clearly w(x) ≥ v(x, 1). The function w is clearly concave. Note, that a p-homogenous u satisfies u(x, y) = u(y, x) if and only if

w(x) = |x|^{p}w(1/x).  (4.7)

Indeed,

u(x, y) = |y|^{p}w(x/y) = |y|^{p}|x/y|^{p}w(y/x) = |y|^{p}w(y/x) = u(y, x).

In this case u can be completely restored from w by putting u(x, 0) = |x|^{p}w(0). Note, that if w is concave (and satisfies (4.7)), then the restored function u is biconcave: indeed, for any fixed y the function u( · , y) is clearly concave, and the symmetry condition u(x, y) = u(y, x) implies that u(x, ·) is concave as well.

Therefore, given C_{p} ≥ 1, the existence of the biconcave u (and so the Burkholder’s Theorem) is equivalent to the existence of a concave w satisfying (4.7) and such, that w(x) ≥ v(x, 1).

Let x_{0} be the solution of the equation v(x, 1) = 0 on the interval [−1, 1], x_{0} = (C_{p} − 1)/(C_{p} + 1).

Lemma 4.2. A concave function w as above exists if and only if C_{p} ≥ p − 1 (we are considering only the case p ≥ 2 here). Moreover, if C_{p} = p − 1, then w is uniquely defined on the interval [x_{0}, 1/x_{0}], and it is an affine function on [x_{0}, 1].

Proof. If

L := lim_{x→1−} \frac{w(x) − w(1)}{x − 1}, R := lim_{x→1+} \frac{w(x) − w(1)}{x − 1}

then (4.7) implies that L = pw(1) − R. Since w is concave, we have

pw(1) = L + R ≤ 2L ≤ 2\frac{w(1) − w(x_{0})}{1 − x_{0}} = (C_{p} + 1)(w(1) − w(x_{0})) ≤ (C_{p} + 1)w(1);

the last inequality here follows from the fact that w(x_{0}) ≥ v(x_{0}, 1) = 0.

So, pw(1) ≤ (C_{p} + 1)w(1), and since w(1) ≥ v(1, 1) > 0, we get C_{p} ≥ p − 1.

If C_{p} = p − 1, then all the inequalities in the above chain become equalities, and that means on the interval [x_{0}, 1] (x_{0} = 1 − 2/p in this case) the function w must be an affine function satisfying w(x_{0}) = 0. Since w is a concave majorant of v(x, 1), the slope must be equal to the derivative \frac{d}{dx}v(x, 1) at x = x_{0}. So, indeed, the function w (if it exists) is uniquely defined on [x_{0}, 1] (and so on [x_{0}, 1/x_{0}]).

To prove that the condition C_{p} ≥ p − 1 is sufficient it is enough to construct w for the critical case C_{p} = p − 1. The function w is uniquely defined on the interval [x_{0}, 1/x_{0}], and outside of this interval we can just put w(x) = v(x, 1). Of course, it remains to check that w is a concave majorant of v(x, 1), and it is nor hard to do.

Note, that function w is not unique, there are other choices possible. For example, we can choose w to be an affine function on [−1, 1] (extension of the affine function on [x_{0}, 1]), and then extend it to ℝ according to (4.7). Again, one has to check that w majorates v(x, 1) and that it is concave (outside of [−1, 1]). The resulting function is given by

w(x) = α_{p} \{|\frac{x + 1}{2}| − (p^{∗} − 1) |\frac{x − 1}{2}|\} \{|\frac{x + 1}{2}| + |\frac{x − 1}{2}|\}^{p−1}  (4.8)

where α_{p} = p · (1 − 1/p^{∗})^{p−1}. Although we only considered the case p > 2, the resulting function works for p < 2 as well, and that is why we are using p^{∗} := max{p, p′}, 1/p+1/p′ = 1 here.

Formula (4.8) gives rise to

u(x, y) = \{|\frac{x + y}{2}| − (p^{∗} − 1) |\frac{x − y}{2}|\} \{|\frac{x + y}{2}| + |\frac{x − y}{2}|\}^{p−1} .

Going back to the original variables f, g we get a Bellman function

B(f, g, F) = α_{p} {|g| − (p^{∗} − 1)|f|} {|f| + |g|}^{p−1} + (p^{∗} − 1)F.

5. Conclusions

• Our Bellman equation in many cases (in all cases we considered here) can be interpreted as a particular case of the classical Bellman equation for stochastic control (see [6]). Recall, that the deterministic bellman equation is a first order PDE. The second order terms in our case are due to the probabilistic nature of the problem (Ito’s formula).

• Every (dyadic) harmonic analysis problem, which scales correctly, has the Bellman equation assigned to it.

• The process that correspond to a dyadic harmonic analysis problem is a discrete time process, so the Bellman equation is in fact a system of discrete (finite difference) inequities. One can consider their continuous (infinitesimal) version by taking infinitely small changes of variables. This infinitesimal version can be also obtained (at least formally) by writing the bellman equation for a continuous time analogue of our discrete time process.

• Moving from a continuous (infinitesimal) Bellman equation (function) is usually pretty easy if the domain is convex. If the domain is not convex, some extra work is needed, and it can be rather tricky to get from the continuous version to the discrete one.

• The Bellman equation is best to interpret as a system of inequalities. There are several reasons for that. First of all, if we just formally write the equation (1.8), in our situation any supersolution, i. e. any function satisfying the inequalities satisfies the equation as well (supremum is attained and equal 0 at α = 0). So something else is needed to get the equation: for example one can require that the supremum is attained at some non-zero α.

Moreover, if we find a way to interpret the equation, its solvability is not equivalent to our original problem. We do not know if it is necessary, because we do not know apriori that B is smooth, and we do not know if it is sufficient.

So, while thinking about equation can provide some useful heuristics, it is better to work with it as with a system of inequalities.

• The Bellman equations, appearing in harmonic analysis are quite different from the classical PDEs: we do not know any existence and uniqueness results (see also above item about interpretation). The boundary values are not essential in our case (although in some situations they can be written), but some bounds on B are very important.

• The domain is very important: it can mean the difference between true and false statements. For example, it is well known that the Muckenhoupt condition is sufficient for the boundedness of Haar multipliers in L^{2}(w), see [16], but its two-weight analogue is not, see[10]. From the point of view of Bellman function, the difference is not in the equation, but in the domain.

• In classical stochastic control, the “retirement bonus” F usually does not appear, but in harmonic analysis it is very often an important part. Unlike profit function f^{α} it does not figure in the equation, but gives rise to bounds on the Bellman function.

• It is very difficult usually (if at all possible) to find the exact solution of the corresponding Bellman equation. Problem of finding a supersolution is often much more tractable.

• Harmonic analysis Bellman equations (inequalities) not always can be written in infinitesimal (continuous) form, see, for example [7].

References

[1] St. Buckley, Summation conditions on weights, Mich. Math. J., 40 (1993), 153-170.

[2] D. L. Burkholder, Explorations in martingale theory and its applications, Ecole d’Eté de Probabilité de Saint-Flour XIX–1989, 1-66, Lecture Notes in Mathematics, 1464, Springer, Berlin, 1991.

[3] D. L. Burkholder, Boundary value problems and sharp inequalities for martingale transforms, Ann. Prob., 12(1984), 647–802.

[4] R. Fefferman, C. Kenig, J. Pipher, The theory of weights and the Dirichlet problem for elliptic equations, Annals of Math., 134, (1991), 65-124.

[5] S. Hucovic, S. Treil, A. Volberg, The Bellman functions and the sharp square estimates for square functions, Operator Theory: Advances and Applications, the volume in memory of S. A. Vinogradov, v. 113, Birkhauser Verlag, 2000.

[6] N. Krylov, Optimal Control of Diffusion Processes, Springer-Verlag, New York, 1980, 308 p.

[7] F. Nazarov, The Beurling lemma via the Bellman function, Preprint, MSU, 1996.

[8] F. Nazarov, S. Treil, The hunt for Bellman function: applications to estimates of singular integral operators and to other classical problems in harmonic analysis, Algebra i Analysis, 8 (1997), no. 5, 32-162.

[9] F. Nazarov, S. Treil, Hunt-Muckenhoupt-Wheeden inequality is now trivial, C. R. Acad. Sci. Paris, Ser. I, 323(1996), 717–722.

[10] F. Nazarov, S. Treil, and A. Volberg, The Bellman functions and two-weght inequalities for Haar multipliers, J. of the Amer. Math. Soc., v. 12, No. 4, 1999, 909-928.

[11] F. Nazarov, A. Volberg, Heating of the Beurling operator and estimates of its norm, Preprint, MSU, 2000, p. 1-12.

[12] F. Nazarov, A. Volberg, Bellman function and T 1 conditions for two-weight Hilbert transform with application to Carleson imbedding theorem for spaces K_{θ}, Preprint, MSU, 2000, p. 1-28.

[13] St. Petermichl, A. Volberg, Heating of the Beurling operator: weakly quasiregular maps on the plane are quasiregular, Preprint, MSU, 2000, p. 1-25.

[14] St. Petermichl, J. Wittwer, A sharp weighted estimate on the norm of Hilbert transform via invariant A_{2} characteristic of the weight, Preprint, MSU, 2000.

[15] E. Stein, Harmonic Analysis: Real-variable Methods, Orthogonality, And Oscillatory Integrals, Princeton Math. Series, 43, Monographs in Harmonic Analysis, Princeton Univ. Press, Princeton, NJ, 1993.

[16] S. Treil, A. Volberg, Wavelets and the angle between past and future, J. Funct. Analysis, 143, No. 2, (1997), 269-308.

[17] J. Wittwer, Ph. D. Thesis, University of Chicago, 2000.