review_iclr_bench/iclr_parsed/1Z5P--ntu8.txt

# ON THE GLOBAL CONVERGENCE OF GRADIENT DE## SCENT FOR MULTI-LAYER RESNETS IN THE MEAN- FIELD REGIME

**Anonymous authors**
Paper under double-blind review

ABSTRACT

Finding the optimal configuration of parameters in ResNet is a nonconvex minimization problem, but first order methods nevertheless find the global optimum
in the overparameterized regime. We study this phenomenon with mean-field
analysis, by translating the training process of ResNet to a gradient-flow partial
differential equation (PDE) and examining the convergence properties of this limiting process. The activation function is assumed to be 2-homogeneous or partially
1-homogeneous; the regularized ReLU satisfies the latter condition. We show that
if the ResNet is sufficiently large, with depth and width depending algebraically
on the accuracy and confidence levels, first order optimization methods can find
global minimizers that fit the training data.

1 INTRODUCTION

Training of multi-layer neural networks (NN) requires us to find weights in the network such that its
outputs perfectly match the prescribed outputs for a given set of training data. The usual approach
is to formulate this problem as a nonconvex minimization problem and solve it with a first-order
optimization method based on gradient descent (GD). Extensive computational experience shows
that in the overparametrized regime (where the total number of parameters in the NN far exceeds the
minimum number required to fit the training data), GD methods run for sufficiently many iterations
consistently find a global minimum achieving the zero-loss property, that is, a perfect fit to the training
data.

What is the mechanism that allows GD to perform so well on this large-scale nonconvex problem?

Part of the explanation is that in the overparametrized case, the parameter space contains many
global minima, and some evidence suggests that they are distributed throughout the space, making
it easier for the optimization process to find one such solution. Many approaches have been taken
to characterize this phenomenon more rigorously, including landscape analysis, the neural tangent
kernel approach, and mean-field analysis. All such viewpoints aim to give an idea of the structure
and size of the NN required to ensure global convergence.

Our approach in this paper is based on mean-field analysis and gradient-flow analysis, the latter being
the continuous and mean-field limit of GD. We will examine residual neural networks (ResNets),
and study how deep and wide a ResNet needs to be to match the data with high accuracy and high
confidence. To relax the assumptions on the activation function as far as possible, we follow the
setup in (Chizat & Bach, 2018), which requires this function to be either 2-homogeneous or partially
1-homogeneous. We show that both depth and width of the NN depend algebraically on ϵ and η,
which are the accuracy and confidence levels, respectively.

Mean-field analysis translates the training process of the ResNet to a gradient-flow partial differential
equation (PDE). The training process evolves weights on connections between neurons. When dealing
with wide neural networks, instead of tracing the evolution of each weight individually, one can
record the evolution of the full distribution of the weight configuration. This perspective translates
the coupled ordinary differential equation system (ODE) that characterizes evolution of individual
weights into a PDE (the gradient-flow equation) characterizing the evolution of the distribution. The
parameters in the PDE naturally depend on the properties of the activation functions. Gradient-flow


-----

analysis is used to show that the PDE drives the solution to a point where the loss function becomes
zero. We obtain our results on zero-loss training of ResNet with GD by translating the zero-loss
property of the gradient-flow PDE back to the discrete-step setting.

This strategy of the proof was taken in an earlier paper (Ding et al., 2021) where multi-layer ResNets
were also analyzed. The main difference in this current paper is that the assumptions on the activation
function and the initial training state for obtaining the global convergence are both much relaxed. This
paper adopts the setup from (Chizat & Bach, 2018) of minimal Lipschitz continuity requirements on
the activation function. Furthermore, the paper (Ding et al., 2021) required a dense support condition
to be satisfied on the final parameter configuration has a support condition. This condition is hard
to justify in any realistic setting, and is discared from current paper. Further details on these issues
appear in Section 3.

We discuss the setup of the problem and formally derive the continuous and mean-field limits in
Section 2. In Section 3, we discuss related work, identify our contribution and present the main
theorem in its general terms. After precise definitions and assumptions are specified in Section 4, we
present the two main ingredients in the proof strategy. The mean-field limit is obtained by connecting
the training process of the ResNet to a gradient-flow PDE in Section 5, and the zero-loss property of
the limiting PDE is verified in Section 6. The main theorem is a direct corollary of Theorem 5.1 and
Theorem 6.1 (or Theorem 6.2).

2 RESNET AND GRADIENT DESCENT

The ResNet can be specified as follows:


_f_ (zl(x), θl,m), _l = 0, 1, . . ., L_ 1, (1)
_−_
_m=1_

X


_zl+1(x) = zl(x) +_

_ML_


where M and L are the width and depth, respectively; z0(x) = x ∈ R[d] is the input data; and
_zL(x) is the output from the last layer. The configuration of the NN is encoded in parameters_
ΘL,M = _θl,m_ _l=0,m=1[, where each parameter][ θ][l,m][ is a vector in][ R][k][ and][ f][ :][ R][d][ ×][ R][k][ →]_ [R][d][ is the]
_{_ _}[L][−][1][,M]_
activation function. The formulation (1) covers“conventional” ResNets, which have the specific form


1

_ML_ _Ul,mσ(Wl,m[⊤]_ _[z][l][(][x][) +][ b][l,m][)][,]_ _l = 0, 1, . . ., L −_ 1,

_m=1_

X


_zl+1(x) = zl(x) +_


where Wl,m, Ul,m ∈ R[d], bl.m ∈ R, and σ is the ReLU activation function. In this example, we have
_θl,m = (Wl,m, Ul,m, bl,m) ∈_ R[k], with k = 2d + 1.

Denote by ZΘL,M (l; x) the output of the ResNet defined by (1). (This quantity is the same as zL(x)
defined above, but we use this alternative notation to emphasize the dependece on parameters ΘL,M .)
The goal of training ResNet is to seek parameters ΘL,M that minimize the following mismatch or
_loss function:_

1 2

_E(ΘL,M_ ) = Ex _µ_ _g(ZΘL,M (L; x))_ _y(x)_ _,_ (2)
_∼_ 2 _−_

 

 


where g(x) : R[d] _→_ R is a given measuring function, y(x) ∈ R is the label corresponding to x, and µ
is the probability from which the data x is drawn.

Classical gradient descent updates the parameters according to the formula


Θ[n]L,M[+1] [= Θ]L,M[n] _[−]_ _[h][∇][Θ][E][(Θ]L,M[n]_ [)][,]

where h is the step length. In the limit as h → 0, the updating process can be characterized by the
following ODE (Chizat & Bach, 2018, Def 2.2) (rescaled by L, M ):

dΘL,M (s)

= _ML_ ΘE(ΘL,M ), for s 0, (3)
ds _−_ _∇_ _≥_

where s represents pseudo-time, the continuous analog of the discrete stepping process.


-----

2.1 THE CONTINUOUS LIMIT AND THE MEAN-FIELD LIMIT

The continuous limit of (1) is obtained when the ResNet is infinitely deep, with L →∞. By
reparametrizing the indices l = [0, · · ·, L − 1] with the continuous variable t ∈ [0, 1], we can view z
in (1) as a function in t that satisfies a coupled ODE, with 1/L being the stepsize in t. Accordingly,
_θl,m can be recast as θm(t = l/L), and denoting Θ(t) = {θm(t)}m[M]=1[, we can write the continuous]_
limit of (1) as


dz(t; x) = [1]

dt _M_


_f_ (z(t; x), θm(t)), _t_ [0, 1], with z(0; x) = x . (4)
_∈_
_m=1_

X


Extending (2), we define the cost functional E as

1

_E(Θ) = Ex_ _µ_ _,_ (5)
_∼_ 2 [(][g][(][Z][Θ][(1;][ x][))][ −] _[y][(][x][))][2]_

 

where ZΘ(t; x) solves (4) for a given collection Θ(t) of the M functions _θm(t)_ . Similar to (3),
_{_ _}_
we can use GD to find the configuration of Θ(t) that minimizes (5) by making Θ(t) flow in the
descending direction of E(Θ). Denote s the pseudo-time of the training process, and Θ(s; t) the
collection of functions at the training time s:


_∂Θ_

_∂s_ [=][ −][M δE]δΘ


Θ(s;·) _,_ _s > 0,_ _t ∈_ [0, 1] (6)


where _[δE]δΘ_ [is the functional derivative of][ E][ with respect to][ Θ][, and thus a list of][ M][ functions of][ t][ for]

every fixed s.

The mean-field limit is obtained by making the ResNet infinitely wide, that is, M →∞. Considering
that the right hand side of (4) has the form of an expectation, it approaches an integral in the limit,
with respect to a certain probability density. Denoting this PDF by ρ(θ, t) ∈C([0, 1]; P [2])[1], and
assuming that the θm are drawn from it, the ODE for z translates to the following:


dz(t; x)

dt


_t ∈_ [0, 1] with z(0; x) = x . (7)
R[k][ f] [(][z][(][t][;][ x][)][, θ][) d][ρ][(][θ, t][)][,]


Mimicking (5), we define the following cost function in the mean-field setting:

1

_E(ρ) = Ex_ _µ_ _,_ (8)
_∼_ 2 [(][g][(][Z][ρ][(1;][ x][))][ −] _[y][(][x][))][2]_

 

where Zρ(t; x) is the solution to (7) for a given ρ. Then, similar to the gradient flow for ΘL,M
and Θ(t), the probability distribution ρ that encodes the configuration of θ flows in the descending
direction of E(ρ) in pseudo-time s. Since ρ(θ, t, s) needs to be a probability density for all s and t,
its evolution in s is characterized by a gradient flow in the Wasserstein metric (Chizat & Bach, 2018;
Lu et al., 2020; Ding et al., 2021):


_∂ρ_
_∂s_ [=][ ∇][θ][ ·]


_δE_
_ρ_ _θ_
_∇_ _δρ_


_s > 0, t_ [0, 1] with _ρ(θ, t, 0) = ρini(θ, t),_ (9)
_∈_


_ρ(·,·,s)_


where _[δE]δρ_ [is the functional derivative with respect to][ ρ][, and thus a function of][ (][θ, t][)][ for every fixed][ s][.]

Using the classical calculus-of-variations method, this functional derivative can be computed as:


_δE_

_δρ_


(θ, t) = Ex _µ_ _p[⊤]ρ_ [(][t][;][ x][)][f] [(][Z][ρ][(][t][;][ x][)][, θ][)] _,_ (10)
_∼_
 



where pρ(·; x), parameterized by x, maps [0, 1] → R[d], and is a vector solution to the following ODE:


dp[⊤]ρ

dt = −p[⊤]ρ


(11)
R[k][ ∂][z][f] [(][Z][ρ][, θ][)][ρ][(][θ, t][) d][θ .]


1A collection of probability distribution that is continuous in t and has bounded second moment in θ for all t.
The definition is to be made rigorous in Def 4.1.


-----

with pρ(t = 1; x) = (g(Zρ(1; x)) _y(x))_ _g(Zρ(1; x)). In the later sections, to emphasize the s_
_−_ _∇_

dependence, we use _[δE][(Θ(]δΘ_ _[s][))]_ and _[δE][(]δρ[ρ][(][s][))]_ to denote _[δE]δΘ_ Θ(s;·) [and][ δE]δρ _ρ(·,·,s)_ [respectively. As a]

summary, to update ρ(θ, t, s) to ρ(θ, t, s + δs) with an infinitesimal δs, we solve (7) for Zρ(t; x),
using the given ρ(θ, t, s), and compute pρ using (11). This then allows us to compute _[δE][(]δρ[ρ][(][s][))]_ (θ, t)

which, in turn, yields ρ(θ, t, s + δs) from (9). In (11), ∂zf is a d _d matrix that stands for the_
_×_
Jacobian of f with respect to its z argument.

3 RELATED WORK AND CONTRIBUTION

There is a vast literature addressing the overparameterization of DNN. Many perspectives have been
taken to justify the success of the application of the first order (gradient descent) optimization methods,
in this overparameterized regime. We briefly review related works, and identify our contribution.

The earliest approach to understanding overparametrization was landscape analysis, in which the
countours of the nonconvex objective function were studied to find which properties make it possible
for a first order method to converge to the optimizer. Different NN structures are then analyzed to see
which have these properties (Jin et al., 2017; Ge et al., 2015; Du et al., 2017; Ge et al., 2018; Nguyen
& Hein, 2018; Du & Lee, 2018; Soltanolkotabi et al., 2019; Nguyen & Hein, 2017; Kawaguchi, 2016;
Yun et al., 2018). This approach naturally limits the types of DNN that can be “explained,” since
most DNN structures do not satisfy the required properties.

Another approach taken in the literature is related to the Neural Tangent Kernel (NTK) regime,
which is the regime in which the nonlinear problem is reduced to a nearly linear model due to the
confinement of the iterates to a small region around the initial values. Insensitivity of the so-called
Gram matrix is evaluated in the limit of the number of weights (Allen-Zhu et al., 2019; Du et al.,
2019a; Zhang et al., 2019; Chatterji et al., 2021; Du et al., 2019b; Jacot et al., 2018; Liu et al., 2020;
Frei et al., 2019). The argument is that zero-loss solutions are close to every point in the space,
and one can find an optimal point within a small region of the initial guess. The NTK arguments
are shown to work well in several real application problems, such as the classification problem (Li
& Liang, 2018; Zou et al., 2019). However, as pointed out by (Ba et al., 2020; Wei et al., 2019;
Fang et al., 2019), NTK approximately views nonlinear DNN as a linear kernel model, a rather
limited description, so the estimates obtained through NTK might not be sharp. Indeed, the empirical
observation in (Allen-Zhu & Li, 2019; Arora et al., 2019) have suggested that the kernel models are
not as general as NN, and certain (nonlinear) features of NN are not captured.

Finally, there is the mean-field limit perspective that we adopt in this paper. The term “mean-field"
indicates that in a system with a large ensemble of particles, the field formed by averaging across
all samples exerts a force on each sample. Instead of tracing the trajectory of each sample, one can
characterize the evolution of the full distribution function that represents the field. This idea originated
in statistical physics, and is made rigorous under the framework of kinetic theory. In training an
overparametrized ResNet context, a large number of weights evolve to decrease the cost function.
In the mean-field limit, the training process evolves the distribution function of these weights. A
significant advantage of the mean-field approach is that once we derive a formula for the gradient flow,
standard PDE techniques can be adopted to describe the convergence behavior. This approach was
taken in (Araújo et al., 2019; Fang et al., 2019; Nguyen, 2019; Du et al., 2019a; Chatterji et al., 2021;
Chizat & Bach, 2018; Mei et al., 2018; Wojtowytsch, 2020; Lu et al., 2020; Sirignano & Spiliopoulos,
2021; 2020). The case of a single hidden layer NN in the regime as M →∞ is studied by Chizat
& Bach (2018); Mei et al. (2018); Wojtowytsch (2020), who justified the mean-field approach and
demonstrated convergence of the gradient flow process to a zero objective. In the multi-layer case,
Lu et al. (2020) showed the convergence of a PDE that can be viewed as a modified version of the
true gradient flow, hinting at convergence of the real mean-field limit. Nguyen & Pham (2021) also
gave the global convergence of the mean-field limit of DNN for a certain class of NN structures, but
their work excludes such important practical NN structures as ResNet. The work most closely related
to ours is (Ding et al., 2021), but this paper makes technical assumptions on ρ and f that restrict
_∞_
the usefulness of the results, as we discuss below following the statement of Theorem 3.1.

We note that in certain parameter regimes, the mean-field and NTK perspectives can sometimes be
unified; see (Chen et al., 2020).


-----

We follow the roadmap of Chizat & Bach (2018); Ding et al. (2021), which shows that the PDE (9)
achieves the global minimum for which E(ρ(θ, t, s = ∞)) = 0, and that the gradient flow in the discrete setting (3) can be closely approximated by the PDE, so that E(ΘL,M (s)) _E(ρ(_ _,_ _, s)). These_
_≈_ _·_ _·_
two results together show that E(ΘL,M (s)) 0 for pseudo-time s sufficiently large. Specifically,
_≈_
the two main tasks of the paper are as follows.

– Task 1: We need to give a rigorous proof of the continuous and mean-field limit. This
will be stated in Theorem 5.1, to justify that for every fixed s < ∞, when M, L →∞,
_E(ΘL,M_ (s)) _E(Θ(s;_ )) _E(ρ(_ _,_ _, s)). The dependence of these approximations on L_
_≈_ _·_ _≈_ _·_ _·_
and M are made precise.

– Task 2: We need to demonstrate the convergence to global minimum. This is stated
in Theorem 6.1 and 6.2, for two different cases. In both theorems, we obtain the global
convergence for the gradient flow, assuming certain homogeneity and the Sard-type regularity
for f . A weak assumption of the initialization of ρini is also imposed.

By combining these two, we obtain the main result of the paper.

**Theorem 3.1 Let the conditions in Theorem 5.1 and 6.1 (or 6.2) hold. Then for any positive ϵ and η,**
_there exist positive constants C0 depending on ρini(θ, t), ϵ and C depending on ρini(θ, t), s such that_
_when_

_s > C0(ρini(θ, t), ϵ),_ _M > [C][(][ρ][ini][(][θ, t][)][, s][)]_ _,_ _L > [C][(][ρ][ini][(][θ, t][)][, s][)]_ _,_

_ϵ[2]η_ _ϵ_

_we have_
P (|E(ΘL,M (s))| ≤ _ϵ) ≥_ 1 − _η,_
_where E is defined in (2) and ΘL,M solves (3)._

This theorem gives quantitative bounds for M and L. The number of weights required to reduce
the cost function below ϵ is O(ML) = O(1/ϵ[3]). The theorem also suggests that L and M are
independent parameters.

The results resonate with those obtained in (Chizat & Bach, 2018) for the 2-layer NN, and extend
those in (Ding et al., 2021) greatly. Specifically, compared with the results in (Chizat & Bach, 2018),
where ρ(θ, s) follows a typical gradient flow in the probability space on θ in time s (Ambrosio et al.,
2008), we have, at each training time s, a “list” of probability measures ρ(θ, t) on θ, for all t. The
members of this list are coupled, flowing together in s in the descending direction of the cost function
_E. New analytical estimates are developed to deal with this non-traditional gradient flow._

Ding et al. (2021) takes a similar approach to ours, but their assumptions on the support of ρ(θ, t, s =
_∞) are quite strong: The limiting probability measure ρ(θ, t, s = ∞) is assumed to have the_
full support over θ. The assumption greatly reduces the technical difficulty of the proof, but it
is impractical and hard to justify, thus preventing the results from being of practical use. In the
current paper, this support condition is replaced by the well-accepted homogeneity condition adopted
by (Chizat & Bach, 2018). As a consequence, the structure of the gradient flow must be examined
closely to demonstrate convergence, requiring considerable technical complications. Lu et al. (2020)
also investigates gradient flow for training multi-layer neural network, but the gradient flow structure
is modified for mathematical convenience. All blocks are integrated together, making ρ a probability
measure over the full (θ, t)-space. This design is inconsistent with the structure of the ResNet design
that we investigate in this paper.

4 NOTATIONS, ASSUMPTIONS, AND DEFINITIONS

Throughout the paper we denote the collection of probability distributions with bounded second
moments as P [2](R[k]), that is, P [2](R[k]) = {ρ : R[k][ |][θ][|][2][ d][ρ][(][θ][)][ <][ ∞}][. We assume certain regularity]

properties for the activation function f, the measuring function g, the data y, and the input measure
_µ, as follows._ R

**Assumption 4.1 (Assumptions on f** **) Let f : R[d]** _× R[k]_ _→_ R[d] _be a C[2]_ _function._

_1. (linear growth) For all x ∈_ R[d], θ ∈ R[k], there is a constant C1 such that
_f_ _C1(_ _θ_ + 1)( _x_ + 1) . (12)
_|_ _| ≤_ _|_ _|[2]_ _|_ _|_


-----

_2. (locally Lipschitz) For all r > 0, and |x| < r, θ ∈_ R[k], we have for C2(r) monotonically
_increasing with respect to r that the following bounds hold:_
_∂xf_ _C2(r)(_ _θ_ + 1), _∂θf_ _C2(r)(_ _θ_ + 1) . (13)
_|_ _| ≤_ _|_ _|[2]_ _|_ _| ≤_ _|_ _|_


_3. (local smoothness) There exists k1_ (0, k] with the following property: Denoting θ =
(monotonically increasing with respect toθ[1], θ[2]), r = max{|x|, |θ[1]|}, where ∈ _r θ that the following bounds hold:[1] ∈_ R[k][1], θ[2] ∈ R[k][−][k][1], we have for C3(r)
_∂x[i]_ _[∂]θ[j][f]_ [(][x, θ][)] _≤_ _C3(r),_ _i + j = 2, i, j ≥_ 0 . (14)

_When k1 < k, we have in addition that_
max _∂xf_ _,_ _f_ _C3(r)_ _θ[2]_ + 1 _,_ _∂θ[1]_ _f_ _C3(_ _x_ ) _θ[1]_ + 1 _._ (15)
_{|_ _|_ _|_ _|} ≤_ _|_ _|_ _≤_ _|_ _|_ _|_ _|_
 
  


_4. (universal kernel) The function set_ _h_ _h =_ _is dense in_

R[k][ f] [(][x, θ][) d][ρ][(][θ][)][, ρ][ ∈P] [2][(][R][k][)]
_x_ _< R; R[d][
]_ _for all R > 0._
_C_ _|_ _|_  R

**Assumption 4.2 (Assumptions on data)** _Let g, y : R[d]_ _→_ R be C[2] _functions, and let µ be the_
_probability distribution of x. We assume the following._

_5. µ(x) is compactly supported, meaning that there is Rµ > 0 such that the support of µ is_
_within a ball of size Rµ around the origin, that is, supp(µ) ⊂BRµ_ ([⃗]0).


_6. y(x)_ _L[∞]loc[(][R][d][)][, that is,][ sup]_ _x_ _R_
_∈_ _|_ _|≤_ _[|][y][(][x][)][|][ <][ ∞][.]_

_7. gthat is,(x) and inf ∇x∈gR(xd |∇) are Lipschitz continuous. Moreover,g(x)| > 0._ _|∇g(x)| has a positive lower bound,_

We note that Assumption 4.1 admits many commonly used activation functions (E et al., 2020). One
example of a function satisfying this assumption is f (x, θ) = f (x, θ[1], θ[2], θ[3]) = θ[3]σ(θ[1]x+θ[2]),
wherefunction, see Remark H.2. θ[1] ∈ R[d][×][d], θ[2] ∈ R[d], θ[3] ∈ R and σ is a component-wise regularized ReLU activation

We now build the metric on the function space for our solutions. Note that the solution ρ(θ, t, s)
to (9) is expected to be a continuous function in (t, s), and a distribution of θ for each (t, s). For this
non-standard probability space, we first introduce the following metric.

**Definition 4.1** [2] _C([0, 1]; P_ [2]) is a collection of continuous paths of probability distribution ν(θ, t)
(limθ ∈t Rt0[k] W, t ∈2 (ν[0( _,, t 1])), ν where 1.(_ _, t0)) = 0 ν(·, where, t) ∈P W[2](R2 is the classical Wasserstein-2 distance. The space[k]) for every fixed t ∈_ [0, 1]; 2. For any t0 ∈ [0, 1],
_→_ _·_ _·_
_C([0, 1]; P_ [2]) is equipped with the following metric d1 (ν1, ν2) = supt W2(ν1(·, t), ν2(·, t)).

_Accordingly, C([0, ∞); C([0, 1]; P_ [2])) is a collection of continuous paths of probability distribution
_ν(θ, t, s) (with θ ∈_ R[k], t ∈ [0, 1], s ∈ [0, ∞)), where 1. ν(·, ·, s) ∈C([0, 1]; P [2]) for every fixed
_s_ [0, ). 2. For any s0 [0, ), lims _s0 d1 (ν(_ _,_ _, s), ν(_ _,_ _, s0)) = 0 (where d1 is defined above)._
_The metric in ∈_ _∞_ _C([0, ∞); C ∈([0, 1]; ∞ P_ [2])) is defined by→ _d·_ _·2 (ν1, ν2·_ ) = sup · _t,s W2(ν1(·, t, s), ν2(·, t, s))._

Since P [2] is complete in W2 distance, C([0, 1]; P [2]) and C([0, ∞); C([0, 1]; P [2])) are complete metric
spaces under d1 and d2 respectively also. To give a rigorous justification of the mean-field limit,
we use the particle representation of ρ(θ, t, s). Thus, at least we need to assume that we can find a
stochastic process that is drawn from the initial condition ρini(θ, t). We call such initial conditions
_admissible._

**Definition 4.2 We call a continuous path of probability distribution ν(θ, t) ∈C([0, 1]; P** [2]) admissible if it has a particle representation, namely there exists a continuous stochastic process
_θ(t) : [0, 1] →_ R[k] _and r > 0 such that for any t0 ∈_ [0, 1], we have

_θ(t0)_ _ν(θ, t0),_ lim _θ(t)_ _θ(t0)_ = 0, _θ(t0)_ _< r ._ (16)
_∼_ _t→t0_ [E] _|_ _−_ _|[2][
]_ _|_ _|_

2 
_C([0, T_ ]; A) is defined to be the set of functions f (a, t) such that for any t ∈ [0, T ], f (·, t) ∈A and f (·, t)
is continuous with respect to t under the metric defined on A. In Definition 4.1, we set T = 1 and A = P [2],
where the natural metric on A is W2.


-----

_Furthermore, ν(θ, t) is called limit-admissible if its M_ _-averaged trajectory is bounded and Lipschitz_
_with high probability. (See the rigorous definition in Definition E.1.)_

We note that without the dependence on t, probability distributions are “admissible", in the sense that
one can draw a sample from a given distribution. In Appendix A, we show that if the initial condition
_ρini(θ, t) is admissible, (9) has a unique solution ρ(θ, t, s) that is admissible for each s._

The global convergence result depends on Sard-type regularity, defined as follows.

**Definition 4.3 Given a metric space Θ, a differentiable function h : Θ →** R, and a subset Θ ⊂ Θ,
_we say h satisfies Sard-type regularity in_ Θ if the set of regular values[3] _of h|Θ_ _[is dense in its range,]_
_where h|Θ_ [:][ e]Θ → R is a confinement of h in Θ. e [e]

[e]

**Remark 4.1e** _This regularity assumption is not a common one; we have adopted it from Chizat &_

[e]

_Bach (2018). This property is essentially that most of the points in the range of h lie in an open set_
_and h is locally monotonic. The assumption is rather mild and can be satisfied by most commonly_
_seen regular functions, unless the function oscillates wildly._

5 MEAN-FIELD AND CONTINUOUS LIMIT

In this section, we focus on the justification of mean-field and continuous-limit result. This is to prove
that E(ρ( _,_ _, s)), E(Θ(s;_ )), and E(ΘL,M (s)) are asymptotically close to each other for every s. In

_·_ _·_ _·_
the next section, we prove convergence of E(ρ(·, ·, s)) as s →∞.

To show the asymptotic equivalence of the three quantities, we need to compare (3), (6), and (9), and
take the measurement in E according to (2), (5) and (8).

**Theorem 5.1 Suppose that Assumptions 4.1 and 4.2 are satisfied. Assume that ρini(θ, t) is limit-**
_admissible and suppθ(ρini(θ, t))_ _θ_ _θ[1]_ _R_ _with some R > 0 for all t_ [0, 1]. Let
_⊂{_ _||_ _| ≤_ _}_ _∈_
_{θm(0; t)}m[M]=1_ _[in][ (6)][ be][ i.i.d.][ drawn from][ ρ][ini][(][θ, t][)][. Let]_

-  ΘL,M (s) = _θl,m(s)_ _be the solution to (3) with initial condition θl,m(s = 0) =_
_{_ _}_
_θm_ 0; _L[l]_ _;_

-  θm(s; t) be the solution to
 (6) with the initial condition θm(0; t);

-  ρ(θ, t, s) be the solution to (9) with initial condition ρini(θ, t).

_Then for any positive ϵ, η, and S, there exists a constant C > 0 that depends on ρini(θ, t) and S such_
_that when_

_M > [C][(][ρ][ini][(][θ, t][)][, S][)]_ _,_ _L > [C][(][ρ][ini][(][θ, t][)][, S][)]_ _,_ _s < S,_

_ϵ[2]η_ _ϵ_

_we have:_


min{P (|E(ΘL,M (s)) − _E(Θ(s; ·))| ≤_ _ϵ/2), P (|E(Θ(s; ·)) −_ _E(ρ(·, ·, s))| ≤_ _ϵ/2)} ≥_ 1 − [1]2 _[η .]_

_It follows that_
P (|E(ΘL,M (s)) − _E(ρ(·, ·, s))| ≤_ _ϵ) ≥_ 1 − _η,_ _∀s < S ._
_Here E(ΘL,M_ (s)), E(Θ(s; )), and E(ρ( _,_ _, s)) are defined in (2), (5), and (8), respectively._
_·_ _·_ _·_

The proof of this result appears in Appendix E. This theorem suggests that for every fixed S > 0, the
gradient descent of ΘL,M is approximately the same as the gradient flow of ρ(θ, t), in the sense that
the two costs are close to each other with high probability, when L and M are sufficiently large. The
size of the ResNet depends negative-algebraically on ϵ (the desired accuracy) and η (the confidence
of success). The result translates the evolution (gradient descent) of ΘL,M to the evolution (gradient
flow) of ρ(θ, t), and thus matches the zero-loss property of the parameter configuration of a finite
sized ResNet to its limiting PDE, whose analysis can be performed with standard PDE tools.

3For a function h : Θ → R, a regular value is a real number α in the range of h such that h[−][1](α) is included
in an open set where h is differentiable and where dh does not vanish.
e


-----

The proof of Theorem 5.1 divides naturally into two components. We show that for all s < S,
_E(ρ(_ _,_ _, s))_ _E(Θ(s;_ )) and E(Θ(s; )) _E(ΘL,M_ (s)) with high probability. The former is

_·_ _·_ _≈_ _·_ _·_ _≈_
obtained from mean-field limit theory, justifying that the particle trajectory θm(t, s) follows ρ(θ, t, s)
for all t in pseudo-time s ∈ [0, S]. The latter makes use of continuity in t and traces the differences
between θm( _L[l]_ [)][ and][ θ][l,m][. These two components of the proof are summarized in Theorems E.1 and]

E.2, respectively. According to the formula of the Fréchet derivatives (10) and (11), the estimates in
these theorems naturally route through the boundedness of pρ, pΘ, pΘL,M, and similarly Zρ,Θ,ΘL,M .
It is technically demanding to derive these bounds, but they are not surprising. We dedicate a large
portion of the appendix to addressing the well-posedness of these systems. See Appendices A-D,
where we show these equations have unique solutions with proper initial conditions, along with the
required bounds. Naturally, these estimates depend on regularity of f, g, y, and µ.

To gain some intuition for the equivalence between (6) and (9), we test them on the same smooth
function h(θ). To test (9), we multiply both sides by h and perform integration by parts to obtain dds R[k][ h][ d][ρ][(][θ][) =][ −] R[k][ ∇][θ][h][∇][θ] _δE(δρρ(s))_ dρ. This is to say dds [E][(][h][) =][ E] _∇θh∇θ_ _δE(δρρ(s))_ .

1 _M_
Testing hR on (6) also gives the same formula.R Supposing that ρ = _M_ m=1 _[δ][θ]m[, we have]_

dds [E][(][h][) =] _M1_ _Mm=1_ ds _[θ][m][ =][ −]_ [P]m[M]=1 _δθm_ [.] The right hand side is alsoP

E _∇θh∇θ_ _δE(δρρP(s))_ if and only if[∇][θ][h][(][θ][m][)][ d] _M_ _[δE][(Θ(]δθm[s][;][·][))]_ = ∇[∇]θ[θ]δE[h]δρ[(]([θ]ρ[m]) [(][)][θ][ δE][m][, t][)][. This will be shown to hold]

true in Appendix F. 

6 CONVERGENCE TO GLOBAL MINIMIZER

After translating the study of ΘL,M to the study of ρ(θ, t), this section presents results on when and
how E(ρ) converges to zero loss by examining the conditions for the global convergence of (9). We
first identify the property of global minimum.

**Proposition 6.1a measure ν(θ) of Suppose that R[k]** _such that ρ ∈CR[k][ d]([0[ν][(], 1];[θ][) = 0] P_ [2])[ and] has E(ρ) > 0. Then for any t0 ∈ [0, 1], there exists
R


_δE_
R[k] _δρ_


(θ, t0) dν(θ) < 0 . (17)


See Appendix H.1 for the proof of this result. At the stationary point of the cost function, _[δE]δρ_ [(][θ, t][0][) =]

0, then there is no ν satisfying (17), so E(ρ) is necessarily trivial. Our task now becomes to check
under what conditions on f we can have _[δE]δρ_ [becoming][ 0][ as][ s][ →∞][. We give two possibilities, both]

requiring a separation initialization and certain homogeneities. In the first case, we require f to be
2-homogeneous, and the result is collected in Section 6.1. In the second case, we require f to be
partially 1-homogeneous; see Section 6.2.

6.1 THE 2-HOMOGENEOUS CASE

The results in this section are obtained under the following assumption on f . (Functions can be
designed to satisfy this assumption easily; see Remark H.1.)

**Assumption 6.1 The function f** (x, θ) : R[d] _× R[k]_ _is 2-homogeneous, meaning that f_ (x, λθ) =
_λ[2]f_ (x, θ) for all (x, θ, λ) ∈ R[d] _× R[k]_ _× R._

**Theorem 6.1 Let Assumption 6.1 and conditions of Theorem 5.1 hold true with k1 = k. Let ρ** (θ, t)
_∞_
_be the long-time limit of (9), that is, ρ(θ, t, s) converges to ρ_ (θ, t) in ([0, 1]; ) as s _. Then_
_∞_ _C_ _P_ [2] _→∞_
_E(ρ_ ) = 0 if the following hold:
_∞_

-  Separation initialization: There exists t0 [0, 1] such that ρini(θ, t0) separates[4] _the spheres_
_raS[k][−][1]_ _and rbS[k][−][1], for some 0 < ra < r ∈b._

4We say that a set C separates the sets A and B if any continuous path in with endpoints in A and B
intersects C.


-----

-  Sard-type regularity: _[δE]δρ_

_ρ∞_ [(][θ, t][0][)][ satisfies the Sard-type regularity condition in][ S][k][−][1][.]

A proof appears in Appendix H.2. A corollary of this theorem, when combined with Theorem 5.1
with k1 = k, gives Theorem 3.1, the main result of our paper.

6.2 THE PARTIALLY 1-HOMOGENEOUS CASE

The following assumption is used in this section. (Functions that satisfy this assumption include
regularized ReLU; see Remark H.2.)

**Assumption 6.2 Let θ = (θ[1], θ[2]) with θ[1] ∈** R and θ[2] ∈ R[k][−][1]:

_1. (partially 1-homogeneous in θ) f can be written as f_ (x, θ) = _f_ (x, θ[1], θ[2]) =
_θ[1]f_ (x, θ[2]),

_2. (locally bounded and smooth) For any r > 0,_ _f_ (x, θ[2]) is bounded and Lipschitz with

[b]

_Lipschitz continuous differential for (x, θ[2]) ∈Br([⃗]0) × R[k][−][1]._

[b]

When Assumption 6.2 holds true, Assumption 4.1 part 3 can be satisfied with k1 = 1. Then, we
introduce the main result in this section.:

**Theorem 6.2 Let Assumption 6.2 and conditions of Theorem 5.1 hold true with k1 = 1. Let ρ** (θ, t)
_∞_
_be the limit of (9) as s_ _. Then E(ρ_ ) = 0 if the following hold:
_→∞_ _∞_

-  Separation initialization: There exists t0 [0, 1] such that ρini(θ[1], θ[2], t0) separates the
_∈_
_spheres {−r0} × R[k][−][1]_ _and {r0} × R[k][−][1]_ _for some r0 > 0, where θ[1], θ[2] are defined by_
_Assumption 4.1 item 3 with k1 = 1._

-  Sard-type regularity: _δEδρ_

_ρ∞_ [((1][, θ][[2]][)][, t][0][) :][ R][k][−][1][ →] [R][ satisfies the Sard-type regularity]

_condition._

-  For any ˜ρ ([0, 1], ), define Hr,ρ˜ _θ[2]_ = _δEδρ(ρ)_ 1, rθ[e][2], t0 _where θ[2] = rθ[˜][2]_
_∈C_ _P_ [2] _ρ˜_

_with r =_ _θ2_ _and_ _θ[˜][2]_ S[k][−][2]. Suppose that  Hr,ρ˜ _[converges in]_ _[ C][1][(][S][k][−][2][)][ as][ r][ →∞]_ _[to a]_
_function H |_ _∞|,ρ˜[. Furthermore, assume that] ∈_ e[ H]∞,ρ∞ _[satisfies the Sard-type regularity condi-]_

_tion in S[k][−][2]_ _and that the intersection of regular values of H∞,ρ∞_ _and_ _[δE]δρ[(][ρ][)]_ _ρ∞_ [(1][, θ][[2]][, t][0][)]

_is also dense in the intersection of their range._


The proof is found in Appendix H.3. This result, combined with Theorem 5.1 in the case of k1 = 1,
gives the main result in Theorem 3.1. The assumptions in Theorem 6.2 are rather technical and
seemingly tedious. However, we note that only the first two assumptions — 1-homogeneous and
the separation assumption — are crucial. The third and fourth assumptions concern the regularity
of the Fréchet derivative of ρ; they are rather mild and serve to exclude wildly oscillating functions;
see Remark 4.1. Most commonly seen functions are regular enough that these two assumptions are
satisfied.

7 ETHICS STATEMENT

This work does not present any foreseeable societal consequence.

8 REPRODUCIBILITY STATEMENT

The notations, assumptions and definitions are clarified in Section 4. And all proofs appear in the
Appendix.


-----

REFERENCES

Z. Allen-Zhu and Y. Li. What can ResNet learn efficiently, going beyond kernels? In Advances in
_Neural Information Processing Systems, volume 32, 2019._

Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over-parameterization. In
_Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings_
_of Machine Learning Research, pp. 242–252. PMLR, 09–15 Jun 2019._

L. Ambrosio, N. Gigli, and G. Savaré. Gradient flows: in metric spaces and in the space of probability
_measures. Springer Science and Business Media. Birkhäuser Basel, 2008._

D. Araújo, R. Oliveira, and D. Yukimura. A mean-field limit for certain deep neural networks.
_arXiv/1906.00193, 2019._

S. Arora, S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang. On exact computation with an
infinitely wide neural net. In NeurIPS, 2019.

J. Ba, M. Erdogdu, T. Suzuki, D. Wu, and T. Zhang. Generalization of two-layer neural networks:
An asymptotic viewpoint. In ICLR, 2020.

N. Chatterji, P. Long, and P. Bartlett. When does gradient descent with logistic loss interpolate using
deep networks with smoothed relu activations? arXiv/2102.04998, 2021.

Z. Chen, Y. Cao, Q. Gu, and T. Zhang. A generalized neural tangent kernel analysis for twolayer neural networks. In Advances in Neural Information Processing Systems, volume 33, pp.
13363–13373, 2020.

L. Chizat and F. Bach. On the global convergence of gradient descent for over-parameterized models
using optimal transport. In Advances in Neural Information Processing Systems, volume 31, 2018.

Z. Ding, S. Chen, Q. Li, and S. Wright. Overparameterization of deep resnet: zero loss and mean-field
analysis. arXiv/2105.14417, 2021.

S. Du and J. Lee. On the power of over-parametrization in neural networks with quadratic activation. In
_Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings_
_of Machine Learning Research, pp. 1329–1338. PMLR, 10–15 Jul 2018._

S. Du, C. Jin, J. Lee, M. Jordan, A. Singh, and B. Póczos. Gradient descent can take exponential time
to escape saddle points. In Advances in Neural Information Processing Systems, volume 30, 2017.

S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural
networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97
of Proceedings of Machine Learning Research, pp. 1675–1685, 09–15 Jun 2019a.

S. Du, X. Zhai, B. Póczos, and A. Singh. Gradient descent provably optimizes over-parameterized
neural networks. In International Conference on Learning Representations, 2019b.

W. E, C. Ma, S. Wojtowytsch, and L. Wu. Towards a mathematical understanding of neural networkbased machine learning: what we know and what we don’t. arXiv/2009.10713, 2020.

C. Fang, Y. Gu, W. Zhang, and T. Zhang. Convex formulation of overparameterized deep neural
networks. arXiv/1911/07626, 2019.

S. Frei, Y. Cao, and Q. Gu. Algorithm-dependent generalization bounds for overparameterized deep
residual networks. In NeurIPS, 2019.

R. Ge, F. Huang, C. Jin, and Y. Yuan. Escaping from saddle points — online stochastic gradient for
tensor decomposition. In Proceedings of The 28th Conference on Learning Theory, volume 40 of
_Proceedings of Machine Learning Research, pp. 797–842. PMLR, 03–06 Jul 2015._

R. Ge, J. Lee, and T. Ma. Learning one-hidden-layer neural networks with landscape design. In
_International Conference on Learning Representations, 2018._


-----

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural
networks. In Advances in Neural Information Processing Systems, volume 31, 2018.

C. Jin, R. Ge, P. Netrapalli, S. Kakade, and M. Jordan. How to escape saddle points efficiently. In
_Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings_
_of Machine Learning Research, pp. 1724–1732. PMLR, 06–11 Aug 2017._

K. Kawaguchi. Deep learning without poor local minima. In Advances in Neural Information
_Processing Systems, volume 29, 2016._

Y. Li and Y. Liang. Learning overparameterized neural networks via stochastic gradient descent
on structured data. In Proceedings of the 32nd International Conference on Neural Information
_Processing Systems, NIPS’18, pp. 8168–8177, 2018._

C. Liu, L. Zhu, and M. Belkin. On the linearity of large non-linear models: when and why the
tangent kernel is constant. In Advances in Neural Information Processing Systems, volume 33, pp.
15954–15964, 2020.

Y. Lu, C. Ma, Y. Lu, J. Lu, and L. Ying. A mean field analysis of deep ResNet and beyond:
Towards provably optimization via overparameterization from depth. In Proceedings of the 37th
_International Conference on Machine Learning, volume 119, pp. 6426–6436, 13–18 Jul 2020._

S. Mei, A. Montanari, and P. M. Nguyen. A mean field view of the landscape of two-layer neural
networks. Proceedings of the National Academy of Sciences, 115(33):E7665–E7671, 2018.

P. M. Nguyen. Mean field limit of the learning dynamics of multilayer neural networks.
_arXiv/1902.02880, 2019._

P. M. Nguyen and H. Pham. A rigorous framework for the mean field limit of multilayer neural
networks. arxiv/2001.11443, 2021.

Q. Nguyen and M. Hein. The loss surface of deep and wide neural networks. In Proceedings of
_the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine_
_Learning Research, pp. 2603–2612. PMLR, 06–11 Aug 2017._

Q. Nguyen and M. Hein. Optimization landscape and expressivity of deep cnns. In ICML, 2018.

J. Sirignano and K. Spiliopoulos. Mean field analysis of neural networks: A law of large numbers.
_SIAM Journal on Applied Mathematics, 80(2):725–752, 2020._

J. Sirignano and K. Spiliopoulos. Mean field analysis of deep neural networks. Mathematics of
_Operations Research, 2021. doi: 10.1287/moor.2020.1118._

M. Soltanolkotabi, A. Javanmard, and J. Lee. Theoretical insights into the optimization landscape of
over-parameterized shallow neural networks. IEEE Trans. Inf. Theor., 65(2):742–769, February
2019. ISSN 0018-9448.

C. Wei, J. Lee, Q. Liu, and T. Ma. Regularization matters: Generalization and optimization of neural
nets v.s. their induced kernel. In NeurIPS, 2019.

S. Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the
mean field regime. arXiv/2005/13530, 2020.

C. Yun, S. Sra, and A. Jadbabaie. Global optimality conditions for deep neural networks. In
_International Conference on Learning Representations, 2018._

H. Zhang, D. Yu, M. Yi, W. Chen, and T. Liu. Convergence theory of learning over-parameterized
resnet: A full characterization. arXiv/1903.07120, 2019.

D. Zou, Y. Cao, D. Zhou, and Q. Gu. Gradient descent optimizes over-parameterized deep relu
networks. Machine Learning, 109:467–492, 2019.


-----

APPENDIX: INTRODUCTION

The appendix contains proofs and supporting analysis for the theorems in the main text. We start by
proving well-posedness of the ResNet ODE and the gradient flow. We then justify the continuous and
mean-field limit. Finally, we prove the global convergence of the gradient-flow PDE. The sections
are organized as follows.

Appendix A: Well-posedness. We summarize the well-posedness of the continuous limit ODE (4),
the mean-field limit ODE (7), and the associated gradient flows (3), (6), and (9). The results are
collected in Theorem A.1.

Appendix B: This section collects the detailed proof for the well-posedness of the ResNet ODE in
its continuous limit (4), and the mean-field limit (7), and finalizes the proof of Theorem A.2.

Appendix C: This section prepares some a-priori estimates for showing the well-posedness of the
gradient flow equations.

Appendix D: This section contains a detailed proof for the well-posedness of gradient flow equations (3) and (9) and finalizes the proofs of Theorem A.3 and A.4.

Appendix E-G: Proof of Theorem 5.1, the continuous and mean-field limit. Section E lays out the
structure of the proof, Section F shows the continuous limit, and Section G shows the mean-field
limit.

Appendix H: Proof of Theorem 6.1 and 6.2: Global convergence of the gradient flow.

The analytical core of the paper lies in Appendices H and E, which describe properties of the gradient
flow PDEs and explain why the gradient descent method for ResNet can be explained by these
equations. The technical results of Appendix B-D can be skipped by readers who are interested to
proofs of the main results.

Throughout, we denote by C(·) a generic constant whose value depends only on its arguments. The
precise value of this constant may change each time it is invoked. Throughout Appendices A-G, we
assume that Assumption 4.1 holds for some 0 < k1 _k._
_≤_

A WELL-POSEDNESS RESULT

In this section, we show the well-posedness of ODEs (4), (7) and gradient flows (3), (6), and (9).

**Theorem A.1 The following claims hold.**

_– Well-posedness of ODE:_

-  If {θm(t)}m[M]=1 _[is continuous, then][ (4)][ has a unique][ C][1][solution.]_

-  If ρ ∈C([0, 1]; P [2]), then (7) has a unique C[1] _solution._

_– Well-posedness of gradient flow:_

-  (3) has a unique solution.

-  If {θm(0; t)}m[M]=1 _[is continuous, then][ (6)][ has a unique solution][ {][θ][m][(][s][;][ t][)][}]m[M]=1_ _[that is continuous]_
_in (s, t)._

-  If ρini(θ, t) is admissible and suppθ(ρini(θ, t)) _θ_ _θ[1]_ _R_ _with some R > 0 for all t_
_⊂{_ _||_ _| ≤_ _}_ _∈_

[0, 1], then (9) has a unique solution ρ(θ, t, s) in C([0, ∞); C([0, 1]; P [2])) with initial condition
_ρini(θ, t). Furthermore, for each s, ρ(θ, t, s) is admissible._

Note that the well-posedness of (4) and (6) are direct corollaries of that of (7), (3), and (9) (the
corresponding continuous versions), according to Remark A.1, A.2. Thus, we merely prove wellposedness on the continuous level, in Theorems A.2, A.3, and A.4. Specifically,

-  Theorem A.2 (Appendix A.1) shows the well-posedness of the dynamical system for z;

-  Theorems A.3 and A.4 (Appendix A.2) justify the well-posedness of the gradient flow of
the parameter configuration.


-----

A.1 WELL-POSEDNESS OF THE OID (7)

As L and M approach ∞, z satisfies the ordinary-integral equation (7). We justify that this differential
equation is well-posed, in the sense that the solution is unique and stable.

**Theorem A.2 Suppose that Assumption 4.1 holds with and that x is in the support of µ, then (7) has**
_a unique_ _solution. Specifically, for ρ1, ρ2_ ([0, 1]; ), we have
_C[1]_ _∈C_ _P_ [2]
_Zρ1_ (t; x) _C(_ 1), (18)
_|_ _| ≤_ _L_
_and for all t ∈_ [0, 1]:
_Zρ1_ (t; x) _Zρ2_ (t; x) _C(_ 1, 2)d1(ρ1, ρ2), (19)
_|_ _−_ _| ≤_ _L_ _L_
_where Li are the second moments of 1 ρi, that is,_

_Li =_ 0 R[k][ |][θ][|][2][dρ][i][(][θ, t][) d][t,] _i = 1, 2._ (20)
Z Z

We leave the proof to Appendix B. Besides the well-posedness result, the theorem also suggests
that a small perturbation to ρ is reflected linearly in Zρ, the solution to (7). This means that a small
perturbation in the parameterization of the ResNet leads to only a small perturbation to the ResNet
output.

**Remark A.1 Although we do not directly show the well-posedness of (4), it follow immediately from**
_Theorem A.2. One way to make this connection is to reformulate the discrete probability distribution_
_as_


_ρ[dis](θ, t) = [1]_


_δθm(t)(θ),_
_m=1_

X


_where Θ(t) = {θm(t)}m[M]=1_ _[is the list of trajectories. Since][ θ][m][(][t][)][ is continuous in][ t][, we have]_
_ρ[dis](θ, t) ∈C([0, 1]; P_ [2]). Since

_M_

1

_f_ (z(t; x), θm(t)) =

_M_ _m=1_ ZR[k][ f] [(][z][(][t][;][ x][)][, θ][) d][ρ][dis][(][θ, t][)][,]

X

_using Theorem A.2, (4) has a unique C[1]_ _solution when Θ(t) is continuous._


A.2 WELL-POSEDNESS OF THE GRADIENT FLOW

The gradient flow of the parameterization is also well-posed, both in the discrete setting and the
continuous mean-field limit. In the discrete setting, we have the following result.

**Theorem A.3 (3) has a unique solution.**

Further, (9) characterizes the dynamics of the continuous mean-field limit of the parameter configuration, and is also well-posed.

**Theorem A.4 If ρini(θ, t) is admissible and suppθ(ρini(θ, t))** _θ_ _θ[1]_ _R_ _with some R > 0 for_
_⊂{_ _||_ _| ≤_ _}_
_all t ∈_ [0, 1]. Then (9) has a unique solution ρ(θ, t, s) in C([0, ∞); C([0, 1]; P [2])) that is admissible
_for each s. Further, we have_
dE(ρ( _,_ _, s))_

_·_ _·_ 0 . (21)

ds _≤_

The proofs of these two theorems can be found in Appendix D.

**Remark A.2 Using the same argument as in Remark A.1, calling**

_M_

_ρ[dis](θ, t, s) = M[1]_ _δθm(s;t)(θ),_ (22)

_m=1_

X

_the well-posedness of (6) follows immediately from Theorem A.4. Furthermore, according to the_
_definition (5) and (8), we have_
_E_ _ρ[dis](·, ·, s)_ = E (Θ(s; ·)) .

_As a consequence, if Θ(s; t) satisfies (6), then ρ[dis]_ _satisfies (9), and vice versa. The well-posedness_

 


_result in Theorem A.4 for (9) then can be extended to justify well-posedness of (6)._


-----

B PROOF OF THEOREM A.2

This section contains the proof of Theorem A.2. We rewrite (7) as follows:

dZρ(t; x)

= F (Zρ, t), _t_ [0, 1] with _z(0; x) = x,_ (23)
dt _∀_ _∈_

where for a given ρ ∈C([0, 1]; P [2]) we use the notation:


_F_ (z, t) =


R[k][ f] [(][z, θ][) d][ρ][(][θ, t][)][ .]


The proof of Theorem A.2 relies on the classical Lipschitz condition for the well-posedness of an
ODE.

**Proof [Proof of Theorem A.2] Since ρ ∈C([0, 1]; P** [2]), we have a constant C such that


sup
0≤t≤1 ZR[k][ |][θ][|][2][ d][ρ][(][θ, t][)][ < C <][ ∞] _[.]_

For any t ∈ [0, 1], using (12) from Assumption 4.1 equation, we have


_|F_ (z, t)| ≤ (24)

R[k][ |][f] [(][z][1][, θ][)][|][ d][ρ][(][θ, t][)][ ≤] _[C][1][(][|][z][|][ + 1)]_ R[k] [(][|][θ][|][2][ + 1) d][ρ][(][θ, t][)][ .]

Z Z

To show the boundedness result (18), we multiply (23) by Zρ1 (t; x) and use (24) to obtain

d|Zρ1d(tt; x)|[2] _≤2C1_ _|Zρ1_ _|[2]_ + |Zρ1 _|_

R[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)]

 
 [Z]

4C1 _Zρ1_ (t; x) + 1 _._
_≤_ _|_ _|[2]_

R[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)]

Z

 



Using Grönwall’s inequality, we have

_Zρ1_ (t; x) exp 2C1
_|_ _| ≤_



1

exp 2C1 ( _x_ + 1)
_| ≤_  Z0 ZR[k][ |][θ][|][2][ d][ρ][1][ d][t][ + 1] _|_ _|_

_≤_ exp (2C1 (L1 + 1)) (|x| + 1),


1
where L1 = 0 R[k][ |][θ][|][2][ d][ρ][1][ d][t][. Since][ x][ ∈] [supp][ µ][ (so that][ |][x][|][ < R][), we have][ (18)][. This gives us an]

a-priori estimate of (23).

R R

Next, using (13) from Assumption 4.1, we have

_f_ (z1, θ) _f_ (z2, θ) _C2(_ _z1_ + _z2_ ) _θ_ + 1 _z1_ _z2_ _._ (25)
_|_ _−_ _| ≤_ _|_ _|_ _|_ _|_ _|_ _|[2]_ _|_ _−_ _|_

Then, using boundedness of the second moment of θ, we have 



_F_ (z1, t) _F_ (z2, t)
_|_ _−_ _| ≤_


R[k][ (][f] [(][z][1][, θ][)][ −] _[f]_ [(][z][2][, θ][)) d][ρ][(][θ, t][)]


_C2(_ _z1_ + _z2_ )
_≤_ _|_ _|_ _|_ _|_

R[k] [(][|][θ][|][2][ + 1) d][ρ][(][θ, t][)][|][z][1][ −] _[z][2][|]_

Z

_< C2(_ _z1_ + _z2_ )(C + 1) _z1_ _z2_ _,_
_|_ _|_ _|_ _|_ _|_ _−_ _|_

which implies that F (z, t) is locally Lipschitz in z for all t ∈ [0, 1]. Combining this with the a-priori
estimate in (18), classical ODE theory implies that (23) has a unique C[1] solution.

To prove the stability resultdenote (19), we define Zρi as in (23) parameterized by ρi ∈C([0, 1]; P [2]), and
∆(t; x) = Zρ1 (t; x) _Zρ2_ (t; x) .
_−_


-----

Then by subtracting the two equations, we obtain


d|∆(t; x)|[2]

dt

= 2 ∆(t; x),


R[k][ f] [(][Z][ρ][1] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)][ −]


R[k][ f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][2][(][θ, t][)]


= 2 ∆(t; x),

-  R[k][ f] [(][Z][ρ][1] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)][ −] R[k][ f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)]+

Z Z

(I)
| {z }

+ 2 ∆(t; x),

-  R[k][ f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)][ −] R[k][ f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][2][(][θ, t][)]

Z Z

(II)

We now bound (I) and (II)|. For (I), we have using (13) and (25) that{z }


= 2

- 

+ 2


∆(t; x),


(26)


_|(I)| ≤2C2 (|Zρ1_ (t; x)| + |Zρ2 (t; x)|) |∆(t; x)|

R[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)]

Z (27)

_C(_ 1, 2) ∆(t; x)
_≤_ _L_ _L_ _|_ _|_

R[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)][,]

Z

where the second inequality comes from (18). For (II), we denote the particle representation
_θ1(t) ∼_ _ρ1(θ, t) and θ2(t) ∼_ _ρ2(θ, t) such that_ E _|θ1 −_ _θ2|[2][

][1][/][2]_ = W2(ρ1(·, t), ρ2(·, t)). Then

(II) E ( _f_ (Zρ2 (t; x), θ1) _f_ (Zρ2 (t; x), θ2) )
_|_ _| ≤_ _|_ _−_ _|_
_≤_ _C2(Zρ2_ (t; x))E ((|θ1| + |θ2| + 1)|θ1 − _θ2|)_
_≤_ _C(L2)E ((|θ1| + |θ2| + 1)|θ1 −_ _θ2|)_

1/2
_≤_ _C(L2)_ E _|θ1|[2]_ + |θ2|[2] + 1 E _|θ1 −_ _θ2|[2][

][1][/][2]_ (28)

_C(_ 2) E  _θ1_ + _θ2_ + 1

1/2  W2(ρ1( _, t), ρ2(_ _, t))_
_≤_ _L_ _|_ _|[2]_ _|_ _|[2]_ _·_ _·_

1/2

  



_C(_ 2) _d1(ρ1, ρ2),_
_≤_ _L_
ZR[k][ |][θ][|][2][dρ][1][(][θ, t][) +] ZR[k][ |][θ][|][2][dρ][2][(][θ, t][) + 1]

where we use mean-value theorem and (13) in the first inequality to obtain
_|f_ (Zρ2 (t; x), θ1) − _f_ (Zρ2 (t; x), θ2)| ≤|∂θf (Zρ2 (t; x), θ1 + λθ2)||θ1 − _θ2|_
_C2(Zρ2_ (t; x))( _θ1_ + _θ2_ + 1) _θ1_ _θ2_ _,_
_≤_ _|_ _|_ _|_ _|_ _|_ _−_ _|_

for some λ ∈ [0, 1]. We use (18) in the second inequality of (28).

Plugging (27)-(28) into (26) and using Hölder’s inequality, we obtain that

d ∆(t; x)
_|_ _|[2]_ _C(_ 1, 2) ∆(t; x)

dt _≤_ _L_ _L_ _|_ _|[2]_

R[k][ |][θ][|][2][ d][ρ][1][(][θ, t][)]

Z

1/2

+ C( 2) ∆(t; x) _d1(ρ1, ρ2)_
_L_ _|_ _|_
ZR[k][ |][θ][|][2][dρ][1][(][θ, t][) +] ZR[k][ |][θ][|][2][dρ][2][(][θ, t][) + 1]

_≤C(L1, L2)_ _|∆(t; x)|[2]_ + d[2]1[(][ρ][1][, ρ][2][)] _,_

R[k][ |][θ][|][2][dρ][1][(][θ, t][) +] R[k][ |][θ][|][2][dρ][2][(][θ, t][) + 1]

where we used Young’s inequality in the last line. SinceZ Z _|∆(0; x)| = 0, we complete the proof of _ (19)

using Grönwall’s inequality.

C A-PRIORI ESTIMATION OF THE COST FUNCTION

Some a-priori estimates are necessary in the proof for the main theorems. We first consider the case
when f satisfies only Assumption 4.1 with 0 < k1 _k. (Better a-priori estimates can be obtained_
when f also satisfies the homogeneity properties of Sections 6.1 or 6.2.) ≤


-----

C.1 A-PRIORI ESTIMATE FOR GENERAL f

According to (10), the Fréchet derivative can be computed, similarly to (Lu et al., 2020), as follows:

_δE(ρ)_

(θ, t) = Ex _µ_ _p[⊤]ρ_ [(][t][;][ x][)][f] [(][Z][ρ][(][t][;][ x][)][, θ][)] _,_ (29)
_δρ_ _∼_
 


where pρ(t; x) is the solution to the following ODE:
 _∂p∂t[⊤]ρ_ = −p[⊤]ρ R[k][ ∂][z][f] [(][Z][ρ][(][t][;][ x][)][, θ][) d][ρ][(][θ, t][)][,] (30)

Z

pρ(1; x) = (g(Zρ(1; x)) _y(x))_ _g(Zρ(1; x)) ._

_−_ _∇_




We now show that pρ is Lipschitz continuous with respect to ρ.

**Lemma C.1 Suppose that x is in the support of µ. Suppose that ρ1, ρ2** ([0, 1]; ) and pρ1 _, pρ2_
_are the corresponding solutions of (30). Denote L1 and L2 as in (20) and ∈C_ _P_ [2]

= min _r_ supp(ρ1) supp(ρ2) _θ_ _θ[1]_ _< r_ _._
_R_ _r_ _∪_ _⊂_

_Then the following two bounds are satisfied:_ 


_pρ1_ (t; x) _C(_ 1), (31)
_|_ _| ≤_ _L_


_and_


_pρ1_ (t; x) _pρ2_ (t; x) _C(_ _,_ ) d1(ρ1, ρ2), (32)
_|_ _−_ _| ≤_ _R_ _L_

_where_ = max 1, 2 _._
_L_ _{L_ _L_ _}_


**Proof From (13) in Assumption 4.1,**

ZR[k][ ∂][z][f] [(][Z][ρ][1] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)] _[≤][C][(][Z][ρ][1]_ [(][t][;][ x][))] ZR[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)] (33)

_C(_ 1)
_≤_ _L_

R[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)][,]

Z


where we use (18) in the second inequality. It follows from the initial condition of (30) that

_pρ1_ (1; x) _C(_ _Zρ1_ (1, x) + 1) _C(_ 1),
_≤_ _|_ _|_ _≤_ _L_

where we use Assumption 4.1 in the first inequality and (18) in the second inequality. Noting that (30)
is a linear equation, (31) follows naturally when we combine (33) with the inequality above.

To prove (32), we define
∆(t; x) = pρ1 (t; x) _pρ2_ (t; x) .
_−_

For t = 1, with x ∈ supp µ, we have

∆(1; x) = _pρ1_ (1; x) _pρ2_ (1; x)
_|_ _|_ _|_ _−_ _|_
= (g(Zρ1 (1; x)) _y(x))_ _g(Zρ1_ (1; x)) (g(Zρ2 (1; x)) _y(x))_ _g(Zρ2_ (1; x))
_|_ _−_ _∇_ _−_ _−_ _∇_ _|_ (34)
_C(_ ) _Zρ1_ (1; x) _Zρ2_ (1; x)
_≤_ _L_ _|_ _−_ _|_
_C(_ )d1(ρ1, ρ2),
_≤_ _L_

where we use Assumption 4.1, (18), and |x| < Rµ in the first inequality and (19) in the second
inequality. The following ODE is satisfied by ∆:


_∂∆[⊤](t; x)_

= ∆[⊤](t; x)
_∂t_ _−_


_ρ2_ [(][t][;][ x][)][D][ρ]1[,ρ]2 [(][t][;][ x][)][,] (35)
R[k][ ∂][z][f] [(][Z][ρ][1] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][) +][ p][⊤]


where


(36)
R[k][ ∂][z][f] [(][Z][ρ][1] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)][ .]


R[k][ ∂][z][f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][2][(][θ, t][)][ −]


_Dρ1,ρ2_ (t; x) =


-----

To show the boundedness ofinto two terms, we obtain _Dρ1,ρ2_ (t; x), we follow the same strategy as that for (26). By splitting

_Dρ1,ρ2_ (t; x)
_|_ _| ≤_

R[k][ ∂][z][f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][2][(][θ, t][)][ −] R[k][ ∂][z][f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)]

Z Z

(I)

(37)

| {z }

+ _._

R[k][ ∂][z][f] [(][Z][ρ][2] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)][ −] R[k][ ∂][z][f] [(][Z][ρ][1] [(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)]

Z Z

(II)

The bound of (II) relies on part 3 of Assumption 4.1: Because the supports of| {z _ρ1, ρ2 are contained in}_
_θ_ _θ[1]_ _<_ and Z is bounded by (18), we have
_R_
 (II) _C(_ _,_ ) _Zρ1_ (t; x) _Zρ2_ (t; x) _C(_ _,_ )d1(ρ1, ρ2), (38)

_|_ _| ≤_ _R_ _L_ _|_ _−_ _| ≤_ _R_ _L_

where we use (19) in the second inequality.

To bound (I), we use the particle representation θ1 _ρ1(θ, t) and θ2_ _ρ2(θ, t) such that_
_∼_ _∼_
E _|θ1 −_ _θ2|[2][

][1][/][2]_ = W2(ρ1(·, t), ρ2(·, t)). We then have
  (I) E ( _∂zf_ (Zρ2 (t; x), θ1) _∂zf_ (Zρ2 (t; x), θ2) )

_≤_ _|_ _−_ _|_
_≤_ _C(R, L)E (|θ1 −_ _θ2|)_ (39)

_≤_ _C(R, L)_ E _|θ1 −_ _θ2|[2][

][1][/][2]_

_C(_ _,_ )d1(ρ1, ρ2),

 

_≤_ _R_ _L_

where we use the mean-value theorem, part 3 of Assumption 4.1 with _θ1,[1]_ _<_ and _θ2,[1]_ _<_,
_|_ _|_ _R_ _|_ _|_ _R_
(18) in the second inequality. Substituting (38) and (39) into (37), we obtain

_Dρ1,ρ2_ (t; x) _C(_ _,_ )d1(ρ1, ρ2) .
_|_ _| ≤_ _R_ _L_

By substituting this bound into (35) and using (33), we have


d ∆(t; x)
_|_ _|[2]_ _C(_ _,_ ) ∆(t; x) + d[2]1[(][ρ][1][, ρ][2][)] _._ (40)

dt _≤_ _R_ _L_ _|_ _|[2]_

The result (32) follows from the initial condition (34) and Grönwall’s inequality. 


The second lemma concerns the continuity of ∇θ _δEδρ(ρ)_ [.]


**Lemma C.2 Suppose that ρ, ρ1, ρ2 ∈C([0, 1]; P** [2]). Define

1

= max = max
_L_ 1≤i≤3 Z0 ZR[k][ |][θ][|][2][ d][ρ][i][(][θ, t][) d][t,] _L[sup]_ 1≤i≤3 _t∈[sup][0,1]_ ZR[k][ |][θ][|][2][ d][ρ][i][(][θ, t][)][,]

_and_
= min _r_ supp(ρ) supp(ρ1) supp(ρ2) _θ_ _θ[1]_ _< r_ _._
_R_ _r_ _∪_ _∪_ _⊂{_ _||_ _|_ _}_

_Then for any (θ, t), (θ1, t1), (θ2, t2) ∈_ R[k] _× [0, 1] and s > 0, the following properties hold._

-  Boundedness:
_δE(ρ)_ _δE(ρ)_

(θ, t) (θ, t) (41)
_δρ_ _δρ_

_[∇][θ]_ _[≤]_ _[C][(][L][)(][|][θ][|][ + 1)][,]_ _[∂][θ][[1]]_ _[≤]_ _[C][(][L][)(][|][θ][[1]][|][ + 1)]_

-  Lipschitz continuity in θ and t: There exists Q1 : R[2] _→_ R[+] _that depends increasingly on_
_both arguments such that_

_δE(ρ)_ _δE(ρ)_

(θ1, t1) _θ_ (θ2, t2)
_δρ_ _−∇_ _δρ_

(42)

_≤Q[∇]1_ _[θ]_ _L, maxi=1,2[(][|][θ][i,][[1]][|][)]_ _|θ1 −_ _θ2| + Q1_ _L[sup], maxi=1,2[(][|][θ][i,][[1]][|][)]_ (|θ2| + 1)|t1 − _t2|,_

   


-----

-  Lipschitz continuity in ρ. There exists Q : R[2] _→_ R[+] _that increasingly depends on both_
_arguments such that_

_δE(ρ1)_ _δE(ρ2)_

(θ, t) _θ_ (θ, t) (43)
_δρ_ _−∇_ _δρ_ _[≤]_ _[Q][(][L][,][ |][θ][[1]][|][,][ R][)(1 +][ |][θ][|][)][d][1][(][ρ][1][, ρ][2][)][,]_

_where d1 is defined in Definition 4.1._

_[∇][θ]_


**Proof To prove the first bound of (41), we restate (29) as follows**

_δE(ρ)_
_θ_ (θ, t) = Ex _µ_ _p[⊤]ρ_ [(][t][;][ x][)][∂][θ][f] [(][Z][ρ][(][t][;][ x][)][, θ][)]
_∇_ _δρ_ _∼_




from which it follows that

_δE(ρ)_

(θ, t)
_δρ_

_[≤]_ [E][x][∼][µ][ (][|][∂][θ][f] [(][Z][ρ][(][t][;][ x][)][, θ][)][| |][p][ρ][(][t][;][ x][)][|][)][ ≤] _[C][(][L][)(][|][θ][|][ + 1)][,]_

where we use (13), (18), and (31) in the second inequality. To prove the second bound in (41), we

_[∇][θ]_

use the bound of _∂θ[1]_ _f_ (x, θ) according to Assumption 4.1 part 3 to obtain

_δE(ρ)_

_δρ_ (θ, t) _[≤]_ [E][x][∼][µ] _∂θ[1]_ _f_ (Zρ(t; x), θ) _|pρ(t; x)|_ _≤_ _C(L)(|θ[1]| + 1),_
 


To prove (42)[∂][θ][[1]], we assume t1 > t2 without loss of generality, and use the triangle inequality to obtain

_δE(ρ)_ _δE(ρ)_

(θ1, t1) _θ_ (θ2, t2)
_δρ_ _−∇_ _δρ_

Ex _µ_ _p[⊤]ρ_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ][(][t][1][;][ x][)][, θ][1][)][ −] _[p]ρ[⊤][(][t][1][;][ x][)][∂][θ][f]_ [(][Z][ρ][(][t][1][;][ x][)][, θ][2][)]
_≤[∇][θ]_ _∼_
 (I) 


(44)

+| Ex _µ_ _p[⊤]ρ_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ][(][t][1][;][ x][)]{z[, θ][2][)][ −] _[p]ρ[⊤][(][t][1][;][ x][)][∂][θ][f]_ [(][Z][ρ][(][t][2][;][ x][)][, θ][2]}[)]
_∼_
 (II) 


+ |Ex _µ_ _p[⊤]ρ_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ][(][t][2][;][ x][)][, θ]{z[2][)][ −] _[p][⊤][(][t][2][;][ x][)][∂][θ][f]_ [(][Z][ρ][(][t][2][;][ x][)][, θ][2][)] } _._
_∼_
 (III) 


To bound (I), we use the mean-value theorem, Assumption 4.1 (14), and (18) to obtain| {z }

_∂θf_ (Zρ(t1; x), θ1) _∂θf_ (Zρ(t1; x), θ2) _∂θ[2][f]_ [(][Z][ρ][(][t][1][;][ x][)][,][ (1][ −] _[λ][)][θ][1]_ [+][ λθ][2][)][||][θ][1]
_|_ _−_ _| ≤|_ _[−]_ _[θ][2][|]_

_C_ _, max_ _θ1_ _θ2_ _._
_≤_ _L_ _i=1,2[(][|][θ][i,][[1]][|][)]_ _|_ _−_ _|_
 

For (II), we note first that


_t1_

_t2_

Z


_Zρ(t1; x)_ _Zρ(t2; x)_
_|_ _−_ _| ≤_


_Zρ(t1; x)_ _Zρ(t2; x)_
_|_ _−_ _| ≤_ _t2_ R[k][ f] [(][Z][ρ][(][t][;][ x][)][, θ][) d][ρ][(][θ, t][) d][t]

Z Z

_t1_

_≤_ _C(L)_ _t2_ R[k] _|θ|[2]_ + 1 dρ(θ, t) dt

Z Z  

_C(_ )( + 1) _t1_ _t2_
_≤_ _L_ _L[sup]_ _|_ _−_ _|_
_C(_ ) _t1_ _t2_ _,_
_≤_ _L[sup]_ _|_ _−_ _|_

where we use Assumption 4.1 (12) together with (18) and L ≤L[sup]. This bound implies that

(II) Ex _µ_ _p[⊤]ρ_ [(][t][1][;][ x][)][| |][∂][θ][f] [(][Z][ρ][(][t][1][;][ x][)][, θ][2][)][ −] _[∂][θ][f]_ [(][Z][ρ][(][t][2][;][ x][)][, θ][2][)][|]
_≤_ _∼_ _|_

_≤_ _C(L)Ex∼µ (|∂θf_ (Zρ(t1; x), θ2) − _∂θf_ (Zρ(t2; x), θ2)|) 

_C(_ _,_ _θ2,[1]_ )Ex _µ (_ _Zρ(t1; x)_ _Zρ(t2; x)_ )
_≤_ _L[sup]_ _|_ _|_ _∼_ _|_ _−_ _|_

_C(_ _,_ _θ2,[1]_ ) _t1_ _t2_ _,_
_≤_ _L[sup]_ _|_ _|_ _|_ _−_ _|_

where we use (31) in the first inequality, Assumption 4.1 (14) for f, and (18) in the second inequality.


-----

To bound term (III), we again use boundedness of Zρ, pρ and the Lipschitz condition of f to obtain

_t1_

_|pρ(t1; x) −_ _pρ(t2; x)| ≤_ _t2_ R[k][ p]ρ[⊤][(][t][;][ x][)][∂][z][f] [(][Z][ρ][(][t][;][ x][)][, θ][) d][ρ][(][θ, t][) d][t]

Z Z

_t1_

_≤_ _C(L)_ _t2_ R[k] [(][|][θ][|][2][ + 1) d][ρ][(][θ, t][) d][t]

Z Z

_C(_ )( + 1) _t1_ _t2_
_≤_ _L_ _L[sup]_ _|_ _−_ _|_
_C(_ ) _t1_ _t2_ _,_
_≤_ _L[sup]_ _|_ _−_ _|_

so that

(III) Ex _µ (_ _∂θf_ (Zρ(t2; x), θ2) _pρ(t1; x)_ _pρ(t2; x)_ ) _C(_ )( _θ2_ + 1) _t1_ _t2_ _,_
_≤_ _∼_ _|_ _||_ _−_ _|_ _≤_ _L[sup]_ _|_ _|_ _|_ _−_ _|_

where we use Assumption 4.1 (13) in the second inequality. By substituting these bounds into (44),
we complete the proof of (42).

Finally, to prove (43), we recall the definition of the Fréchet derivative, to obtain

_δE(ρ1)_ _δE(ρ2)_

(θ, t) _θ_ (θ, t)
_δρ_ _−∇_ _δρ_

Ex _µ (_ _∂θf_ (Zρ1 (t; x), θ)pρ1 (t; x) _∂θf_ (Zρ2 (t; x), θ)pρ2 (t; x) )
_≤_ _[∇]∼[θ]_ _|_ _−_ _|_
Ex _µ (_ _∂θf_ (Zρ1 (t; x), θ) _∂θf_ (Zρ2 (t; x), θ) _pρ1_ (t; x) )
_≤_ _∼_ _|_ _−_ _| |_ _|_
+ Ex _µ (_ _∂θf_ (Zρ2 (t; x), θ) _pρ1_ (t; x) _pρ2_ (t; x) )
_∼_ _|_ _| |_ _−_ _|_
_C(_ _,_ _θ[1]_ )Ex _µ (_ _Zρ1_ (t; x) _Zρ2_ (t; x) ) + C( )( _θ_ + 1)Ex _µ (_ _pρ1_ (t; x) _pρ2_ (t; x) )
_≤_ _L_ _|_ _|_ _∼_ _|_ _−_ _|_ _L_ _|_ _|_ _∼_ _|_ _−_ _|_

_C(_ _,_ _θ[1]_ _,_ )(1 + _θ_ )d1(ρ1, ρ2),
_≤_ _L_ _|_ _|_ _R_ _|_ _|_

where we use Assumption 4.1 for f with (18) and (31) in the third inequality. In the last inequality,
we use (19), (32).

C.2 A-PRIORI ESTIMATE UNDER THE HOMOGENEOUS ASSUMPTION

In Theorem 6.1 and 6.2, 2-homogeneity or partial 1-homogeneity are assumed. When these properties
hold, we can sharpen the estimates obtained in the previous section. We summarize our results here.

**Lemma C.3 Suppose that Assumption 4.1 holds, then there exists a constant C3(r) depending**
_increasingly on r such that for any r > 0 with |x| < r and θ ∈_ R[k], we have the following.



-  If f is 2-homogeneous (Assumption 6.1), then
_∂x[2][f]_ _≤_ _C3(r)_ _|θ|[2]_ + 1 _,_ _|∂x∂θf_ _| ≤_ _C3(r) (|θ| + 1),_ _∂θ[2][f]_ _≤_ _C3(r) ._ (45)
 


-  If f is partially 1-homogeneous (Assumption 6.2), then
_∂x[2][f]_ _≤_ _C3(r)(|θ| + 1),_ _|∂x∂θf_ _| ≤_ _C3(r)(|θ| + 1),_ _∂θ[2][f]_ _≤_ _C3(r)(|θ| + 1), (46)_

_where | · | denotes the Frobenius norm._

**Proof When f satisfies Assumption 4.1 and Assumption 6.2, (46) can be obtained from direct**
calculation. When f is 2-homogeneous in θ, we have

_f_ (x, θ) = _θ_ _f (x, θ/_ _θ_ ) = f1(θ)f2(x, θ),
_|_ _|[2]_ _|_ _|_

where f1(θ) = _θ_ _, f2(x, θ) = f (x, θ/_ _θ_ ). Naturally, with product rule, the derivatives of f
_|_ _|[2]_ _|_ _|_
becomes the products of derivatives of f1 and f2. We then obtain (45), noting that when |θ| > 1, we
have
_∂x[2][f][1]_ = 0, _∂x∂θf1_ 0, _∂xf1_ = 0, _∂θf1_ 2 _θ_ _,_ _∂θ[2][f][1]_ 2k,
_|_ _| ≤_ _|_ _|_ _|_ _| ≤_ _|_ _|_ _≤_
and

_∂x[2][f][2]_ = C(r), _∂x∂θf2_ _∂xf2_ = C(r), _∂θf2_ _∂θ[2][f][2]_
_|_ _| ≤_ _[C]θ[(][r][)]_ _[,]_ _|_ _|_ _|_ _| ≤_ _[C]θ[(][r][)]_ _[,]_ _≤_ _[C]θ[(][r][)]_

_|_ _|_ _|_ _|_ _|_ _|[2][ .]_


-----

The estimates above allow us to improve the stability results in Lemmas C.1 and C.2.

**Lemma C.4in (20) and** _Suppose that Assumption 4.1 holds and let ρ1, ρ2 ∈C([0, 1]; P_ [2]). Define L1 and L2 as


1 = sup
_L[sup]_ _t∈[0,1]_

_Then we have the following results._


R[k][ |][θ][|][2][dρ][1][(][θ, t][)][ .]



-  If f either satisfies Assumption 6.1 or Assumption 6.2, we have the following properties.

_1. Stability of pρ:_
_pρ1_ (t; x) _pρ2_ (t; x) _C(_ 1, 2)d1(ρ1, ρ2) . (47)
_|_ _−_ _| ≤_ _L_ _L_

_2. Lipschitz continuity in ρ:_
_δE(ρ1)_ _δE(ρ2)_

(θ, t) _θ_ (θ, t) (48)
_δρ_ _−∇_ _δρ_ _[≤]_ _[C][(][L][1][,][ L][2][)][d][1][(][ρ][1][, ρ][2][) (][|][θ][|][ + 1)][,]_

_where d1 is defined in (4.1)._

_[∇][θ]_

-  Lipschitz continuity in θ and t: If f satisfies Assumption 6.1, then for any (θ1, t1), (θ2, t2)
_∈_
R[k] _× [0, 1], we have_
_δE(ρ1)_ _δE(ρ1)_

_δρ_ (θ1, t1) −∇θ _δρ_ (θ2, t2) _[≤]_ _[C][(][L][1][)][|][θ][1][ −]_ _[θ][2][|]_ [+] _[C][(][L]1[sup])(|θ2|_ +1)|t1 − _t2| ._

(49)

_[∇][θ]_

-  Lipschitz continuity in θ and t: If f satisfies Assumption 6.2, then for any (θ1, t1), (θ2, t2)
_∈_
R[k] _× [0, 1], we have_
_δE(ρ1)_ _δE(ρ1)_

_δρ_ (θ1, t1) −∇θ _δρ_ (θ2, t2) _[≤]_ _[C][(][L]1[sup])(|θ1|+|θ2|+1)(|θ1_ _−θ2|+|t1_ _−t2|),_

(50)

_[∇][θ]_

**Remark C.1 We note that in comparing (32) with (47) and (42)-(43) with (48)-(50), the main**
_differences are the dependence of the bounds on θ. The new estimates have explicit (and mild)_
_dependence on θ._

**Proof First, to prove (47), we let ∆(t; x) = pρ1** (t; x) _pρ2_ (t; x), and recall (35) and (36). Using
_−_
the boundedness of Zρ in (18) and (19), and calling (45) (or (46)), we have from (36) that

_Dρ1,ρ2_ (t; x) _C(_ 1, 2) _d1(ρ1, ρ2) ._
_|_ _| ≤_ _L_ _L_
ZR[k][ |][θ][|][2][dρ][1][(][θ, t][) +] ZR[k][ |][θ][|][2][dρ][2][(][θ, t][) + 1]

By substituting into (35), and using (31), (33), and Hölder’s inequality, we have
d ∆(t; x)
_|_ _|[2]_ _C(_ 1, 2) ∆(t; x)

dt _≤_ _L_ _L_ _|_ _|[2]_
ZR[k][ |][θ][|][2][dρ][1][(][θ, t][) +] ZR[k][ |][θ][|][2][dρ][2][(][θ, t][) + 1]

+ C(L1, L2) _d[2]1[(][ρ][1][, ρ][2][)][ .]_
ZR[k][ |][θ][|][2][dρ][1][(][θ, t][) +] ZR[k][ |][θ][|][2][dρ][2][(][θ, t][) + 1]

The result (47) follows from the initial condition (34) together with Grönwall’s inequality.


Next, to prove (48), we have
_δE(ρ1)_ _δE(ρ2)_

(θ, t) _θ_ (θ, t)
_δρ_ _−∇_ _δρ_

Ex _µ (_ _∂θf_ (Zρ1 (t; x), θ)pρ1 (t; x) _∂θf_ (Zρ2 (t; x), θ)pρ2 (t; x) )
_≤_ _[∇]∼[θ]_ _|_ _−_ _|_
Ex _µ (_ _∂θf_ (Zρ1 (t; x), θ) _∂θf_ (Zρ2 (t; x), θ) _pρ1_ (t; x) )
_≤_ _∼_ _|_ _−_ _| |_ _|_
+ Ex _µ (_ _∂θf_ (Zρ2 (t; x), θ) _pρ1_ (t; x) _pρ2_ (t; x) )
_∼_ _|_ _| |_ _−_ _|_
_C(_ 1, 2)( _θ_ + 1) (Ex _µ (_ _Zρ1_ (t; x) _Zρ2_ (t; x) ) + Ex _µ (_ _pρ1_ (t; x) _pρ2_ (t; x) ))
_≤_ _L_ _L_ _|_ _|_ _∼_ _|_ _−_ _|_ _∼_ _|_ _−_ _|_
_C(_ 1, 2)d1(ρ1, ρ2) ( _θ_ + 1),
_≤_ _L_ _L_ _|_ _|_


-----

where we also use (13), (18), (31), (45) (or (46)) in the second inequality and (19) and (47) in the
final inequality.


Finally, to prove (49) and (50), we have as in (44) that

_δE(ρ1)_ _δE(ρ1)_

(θ1, t1) _θ_ (θ2, t2)
_δρ_ _−∇_ _δρ_

_≤_ E[∇]x[θ]∼µ _p[⊤]ρ1_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ]1 [(][t][1][;][ x][)][, θ][1][)][ −] _[p][⊤]ρ1_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ]1 [(][t][1][;][ x][)][, θ][2][)]
 (I) 

|+ Ex∼µ _p[⊤]ρ1_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ]1 [(][t][1][;][ x][)]{z[, θ][2][)][ −] _[p][⊤]ρ1_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ]1 [(][t][2][;][ x][)][, θ][2]}[)]

 (II)

+ |Ex∼µ _p[⊤]ρ1_ [(][t][1][;][ x][)][∂][θ][f] [(][Z][ρ]1 [(][t][2][;][ x][)][, θ]{z[2][)][ −] _[p][⊤]ρ1_ [(][t][2][;][ x][)][∂][θ][f] [(][Z][ρ]1 [(][t][2][;][ x][)][, θ][2][)]
 (III)

The boundedness of the three terms above relies on Assumption 4.1 and (45) (or (46)).| {z

To bound (I), if f is 2-homogeneous, we have


(51)


(I) _C(_ 1) _θ1_ _θ2_ _,_
_|_ _| ≤_ _L_ _|_ _−_ _|_

where we use (18), (31), and ∂θ[2][f][ ≤] _[C][(][|][z][|][)][. If][ f][ satisfies Assumption 6.2, we have]_

(I) _C(_ 1)( _θ1_ + _θ2_ + 1) _θ1_ _θ2_ _,_
_|_ _| ≤_ _L_ _|_ _|_ _|_ _|_ _|_ _−_ _|_

where we use (18), (31), and ∂θ[2][f] [(][z, θ][)][ ≤] _[C][(][|][z][|][)(][|][θ][|][ + 1)][.]_

The bounds of (II) and (III) are same for both homogeneity assumptions and similar to the proof of
Lemma C.2. For (II), we have


_∂θf_ (Zρ1 (t1; x), θ2) _∂θf_ (Zρ1 (t2; x), θ2) _C(_ 1)( _θ2_ + 1) _Zρ1_ (t1; x) _Zρ1_ (t2; x)
_|_ _−_ _| ≤_ _L_ _|_ _|_ _|_ _−_ _|_

and
_Zρ1_ (t1; x) _Zρ1_ (t2; x) _C(_ 1)( 1 + 1) _t1_ _t2_ _._
_|_ _−_ _| ≤_ _L_ _L[sup]_ _|_ _−_ _|_

For (III), we also use
_∂θf_ (Zρ1 (t2; x), θ2) _C(_ 1)( _θ2_ + 1)
_|_ _| ≤_ _L_ _|_ _|_

and
_pρ1_ (t1; x) _pρ1_ (t2; x) _C(_ 1)( 1 + 1) _t1_ _t2_ _._
_|_ _−_ _| ≤_ _L_ _L[sup]_ _|_ _−_ _|_

These estimates, together with 1 1 and (51), prove the claims (49) and (50).
_L_ _≤L[sup]_

D WELL POSEDNESS OF GRADIENT FLOW

Theorem A.4 is about well-posedness of the gradient flow equation (9) in the mean-field limit, while
Theorem A.3 shows the corresponding result in the discrete setting. The proof for the two are similar.
We first show the mean-field limit well-posedness, Theorem A.4.

D.1 PROOF OF THEOREM A.4

We use the fixed-point argument. To do so, we first define a subset of C([0, S]; C([0, 1]; P [2])) with
compact support measures, as follows:


ΩS = _φ(θ, t, s) ∈C([0, S]; C([0, 1]; P_ [2])) _∃r > 0, supp(φ) ⊂{θ||θ[1]| < r}, ∀(t, s) ∈_ [0, 1] × [0, S]
 _φ(θ, t, s)_ ([0, S]; ([0, 1]; )) _φ(θ, t, 0) = ρini(θ, t)_

_∩_ _∈C_ _C_ _P_ [2]

For any φ(θ, t, s) ∈ ΩS with φ(θ, t, 0) = ρini(θ, t), we define a map

_ϕ =_ _S(φ) : ΩS_ ΩS (52)
_T_ _→_


-----

where ϕ solves


_∂ϕ(θ, t, s)_ _δE(φ(s))_

= _θ_ _ϕ(θ, t, s)_ _θ_ (θ, t) _,_
_∂s_ _∇_ _·_ _∇_ _δρ_ (53)

  
ϕ(θ, t, 0) = ρini(θ, t) .

The strategy is to show this map is a contraction map, so that there is a fixed point in the set ΩS,



which is then the solution to equation (9).

The proof of Theorem A.4 is divided into three steps:

Step 1: We show TS is well-defined map from ΩS to ΩS.
Step 2: We give a bound of d2(TS(φ1), TS(φ2)) in terms of d2(φ1, φ2). One can then tune S to
ensure TS is a contraction map, meaning there is a unique fixed point φ[∗] so that φ[∗] = TS(φ[∗]),
and thus φ[∗] solves (53) for s < S.

Step 3: We extend the local solution to a global solution.

**Step 1.** According to the definition of (53), for a fixed φ(θ, t, s), let
dθφd(ss;t) = −∇θ _δE(δρφ(s))_ (θφ(s; t), t), (54)
(θφ(0; t) _ρini(θ, t) ._

_∼_

Then θφ _ϕ =_ _S(φ). To show the existence of ϕ amounts to showing the wellposedness of (54)._
_∼_ _T_

According to Lemma C.2 (41) and (42), the force ∇θ _δφδE(s)_ [(][·][, t][)][ has a linear growth and is locally]

Lipschitz. Classical ODE theory then suggests there is a unique solution for s ∈ [0, S], which
depends continuously on the initial value θ(0; t).


Denoting


1

_LS,φ =_ 0≤sups≤S Z0 ZR[k][ |][θ][|][2][ d][φ][(][θ, t, s][) d][t,]

_Lini[sup]_ [= sup]0≤t≤1 ZR[k][ |][θ][|][2][ d][ρ][ini][(][θ, t][)][,]

_LS,φ[sup]_ [=] 0≤t≤sup1,0≤s≤S ZR[k][ |][θ][|][2][ d][φ][(][θ, t, s][)]

ini = inf supp(ρini(θ, t)) _θ_ _θ[1]_ _< r_ _,_ _t_ [0, 1]
_R_ _r>0_ _⊂{_ _||_ _|_ _}_ _∀_ _∈_



(55)


_S,φ = inf_ supp(φ(θ, t, s)) _θ_ _θ[1]_ _< r_ _,_ (t, s) [0, 1] [0, S] _,_
_R_ _r>0_ _⊂{_ _||_ _|_ _}_ _∀_ _∈_ _×_

we have the following proposition.

**Proposition D.1 Suppose that θφ(s; t) solves (54) and φ ∈** ΩS. Then for any (t1, s1), (t2, s2) ∈

[0, 1] × [0, S], we have

E _|θφ(s1; t1)|[2][]_ _≤_ exp(SC(LS,φ)) (Lini[sup] [+ 1)][,][ |][θ][φ,][[1]][(][s][1][;][ t][1][)][|]
 _≤_ exp(SC(LS,φ)) (Rini + 1) (56)

_and_

E _|θφ(s1; t1) −_ _θφ(s2; t2)|[2][]_

(57)



_≤_ _C_ _LS,φ[sup][,][ R][ini][, S]_ E _|θφ(0; t1) −_ _θφ(0; t2)|[2][]_ + |t1 − _t2|[2]_ + |s1 − _s2|[2][]_ _._
   

**Proof To prove the first bound in (56), we multiply (54) on both sides by θφ and use boundedness of**
the forcing term (41) to obtain


d _θφ(s; t1)_
_|_ _|[2]_ _C(_ _S,φ)(_ _θφ(s; t1)_ + 1) .

ds _≤_ _L_ _|_ _|[2]_


-----

Using Grönwall’s inquality together with E _|θφ(0; t1)|[2][
]_ _≤Lini[sup][, we obtain the first bound in][ (56)][.]_

To prove the second bound in (56), we use the bound of _∂θ[1]_ _δEδρ(ρ)_ [(][θ, t][)] according to Lemma C.2

(41):
d _θφ,[1](s; t1)_

ds _≤_ _C(LS,φ)(_ _θφ,[1](s; t1)_ + 1) .

Using Grönwall’s inquality together with[2] _θφ,[1](0; t1)_ ini, we obtain the second bound in (56).
_|_ _| ≤R_ [2]

To show (57), we first write

E |θφ(s1; t1) − _θφ(s2; t2)|[2][]_

_≤_ 2 E |θφ(s1; t1) − _θφ(s1; t2)|[2][]_ +2 E |θφ(s1; t2) − _θφ(s2; t2)|[2][]_ _,_ (58)

(I) (II)

then bound the two terms| (I) and (II){z separately. For} (I)|, we use (54){z, the second bound of} (56), and
Lemma C.2 (42)

d _θφ(s; t1)_ _θφ(s; t2)_
_|_ _−_ _|[2]_

ds
_C(_ _S,φ,_ ini) _θφ(s; t1)_ _θφ(s; t2)_ + C( _S,φ[,][ R][ini][)(][|][θ][φ][(][s][1][;][ t][1][)][|][2][ +][ |][θ][φ][(][s][1][;][ t][2][)][|][2][ + 1)][|][t][1][ −]_ _[t][2][|][2][,]_
_≤_ _L_ _R_ _|_ _−_ _|[2]_ _L[sup]_

Using the first bound in (56) and Grönwall’s inequality, this implies that

E _|θφ(s1; t1) −_ _θφ(s1; t2)|[2][]_ _≤_ _C_ _LS,φ[sup][,][ R][ini][, S]_ E _|θφ(0; t1) −_ _θφ(0; t2)|[2][]_ + |t1 − _t2|[2][]_ _._
     (59)
For (II), we obtain an estimate by integrating (54) from s1 to s2 and using the boundedness of
_θ_ _δEδρ(ρ)_ in (41). From the Grönwall inequality, we have
_∇_

_s2_
_θφ(s1; t2)_ _θφ(s2; t2)_ _C (_ _S,φ)_ _θφ(s; t2)_ ds + _s1_ _s2_ _._
_|_ _−_ _| ≤_ _L_ _s1_ _|_ _|_ _|_ _−_ _|_
Z 

Using first bound in (56) and Hölder’s inequality, this implies


E |θφ(s1; t2) − _θφ(s2; t2)|[2]_ _≤_ _C (LS,φ, Lini[sup][, S][)][ |][s][1][ −]_ _[s][2][|][2]_ (60)

and thus combining with (59) and substituting in (58), we complete the proof.


An immediate corollary of Proposition D.1 is that the map _S(_ ) is well defined.
_T_ _·_

**Corollary D.1 For every fixed S > 0, the map TS is well defined. That is, for any φ ∈** ΩS, one can
_find ϕ = TS(φ) ∈_ ΩS as the unique solution of (53). In particular, for any (t, s) ∈ [0, 1] × [0, S],
_we have_

ini [+ 1)][,]
R[k][ |][θ][|][2][ d][ϕ][(][θ, t, s][)][ ≤] [exp(][SQ][1][(][L][S,φ][)) (][L][sup] (61)

Z

suppθ(ϕ(θ, t, s)) _θ_ _θ[1]_ exp(SQ1( _S,φ)) (_ ini + 1) _,_
_⊂_ _|_ _| ≤_ _L_ _R_

_where Q1 : R+_ R+ is depends only on _S,φ and is an increasing function of its argument._
_→_ _L_

**Proof For fixed (t, s) ∈** [0, 1] _×_ [0, S], define ϕ(θ, t, s) as the distribution of θφ(s; t). Using classical
stochastic theory (Ambrosio et al., 2008, Prop 8.1.8), ϕ(θ, t, s) is a solution to (53). The estimate of
the support is a consequence of (56). Finally, using (16) and (57), we obtain that

lim E _θφ(s; t)_ _θφ(s0; t0)_ = 0,
(t,s) (t0,s0) _[W][2][(][ϕ][(][·][, t, s][)][, ϕ][(][·][, t][0][, s][0][))][ ≤]_ ([lim]t,s) _|_ _−_ _|[2][][1][/][2]_
_→_

 

which proves the continuity in s and t, so that ϕ ∈C([0, S]; C([0, 1]; P [2])). By combining all the
factors above, we conclude that ϕ ΩS.
_∈_


-----

**Step 2.** We show now that TS is a contraction map for S sufficiently small.

**Proposition D.2 For any φ1, φ2 ∈** ΩS, we have

_d2(_ _S(φ1),_ _S(φ2))_ _SQ2(_ _S,_ _S, S)d2(φ1, φ2),_ (62)
_T_ _T_ _≤_ _L_ _R_

_where Q2 : R[3]_ _→_ R+ is an increasing function and LS = max{LS,φ1, LS,φ2 _}, RS =_
max{RS,φ1, RS,φ2 _}, with LS,φ, RS,φ defined in (55)._


**Proof Denote by θφi** (s; t) the solutions to (54) with φ = φi using the same initial data, that is,

_θφ1_ (0; t) = θφ2 (0; t) .

As in the previous subsection, we translate the study of ϕi to the study of θφi . Defining

∆t(s) = _θφ1_ (s; t) _θφ2_ (s; t) _,_
_|_ _−_ _|_

we have according to the definition of Wasserstein distance that

_d2(TS(φ1), TS(φ2)) ≤_ (t,s)∈[0sup,1]×[0,S] E ∆[2]t [(][s][)] _._
 


Using (54), we obtain


2

d(∆t(s))[2] _δE(φ1(s))_ _δE(φ2(s))_

ds _≤_ 4(∆t(s))[2] + 4 _δρ_ (θφ1 _, t) −∇θ_ _δρ_ (θφ2 _, t)_

2

_≤_ 4(∆t(s))[2] + 8 _[∇][θ]_ _δE(φδρ1(s))_ (θφ1 _, t) −∇θ_ _δE(φδρ1(s))_ (θφ2 _, t)_ (63)

2

_δE(φ1(s))_ _δE(φ2(s))_
+ 8 _δρ_ _[∇](θ[θ]φ2_ _, t) −∇θ_ _δρ_ (θφ2 _, t)_ _._

The second term on the right hand side involves the continuity addressed in Lemma C.2 (42)

_[∇][θ]_

2

_δE(φ1(s))_ _δE(φ1(s))_

_δρ_ (θφ1 _, t) −∇θ_ _δρ_ (θφ2 _, t)_ _≤_ _C(LS, RS, S)(∆t(s))[2]_ _,_

where we use the second bound in (56) to substitute the constant in (42). Then the last term of (63)

_[∇][θ]_

involves the continuity discussed in (43). In particular, we have

_δE(φ1(s))_ _δE(φ2(s))_

_δρ_ (θφ2 _, t) −∇θ_ _δρ_ (θφ2 _, t)_

_C(_ _S,_ _θφ2,[1]_ )(1 + _θφ2_ )d1(φ1, φ2) _C(_ _S,_ _S, S)(1 +_ _θφ2_ )d2(φ1, φ2),
_≤_ _[∇]R[θ]_ _|_ _|_ _|_ _|_ _≤_ _L_ _R_ _|_ _|_

where d2 is defined in Definition 4.1 and in the second inequality we use the second bound in (56) to
substitute the constant in (43).

By substituting in (63), we obtain


d(∆t(s))[2]

_C(_ _S,_ _S, S)_ (∆t(s))[2] + (1 + _θφ2_ )d[2]2[(][φ][1][, φ][2][)]
ds _≤_ _L_ _R_ _|_ _|[2]_



which implies

d E(∆t(s))[2][
]

_C(_ _S,_ _S, S)_ E(∆t(s))[2][
] + d[2]2[(][φ][1][, φ][2][)] _,_
ds _≤_ _L_ _R_



 

where we use the first inequality in (56). From the Grönwall inequality, there exists Q2 : R[3] _→_ R+
is an increasing function such that

E (∆t(s))[2][
] _≤_ _SQ2(LS, RS, S)d[2]2[(][φ][1][, φ][2][)][,]_

completing the proof. 


To apply the contraction mapping theorem, we need to verify two conditions in order to show that
there exists a fixed point φ[∗] = _S(φ[∗]):_
_T_


-----

-  There is a closed subset in ΩS such that TS maps this set to itself.

-  TS is a contraction map in this subset.

For the closed subset, we define
_Bρ0 =_ _φ ∈_ ΩS supp(φ(t, s)) ⊂{θ||θ[1]| ≤ 4(Rini + 1)}, _∀(t, s) ∈_ [0, 1] × [0, S]

(64)



_∩_ _φ ∈_ ΩS ini [+ 1)][,] _∀(t, s) ∈_ [0, 1] × [0, S] _._
 ZR[k][ |][θ][|][2][ d][φ][(][θ, t, s][)][ ≤] [4 (][L][sup] 

We now claim that for small enough S, TS is a contraction map in Bρ0 .

**Proposition D.3 Suppose that S is small enough that**
exp(SQ1(4(Lini[sup] [+ 1))) (][L]ini[sup] [+ 1)][ ≤] [4(][L]ini[sup] [+ 1][,]

exp(SQ1(4(Lini[sup] [+ 1))) (][R][ini][ + 1)][ ≤] [4(][R][ini][ + 1)][,]

_SQ2(4(_ ini [+ 1)][,][ 4(][R][ini][ + 1)][, S][)][ <][ 1]
_L[sup]_ 2 _[,]_

_where Q1 and Q2 are defined in Corollary D.1 and Proposition D.2, respectively. Then we have the_
_following._

-  If φ _Bρ0_ _, then_ _S(φ)_ _Bρ0_ _, that is, for any (t, s)_ [0, 1] [0, S], we have
_∈_ _T_ _∈_ _∈_ _×_
supp(TS(φ)(t, s)) ⊂{θ||θ[1]| ≤ 4(Rini + 1)}, (65)
_and_

ini [+ 1)][ .] (66)
R[k][ |][θ][|][2][ d][T][S][(][φ][)(][θ, t, s][)][ ≤] [4 (][L][sup]

Z

-  TS is a contraction map in this subset, meaning that for any φ1, φ2 ∈ _Bρ0_ _, we have_

_d2(_ _S(φ1),_ _S(φ2)) <_ [1] (67)
_T_ _T_ 2 _[d][2][(][φ][1][, φ][2][)][ .]_


using (62) with (65) and (66), we haveProof First, using Corollary D.1 (61) and noticing LS,φ ≤ 4 (Lini[sup] [+ 1)][, we prove][ (65)][,][ (66)][. Then,]

_d2(_ _S(φ1),_ _S(φ2))_ _SQ2(4(_ ini [+ 1)][,][ 4(][R][ini][ + 1)][, S][)][ <][ 1]
_T_ _T_ _≤_ _L[sup]_ 2 _[d][2][(][φ][1][, φ][2][)][,]_

which proves (67).


Using the contraction mapping theorem, we can obtain directly that _S(φ) has a fixed point in Bρ0_
_T_
when S is small enough.

**Corollary D.2 If S satisfies conditions in Proposition D.3, then there exists a unique φ[∗](θ, t, s) ∈**
_Bρ0 ⊂_ ΩS such that φ[∗](θ, t, s) is a solution to (9) with initial condition ρini(θ, t).

This is a direct consequence of the application of contraction mapping theorem.

Finally, we prove that the cost function decreases along the flow.

**Lemma D.1 Suppose that φ[∗](θ, t, s) ∈C([0, S]; C([0, 1]; P** [2])) solves (9) with initial condition
_ρini(θ, t). Then for 0 < s < S, we have_

1 2

dE(φ[∗](θ, t, s)) _δE(φ[∗](s))_

= (θ, t) dφ[∗](θ, t, s) dt 0 . (68)
ds _−_ 0 R[k] _δρ_ _≤_
Z Z

**Proof Denote by θ[∗](s; t) the associated path, meaning that θ[∗](s; t) solves (54) with φ = φ[∗], then**

_[∇][θ]_

_θ[∗]_ _∼_ _φ[∗], meaning the distribution of θ[∗]_ is φ[∗]. According to (9), we obtain using a change of variable
that

1 2

dE(φ[∗](θ, t, s)) _δE(φ[∗](s))_

= (θ, t) dφ[∗](θ, t, s) dt 0, (69)
ds _−_ 0 R[k] _δρ_ _≤_
Z Z

which proves the result. We note that the derivation in (69) is formal. A rigorous proof can be found
in (Ding et al., 2021, Appendix I). _[∇][θ]_


-----

**Step 3.** In this final step of the proof, we extend the local solution from Corollary D.2 to a global
solution. Lemma D.1 shows that the formula of [d]d[E]s [, so we can then use this formula to improve]

the bound for the support of the solution (61). This improvement will be shown in the following
corollary. This improved estimate helps in extending the local solution to the global solution.

**Corollary D.3 For fixed S satisfying the condition in Proposition D.3, denote by φ[∗](θ, t, s) ∈**
([0, S]; ([0, 1]; )) the solution to (9) with initial condition ρini(θ, t). Then for any (t, s)
_C_ _C_ _P_ [2] _∈_

[0, 1] × [0, S], we have

ini [)][,]
R[k][ |][θ][|][2][ d][φ][∗][(][θ, t, s][)][ ≤] _[C][(][S,][ R][ini][,][ L][sup]_ (70)

Z

supp(φ[∗](t, s)) ⊂ _θ_ _|θ[1]| ≤_ _C(S, Rini, Lini[sup][)][}][,]_

_where the quantity C depends only on S, Rini, and Lini[sup][)][.]_

**Proof According to (61), it suffices to prove**

1

_LS,φ∗_ = 0≤sups≤S Z0 ZR[k][ |][θ][|][2][ d][φ][∗][(][θ, t, s][) d][t][ ≤] _[C][(][S,][ R][ini][,][ L]ini[sup][)][ .]_

Denote by θ[∗](s; t) the particle representation of φ[∗], meaning that θ[∗](s; t) solves (54) with φ = φ[∗].
Since θ[∗](s; t) ∼ _φ[∗](s; t),_

1 1

_S,φ∗_ = sup sup E _θ[∗](s; t)_ dt .
_L_ 0≤s≤S Z0 ZR[k][ |][θ][|][2][ d][φ][∗][(][θ, t, s][) d][t][ =] 0≤s≤S Z0 _|_ _|[2][
]_



Using (54), we obtain that

d _θ[∗](s; t)_ _δE(φ[∗](s))_
_|_ _|[2]_ _θ[∗](s; t)_ (θ[∗](s; t), t)

_[,]_

ds _≤|_ _|_ _δρ_

which gives

_[∇][θ]_

1
d 0 [E] _|θ[∗](s; t)|[2][
]_ _dt_

ds

R 

1 1/2 1 2[!]

_δE(φ[∗](s))_

E _θ[∗](s; t)_ _dt_ E (θ[∗](s; t), t) _dt_

_≤_ Z0 _|_ _|[2][
]_  Z0 _δρ_ !

1  1/2

dE(φ[∗](θ, t, s[∇][θ] ))

= E _θ[∗](s; t)_ _dt_ _[,]_

0 _|_ _|[2][
]_ ds

Z 



where we use the Hölder inequality and (68) from Lemma D.1 in the last equality. Since
1

0 E _|θ[∗](0; t)|[2][
]_ _dt ≤Lini[sup][,]_

Z

_S_ 

dE(φ[∗](θ, t, s))

[d][s][ ≤] _[E][(][ρ][ini][(][θ, t][))][ −]_ _[E][(][φ][∗][(][θ, t, S][))][ ≤]_ _[C][(][L]ini[sup][)][,]_

0 ds

Z

we obtain

1

_LS,φ∗_ = sup0≤u≤s Z0 E _|θ[∗](u; t)|[2][
]_ _dt ≤_ _C(S, Lini[sup][)]_



by Grönwall’s inequality. This proves (70).


By contrast with (61), this estimate removes the dependence of the bound on _S,φ. This improvement_
_L_
is important because it relaxes the fixed-point argument from the dependence on the initial guess φ.

We are now ready to prove Theorem A.4.

**Proof [Proof of Theorem A.4] From Corollary D.2, let S1 be a constant satisfying the conditions in**
Proposition D.3. Then there is a local solution φ[∗] _C([0, S1];_ ([0, 1]; )) to (9).
_∈_ _C_ _P_ [2]


-----

We now denote by S[∗] the largest time within which the solution exists, where we denote this solution
by φ[∗] _∈_ _C_ [0, S[∗]); C([0, 1]; P [2]) . We aim to show that S[∗] = ∞. According to Corollary D.3 (70),
for any s < S[∗] and t [0, 1], we have
 _∈_ 


ini [)][,]
R[k][ |][θ][|][2][ d][φ][∗][(][θ, t, s][)][ ≤] _[C][(][S][∗][,][ R][ini][,][ L][sup]_

Z

supp(φ[∗](t, s)) ⊂ _θ_ _|θ[1]| ≤_ _C(S[∗], Rini, Lini[sup][)]_ _,_

Define R[∗] = RS[∗],φ[∗], L[∗] = LS[sup][∗],φ[∗] [according to] [ (55)][. Since][ R][∗][,][ L][∗] _[<][ ∞][, let us choose][ ∆][S][∗]_ [small]
enough to satisfy

exp(∆S∗ _Q1(4(_ +1))) ( + 1) 4( +1), exp(∆S∗ _Q1(4(_ +1))) ( + 1) 4( +1),
_L[∗]_ _L[∗]_ _≤_ _L[∗]_ _L[∗]_ _R[∗]_ _≤_ _R[∗]_

and
∆S∗ _Q2(4(_ + 1), 4( + 1), ∆S∗ )
_L[∗]_ _R[∗]_ _≤_ [1]2 _[,]_

If S[∗] is finite, then, using Proposition D.3 and Corollary D.2, we can further extend φ[∗] to be supported
on C [0, S[∗] + ∆S∗ ); ([0, 1]; ), giving a contradiction. If follows that S[∗] =, as desired.
_C_ _P_ [2] _∞_

Finally, (21) is a direct result of Lemma D.1. 


D.2 PROOF OF THEOREM A.3

This section is dedicated to Theorem A.3 — we show the well posedness of the gradient flow in the
finite-layer case. We rewrite the gradient of (2) as follows


_∂E(ΘL,M_ )

_∂θl,m_

where pΘL,M (l; x) solves:


1

_∂θf_ (ZΘL,M (l; x), θl,m)pΘL,M (l; x) _,_ (71)
_ML_ [E][x][∼][µ]
 



_p[⊤]ΘL,M_ [(][l][;][ x][) =][ p]Θ[⊤]L,M [(][l][ + 1;][ x][)]


_I +_


_∂zf_ _ZΘL,M (l + 1; x), θl+1,i_
_m=1_

X 


(72)


_ML_


_pΘL,M (L −_ 1; x) = _g(ZΘL,M (L; x)) −_ _y(x)_ _∇g(ZΘL,M (L; x)),_

for 0 ≤ _l ≤_ _L −_ 2. We unify the space in a similar fashion to Definition 4.1. 


**Definition D.1 ΘL,M = {θl,m}l[L]=0[−][1],m[,M]=1** _[∈]_ _[L]L,M[∞]_ _[if and only if]_

sup _θl,m_ _<_ _._
_l,m_ _|_ _|_ _∞_


_The metric in L[∞]L,M_ _[is defined as]_

1/2

_M_

1

_d1,L,M_ ΘL,M _,_ ΘL,M = max _θl,m_ _θl,m_ _._
_l_ _M_ _m=1_ _|_ _−_ [e] _|[2]!_
  X

[e]

**Definition D.2 For s** 0, we have ΘL,M (s) = _θl,m(s)_ _l=0,m=1_ _L,M_ [)][ if and only if]
_≥_ _{_ _}[L][−][1][,M]_ _[∈C][([0][,][ ∞][);][ L][∞]_


_1. For fixed s ∈_ [0, ∞), ΘL,M (s) ∈ _L[∞]L,M_ _[.]_


_2. For any s0_ [0, ),
_∈_ _∞_

lim
_s_ _s0_ _[d][1][,L,M][ (Θ][L,M]_ [(][s][)][,][ Θ][L,M] [(][s][0][)) = 0][,]
_→_

_where d1,L,M is defined in Definition D.1._

_The metric in C([0, ∞); L[∞]L,M_ [)][ is defined by]

_d2,L,M_ ΘL,M _,_ ΘL,M = sup ΘL,M (s)) .
_s_ _[d][1][,L,M]_ [(Θ][L,M] [(][s][)][,][ e]
 

[e]


-----

Theorem A.3 is to say that the solution to (3) is unique in C([0, ∞); L[∞]L,M [)][ if][ Θ][L,M] [(0)][ ∈] _[L][∞]L,M_ [.]

Before proving the theorem, prepare some a-priori estimates of ZΘL,M and pΘL,M .

**Lemma D.2 Suppose that Assumption 4.1 holds and that x is in the support of µ. Let**


_L−1,M_
ΘL,M = {θl,m}[L]l=0[−][1],m[,M]=1 _[,]_ _and_ ΘL,M = _θl,m_ _l=0,m=1_ _[,]_
n o
e e


_and denote_


_L−1_

_l=0_

X

_L−1_

_l=0_

X


_θl,m_ _,_
_|_ _|[2]_
_m=1_

X


_LΘL,M =_


_LM_

1

_LM_


_θl,m_ _,_
_|[e]_ _|[2]_

_l=0_ _m=1_

X X

_θl,m_ _,_ _θl,m_ _._
_|_ _|_
o

[e]


_LΘL,M_ [=] _LM_
e

_RL,M = supl,m_


_Then for 0 ≤_ _l ≤_ _L −_ 1, we have the following properties:

-  Boundedness in ZΘL,M :
_ZΘL,M (l + 1; x)_ _≤_ _C(LΘL,M ),_ (73)

-  Lipschitz in ZΘL,M :

_ZΘL,M (l + 1; x) −_ _ZΘL,M_ [(][l][ + 1;][ x][)] _≤_ _C_ _LΘL,M, LΘL,M_ _d1,L,M_ ΘL,M _,_ ΘL,M _,_
   (74)
e e

[e]

-  Boundedness in pΘL,M :
_pΘL,M (l; x)_ _≤_ _C(LΘL,M ),_ (75)

-  Lipschitz in pΘL,M :
_pΘL,M (l; x) −_ _pΘL,M_ [(][l][;][ x][)] _≤_ _C (RL,M_ ) d1,L,M ΘL,M _,_ ΘL,M _._ (76)
 
e

**Proof From (1) and (12) we obtain** [e]


( _θl,m_ + 1)
_|_ _|[2]_
_m=1_

X

_M_

( _θl,m_ + 1)
_|_ _|[2]_
_m=1_

X


_ZΘL,M (l + 1; x)_ + 1 _≤_ _C1_ 1 +
 


_≤_ _C1 exp_

which proves (73) by iteration on l.

From (13) and (73) we obtain


_ZΘL,M (l; x)_ + 1)

_ZΘL,M (l; x)_ + 1),


_LM_

1

_LM_


_M_ _M_

1

_∂zf_ _ZΘL,M (l + 1; x), θl,m_ ( _θl,m_ + 1),

_ML_ _|_ _| ≤_ _[C][(][L]ML[Θ][L,M][ )]_ _|_ _|[2]_

_m=1_ _m=1_

X  
 X

which by (72) implies

_M_

_pΘL,M (l; x)_ 1 + _[C][(][L][Θ][L,M][ )]_ ( _θl,m_ + 1) _pΘL,M (l + 1; x)_ _._
_|_ _| ≤_ _ML_ _m=1_ _|_ _|[2]_ ! _|_ _|_

X

From this bound, together with |pΘL,M (x, L − 1)| ≤ _C|ZΘL,M (L; x)| ≤_ _C(LΘL,M ), we prove (75)_
by iteration on l.


-----

To prove (74), we subtract the two updating formulas and split the estimate to obtain

_ZΘL,M (l + 1; x) −_ _ZΘL,M_ [(][l][ + 1;][ x][)]

_M_

e

_≤_ 1 + _[C][(][L]ML[Θ][L,M][ )]_ (|θl,m|[2] + 1) _ZΘL,M (l; x) −_ _ZΘL,M_ [(][l][;][ x][)]

_m=1_

+ _C(LΘL,M, LΘL,MX[)]_ 1 _M_ ( _θl,m!_ + _θl,m_ + 1) e _d1,L,M_ ΘL,M _,_ ΘL,M _,_

_L_ e _M_ _m=1_ _|_ _|[2]_ _|[e]_ _|[2]_ !

X  

where we use (1) together with the bounds (13), and (73). Noting that |ZΘL,M (0; x)−Z[e]ΘL,M [(0;][ x][)][|][ =]
0, we prove (74) by iteration on l.
e

Finally, for (76), we subtract two equations in the form of (72), and use (72)-(75) together with
Lipschitz continuity to obtain

_pΘL,M (l; x) −_ _pΘL,M_ [(][l][;][ x][)]

_M_

e 1

_≤_ ΘL,M [(][l][ + 1;][ x][))][⊤] _I +_ _ML_ _m=1_ _∂zf_ _ZΘL,M (l + 1; x), θl+1,m_ !

_M_ X  


1

+ [(][p]Θ[Θ]L,M[L,M][(][ (][l][ + 1;][l][ + 1;][ x][ x][)] [)][ −]ML[p] [e] _∂zf_ _ZΘL,M (l + 1; x), θl+1,i_ _−_ _∂zf_ _ZΘL,M_ [(][l][ + 1;][ x][)][,][ e]θl+1,m

_m=1_

e _M_ X   
  e [!]

_≤_ _[p][⊤]1 +_ _[C][(]ML[R][L,M]_ [)] (|θl,m|[2] + 1) _pΘL,M (l + 1; x) −_ _pΘL,M_ [(][l][ + 1;][ x][)]

_m=1_

X ! e

+ _[C][(][R][L,M]_ [)] _d1,L,M_ ΘL,M _,_ ΘL,M _._

_L_
  (77)
The initial data is also controlled, as follows:[e]

_|pΘL,M (L −_ 1; x) − _pΘL,M_ [(][L][ −] [1;][ x][)][| ≤] _[C][|][Z][Θ][L,M]_ [(][L][;][ x][)][ −] _[Z]Θ[e]_ _L,M_ [(][L][;][ x][)][|]
e _C(_ _L,M_ )d1,L,M ΘL,M _,_ ΘL,M _._

_≤_ _R_
 

By combining this with (77), we prove (76) by iteration on l.

[e]


Lemma D.2 resembles Theorem A.2 and Lemma C.1. These estimates allow us to prove Theorem A.3.
Since the proof strategy is exactly the same, we omit details. Essentially we define a map

Θ(s) = _S[L,M]_ (Θ[′](s)) : ([0, ); L[∞]L,M [)][ →] _[,][ C][([0][,][ ∞][);][ L][∞]L,M_ [)][,]
_T_ _C_ _∞_

where Θ(s) solves:

e

dΘ([e] _s)_

= _ML_ ΘE(Θ[′](s)), for s 0,

[e] ds _−_ _∇_ _≥_

where Θ defines the forcing term. The estimates above provide all the ingredients to show the map is
well-defined, and for a small enough S, the map is also contracting, leading to the uniqueness of the
solution to (3). Similar to Lemma D.1, one can also show [d]d[E]s [=][ −][ML][|∇][Θ][E][(Θ][L,M] [)][|][2][, improving]

the estimates and removing the constants’ dependence on the initial guess. This extends the local
solution to the global one, as done in Step 3 for the continuous case.

E PROOF OF THEOREM 5.1

Theorem 5.1 links the cost defined by ΘL,M (s) with that defined by ρ(θ, t, s) for all s. The continuous
and mean-field limits are obtained, with both L and M sent to infinity. We decompose this result into
two parts, discussing mean-field and continuous limits separately.

We start with the full definition of "limit-admissible" for a distribution ρ.


-----

**Definition E.1 For an admissible ρ(θ, t), we say ρ(θ, t) is limit-admissible if the average of a large**
_number of particle presentations is bounded and Lipschitz with high probability. That is, for an_
_admissible ρ(θ, t), there are two constants C3 and C4, both greater than supt∈[0,1]_ R[k][ |][θ][|][2][dρ][(][θ, t][)]

_such that, for any M stochastic process presentation {θm(t)}m[M]=1_ _[that are][ i.i.d.][ drawn from]R_ _[ ρ][(][θ, t][)][,]_
_the following properties are satisfied for any η > 0 and M >_ _[C]η[3]_ _[:]_

_1. Second moment boundedness in time:_


_θm(t)_ _C4_
_|_ _|[2]_ _≤_
_m=1_

X


_≥_ 1 − _η ._ (78)


sup
_t∈[0,1]_

_M_ _l+1_

_L_

_l_

_m=1_ Z _L_

X


_2. For all L > 0, we have_

1 _L−1_

P

_M_

_l=0_

X


_l_

_L_




dt
_≤_ _[C]L[2][4]_


_≥_ 1 − _η ._ (79)



[4]

P dt 1 _η ._ (79)

_M_ _l=0_ _m=1_ Z _Ll_  _L_  _≤_ _L[2]_ ! _≥_ _−_

X X

_[θ][m][(][t][)][ −]_ _[θ][m]_

We now state the two theorems that play complementary parts in Theorem 5.1. The first theorem
addresses the limit in M under the assumption that L = ∞. This is the mean-field part of the analysis.

**Theorem E.1limit-admissible and Let Assumptions 4.1 and 4.2 hold with some suppθ(ρini(θ, t))** _θ_ _θ[1]_ _R_ _with some 0 < k R >1 ≤_ 0k. Assume that for all t [0, 1] ρini. Suppose(θ, t) is
_⊂{_ _||_ _| ≤_ _}_ _∈_
_that {θm(0; t)}m[M]=1_ _[are][ i.i.d][ drawn from][ ρ][ini][(][θ, t][)][. Suppose in addition that]_

-  ρ(θ, t, s) solves (9) with the initial condition ρini(θ, t), and

-  θm(s; t) solves (6) with the initial condition θm(0; t).

_Then for any ϵ, η, S > 0, there exists a constant C(ρini(θ, t), S) > 0 depending on ρini(θ, t), S such_
_that when M >_ _[C][(][ρ][ini]ϵ[2][(][θ,t]η_ [)][,S][)] _, we have_

P (|E(Θ(s; ·)) − _E(ρ(·, ·, s))| ≤_ _ϵ) ≥_ 1 − _η,_ _∀s < S ._

**Proof See Appendix F.**

The conclusion of this result suggests that for a 1 − _η confidence of an ϵ accuracy, M grows_
polynomially with respect to 1/ϵ and 1/η.

The second result considers the convergence of the parameter configuration for the discrete ResNet
(1) to that for the continuous ResNet (4) as L →∞. This is the continuous-limit part of the analysis.

**Theorem E.2limit-admissible and Let Assumptions 4.1 and 4.2 hold with some suppθ(ρini(θ, t))** _θ_ _θ[1]_ _R_ _with some 0 < k R >1 ≤_ 0k. Assume that for all t [0, 1] ρini. Suppose(θ, t) is
_⊂{_ _||_ _| ≤_ _}_ _∈_
_that {θm(0; t)}m[M]=1_ _[are][ i.i.d][ drawn from][ ρ][ini][(][θ, t][)][. Suppose in addition that]_

-  θm(s; t) solves (6) with initial condition θm(0; t),

-  θl,m(s) solves (3) with initial condition θm 0; _L[l]_ _._

_Then for any ϵ, η, S > 0, there exists a constant C(ρini_ (θ, t
), S) > 0 depending on ρini(θ, t), S such
_that when M_ _η_ _and L_ _ϵ_ _, we have for all s < S that_
_≥_ _[C][(][ρ][ini][(][θ,t][)][,S][)]_ _≥_ _[C][(][ρ][ini][(][θ,t][)][,S][)]_


P (|E(Θ(s; ·)) − _E(ΘL,M_ (s))| ≤ _ϵ) ≥_ 1 − _η ._

**Proof See Appendix G.**

This theorem shows that when the width is large enough, then with high probability, in the whole
training process with s < S, the difference between the loss functions defined by the discrete ResNet
and its continuous counterpart decreases to 0 as L →∞.


-----

F CONVERGENCE TO THE MEAN-FIELD PDE

This section is dedicated to mean-field analysis and the proof of Theorem E.1. The intuition of this
theorem is largely aligned with many other mean-field results, as demonstrated in (Ding et al., 2021).

As argued in Section 5, to show “equivalence" between (6) and (9), we can test them on the same
smooth function h(θ). Testing (9) on amounts to multiplying h on both sides of the equation by h.
From integration by parts we have


_δE(ρ(s))_

dρ,

R[k][ ∇][θ][h][∇][θ] _δρ_


R[k][ h][ d][ρ][(][θ][) =][ −]


ds


that is,

To test (6) on h, we let ρ =


d _δE(ρ(s))_

_θh_ _θ_ _._
ds [E][(][h][) =][ E] _∇_ _∇_ _δρ_
 

, we let ρ = _M1_ _Mm=1_ _[δ][θ][m]_ [and obtain]

_M_ _M_

P

d

_θh(θm) [d]_ _θh(θm)_ _[δE]_

ds [E][(][h][) = 1]M _∇_ ds _[θ][m][ =][ −]_ _∇_ _δθm_

_m=1_ _m=1_

X X


We see that (9) and (6) are equivalent when tested by h, if and only if the right hand sides of the two
equations above are the same, that is,

_δE(ρ)_

_M [δE]_ = _θ_ (θm, t) . (80)

_δθm_ _∇_ _δρ_


This claim can be established from the definitions of the Fréchet derivatives for _[δE]δρ[(][ρ][)]_ and _δθδEm_ [; see]

(Ding et al., 2021, Lemma 33).

To give a quantitative estimate on how quickly (6) converges to of (9), we utilize the particle
method, a classical strategy for the mean-field limit. We sketch the proof here and will it more
rigorous in the following subsections. We make use of two particle systems. In one system, the
particles evolve themselves, while in the second, the particles are moved forward according to the
underlying field constructed by the limit. In our situation, the former particle system consists of the
_M stochastic processes_ _θm(s; t)_ that descend according to E(Θ(s; )). The latter particle system
_{_ _}_ _·_
will be termed Θ(s; t) = {θ[e]m(s; t)}m[M]=1[; it descends according to][ E][(][ρ][(][·][,][ ·][, s][))][, the limiting cost]
function. Essentially, we prove

[e]

_E(Θ(s; ·)) ≈_ _E_ Θ(s; t) _≈_ _E(ρ(·, ·, s)) ._
 

The latter approximation arises roughly from the law of large numbers, but the former needs to bee
proved rigorously by tracing the two different evolving ODEs.

To be more specific, let ρ(θ, t, s) be the solution to (9) with admissible initial conditions ρini(θ, t),
and let θm(s; t) be the solution to (6) with initial conditions {θm(t, 0)}m[M]=1 [that are][ i.i.d][ drawn from]
_ρini(θ, t). Using the definition of E in (8), we have_

_|E(ρ(·, ·, s)) −_ _E(Θ(s; ·))|_

1 2 1 2

Ex _µ_ _g(Zρ(s)(1; x))_ _y(x)_ _g(ZΘ(s)(1; x))_ _y(x)_
_≤_ _∼_ 2 _−_ _−_ 2 _−_

 

Ex _µ_ _g(Z_ _ρ(s)(1; x))_ _g(ZΘ(s)(1;
_ _x))_  _g(Zρ(s)(1; x)) + g(ZΘ(
_ _s)(1; x))_ + _y(x)_ (81)
_≤_ _∼_ _−_ _|_ _|_

_C(_ _s)g(Zρ(s)(1; x))_ _g(ZΘ(s)(1; x))_ 


_≤_ _L_ _−_ []

_C(_ _s)_ _Zρ(s)(1; x)_ _ZΘ(s)(1; x)_ _,_
_≤_ _L_ _−_

where Ls = max 10 R[k][ |][θ][|][2][ d][ρ][(][θ, t, s][) d][t,][ 1]M 10 _Mm=1_ _[|][θ][m][(][s][;][ t][)][|][2][ d][t]_ . In this derivation, we

used the Lipschitz property ofnR R _g, the boundedness ofR_ P y (required in Assumptions 4.1-4.2), and theo
boundedness of Zρ and ZΘ(s). Boundedness of Zρ was shown in Theorem A.2, while the bound for
_ZΘ(s) will be addressed in Lemma F.3. The constant C depends on the support of ρini, as well as on_


-----

_s and the Lipschitz constant of g. It follows from (81) that to control E(ρ(·, ·, s)) −_ _E(Θ(s; ·)), we_
need to control
_Zρ(s)(1; x)_ _ZΘ(s)(1; x)_ _._ (82)
_−_

To do so, we employ the particle method and invent a new particle system.

According to (80), we can reformulate the original particle system as

dθm(s; t) _δE(ρ[dis](s))_

= _θ_ (θm(s; t)), (t, s) [0, 1] [0, ), (83)
ds _−∇_ _δρ_ _∀_ _∈_ _×_ _∞_


where we denote


_ρ[dis](θ, t, s) = [1]_


_δθm(s;t)(θ) ._ (84)
_m=1_

X


We invent a new system that follows the underlying flow governed by the limit. Define Θ(s) =
_{θ[e]m(s; t)}m[M]=1[, where][ e]θm solves_

[e]

dθ[e]m(s; t) _δE(ρ(s))_

= _θ_ _θm(s; t)_ _,_ (t, s) [0, 1] [0, ), (85)
ds _−∇_ _δρ_ _∀_ _∈_ _×_ _∞_

 

with initial condition e
_θm(0; t) = θm(0; t) ._ (86)

As a consequence, we have _θm(s; t)_ _ρ(θ, t, s) for all (t, s). The corresponding ensemble distribu-_
_∼_ e
tion is

_M_

[e]

_ρ[dis](θ, t, s) = M[1]_ _δθm(s;t)[(][θ][)][ .]_ (87)

_m=1_

X e

Now we have available particle systeme Θ(s) = _θm(s)_, a newly invented particle system Θ(s) =
_{_ _}_
_{θ[e]m(s)} and the mean-field flow ρ. Accordingly, there are three versions of Z: Zρ(s) that solves (7)_
using ρ(s) and ZΘ(s) and ZΘ(s) [that solve][ (4)][ using][ Θ(][s][)][ and][ e]Θ(s), respectively. We use the[e]
following relabelling for convenience:
e

_Zs = Zρ(s),_ _Zs[dis]_ = ZΘ(s), _Zs[dis]_ = ZΘ(s) _[.]_ (88)

Similarly, there are three sets of p: pρ(s), pρdis(s), and pρ[dis](se) [that solve]e [ (11)][ using][ ρ][(][s][)][,][ ρ][dis][(][s][)][, and]
_ρ[dis](s), respectively. We relabel similarly to (88) and write_
e

_ps = pρ(s),_ _p[dis]s_ = pρdis(s), _p[dis]s_ = pρ[dis](s) _[.]_ (89)

e

e

Since Θ(s) serves as a bridge, we translate the control of (82) to:e

[e] _Zs(t; x) −_ _Zs[dis][(][t][;][ x][)]_ _≤_ _Zs(t; x) −_ _Z[e]s[dis][(][t][;][ x][)]_ + _Z[e]s[dis][(][t][;][ x][)][ −]_ _[Z]s[dis][(][t][;][ x][)]_ _._ (90)

Bounding _Zs(t; x) −_ _Z[e]s[dis][(][t][;][ x][)]_ can be done using the law of large numbers. Bounding

1

_Zs[dis][(][t][;][ x][)][ −]_ _[Z]s[dis][(][t][;][ x][)]_ translates to controlling _m_ 0 _θm(s; t)_ dt, for which we will

_[|][θ][m][(][s][;][ t][)][ −]_ [e] _|_
evaluate the difference between equations (83) and (85). Since these two equations have the same
initial data, the difference between[e] _θm and_ _θm can then be controlled when the right-hand side forcing[P]_ R
terms are close.

This approach divides the proof naturally into two components. In Section F.1, we give the rigorous[e]

1

bound of (90), while in Section F.2, we trace the evolution of the difference _m_ 0

_[|][θ][m][(][s][;][ t][)][ −]_
_θm(s; t)_ dt in s, thus finalizing the proof for Theorem E.1.
_|_ R

[P]

e


-----

F.1 STABILITY IN THE MEAN-FIELD REGIME

Here we discuss control of the two terms on the right-hand side of (90). Recall that Θ(s) and Θ(s)
satisfy (83) and (85), and the two corresponding ensemble distribution are defined in (84) and (87),
respectively. For any S > 0, define

[e]

_M_ _M_

_LS[sup]_ = 0≤t≤sup1,0≤s≤S (ZR[k][ |][θ][|][2][ d][ρ][(][θ, t, s][)][,][ 1]M _i=1_ _|θm(s; t)|[2],_ _M[1]_ _i=1_ _|θ[e]m(s; t)|[2])_ (91)

X X

_S = inf_ supp(ρ(θ, t, s)) _θm(s; t)_ _m=1_ [1][|][ < r][}][,][ ∀][(][t, s][)][ ∈] [[0][,][ 1]][ ×][ [0][, S][]] _,_
_R_ _r>0_ _∪{_ _}[M]_ _[⊂{][θ][||][θ]_



We have the following lemma:

**Lemma F.1 For every fixed s, let Zs and Zs[dis]** _be as defined in (88). Then there exists a constant_
_C(_ _s_ ) such that for all t [0, 1], s [0, ), we have
_L[sup]_ _∈_ _∈_ _∞_

_Zs(t; x) −_ _Zs[dis][(][t][;][ x][)]_

1/2

1 _M_ 1 2

_≤_ _C(Ls[sup])_ _M_ _m=1_ Z0 _θm(s; τ_ ) − _θ[e]m(s; τ_ ) dτ ! (92)

X

1 2 1/2

+ C( _s_ ) _ρ[dis](θ, τ, s))_ dτ _._
_L[sup]_ 0 R[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e] !

Z Z

**Proof Since the statement holds for a fixed s, we eliminate all s dependence in all calculations in the**
proof, for conciseness.

Recalling the definitions in (88), we denote

∆(t; x) = Zs(t; x) − _Z[e]s[dis][(][t][;][ x][)][,]_ ∆(t; x) = _Zs[dis][(][t][;][ x][)][ −]_ _[Z]s[dis][(][t][;][ x][)][ .]_

It follows from the triangle inequality that

e [e]

_Zs(t; x) −_ _Zs[dis][(][t][;][ x][)]_ _≤_ ∆(t; x) + |∆(t; x)| .

We now bound these two terms. We first apply the same argument as in the proof of Theorem A.2

[e]

(see (26) in Appendix B) to obtain


d ∆(t; x) 2

dt _≤_ _C(Ls[sup])_ ∆(t; x) + _ρ[dis](θ, t, s))_

R[k][ f][ (][Z][s][(][t][;][ x][)][, θ][) d(][ρ][(][θ, t, s][)][ −] [e]

[e] Z

Using the Grönwall inequality and the fact that[e] ∆(0; x) = 0, we have

1 2 1/2

[e]

∆(t; x) _C(_ _s_ ) _ρ[dis](θ, t, s))_ dτ
_≤_ _L[sup]_ 0 R[k][ f][ (][Z][s][(][t][;][ x][)][, θ][) d(][ρ][(][θ, t, s][)][ −] [e] !

Z Z

for all[e] t ∈ [0, 1]. Similarly, to bound ∆(t; x), we have


(93)


2

d ∆(t; x)
_|_ _|[2]_ _C(_ _s_ ) ∆(t; x) + _Zs[dis][(][t][;][ x][)][, θ]_ d(ρ[dis](θ, t, s) _ρ[dis](θ, t, s))_ _._

dt _≤_ _L[sup]_ _|_ _|[2]_ _−_

R[k][ f]

Z  

From Assumption 4.1 and the fact that _Zdis is bounded in Theorem A.2, we havee_ e

2 _M_

1

[e]

_Zs[dis][(][t][;][ x][)][, θ]_ d(ρ[dis](θ, t, s) _ρ[dis](θ, t, s))_ _C(_ _s_ ) _θm(s; t)_ _θm(s; t)_
_−_ _≤_ _L[sup]_ _M_ _−_ [e]

ZR[k][ f]   _mX=1_

similar to the proof of Theorem A.2 (see (28) in Appendix B).e e


2
!


-----

Using ∆(0; x) = 0, we apply Grönwall’s inequality to obtain


1/2
!


1

0

Z


2
_θm(s; τ_ ) _θm(s; τ_ ) dτ
_−_ [e]


∆(t; x) _C(_ _s_
_|_ _| ≤_ _L[sup]_


(94)


_m=1_


The result is obtained from adding (93) and (94).

The difference in ps and p[dis]s [:]


**Lemma F.2 For every fixed s** [0, ), let ps and p[dis]s _be defined in (89). There exists a constant_
_∈_ _∞_
_C(_ _s) with_ _s,_ _s_ _defined in (91) such that for all t_ [0, 1]:
_R_ _R_ _L[sup]_ _∈_

_ps(t; x) −_ _p[dis]s_ [(][t][;][ x][)]

1/2

1 _M_ 1 2

_≤_ _C(Rs, Ls[sup])_ _M_ _m=1_ Z0 _θm(s; τ_ ) − _θ[e]m(s; τ_ ) dτ !

X

1 2 1/2 (95)

+ C( _s,_ _s_ ) _ρ[dis](θ, τ, s))_ dτ
_R_ _L[sup]_ 0 R[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e] !

Z Z

1 2 1/2

+ C( _s,_ _s_ ) _ρ[dis](θ, τ, s))_ dτ _._
_R_ _L[sup]_ 0 R[k][ ∂][z][f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e] !

Z Z

**Proof As in the previous proof, we eliminate dependence on s in some notation, for conciseness.**
Denoting ∆p(t; x) = ps(t; x) − _p[dis]s_ [(][t][;][ x][)][, we recall (11) to have:]

d ∆p(t; x)
_|_ _|[2]_

dt


_≤_ 2

ZR[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][(][θ, t, s][)] _[|][∆][p][(][t][;][ x][)][|][2]_

+ 2 ∆p(t; x) _p[dis]s_
_|_ _|_

R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][(][θ, t, s][)][ −]

Z

_C(_ _s_ ) ∆p(t; x)
_≤_ _L[sup]_ _|_ _|[2]_


_s_ [(][t][;][ x][)][, θ][) d][ρ][dis][(][θ, t, s][)]
R[k][ ∂][z][f] [(][Z] [dis]


_s_ [(][t][;][ x][)][, θ][) d][ρ][dis][(][θ, t, s][)]
R[k][ ∂][z][f] [(][Z] [dis]


+ 2


R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][(][θ, t, s][)][ −]


_C(_ _s_ ) ∆p(t; x)
_≤_ _L[sup]_ _|_ _|[2]_

2

+ 6 _ρ[dis](θ, t, s)_

R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][(][θ, t, s][)][ −] R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][e]

Z Z

2

+ 6 _ρ[dis](θ, t, s) −_

R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][e] R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][dis][(][θ, t, s][)]

Z Z

(I)

2

| {z }

+ 6 _s_ [(][t][;][ x][)][, θ][) d][ρ][dis][(][θ, t, s][)] _,_

R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][dis][(][θ, t, s][)][ −] R[k][ ∂][z][f] [(][Z] [dis]

Z Z

(II)

where we use| (13) from Assumption 4.1 together with{z (18) and (31) in the second inequality. To}
bound (I) on the right-hand side, we recall the definition (84) and (87), and use Assumption 4.1 (14)


-----

along with the boundedness of Z (as shown in (18)) to obtain

_M_

1

(I) _∂zf_ (Zs(t; x), θm(s; t)) _∂zf_ (Zs(t; x), _θm(s; t))_
_≤_ _M_ _−_

_m=1_

X 2

_M_

[e]

1

_≤C(Rs, Ls[sup])_ _M_ _m=1_ _|θm(s; t) −_ _θ[e]m(s; t)|!_

X

_M_

1 2

_≤C(Rs, Ls[sup])_ _M_ _m=1_ _θm(s; t) −_ _θ[e]m(s; t)_ ! _,_

X

where we use Hölder’s inequality in the last inequality. For (II), we have


2
!


(96)

(97)


2

_M_

1

(II) _∂zf_ (Zs(t; x), θm(s; t)) _∂zf_ (Zs[dis][(][t][;][ x][)][, θ][m][(][s][;][ t][))]
_≤_ _M_ _m=1_ _−_ !

X

_C(_ _s,_ _s_ ) _Zs(t; x)_ _Zs[dis][(][t][;][ x][)][|][2][ .]_
_≤_ _R_ _L[sup]_ _|_ _−_

By substituting (96) and (97) into the bound above, we obtain

d ∆p(t; x)
_|_ _|[2]_

dt


1 2

=C( _s_ ) ∆p(t; x) + C( _s,_ _s_ ) _θm(s; t)_ _θm(s; t)_ + _Zs(t; x)_ _Zs[dis][(][t][;][ x][)][|][2]_
_L[sup]_ _|_ _|[2]_ _R_ _L[sup]_ _M_ _m=1_ _−_ [e] _|_ _−_ !

X 2

+ 6 _ρ[dis](θ, t, s)_ _._

R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][ρ][(][θ, t, s][)][ −] R[k][ ∂][z][f] [(][Z][s][(][t][;][ x][)][, θ][) d][e]

Z Z

(98)
The “initial condition" for ps and p[dis]s yields


_|∆p(1; x)| ≤_ _C(Ls[sup])|Zs(1; x) −_ _Zs[dis][(1;][ x][)][|][ .]_

The result is obtained when we substitute (92) into (98) and use the Grönwall’s inequality.

F.2 PROOF OF THEOREM E.1

With the quantitative description of (90) presented in Lemma F.1, we complete the proof for Theorem E.1 in this section. Recall from (55) that


R[k][ |][θ][|][2][ d][ρ][ini][(][θ, t][)]


_Rini = infr>0_ supp(ρini(t)) ⊂ _θ_ _|θ[1]| < r_ _, ∀t ∈_ [0, 1] _,_ _Lini[sup]_ [= sup]0≤t≤1
 

and note that ini _θm,[1](0; t)_ for all m, t. Define
_R_ _≥|_ _|_


_θm(0; t)_ _,_
_|_ _|[2]_
_m=1_

X


ini = sup
_L[dis][,][sup]_ 0≤t≤1


We note that when M is large, L[dis]ini _[,][sup]_ is close to Lini[sup] [(which has no randomness) with high]
probability. We have the following lemma:

**Lemma F.3 For a given S > 0, there exists a constant C (S, Rini) such that for any t ∈** [0, 1], s ∈

[0, S], we have


1

_M_ _|θm(s; t)|[2]_ _≤_ _C(S, Rini, Lini[dis][,][sup]),_

_m=1_

X

_θm(s; t)_ _m=1_ [1][| ≤] _[C][(][S,][ R][ini][,][ L][dis]ini_ _[,][sup])_ _,_
_{_ _}[M]_ _[⊂{][θ][||][θ]_ _}_


(99)


-----

_Furthermore, for any x with |x| < Rµ (as in Assumption 4.2, item 4), and any s ∈_ [0, S] and
_t ∈_ [0, 1], the ODE solution is bounded as follows:
_Zs[dis][(][t][;][ x][)]_ _≤_ _C(S, Rini, Lini[dis][,][sup]),_ (100)

_while the following bound holds on p[dis]s_ _[:]_
_p[dis]s_ [(][t][;][ x][)] _≤_ _C(S, Rini, Lini[dis][,][sup]) ._ (101)

The bound (99) is a result of Corollary D.3. The bounds (100) and (101) are obtained using the same
arguments as in Theorem A.2 and Lemma C.1.

The next lemma bounds the support and second moment of _ρ[dis](θ, t, s)._

**Lemma F.4 Under conditions of Theorem E.1, for any S > 0, there is a constant C(S,** ini, ini )

e _R_ _L[dis][,][sup]_

_depending only on_ ini, ini _, S such that for any t_ [0, 1], s [0, S], we have
_R_ _L[dis][,][sup]_ _∈_ _∈_


1

_M_ _m=1_ _|θ[e]m(s; t)|[2]_ _≤_ _C(S, Rini, Lini[dis][,][sup]),_ (102)

X

_θm(s; t)_ _m=1_ [1][| ≤] _[C][(][S,][ R][ini][,][ L][dis]ini_ _[,][sup])_ _,_
_{[e]_ _}[M]_ _[⊂{][θ][||][θ]_ _}_

**Proof Similar to the proof of Proposition D.1, we multiply (85) by** _θm(s; t) on both sides and utilize_
the bound (41) from Lemma C.2, where in (41) is replaced by C(S, ini, ini ) according to
_L_ _R_ _L[dis][,][sup]_
Corollary D.3. We thus obtain [e]

_θm(s; t)_ _C(S,_ ini, ini ) _θm(0; t)_ + 1 _,_
_|[e]_ _| ≤_ _R_ _L[dis][,][sup]_ _|[e]_ _|_

and  
_θm,[1](s; t)_ _C(S,_ ini, ini ) _θm,[1](0; t)_ + 1
_|[e]_ _| ≤_ _R_ _L[dis][,][sup]_ _|[e]_ _|_

which implies (102) by (86).  


Denote 0 = max ini _,_ ini 0 should be bound by
_C4 with high probability. We are now ready for the main proof of this section. L[sup]_ _{L[dis][,][sup]_ _L[sup][}][. According to Definition E.1][ (78)][,][ L][sup]_

**Proof [Proof of Theorem E.1] First, using Lemmas F.3, F.4, there is a constant C(S,** ini, 0 )
_R_ _L[sup]_
depending only on ini, 0 _, S such that for any t_ [0, 1], s [0, S]
_R_ _L[sup]_ _∈_ _∈_

_M_ _M_

1 2 1

max (ZR[k][ |][θ][|][2][ d][ρ][(][θ, t, s][)][,] _M_ _m=1_ _θm(s; t)_ _,_ _M_ _m=1_ _|θm(s; t)|[2])_ _≤_ _C(S, Rini, L0[sup]),_

X X

_M_
supp(ρ(θ, t, s)) _θm(s; t)_ _m=1_ [e]θm(s; t) _θ_ _θ[1]_ _C(S,_ ini, 0 ) _,_
_∪{_ _}[M]_ _[∪]_ n om=1 _[⊂]_  _|_ _| ≤_ _R_ _L[sup]_ (103)
e

Recalling (81), we have _s_ _C(S,_ ini, 0 ) and
_L_ _≤_ _R_ _L[sup]_

_|E(ρ(·, ·, s)) −_ _E(Θ(s; ·))| ≤_ _C(S, Rini, L0[sup])_ _Zs(1; x) −_ _Zs[dis][(1;][ x][)]_ _._ (104)
Furthermore, according to Lemma F.1 (92), from (103)
_Zs(t; x) −_ _Zs[dis][(][t][;][ x][)]_

1/2

1 _M_ 1 2

_≤_ _C(S, Rini, L0[sup])_ _M_ _m=1_ Z0 _θm(s; τ_ ) − _θ[e]m(s; τ_ ) dτ ! (105)

X

1 2 1/2

+ C(S, ini, 0 ) _ρ[dis](θ, τ, s))_ dτ _._
_R_ _L[sup]_ 0 R[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e] !

Z Z

The second term in this bound can be treated using the law of large numbers. We focus on controlling
the first term.


-----

Step 1: Estimating _M1_ _Mm=1_ 10 _θm(s; t)_ _θm(s; t)_ 2 dt. Defining

_−_ [e]

P R

∆t,m(s) = θm(s; t) _θm(s; t),_
_−_ [e]

we note that ∆t,m(0) = 0. By taking the difference of (83) and (85) and multiplying both sides by
_|_ _|_
∆t,m(s), we obtain


d ∆t,m(s)
_|_ _|[2]_

ds

= 2 ∆t,m(s), Ex _µ_ _∂θf_ (Zs(t; x), _θm)ps(t; x)_ _∂θf_ (Zs[dis][(][t][;][ x][)][, θ][m][)][p]s[dis][(][t][;][ x][)]
_−_ _∼_ _−_
D  E

= 2 ∆t,m(s), Ex _µ_ _∂θf_ (Zs(t; x),[e]θm)ps(t; x) _∂θf_ (Zs(t; x), _θm)p[dis]s_ [(][t][;][ x][)]
_−_ -  _∼_ _−_ +

 

(I)

[e] [e]

| {z }

2 ∆t,m(s), Ex _µ_ _∂θf_ (Zs(t; x), _θm)p[dis]s_ [(][t][;][ x][)][ −] _[∂][θ][f]_ [(][Z]s[dis][(][t][;][ x][)][, θ][m][)][p]s[dis][(][t][;][ x][)]
_−_ -  _∼_

 

(II)

[e]

For (I), we have from the bounds of| _Zs in (18), respectively, that{z_

(I) _C(S,_ ini, 0 ) _θm_ + 1 Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)] _,_
_|_ _| ≤_ _R_ _L[sup]_ _|[e]_ _|_ _∼_ _−_

which can be controlled using Lemma F.2 (95). For  (II), we have [
]


(106)


(II) _C(S,_ ini, 0 )Ex _µ_ _∂θf_ (Zs(t; x), _θm)_ _∂θf_ (Zs[dis][(][t][;][ x][)][, θ][m][)]
_|_ _| ≤_ _R_ _L[sup]_ _∼_ _−_
 

_≤C(S, Rini, L0[sup])_ Ex∼µ _Zs(t; x) −_ _Zs[e][dis][(][t][;][ x][)]_ + |θ[e]m − _θm|_ _,_

where the first term can be controlled using Lemma F.1h  (92). In both estimates, we used the propertyi

[
]

of f in Assumption 4.1 and bounds on Z, p, θm,[1], and _θm,[1]. By substituting these estimates into_
(106), we obtain

d ∆t,m(s) [e]
_|_ ds _|[2]_ _≤_ _C(S, Rini, L0[sup])|∆t,m(s)|[2]_

+ C(S, ini, 0 ) ∆t,m(s) Ex _µ_ _Zs(t; x)_ _Zs[dis][(][t][;][ x][)]_
_R_ _L[sup]_ _|_ _|_ _∼_ _−_

+ C(S, ini, 0 ) ∆t,m(s) _θm_  + 1 Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)] _,_
_R_ _L[sup]_ _|_ _|_ _|[e]_ _|_ _∼_ _−[
]_

which implies that   

[
]


d ∆t,m(s)
_|_ _|[2]_

ds


_m=1_


_C(S,_ ini, 0 )
_≤_ _R_ _L[sup]_

+ C(S, ini, 0
_R_ _L[sup]_

+ C(S, ini, 0
_R_ _L[sup]_


∆t,m(s)
_|_ _|[2]_
_m=1_

X


1

+ C(S, ini, 0 ) ∆t,m(s) Ex _µ_ _Zs(t; x)_ _Zs[dis][(][t][;][ x][)]_
_R_ _L[sup]_ _M_ _m=1_ _|_ _|!_ _∼_ _−_

XM  [
]

1

+ C(S, ini, 0 ) ∆t,m(s) _θm_ + 1 Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)]
_R_ _L[sup]_ _M_ _|_ _|_ _|[e]_ _|_ _∼_ _−_

_m=1_

_MX_  [!]  [
]

1

_C(S,_ ini, 0 ) ∆t,m(s) + C(S, ini, 0 )Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)]
_≤_ _R_ _L[sup]_ _M_ _m=1_ _|_ _|[2]!_ _R_ _L[sup]_ _∼_ _−_

X 

[2][]

+ C(S, ini, 0 )Ex _µ_ _Zs(t; x)_ _Zs[dis][(][t][;][ x][)]_ _,_
_R_ _L[sup]_ _∼_ _−_
(107)


[2][]


-----

where in the last inequality we use Hölder’s inequality

_M_

1

∆t,m(s) _θm_ + 1 Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)]

_M_ _|_ _|_ _|[e]_ _|_ _∼_ _−_

_m=1_

X  1/2 [!]  1/2

_M_ _M_ [
]

1 1 2

∆t,m(s) _θm_ + 1 Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)]

_≤_ _M_ _m=1_ _|_ _|[2]!_ _M_ _m=1_ _|[e]_ _|_ ! _∼_ _−_

X X  1/2  

_M_

1

_C(S,_ ini, 0 ) ∆t,m(s) Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)]
_≤_ _R_ _L[sup]_ _M_ _m=1_ _|_ _|[2]!_ _∼_ _−_

XM  [
]

1

_C(S,_ ini, 0 ) ∆t,m(s) + Ex _µ_ _ps(t; x)_ _p[dis]s_ [(][t][;][ x][)]
_≤_ _R_ _L[sup]_ " _M_ _m=1_ _|_ _|[2]!_ _∼_ _−_

X 

[2][#]

Noting the estimate in Lemma F.1-F.2, we obtain


_M_ 1
_m=1_ 0

_[|][∆][t,m][(][s][)][|][2][ d][t]_
ds
P R


d _M[1]_


_M_ 1

1

_C(S, Rini, L0[sup])_ _M_ _m=1_ Z0

X

1

+ C(S, Rini, L0[sup])Ex∼µ 0

Z

1

+ C(S, Rini, L0[sup])Ex∼µ 0

Z


∆t,m(s) dt
_|_ _|[2]_


_C(S,_ ini, 0
_≤_ _R_ _L[sup]_


+ C(S, Rini, L0[sup])Ex∼µ 0 R[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][−]ρ[e][dis](θ, τ, s))

Z Z

1

+ C(S, Rini, L0[sup])Ex∼µ 0 R[k][ ∂][z][f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][−]ρ[e][dis](θ, τ, s))

Z Z

which implies, using Grönwall’s inequality, that


dτ

!

2
dτ


1

0

Z


∆t,m(s) dt
_|_ _|[2]_


_m=1_


_m=1_

_S_ 1 2

_≤_ _C(S, Rini, L0[sup])Ex∼µ_ 0 0 R[k][ f][ (][Z][s][(][t][;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] _ρ[e][dis](θ, τ, s))_ dτ ds!

Z Z Z

_S_ 1 2

+ C(S, Rini, L0[sup])Ex∼µ 0 0 R[k][ ∂][z][f][ (][Z][s][(][t][;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] _ρ[e][dis](θ, τ, s))_ dτ ds!

Z Z Z

(108)
where we use ∆t,m(0) = 0.
_|_ _|_


_≤_ _C(S, Rini, L0[sup])Ex∼µ_


-----

Step 2: Completing the proof. By substituting (108) into (92), noticing _s_ _C(_ ini, S), we obtain
_R_ _≤_ _R_


_Zs(t; x) −_ _Zs[dis][(][t][;][ x][)]_

1/2

_S_ 1 2

 

_≤C(S, Rini, L0[sup])_ Ex∼µ Z0 Z0 ZR[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] _ρ[e][dis](θ, τ, s))_ dτ ds!

 
 (I) 
 
 

| {z }

_S_ 1 2



+ C(S, Rini, L0[sup]) Ex∼µ Z0 Z0 ZR[k][ ∂][z][f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] _ρ[e][dis](θ, τ, s))_ dτ ds!


 (II)

 1/2

| {z }

1 2

+ C(S, ini, 0 )  _ρ[dis](θ, τ, s))_ dτ  _,_
_R_ _L[sup]_ 0 R[k][ ∂][z][f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e]

Z Z 
 (III) 
 
  (109)

| {z }

All three terms in (109) can be controlled in expectation. Here we take the expectation with respect
to the randomness initial drawing of {θm(0; t)}m[M]=1[. For (I), we have]

_S_ 1 2

E(I) = E Ex _µ_ _ρ[dis](θ, τ, s))_ dτ ds

_∼_ 0 0 R[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e] !!

Z Z Z

_S_ 1 2[!]

= Ex _µ_ E _ρ[dis](θ, τ, s))_ dτ ds
_∼_ 0 0 R[k][ f][ (][Z][s][(][τ] [;][ x][)][, θ][) d(][ρ][(][θ, τ, s][)][ −] [e] !

Z Z Z

_S_ 1

ini [)]

Ex _µ_

_≤_ _[C][(][S,][ R]M[ini][,][ L][sup]_ _∼_ 0 0 R[k][ |][f][ (][Z][s][(][τ] [;][ x][)][, θ][)][ |][2][ d][ρ][(][θ, τ, s][) d][τ][ d][s]!

Z Z Z

ini [)]

_,_

_≤_ _[C][(][S,][ R]M[ini][,][ L][sup]_


1/2


where we use _θm(s; t) ∼_ _ρ(θ, t, s) in the first inequality. In second inequality, if k1 < k, we use first_
inequality of Assumption 4.1 (15) with _θ[1]_ _≤_ _C(S, Rini, Lini[sup][)][ and][ |][Z][s][| ≤]_ _[C][(][S,][ R][ini][,][ L]ini[sup][)][. If]_
_k1 = k, we use[e] (12) and |θ| =_ _θ[1]_ _≤_ _C(S, Rini, Lini[sup][)][ and][ |][Z][s][| ≤]_ _[C][(][S,][ R][ini][,][ L]ini[sup][)][ in the second]_
inequality. By similar reasoning, we obtain

ini [)] ini [)]
E(II) _,_ E(III) _._
_≤_ _[C][(][S,][ R]M[ini][,][ L][sup]_ _≤_ _[C][(][S,][ R]M[ini][,][ L][sup]_

From Markov’s inequality, these bounds imply that when M > _[C][(][R][ini]ϵ[2][,S,]η_ _[L]ini[sup][)]_, we have

P (I) < ϵ[2] _∩_ (II) < ϵ[2] _∩_ (III) < ϵ[2][ 
] _> 1 −_ _η/2 ._ (110)

Finally, using Definition E.1 (78), when  _M >_ [2][C]η [3] [,] 

P ( 0 _C4)_ 1 _η/2 ._ (111)
_L[sup]_ _≤_ _≥_ _−_

By substituting (110) and (111) into (109), we see that there exists a constant C( ini, C3, C4, S)
_R_
such that for any ϵ, η > 0, when M > _[C][(][R][ini]ϵ[,C][2]η[3][,C][4][,S][)]_ we obtain that

P _Zs(1; x) −_ _Zs[dis][(1;][ x][)]_ _< ϵ_ _> 1 −_ _η ._

By using this result and (111) in conjunction with (104), we complete the proof.

 



-----

G CONVERGENCE TO THE CONTINUOUS LIMIT

This section is dedicated to the continuous limit and, in particular, the proof of Theorem E.2.

G.1 STABILITY WITH DISCRETIZATION

Before proving Theorem E.2, and similarly to Appendix F.1, we first consider the stability of Z and p
under discretization. Defining the path of parameters Θ(t) = {θm(t)}m[M]=1 [and the set of parameters]
ΘL,M = {θl,m}l[L]=0[−][1],m[,M]=1[, we have the following lemma.]

**Lemma G.1 Suppose that Assumption 4.1 holds and that x is in the support of µ. Denoting**


_M_

_θl,m_ _,_ [1]
_|_ _|[2]_ _M_
_i=1_

X


_L[sup]_ = 0≤supt≤1,l


_θm(t)_
_|_ _|[2]_
_i=1_

X


(112)


= inf _θl,m_ _m=1,l=1_ _m=1_ [1][|][ < r][}][,][ ∀][t][ ∈] [[0][,][ 1]] _,_
_R_ _r>0_ _{_ _}[M,L]_ _[∪{][θ][m][(][t][)][}][M]_ _[⊂{][θ][||][θ]_

_there exists a constantn C(R, L[sup]) depending only on R, L[sup]_ _such that for any 0 ≤ol ≤_ _L −_ 1, we
_have_

sup _ZΘ(t; x)_ _ZΘL,M (l; x)_ _,_ _ZΘ(t; x)_ _ZΘL,M (l + 1; x)_
_l_ _−_ _−_

_L_ _L_

_[≤][t][≤]_ _[l][+1]_ 

_L_ 1 _M_ _l+1_ 1/2 (113)

1 _−_ _L_

_C(_ _,_ ) _θl,m_ _θm(τ_ ) dτ + _[C][(][R][,][ L][sup][)]_ _,_
_≤_ _R_ _L[sup]_ _M_ _l=0_ _m=1_ Z _Ll_ _|_ _−_ _|[2]_ ! _L_

X X

_and_
sup _pΘ(t; x)_ _pΘL,M (l; x)_
_l_ _−_

_L_ _L_

_[≤][t][≤]_ _[l][+1]_

_L_ 1 _M_ _l+1_ 1/2 (114)

1 _−_ _L_

_C(_ _,_ ) _θl,m_ _θm(τ_ ) dτ + _[C][(][R][,][ L][sup][)]_ _._
_≤_ _R_ _L[sup]_ _M_ _l=0_ _m=1_ Z _Ll_ _|_ _−_ _|[2]_ ! _L_

X X

**Proof Define**
_Z(t; x) = ZΘ(t; x),_ _p(t; x) = pΘ(t; x),_
and


_L−1_ _L−1_

_Z(t; x) =_ _ZΘL,M (l; x)1 lL_ _L_ _[,]_ _p(t; x) =_ _pΘL,M (l; x)1 lL_ _[<t][≤]_ _[l][+1]L_ _[,]_ (115)

_l=0_ _[≤][t<][ l][+1]_ _l=0_

X X

with e e
_Z(1; x) = ZΘL,M (L; x),_ _p(0; x) = pΘL,M (0; x) ._ (116)

Using (1), (72), Assumption 4.1, and Lemma D.2 (73) and (75), we obtain for all l = 0, 1, . . ., L 1
that e e _−_
_ZΘL,M (l + 1; x)_ _ZΘL,M (l; x)_ _< [C][(][L][sup][)]_ _,_
_−_ _L_

(117)

_pΘL,M (l + 1; x)_ _pΘL,M (l; x)_ _< [C][(][L][sup][)]_ _._
_−_ _L_

Now define ∆t by
∆t = Z(t; x) − _Z[e](t; x) ._
For t ∈ [ _L[l]_ _[,][ l][+1]L_ []][, we have from (4) that]

_M_ _t_

∆t ∆ _l_ + [1] _f_ (Z(τ ; x), θm(τ )) dτ
_|_ _| ≤_ _L_ _M_ _l_ _|_ _|_

_m=1_ Z _L_

X _M_ _l+1_

_L_ (118)

∆ _l_ + _[C][(][L][sup][)]_ ( _θm(τ_ ) + 1) dτ
_≤_ _L_ _M_ _l_ _|_ _|[2]_

_m=1_ Z _L_

X

∆ _l_ + _[C][(][L][sup][)]_ _,_
_≤_ _L_ _L_


-----

where we use (12), (18), and (112) in the last two inequalities. From (1) and (4), we obtain further
that

_M_ _l+1_

1 _L_

∆ _l+1_ = ∆ _l_ + _f_ (Z(τ ; x), θm(τ )) _f_ _Z(τ_ ; x), θl,m dτ

_L_ _L_ _M_ _l_ _−_

1 _mXM=1_ Z _Ll+1L_  e 

∆ _l_ + _f_ (Z(τ ; x), θm(τ )) _f (Z(τ_ ; x), θl,m) dτ
_≤_ _L_ _M_ _l_ _−_

_m=1_ Z _L_

_M_ Xl+1

1 _L_

+ _f_ (Z(τ ; x), θl,m) _f_ _Z(τ_ ; x), θl,m dτ

_M_ _l_ _−_

(I) _mX=1_ Z _L_ 1 _M_ _l+1L_  e 
∆ _l_ + C( ) ( _θm(τ_ ) + _θl,m_ + 1) _θm(τ_ ) _θl,m_ dτ
_≤_ _L_ _L[sup]_ _M_ _m=1_ Z _Ll_ _|_ _|_ _|_ _|_ _|_ _−_ _|_ !

X

_M_

1

+ C( ) ∆ξ ( _θl,m_ + 1)
_L[sup]_ _|_ _|_ _ML_ _m=1_ _|_ _|[2]_ !

X _M_ _l+1_

(II) 1 _L_
1 + _[C][(][L][sup][)]_ ∆ _l_ + C( ) ( _θm(τ_ ) + _θl,m_ + 1) _θm(τ_ ) _θl,m_ dτ
_≤_ _L_ _L_ _L[sup]_ _M_ _l_ _|_ _|_ _|_ _|_ _|_ _−_ _|_
  _mX=1_ Z _L_

+ _[C][(][L][sup][)]_ _,_

_L[2]_

where ξ ∈ [ _L[l]_ _[,][ l][+1]L_ []][, and we used the mean-value theorem with][ (13)][,][ (18)][,][ (73)][ in (I) and][ (112)][,][ (118)]

in (II). By applying this bound iteratively, we obtain


_l−1_

_j=0_

X


_j+1_


_l_ 1 _M_ _j+1_
_−_ _L_

∆ _l_ _C(_ ) ∆0 +C( ) ( _θm(τ_ ) + _θl,m_ + 1) _θm(τ_ ) _θl,m_ dτ + _[C][(][L][sup][)]_

_L_ _≤_ _L[sup]_ _|_ _|_ _L[sup]_  _M_ _j=0_ _m=1_ Z _Lj_ _|_ _|_ _|_ _|_ _|_ _−_ _|_  _L_

X X

 [1] 

where ∆0 = 0. By combining this bound with (118) and using Hölder’s inequality with (112), we
_|_ _|_
obtain that


_L_ 1 _M_ _l+1_ 1/2

1 _−_ _L_

∆t _C(_ ) _θl,m_ _θm(τ_ ) dτ + _[C][(][L][sup][)]_ _._
_|_ _| ≤_ _L[sup]_ _M_ _l=0_ _m=1_ Z _Ll_ _|_ _−_ _|[2]_ ! _L_

X X

By combining (119) with (117), we prove (113).

To prove (114), we define
∆p(t; x) = p(t; x) _p(t; x) ._
_−_

Similarly to (34), we obtain

e

∆p(1; x) _C(_ ) _Z(1; x)_ _Z(1; x)_
_|_ _| ≤_ _L[sup]_ _−_

_L_ 1 _M_ _l+1_ 1/2

[e] 1 _−_ _L_

_C(_ ) _θl,m_ _θm(τ_ ) dτ + _[C][(][L][sup][)]_ _._
_≤_ _L[sup]_ _M_ _l=0_ _m=1_ Z _Ll_ _|_ _−_ _|[2]_ ! _L_

X X

_l_
For t _L_ _[,][ l][+1]L_ and using (11), we obtain that
_∈_
  _l + 1_ _M_ _l+1L_

∆p (t; x) ; x [+ 1] _∂zf_ (Z(τ ; x), θm(τ )) _p(τ_ ; x) dτ
_|_ _| ≤_  _L_  _M_ _m=1_ Zt _|_ _| |_ _|_

X _M_ _l+1_

[∆][p] _l + 1_ _L_

_≤_ _L_ ; x [+][ C][(][L]M[sup][)] _l_ (|θm(τ )|[2] + 1) dτ

  _m=1_ Z _L_

X

_l + 1_

[∆][p] ; x [+][ C][(][L][sup][)] _,_

_≤_ _L_ _L_

 

[∆][p]


(119)

(120)

(121)


-----

where we use (13), (18), and (31) in the second inequality and the definition of L[sup] from (112) in
the last line.

From (11), we obtain that


_l_ _l + 1_
_p[⊤]_ = p[⊤]

_L_ [;][ x] _L_

  

while (72) implies that


_l+1_


; x + [1]



_p[⊤]_ (τ ; x) ∂zf (Z(τ ; x), θm(τ )) dτ,


_m=1_ _L_

while (72) implies that

_M_ _l+1_

_L_

_l_ _l + 1_ _l + 1_
_p[⊤]_ _L_ [;][ x] = _p[⊤]_ _L_ ; x + M[1] _l_ _p[⊤]_ _L_ ; x _∂zf_ (ZΘL,M (l; x), θl,m) dτ,
    _m=1_ Z _L_  

X

wheree _p is defined in (115), (116). e_ e

By bounding differences of these two expressions, we have

e

_l_ _l + 1_

; x

_L_ [;][ x] _L_

  _[≤]_  

_M_ _l+1_

_L_ _l + 1_

+ [∆][p]M[1] _l_ [∆][p] _p[⊤]_ _L_ ; x _∂zf_ (Z(τ ; x), θm(τ )) [d][τ]

_m=1_ Z _L_  

X

(I)

_[p][⊤]_ [(][τ] [;][ x][)][ ∂][z][f] [(][Z][(][τ] [;][ x][)][, θ][m][(][τ] [)) d][τ][ −] [e]

_M_ _l+1_

_L_

| _l + 1_ {z _l + 1_ }

+ M[1] _l_ _p[⊤]_ _L_ ; x _∂zf_ (Z(τ ; x), θm(τ )) − _p[⊤]_ _L_ ; x _∂zf_ (ZΘL,M (l; x), θm(τ )) [d][τ]

_m=1_ Z _L_    

X

[e] (II) e

_M_ _l+1_

_L_

| _l + 1_ {z _l + 1_ }

+ M[1] _l_ _p[⊤]_ _L_ ; x _∂zf_ (ZΘL,M (l; x), θm(τ )) − _p[⊤]_ _L_ ; x _∂zf_ (ZΘL,M (l; x), θl,m) [d][τ]

_m=1_ Z _L_    

X

[e] (III) e

(122)

| {z }

We bound (I), (II), and (III) in turn.


(I): Using (13), (18), and (112), we obtain that

(I)
_≤_ _[C][(][L]L[sup][)]_


∆p(t; x)
_|_ _|_


(II): Using Assumption 4.1 (14) together with (18), (73), (75), and (112), we obtain that


_l+1_


(II)
_≤_ _[C][(][R]M[,][ L][sup][)]_

_≤_ _[C][(][R][,]L[ L][sup][)]_


_|Z(τ_ ; x) − _ZΘL,M (l; x)| dτ_


_m=1_


1/2

+ [1]

_L_

!


_l[′]_ +1

_L_

_θl[′],m_ _θm(τ_ ) dτ
_lL[′]_ _|_ _−_ _|[2]_


_L−1_

_l[′]=0_

X


_,_






_m=1_


where we make use of (113) in the last inequality.


(III): Using Assumption 4.1 item 3 together with (73), (75), and (112), we obtain that


_l+1_


(III)
_≤_ _[C][(][R]M[,][ L][sup][)]_


_θm(τ_ ) _θl,m_ dτ
_|_ _−_ _|_

1/2

_M_

_θm(τ_ ) _θl,m_
_m=1_ _|_ _−_ _|[2]!_

X


_m=1_


_l+1_

_L_

_C(_ _,_ )
_≤_ _R_ _L[sup]_ _l_
Z


dτ


-----

By substituting these three bounds and (121) into (122), we obtain

_l_ _l + 1_

1 + _[C][(][R][,][ L][sup][)]_ ; x
_L_ [;][ x] _L_ _L_

  _[≤]_    

_L_ 1 _M_ _l[′]_ +1

[∆][p] 1[∆][p]− _L_

+ _[C][(][R][,][ L][sup][)]_ _θl′,m_ _θm(τ_ ) dτ

_L_  _M_ _l[′]=0_ _m=1_ Z _lL[′]_ _|_ _−_ _|[2]_

X X




1/2
!


+ [1]


1/2

_M_

_θm(τ_ ) _θl,m_
_m=1_ _|_ _−_ _|[2]!_

X


_l+1_

_L_

+ C( _,_ )
_R_ _L[sup]_ _l_
Z


dτ + _[C][(][R][,][ L][sup][)]_

_L[2]_


By applying this bound iteratively, and using (120) and (121), we obtain

_L_ 1 _M_ _l+1_ 1/2

1 _−_ _L_

∆p(t; x) _C(_ _,_ ) _θl,m_ _θm(τ_ ) dτ
_|_ _| ≤_ _R_ _L[sup]_  _M_ _l=0_ _m=1_ Z _Ll_ _|_ _−_ _|[2]_ !

X X



where we also use Hölder’s inequality to write

_L_ 1 _l+1_ _M_ 1/2
_−_ _L_ 1

_C(_ _,_ ) _θm(τ_ ) _θl,m_ dτ
_R_ _L[sup]_ _l=0_ Z _Ll_ _M_ _m=1_ _|_ _−_ _|[2]!_

X X

_L_ 1 _M_ _l+1_ 1/2

1 _−_ _L_

_C(_ _,_ ) _θl,m_ _θm(τ_ ) dτ
_≤_ _R_ _L[sup]_ _M_ _l=0_ _m=1_ Z _Ll_ _|_ _−_ _|[2]_ !

X X


+ [1]


_,_ (123)






We obtain (114) by combining (123) with (121).

G.2 PROOF OF THEOREM E.2

We denote by θm(s; t) the solution to (6) with initial {θm(0; t)}m[M]=1 [that are][ i.i.d][ drawn from]
_ρini(θ, t) . Further, θl,m(s) is a solution to (3) with initial value θl,m(0) = θm_ 0; _L[l]_ for 0 _l_

_≤_ _≤_
_L −_ 1 and 1 ≤ _i ≤_ _M_ . Define  



ini = sup
_L[dis][,][sup]_ 0≤t≤1


_θm(0; t)_
_|_ _|[2]_
_i=1_

X


and recall ini defined in (55). According to Definition E.1, ini is bounded with high probability.
_R_ _L[dis][,][sup]_
Then, we have the following lemma:

**Lemma G.2 For fixed S > 0 and any s** [0, S], there exists a constant C(S, ini, ini )
_∈_ _R_ _L[dis][,][sup]_
_depending only on S,_ ini, ini _such that_
_R_ _L[dis][,][sup]_


_θl,m(s)_ _C(S,_ ini, ini ),
_|_ _|[2]_ _≤_ _R_ _L[dis][,][sup]_
_m=1_

X


(124)


_θl,m(s)_ _m=1,l=1_ _θ_ _θ[1]_ _C(S,_ ini, ini ) _,_
_{_ _}[M,L]_ _[⊂]_ _|_ _| ≤_ _R_ _L[dis][,][sup]_
n o

_Furthermore, the ODE solution and pΘL,M_ (s) are bounded as follows, for any x in the support of µ
_and l = 0, 1, . . ., L −_ 1:
_ZΘL,M_ (s)(l + 1; x) _C_ _S,_ ini _,_ ini _,_ (125)
_≤_ _L[dis][,][sup]_ _R_
 

_and_
_pΘL,M_ (s)(l; x) _C_ _S,_ ini _,_ ini _._ (126)
_≤_ _L[dis][,][sup]_ _R_
 


-----

The proof is quite similar to that of Lemma F.3, so we omit the details.

We are now ready to prove Theorem E.2.

**Proof [Proof of Theorem E.2] From (99) and (124) we obtain for all t ∈** [0, 1], s ∈ [0, S],


_M_

_θl,m(s)_ _,_ [1]
_|_ _|[2]_ _M_
_m=1_

X


_C(S,_ ini, ini ),
_≤_ _R_ _L[dis][,][sup]_


_θm(s; t)_
_|_ _|[2]_
_m=1_

X


max


(127)


_θm(s; t)_ _m=1_ _m=1,l=1_ _θ_ _θ[1]_ _C(S,_ ini, ini ) _,_
_{_ _}[M]_ _[∪{][θ][l,m][(][s][)][}][M,L]_ _[⊂]_ _|_ _| ≤_ _R_ _L[dis][,][sup]_
n o

where C(S, ini, ini ) is a constant depending on S, ini, and ini .
_R_ _L[dis][,][sup]_ _R_ _L[dis][,][sup]_

From a similar derivation to (81), we obtain

_E(Θ(s;_ )) _E(ΘL,M_ (s)) _C(S,_ ini, ini ) _ZΘ(s)(1; x)_ _ZΘL,M_ (s)(L; x) _._ (128)
_|_ _·_ _−_ _| ≤_ _R_ _L[dis][,][sup]_ _−_

Thus, to prove the theorem, it suffices to prove that _ZΘL,M_ (s)(L; x) _ZΘ(s)(1; x)_ is small. For this
_−_
purpose, according to (113), we need to bound the quantity

_L_ 1 _M_ _l+1_

1 _−_ _L_

∆t,m(s) dt, (129)

_M_ _l_ _|_ _|[2]_

_l=0_ _m=1_ Z _L_

X X


where ∆t,m(s) = θl,m(s) _θm(s; t). The next part of the proof contains the required bound._
_−_

First, using (3) and (6), we obtain that

d ∆t,m(s)
_|_ _|[2]_

ds
= 2 ∆t,m(s), Ex _µ_ _∂θf_ (ZΘL,M (s)(l; x), θl,m(s))pΘL,M (s)(l; x) _∂θf_ (ZΘ(s)(t; x), θm(s; t))pΘ(s)(t; x)
_−_ _∼_ _−_



= 2 ∆t,m(s), Ex _µ_ _∂θf_ (ZΘL,M (s)(l; x), θl,m(s))pΘL,M (s)(l; x) _∂θf_ (ZΘ(s)(t; x), θm(s; t))pΘL,M (s)(l; x)
_−_ -  _∼_ _−_

 (I)

| {z

2 ∆t,m(s), Ex _µ_ _∂θf_ (ZΘ(s)(t; x), θm(s; t))pΘL,M (s)(l; x) _∂θf_ (ZΘ(s)(t; x), θm(s; t))pΘ(s)(t; x) _._
_−_ -  _∼_ _−_ +

 (II) 


(130)

| {z }

To bound (I), we use (126) to obtain

_|(I)| ≤_ _C(S, Rini, Lini[dis][,][sup])Ex∼µ_ _∂θf_ (ZΘL,M (s)(l; x), θl,m(s)) − _∂θf_ (ZΘ(s)(t; x), θm(s; t))

_≤_ _C(S, Rini, L[dis]ini_ _[,][sup])_ Ex∼µ _ZΘL,M_ (s)(l; x) − _ZΘ(s)(t; x)_ + |θl,m(s) − _θm(s; t)|_ _,_ [
]
(131)
  

where we use Assumption 4.1 (14), (100), (125), and (127) in the second inequality. To bound (II),
we use (13), (100), and (127) to obtain

_|(II)| ≤_ _C(S, Rini, Lini[dis][,][sup])(|θm(s; t)| + 1)Ex∼µ_ _pΘL,M_ (s)(l; x) − _pΘ(s)(t; x)_ _._ (132)


By substituting (132) and (131) into (130), we obtain [
]


d ∆t,m(s)
_|_ _|[2]_

ds
_C(S,_ ini, ini ) ∆t,m(s)
_≤_ _R_ _L[dis][,][sup]_ _|_ _|[2]_

+ C(S, Rini, L[dis]ini _[,][sup])|∆t,m(s)|Ex∼µ_ _ZΘL,M_ (s)(l; x) − _ZΘ(s)(t; x)_

+ C(S, Rini, L[dis]ini _[,][sup])|∆t,m(s)|(|θm(s; t)| + 1)Ex∼µ_ _pΘL,M_ (s)(l; x)[
] − _pΘ(s)(t; x)_



-----

Using Hölder’s inequality similar to (107), we obtain


_M_
_m=1_

_[|][∆][t,m][(][s][)][|][2]_

ds

P


d _M[1]_


1

_≤_ _C(S, Rini, Lini[dis][,][sup])_ _M_ _m=1_ _|∆t,m(s)|[2]!_

X

+ C(S, Rini, Lini[dis][,][sup])Ex∼µ _pΘL,M_ (s)(t; x) − _pΘ(s)(t; x)_


+ C(S, ini, ini )Ex _µ_ _ZΘL,M_ (s)(t; x) _ZΘ(s)(t; x)[2][]_
_R_ _L[dis][,][sup]_ _∼_ _−_


By substituting (113) and (114) into (133), we obtain [2][]


(133)

∆t,m(s) dt + [1]
_|_ _|[2]_ _L[2]_


_L_ 1 _M_ _[l][+1]L_
_l=0−_ _m=1_ _l_ ∆t,m(s) dt

_L_ _|_ _|[2]_

P P dRs _≤_ _C(S, Rini, Lini[dis][,][sup]_


d _M[1]_


_L−1_

_l=0_

X


_l+1_


_m=1_

_l+1_


which implies, from Grönwall’s inequality, that


_L−1_


_l+1_


_L−1_


_L_ 1 _M_ _L_ 1 _M_

1 _−_ _L_ 1 _−_ _L_

∆t,m(s) dt _C(S,_ ini, ini ) ∆t,m(0) dt + [1]

_M_ _l=0_ _m=1_ Z _Ll_ _|_ _|[2]_ _≤_ _R_ _L[dis][,][sup]_ _M_ _l=0_ _m=1_ Z _Ll_ _|_ _|[2]_ _L[2]_ !

X X X X

(134)
We have thus established the bound (129). We also have


_L−1_

_l=0_

X


_l+1_


_L−1_

_l=0_

X


_l+1_

_L_

_l_

_L_

_[θ][m]_

to obtain


∆t,m(0) dt = [1]
_|_ _|[2]_ _M_


0; _[l]_


dt . (135)


_θm(0; t)_
_−_


_m=1_


_m=1_


To complete the proof, we use (79) and take M _η_
_≥_ [8][C][3]


_L−1_ _M_

_l=0_ _m=1_

X X


_l+1_

_L_

∆t,m(0) dt
_|_ _|[2]_ _≤_ _[C]L[2][4]_


1
_≥_ _−_ _[η]8_ _[.]_


According to (78), when M > [8][C]η [3] [,]


P _Lini[dis][,][sup]_ _≤_ _C4_ _≥_ 1 − _[η]8_ _[.]_ (136)
 

By using these expressions to substitute _M1_ _Ll=0−1_ _Mm=1_ _Ll[l][+1]L_ _|∆t,m(0)|[2]_ and Lini[dis][,][sup] into (134),

we find that there exists a constant C _[′](C4,_ ini, S) depending on C4, ini, and S such that if
_RP_ P R _R_

_M ≥_ [8][C]η [3] _[,]_ _L ≥_ _[C]_ _[′][(][C][4][,][ R]ϵ_ [ini][, S][)] _,_


then we have


_L−1_ _M_

_l=0_ _m=1_

X X


_l+1_


1 (137)
_≥_ _−_ _[η]4_ _[.]_


∆t,m(s) dt _ϵ[2]_
_|_ _|[2]_ _≤_


Using (137), (136), and (127) to bound the right hand side of (113), we find that there exists another
constant C _[′′](C4,_ ini, S) depending on C4, ini, and S such that if
_R_ _R_

_M ≥_ [8][C]η [3] _[,]_ _L ≥_ _[C]_ _[′′][(][C][4][,]ϵ[ R][ini][, S][)]_ _,_

then we have
P _ZΘ(s)(1; x)_ _ZΘL,M_ (s)(L; x) _ϵ_ 1
_−_ _≤_ _≥_ _−_ _[η]2_ _[,]_

By using this result and (136) in conjunction with (128), we complete the proof. 



-----

H PROOF OF GLOBAL CONVERGENCE RESULT

Intuitively, if the equation (9) converges to a stationary point, denote by ρ∞, so that ∂sρ∞ = 0, then


_δE_
_θ_
_∇_ _δρ_


(θ, t) = 0, _ρ_ (θ, t)-a.e. θ R[k] _,_ a.e. t [0, 1] .
_ρ∞(·,·)_ _∞_ _∈_ _∈_


The rest of the analysis shows that E(ρ ) = 0 when this happens. However, it is not direct because
_∞_
the condition above only suggests the fact that _[δE]δρ_ _ρ∞(˙,)[˙]_ [is a piecewise constant function. We need]

a stronger result that shows this constant has to be zero. This is achieved by Proposition 6.1. To[|]
show this proposition, we follow the proof in (Lu et al., 2020) that explores the expressive power
of f (x, θ), particularly the universal kernel property of Assumption 4.1. It is this proposition that
identifies stationary points with the global minimizer.

We should mention that the zero loss was demonstrated by Chizat & Bach (2018) for the 2-layer
problem where the stability equates to zero-loss due to convexity. The extension to the multi-layer
case is more difficult since convexity is not present.

H.1 PROOF OF PROPOSITION 6.1

We first prove a lower bound for pρ in the following lemma.

**Lemma H.1 Suppose that ρ ∈C([0, 1]; P** [2]) and that pρ is a solution to (11). Denoting

1
_Lρ =_ 0 R[k][ |][θ][|][2][dρ][(][θ, t][)][,]
Z Z

_then for any t ∈_ [0, 1] we have that

Ex _µ_ _pρ(t; x)_ _Q(_ _ρ)E(ρ),_ (138)
_∼_ _|_ _|[2][
]_ _≥_ _L_

_where Q : R+ →_ R+ is a decreasing function.

**Proof Recall that the initial condition for pρ in (11) is:**

_pρ(1; x) = (g(Zρ(1; x))_ _y(x))_ _g(Zρ(1; x)),_
_−_ _∇_

so from Assumption 4.1, we have

2
Ex _µ_ _pρ(1; x)_ inf _E(ρ) ._
_∼_ _|_ _|[2][
]_ _≥_ _x_ R[d][ |∇][g][(][x][)][|]
 _∈_ 


Further, since the equation is linear, we have


_∂p[⊤]ρ_

_∂t_ = −p[⊤]ρ


R[k][ ∂][z][f] [(][Z][ρ][, θ][)][dρ][(][θ, t][)][ .]


According to equation (13) in Assumption 4.1, we obtain


R[k][ ∂][z][f] [(][Z][ρ][(][t][;][ x][)][, θ][) d][ρ][1][(][θ, t][)] _[≤]_ _[C][(][Z][ρ][(][t][;][ x][))]_ ZR[k] [(][|][θ][|][2][ + 1) d][ρ][(][θ, t][)]

_C(_ _ρ)_
_≤_ _L_

R[k] [(][|][θ][|][2][ + 1) d][ρ][1][(][θ, t][)][,]

Z


where we use (18) in the second inequality. By combining the last two bounds in the usual way, we
obtain
d _pρ(t; x)_
_|_ _|[2]_ 2C( _ρ)_ _pρ(t; x)_ _,_

dt _≤_ _L_ _|_ _|[2]_
 ZR[k] [(][|][θ][|][2][ + 1)][dρ][(][θ, t][)]

By solving the equation, we have


1
_pρ(t; x)_ _pρ(1; x)_ exp 2C( _ρ)_
_|_ _|[2]_ _≥|_ _|[2]_ _−_ _L_ _t_
 Z


_C(_ _ρ)_ _pρ(1; x)_ _._
_≥_ _L_ _|_ _|[2]_
R[k] [(][|][θ][|][2][ + 1)][dρ][(][θ, t][)]


-----

The proof is finalized by taking expectation on both sides, and note that monotonicity comes from the
format of the exponential term.

We are now ready to prove Proposition 6.1.

**Proof [Proof of Proposition 6.1] Denote**

1
_Lρ =_ 0 R[k][ |][θ][|][2][ d][ρ][(][θ, t][) d][t .]
Z Z

According to existence and uniqueness of the solution to (7), for any t ∈ [0, 1], we can construct a
map Zt such that
_Zt(x) = Zρ (t; x) ._
Since the trajectory can be computed backwards in time, _t_ is well defined. Further, we denote
_Z_ _[−][1]_
_µ[∗]t_ [= (][Z][t][)][♯][µ][ to be the pushforward of][ µ][ under map][ Z][t] [and let]

_p[∗](t; x) = pρ_ _t;_ _t_ (x) _._
_Z_ _[−][1]_

By Assumption 4.1 and classical ODE theory, Zt and _Zt[−][1]_ are both continuous maps in
 _x, and so are_
_pρ(t; x) and p[∗](t; x). With the change of variables, we have for all t_ [0, 1] that
_∈_

_δE(ρ)_

(θ, t) = _ρ_ [(][t][;][ x][)][f] [(][Z][t][(][x][)][, θ][) d][µ][ =] _t_ _[.]_ (139)
_δρ_ R[d][ p][⊤] R[d] [(][p][∗][(][t][;][ x][))][⊤][f] [(][x, θ][) d][µ][∗]
Z Z

For a fixed t0 [0, 1], we have boundedness of the Jacobian from Lemma C.1, meaning that
_∈_

dµ[∗]t0 [(][Z]t[−][1](x))

sup _C(_ _ρ)._
_x_ supp(µ) dµ(x) _≤_ _L_
_∈_ 2


As a consequence, µ[∗]t0 [(][x][)][ has a compact support since][ µ][(][x][)][ does. We denote the size of the support]
by R[∗], defined to be a real number such that supp _µ[∗]t0_ [(][x][)] _⊂{x : |x| < R[∗]}._

_δE(ρ)_
We now derive a general formula for _δρ_ [(][θ, t][) d] _[ν][. Recalling (139), we have]
_

_δE(ρ)_ R

(θ, t0) dν(θ) = dµ[∗]t0 [(][x][)]

ZR[k] _δρ_ ZR[d] [(][p][∗][(][t][0][;][ x][))][⊤] ZR[k][ f] [(][x, θ][) d][ν][(][θ][)]

= dµt0 (x) (140)
ZR[d] [(][p][∗][(][t][0][;][ x][))][⊤] ZR[k][ f] [(][x, θ][) d][ν][(][θ][) +][ p][∗][(][x, t][0][)]

_−_ _t0_ [(][x][)][ .]

R[d] [(][p][∗][(][t][0][;][ x][))][⊤][p][∗][(][x, t][0][) d][µ][∗]

Z

Noticing that according to Lemma H.1, if E(ρ) ̸= 0, the second term above is strictly negative (less
than _Q(_ _ρ)E(ρ)), the goal then is to find ν for which_ dν = 0 that makes the first term small, so
_−_ _L_
that the right-hand side in (140) is negative. Defining the continuous function h to be
R


_h(x) = p[∗]_ (t0; x) +

R[k][ f] [(][x, θ][) d][ρ][(][θ, t][0][)][,]

Z

then according to Assumption 4.1, for arbitrarily small ϵ, there is a ˆν so that _νˆ = 0 and_
R
ZR[k][ f] [(][x, θ][) dˆ]ν(θ) _L[∞]|x|<R[∗]_ _≤_ _ϵ ._

Setting ν = ρ _νˆ and substituting into the first term of (140), we obtain_
_−_ _[h][(][x][)][ −]_

dµt0 (x)

ZR[d] [(][p][∗][(][t][0][;][ x][))][⊤] ZR[k][ f] [(][x, θ][) d][ν][(][θ][) +][ p][∗][(][x, t][0][)]


_≤_ _ν(θ)_ [d][µ]t[∗]0 [(][x][)]

R[d][ |][p][∗][(][t][0][;][ x][)][|] R[k][ f] [(][x, θ][) dˆ]

Z Z

_≤∥p[∗](t0; x)∥L∞|x|<R[h][(][∗][x][)][ −]_ ZR[k][ f] [(][x, θ][) dˆ]ν(θ) _L[∞]|x|<R[∗]_

_[h][(][x][)][ −]_


(141)


-----

By choosing ϵ small enough that (141) is less than 12 _[Q][(][L][ρ][)][E][(][ρ][)][, we have from][ (140)]_
_δE(ρ)_

_δρ_ [(][θ, t][0][) d][ν][(][θ][)][ <][ 0][, completing the proof.]

R

H.2 2-HOMOGENEOUS CASE: PROOF OF THEOREM 6.1

We first give an example of 2-homogeneous activation function that satisfy Assumption 4.1, and 6.1.

**Remark H.1 A function that satisfies Assumption 4.1 and the 2-homogeneous property of Assump-**
_tion 6.1 is f_ (x, θ) = f (x, θ[1], θ[2]) = σ(θ[1]x + θ[2]) exp(−|x|[2]), where θ[1] ∈ R[d][×][d], θ[2] ∈ R[d],
_and σ(x) = | max{x, 0}|[2]_ _applied componentwise._

Before proving the Theorem 6.1, we first introduce the following lemma, which shows that the
separation property is preserved in the training process. Our proof of this result is adapted from
(Chizat & Bach, 2018).

**Lemma H.2 Let Assumptions 4.1 and 4.2 hold, and suppose that ρini(θ, t) is admissible with compact**
_support. Let ρ(θ, t, s)_ ([0, ); ([0, 1]; )) solve (9). If there exists t0 [0, 1], so that the
_initial condition ρini(θ, t ∈C0) separates the spheres ∞_ _C_ _P_ [2] raS[k][−][1] _and rbS[k][−][1]_ _for some 0 ∈ < ra < rb, then for_
_any s0 ∈_ [0, ∞), ρ(θ, t0, s0) separates the spheres ra[′] [S][k][−][1][ and][ r]b[′] [S][k][−][1][ for some][ 0][ < r]a[′] _[< r]b[′]_ _[.]_

**Proof For every fixed 0 < s0 < S < ∞, we note that the particle representation θρ(s; t0) of**
_ρ(θ, t0, s) updates the following equation:_

dθρ(s; t0) _δE(ρ(s))_

= _θ_ (θρ(s; t0), t0), _s_ (0, S)
ds _−∇_ _δρ_ _∈_ (142)


 _θρ(0; t0) = θ ._


Define the map _s(θ) to be the solution map that solves the equation above for given initial condition_
_P_
_θ up to time s. Our proof amounts to showing that this map preserves the separation property._
According to (Chizat & Bach, 2018, Proposition C.11), we need only show that the inverse map of
_s(θ) is stable near 0 for any fixed 0 < s < S. That is, for any ϵ > 0, we need to identify η > 0_
_P_
such that
_s_ [(][θ][)][ ⊂B][ϵ] [(0)][,] _θ_ _η (0),_ (143)
_P_ _[−][1]_ _∀_ _∈B_
where _η(0) is the k-dimensional ball around original 0 with radius η._
_B_

Since f is 2-homogeneous in θ, we have that _∂θf_ (z, 0) = 0 for all z. Thus, from (10),
_|_ _|_
_δE(ρ(s))_

(0, t0) [= 0][.]
_δρ_

Using estimate (49) from Lemma C.4, we obtain

_[∇][θ]_

_δE(ρ(s))_

(θ, t0) _S_ [)][|][θ][|][,]
_δρ_

_[≤]_ _[C][(][L][sup]_

where LS[sup] = sup0≤s≤S,t∈[0,1][∇]R[θ][k][ |][θ][|][2][ d][ρ][(][θ, t, s][) d][t][. This upper bound on the velocity implies in]

particular that

R

_|Ps[−][1][(][θ][)][| ≤|][θ][|][ exp(][C][(][L]S[sup][)][s][)][,]_
which establishes (143) when we choose η to satisfy η < ϵ exp(−C(LS[sup][)][s][)][, concluding the proof.]

We are now ready to prove Theorem 6.1.

**Proof [Proof of Theorem 6.1] Since ρ(θ, t, s) converges to ρ** (θ, t) in ([0, 1]; ), we have for any
_∞_ _C_ _P_ [2]
_t0 that_

sup (144)
_s≥0_ ZR[k][ |][θ][|][2][ d][ρ][(][θ, t][0][, s][)][ <][ ∞] _[.]_

According to Proposition 6.1, it suffices to prove that


_δE(ρ_ )
_∞_ (θ, t0) = 0, _θ_ R[k] _._ (145)

_δρ_ _∀_ _∈_


-----

We use a contradiction argument: We will assume that (145) is not satisfied and show that

R[k][ |][θ][|][2][ d][ρ][(][θ, t][0][, s][)][ blows up to infinity as][ s][ →∞][, contradicting][ (144)][. In particular, we will]
use homogeneity to construct a set in which the second moment blows up.

R

Define the functions h and hs as follows:
_∞_

_h∞(θ) =_ _[δE]δρ[(][ρ][∞][)]_ (θ, t0), _hs(θ) =_ _[δE][(]δρ[ρ][(][s][))]_ (θ, t0) .


Recall from (10) that


_δE(ρ)_

(θ, t0) = Eµ _p[⊤]ρ_ [(][t][0][, x][)][f] [(][Z][(][t][0][;][ x][)][, θ][)] _._ (146)
_δρ_
 


Since (145) is not satisfied, there exists a θ[∗] such that _[δE]δρ[(][ρ][∞][)]_ (θ[∗], t0) = 0. From (146), by Hölder’s

_̸_
inequality,

_δE(ρ_ )
0 < _∞_ (θ[∗], t0) Ex _µ_ _pρ_ (t0; x) Ex _µ_ _f_ (Z(t0; x), θ[∗]) _,_

_δρ_ _∼_ _|_ _∞_ _|[2][

][1][/][2][ ]_ _∼_ _|_ _|[2][

][1][/][2]_

_[≤]_
  

which implies
Ex _µ_ _pρ_ (t0; x) _> 0 ._
_∼_ _|_ _∞_ _|[2][
]_
Then, Since f is a universal kernel according to Assumption 4.1, we can find ν such that

_f_ (Z(t0; x), θ) dν approximates −pρ∞ (t0, x). leading to
R _δE(ρ_ )

_δρ∞_ (θ, t0) dν(θ) < − 2[1] [E][x][∼][µ] _|pρ∞_ (t0; x)|[2][
] _< 0 ._

R[k][ h][∞][(][θ][) d][ν][(][θ][) =] R[k]

Z Z



As a consequence, there exists at least one point θ0 R[k] such that h (θ0) < 0. Since f is
2-homogeneous, by (10), h is also 2-homogeneous, so that ∈ _∞_

_h∞(θ0/|θ0|) < 0 ._

Because of continuity, there is a small neighborhood around θ0/|θ0| in S[k][−][1] where h is strictly
negative. Moreover, since h is Sard-type regular, there exist ϵ > 0 and η > 0 such that

_A =_ _θ ∈_ S[k][−][1] _h∞|Sk−1_ _θ_ _< −ϵ_ ≠ _∅_ _,_

∇θ[h][∞]n[|]e[S][k][−][1] _θ_ _· nθ_ _[> η,]e∀_ _θ[e] ∈_ _∂A,o_

 

where h∞|Sk−1 is the confinement of e _h∞eon S[k]e[−][1], and nθ_ [is the outer normal vector to][ ∂A][.]

This statement of h∞ can be extended to hs for sufficiently largere _s as well. Using estimate (48) from_
Lemma C.4, we obtain that


_hs(θ)_ _h_ (θ) in loc[(][R][k][)][,] as s _,_
_→_ _∞_ _C[1]_ _→∞_

meaning there exists S > 0 such that for any s ≥ _S, we have_

_hs|Sk−1_ _θ_ _< −ϵ/2,_ _∀_ _θ[e] ∈_ _A,_

∇θ[h][s][|][S][k][−]e[1] _θ_ _· nθ_ _[>][ 1]2_ _[η,]_ _∀_ _θ[e] ∈_ _∂A ._

 
e e

Extending this patch on the unit sphere to the whole domain, we define the cone: e


_A =_ _θ ∈_ R[k] _|θ| > 0, θ/|θ| ∈_ _A_ _._

Using the 2-homogeneous property of hs, we have for s _S that_
_≥_
_hs(θ) < −_ _[ϵ][|][θ]2[|][2]_ _[,]_ _∀θ ∈A,_ (147)
( _θhs(θ)_ _⃗nθ > 0,_ _θ_ _∂_ _θ_ _> 0_ _,_

_∇_ _·_ _∀_ _∈_ _A ∩{|_ _|_ _}_

where ⃗nθ is the outer normal vector to ∂A.


-----

We now define a new system that follows the gradient flow corresponding to ρs. Denote by _θ (s; α)_
the solution to the following ODE:

dθ[b] (s; α) _δE(ρ(s))_ [b]

= _θ_ _θ (s; α), t0_ = _θhs_ _θ (s; α)_ _,_ _s > S_
ds _−∇_ _δρ_ _−∇_ (148)



   

 _θ (S; α) = α,_ b b

where α R[k]. According to (147), when θ _∂_ _θ_ _> 0_, _θhs(θ) points outwards, away from_
_∈_ b _∈_ _A ∩{|_ _|_ _}_ _∇_

_A. We also notice that_ _θ_ _s;[⃗]0_ = _[⃗]0. Thus if the ODE starts with from some α ∈A, then for any_
_s_ _S, the particle stays within_ , that is,
_≥_ _A_

[b]

_θ (s; α) ∈A ._ (149)

As a consequence, we have

2 b
d _θ (s; α)_ 2

= 2 _θ (s; α),_ _θhs_ _θ (s; α)_ = 4hs _θ (s; α)_ _> 2ϵ_ _θ (s; α)_ _,_ (150)
ds _−_ _∇_ _−_

where we use the 2-homogeneous property of[b] D  _hEs in the second equality and_  (147) in the final

b b b [b]

inequality.

According to Lemma H.2, there exist two spheres separated by ρ(θ, t0, S), meaning that there exist
_β > 0 and γ > 0 relatively small (for example, with β < ra[′]_ [) such that]

(151)
_c_ [d][ρ][(][θ, t][0][, S][)][ > γ .]

ZA∩(Bβ([⃗]0))

By tracing the trajectory of (148), we have


R[k][ 1]θ[b](s;α)∈A∩(Bβ([⃗]0))c dρ(α, t0, S)

_s_ _S,_
_A∩(Bβ([⃗]0))c_ [d][ρ][(][α, t][0][, S][)][ > γ,] _≥_


_c_ [d][ρ][(][θ, t][0][, s][) =]
_A∩(Bβ([⃗]0))_


d _θ(s;α)_ 2
where in the first inequality we also use _|[b]_ ds _|_ 0 when α . Further, we have

_≥_ _∈A_

2

d _A∩(Bβ([⃗]0))c 1θ(s;α)∈A∩(Bβ([⃗]0))c_ _θ (s; α)_ dρ(α, t0, S)
R b ds

2 [b]

= d _A∩(Bβ([⃗]0))c_ _θ (s; α)_ dρ(α, t0, S)
R ds

[b] 2

2ϵ _c_ _θ (s; α)_ dρ(α, t0, S)
_≥_ ZA∩(Bβ([⃗]0))

2ϵγβ[2] [b]
_≥_

where we use (150) in the second inequality and (151) in the final inequality. It follows that

2

_slim→∞_ ( _β([⃗]0))c_ **[1]θ[b](s;α)∈A∩(Bβ([⃗]0))c** _θ (s; α)_ dρ(α, t0, S) = ∞ _._

ZA∩ _B_

It follows from this result that [b]


lim
_s→∞_ R[k][ |][θ][|][2][ d][ρ][(][θ, t][0][, s][) =][ ∞] _[,]_

Z

contradicting (144). Therefore, we must have

_δE(ρ_ )
_∞_ (θ, t0) = 0, _θ_ R[k] _,_

_δρ_ _∀_ _∈_


which completes the proof.


-----

H.3 PARTIALLY 1-HOMOGENEOUS CASE: PROOF OF THEOREM 6.2

We first give an example of partially 1-homogeneous activation function that satisfy Assumption 4.1,
and 6.2.

**Remark H.2 The following function satisfies Assumptions 4.1 and 6.2: Let θ = (θ[1], θ[2], θ[3])**
_with θ θ[1][2]σ2∈(_ _θ[2]R)[d], θ[2]θ[3]σ∈2(_ _θ[3]R[d])[×][d], θ[3]_ _∈_ R[d]. _Define f_ (x, θ) = _f_ (x, θ[1], θ[2], θ[3]) =

_θ[1]σ_ _θ[2]|_ _|_ _x +_ _θ[3]|_ _|_ _, where σ(x) is a regularized ReLU activation function, and_

_|_ _|_ _|_ _|_

_σway (of many) to define a regularized ReLU activation function is2, σ2(x)/x : R+ →_ R+ are bounded, Lipschitz, and differentiable with Lipschitz differential. One _σ(x) = (x + η)[2]/(4η)1x_ [ _η,η] +_
_∈_ _−_
_x1x∈(η,∞), for some small η._

As in the previous theorem, we prove a lemma, adapted from (Chizat & Bach, 2018, Lemma C.13),
that asserts preservation of the separation property.

**Lemma H.3 Let Assumptions 4.1 and 4.2 with k1 = 1. Suppose that ρini(θ, t) is admissible**
_and suppθ(ρini(θ, t))_ _θ_ _θ[1]_ _R_ _with some R > 0 for all t_ [0, 1]. Let ρ(θ, t, s)
_⊂{_ _||_ _| ≤_ _}_ _∈_ _∈_
_C([0, ∞); C([0, 1]; P_ [2])) solve (9). Suppose in addition that

-  f satisfies the partial 1-homogeneous condition (see Assumption 6.2),

-  The initial conditions satisfy the separation condition, that is, there exists t0 [0, 1] such
_that ρini(θ[1], θ[2], t0) separates the spheres {−r0} × R[k][−][1]_ _and {r0} × R[k] ∈[−][1]_ _for some_
_r0 > 0._

_Then for any s0 ∈_ [0, ∞), ρ(θ[1], θ[2], t0, s0) separates {−r[′]} × R[k][−][1] _and {r[′]} × R[k][−][1]_ _for some_
_r[′]_ _> 0._

**Proof Note that the particle representation θρ(s; t0) of ρ(θ, s, t0) satisfies**
dθρd(ss;t0) = −∇θ _δE(δρρ(s))_ (θρ(s; t0), t0), (152)
(θρ(0; t0) = θ .

Define a continuous map P : R[k] _× [0, ∞) →_ R[k] as the solution to (152), that is, P(θ, s) is the
solution to (152) with initial condition θρ(0; t0) = θ, where t0 is fixed. Define a diffeomorphism
_ψ : R × R[k][−][1]_ _→_ R × B1 (0) as follows:

_ψ(θ[1], θ[2]) =_ _θ[1],_ _θ[2]/|θ[2]|_ _· tanh_ _|θ[2]|_ _,_ _θ[2] ̸= 0_
(θ[1], 0), _θ[2] = 0,_
  
  



wheremap keeps the first component of θ[1] ∈ R is the first component of θ[1] intact and shrinks θ and θ[2] ∈ R[k][−][1] θcontains the remaining components. This[2] to push its amplitude below 1. This
diffeomorphism preserves the connection/separation property.

Define the continuous map Q as follows:

_Q(θ, s) = ψ ◦P(ψ[−][1](θ), s) : R × B1 (0) × [0, ∞) →_ R × B1 (0) .

Since ψ preserves the connection property, the lemma is proved if we can show ψ (supp(ρini), s)
_◦P_
separates {−r[′]} × B1 (0) and {r[′]} × B1 (0) for some r[′] _> 0. Since ψ ◦P(supp(ρini), s) =_
_Q(ψ(supp(ρini)), s), we trace the evolution of Q(θ, s) for θ ∈_ R × B1(0). According to (Chizat &
Bach, 2018, Proposition C.14), this translates to showing Q(θ, s) can be continuously extended to
R × B1 (0) × [0, S] → R × B1 (0), with the extension satisfying

_Q(θ, s) ∈_ R × ∂B1 _⃗0_ _,_ _∀θ ∈_ R × ∂B1 _⃗0_ _,_ _s ∈_ [0, ∞), (153)
   

meaning that the extension Q(·, s) maps R × ∂B1([⃗]0) to itself for all s ∈ [0, ∞).


-----

Denoting _s(θ) =_ (θ, s), we consider the velocity field of this flow (similar to the proof of (Chizat
_Q_ _Q_
& Bach, 2018, Lemma C.13)):

d _s_ d _s_ _ψ[−][1]_
_Q_ _θψ_ _s_ _ψ[−][1]_ (θ) _P_ _◦_

ds [=] _∇_ _P_ _◦_ ds
  

 _δE(ρ(s))_

= _θψ_ _s_ _ψ[−][1]_ (θ) _θ_ _s_ _ψ[−][1]_ (θ), t0
_−_ _∇_ _P_ _◦_ _∇_ _δρ_ _P_ _◦_
  

 []δE(ρ(s))  
[]

= _θψ_ _ψ[−][1](_ _s)_ _θ_ _ψ[−][1](_ _s), t0_ = V ( _s, s) ._
_−_ _∇_ _Q_ _∇_ _δρ_ _Q_ _Q_
  

 []  
[]

From the fourth condition of Theorem 6.2, the velocity field V (θ, s) can be continuously extended to
R × ∂B1 (0) as follows:

_V (θ, s) = V (θ[1], θ[2], s) =_ _−_ _∇θψ_ _ψ[−][1](θ)_ _∇θ_ _δE(δρρ(s))_ _ψ[−][1](θ), t0_ _,_ _|θ[2]| < 1_

( _−H∞,ρ(s)(θ[2]), 0

 [],_  
[] _|θ[2]| = 1,_

where H∞,ρ(s) is the limit of _[δE][(]δρ[ρ][(][s][))]_ 1, rθ[2], t0 as
 _r →∞. Within this velocity field, Qs can be_

continuously extended to R 1 (0)  [0, ) 
 R 1 (0) with the extension satisfying (153).
_× B_ _×_ _∞_ _→_ _× B_
This completes the proof.


We are now ready to prove Theorem 6.2.

**Proof [Proof of Theorem 6.2] The technique of proof is similar to the 2-homogeneous case. Since**
_ρ(θ, t, s) converges to ρ_ (θ, t) in ([0, 1]; ), we have
_∞_ _C_ _P_ [2]

sup (154)
_s≥0_ ZR[k][ |][θ][|][2][dρ][(][θ, t][0][, s][)][ <][ ∞] _[.]_

According to Proposition 6.1, it suffices to prove that

_δE(ρ_ ) _δE(ρ_ )
_∞_ (θ, t0) = _[δE][(][ρ][∞][)]_ (θ[1], θ[2], t0) = θ[1] _∞_ (1, θ[2], t0) = 0, _θ_ R[k] _._ (155)

_δρ_ _δρ_ _δρ_ _∀_ _∈_


In the following, we will show that

R[k][ |][θ][|][2][dρ][(][θ, t][0][, s][)][ blows up as][ s][ →∞] [if][ (155)][ fails to hold, in]
contradiction to (154).

R


Denote


_h∞(θ[2]) =_ _[δE]δρ[(][ρ][∞][)]_ (1, θ[2], t0), _hs(θ[2]) =_ _[δE][(]δρ[ρ][(][s][))]_ (1, θ[2], t0) . (156)


Then if (155) is not satisfied, without loss of generality, we can assume that there exists θ[2] such that
_h∞(θ[2]) < 0. Since h∞_ satisfies Sard-type regularity, there exist ϵ > 0 and η > 0 such that

_A =_ _θ[2] ∈_ R[k][−][1] _h∞_ _θ[2]_ _< −ϵ_ ≠ _∅;_ _∇θ[2]_ _h∞(θ[2]) · ⃗nθ[2] > η,_ _∀_ _θ[2] ∈_ _∂A,_

where nθ[2] is the outer normal vector on  
 _∂A._

Using the definition of _[δE]δρ_ [as in (10), we have]

_δE(ρ(s))_ _δE(ρ_ )

_δρ_ (1, θ[2], t) −∇θ[2] _δρ∞_ (1, θ[2], t)

=[∇] E[θ][[2]]x∼µ _∂θ[2]_ _f[ˆ](Zρ(s)(t; x), θ[2])pρ(s)(t; x) −_ _∂θ[2]_ _f[ˆ](Zρ∞_ (t; x), θ[2])pρ∞ (t; x)
 


_≤_ Ex∼µ _∂θ[2]_ _f[ˆ](Zρ(s)(t; x), θ[2]) −_ _∂θ[2]_ _f[ˆ](Zρ∞_ (t; x), θ[2]) _|pρ(s)(t; x)|_
 

+ Ex∼µ _∂θ[2]_ _f[ˆ](Zρ∞_ (t; x), θ[2]) _pρ(s)(t; x) −_ _pρ∞_ (t; x)

_C_ Ex _µ_ _Zρ(s)(t; x)_ _Zρ_ (t; x) + Ex _µ_ _pρ(s)(t; x)_ _pρ_ (t; x)
_≤_ _∼_ _−_ _∞_ _∼_ _−_ _∞_

_≤_ _Cd1(ρ(s), ρ∞),_ [
]  [

]


-----

where we have used the Lipschitz continuity of f and its derivatives as in Assumption 6.2 and the
estimates (18), (31), and (47). We thus have

_hs(θ[2])_ _h_ (θ[2]) in (R[k][−][1]), as s _._ (157)
_→_ _∞_ _C[1]_ _→∞_

As a consequence, there exists S > 0 such that for any s ≥ _S_
_hs_ _θ[2]_ _< −ϵ/2,_ _∀_ _θ[2] ∈_ _A,_ (158)
∇θ[2] _hs
(θ[2]) · ⃗nθ[2] >_ [1]2 _[η,]_ _∀_ _θ[2] ∈_ _∂A ._

Extending this set to the whole space, we define

= (θ[1], θ[2]) (0, ) _A_ _._
_A_ _∈_ _∞_ _×_

Sincehs, we have ∂A = (θ[1], θ[2]) ∈ (0, ∞) × ∂A _∪{θ[1] = 0, θ[2] ∈_ _A}, from (158) and the definition of_


_δE(ρ(s))_
_∇θ_ _δρ_ (θ, t0) · ⃗nθ = θ[1] _∇θ[2]_ _hs(θ[2]) · ⃗nθ[2]_ _> 0,_ _∀θ ∈_ (0, ∞) × ∂A, (159)

 



where ⃗nθ is the outer normal direction on ∂A, and ⃗nθ[2] is the outer normal vector on ∂A. When
_θ ∈{θ[1] = 0, θ[2] ∈_ _A},_
_δE(ρ(s))_ _δE(ρ(s))_
_∇θ_ _δρ_ (θ, t0) 1 = hs(θ[2]) < 0, _∇θ_ _δρ_ (θ, t0) _i_ = 0, _i = 2, . . ., k,_ (160)
   

where [ ]i means the i-component of the vector. This implies that _θ_ _δE(δρρ(s))_ (θ, t0) points strictly

_·_ _∇_

downward when θ ∈{θ[1] = 0, θ[2] ∈ _A}._

As in the 2-homogeneous case, we consider the gradient flow corresponding to ρs. Denote by _θ (s; α)_
the solution to the following ODE:
dθ(s;α) _δE(ρ(s))_ [b]

ds = −∇θ _δρ_ _θ (s; α), t0_ = −∇θ _θ[1]hs_ _θ[2]_ _θ (s; α)_ _,_ _s > S_ (161)

b
(θ (S; α) = α,     

 [] 

b b

whereb α ∈ R[k]. Since the minus gradient is pointing inward to A, as stated in (159) and (160), _θ (s; α)_
stays in A if _θ (s[′]; α) ∈A for some s[′]_ _∈_ [S, s]. Moreover, using (161), if _θ (s; α) ∈A, we have_

[b]

[b] d|θ[b][1] (s; α) |[2] = 2θ[b][1] (s; α) hs _θ[2] (s; α)_ _> ϵθ[b][1] ([b]s; α),_ (162)

ds _−_
 

where the last inequality uses (158).

b

Similar to (Chizat & Bach, 2018, Proposition C.4), we claim that there exists S1 _S, β > 0, and_
_γ > 0 such that (see detailed proof in Appendix H.3.1)_ _≥_


dρ(θ, t0, S1) > γ . (163)

Z(β,∞)×A

2
R[k][ 1]θ[b](S1;α)∈(β,∞)×A _θ[1] (s; α)_ dρ(α, t0, S)

ds

[b]


Then

Since

lim
_s→∞_


R[k][ 1]θ[b](S1;α)∈(β,∞)×Aθ[b][1] (s; α) dρ(α, t0, S)

_θ(S1;α)_ (β, ) _A_ [d][ρ][(][α, t][0][, S][)]
R[k][ 1][b] _∈_ _∞_ _×_

Z

dρ(θ, t0, S1) _ϵβγ ._

Z(β,∞)×A _≥_


_≥_ _ϵ_
Z

_≥_ _ϵβ_

= ϵβ


(164)


_θ[1]_ dρ(θ, t0, s) lim
(β, ) _A_ _|_ _|[2]_ _≥_ _s→∞_
_∞_ _×_


_θ(S1;α)_ (β, ) _A_ _θ[1] (s; α)_ dρ(α, t0, S) = _,_
R[k][ 1][b] _∈_ _∞_ _×_ _∞_

[b]


-----

we finally obtain, using (164), that


lim _θ[1]_ dρ(θ, t0, s) = _._
_s→∞_ R[k][ |][θ][|][2][ d][ρ][(][θ, t][0][, s][)][ ≥] _s[lim]→∞_ (β, ) _A_ _|_ _|[2]_ _∞_

Z Z _∞_ _×_

This limit contradicts (154), implying that (155) must hold, as claimed.

H.3.1 CLAIM IN THE PROOF OF THEOREM 6.2

In this section, we prove the statement in (163), meaning that we need to find S1 _S, β > 0, and_
_γ > 0 such that_ _≥_

dρ(θ, t0, S1) > γ . (165)

Z(β,∞)×A

Supposing that [d][ρ][(][θ, t][0][, S][)][ >][ 0][, then by making][ β][ and][ γ][ small enough,][ (165)][ is satisfied]

_A_
naturally.

R


If



[d][ρ][(][θ, t][0][, S][) = 0][, then it suffices to show that there exists][ S][1][ > S][ such that]
_A_

dρ(θ, t0, S1) > 0 . (166)

Z


Define hs(θ[2]) and h∞(θ[2]) as in (156). Because of the fourth condition in Theorem 6.2, there exists
a function h[′](θ[e]) on S[k][−][2][
] such that
_C[1][ ]_

_h_ (rθ[e]) _r→∞_ _h[′](θ[e]),_ in (S[k][−][2] _._
_∞_ _−−−→_ _C[1]_

Combining this with (157), there exists h[∗] _> ϵ/2 such that_ _hs_ _L[∞]A_
we can find θ[2][∗] _∥_ _∥_ _[≤]_ _[h][∗][. Further, for any][ ξ >][ 0][,]_

_[∈]_ _[A][ and][ S][′][ large enough such that]_

_|∇θ[2][∗]_ _[h][s][(][θ][2][∗]_ [)][|][ < ξ,] _∀s ≥_ _S[′]_ _._ (167)

According to Lemma H.3, there exists rS′ > 0 such that ρ(θ[1], θ[2], t0, S[′]) separates _rS′_
_{−_ _} ×_
R[k][−][1] and {rS′ _} × R[k][−][1]. Considering the set [−rS′_ _, rS′_ ] × {θ[2][∗] _[}][, it must intersect the support of]_
_ρ(θ[1], θ[2], t0, S[′]) due to the separation property. Thus, any open set that contains [−rS′_ _, rS′_ ]×{θ[2][∗] _[}]_
must have a positive measure in ρ(θ[1], θ[2], t0, S[′]). Because [d][ρ][(][θ, t][0][, S][′][) = 0][ and][ (][−][r][S][′][ −]

_A_
1, ) _A is a open set that covers [_ _rS[′]_ _, rS[′]_ ] _θ[2][∗]_ _[}][, there exists a open set][ U][ ⊂]_ [(][−][r][S][′][ −] [1][,][ 0]] _[×]_ _[A]_
_∞_ _×_ _−_ _×{_ R
such that

dρ(θ[1], θ[2], t0, S[′]) > 0 .
_U_

Z

Thus, we can find a point 0 < r[∗] _rS′ + 1 and an arbitrary small σ > 0 such that_
_≤_


_Bσ(−r[∗], θ[2][∗]_ [)][ ⊂] [(][−][r][S][′][ −] [1][ −] _[σ, σ][)][ ×][ A,]_ and

Recalling the system (161), we claim the following:


dρ(θ, t0, S[′]) > 0 .
_Bσ(−r[∗],θ[2][∗]_ [)]


_When ξ, σ are small enough, there exists S1 > S[′]_ _such that_ _θ (S1; α) ∈A for any_
_α ∈Bσ(−r[∗], θ[2][∗]_ [)][.]

[b]

If this claim is true, then


dρ(θ, t0, S[′]) > 0 .
_Bσ(−r[∗],θ[2][∗]_ [)]


dρ(θ, t0, S1)
_≥_

ZA

which proves (166) and the lemma.


-----

Now, we prove the claim. Because f satisfies Assumption 6.2 and the second moment of ρ is
uniformly bounded in s, for all s > 0, we have

_δE(ρ(s))_

(1, θ[2], t0) (1, θ[2][′] _[, t][0][)]_ [2][|][,]
_δρ_ _−_ _[δE][(]δρ[ρ][(][s][))]_

_[≤]_ _[L][|][θ][[2]][ −]_ _[θ][′]_

(168)

_δE(ρ(s))_ _δE(ρ(s))_

(1, θ[2], t0) _θ_ (1, θ[2][′] _[, t][0][)]_ [2][|][,]
_δρ_ _−∇_ _δρ_ _[≤]_ _[L][|][θ][[2]][ −]_ _[θ][′]_

for some constant L. According to (161), we have

_[∇][θ]_

dθ[b][1](s; α)

ds = −hs _θ[2](s; α)_

 2   _,_ _s_ _S[′]_ (169)
 d _θ[2](s; α) −_ _θ[2][∗]_ b _≥_

ds _≤_ 2 _θ[2](s; α) −_ _θ[2][∗]_ _θ[1](s; α)_ _∇θ[2]_ _hs_ _θ[2](s; α)_

where [b] [b] [b] b 
_r[∗]_ _σ_ _θ[1](S[′]; α)_ _σ,_ and _θ[2](S[′]; α)_ _θ[2][∗]_ _σ ._ (170)
_−_ _−_ _≤_ [b] _≤_ _−_ _≤_

To prove the claim, it suffices to show that there exists S1 > S[′] such that

[b]

We first show that _θ[1] increases and the right hand-side ofθb[1] (S1; α) > 1_ and _θb[2] (S (169)1; α) ∈ is bounded. SinceA ._ _A is a open set,(171)_
there exists an arbitrary small Σ > 0 such that BΣ(θ[2][∗] [)][ ⊂] _[A][. We first choose][ σ <][ min][{][Σ][,][ 2][}][. When]_

[b]
_θ[1](s; α)_ _≤_ [2][r][S][′]ϵ[ + 2] _h[∗]_ + 2, _θ[2](s; α) −_ _θ[2][∗]_ _≤_ Σ, (172)

we have from (158), (167), and (168) that

[b] [b]

dθ[b][1](s; α) dθ[b][1](s; α)

ds = −hs _θ[2](s; α)_ _< h[∗],_ ds = −hs _θ[2](s; α)_ _> ϵ/2_ (173)
   

and

b 2 b

d _θ[2](s; α)_ _θ[2][∗]_
_−_

ds

[b] 2rS′ + 2 (174)

_≤_ 2 _ϵ_ _h[∗]_ + 2 _θ[2](s; α) −_ _θ[2][∗]_ _L_ _θ[2](s; α) −_ _θ[2][∗]_ + ξ
 2rS′ + 2  2 2rS′ + 2 

_≤_ 2L _ϵ_ _h[∗]_ + 2 [b]θ[2](s; α) − _θ[2][∗]_ + 2[b] _ϵ_ _h[∗]_ + 2 Σξ .
   

When s = S[′] and α ∈Bσ(−r[∗], θ[2][∗] [)][, we have from (170) that][b]

_θ[1](S[′]; α)_ _rS′ + 1 + σ <_ [2][r][S][′][ + 2] _h[∗]_ + 2, _θ[2](S[′]; α)_ _θ[2][∗]_ _[| ≤]_ _[σ <][ Σ][ .]_
_|[b]_ _| ≤_ _ϵ_ _|[b]_ _−_

Thus, for s slightly larger than S[′], we still have that
_θ[1](s; α)_ _<_ [2][r][S][′]ϵ[ + 2] _h[∗]_ + 2 and _θ[2](s; α) −_ _θ[2][∗]_ _< Σ._

Denote by S[∗] the first time that

[b] [b]

_θ[1](S[∗]; α)_ _≥_ [2][r][S][′]ϵ[ + 2] _h[∗]_ + 2 or _θ[2](S[∗]; α) −_ _θ[2][∗]_ _≥_ Σ.

Then we show that there exists S1 [S[′], S[∗]] such that (171) is satisfied when σ, Σ, and ξ are small
enough. From (170), we have for[b] _s ∈ ∈_ (S[′], S[∗]) that [b]

_θ[1](s; α)_ _σ + (s_ _S[′])h[∗]_
_≤_ _−_

_θ[1](s; α) >_ _rS′_ _σ + (s_ _S[′])ϵ/2,_

b _−_ _−_ _−_

1/2

2rS′ + 2 2rS′ + 2
_θ[2](s; αb) −_ _θ[2][∗]_ _< exp_ _L_ _ϵ_ _h[∗]_ + 2 (s − _S[′])_ _σ[2]_ + 2(s − _S[′])_ _ϵ_ _h[∗]_ + 2 Σξ
       

[b]


-----

where the first two inequlaties come from (173) and last inequality comes from (174) via Grönwall’s
inequality. Defining

_S1 = [2][r][S][′][ + 2]_ + S[′],

_ϵ_

we can choose the positive values σ, Σ, and ξ small enough that


_θ[1](S1, θ[S][′]_ ) > _rS′ + (S1_ _S[′])ϵ/2 = 1,_
_−_ _−_

and for s [S[′], S1]
_∈_ b
_θ[1](s; α)_ _<_ [2][r][S][′]ϵ[ + 2] _h[∗]_ + 2, _θ[2](s; α) −_ _θ[2][∗]_ _< Σ ._ (175)

According to (175), the bounds (172) are satisfied for s [S[′], S1], which implies that S1 < S[∗]

[b] [b] ∈

and _θ[2](S1, θ[S][′]_ ) _A. Further, we have_ _θ[1](S1, θ[S][′]_ ) > 1. By combining these two results,
_∈_
we conclude that (171) is satisfied with the chosen values of σ, Σ, ξ, and S1. Thus, we have
_θ (S1[b]; α)_ (1, ) _A_ for any α _σ[b](_ _r[∗], θ[2][∗]_ [)][, which proves the claim.]
b _∈_ _∞_ _×_ _⊂A_ _∈B_ _−_


-----