Joint Behaviour of Semirecursive Kernel Estimators of the Location and of the Size of the Mode of a Probability Density Function

Abstract

Let 𝜃 and 𝜇 denote the location and the size of the mode of a probability density. We study the joint convergence rates of semirecursive kernel estimators of 𝜃 and 𝜇. We show how the estimation of the size of the mode allows measuring the relevance of the estimation of its location. We also enlighten that, beyond their computational advantage on nonrecursive estimators, the semirecursive estimators are preferable to use for the construction of confidence regions.

1. Introduction

Let 𝑋₁,…,𝑋_𝑛 be independent and identically distributed ℝ^𝑑-valued random variables with unknown probability density 𝑓. The aim of this paper is to study the joint kernel estimation of the location 𝜃 and of the size 𝜇=𝑓(𝜃) of the mode of 𝑓. The mode is assumed to be unique, that is, 𝑓(𝑥)<𝑓(𝜃) for any 𝑥≠𝜃, and nondegenerated, that is, the second order differential 𝐷²𝑓(𝜃) at the point 𝜃 is nonsingular (in the sequel, 𝐷^𝑚𝑔 will denote the differential of order 𝑚 of a multivariate function 𝑔).

The problem of estimating the location of the mode of a probability density was widely studied. Kernel methods were considered, among many others, by Parzen [1], Nadaraya [2], Van Ryzin [3], Rüschendorf [4], Konakov [5], Samanta [6], Eddy ([7, 8]), Romano [9], Tsybakov [10], Vieu [11], Mokkadem and Pelletier [12], and Abraham et al. ([13, 14]). The problem of estimating the size of the mode was brought up by several authors (see, e.g., Romano [9] and Vieu [11]), but, at our knowledge, the behaviour of estimators of the size of the mode has not been investigated in detail, whereas there are at least two statistical motivations for estimating this parameter. First, the use of an estimator of the size is necessary for the construction of confidence regions for the location of the mode (see, e.g., Romano [9]). As a more important motivation, the estimation of the high of the peak gives information on the shape of a density in a neighbourhood of its mode and, consequently, allows measuring the pertinence of the parameter location of the mode; this motivation must be related to the remark made by Vieu [11], who pointed out that the location of the mode is more related to the shape of the derivative of 𝑓, whereas the size of the mode is more related to the shape of the density itself.

Let us mention that, even though the problem of estimating the size of the mode was not investigated in the framework of density estimation, it was studied in the framework of regression estimation. Müller [15] proves in particular the joint asymptotic normality and independence of kernel estimators of the location and of the size of the mode in the framework of nonparametric regression models with fixed design. In the framework of nonparametric regression with random design, a similar result is obtained by Ziegler ([16, 17]) for kernel estimators and by Mokkadem and Pelletier [18] for estimators issued from stochastic approximation methods.

This paper is focused on semirecursive kernel estimators of 𝜃 and 𝑓(𝜃). To explain why we chose this option of semirecursive estimators, let us first recall that the (nonrecursive) wellknown kernel estimator of the location of the mode introduced by Parzen [1] is defined as a random variable 𝜃^∗_𝑛 satisfying 𝑓^∗_𝑛𝜃^∗_𝑛=sup_{𝑦∈ℝ𝑑}𝑓^∗_𝑛𝑦, (1.1) where 𝑓^∗_𝑛 is Rosenblatt's estimator of 𝑓; more precisely, 𝑓^∗_𝑛(𝑥)=1𝑛ℎ^𝑑_𝑛^𝑛_𝑖=1𝐾𝑥−𝑋_𝑖ℎ_𝑛, (1.2) where the bandwidth (ℎ_𝑛) is a sequence of positive real numbers going to zero and the kernel 𝐾 is a continuous function satisfying lim_{‖𝑥‖→+∞}𝐾(𝑥)=0, ∫_ℝ𝑑𝐾(𝑥)𝑑𝑥=1. The asymptotic behaviour of 𝜃^∗_𝑛 was widely studied (see, among others, [1–9, 11, 12]), but, on a computational point of view, the estimator 𝜃^∗_𝑛 has a main drawback: its update, from a sample size 𝑛 to a sample size 𝑛+1, is far from being immediate. Applying the stochastic approximation method, Tsybakov [10] introduced the recursive kernel estimator of 𝜃 defined as 𝑇_𝑛=𝑇_𝑛−1+𝛾_𝑛1ℎ_𝑛^𝑑+1𝑇∇𝐾^𝑛−1−𝑋_𝑛ℎ_𝑛, (1.3) where 𝑇₀∈ℝ^𝑑 is arbitrarily chosen and the stepsize (𝛾_𝑛) is a sequence of positive real numbers going to zero. The great property of this estimator is that its update is very rapid. Unfortunately, for reasons inherent to stochastic approximation algorithms properties, very strong assumptions on the density 𝑓 must be required to ensure its consistency. A recursive version 𝑓_𝑛 of Rosenblatt's density estimator was introduced by Wolverton and Wagner [19] (and discussed, among others, by Yamato [20], Davies [21], Devroye [22], Menon et al. [23], Wertz [24], Wegman and Davies [25], Roussas [26], and Mokkadem et al. [27]). Let us recall that 𝑓_𝑛 is defined as𝑓_𝑛(𝑥)=1𝑛^𝑛_𝑖=11ℎ^𝑑_𝑖𝐾𝑥−𝑋_𝑖ℎ_𝑖. (1.4) Its update from a sample of size 𝑛 to one of size 𝑛+1 is immediate since 𝑓_𝑛 clearly satisfies the recursive relation 𝑓_𝑛(𝑥)=11−𝑛𝑓_𝑛−1(𝑥)+1𝑛ℎ^𝑑_𝑛𝐾𝑥−𝑋_𝑛ℎ_𝑛. (1.5) This property of rapid update of the density estimator is particularly important in the framework of mode estimation, since the number of points where 𝑓 must be estimated is very large. We thus define a semirecursive version of Parzen's estimator of the location of the mode by using Wolverton-Wagner's recursive density estimator, rather than Rosenblatt's density estimator. More precisely, our estimator 𝜃_𝑛 of the location 𝜃 of the mode is a random variable satisfying𝑓_𝑛𝜃_𝑛=sup_{𝑦∈ℝ𝑑}𝑓_𝑛𝑦. (1.6)

Let us now come back to the problem of estimating the size 𝑓(𝜃) of the mode. The ordinarily used estimator is defined as 𝜇^∗_𝑛=𝑓^∗_𝑛(𝜃^∗_𝑛) (𝑓^∗_𝑛 being Rosenblatt's density estimator and 𝜃^∗_𝑛 Parzen's mode estimator); the consistency of 𝜇^∗_𝑛 is sufficient to allow the construction of confidence regions for 𝜃 (see, e.g., Romano [9]). Adapting the construction of 𝜇^∗_𝑛 to the semirecursive framework would lead us to estimate 𝑓(𝜃) by𝜇_𝑛=𝑓_𝑛𝜃_𝑛. (1.7) However, this estimator has two main drawbacks (as well as 𝜇^∗_𝑛). First, the use of a higher order kernel 𝐾 is necessary for (𝜇_𝑛−𝜇) to satisfy a central limit theorem and thus for the construction of confidence intervals of 𝜇 (and of confidence regions for (𝜃,𝜇)). Moreover, in the case when a higher order kernel is used, it is not possible to choose a bandwidth for which both estimators 𝜃_𝑛 and 𝜇_𝑛 converge at the optimal rate. These observations lead us to use two different bandwidths, one for the estimation of 𝜃, the other one for the estimation of 𝜇. More precisely, let 𝑓_𝑛 be the recursive kernel density estimator defined as 𝑓_𝑛(𝑥)=1𝑛^𝑛_𝑖=11ℎ^𝑑_𝑖𝐾𝑥−𝑋_𝑖ℎ_𝑖, (1.8) where the bandwidth (ℎ_𝑛) may be different from (ℎ_𝑛) used in the definition of 𝑓_𝑛 (see (1.4)); we estimate the size of the mode by𝜇_𝑛=𝑓_𝑛𝜃_𝑛, (1.9) where 𝜃_𝑛 is still defined by (1.6) and thus with the first bandwidth (ℎ_𝑛).

The purpose of this paper is the study of the joint asymptotic behaviour of 𝜃_𝑛 and 𝜇_𝑛. We first prove the strong consistency of both estimators. We then establish the joint weak convergence rate of 𝜃_𝑛 and 𝜇_𝑛. We prove in particular that adequate choices of the bandwidths lead to the asymptotic normality and independence of these estimators and that the use of different bandwidths allow obtaining simultaneously the optimal convergence rate of both estimators. We then apply our weak convergence rate result to the construction of confidence regions for (𝜃,𝜇) and illustrate this application with a simulations study. This application enlightens the advantage of using semirecursive estimators rather than nonrecursive estimators. It also shows how the estimation of the size of the mode gives information on the relevance of estimating its location. Finally, we establish the joint strong convergence rate of 𝜃_𝑛 and 𝜇_𝑛.

2. Assumptions and Main Results

Throughout this paper, (ℎ_𝑛) and (ℎ_𝑛) are defined as ℎ_𝑛=ℎ(𝑛) and ℎ_𝑛=ℎ(𝑛) for all 𝑛≥1, where ℎ and ℎ are two positive functions.

2.1. Strong Consistency

The conditions we require for the strong consistency of 𝜃_𝑛 and 𝜇_𝑛 are the following. (A1)(i)𝐾 is an integrable, differentiable, and even function such that ∫_ℝ𝑑𝐾(𝑧)𝑑𝑧=1.(ii)There exists 𝜁>0 such that ∫_ℝ𝑑‖𝑧‖^𝜁|𝐾(𝑧)|𝑑𝑧<∞.(iii)𝐾 is Hölder continuous.(iv)There exists 𝛾>0 such that 𝑧↦‖𝑧‖^𝛾|𝐾(𝑧)| is a bounded function. (A2)(i)𝑓 is uniformly continuous on ℝ^𝑑.(ii)There exists 𝜉>0 such that ∫_ℝ𝑑‖𝑥‖^𝜉𝑓(𝑥)𝑑𝑥<∞.(iii)There exists 𝜂>0 such that 𝑧↦‖𝑧‖^𝜂𝑓(𝑧) is a bounded function.(iv)There exists 𝜃∈ℝ^𝑑 such that 𝑓(𝑥)<𝑓(𝜃) for all 𝑥≠𝜃. (A3)(i)The function ℎ is locally bounded and varies regularly with exponent (−𝑎) with 𝑎∈]0,1/𝑑[. (ii)The function ℎ is locally bounded and varies regularly with exponent (−𝑎) with 𝑎∈]0,1/𝑑[.

Remark 2.1. Note that (A1)(iv) implies that 𝐾 is bounded.

Remark 2.2. Let us recall that a positive function (not necessarily monotone) ℒ defined on ]0,∞[ is slowly varying if lim_𝑡→∞ℒ(𝑡𝑥)/ℒ(𝑡)=1 and that a function 𝐺 varies regularly with exponent 𝜌, 𝜌∈ℝ, if and only if it is of the form 𝐺(𝑥)=𝑥^𝜌ℒ(𝑥) with ℒ slowly varying (see, e.g., Feller [28, page 275]). Typical examples of regularly varying functions are 𝑥^𝜌, 𝑥^𝜌log𝑥, 𝑥^𝜌loglog𝑥, 𝑥^𝜌log𝑥/loglog𝑥, and so on.

Proposition 2.3. Let 𝜃_𝑛 and 𝜇_𝑛 be defined by (1.6) and (1.9), respectively. (i)Under (A1), (A2), and (A3)(i), lim_𝑛→∞𝜃_𝑛=𝜃 a.s. (ii)Under (A1)–(A3), lim_𝑛→∞𝜇_𝑛=𝜇 a.s.

Let us mention that the assumptions required on the probability density to establish the strong consistency of the semirecursive estimator of the location of the mode are slightly stronger than those needed for the nonrecursive estimator 𝜃^∗_𝑛 (see, e.g., [9, 12]), but are much weaker than those needed for the recursive estimator (see [10]). Let us also note that the strong consistency of 𝜇^∗_𝑛 can be proved in the same way as that of 𝜇_𝑛.

2.2. Weak Convergence Rate

In order to state the weak convergence rate of 𝜃_𝑛 and 𝜇_𝑛, we need the following additional assumptions on 𝐾, 𝑓, ℎ, and ℎ. (A4)(i)𝐾 is twice differentiable on ℝ^𝑑.(ii)𝑧↦𝑧∇𝐾(𝑧) is integrable.(iii)For any (𝑖,𝑗)∈{1,…,𝑑}², 𝜕²𝐾/𝜕𝑥_𝑖𝜕𝑥_𝑗 is bounded integrable and Hölder continuous.(iv)𝐾 is a kernel of order 𝑞≥2, that is, forall𝑠∈{1,…,𝑞−1}, forall𝑗∈{1,…,𝑑}, ∫_ℝ𝑑𝑦^𝑠_𝑗𝐾(𝑦)𝑑𝑦_𝑗=0 and ∫_ℝ𝑑|𝑦^𝑞_𝑗𝐾(𝑦)|𝑑𝑦<∞.(A5)(i)𝐷²𝑓(𝜃) is nonsingular.(ii)𝐷²𝑓 is 𝑞-times differentiable; ∇𝑓 and 𝐷^𝑞𝑓 are bounded.(iii)For any (𝑖,𝑗)∈{1,…,𝑑}², sup_{𝑥∈ℝ𝑑}‖𝐷^𝑞(𝜕²𝑓/𝜕𝑥_𝑖𝜕𝑥_𝑗)‖<∞, and for any 𝑘∈{1,…,𝑑}, sup_{𝑥∈ℝ𝑑}‖𝐷^𝑞(𝜕𝑓/𝜕𝑥_𝑘)‖<∞. (A6)(i)𝑎∈]0,1/(𝑑+4)[.(ii)𝑎∈]0,1/(𝑑+2)[.

Remark 2.4. Note that (A4)(ii) and (A4)(iii) imply that ∇𝐾 is Lipschitz-continuous and integrable; it is thus straightforward to see that lim_{‖𝑥‖→∞}‖∇𝐾(𝑥)‖=0 (and in particular ∇𝐾 is bounded).

Let 𝐺 be the 𝑑×𝑑 matrix defined by 𝐺^(𝑖,𝑗)=∫_ℝ𝑑(𝜕𝐾/𝜕𝑥_𝑖)(𝑥)(𝜕𝐾/𝜕𝑥_𝑗)(𝑥)𝑑𝑥 and set 𝛽^𝑞_𝑗=_ℝ𝑑𝑦^𝑞_𝑗𝐾𝑦𝐵𝑑𝑦,_{(𝜃)𝑎,𝑞}=(−1)^𝑞∇𝑞!1−𝑎𝑞^𝑑_𝑗=1𝛽^𝑞_𝑗𝜕^𝑞𝑓𝜕𝑥^𝑞_𝑗(𝜃)Σif𝑎𝑞<1,0otherwise,_𝑎(𝜃)=𝑓(𝜃)1+𝑎(𝑑+2)𝐺. (2.1) The following theorem gives the weak convergence rate of the semirecursive kernel mode estimator 𝜃_𝑛.

Theorem 2.5 (Weak convergence rate of 𝜃_𝑛). Let 𝜃_𝑛 be defined by (1.6), and assume that (A1), (A2), (A3)(i), (A4), (A5), and (A6)(i) hold. (1)If there exists 𝑐≥0 such that lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞=𝑐, then 𝑛ℎ_𝑛^𝑑+2𝜃_𝑛−𝜃⟶^𝒟𝒩−√𝑐𝐷²𝑓(𝜃)⁻¹𝐵^(𝜃)_𝑎,𝑞,𝐷²𝑓(𝜃)⁻¹Σ_𝑎^(𝜃)𝐷²𝑓(𝜃)⁻¹. (2.2)(2)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞=∞, then ℎ_𝑛^−𝑞𝜃_𝑛−𝜃⟶^ℙ−𝐷²𝑓(𝜃)⁻¹𝐵^(𝜃)_𝑎,𝑞. (2.3)

Remark 2.6. If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞=𝑐≠0 or if lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞=∞, then 𝑎𝑞<1, and thus 𝐵^(𝜃)_𝑎,𝑞 is usually nonzero.

In order to compare the semirecursive estimator 𝜃_𝑛 with the well-known nonrecursive Parzen's estimator 𝜃^∗_𝑛, let us recall that Theorem 2.5 holds when 𝜃_𝑛 is replaced by 𝜃^∗_𝑛 and 𝑎 by zero (see e.g., Parzen [1] in the case 𝑑=1, and Mokkadem and Pelletier [12] in the case 𝑑≥1). The main advantage of 𝜃_𝑛 on 𝜃^∗_𝑛 is that, due to the factor [1+𝑎(𝑑+2)]⁻¹ standing in the definition of Σ_𝑎^(𝜃), the asymptotic covariance of 𝜃^∗_𝑛 is smaller than that of 𝜃_𝑛; this property will be discussed again in Section 2.3.

In order to state the weak convergence rate of 𝜇_𝑛, we need the following notation (where 𝛾>0 and where (𝑏_𝑛) is a positive sequence): 𝑤𝑏_𝑛,𝛾=log𝑛^1+𝛾𝑛𝑏_𝑛^𝑑+2+𝑏_𝑛^2𝑞,𝑃_𝑛𝛾=𝑤ℎ_𝑛,𝛾²+𝑤ℎ_𝑛,𝛾𝑤ℎ_𝑛,𝐵,𝛾_{(𝜇)𝑎,𝑞}=(−1)^𝑞𝑞!1−𝑎𝑞^𝑑_𝑗=1𝛽^𝑞_𝑗𝜕^𝑞𝑓𝜕𝑥^𝑞_𝑗(𝜃)Σif𝑎𝑞<1,0otherwise,_𝑎(𝜇)=𝑓(𝜃)1+𝑎𝑑_ℝ^𝑑𝐾²(𝑧)𝑑𝑧. (2.4)

Theorem 2.7 (Weak convergence rate of 𝜇_𝑛). Let 𝜇_𝑛 be defined by (1.9), and assume that (A1)–(A6) hold. (1)If there exists 𝑐̃≥0 such that lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=𝑐̃ and if there exists 𝛾>0 such that lim_𝑛→∞𝑛ℎ^𝑑_𝑛𝑃_𝑛(𝛾)=0, then 𝑛ℎ^𝑑_𝑛𝜇_𝑛−𝜇⟶^𝒟𝒩−√𝑐𝐵̃^(𝜇)_𝑎,𝑞,Σ_𝑎^(𝜇). (2.5)(2)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=∞ and if there exists 𝛾>0 such that lim_𝑛→∞ℎ_𝑛^−2𝑞𝑃_𝑛(𝛾)=0, then ℎ_𝑛^−𝑞𝜇_𝑛−𝜇⟶^ℙ𝐵^(𝜇)_𝑎,𝑞. (2.6)

Remark 2.8. If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=𝑐̃≠ or if lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=∞, then 𝑎𝑞<1, so that 𝐵^(𝜇)_𝑎,𝑞 is usually nonzero.

Remark 2.9. Following the proof of Theorem 2.7, it can be shown that the results of Theorem 2.7 also hold when 𝜇_𝑛 is replaced by 𝜇^∗_𝑛 and 𝑎 by zero.

Let us consider the case when (ℎ_𝑛)=(ℎ_𝑛), that is, the case when the same bandwidth is used to define 𝜃_𝑛 and 𝜇_𝑛 (or 𝜃^∗_𝑛 and 𝜇^∗_𝑛). If 𝐾 is a two-order kernel (i.e., if 𝑞=2), then the condition lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=𝑐̃≥0 implies that lim_𝑛→∞𝑛ℎ^𝑑_𝑛𝑃_𝑛(𝛾)=∞ for all 𝛾>0, so that the condition required in Part 1 of Theorem 2.7 is not satisfied: the limit of (𝜇_𝑛−𝜇) (or of (𝜇^∗_𝑛−𝜇)) suitably normalized is then necessary degenerated. This is the first main drawback of using the same bandwidth to estimate the location and the size of the mode: to construct confidence intervals for 𝜇, the use of higher-order kernels is unavoidable.

The estimation of the size of the mode is of course not independent of the estimation of the location, since the estimator 𝜇_𝑛 is constructed with the help of the estimator 𝜃_𝑛. To get a good estimation of the size of the mode, it seems obvious that 𝜃_𝑛 should be computed with a bandwidth (ℎ_𝑛) leading to its optimal convergence rate (or, at least, to a convergence rate close to the optimal one). The main information given by Theorem 2.7 is that, for 𝜇_𝑛 to converge at the optimal rate, the use of a second bandwidth (ℎ_𝑛) is then necessary.

Now, set 𝐵_𝑎,𝑎_,𝑞=𝐵^(𝜃)_𝑎,𝑞𝐵^(𝜇)_𝑎,𝑞,Σ_𝑎,𝑎=Σ_𝑎^(𝜃)00Σ_𝑎^(𝜇)−𝐷,𝐴=²𝑓(𝜃)⁻¹001, (2.7) and, for any 𝑐,𝑐̃≥0, 𝐷(𝑐,𝑐)̃=^{√𝑐𝐼𝑑0}₀𝑐̃ where 𝐼_𝑑 is the 𝑑×𝑑 identity matrix. The following corollary gives a central limit theorem for the couple (𝜃_𝑛,𝜇_𝑛).

Corollary 2.10 (Joint weak convergence rate of 𝜃_𝑛 and 𝜇_𝑛). Let 𝜃_𝑛 and 𝜇_𝑛 be defined by (1.6) and (1.9), respectively, and let the assumptions of the first parts of Theorems 2.5 and 2.7 hold. Then, 𝑛ℎ_𝑛^𝑑+2𝜃_𝑛−𝜃𝑛ℎ^𝑑_𝑛𝜇_𝑛−𝜇^𝒟⟶𝒩𝐷(𝑐,𝑐)̃𝐴𝐵_𝑎,𝑎_,𝑞,𝐴Σ_𝑎,𝑎𝐴. (2.8)

Remark 2.11. Following the proof of Corollary 2.10, it can be shown that the result of Corollary 2.10 also holds when 𝜃_𝑛 is replaced by 𝜃^∗_𝑛, 𝜇_𝑛 by 𝜇^∗_𝑛, and 𝑎 and 𝑎 by zero.

Let us enlighten that, in view of Corollary 2.10, in the case when the couple (𝜃_𝑛,𝜇_𝑛) satisfies a central limit theorem, the estimators 𝜃_𝑛 and 𝜇_𝑛 are asymptotically independent, although, in its definition, the estimator of the size of the mode is heavily connected to that of the location of the mode. This property is not quite surprising since, as pointed out by Vieu [11], the location of the mode gives information on the shape of the density derivative, whereas the size of the mode gives information on the shape of the density itself. This constatation must be related to the fact that the weak (and strong) convergence rate of 𝜃_𝑛 is given by that of the gradient of 𝑓_𝑛, whereas the weak (and strong) convergence rate of 𝜇_𝑛 is given by that of 𝑓_𝑛 itself; the variance of the density estimators converging to zero faster than that of the estimators of the density derivatives, the asymptotic independence of 𝜃_𝑛 and 𝜇_𝑛 is completely explained.

Let us finally say one word on our assumptions on the bandwidths. In the framework of nonrecursive estimation, there is no need to assume that (ℎ_𝑛) and (ℎ_𝑛) are regularly varying sequences. In the case of semirecursive estimation, this assumption cannot obviously be omitted, since the exponents 𝑎 and 𝑎 stand in the expressions of the asymptotic bias 𝐵_𝑎,𝑎_,𝑞 and variance Σ_𝑎,𝑎. This might be seen as a slight inconvenient of semirecursive estimation; however, as it is enlightened in the following section, it turns out to be an advantage, since the asymptotic variances of the semirecursive estimators are smaller than those of the nonrecursive estimators.

2.3. Construction of Confidence Regions and Simulations Studies

The application of Theorems 2.5, 2.7, and Corollary 2.10 allows the construction of confidence regions of the location and of the size of the mode, as well as confidence ellipsoids of the couple (𝜃,𝜇). Hall [29] shows that, in order to construct confidence regions, avoiding bias estimation by a slight undersmoothing is more efficient than explicit bias correction. In the framework of undersmoothing, the asymptotic bias of the estimator is negligible in front of its asymptotic variance; according to the estimation by confidence regions point of view, the parameter to minimize is thus the asymptotic variance. Now, set Σ^∗=[1+𝑎(𝑑+2)]𝐼_𝑑00[]Σ1+𝑎𝑑^𝑎,𝑎, (2.9) and note that, in view of Corollary 2.10 and of Remark 2.11, 𝐴Σ_𝑎,𝑎𝐴 (resp. 𝐴Σ^∗𝐴) is the asymptotic covariance matrix of the semirecursive estimators (𝜃_𝑛,𝜇_𝑛) (resp., of the nonrecursive estimators (𝜃^∗_𝑛,𝜇^∗_𝑛)). In order to construct confidence regions for the location and/or size of the mode, it is thus much preferable to use semirecursive estimators rather than nonrecursive estimators. Simulations studies confirm this theoretical conclusion, whatever the parameter (𝜃, 𝜇 or (𝜃,𝜇)) for which confidence regions are constructed is. For sake of succinctness, we do not give all these simulations results here but focus on the construction of confidence ellipsoid for (𝜃,𝜇); the aim of this example is of course to enlighten the advantage of using semirecursive estimators rather than nonrecursive estimators but also to show how this confidence region gives information on the shape of the density and consequently allows measuring the pertinence of the parameter location of the mode.

To construct confidence regions for (𝜃,𝜇), we consider the case 𝑑=1. The following corollary is a straightforward consequence of Corollary 2.10.

Corollary 2.12. Let 𝜃_𝑛 and 𝜇_𝑛 be defined by (1.6) and (1.9), respectively, and let the assumptions of the first parts of Theorems 2.5 and 2.7 hold. We then have (1+3𝑎)𝑛ℎ³_𝑛𝑓^′′(𝜃)²𝑓(𝜃)∫_ℝ𝐾^'2(𝑥)𝜃𝑑𝑥^𝑛−𝜃²+(1+𝑎)𝑛ℎ_𝑛𝑓(𝜃)∫_ℝ𝐾²(𝑥)𝜇𝑑𝑥^𝑛−𝜇⟶^2𝒟𝜒²(2). (2.10) Moreover, (2.10) still holds when the parameters 𝑓(𝜃) and 𝑓^′′(𝜃) are replaced by consistent estimators.

Remark 2.13. In view of Remark 2.11, in the case when the nonrecursive estimators 𝜃^∗_𝑛 and 𝜇^∗_𝑛 are used, (2.10) becomes 𝑛ℎ³_𝑛𝑓^′′(𝜃)²𝑓(𝜃)∫_ℝ𝐾^'2(𝑥)𝜃𝑑𝑥^∗𝑛−𝜃²+𝑛ℎ_𝑛𝑓(𝜃)∫_ℝ𝐾²(𝑥)𝜇𝑑𝑥^∗𝑛−𝜇⟶^2𝒟𝜒²(2) (2.11) (and, again, this convergence still holds when the parameters 𝑓(𝜃) and 𝑓^′′(𝜃) are replaced by consistent estimators).

Let 𝑓̌_𝑛^′′ (resp., 𝑓̌^∗_𝑛′′) be the recursive estimator (resp., the nonrecursive Rosenblatt's estimator) of 𝑓^′′ computed with the help of a bandwidth ℎ̌_𝑛, and set 𝑃_𝑛=(1+3𝑎)𝑛ℎ³_𝑛𝑓̌_𝑛^′′𝜃_𝑛²𝑓_𝑛𝜃_𝑛∫_ℝ𝐾^'2(𝑥)𝑑𝑥,𝑄_𝑛=(1+𝑎)𝑛ℎ_𝑛𝑓_𝑛𝜃_𝑛∫_ℝ𝐾²(𝑥),𝑃𝑑𝑥_∗𝑛=𝑛ℎ³_𝑛𝑓̌^∗_𝑛′′𝜃^∗_𝑛²𝑓^∗_𝑛𝜃^∗_𝑛∫_ℝ𝐾^'2(𝑥)𝑑𝑥,𝑄^∗_𝑛=𝑛ℎ_𝑛𝑓^∗_𝑛𝜃^∗_𝑛∫_ℝ𝐾²(𝑥).𝑑𝑥 (2.12) Moreover, let 𝑐_𝛼 be such that ℙ(𝑍≤𝑐_𝛼)=1−𝛼, where 𝑍 is 𝜒²(2)-distributed; in view of Corollary 2.12 and Remark 2.13, the sets ℰ_𝛼=𝜃,𝜇𝑃_𝑛𝜃_𝑛−𝜃²+𝑄_𝑛𝜇_𝑛−𝜇²≤𝑐_𝛼ℰ^∗_𝛼=𝜃,𝜇𝑃^∗_𝑛𝜃^∗_𝑛−𝜃²+𝑄^∗_𝑛𝜇^∗_𝑛−𝜇²≤𝑐_𝛼 (2.13) are confidence ellipsoids for (𝜃,𝜇) with asymptotic coverage level 1−𝛼. Let us dwell on the fact that both confidence regions have the same asymptotic level, but the lengths of the axes of the first one (constructed with the help of the semirecursive estimators 𝜃_𝑛 and 𝜇_𝑛) are smaller than those of the second one (constructed with the help of the nonrecursive estimators 𝜃^∗_𝑛 and 𝜇^∗_𝑛).

We now present simulations results. In order to see the relationship between the shape of the confidence ellipsoids and that of the density, the density 𝑓 we consider is the density of the 𝒩(0,𝜎²)-distribution, the parameter 𝜎 taking the values 0.3, 0.4, 0.5, 0.7, 0.75, 1, 1.5, 2, and 2.5. We use the sample size 𝑛=100 and the coverage level 1−𝛼=95% (and thus 𝑐_𝛼=5.99). In each case, the number of simulations is 𝑁=5000. The kernel we use is the standard Gaussian density; the bandwidths are ℎ_𝑛=𝑛^−1/7,ℎlog𝑛^𝑛=𝑛^−1/5,ℎ̌log𝑛^𝑛=𝑛^−1/9. (2.14) Table 1 gives, for each value of 𝜎, the empirical values of 𝜃_𝑛, 𝜃^∗_𝑛, 𝜇_𝑛, 𝜇^∗_𝑛 (with respect to the 5000 simulations) and

Table 1 

𝑏: the empirical length of the 𝜃-axis of the confidence ellipsoid ℰ_5%;

𝑏^∗: the empirical length of the 𝜃-axis of the confidence ellipsoid ℰ^∗_5%;

𝑎: the empirical length of the 𝜇-axis of the confidence ellipsoid ℰ_5%;

𝑎^∗: the empirical length of the 𝜇-axis of the confidence ellipsoid ℰ^∗_5%;

𝑝: the empirical coverage level of the confidence ellipsoid ℰ_5%;

𝑝^∗: the empirical coverage level of the confidence ellipsoid ℰ^∗_5%.

Confirming our theoretical results, we see that the empirical coverage levels of both confidence ellipsoids ℰ^∗_5% and ℰ_5% are similar but that the empirical areas of the ellipsoids ℰ_5% (constructed with the help of the semirecursive estimators) are always smaller than those of the ellipsoids ℰ^∗_5% (constructed with the help of the nonrecursive estimators).

Let us now discuss the interest of the estimation of the size of the mode and that of the joint estimation of the location and size of the mode. Both estimations give information on the shape of the probability density and, consequently, allow measuring the pertinence of the parameter location of the mode. Of course, the parameter 𝜃 is significant only in the case when the high of the peak is large enough; since we consider here the example of the 𝒩(0,𝜎²)-distribution, this corresponds to the case when 𝜎 is small enough. Estimating only the size of the mode gives a first idea of the shape of the density around the location of the mode (for instance, when the size is estimated around 0.16, it is clear that the density is very flat). Now, the shape of the confidence ellipsoids allows getting a more precise idea. As a matter of fact, for small values of 𝜎, the length of the 𝜇-axis is larger than that of the 𝜃-axis; as 𝜎 increases, the length of the 𝜇-axis decreases, and the one of the 𝜃-axis increases (for 𝜎=2.5, the length of the 𝜃-axis is larger than 20 times the one of the 𝜇-axis). Let us underline that these variations of the lengths of the axes are not due to bad estimations results; Table 2 gives the values of the lengths 𝑏 (resp., 𝑏^∗) of the 𝜃-axis, 𝑎 (resp., 𝑎^∗) of the 𝜇-axis of the ellipsoids computed with the semirecursive estimators 𝜃_𝑛 and 𝜇_𝑛 (resp., with the nonrecursive estimators 𝜃^∗_𝑛 and 𝜇^∗_𝑛) in the case when the true values of the parameters 𝑓(𝜃) and 𝑓^′′(𝜃) are used (that is, by straightforwardly applying (2.10) and (2.11)).

Table 2 

2.4. Strong Convergence Rate

To establish the strong convergence rate of 𝜃_𝑛 and 𝜇_𝑛, we need the following additional assumption. (A7)(i)ℎ is differentiable, and its derivative varies regularly with exponent (−𝑎−1).(ii)ℎ is differentiable, and its derivative varies regularly with exponent (−𝑎−1).

The following two theorems give the almost sure convergence rate of 𝜃_𝑛 and of 𝜇_𝑛, respectively. Before stating them, let us enlighten that Proposition 2.3 in Mokkadem and Pelletier [12] ensures that the matrix 𝐺 (and thus the matrix Σ_𝑎^(𝜃)) is nonsingular.

Theorem 2.14 (Strong convergence rate of 𝜃_𝑛). Let 𝜃_𝑛 be defined by (1.6), and assume that (A1), (A2), (A3)(i), (A4), (A5), (A6)(i), and (A7)(i) hold. (1)If there exists 𝑐≥0 such that lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞/[2loglog𝑛]=𝑐, then, with probability one, the sequence ((𝑛ℎ_𝑛^𝑑+2/2loglog𝑛)(𝜃_𝑛−𝜃)) is relatively compact and its limit set is the ellipsoid ℰ=𝜈∈ℝ^𝑑suchthat−𝐷²𝑓(𝜃)𝜈−√𝑐𝐵^(𝜃)_𝑎,𝑞^𝑇Σ_𝑎^(𝜃)−1−𝐷²𝑓(𝜃)𝜈−√𝑐𝐵^(𝜃)_𝑎,𝑞≤1. (2.15)(2)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞/[2loglog𝑛]=∞, then, with probability one, lim_𝑛→∞ℎ_𝑛^−𝑞(𝜃_𝑛−𝜃)=−[𝐷²𝑓(𝜃)]⁻¹𝐵^(𝜃)_𝑎,𝑞.

Theorem 2.15 (Strong convergence rate of 𝜇_𝑛). Let 𝜇_𝑛 be defined by (1.9), and assume that (A1)–(A7) hold. (1)If there exists 𝑐̃≥0 such that lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞/[2loglog𝑛]=𝑐̃ and if there exists 𝛾>0 such that lim_𝑛→∞𝑛ℎ^𝑑_𝑛𝑃_𝑛(𝛾)/loglog𝑛=0, then, with probability one, the sequence 𝑛ℎ^𝑑_𝑛𝜇2loglog𝑛^𝑛−𝜇 (2.16) is relatively compact, and its limit set is the interval [√𝑐𝐵̃^(𝜇)_𝑎,𝑞−Σ_𝑎^(𝜇);√𝑐𝐵̃^(𝜇)_𝑎,𝑞+Σ_𝑎^(𝜇)]. (2)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞/loglog𝑛=∞ and if there exists 𝛾>0 such that lim_𝑛→∞ℎ_𝑛^−2𝑞𝑃_𝑛(𝛾)=0, then, with probability one, lim_𝑛→∞ℎ_𝑛^−𝑞(𝜇_𝑛−𝜇)=𝐵^(𝜇)_𝑎,𝑞.

To establish a law of the iterated logarithm for the couple (𝜃_𝑛,𝜇_𝑛), we need the following additional assumption.(A8)There exists 𝑛₀∈ℕ such that 𝑛≥𝑚≥𝑛₀⟹max𝑚ℎ_𝑚^−(𝑑+2)𝑛ℎ_𝑛^−(𝑑+2);𝑚ℎ_𝑚^−𝑑𝑛ℎ_𝑛^−𝑑=min𝑚ℎ_𝑚^−(𝑑+2);𝑚ℎ_𝑚^−𝑑min𝑛ℎ_𝑛^−(𝑑+2);𝑛ℎ_𝑛^−𝑑. (2.17)

Remark 2.16. Assumption (A8) holds when 𝑎≠𝑎. In the case when 𝑎=𝑎, set ℒ_𝜃(𝑛)=𝑛^𝑎ℎ_𝑛 and ℒ_𝜇(𝑛)=𝑛^𝑎ℎ_𝑛; (A8) is then satisfied when ℒ_𝜃(𝑛)=(ℒ_𝜇(𝑛))^{𝑑/(𝑑+2)} for 𝑛 large enough.

Corollary 2.17 (Joint strong convergence rate of 𝜃_𝑛 and 𝜇_𝑛). Let 𝜃_𝑛 and 𝜇_𝑛 be defined by (1.6) and (1.9), respectively; let the assumptions of Parts 1 of Theorems 2.14 and 2.15 hold, as well as (A8). Then, with probability one, the sequence 12loglog𝑛𝑛ℎ_𝑛^𝑑+2𝜃_𝑛−𝜃𝑛ℎ^𝑑_𝑛𝜇_𝑛−𝜇 (2.18) is relatively compact, and its limit set is the ellipsoid ℰ=𝜈∈ℝ^𝑑+1suchthat𝐴⁻¹𝜈−𝐷(𝑐,𝑐)̃𝐵_𝑎,𝑎_,𝑞^𝑇Σ⁻¹_𝑎,𝑎𝐴⁻¹𝜈−𝐷(𝑐,𝑐)̃𝐵_𝑎,𝑎_,𝑞≤1. (2.19)

Laws of the iterated logarithm for Parzen's nonrecursive kernel mode estimator in the multivariate framework were established by Mokkadem and Pelletier [12]. The technics of demonstration used in the framework of nonrecursive estimators are totally different from those employed to prove Theorem 2.14. This is due to the following fundamental difference between the nonrecursive estimator 𝜃^∗_𝑛 and the semirecursive estimator 𝜃_𝑛: the study of the asymptotic behaviour of 𝜃^∗_𝑛 comes down to the one of a triangular sum of independent variables, whereas the study of the asymptotic behaviour of 𝜃_𝑛 reduces to the one of a sum of independent variables. Of course, this difference is not quite important for the study of the weak convergence rate. But, for the study of the strong convergence rate, it makes the case of the semirecursive estimation much easier than the case of the nonrecursive estimation. In particular, on the opposite to the weak convergence rate, the joint strong convergence rate of the nonrecursive estimators 𝜃^∗_𝑛 and 𝜇^∗_𝑛 cannot be obtained by following the lines of the proof of Theorem 2.14 and remains an open question.

3. Proofs

Let us first note that an important consequence of (A3)(i) is thatif𝛽𝑎<1,thenlim_𝑛→∞1𝑛ℎ^𝛽_𝑛^𝑛_𝑖=1ℎ^𝛽_𝑖=11−𝑎𝛽. (3.1) Moreover, for all 𝜀>0,1𝑛^𝑛_𝑖=1ℎ^𝑞_𝑖=𝑂ℎ^𝑞_𝑛+𝑛^𝜀−1. (3.2) As a matter of fact: (i) if 𝑎𝑞<1, (3.2) follows easily from (3.1); (ii) if 𝑎𝑞>1, since ∑_𝑖ℎ^𝑞_𝑖 is summable, (3.2) holds; (iii) if 𝑎𝑞=1, then ℎ^𝑞_𝑛=𝑂(𝑛^𝜀−1), so that ∑^𝑛_𝑖=1ℎ^𝑞_𝑖=𝑂(𝑛^𝜀), and thus (3.2) follows.

Of course, in view of (A3)(ii), (3.1) and (3.2) also hold when (ℎ_𝑛) and 𝑎 are replaced by (ℎ_𝑛) and 𝑎, respectively.

Our proofs are now organized as follows. Section 3.1 is devoted to the proof of Proposition 2.3. In Section 3.2, we state some preliminary lemmas, which are crucial in the proof of the convergence rates of 𝜃_𝑛 and 𝜇_𝑛 and which are proved in Section 3.6. Section 3.3 is reserved to the proof of Theorems 2.5 and 2.14, Section 3.4 to that of Theorems 2.7 and 2.15, and Section 3.5 to that of Corollaries 2.10 and 2.17.

3.1. Consistency of 𝜃_𝑛 and 𝜇_𝑛: Proof of Proposition 2.3

Since 𝜃_𝑛 is the mode of 𝑓_𝑛 and 𝜃 the mode of 𝑓, we have0≤𝑓(𝜃)−𝑓𝜃_𝑛=𝑓(𝜃)−𝑓_𝑛𝜃_𝑛+𝑓_𝑛𝜃_𝑛−𝑓𝜃_𝑛≤𝑓(𝜃)−𝑓_𝑛(𝜃)+𝑓_𝑛𝜃_𝑛−𝑓𝜃_𝑛≤𝑓(𝜃)−𝑓_𝑛(𝜃)+𝑓_𝑛𝜃_𝑛−𝑓𝜃_𝑛≤2𝑓_𝑛−𝑓_∞. (3.3) The application of Theorem 5 in Mokkadem et al. [27] with |𝛼|=0 and 𝑣_𝑛=log𝑛ensures that, for any 𝛿>0, there exists 𝑐(𝛿)>0 such that ℙ[(log𝑛)‖𝑓_𝑛−𝔼(𝑓_𝑛)‖_∞≥𝛿]≤exp(−𝑐(𝛿)∑^𝑛_𝑖=1ℎ^𝑑_𝑖/(log𝑛)²). In view of (3.1), since 𝑎𝑑<1, we can write 𝑛²(𝛿)∑exp−𝑐^𝑛𝑖=1ℎ^𝑑_𝑖log𝑛²=𝑛²exp−𝑐(𝛿)𝑛ℎ^𝑑_𝑛log𝑛²∑^𝑛_𝑖=1ℎ^𝑑_𝑖𝑛ℎ^𝑑_𝑛=𝑜(1). (3.4) Borell-Cantelli's Lemma then ensures that lim_𝑛→∞‖𝑓_𝑛−𝔼(𝑓_𝑛)‖_∞=0 a.s. Since lim_𝑛→∞‖𝔼(𝑓_𝑛)−𝑓‖_∞=0, it follows from (3.3) that lim_𝑛→∞𝑓(𝜃_𝑛)=𝑓(𝜃) a.s. Since 𝑓 is continuous, since lim_{‖𝑧‖→∞}𝑓(𝑧)=0, and since 𝜃 is the unique mode of 𝑓, we deduce that lim_𝑛→∞𝜃_𝑛=𝜃 a.s. Now, we have 𝜇_𝑛−𝜇≤𝑓_𝑛𝜃_𝑛−𝑓𝜃_𝑛+𝑓𝜃_𝑛−𝑓(𝜃)≤𝑓_𝑛−𝑓_∞+2𝑓_𝑛−𝑓_∞, (3.5) where the last inequality follows from (3.3). Following the proof of the strong uniform convergence of 𝑓_𝑛, we show that lim_𝑛→∞‖𝑓_𝑛−𝑓‖_∞=0 a.s. It follows that lim_𝑛→∞𝜇_𝑛=𝜇 a.s., which concludes the proof of Proposition 2.3.

3.2. Some Preliminary Lemmas

The aim of this section is to state some properties of the density estimators and of their derivatives, which are crucial in the proof of Theorems 2.5–2.15 and of Corollaries 2.10 and 2.17.

3.2.1. Strong Uniform Convergence Rate of the Derivatives of the Density Estimators

For any 𝑑-uplet [𝛼]=(𝛼₁,…,𝛼_𝑑)∈ℕ^𝑑, set |𝛼|=𝛼₁+⋯+𝛼_𝑑 and, for any function 𝑔, let 𝜕^[𝛼]𝑔(𝑥)=𝜕^|𝛼|𝑔/(𝜕𝑥^𝛼1₁…𝜕𝑥^𝛼𝑑_𝑑)(𝑥) denote the [𝛼]-th partial derivative of 𝑔 (if |𝛼|=0, then 𝜕^[𝛼]𝑔=𝑔). In order to prove Theorems 2.5–2.15 and Corollaries 2.10 and 2.17, we need to know the behaviour of 𝜕^[𝛼]𝑓_𝑛 for |𝛼|∈{0,1,2} and that of 𝜕^[𝛼]𝑓_𝑛 for |𝛼|∈{0,1}. For the sake of conciseness, we state the preliminary lemmas 3.1 and 3.2 for 𝜕^[𝛼]𝑔_𝑛 where either 𝑔_𝑛=𝑓_𝑛 and |𝛼|∈{0,1,2} or 𝑔_𝑛=𝑓_𝑛 and |𝛼|∈{0,1}. Moreover, we set (𝑏_𝑛)=(ℎ_𝑛) if 𝑔_𝑛=𝑓_𝑛, and (𝑏_𝑛)=(ℎ_𝑛) if 𝑔_𝑛=𝑓_𝑛.

Lemma 3.1. Let (A1), (A2), (A3)(i), (A4), (A5), and (A6)(i) hold. Moreover, if 𝑔_𝑛=𝑓_𝑛, then let (A3)(ii) and (A6)(ii) hold. We have lim_𝑛→∞𝑛∑^𝑛_𝑖=1𝑏^𝑞_𝑖𝔼𝜕^[𝛼]𝑔_𝑛(𝑥)−𝜕^[𝛼]𝑓(𝑥=())−1^𝑞𝜕𝑞!^[𝛼]𝑑_𝑗=1𝛽^𝑞_𝑗𝜕^𝑞f𝜕𝑥^𝑞_𝑗(𝑥), (3.6) where 𝛽^𝑞_𝑗 is defined in (2.1). Moreover, if we set 𝑀_𝑞=sup_{𝑥∈ℝ𝑑}‖𝐷^𝑞𝜕^[𝛼]𝑓(𝑥)‖, then lim_𝑛→∞𝑛∑^𝑛_𝑖=1𝑏^𝑞_𝑖sup_{𝑥∈ℝ𝑑}𝔼𝜕^[𝛼]𝑔_𝑛(𝑥)−𝜕^[𝛼]𝑓(𝑥≤𝑀)^𝑞𝑞!_ℝ^𝑑‖𝑧‖^𝑞|𝐾(𝑧)|𝑑𝑧. (3.7)

Lemma 3.2. Let 𝑈 be a compact set of  ℝ^𝑑, and assume that (A1), (A2), (A3)(i), (A4), (A5), and (A6)(i) hold. Moreover, if 𝑔_𝑛=𝑓_𝑛, then let (A3)(ii) and (A6)(ii) hold. Then, for all 𝛾>0, we have sup_𝑥∈𝑈𝜕^[𝛼]𝑔_𝑛(𝑥)−𝔼𝜕^[𝛼]𝑔_𝑛(𝑥)=𝑂log𝑛^1+𝛾∑^𝑛_𝑖=1𝑏_𝑖^𝑑+2|𝛼|a.s. (3.8)

Lemma 3.1 is proved in Mokkadem et al. [27], Lemma 3.2 in Section 3.6.

3.2.2. Convergence Rate of (∇𝑓_𝑛(𝜃)−𝔼(∇𝑓_𝑛(𝜃)),𝑓_𝑛(𝜃)−𝔼(𝑓_𝑛(𝜃)))

Lemma 3.3.   (1)Under Assumptions (A1), (A2), (A3)(i), (A4), (A5), and (A6)(i), we have 𝑛ℎ_𝑛^𝑑+2∇𝑓_𝑛(𝜃)−𝔼∇𝑓_𝑛(𝜃)⟶^𝒟𝒩0,Σ_𝑎^(𝜃). (3.9)(2)Under Assumptions (A1)–(A6), we have 𝑊_𝑛=𝑛ℎ_𝑛^𝑑+2∇𝑓_𝑛(𝜃)−𝔼∇𝑓_𝑛(𝜃)𝑛ℎ^𝑑_𝑛𝑓_𝑛(𝜃)−𝔼𝑓_𝑛(𝜃)^𝒟⟶𝒩0,Σ_𝑎,𝑎. (3.10)

Lemma 3.4.   (1)Under Assumptions (A1), (A2), (A3)(i), (A4), (A5), (A6)(i), and (A7)(i), with probability one, the sequence ((𝑛ℎ_𝑛^𝑑+2[∇𝑓_𝑛(𝜃)−𝔼(∇𝑓_𝑛(𝜃))])/2loglog𝑛) is relatively compact, and its limit set is {𝜈∈ℝ^𝑑suchthat𝜈^𝑇[Σ_𝑎^(𝜃)]⁻¹𝜈≤1}. (2)Under Assumptions (A1)–(A7), with probability one, the sequence ((𝑛ℎ^𝑑_𝑛[𝑓_𝑛(𝜃)−𝔼(𝑓_𝑛(𝜃))])/2loglog𝑛) is relatively compact, and its limit set is the interval [−Σ_𝑎^(𝜇);Σ_𝑎^(𝜇)].(3)Under Assumptions (A1)–(A8), with probability one, the sequence 12loglog𝑛𝑛ℎ_𝑛^𝑑+2∇𝑓_𝑛(𝜃)−𝔼∇𝑓_𝑛(𝜃)𝑛ℎ^𝑑_𝑛𝑓_𝑛(𝜃)−𝔼𝑓_𝑛(𝜃) (3.11) is relatively compact, and its limit set is {𝜈∈ℝ^𝑑+1suchthat𝜈^𝑇Σ⁻¹_𝑎,𝑎𝜈≤1}.

3.3. Convergence Rate of 𝜃_𝑛: Proof of Theorems 2.5 and 2.14

In order to prove Theorems 2.5 and 2.14, we first show that the weak and strong asymptotic behaviours of 𝜃_𝑛−𝜃 are given by those of −[𝐷²𝑓(𝜃)]⁻¹∇𝑓_𝑛(𝜃) (see Section 3.3.1) and then deduce the convergence rates of 𝜃_𝑛−𝜃 from those of ∇𝑓_𝑛(𝜃) (see Section 3.3.2).

3.3.1. Relationship between (𝜃_𝑛−𝜃) and ∇𝑓_𝑛(𝜃)

By definition of 𝜃_𝑛, we have ∇𝑓_𝑛(𝜃_𝑛)=0, so that∇𝑓_𝑛𝜃_𝑛−∇𝑓_𝑛(𝜃)=−∇𝑓_𝑛(𝜃). (3.12) For each 𝑖∈{1,…,𝑑}, a Taylor expansion applied to the real valued application 𝜕𝑓_𝑛/𝜕𝑥_𝑖 implies the existence of 𝜀_𝑛(𝑖)=(𝜀_𝑛⁽¹⁾(𝑖),…,𝜀_𝑛^(𝑑)(𝑖))^𝑡 such that 𝜕𝑓_𝑛𝜕𝑥_𝑖𝜃_𝑛−𝜕𝑓_𝑛𝜕𝑥_𝑖(𝜃)=^𝑑_𝑗=1𝜕²𝑓_𝑛𝜕𝑥_𝑖𝜕𝑥_𝑗𝜀_𝑛(𝑖)𝜃_𝑛^(𝑗)−𝜃^(𝑗),𝜀_𝑛^(𝑗)(𝑖)−𝜃^(𝑗)≤𝜃_𝑛^(𝑗)(𝑖)−𝜃^(𝑗)∀𝑗∈{1,…,𝑑}. (3.13) Define the 𝑑×𝑑 matrix 𝐻_𝑛=(𝐻_𝑛^(𝑖,𝑗))_{1≤𝑖,𝑗≤𝑑} by setting 𝐻_𝑛^(𝑖,𝑗)=(𝜕²𝑓_𝑛/𝜕𝑥_𝑖𝜕𝑥_𝑗)(𝜀_𝑛(𝑖)); (3.12) can then be rewritten as𝐻_𝑛𝜃_𝑛−𝜃=−∇𝑓_𝑛(𝜃). (3.14) Now, let 𝑈 be a compact set of ℝ^𝑑 containing 𝜃. The combination of Lemmas 3.1 and 3.2 with |𝛼|=2, 𝑔_𝑛=𝑓_𝑛, and 𝑏_𝑛=ℎ_𝑛 ensures that, for any 𝛾>0 and any 𝜀∈]0,1[,sup_𝑥∈𝑈𝜕^[𝛼]𝑓_𝑛(𝑥)−𝜕^[𝛼]𝑓(𝑥)=𝑂log𝑛^1+𝛾∑^𝑛_𝑖=1ℎ_𝑖^𝑑+4+∑^𝑛_𝑖=1ℎ^𝑞_𝑖𝑛a.s.=𝑂log𝑛^1+𝛾𝑛ℎ_𝑛^𝑑+4+ℎ^𝑞_𝑛+𝑛^𝜀−1=𝑜(1)a.s. (3.15)

Since 𝐷²𝑓 is continuous in a neighbourhood of 𝜃 and since lim_𝑛→∞𝜃_𝑛=𝜃 a.s., (3.15) ensures thatlim_𝑛→∞𝐻_𝑛=𝐷²𝑓(𝜃)a.s. (3.16) In view of (3.14), we can thus state the following lemma.

Lemma 3.5. Under Assumptions (A1), (A2), (A3)(i), (A4), (A5), and (A6)(i), the weak and strong asymptotic behaviour of (𝜃_𝑛−𝜃)^𝑇 is given by that of −[𝐷²𝑓(𝜃)]⁻¹∇𝑓_𝑛(𝜃).

3.3.2. Proof of Theorems 2.5 and 2.14

Theorem 2.5 (resp., Theorem 2.14) straightforwardly follows from the application of Lemma 3.5 of the first part of Lemma 3.3 (resp., of the first part of Lemma 3.4) and of the following lemma.

Lemma 3.6. Let (A1), (A2), (A3)(i), (A4), (A5), and (A6)(i) hold. (1)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞∈]0,∞], then lim_𝑛→∞ℎ_𝑛^−𝑞𝔼(∇𝑓_𝑛(𝜃))=𝐵^(𝜃)_𝑎,𝑞. (2)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞=0, then lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝔼(∇𝑓_𝑛(𝜃))=0.

Let us now prove Lemma 3.6. The application of Lemma 3.1 (with 𝑔_𝑛=𝑓_𝑛 and |𝛼|=1) ensures thatlim_𝑛→∞𝑛∑^𝑛_𝑖=1ℎ^𝑞_𝑖𝔼∇𝑓_𝑛(𝜃)=(−1)^𝑞∇𝑞!^𝑑_𝑗=1𝛽^𝑞_𝑗𝜕^𝑞𝑓𝜕𝑥^𝑞_𝑗(𝜃). (3.17) Let us first consider the case when 𝑎𝑞<1. By application of (3.1), we have lim_𝑛→∞ℎ_𝑛^−𝑞𝔼∇𝑓_𝑛(𝜃)=𝐵^(𝜃)_𝑎,𝑞. (3.18)(i)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞∈]0,∞], then 1−𝑎(𝑑+2+2𝑞)≥0, and thus 𝑎𝑞<1/2; Part 1 of Lemma 3.6 thus follows. (ii)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2+2𝑞=0, then lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2ℎ^𝑞_𝑛=0, and Part 2 of Lemma 3.6 also follows.

It remains to prove Part 2 of Lemma 3.6 in the case when 𝑎𝑞≥1. The application of (3.2) then ensures that, for all 𝜀∈]0,1/2[, 𝑛ℎ_𝑛^𝑑+2∑^𝑛_𝑖=1ℎ^𝑞_𝑖𝑛=𝑂𝑛ℎ_𝑛^𝑑+2ℎ^𝑞_𝑛+𝑛^𝜀−1=𝑜(1). (3.19) In view of (3.17), Part 2 of Lemma 3.6 then follows.

3.4. Convergence Rate of 𝜇_𝑛: Proof of Theorems 2.7 and 2.15

In order to prove Theorems 2.7 and 2.15, we show that the weak and strong convergence rates of 𝜇_𝑛−𝜇 are given by those of 𝑓_𝑛(𝜃)−𝑓(𝜃). More precisely, set𝑅_𝑛=𝜇_𝑛−𝜇−𝑓_𝑛(𝜃)−𝑓(𝜃), (3.20) and assume that the following two lemmas hold.

Lemma 3.7. Let (A1)–(A6) hold. (1)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞∈]0,∞], then lim_𝑛→∞ℎ_𝑛^−𝑞[𝔼(𝑓_𝑛(𝜃))−𝑓(𝜃)]=𝐵^(𝜇)_𝑎,𝑞. (2)If lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=0, then lim_𝑛→∞𝑛ℎ^𝑑_𝑛[𝔼(𝑓_𝑛(𝜃))−𝑓(𝜃)]=0.

Lemma 3.8. Let (A1)–(A6) hold. For all 𝛾>0, 𝑅²_𝑛=𝑂(𝑃_𝑛(𝛾)) a.s.

In order to prove Theorem 2.7, we first note that the application of the second part of Lemma 3.3 yields𝑛ℎ^𝑑_𝑛𝑓_𝑛(𝜃)−𝔼𝑓_𝑛(𝜃)⟶^𝒟𝒩0,Σ_𝑎^(𝜇). (3.21) In view of Lemma 3.7, it follows that if lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=𝑐̃≥0, then𝑛ℎ^𝑑_𝑛𝑓_𝑛(𝜃)−𝑓(𝜃)⟶^𝒟𝒩√𝑐𝐵̃^(𝜇)_𝑎,𝑞,Σ_𝑎^(𝜇), (3.22) and if lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2𝑞=∞, thenℎ_𝑛^−𝑞𝑓_𝑛(𝜃)−𝑓(𝜃)⟶^ℙ𝐵^(𝜇)_𝑎,𝑞. (3.23) Part 1 (resp., Part 2) of Theorem 2.7 straightforwardly follows from the combination of (3.22) (resp., of (3.23)) and of Lemma 3.8.

The proof of Theorem 2.15 follows that of Theorem 2.7 (except that the second part of Lemma 3.4 stands instead of (3.21)); this proof is thus omitted.

It remains to prove Lemmas 3.7 and 3.8. The proof of Lemma 3.7 follows that of Lemma 3.6 and is omitted. We now prove Lemma 3.8. We first note that 𝑅_𝑛=𝑓_𝑛(𝜃_𝑛)−𝑓_𝑛(𝜃); a Taylor's expansion implies the existence of 𝜁_𝑛 such that ‖𝜁_𝑛−𝜃_𝑛‖≤‖𝜃_𝑛−𝜃‖ and 𝑅_𝑛=𝜃_𝑛−𝜃^𝑇∇𝑓_𝑛𝜁_𝑛=𝜃_𝑛−𝜃^𝑇∇𝑓_𝑛𝜁_𝑛−∇𝑓𝜁_𝑛+∇𝑓𝜁_𝑛−∇𝑓(𝜃). (3.24) Let 𝒱 be a compact set that contains 𝜃; for 𝑛 large enough, we get 𝑅_𝑛=𝑂𝜃_𝑛−𝜃sup_𝑥∈𝒱∇𝑓_𝑛(𝑥)−∇𝑓(𝑥)+𝜁_𝑛𝜃−𝜃=𝑂_𝑛−𝜃sup_𝑥∈𝒱∇𝑓_𝑛(𝑥)−∇𝑓(𝑥)+𝜃_𝑛−𝜃². (3.25) To get an upper bound of |𝑅_𝑛|, we thus need to establish upper bounds of ‖𝜃_𝑛−𝜃‖ and of sup_𝑥∈𝒱‖∇𝑓_𝑛(𝑥)−∇𝑓(𝑥)‖.(i)Let us recall that the a.s. convergence rate of (𝜃_𝑛−𝜃) is given by that of [𝐷²𝑓(𝜃)]⁻¹∇𝑓_𝑛(𝜃) (see Lemma 3.5). Theorem 2.14 can be applied to obtain the exact a.s. convergence rate of 𝜃_𝑛−𝜃. However, to avoid assuming (A7)(i), we apply here Lemmas 3.1 and 3.2 (with |𝛼|=1 and (𝑔_𝑛,𝑏_𝑛)=(𝑓_𝑛,ℎ_𝑛)) and get the following upper bound of the a.s. convergence rate of ∇𝑓_𝑛(𝜃) and thus of 𝜃_𝑛−𝜃: for any 𝛾>0 and 𝜀∈]0,(1+𝑎(𝑑+2))/2[, 𝜃_𝑛−𝜃²=𝑂log𝑛^1+𝛾∑^𝑛_𝑖=1ℎ_𝑖^𝑑+2+∑^𝑛_𝑖=1ℎ^𝑞_𝑖𝑛²=𝑂log𝑛^1+𝛾𝑛ℎ_𝑛^𝑑+2+ℎ^𝑞_𝑛+𝑛^𝜀−12𝑤ℎa.s.=𝑂_𝑛,𝛾a.s. (3.26)(ii)Noting that sup_𝑥∈𝒱∇𝑓_𝑛(𝑥)−∇𝑓(𝑥)≤sup_𝑥∈𝒱∇𝑓_𝑛(𝑥)−𝔼∇𝑓_𝑛(𝑥)+sup_𝑥∈𝒱𝔼∇𝑓_𝑛(𝑥)−∇𝑓(𝑥), (3.27) the application of Lemmas 3.1 and 3.2 with |𝛼|=1, (𝑔_𝑛,𝑏_𝑛)=(𝑓_𝑛,ℎ_𝑛) ensures that, for any 𝛾>0 and 𝜀∈]0,(1+𝑎(𝑑+2))/2[,sup_𝑥∈𝒱∇𝑓_𝑛(𝑥)−∇𝑓(𝑥)²=𝑂log𝑛^1+𝛾𝑛ℎ_𝑛^𝑑+2+∑^𝑛_𝑖=1ℎ^𝑞_𝑖𝑛²a.s.=𝑂log𝑛^1+𝛾𝑛ℎ_𝑛^𝑑+2+ℎ^𝑞_𝑛+𝑛^𝜀−12𝑤ℎa.s.=𝑂_𝑛,𝛾a.s. (3.28) It follows from (3.25), (3.26), and (3.28) that 𝑅_𝑛²=𝑂𝑤ℎ_𝑛,𝛾𝑤ℎ_𝑛,𝛾+𝑤ℎ_𝑛,𝛾²𝑃a.s.=𝑂_𝑛𝛾a.s., (3.29) which concludes the proof of Lemma 3.8.

3.5. Joint Convergence Rate of 𝜃_𝑛 and 𝜇_𝑛: Proof of Corollaries 2.10 and 2.17

In view of (3.14) and (3.20), we have 𝐻_𝑛0𝜃01𝜇^𝑛−𝜃_𝑛=−𝜇−∇𝑓_𝑛(𝜃)𝑓_𝑛(𝜃)(𝜃)+0−𝑓𝑅_𝑛, (3.30) and, in view of (3.16), the weak and strong asymptotic behaviour of 𝜇^{𝜃𝑛−𝜃}_𝑛−𝜇 is given by that of 𝐷²𝑓(𝜃)⁻¹001−∇𝑓_𝑛(𝜃)𝑓_𝑛(𝜃)(𝜃)+0−𝑓𝑅_𝑛=−𝐷²𝑓(𝜃)⁻¹∇𝑓_𝑛(𝜃)−𝔼∇𝑓_𝑛(𝜃)𝑓_𝑛(𝜃)−𝔼𝑓_𝑛(𝜃)+−𝐷²𝑓(𝜃)⁻¹𝔼∇𝑓_𝑛(𝜃)𝔼𝑓_𝑛(𝜃)−𝑓(𝜃)+𝑅_𝑛. (3.31) Under the assumptions of Corollary 2.10, the application of Part 1 of Lemma 3.6, that of Part 1 of Lemma 3.7, and that of Lemma 3.8 ensure that lim_𝑛→∞𝑛ℎ_𝑛^𝑑+2−𝐷²𝑓(𝜃)⁻¹𝔼∇𝑓_𝑛(𝜃)𝑛ℎ^𝑑_𝑛𝔼𝑓_𝑛(𝜃)−𝑓(𝜃)+𝑅_𝑛=𝐷(𝑐,𝑐)̃𝐴𝐵_𝑎,𝑎_,𝑞. (3.32) Corollary 2.10 then follows from the application of Part 2 of Lemma 3.3. The proof of Corollary 2.17 follows that of Corollary 2.10 (except that Part 3 of Lemma 3.4 is applied instead of Part 2 of Lemma 3.3) and is omitted.

3.6. Proof of Lemmas 3.2–3.4

3.6.1. Proof of Lemma 3.2

Set 𝑣_𝑛=[∑^𝑛_𝑖=1𝑏_𝑖^𝑑+2|𝛼|]^1/2[(log𝑛)^1+𝛾]^−1/2. Applying Proposition 3 in Mokkadem et al. [27], it holds that for any 𝛿>0, there exists 𝑐(𝛿)>0 such that ℙsup_𝑥∈𝑈𝑣_𝑛𝜕^[𝛼]𝑔_𝑛(𝑥)−𝔼𝜕^[𝛼]𝑔_𝑛(𝑥)(𝛿)∑≥𝛿≤exp−𝑐^𝑛𝑖=1𝑏_𝑖^𝑑+2|𝛼|2𝑣²_𝑛. (3.33) Since lim_𝑛→∞∑^𝑛_𝑖=1𝑏_𝑖^𝑑+2|𝛼|/(𝑣²_𝑛log𝑛)=∞, we have, for 𝑛 large enough, 𝑐(𝛿)∑^𝑛_𝑖=1𝑏_𝑖^𝑑+2|𝛼|/2𝑣²_𝑛≥2log𝑛, and Lemma 3.2 follows from the application of Borel-Cantelli's Lemma.

3.6.2. Proof of Lemma 3.3

We only prove the second part of Lemma 3.3: the proof of its first part can be easily deduced. We first establish thatlim_𝑛→∞𝔼𝑊_𝑛𝑊^𝑇_𝑛=Σ_𝑎,𝑎 (3.34) and then check that (𝑊_𝑛) satisfies Lyapounov's condition. Set 𝑌_𝑘,𝑛=1𝑛ℎ_𝑛^−𝑑−2ℎ_𝑘^−𝑑−1∇𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘,𝑍_𝑘,𝑛=1𝑛ℎ_𝑛^−𝑑ℎ_𝑘^−𝑑𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼𝐾𝜃−𝑋_𝑘ℎ_𝑘, (3.35) and note that 𝔼𝑊_𝑛𝑊^𝑇_𝑛=^𝑛_𝑘=1𝔼𝑌_𝑘,𝑛𝑌^𝑇_𝑘,𝑛𝔼𝑌_𝑘,𝑛𝑍_𝑘,𝑛𝔼𝑌^𝑇_𝑘,𝑛𝑍_𝑘,𝑛𝔼𝑍²_𝑘,𝑛. (3.36) Now, for any 𝑠,𝑡∈{1,…,𝑑}, we have 𝔼𝜕𝐾𝜕𝑥_𝑠𝜃−𝑋_𝑘ℎ_𝑘𝜕𝐾𝜕𝑥_𝑡𝜃−𝑋_𝑘ℎ_𝑘=_ℝ𝑑𝜕𝐾𝜕𝑥_𝑠𝜃−𝑦ℎ_𝑘𝜕𝐾𝜕𝑥_𝑡𝜃−𝑦ℎ_𝑘𝑓𝑦𝑑𝑦=ℎ^𝑑_𝑘𝑓(𝜃)𝐺_𝑠,𝑡+𝑜ℎ^𝑑_𝑘, (3.37) and since 𝔼[(𝜕𝐾/𝜕𝑥_𝑠)(𝜃−𝑋_𝑘/ℎ_𝑘)]=𝑂(ℎ^𝑑_𝑘), we deduce that𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘∇𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘^𝑇=𝑓(𝜃)𝐺ℎ^𝑑_𝑘[1+𝑜(1)], (3.38) which implies that lim_𝑛→∞∑^𝑛_𝑘=1𝔼(𝑌_𝑘,𝑛𝑌^𝑇_𝑘,𝑛)=𝑓(𝜃)[1+𝑎(𝑑+2)]⁻¹𝐺. In the same way, we have𝔼𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼𝐾𝜃−𝑋_𝑘ℎ_𝑘²=ℎ^𝑑_𝑘𝑓(𝜃)_ℝ𝑑𝐾²(𝑧)𝑑𝑧[1+𝑜(1)], (3.39) and thus lim_𝑛→∞∑^𝑛_𝑘=1𝔼(𝑍²_𝑘,𝑛)=𝑓(𝜃)[1+𝑎𝑑]⁻¹∫_ℝ𝑑𝐾²(𝑧)𝑑𝑧. Moreover, set ℎ^∗_𝑛=min(ℎ_𝑛,ℎ_𝑛); we have 𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘𝐾𝜃−𝑋_𝑘ℎ_𝑘=ℎ_𝑘^∗𝑑_ℝ𝑑ℎ∇𝐾^∗𝑘ℎ_𝑘𝑧𝐾ℎ^∗_𝑘ℎ_𝑘𝑧𝑓𝜃−ℎ^∗_𝑘𝑧𝑑𝑧. (3.40) Noting that 𝑓(𝜃−ℎ^∗_𝑘𝑧)=𝑓(𝜃)+ℎ^∗_𝑘𝑅_𝑘(𝜃,𝑧) with |𝑅_𝑘(𝜃,𝑧)|≤‖∇𝑓‖_∞‖𝑧‖, we get 𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘𝐾𝜃−𝑋_𝑘ℎ_𝑘=ℎ_𝑘^∗𝑑𝑓(𝜃)_ℝ𝑑ℎ∇𝐾^∗𝑘ℎ_𝑘𝑧𝐾ℎ^∗_𝑘ℎ_𝑘𝑧𝑑𝑧+ℎ^∗_𝑘ℝ𝑑ℎ∇𝐾^∗𝑘ℎ_𝑘𝑧𝐾ℎ^∗_𝑘ℎ_𝑘𝑧𝑅_𝑘(𝜃,𝑧)𝑑𝑧. (3.41) Since the function 𝑧↦[∇𝐾(𝑧)]𝐾(𝑧 is odd (in each coordinate), the first right-handed integral is zero, and since ℎ^∗_𝑘 equals either ℎ_𝑘 or ℎ_𝑘, we get 𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘𝐾𝜃−𝑋_𝑘ℎ_𝑘≤ℎ_𝑘^∗(𝑑+1)∇𝑓_∞‖𝐾‖_∞ℝ𝑑‖𝑧‖‖∇𝐾(𝑧)‖𝑑𝑧+‖∇𝐾‖_∞ℝ𝑑‖𝑧‖|𝐾(𝑧)|𝑑𝑧=𝑂ℎ_𝑘^∗(𝑑+1). (3.42) We then deduce that𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼𝐾𝜃−𝑋_𝑘ℎ_𝑘=𝑂minℎ_𝑘,ℎ_𝑘^𝑑+1+𝑂ℎ^𝑑_𝑘ℎ^𝑑_𝑘=𝑂ℎ_𝑘^(𝑑+1)/2ℎ_𝑘^(𝑑+1)/2, (3.43) and thus, in view of (3.1), ^𝑛_𝑘=1𝔼𝑌_𝑘,𝑛𝑍_𝑘,𝑛1=𝑂𝑛ℎ_𝑛^−𝑑−2𝑛ℎ_𝑛^{−𝑑𝑛𝑘=1}ℎ_𝑘^{−(𝑑+1)/2}ℎ_𝑘^(1−𝑑)/2=𝑜(1), (3.44) which concludes the proof of (3.34). Let us now check that (𝑊_𝑛) satisfies the Lyapounov's condition. Set 𝑝>2. Since 𝐾 and ∇𝐾 are bounded and integrable, we have ∫_ℝ𝑑‖∇𝐾(𝑧)‖^𝑝𝑑𝑧<∞ and ∫_ℝ𝑑|𝐾(𝑧)|^𝑝𝑑𝑧<∞. It follows that ^𝑛_𝑘=1𝔼𝑌_𝑘,𝑛^𝑝1=𝑂𝑛ℎ_𝑛^{−𝑑−2𝑛𝑝/2𝑘=1}ℎ_𝑘^{(−𝑑−1)𝑝}_ℝ𝑑∇𝐾𝜃−𝑦ℎ_𝑘^𝑝𝑓𝑦1𝑑𝑦=𝑂𝑛ℎ_𝑛^{−𝑑−2𝑛𝑝/2𝑘=1}ℎ_𝑘^{(−𝑑−1)𝑝}ℎ^𝑑_𝑘=𝑜_𝑛𝑘=1(1),𝔼𝑍_𝑘,𝑛^𝑝1=𝑂𝑛ℎ_𝑛^{−𝑑𝑛𝑝/2𝑘=1}ℎ_𝑘^−𝑑𝑝_ℝ𝑑𝐾𝜃−𝑦ℎ_𝑘^𝑝𝑓𝑦1𝑑𝑦=𝑂𝑛ℎ_𝑛^{−𝑑𝑛𝑝/2𝑘=1}ℎ_𝑘^−𝑑𝑝ℎ^𝑑_𝑘=𝑜(1), (3.45) which concludes the proof of Lemma 3.3.

3.6.3. Proof of Lemma 3.4

We only prove the third part of Lemma 3.4: the proofs of its first two parts can be easily deduced. Set (𝜃)0Γ=𝑓𝐺0_ℝ𝑑𝐾²(𝑧)𝑑𝑧,Δ_𝑛=1𝑛ℎ_𝑛^−𝑑−2𝐼_𝑑001𝑛ℎ_𝑛^−𝑑,𝑄_𝑛=ℎ_𝑛^−𝑑−2𝐼_𝑑00ℎ_𝑛^−𝑑, (3.46) let (𝜀_𝑛) be a sequence of ℝ^𝑑+1-valued, independent, and 𝒩(0,Γ)-distributed random vectors, and set 𝑆_𝑛=∑^𝑛_𝑘=1𝑄_𝑘𝜀_𝑘 and 𝐵_𝑛=𝔼(𝑆_𝑛𝑆^𝑇_𝑛). We clearly have lim_𝑛→+∞Δ_𝑛𝐵_𝑛Δ_𝑛=Σ_𝑎,𝑎. Moreover, all the eigenvalues 𝜌_𝑛^(𝑖)(1≤𝑖≤𝑑+1) of 𝐵_𝑛 satisfy lim_𝑛→+∞loglog𝜌_𝑛^(𝑖)/loglog𝑛=1. The application of Theorem 1 in Mokkadem and Pelletier [30] then ensures that, with probability one, the sequenceΔ_𝑛𝑆_𝑛2loglog𝑛 (3.47) is relatively compact, and its limit set is the ellipsoid {𝜈∈ℝ^𝑑+1suchthat𝜈^𝑇Σ⁻¹_𝑎,𝑎𝜈≤1}. Now, set 𝑉_𝑘=ℎ_𝑘^−𝑑/2∇𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼∇𝐾𝜃−𝑋_𝑘ℎ_𝑘ℎ_𝑘^−𝑑/2𝐾𝜃−𝑋_𝑘ℎ_𝑘−𝔼𝐾𝜃−𝑋_𝑘ℎ_𝑘 (3.48) and Γ_𝑘=𝔼(𝑉_𝑘𝑉^𝑇_𝑘). In view of (3.38), (3.39), and (3.43), we have lim_𝑘→∞Γ_𝑘=Γ. It follows that ∃𝑘₀≥1 such that forall𝑘≥𝑘₀, Γ_𝑘 is inversible; without loss of generality, we assume 𝑘₀=1, and set 𝑈_𝑘=Γ_𝑘^−1/2𝑉_𝑘. Set 𝑝∈]2,4[ and let ℒ be a slowly varying function; we have 𝔼𝑈_𝑘^𝑝𝑘loglog𝑘^𝑝/2ℎ=𝑂^{𝑘−𝑑𝑝/2}𝔼∇𝐾(𝜃−𝑋_𝑘)/ℎ_𝑘^𝑝+ℎ_𝑘^{−𝑑𝑝/2}𝔼𝐾(𝜃−𝑋_𝑘)/ℎ_𝑘^𝑝𝑘loglog𝑘^𝑝/2ℎ=𝑂^{𝑘𝑑−𝑑𝑝/2}+ℎ_𝑘^{𝑑−𝑑𝑝/2}𝑘loglog𝑘^𝑝/2=𝑂ℒ(𝑘)𝑘^{−[1+((𝑝/2)−1)(1−𝑎𝑑)]}+𝑘^{−[1+((𝑝/2)−1)(1−}𝑎_𝑑)] (3.49) so that ∑_𝑘(𝑘loglog𝑘)^−𝑝/2𝔼(‖𝑈_𝑘‖^𝑝)<∞. By application of Theorem 2 of Einmahl [31], we deduce that ∑^𝑛_𝑘=1𝑈_𝑘−∑^𝑛_𝑘=1𝜂_𝑘=𝑜(𝑛loglog𝑛) a.s., where 𝜂_𝑘 are independent and 𝒩(0,𝐼_𝑑+1)-distributed random vectors. It follows that^𝑛_𝑘=1Γ^1/2Γ_𝑘^−1/2𝑉_𝑘−^𝑛_𝑘=1𝜀_𝑘=𝑜𝑛loglog𝑛a.s. (3.50) Now, Δ_𝑛^𝑛_𝑘=1𝑄_𝑘Γ^1/2Γ_𝑘^−1/2𝑉_𝑘−^𝑛_𝑘=1𝑄_𝑘𝜀_𝑘=Δ_𝑛^𝑛_𝑘=1𝑄_𝑘Γ^1/2Γ^−1/2_k𝑉_𝑘−𝜀_𝑘=Δ_𝑛^𝑛_𝑘=1𝑄_𝑘^𝑘_𝑗=1Γ^1/2Γ_𝑗^−1/2𝑉_𝑗−𝜀_𝑗−^𝑘−1_𝑗=1Γ^1/2Γ_𝑗^−1/2𝑉_𝑗−𝜀_𝑗with⁰_𝑗=1=0=Δ_𝑛^𝑛−1_𝑘=1𝑄_𝑘−𝑄_𝑘+1^𝑘_𝑗=1Γ^1/2Γ_𝑗^−1/2𝑉_𝑗−𝜀_𝑗+Δ_𝑛𝑄_𝑛^𝑛_𝑗=1Γ^1/2Γ_𝑗^−1/2𝑉_𝑗−𝜀_𝑗=Δ_𝑛^𝑛−1_𝑘=1𝑄_𝑘−𝑄_𝑘+1𝑜𝑘loglog𝑘+Δ_𝑛𝑄_𝑛𝑜𝑛loglog𝑛a.s. (3.51) Moreover, Δ_𝑛^𝑛−1_𝑘=1𝑄_𝑘−𝑄_𝑘+1𝑜=𝑘loglog𝑘ℎ_𝑛^𝑑+2𝑛^𝑛−1_𝑘=1ℎ_𝑘^{−(𝑑+2)/2}−ℎ^{−(𝑑+2)/2}_𝑘+1𝑜00𝑘loglog𝑘ℎ^𝑑_𝑛𝑛^𝑛−1_𝑘=1ℎ_𝑘^−𝑑/2−ℎ^−𝑑/2_𝑘+1=𝑜ℎ_𝑛^𝑑+2loglog𝑛^𝑛−1_𝑘=1ℎ_𝑘^{−(𝑑+2)/2}−ℎ^{−(𝑑+2)/2}_𝑘+100𝑜ℎ^𝑑_𝑛loglog𝑛^𝑛−1_𝑘=1ℎ_𝑘^−𝑑/2−ℎ^−𝑑/2_𝑘+1. (3.52) Set 𝜙(𝑠)=[ℎ(𝑠)]^{−(𝑑+2)/2} and 𝜙(𝑠)=[ℎ(𝑠)]^−𝑑/2, and let 𝑢_𝑘∈[𝑘,𝑘+1]; since 𝜙^′ and 𝜙^′ vary regularly with exponent (𝑎(𝑑+2)/2−1) and (𝑎𝑑/2−1), respectively, we have ^𝑛−1_𝑘=1ℎ_𝑘^{−(𝑑+2)/2}−ℎ^{−(𝑑+2)/2}_𝑘+1=𝑂^𝑛−1_𝑘=1𝜙^′𝑢_𝑘=𝑂^𝑛₁𝜙^′(𝑠)𝑑𝑠=𝑂ℎ_𝑛^{−(𝑑+2)/2},_{𝑛−1𝑘=1}ℎ_𝑘^−𝑑/2−ℎ^−𝑑/2_𝑘+1=𝑂^𝑛−1_𝑘=1𝜙^′𝑢_𝑘=𝑂^𝑛₁𝜙^′(𝑠)𝑑𝑠=𝑂ℎ_𝑛^−𝑑/2 (3.53) so that Δ_𝑛^𝑛−1_𝑘=1𝑄_𝑘−𝑄_𝑘+1𝑜𝑘loglog𝑘=𝑜loglog𝑛. (3.54) Since Δ_𝑛𝑄_𝑛[𝑜(𝑛loglog𝑛)]=𝑜(loglog𝑛), we deduce that Δ_𝑛∑^𝑛_𝑘=1𝑄_𝑘Γ^1/2Γ_𝑘^−1/2𝑉_𝑘−Δ2loglog𝑛^𝑛∑^𝑛_𝑘=1𝑄_𝑘𝜀_𝑘(1)2loglog𝑛=𝑜a.s. (3.55) The application of (3.47) then ensures that, with probability one, the sequence Δ_𝑛∑^𝑛_𝑘=1𝑄_𝑘Γ^1/2Γ_𝑘^−1/2𝑉_𝑘2loglog𝑛 (3.56) is relatively compact, and its limit set is ℰ={𝜈∈ℝ^𝑑+1suchthat𝜈^𝑇Σ⁻¹_𝑎,𝑎𝜈≤1}. Since Δ_𝑛∑^𝑛_𝑘=1𝑄_𝑘𝑉_𝑘=Δ2loglog𝑛^𝑛∑^𝑛_𝑘=1𝑄_𝑘Γ^1/2Γ_𝑘^−1/2𝑉_𝑘+Δ2loglog𝑛^𝑛∑^𝑛_𝑘=1𝑄_𝑘𝐼_𝑑+1−Γ^1/2Γ_𝑘^−1/2𝑉_𝑘2loglog𝑛 (3.57) with lim_𝑘→∞(𝐼_𝑑+1−Γ^1/2Γ_𝑘^−1/2)=0, Lemma 3.4 follows.