Optimal Estimators for Threshold-Based Quality Measures

Abstract

We consider a problem in parametric estimation: given 𝑛 samples from an unknown distribution, we want to estimate which distribution, from a given one-parameter family, produced the data. Following Schulman and Vazirani (2005), we evaluate an estimator in terms of the chance of being within a specified tolerance of the correct answer, in the worst case. We provide optimal estimators for several families of distributions on ℝ. We prove that for distributions on a compact space, there is always an optimal estimator that is translation invariant, and we conjecture that this conclusion also holds for any distribution on ℝ. By contrast, we give an example showing that, it does not hold for a certain distribution on an infinite tree.

1. Introduction

Estimating probability distribution functions is a central problem in statistics. Specifically, beginning with an unknown probability distribution on an underlying space 𝑋, one wants to be able to do two things: first, given some empirical data sampled from the unknown probability distribution, estimate which one of a presumed set of possible distributions produced the data; and second, obtain bounds on how good this estimate is. For example, the maximum likelihood estimator selects the distribution that maximizes the probability (among those under consideration) of producing the observed data. Depending on what properties of the estimator one is trying to evaluate, this may or may not be optimal. An extensive literature, dating back to the early 20th century, addresses problems of this sort; see for example [1–6].

In this paper, we consider one such problem. We presume that samples are coming from an unknown “translate” of a fixed known distribution. The challenge is to guess the translation parameter. More precisely, we are given a distribution 𝜇 on a space 𝑋, along with an action of a group 𝐺 on 𝑋, which defines a set of translated distributions 𝜇_𝜃 as follows: 𝜇_𝜃(𝐴)=𝜇({𝑥∶𝜃x∈𝐴}), (1.1) for 𝐴⊂𝑋.

Thus, in this context an estimator is a (measurable) function 𝑒∶𝑋^𝑛→𝐺; the input 𝐱=(𝑥₁,…,𝑥_𝑛) is the list of samples, and the output 𝑒(𝐱) is the estimate of 𝜃, the translation parameter. For the majority of the paper, we will study the case of 𝐺=ℝ acting by translations (changes in location) on 𝑋=ℝ, and the group action will be written additively, as seen beginning from Section 2.

We are interested in finding good estimators; thus we need a way of measuring an estimator's quality. A common way to do this is to measure the mean squared error, in which case an optimal estimator minimizes this error. Various results are known in this case; for instance, the maximum likelihood estimator (which agrees with the sample mean estimator), 𝑒(𝐱)=1𝑛𝑥_𝑖−𝔼𝜇, (1.2) minimizes the mean squared error if 𝜇 is a Gaussian distribution on ℝ.

In this paper, we investigate a different and natural measure of quality whereby we consider an estimator to succeed or fail according to whether or not its estimate is within a certain threshold 𝛿>0 of the correct answer. We then define the quality of the estimator to be the chance of success in the worst case. This notion was introduced in [7] to analyze certain approximation algorithms in computer science. Precisely, the 𝛿-quality of 𝑒 is defined as 𝑄_𝛿(𝑒)=inf_𝜃𝑄^𝜃_𝛿(𝑒)=inf_𝜃Pr𝑑(𝑒(𝐱),𝜃)<𝛿∶𝑥_𝑖arechosenfrom𝜇_𝜃=inf_𝜃𝜇^𝑛_𝜃({𝐱∶𝑑(𝑒(𝐱),𝜃)<𝛿}), (1.3) where 𝑑 is a metric on 𝑋 and 𝜇^𝑛_𝜃 is the product measure 𝜇_𝜃×⋯×𝜇_𝜃 on 𝑋^𝑛. (In the case of perverse measures, 𝜇, we must consider the probability as the sup of the intersection of the set {𝑑(𝑒(𝐱),𝜃)<𝛿} with all measurable sets. We will ignore this caveat throughout. Indeed, we primarily focus on absolutely continuous measures (as [8, 9] have done, e.g.) and purely atomic measures.) Note that, depending on context, it is sometimes advantageous to define the quality using a closed interval rather than an open one; for example, in the discrete case we could then interpret 𝑄^𝜃₀(𝑒) as the probability that 𝑒 is exactly equal to 𝜃. We write 𝑄(𝑒) when the value of 𝛿 is unambiguous. For fixed 𝛿, an (𝑛-sample) estimator 𝑒 is optimal if 𝑄_𝛿(𝑒)≥𝑄_𝛿(𝑒′) for all (𝑛-sample) estimators 𝑒′. Many authors use the term minimax to describe optimal estimators. Note that much of the literature on this subject uses the notion of loss functions and the associated risk 𝑅=1−𝑄; our point of view is equivalent but more convenient for our purposes.

Motivated initially by analyzing an approximate algorithm for determining the average matching size in a graph, Schulman and Vazirani [7] introduce the stronger notion of a majorizing estimator, which is optimal (by the above definition) simultaneously for all 𝛿>0. This was previously studied by Pitman [5], who considered several different optimality criteria and, for each one, constructed optimal “shift-invariant” estimators (defined below). Schulman and Vazirani focus on the Gaussian distribution and prove that the mean estimator is the unique majorizing estimator in this case.

In the first part of this paper, we investigate the optimal estimators for several different classes of distributions on ℝ. We conjecture that there is always an optimal estimator 𝑒 that is shift invariant, that is, 𝑒 satisfies 𝑒𝑥₁+𝑐,…,𝑥_𝑛+𝑐=𝑒𝑥₁,…,𝑥_𝑛+𝑐, (1.4) for all 𝑐,𝑥_𝑖∈ℝ. These estimators are typically easier to analyze than general estimators because the quality is the same everywhere, that is, 𝑄(𝑒)=𝑄^𝜃(𝑒) for every 𝜃. Conditions under which invariant minimax estimators can be obtained have been studied, for example, in [10–12]. Indeed, some of our existence results follow from the quite general Hunt-Stein theorem [12, Theorem 9.5.5], but we give constructions that are very natural and explicit. We obtain general bounds on the quality of shift invariant estimators (Section 2) and general estimators (Section 3), and then we apply these bounds to several families of distributions (Section 4). In each case, we are able to construct an optimal estimator that is shift-invariant. These examples include the Gaussian and exponential distributions, among others.

These results motivate our study of shift-invariant estimators on other spaces; these are estimators that are equivariant with respect to the induced diagonal action of 𝐺 on either the left or the right on 𝑋^𝑛. That is, a left-invariant estimator satisfies 𝑒𝑔𝐱=𝑔𝑒(𝐱), (1.5) where 𝑔𝑥₁,…,𝑥_𝑛=𝑔𝑥₁,…,𝑔𝑥_𝑛. (1.6) Right invariance is defined similarly.

In Section 5, we show that on a compact space 𝑋, if 𝑒 is an estimator for 𝜇, then there is always a shift-invariant estimator with quality at least as high as that of 𝑒. The idea is to construct a shift-invariant estimator 𝑠 as an average of the translates of 𝑒; this is essentially a simple proof of a special case of the Hunt-Stein theorem. As there is no invariant probability measure on ℝ, the proof does not extend to the real case.

Finally, in the last section, we give an example due to Schulman which shows that (on noncompact spaces) there may be no shift-invariant estimator that is optimal. It continues to be an interesting problem to determine conditions under which one can guarantee the existence of a shift-invariant estimator that is optimal.

2. The Real Case: Shift-Invariant Estimators

Let 𝐺=𝑋=ℝ, and consider the action of 𝐺 on 𝑋 by translations. Because much of this paper is concerned with this context, we spell out once more the parameters of the problem. We assume that 𝛿>0 is fixed throughout. We are given a probability distribution 𝜇 on ℝ, and we are to guess which distribution 𝜇_𝜃 produced a given collection 𝐱=(𝑥₁,…,𝑥_𝑛) of data, where 𝜇_𝜃(𝐴)=𝜇({𝑥∶𝑥+𝜃∈𝐴}). An estimator is a function 𝑒∶ℝ^𝑛→ℝ, and we want to maximize its quality, which is given by 𝑄(𝑒)=inf_𝜃𝑄^𝜃(𝑒)=inf_𝜃Pr|𝑒(𝐱)−𝜃|<𝛿∶𝑥_𝑖arechosenfrom𝜇_𝜃=inf_𝜃𝜇^𝑛_𝜃({𝐱∶|𝑒(𝐱)−𝜃|<𝛿}). (2.1)

First we present some notation. We will write the group action additively and likewise the induced diagonal action of 𝐺 on ℝ^𝑛; in other words, if 𝐱=(𝑥₁,…,𝑥_𝑛)∈ℝ^𝑛 and 𝑎∈ℝ, then 𝐱+𝑎 denotes the point (𝑥₁+𝑎,…,𝑥_𝑛+𝑎)∈ℝ^𝑛. Similarly, if 𝑌⊂ℝ^𝑛 and 𝐴⊂ℝ, then 𝑌+𝐴={𝐲+𝑎∶𝐲∈𝑌,𝑎∈𝐴}. We also use the “interval notation” (𝐱+𝑎,𝐱+𝑏) for the set {𝐱+𝑡∶𝑎<𝑡<𝑏}; this is a segment of length (𝑏−𝑎)√𝑛 in ℝ^𝑛 if 𝑎 and 𝑏 are finite. If 𝑓∶ℝ^𝑛→ℝ is any function, and 𝜃∈ℝ, define 𝑓_𝜃(𝐱)=𝑓(𝐱−𝜃). If 𝑓∶ℝ→ℝ, then define 𝑓^[𝑛]∶ℝ^𝑛→ℝ by 𝑓^[𝑛](𝐱)=𝑓(𝑥₁)𝑓(𝑥₂)⋯𝑓(𝑥_𝑛).

We now establish our upper bound on the quality of shift-invariant estimators. Note that a shift invariant estimator has the property that 𝑒(𝐱−𝑒(𝐱))=0. Also note that a shift-invariant estimator is determined uniquely by its values on the coordinate hyperplane 𝑋₀=𝐱∈ℝ^𝑛∶𝑥₁=0, (2.2) and that a shift-invariant estimator exists for any choice of such values on 𝑋₀. In addition, for 𝑒 shift-invariant, 𝜇^𝑛_𝜃({𝐱∶|𝑒(𝐱)−𝜃|<𝛿})=𝜇^𝑛_𝜃({𝐱∶|𝑒(𝐱−𝜃)|<𝛿})=𝜇^𝑛({𝐱∶|𝑒(𝐱)|<𝛿}), (2.3) so the quality can be ascertained by setting 𝜃=0.

Definition 2.1. For fixed 𝑛, let 𝒜 denote the collection of all Borel subsets 𝐴 of the form 𝐴=_𝐱∈𝑋0𝐱+𝑓(𝐱)−𝛿,𝑓(𝐱)+𝛿, (2.4) where 𝑓∶𝑋₀→ℝ is a Borel function. For fixed 𝜇 and 𝑛, define 𝑆_𝜇,𝑛=𝑆_𝜇,𝑛(𝛿)=sup_𝐴∈𝒜𝜇^𝑛(𝐴). (2.5)

Theorem 2.2. Let 𝜇 and 𝑛 be given, then any shift-invariant 𝑛-sample estimator 𝑒 satisfies 𝑄(𝑒)≤𝑆_𝜇,𝑛.

Proof. Due to the observation above, it suffices to bound the quality of 𝑒 at 𝜃=0. But this quality is just 𝜇^𝑛(𝐴), where 𝐴=𝑒⁻¹((−𝛿,𝛿)). Note that 𝐴=_𝐱∈𝑋0(𝐱−𝑒(𝐱)−𝛿,𝐱−𝑒(𝐱)+𝛿), (2.6) and in particular 𝐴∈𝒜. Thus, the quality of 𝑒 is at most 𝑆_𝜇,𝑛.

Theorem 2.3. Let 𝜇 and 𝑛 be given. If the sup in Definition 2.1 is achieved, then there is a shift-invariant 𝑛-sample estimator with quality 𝑆_𝜇,𝑛. For any 𝜖>0, there is a shift-invariant 𝑛-sample estimator 𝑒 with quality greater than 𝑆_𝜇,𝑛−𝜖.

Proof. For a given 𝐴∈𝒜, let 𝑓 be the corresponding Borel function (see Definition 2.1). Define the estimator 𝑒 to be −𝑓(𝐱) on 𝑋₀ and then extend to all of ℝ^𝑛 to make it shift invariant. Note that 𝐴⊂𝑒⁻¹((−𝛿,𝛿)), so 𝑄(𝑒)≥𝜇^𝑛(𝐴). The theorem now follows from the definition of 𝑆_𝜇,𝑛.

3. The Real Case: General Estimators

In this section, we obtain a general upper bound on the quality of randomized estimators, still in the case 𝐺=𝑋=ℝ. The arguments are similar to those of the previous section.

Again 𝛿 is fixed throughout. A randomized estimator is a function 𝑋^𝑛×Ω→ℝ, where Ω is a probability space of estimators; thus for fixed 𝜔∈Ω, 𝑒=𝑒(̃⋅,𝜔) is an estimator. The 𝛿-quality of a randomized estimator 𝑒̃ is 𝑄(𝑒)̃=inf_𝜃𝑄^𝜃(𝑒)̃, (3.1) where 𝑄^𝜃(𝑒)̃=𝔼𝜇^𝑛_𝜃{𝐱∣|𝑒̃(𝐱)−𝜃|<𝛿}. (3.2)

Definition 3.1. For fixed 𝑛, let ℬ=𝐵⊂ℝ^𝑛∶𝐵∩(𝐵+2𝑘𝛿)=∅forallnonzerointegers𝑘. (3.3) For fixed 𝜇 and 𝑛, define 𝑇_𝜇,𝑛=𝑇_𝜇,𝑛(𝛿)=sup_𝐵∈ℬ𝜇^𝑛(𝐵). (3.4)

Comparing with Definition 2.1, we observe that 𝒜⊂ℬ and hence 𝑆_𝜇,𝑛≤𝑇_𝜇,𝑛.

Theorem 3.2. Let 𝜇 and 𝑛 be given. Any 𝑛-sample randomized estimator satisfies 𝑄(𝑒)̃≤𝑇_𝜇,𝑛.

Proof. We will give a complete proof in the case that 𝜇 is defined by a density function 𝑓 and then indicate the modifications required for the general case. The difference is purely technical; the ideas are the same.
Consider first a nonrandomized estimator 𝑒. The performance of 𝑒 at 𝜃 is 𝜇^𝑛_𝜃(𝑒⁻¹((𝜃−𝛿,𝜃+𝛿))). To simplify notation, we will let 𝐼_𝑒,𝜃 denote the set 𝑒⁻¹((𝜃−𝛿,𝜃+𝛿)), and we will suppress the subscript 𝑒 when no ambiguity exists. Since 𝑄(𝑒) is an infimum, the average performance of 𝑒 at the 𝑘 points 𝜃_𝑖=2𝛿𝑖 (𝑖=1,2,…,𝑘) is at least 𝑄(𝑒)as follow: 𝑄(𝑒)≤1𝑘^𝑘_𝑖=1𝜇^𝑛_𝜃𝑖𝐼_𝜃𝑖. (3.5)
Now, we use the density function 𝑓. Recall that 𝑓_𝜃(𝑥)=𝑓(𝑥−𝜃). Define 𝑓 on ℝ^𝑛 by 𝑓(𝐱)=max_𝑖𝑓_𝜃^[𝑛]_𝑖(𝐱)=max_𝑖𝑓_𝜃𝑖𝑥₁𝑓_𝜃𝑖𝑥₂⋯𝑓_𝜃𝑖𝑥_𝑛. (3.6)
Since the 𝐼_𝜃𝑖 are disjoint, we now have 𝑄(𝑒)≤1𝑘^𝑘_{𝑖=1𝐼𝜃𝑖}𝑓_𝜃^[𝑛]_𝑖(𝐱)1𝑑𝐱≤𝑘^𝑘_{𝑖=1𝐼𝜃𝑖}𝑓(𝐱)1𝑑𝐱≤𝑘_{∪𝐼𝜃𝑖}𝑓(𝐱)1𝑑𝐱≤𝑘_ℝ^𝑛𝑓(𝐱)=1𝑑𝐱𝑘_𝑥1≤0𝑓(𝐱)1𝑑𝐱+𝑘_{0<𝑥1<2𝛿𝑘}𝑓(𝐱)1𝑑𝐱+𝑘_{𝑥1≥2𝛿𝑘}𝑓(𝐱)𝑑𝐱. (3.7)
We will bound the middle term by 𝑇_𝜇,𝑛 and show that the first and last terms go to zero (independently of 𝑒) as 𝑘 gets large. The bound on the middle term is a consequence of the following claim.
Claim 3.3. For any 𝑎∈ℝ, _{{𝑎≤𝑥1≤𝑎+2𝛿}}𝑓(𝐱)𝑑𝐱≤𝑇_𝜇,𝑛. (3.8) To prove the claim, set 𝑍={𝐱∈ℝ^𝑛∶𝑎≤𝑥₁<𝑎+2𝛿}, and set 𝑍_𝑖={𝐱∈𝑍∶𝑖asthesmallestindexsuchthat𝑓(𝐱)=𝑓_𝜃^[𝑛]_𝑖(𝐱)}. Thus, the 𝑍_𝑖 are disjoint and cover 𝑍. Now, _{{𝑎≤𝑥1≤𝑎+2𝛿}}𝑓(𝐱)𝑑𝐱=_𝑖𝑍𝑖𝑓(𝐱)𝑑𝐱=_𝑖𝑍𝑖𝑓_𝜃^[𝑛]_𝑖(𝐱)𝑑𝐱=_{𝑖𝑍𝑖−𝜃𝑖}𝑓^[𝑛](𝐱)=𝑑𝐱_{∪(𝑍𝑖−𝜃𝑖)}𝑓^[𝑛](𝐱)𝑑𝐱≤𝑇_𝜇,𝑛. (3.9) The last equality follows from the fact that the 𝑍_𝑖−𝜃_𝑖 are disjoint (recall that 𝜃_𝑖=2𝛿𝑖), and the final step follows because the set 𝐵=∪(𝑍_𝑖−𝜃_𝑖) is in ℬ. This proves the claim.
Next, we show that (1/𝑘)∫_{𝑥1≤0}𝑓(𝐱)𝑑𝐱 approaches zero as 𝑘 grows. Recall that 𝜃_𝑖=2𝛿𝑖, and set 𝑧_𝑖=∫_{𝑥1≤0}𝑓^[𝑛]_2𝛿𝑖(𝐱)𝑑𝐱=∫_{{𝑥1≤−2𝛿𝑖}}𝑓^[𝑛](𝐱)𝑑𝐱. The function 𝑓 is a probability density function, so 𝑓 is nonnegative and has total integral 1. The Dominated Convergence Theorem then implies that the sequence {𝑧_𝑖} is decreasing to 0. Bounding 𝑓(𝐱) by ∑^𝑘_𝑖=1𝑓_𝜃^[𝑛]_𝑖(𝐱), we have 1𝑘_{𝑥1≤0}𝑓(𝐱)1𝑑𝐱≤𝑘_{𝑥1^𝑘_≤0}𝑖=1𝑓_𝜃^[𝑛]_𝑖(𝐱)1𝑑𝐱=𝑘^𝑘_𝑖=1𝑧_𝑖⟶0. (3.10)
A similar argument shows that the term (1/𝑘)∫_{{𝑥1≥2𝛿𝑘}}𝑓(𝐱)𝑑𝐱 goes to 0 as 𝑘 grows. Since (3.7) holds for all 𝑘, we have 𝑄(𝑒)≤𝑇_𝜇,𝑛 for any estimator 𝑒.
We have shown that for any 𝜖>0, we can find 𝑘 depending on 𝜖 and 𝑓, such that the average performance of an arbitrary estimator 𝑒 on the 𝑘 points 𝜃_𝑖=2𝛿𝑖 is bounded above by 𝑇_𝜇,𝑛+2𝜖. Now, for a randomized estimator 𝑒̃, the quality is bounded above by its average performance on the same 𝑘 points, and that performance can be no better than the best estimator's performance. We conclude that 𝑄(𝑒)̃≤𝑇_𝜇,𝑛+2𝜖, and the theorem follows.
The proof is now complete in the case that 𝜇 has a density 𝑓. In general, the argument requires minor technical adjustments. The first step that requires modification is the definition of the function 𝑓. Let 𝜈=^𝑘_𝑖=1𝜇^𝑛_𝜃𝑖,𝑔_𝑖=𝑑𝜇^𝑛_𝜃𝑖𝑑𝜇,𝑓=max𝑔₁,…𝑔_𝑘. (3.11) Then 𝑓⋅𝜈=𝜇, and we work with 𝜇 rather than 𝑓. From here, one defines the 𝑍_𝑖 accordingly, and the remainder of the argument goes through with corresponding changes.

4. The Real Case: Examples

We have obtained bounds on quality for general estimators and for shift-invariant ones. In this section, we give several situations where the bounds coincide, and therefore the optimal shift-invariant estimators constructed in Section 2 are in fact optimal estimators, as promised by the Hunt-Stein theorem. These examples include many familiar distributions, and they provide evidence for the following conjecture.

Conjecture 4.1. Let 𝜇 be a distribution on ℝ. Then there is an optimal estimator for 𝜇 that is shift invariant.

4.1. Warmup: Unimodal Densities—One Sample

Our first class of examples generalizes Gaussian distributions and many others. The argument works only with one sample, but we will refine it in Section 4.2. Note that the optimal estimator in this case is the maximum likelihood estimator.

We say that a density function is unimodal if for all 𝑦, {𝑥∶𝑓(𝑥)≥𝑦} is convex.

Example 4.2. Let 𝜇 be defined by a unimodal density function 𝑓. Then there is a shift-invariant one-sample estimator that is optimal.

Proof. We first show that 𝑇_𝜇,1=𝑆_𝜇,1. It follows from the definition of ℬ that any set 𝐵∈ℬ must have Lebesgue measure less than or equal to 2𝛿. Since 𝑓 is unimodal, ∫_𝐵𝑓(𝑥)𝑑𝑥 is maximized by concentrating 𝐵 around the peak of 𝑓; thus the best 𝐵 will be an interval that includes the peak of 𝑓. But any interval in ℬ is contained in 𝒜 and thus 𝑇_𝜇,1≤𝑆_𝜇,1. Since 𝑆_𝜇,1≤𝑇_𝜇,1 always, we have 𝑇_𝜇,1=𝑆_𝜇,1.
Now, recalling that 𝑆_𝜇,1 and 𝑇_𝜇,1 are defined as suprema, we observe that the above argument shows that if one is achieved, then so is the other. Therefore, the result follows from Theorems 2.3 and 3.2.

4.2. A Sufficient Condition

The next class is more restrictive than the preceding, but with the stronger hypothesis we get a result for arbitrary 𝑛. Any Gaussian distribution continues to satisfy the hypothesis.

Example 4.3. Let 𝜇 be a distribution defined by a density function of the form 𝑓=𝑒^𝜆(𝑥) with 𝜆′(𝑥) continuous and decreasing. Then for any 𝑛, there is a shift-invariant 𝑛-sample estimator that is optimal.

Proof. For any fixed 𝐱∈𝑋₀, we define a function ℎ_𝐱∶ℝ→ℝ by ℎ_𝐱(𝑡)=𝑓^[𝑛](𝐱+𝑡)=𝑒^{𝜆(𝑥1+𝑡)+⋯+𝜆(𝑥𝑛+𝑡)}. (4.1) Since ℎ_𝐱′(𝑡)=𝑒^{𝜆(𝑥1+𝑡)+⋯+𝜆(𝑥𝑛+𝑡)}𝜆^′𝑥₁+𝑡+⋯+𝜆^′𝑥_𝑛+𝑡 (4.2) and 𝜆′ is decreasing, it is clear that for each 𝐱, ℎ_𝐱′(𝑡)=0 for at most one value of 𝑡. Since ℎ_𝐱(𝑡)→0 as 𝑡→±∞, it follows that for any 𝐱, ℎ_𝐱 is a unimodal function of 𝑡.
Now the argument is similar to Example 4.2. We will show that 𝑇_𝜇,𝑛=𝑆_𝜇,𝑛. Since 𝑓^[𝑛] restricted to each orbit 𝐱+ℝ is unimodal as we have just shown, a set 𝐵∈ℬ on which the integral of 𝑓^[𝑛] is maximized is obtained by choosing an interval from each orbit. To make this more precise, for each 𝐱∈𝑋₀, let 𝑡_𝐱 be the center of the length 2𝛿 interval 𝐼=(𝑡_𝐱−𝛿,𝑡_𝐱+𝛿) that maximizes ∫_𝐼ℎ_𝐱𝑑𝑡. Then let 𝐴=_𝐱∈𝑋0𝐱+𝑡_𝑥−𝛿,𝐱+𝑡_𝑥+𝛿. (4.3) Now 𝐴∈𝒜, and moreover, 𝜇^𝑛(𝐴)≥𝜇^𝑛(𝐵) for any 𝐵∈ℬ because ∫_{𝐴∩(𝐱+ℝ)}𝑓^[𝑛]≥∫_{𝐵∩(𝐱+ℝ)}𝑓^[𝑛] for each 𝐱∈𝑋₀.
Thus, sup_𝐵∈ℬ{𝜇^𝑛(𝐵)} is achieved by 𝐵=𝐴∈𝒜, and it follows that 𝑆_𝜇,𝑛=𝑇_𝜇,𝑛 and that the best shift-invariant estimator is optimal.

4.3. Monotonic Distributions on ℝ⁺

The third class of examples generalizes the exponential distribution, defined by the density 𝑓(𝑥)=𝜆𝑒^−𝜆𝑥 for 𝑥≥0 and 𝑓(𝑥)=0 for 𝑥<0. The optimal estimator in this case is not the maximum likelihood estimator. (Note that in a typical estimation problem involving a family of exponential distributions, one is trying to estimate the "scale" parameter 𝜆 rather than the "location" 𝜃).

Example 4.4. Let 𝜇 be defined by a density function 𝑓 that is decreasing for 𝑥≥0 and identically zero for 𝑥<0. Then for any 𝑛, there is a shift-invariant 𝑛-sample estimator that is optimal.

Proof. We construct the estimator as follows: for 𝐱∈ℝ^𝑛, define 𝑒(𝐱)=min{𝑥₁,…,𝑥_𝑛}−𝛿. Note that this is shift invariant; therefore 𝑄(𝑒) can be computed at 𝜃=0. That is, it suffices to show that 𝑄⁰(𝑒)=𝑇_𝜇,𝑛.
Let 𝐵={𝐱∈ℝ^𝑛∶0≤min{𝑥₁,…,𝑥_𝑛}<2𝛿}. Note that 𝐵=𝑒⁻¹([−𝛿,𝛿)), and so 𝜇^𝑛(𝐵) is the quality of 𝑒. Note also that 𝐵∈ℬ (in fact 𝐵∈𝒜), so certainly 𝜇^𝑛(𝐵)≤𝑇_𝜇,𝑛. We will show that any 𝐶∈ℬ can be modified to a set 𝐶′∈ℬ such that 𝐶′⊂𝐵 and 𝜇^𝑛(𝐶)≤𝜇^𝑛(𝐶′). It then follows that 𝑇_𝜇,𝑛≤𝜇^𝑛(𝐵), and this will complete the proof.
So, let 𝐶∈ℬ, and define 𝐶′={𝐱∈𝐵∶𝐱+2𝑘𝛿∈𝐶forsome𝑘∈𝑍}. Note that𝑘 is determined uniquely by 𝐱. Now 𝐶′⊂𝐵 is in ℬ, and by our hypotheses on 𝑓, if 𝐱∈𝐵, then 𝑓^[𝑛](𝐱)≥𝑓^[𝑛](𝐱+2𝑘𝛿) for every integer 𝑘. Therefore, 𝜇^𝑛(𝐶′)−𝜇^𝑛(𝐶)=∫_𝐶′[𝑓^[𝑛](𝐱)−𝑓^[𝑛](𝐱+2𝑘𝛿)]𝑑𝜇^𝑛≥0.

4.4. Discrete Distributions

Here, we discuss purely atomic distributions on finite sets of points. Because we are only trying to guess within 𝛿 of the correct value of 𝜃, there are many possible choices of estimators with the same quality. Among the optimal ones is the maximum likelihood estimator.

Example 4.5. Let 𝜇 be a distribution on a finite set of points 𝑍. There is a shift-invariant one-sample estimator that is optimal. Furthermore, if all of the pairwise distances between points of 𝑍 are distinct, then for every 𝑛 there is a shift-invariant 𝑛-sample estimator that is optimal.

Proof. We first treat the case 𝑛=1. Since 𝜇 is discrete, the supremum defining 𝑆_𝜇,1 is attained; therefore, by Theorems 2.3 and 3.2, it suffices to show that every estimator has quality at most 𝑆_𝜇,1.
Let 𝑍={𝑧₁,…,𝑧_𝑟}, and for any 𝑧∈𝑍, let 𝑝_𝑧 denote the mass at 𝑧. For a finite set, we use |⋅| to denote the cardinality. Suppose that 𝑒 is any estimator.
Lemma 4.6. Let 𝑌 be any finite subset of ℝ, then 𝑄(𝑒)≤𝑆_𝜇,1|𝑌+𝑍||𝑌|, (4.4) where 𝑌+𝑍 denotes the set {𝑦+𝑧∣𝑦∈𝑌,𝑧∈𝑍}.Proof of Lemma 4.6. To prove the lemma, we estimate the average quality 𝑄^𝜃(𝑒) over 𝜃∈𝑌. We have _𝜃∈𝑌𝑄^𝜃(𝑒)=_𝜃∈𝑌𝜇𝑒⁻¹(𝜃−𝛿,𝜃+𝛿)−𝜃=_{𝜃∈𝑌𝑧}𝑝_𝑧, (4.5) with the inner part of the last sum taken over those 𝑧∈𝑍 that lie in 𝑒⁻¹(𝜃−𝛿,𝜃+𝛿)−𝜃. Using 𝑥 to denote 𝜃+𝑧, this condition becomes 𝑒(𝑥)∈(𝜃−𝛿,𝜃+𝛿), and the right hand side above may be rewritten as _{𝜃∈𝑌𝑧}𝑝_z=_{𝜃∈𝑌𝑥}𝑝_𝑥−𝜃=_{𝑥∈𝑌+𝑍𝜃}𝑝_𝑥−𝜃, (4.6) with the inner sum now taken over all 𝜃 with 𝑒(𝑥)∈(𝜃−𝛿,𝜃+𝛿). This latter condition implies that 𝑧 is within 𝛿 of 𝑥−𝑒(𝑥). But by definition, 𝑆_𝜇,1 is the maximum measure of any interval of length 2𝛿. Hence, for any fixed 𝑥∈𝑌+𝑍, the inner sum is at most 𝑆_𝜇,1, and the entire sum is thus bounded above by 𝑆_𝜇,1⋅|𝑌+𝑍|. Dividing by |𝑌| gives a bound for the average quality over 𝑌, and since 𝑄(𝑒) is defined as an infimum, the lemma follows.
We now apply the lemma to complete the example. Let 𝑘 be a natural number, and let 𝑌_𝑘=ℎ₁𝑧₁+⋯+ℎ_𝑟𝑧_𝑟∶ℎ_𝑖∈ℤ,0≤ℎ_𝑖<𝑘. (4.7) Note that 𝑌_𝑘⊆𝑌_𝑘+1 and |𝑌_𝑘|≤𝑘^𝑟. It follows that for any 𝜖>0, there exists 𝑘 such that |𝑌_𝑘+1|/|𝑌_𝑘|<1+𝜖, for otherwise, |𝑌_𝑘| would grow at least exponentially in 𝑘. Using the fact that 𝑌_𝑘+𝑍⊆𝑌_𝑘+1, the lemma applied to 𝑌_𝑘 implies that 𝑄(𝑒)≤𝑆_𝜇,1(1+𝜖). Therefore,𝑄(𝑒)≤𝑆_𝜇,1, and this finishes the case 𝑛=1.
Lastly, we consider an arbitrary 𝑛. If we are given samples 𝑥₁,…,𝑥_𝑛 and if any 𝑥_𝑖≠𝑥_𝑗 for some 𝑖 and 𝑗, then by our hypothesis the shift 𝜃 is uniquely determined. Thus, we may assume that any optimal estimator picks the right 𝜃 in these cases, and the only question is what value the estimator returns if all the samples are identical. The above analysis of the one sample case can be used to show that any optimal shift-invariant estimator is optimal.

5. The Compact Case

So far, we have dealt only with distributions on ℝ, where the shift parameter is a translation. In every specific case that we have analyzed, we have found a shift-invariant estimator among the optimal estimators. In this section, we prove that if 𝐺=𝑋 is a compact group acting on itself by (left) multiplication, then at least for measures defined by density functions, there is always a shift-invariant estimator as good as any given estimator. In Section 6, we show that the compactness hypothesis cannot be eliminated entirely; we do not know how much it can be weakened, if at all.

We will continue to use both 𝐺 and 𝑋 as notations, in order to emphasize the distinction between the two roles played by this object. Eaton [2] discusses estimators in a context in which the group 𝐺 acts on both the parameter space Θ and the sample space 𝑋. In his work, the sample space 𝑋 is an arbitrary homogeneous space (i.e., a space with a transitive 𝐺-action). In this generality, shift-invariant estimators may not exist, since there may not even exist a function from 𝑋^𝑛 to Θ that preserves the 𝐺 action. For this reason, we choose to identify the sample space with the group 𝐺.

As usual 𝐺 acts diagonally on 𝑋^𝑛; we denote the orbit space by 𝑌. An element 𝐲 of 𝑌 is an equivalence class 𝐲=[𝐱]={(𝑔𝑥₁,…,𝑔𝑥_𝑛)∶𝑔∈𝐺}, which we identify with 𝐺 via (𝑔𝑥₁,…,𝑔𝑥_𝑛)↦𝑔𝑥₁. For 𝐱=(𝑥₁,…,𝑥_𝑛)∈𝑋, we denote by 𝐱₀ the point 𝑥₁⁻¹𝐱; thus, 𝐱₀ is in the orbit of 𝐱 and has first coordinate 1. The set 𝑋₀={𝐱₀∶𝐱∈𝑋}⊂𝑋^𝑛 is naturally identified with 𝑌.

Equip 𝐺=𝑋 with a left- and right-invariant metric 𝑑, meaning that 𝑑(𝑔𝑥,𝑔𝑦)=𝑑(𝑥,𝑦)=𝑑(𝑥𝑔,𝑦𝑔) for all 𝑥,𝑦,𝑔∈𝐺. Let 𝐵(𝑔)=𝐵_𝛿(𝑔) denote the ball of radius 𝛿 around 𝑔∈𝐺. If 𝑆 is a subset of a measure space (𝑇,𝛼), then we denote the measure of 𝑆 variously by 𝛼(𝑆), ∫_𝑆𝑑𝛼, or ∫_𝑇𝜒_𝑆𝑑𝛼. (The notation 𝜒_𝑆 refers to the characteristic function of the set 𝑆.)

Fix 𝛿 and 𝑛, and let 𝜇 be an arbitrary measure on 𝑋. The following technical lemma says that to evaluate an integral over 𝑋^𝑛, we can integrate over each 𝐺-orbit and then integrate the result over the orbit space.

Lemma 5.1. There exist measures 𝜈 on 𝑌 and 𝛼_𝐲 on each orbit 𝐲 such that for any function 𝐹 on 𝑋^𝑛, _𝑋𝑛𝐹(𝐱)𝑑𝜇^𝑛=_𝑌𝐺𝐹𝑔𝐱₀𝑑𝛼_𝐲𝑑𝜈. (5.1)

Proof. Let 𝜑∶𝑋^𝑛→𝑋^𝑛 be defined by 𝜑(𝐱)=(𝑥₁,𝑥₁⁻¹𝑥₂,…,𝑥₁⁻¹𝑥_𝑛) and let 𝜋 be the image of 𝜇^𝑛 with respect to 𝜑, that is, 𝜋(𝐴)=𝜇^𝑛{𝜑∈𝐴} for all Borel 𝐴⊂𝑋^𝑛, then 𝐹𝜑(𝐱)𝑑𝜇^𝑛(𝐱)=𝐹(𝐲)𝑑𝜋(𝐲) (5.2) for nonnegative Borel functions 𝐹 on 𝑋^𝑛. Taking 𝐹(𝐲)=𝐹(𝑦₁,𝑦₁𝑦₂,…,𝑦₁𝑦_𝑛) yields 𝐹(𝐱)𝑑𝜇^𝑛(𝐱)=𝐹𝑦₁,𝑦₁𝑦₂,…,𝑦₁𝑦_𝑛(𝐲)𝜈𝑦𝑑𝜋=𝑑₂,…,𝑦_𝑛𝐹𝑦₁,𝑦₁𝑦₂,…,𝑦₁𝑦_𝑛𝑑𝛼_{(𝑦2,…,𝑦𝑛)}𝑦₁, (5.3) for some measures 𝜈 and 𝛼. The right-hand side can then be written as 𝑑𝜈𝐱_𝑋00𝐺𝐹𝑔𝐱₀𝑑𝛼_𝐱0𝑔, (5.4) where 𝜈 is an image of 𝜈 with respect to the function (𝑦₂,…,𝑦_𝑛)↦(1,𝑦₂,…,𝑦_𝑛) and 𝛼_𝐱0=𝛼_{(𝑦2,…,𝑦𝑛)} for 𝐱₀=(1,𝑦₂,…,𝑦_𝑛), completing the proof.

Lemma 5.2. If 𝑠 is a shift-invariant (𝑛-sample) estimator, then 𝑄(𝑠)=_𝑌𝐺𝜒_{𝐵(𝑠(𝐱0)}⁻¹₎𝑑𝛼_𝐱0𝑑𝜈. (5.5)

Proof. Since 𝑠 is shift invariant, its quality can be computed at the identity. Thus, 𝑄(𝑠)=𝑄¹(𝑠)=𝜇^𝑛(𝑠⁻¹(𝐵(1)))=∫_𝑋𝑛𝜒_{𝑠−1(𝐵(1))}𝑑𝜇^𝑛. By Lemma 5.1, this integral can be decomposed as _𝑌𝐺𝜒_{𝑠−1(𝐵(1))}𝑔𝐱₀𝑑𝛼_𝐱0𝑑𝜈. (5.6) Now, note that 𝑔𝐱₀∈𝑠⁻¹(𝐵(1)) if and only if 𝑔𝑠(𝐱₀)∈𝐵(1) if and only if 𝑔∈𝐵(𝑠(𝐱₀)⁻¹). Thus, the integral above is the same as the one in the statement of the lemma, and we are done.

We are now ready to prove the result. Note that we do not prove that optimal estimators exist—only that if they exist, then one of them is shift invariant.

Theorem 5.3. Let 𝐺=𝑋 be a compact group, let 𝛿 and 𝑛 be given, and let 𝜇 be defined by a density function. If 𝑒∶𝑋^𝑛→𝐺 is any estimator, then there exists a shift-invariant estimator 𝑠 with 𝑄(𝑠)≥𝑄(𝑒).

Proof. Let 𝑒∶𝑋^𝑛→𝐺 be any estimator. For each group element 𝛾∈𝐺, we define a shift-invariant estimator 𝑠_𝛾 that agrees with 𝑒 on the coset 𝛾𝑋₀ as follows: 𝑠_𝛾𝑔,𝑔𝑥₂,…,𝑔𝑥_𝑛=𝑔𝛾⁻¹𝑒𝛾,𝛾𝑥₂,…,𝛾𝑥_𝑛. (5.7) We will show that there exists 𝛾 such that 𝑄(𝑠_𝛾)≥𝑄(𝑒).
Denote by 𝜌 the invariant (Haar) measure on 𝐺. Since 𝑄(𝑒) is defined as an infimum, we have 𝑄(𝑒)≤_𝜃∈𝐺𝑄^𝜃(𝑒)𝑑𝜌=_{𝜃∈𝐺𝑋𝑛}𝜒_𝜃⁻¹_{𝑒−1(𝐵(𝜃))}(𝐱)𝑑𝜇^𝑛=𝑑𝜌_{𝑋𝑛𝜃∈𝐺}𝜒_𝜃⁻¹_{𝑒−1(𝐵(𝜃))}(𝐱)𝑑𝜌𝑑𝜇^𝑛=_{𝑌𝐺𝜃∈𝐺}𝜒_𝜃⁻¹_{𝑒−1(𝐵(𝜃))}𝑔𝐱₀𝑑𝜌𝑑𝛼_𝐱0𝑑𝜈, (5.8) where the last equality comes from Lemma 5.1. The condition that 𝑔𝐱₀∈𝜃⁻¹𝑒⁻¹(𝐵(𝜃)) is equivalent to 𝑑(𝑒(𝜃𝑔𝐱₀),𝜃)<𝛿. Now, we make the substitution 𝛾=𝜃𝑔. Thus, 𝜃=𝛾𝑔⁻¹, and the condition becomes 𝑑(𝑒(𝛾𝐱₀),𝛾𝑔⁻¹)<𝛿, or, by invariance of the metric, 𝑑(𝛾⁻¹𝑒(𝛾𝐱₀),𝑔⁻¹)<𝛿. This says that 𝑔⁻¹∈𝐵(𝛾⁻¹𝑒(𝛾𝐱₀)), or equivalently, 𝑔∈𝐵(𝑒(𝛾𝐱₀)⁻¹𝛾).
This allows us to rewrite the triple integral (5.8), using the measure-preserving transformation 𝜃↦𝛾=𝜃𝑔, as _{𝑌𝐺𝜃∈𝐺}𝜒_𝜃⁻¹_{𝑒−1(𝐵(𝜃))}𝑑𝜌𝑑𝛼_𝐱0𝑑𝜈=_{𝑌𝐺𝛾∈𝐺}𝜒_{𝐵(𝑒(𝛾𝐱0)}⁻¹_𝛾)𝑑𝜌𝑑𝛼_𝐱0𝑑𝜈=_{𝛾∈𝐺𝑌𝐺}𝜒_{𝐵(𝑒(𝛾𝐱0)}⁻¹_𝛾)𝑑𝛼_𝐱0𝑑𝜈𝑑𝜌. (5.9) Now, comparing with Lemma 5.2, we see that the inner integral above is exactly the quality of the shift-invariant estimator 𝑠_𝛾.
We therefore have 𝑄(𝑒)≤_𝛾𝑄𝑠_𝛾𝑑𝜌, (5.10) or in other words, the average quality of the shift-invariant estimators {𝑠_𝛾} is at least 𝑄(𝑒). Therefore, at least one of the 𝑠_𝛾 satisfies 𝑄(𝑠_𝛾)≥𝑄(𝑒).

6. A Non-Shift-Invariant Example

In the following example, suggested by Schulman, the optimal shift-invariant estimator is not optimal. This provides an interesting complement to Conjecture 4.1. Lehmann and Casella [11, Section 5.3] also give examples of this phenomenon.

Let 𝑋 be the infinite trivalent tree, which we view as the Cayley graph of the group 𝐺=(𝑍/2𝑍)∗(𝑍/2𝑍)∗(𝑍/2𝑍)=⟨𝑎,𝑏,𝑐∣𝑎²=𝑏²=𝑐²=1⟩. In words, 𝐺 is a discrete group generated by three elements 𝑎,𝑏, and 𝑐, each of order two and with no other relations. Each nonidentity element of 𝐺 can be written uniquely as a finite word in the letters 𝑎, 𝑏, and 𝑐 with no letter appearing twice in a row; we refer to such a word as the reduced form of the group element. (We write 1 for the identity element of 𝐺.) Multiplication in the group is performed by concatenating words and then canceling any repeated letters in pairs. Evidently, 𝐺 is infinite. The Cayley graph 𝑋 is a graph with one vertex labeled by each group element of 𝐺, and with an edge joining vertices 𝑤 and 𝑥 if and only if 𝑤=𝑥𝑎, 𝑤=𝑥𝑏, or 𝑤=𝑥𝑐. Note that this relation is symmetric, since 𝑎, 𝑏, and 𝑐 each has order 2. Each vertex of 𝑋 has valence 3, and 𝑋 is connected and contains no circuits, that is, it is a tree. Finally, 𝑋 becomes a metric space by declaring each edge to have length one.

Because of how we defined the edges of 𝑋, 𝐺 acts naturally on the left of 𝑋: given 𝑔∈𝐺, the map 𝑔∶𝑋→𝑋 defined by 𝑔(𝑥)=𝑔𝑥 is an isometry of 𝑋. So if 𝛿>0 is given, 𝜇 is a probability distribution, 𝜃∈𝐺, and 𝑒 is an estimator, then the shift 𝜇_𝜃 and the quality 𝑄_𝛿(𝑒) are defined as usual by (1.1) and (1.3).

We are ready to present the example. Suppose that 0<𝑑<1 is fixed, and let 𝜇 be the probability distribution with atoms of weight 1/3 at the three vertices 𝑎,𝑏,and𝑐. Thus for 𝜃∈𝐺, the distribution 𝜇_𝜃 has atoms of weight 1/3 at the three neighbors 𝜃𝑎,𝜃𝑏,and𝜃𝑐 of the vertex 𝜃 in 𝑋.

Example 6.1. There is an optimal one-sample estimator with quality 2/3, but the quality of any shift-invariant one-sample estimator is at most 1/3.

Proof. Consider the one-sample estimator 𝑒 that truncates the last letter of the sample (unless the sample is the identity, in which case we arbitrarily assign the value 𝑎). That is, for a vertex 𝑥 of 𝑋, 𝑒(𝑥)=𝑤if𝑥=𝑤ℓisreduced,ℓ=𝑎,𝑏,or𝑐,𝑎if𝑥=1. (6.1) Geometrically, this estimator takes a sample 𝑥 and, unless 𝑥=1, guesses that the shift is the (unique) neighbor of 𝑥 that is closer to 1.
We compute the quality of 𝑒. Note that 𝑄¹(𝑒)=1, because if 𝜃=1, then the sample will be 𝑎,𝑏, or 𝑐, and the estimator is guaranteed to guess correctly. In fact, 𝑄^𝑎(𝑒)=1 also, as is easily verified. For any other shift 𝜃, the sample is 𝜃ℓ for ℓ=𝑎,𝑏, or 𝑐, and the estimator guesses correctly exactly when ℓ differs from the last letter of 𝜃. So 𝑄^𝜃(𝑒)=2/3, and 𝑄(𝑒)=inf_𝜃𝑄^𝜃(𝑒)=2/3.
It is easy to see that this estimator is optimal. Suppose that 𝑒^′ is another estimator and 𝑄(𝑒^′)>2/3. Since each local quality 𝑄^𝜃(𝑒^′) is either 0, 1/3, 2/3, or 1, we must have 𝑄^𝜃(𝑒^′)=1 for all 𝜃. This means that 𝑒^′ always guesses correctly. But since there are different values of 𝜃 that can produce the same sample, this is impossible.
Observe that the estimator 𝑒 above is neither left nor right invariant. For instance right invariance fails, as 𝑒(𝑏𝑎⋅𝑎)=𝑒(𝑏)=1≠𝑏𝑎=𝑒(𝑏𝑎)⋅𝑎, and the same example shows the failure of left invariance: 𝑒(𝑏⋅𝑎𝑎)=𝑖𝑑≠𝑏𝑎=𝑏𝑒(𝑎𝑎).
Indeed, we conclude by showing that the quality of any shift invariant one-sample estimator 𝑒^′ is at most 1/3. Suppose 𝑒^′(1)=𝑤. If 𝑒^′ is left invariant, it follows that 𝑒^′(𝑥)=𝑥𝑤 for all 𝑥; if 𝑒^′ is right invariant, it follows that 𝑒^′(𝑥)=𝑤𝑥 for all 𝑥.
Since 𝛿<1, the quality of 𝑒^′ at 𝜃=1 is equal to the probability that 𝑒^′(𝑥)=1, given that 𝑥 was sampled from 𝜇. With equal probability 𝑥 is 𝑎, 𝑏, or 𝑐; since at most one of 𝑤𝑎, 𝑤𝑏, and 𝑤𝑐 and one of 𝑎𝑤, 𝑏𝑤, and 𝑐𝑤 can equal 1, we conclude that 𝑄(𝑒^′)≤𝑄¹(𝑒^′)≤1/3.

We remark that this example readily generalizes to other non-amenable groups; the key is that 𝑒 is a two-to-one map but with only one sample, a shift-invariant estimator is necessarily one-to-one.

Acknowledgments

The authors thank V. Vazirani and L. Schulman for suggesting the problem that motivated this paper along with subsequent helpful discussions. They are grateful to the reviewers for many helpful suggestions, especially for correcting our proof of Lemma 5.1. As always, they thank Julie Landau, without whose support and down-the-line backhand this work would not have been possible.