The Generalization Error Bound for the Multiclass Analytical Center Classifier

Abstract

This paper presents the multiclass classifier based on analytical center of feasible space (MACM). This multiclass classifier is formulated as quadratic constrained linear optimization and does not need repeatedly constructing classifiers to separate a single class from all the others. Its generalization error upper bound is proved theoretically. The experiments on benchmark datasets validate the generalization performance of MACM.

1. Introduction

Multiclass classification is an important and on-going research subject in machine learning. Its application is immense, such as machine vision [1, 2], text and speech categorization [3, 4], natural language processing [5], and disease diagnosis [6, 7]. Two kinds of approaches have been proposed to solve multiclass classification problem [8]. The first multiclass classification approach is extending binary classifier to handle the multiclass case directly. This included neural networks, decision trees, support vector machines, naive Bayes, and 𝐾-nearest neighbors. The second approach decomposes the multiclass classification problem into several binary classification tasks. Several methods are used for this decomposition: one-versus-all [9], all-versus-all [10], and error-correcting output coding [11].

The one-versus-all approach reduces the problem of classifying among 𝐾 classes into 𝐾 binary problems, where each problem discriminates a given class from the other 𝐾−1 classes.

For the all-versus-all method, a binary classifier is built to discriminate between each pair of classes, while discarding the rest of the classes. This requires building 𝐾(𝐾−1)/2 binary classifiers for 𝐾 classes problem. When testing a new example, voting is performed among the classifiers and the class with the maximum number of votes wins.

For error-correcting output coding, it works by training 𝑁 binary classifiers to distinguish between the 𝐾 different classes. Each class is given a codeword of length 𝑁 according to a binary matrix 𝑀. Each row of 𝑀 corresponds to a certain class.

The above multiclass classification algorithms need construct binary classifier repeatedly to separate a single class from all the others for 𝐾 classes problem, which leads to daunting computation and low efficiency of classification. Reference [12] proposes multiclass support vector machine (MSVM), which corresponds to simple quadratic optimization and need not repeat constructing binary classifier. However, support vector machine corresponds to the center of the largest inscribed hypersphere of feasible space. When the feasible space, that is, the space of hypotheses consistent with the training data, is elongated or asymmetric, support vector machine is not effective [13]. To address the above problems, multiclass classifier based on the analytical center of feasible space (MACM) is proposed. At the same time, in order to validate its generalization performance theoretically, its generalization error upper bound is formulated and proved. And the experiments on benchmark dataset validate the generalization performance of MACM.

2. Multiclass Analytical Center Classifier

To facilitate the discussion of multiclass analytical center classifier, the following definitions are introduced.

Definition 1 (chunk). A vector, 𝐯=(𝑣₁,…,𝑣_𝑘𝑑)∈ℜ^𝑘𝑑, is broken into 𝑘 chunks (𝐯₁,…,𝐯_𝑘) where the 𝑖th chunk 𝐯_𝑖=(𝑣_{(𝑖−1)∗𝑑+1},…,𝑣_𝑖∗𝑑).

Definition 2 (expansion). Let Vec(𝑥,𝑖)∈ℜ^𝑘𝑑 be a vector where 𝐱∈ℜ^𝑑 is embedded in 𝑘𝑑 dimensions space by writing the coordinates of 𝐱 in the 𝑖th chunk of a vector in ℜ^𝑘𝑑. 𝟎^ℓ denotes the zero vector of length ℓ. Then, Vec(𝐱,𝑖) can be written formally as the concatenation of three vectors, Vec(𝐱,𝑖)=(𝟎^{(𝑖−1)∗𝑑},𝐱,𝟎^{(𝑘−𝑖)∗𝑑})∈ℜ^𝑘𝑑. And define Vec(𝐱,𝑖,𝑗)=Vec(𝐱,𝑖)−Vec(𝐱,𝑗) as a vector where 𝐱 is embedded in the 𝑖th chunk and −𝐱 is embedded in the 𝑗th chunk of a vector in ℜ^𝑘𝑑.

Definition 3. Given the sample (𝐱,𝑦)∈ℜ^𝑑×{1,…,𝑘}, its expansion is defined as 𝐺(𝐱,𝑦)={Vec(𝐱,𝑖,𝑗)∣𝑖=𝑦,𝑗={1,…,𝑘}/𝑖}; the expansion of the whole sample set 𝑆={(𝐱₁,𝑦₁),(𝐱₂,𝑦₂),…} is defined as 𝐺(𝑆)=⋃_{(𝐱,𝑦)∈𝑆}𝐺(𝐱,𝑦).

Definition 4 (piecewise linear separability). The point sets 𝐴^𝑖∈ℜ^{𝑑×𝑚𝑖}, 𝑖=1,…,𝑘 (𝑘 represents the class labels and𝑚_𝑖 the number of samples belonging to 𝑖th class), are piecewise linear separable if there exists 𝐰^𝑖∈ℜ^𝑑, 𝛾^𝑖∈ℜ, 𝑖=1,…,𝑘, where 𝑑 represents the dimension of point, such that 𝐴^𝑖_ℓ𝐰^𝑖−𝛾^𝑖>𝐴^𝑖_ℓ𝐰^𝑗−𝛾^𝑗,𝑖,𝑗=1,…,𝑘,𝑖≠𝑗,ℓ=1,…,𝑚_𝑖. (1)

Definition 5 (piecewise linear classifier). Assume 𝐰=(𝐰₁,…,𝐰_𝑘), where (𝐰₁,…,𝐰_𝑘)∈ℜ^{𝑘∗(𝑑+1)}=ℜ^𝑑+1×⋯×ℜ^𝑑+1. Given a new point 𝐱∈ℜ^𝑑+1, a piecewise linear classifier is a function 𝑓∶ℜ^𝑑+1→{1,…,𝑘} as follows: 𝑓(𝐱)=argmax_{𝑖=1,…,𝑘}𝐰_𝑖𝐱^′, (2) where arg max returns to a class label corresponding to the maximum value.

To simplify the notation for the formulation of multiclass analytical center classifier, we consider an augmented weight space as follows.

Let 𝐴^𝑖_ℓ=𝐴^𝑖_ℓ1∈ℜ^𝑑+1,𝐰^𝑖=𝐰^𝑖𝛾∈ℜ^𝑑+1,𝑖=1,…,𝑘,ℓ=1,…,𝑚_𝑖; (3) then, inequality (1) can be rewritten as 𝐴^𝑖_ℓ𝐰^𝑖−𝐴^𝑖_ℓ𝐰^𝑗>0,𝑖,𝑗=1,…,𝑘,𝑖≠𝑗,ℓ=1,…,𝑚_𝑖. (4) Let 𝐰=(𝐰₁,…,𝐰_𝑘)∈ℜ^{𝑘∗(𝑑+1)}=ℜ^𝑑+1×⋯×ℜ^𝑑+1. According to Definition 2, embedding 𝐴^𝑖_ℓ into ℜ^{𝑘∗(𝑑+1)} space, inequality (4) has the following form: Vec𝐴^𝑖_ℓ,𝑖−Vec𝐴^𝑖_ℓ,𝑗𝐰>0,𝑖,𝑗=1,…,𝑘,𝑖≠𝑗,ℓ=1,…,𝑚_𝑖. (5) Consider that Vec𝐴^𝑖_ℓ,𝑖,𝑗=Vec𝐴^𝑖_ℓ,𝑖−Vec𝐴^𝑖_ℓ,𝑗,𝑖,𝑗=1,…,𝑘,𝑖≠𝑗,ℓ=1,…,𝑚_𝑖. (6) Thus, inequality (6) can be rewritten as follows: Vec𝐴^𝑖_ℓ,𝑖,𝑗𝐰>0,𝑖,𝑗=1,…,𝑘,𝑖≠𝑗,ℓ=1,…,𝑚_𝑖. (7)

Inequality (7) represents the feasible space of 𝐰 in the higher dimension space ℜ^{𝑘∗(𝑑+1)}. Similar to the binary classification based on the analytical center of version space [8], we define the slack variable 𝑆_{𝑖,𝑗,ℓ}=Vec(𝐴^𝑖_ℓ,𝑖,𝑗)𝐰, 𝑖,𝑗=1,…,𝑘, 𝑖≠𝑗, ℓ=1,…,𝑚_𝑖 and then have the following minimization problem, whose solver corresponds to the analytical center of higher dimension space: minΦ(𝐰)=−∑^𝑚𝑖𝑘_ℓ=1∑_{𝑖≠𝑗,𝑖,𝑗=1}ln𝑆_{𝑖,𝑗,ℓ}(𝐰)=1s.t.ℎ2𝐰𝐰^′−1=0. (8)

In order to further simplify the formulation of multiclass analytical center classifier, we introduce some notations as follows: 𝑀=𝑘∗(𝑘−1)∗∑^𝑘_𝑖=1𝑚_𝑖, 𝐵={Vec(𝐴^𝑖_ℓ,𝑖,𝑗)∣𝑖,𝑗=1,…,𝑘,𝑖≠𝑗,ℓ=1,…,𝑚_𝑖}∈ℜ^{𝑀×𝑘∗(𝑑+1)}; let 𝐵_𝑖 represent the 𝑖th row vector of 𝐵. Then, the optimization problem (8) can be rewritten as follows: minΦ(𝐰)=−∑^𝑀_𝑖=1ln𝑆_𝑖where𝑆_𝑖=𝐵_𝑖𝐰(𝐰)=1s.t.ℎ2𝐰𝐰^′−1=0. (9)

After solving the optimization problem (9) to get the optimal weight 𝐰^∗, we have a piecewise linear classifier 𝑓∶ℜ^𝑑+1→{1,…,𝑘} computed in the following way: 𝑓(𝐱)=argmax_{𝑖=1,…,𝑘}𝐰^∗_𝑖𝐱^′, (10) where argmax returns to a class label corresponding to the maximum value.

If the dataset is not piecewise linear separable, the kernel function is used to map the data into high dimension linear space.

3. Generalization Error Bound of Multiclass Analytical Center Classifier

In order to analyze the generalization error bound theoretically, we introduce the definition of classification margin and data radius and then deduce the margin-based generalization error bound of MACM.

Definition 6 (classification margin). Given the linear classifier ℎ(𝐱_𝑗)=argmax_{𝑖=1,…,𝑘}𝐰_𝑖𝐱^′_𝑗, the classification margin of the sample (𝐱_𝑗,𝑦_𝑗)∈ℜ^𝑑×{1,…,𝑘} is defined as follows: 𝑟_𝑗=min_{𝑙=𝑦𝑗{},𝑖=1,…,𝑘/𝑙}𝐰_𝑙𝐱^′_𝑗−𝐰_𝑖𝐱^′_𝑗. (11) For the whole training set 𝑆={(𝐱₁,𝑦₁),(𝐱₂,𝑦₂),…,(𝐱_𝑚,𝑦_𝑚)}, the minimal margin is as follows: margin_𝑆(ℎ)=min_{𝐱𝑗,𝑦𝑗∈𝑆}𝑟_𝑗. (12)

Definition 7 (data radius). Given dataset 𝑆={(𝐱₁,𝑦₁),(𝐱₂,𝑦₂),…,(𝐱_𝑚,𝑦_𝑚)}, the data radius is defined as follows: 𝜍(𝑆)=max_{𝐱𝑖∈𝑆}𝐱_𝑖. (13)

Theorem 8. Define data radius of dataset 𝑆 as 𝜍(𝑆) and data radius of dataset Vec(𝑆,𝑖,𝑗) as 𝜍(Vec(𝑆,𝑖,𝑗)); if 𝜍(𝑆)≤𝑅, then 𝜍(Vec(𝑆,𝑖,𝑗))≤2𝑅.

Proof. Consider the following: 𝜍Vec𝑆,𝑖,𝑗=max_{𝐱𝑙∈𝑆}Vec𝐱_𝑙,𝑖,𝑗=max_{𝐱𝑙∈𝑆}Vec𝐱_𝑙,𝑖−Vec𝐱_𝑙,𝑗≤max_{𝐱𝑙∈𝑆}Vec𝐱_𝑙,𝑖+Vec𝐱_𝑙,𝑗=max_{𝐱𝑙∈𝑆}Vec𝐱_𝑙,𝑖+max_{𝐱𝑙∈𝑆}Vec𝐱_𝑙,𝑗. (14) Because ‖Vec(𝐱,𝑖)‖=‖(𝟎,𝐱,𝟎)‖=‖𝐱‖, 𝜍(Vec(𝑆,𝑖,𝑗))≤2∗max_{𝐱𝑖∈𝑆}‖𝐱_𝑖‖=2𝑅. This ends the proof of Theorem 8.

Theorem 9 (see [14]). Consider thresholding of a real-valued function ℋ with unit weight vectors ‖𝐰‖=1 on the inner product space 𝑋 and fix margin 𝑟∈ℜ⁺. For any probability distribution 𝒟 on 𝑋×{−1,1} with support in a ball of radius 𝑅 around the origin, with probability 1−𝛿 over 𝑚 random samples 𝑆, any hypothesis ℎ∈ℋ with 𝑚𝑎𝑟𝑔𝑖𝑛_𝑆(ℎ)≥𝑟 on 𝑆 has error more than 𝑒𝑟𝑟_𝒟(ℎ)()=2=𝜀𝑟,𝑚,𝛿,𝑅𝑚64𝑅²𝑟²log𝑒𝑚𝑟8𝑅²log32𝑚𝑟²4+log𝛿, (15) provided 𝑚>2/𝜀,64𝑅²/𝑟²<𝑚.
From Definition 3 and inequality (7), it is shown that to correctly classify the sample 𝐴^𝑖,𝑖=1,…,𝑘, 𝐺(𝐴)𝐰=𝐺(⋃^𝑘_𝑖=1𝐴^𝑖)𝐰>𝟎 is satisfied. Here, one introduces the samples’ pairs 𝑃₊(𝐴)={𝐺(𝐴),+𝟏}∈ℜ^{𝑘∗(𝑑+1)}×{+1} and 𝑃₋(𝐴)={−𝐺(𝐴),−𝟏}∈ℜ^{𝑘∗(𝑑+1)}×{−1}, where 𝟎,𝟏 denote the corresponding dimension vector with elements 0 or 1, respectively. So, one can construct the new samples’ set 𝑃(𝐴)=𝑃₊(𝐴)⋃𝑃₋(𝐴)∈ℜ^{𝑘∗(𝑑+1)}×{−1,+1}.

Theorem 10. The binary classification of sample 𝑃(𝐴) by analytical center classifier is equivalent to the multiclass classification of sample 𝐴={𝐴^𝑖}^𝑘_𝑖=1 by multiclass analytical center classifier.

Proof. Assume that {𝑋⁺_𝑖,𝑌⁺}∈𝑃₊(𝐴),{𝑋⁻_𝑖,𝑌⁻}∈𝑃₋(𝐴),𝑖=1,…,|𝑃₊(𝐴)|; then, binary classification is to solve the following feasible problem: 𝑌⁺𝑋⁺_𝑖𝐰+𝑏>0,𝑖=1,…,𝑃₊(𝐴,𝑌₋)𝑋⁻_𝑖𝐰+𝑏>0,𝑖=1,…,𝑃₋(𝐴). (16) Suppose the bias 𝑏 equals 0, because 𝑃₋(𝐴) and 𝑃₊(𝐴)are symmetrical on the origin. The feasible constraints can be rewritten as follows: 𝑋⁺_𝑖𝐰>0,𝑖=1,…,𝑃₊(𝐴). (17) The feasible constraints (17) define the feasible space of weight vector 𝐰∈ℜ^{𝑘∗(𝑑+1)}; the binary classification by analytical center classifier can be formulated as follows: minΦ(𝐰)=−∑^{|𝑃+(𝐴)|}_𝑖=1𝑋⁺_𝑖1𝐰>0s.t.2𝐰^′𝐰−1=0. (18) Because 𝑋⁺_𝑖∈𝐺(𝐴) and |𝑃₊(𝐴)|=|𝐺(𝐴)|, the problem (18) is equivalent to problem (8). This ends the proof of Theorem 10.

Theorem 11. Consider the classifiers’ set ℋ from Definition 5 with ∑_{𝑖=1,…,𝑘}‖𝐰_𝑖‖=1 on the inner product space 𝑋, where ℎ∶ℜ^𝑑+1→{1,…,𝑘} and fix margin 𝑟∈ℜ⁺. For any probability distribution 𝒟 on 𝑋×{1,…,𝑘} with support in a ball of radius 𝑅 around the origin, with probability 1−𝛿 over 𝑚 random samples 𝑆, any hypothesis ℎ∈ℋ with 𝑚𝑎𝑟𝑔𝑖𝑛_𝑆(ℎ)≥𝑟 on 𝑆 has error more than err_𝒟(ℎ)()=4()=𝜀𝑟,𝑚,𝛿,𝑅,𝑘𝑘−1𝑚256𝑅²𝑟²logemr32𝑅²log32𝑚𝑟²4+log𝛿 (19) provided 𝑚>2/𝜀,64𝑅²/𝑟²<𝑚.

Proof. Because the sample in 𝑃(𝑆) is not independent, the generalization error bound cannot be attained from Theorem 9. Theorem 9 is independent of the sample distribution, so we can construct a new sample distribution 𝒟^′. According to the new distribution and dataset 𝑆 to generate the independent sample set 𝑃^′(𝑆) with 𝑚 samples, that is, for every (𝐱,𝑦)∈𝑆, define 𝑃^′(𝐱,𝑦) as the point sampled uniformly and randomly from 𝑃(𝑆) according to the distribution 𝒟^′; then, we have 𝑃^′(𝑆)=⋃_{(𝐱,𝑦)∈𝑆}𝑃^′(𝐱,𝑦). From Theorem 8, the data radius 𝜍(𝑃^′(𝑆)) of 𝑃^′(𝑆) satisfies 𝜍(𝑃^′(𝑆))≤2𝑅. The generalization error of hypothesis ℎ over 𝑃^′(𝑆) from Theorem 9 can be calculated as follows: err_𝒟′(ℎ)()=2=𝜀𝑟,𝑚,𝛿,𝑅𝑚256𝑅²𝑟²logemr32𝑅²log32𝑚𝑟²4+log𝛿. (20)
Event 𝐸_𝑖which denotes a sample in 𝑃(𝑆) is wrongly classified and event 𝐶which denotes the misclassification occurs in 𝑃^′(𝑆). From the above analysis, the misclassification of any sample in 𝑃(𝑆) causes the misclassification of the point in 𝑃^′(𝑆), so that the probability of events 𝐸_𝑖 and 𝐶 satisfies the following inequality: 𝑃𝐸_𝑖≤𝑃(𝐶). (21) Because the cardinality of 𝑃(𝑆) equals 2(𝑘−1), the probability of sample misclassification in 𝑃(𝑆) is written as follows: 𝑃^2(𝑘−1)_𝑖=1𝐸_𝑖. (22) From union bound theorem, we have the following inequality: 𝑃^2(𝑘−1)_𝑖=1𝐸_𝑖≤^2(𝑘−1)_𝑖=1𝑃𝐸_𝑖≤^2(𝑘−1)_𝑖=1𝑃(𝐶). (23) So the generalization error of hypothesis ℎ over 𝑆 is err_𝒟(ℎ)()==𝜀𝑟,𝑚,𝛿,𝑅,𝑘_{2(𝑘−1)𝑖=1}2𝑚256𝑅²𝑟²logemr32𝑅²log32𝑚𝑟²4+log𝛿=4(𝑘−1)𝑚256𝑅²𝑟²logemr32𝑅²log32𝑚𝑟²4+log𝛿. (24) This ends the proof of Theorem 11.

4. Computational Experiments

In this section, we present the computational results comparing multiclass analytical center classifier (MACM) and multiclass support vector machine (MSVM) [12]. A description of each of the datasets follows this paragraph. The kernel function for the piecewise nonlinear MACM and MSVM methods is 𝑘(𝐱,𝐱_𝑖)=(𝐱𝐱^′_𝑖/𝑁+1)^𝑛, where 𝑛 is the desired polynomial.

Wine Recognition Data. The wine dataset uses the chemical analysis of wine to determine the cultivar. There are 178 points with 13 features. This is a three class dataset distributed as follows: 59 points in class 1, 71 points in class 2, and 48 points in class 3.

Glass Identification Database. The Glass dataset is used to identify the origin of a sample of glass through chemical analysis. This dataset is comprised of six classes of 214 points with 9 features. The distribution of points by class is as follows: 70 float processed building windows, 17 float processed vehicle windows, 76 nonfloat processed building windows, 13 containers, 9 tableware, and 29 headlamps.

Table 1 contains the results for MACM and MSVM on wine and glass datasets. As anticipated, MACM produces better testing generalization than MSVM.

Table 1 

The generalization error of MACM and MSVM.

5. Summary

In this paper, the multiclass analytical center classifier based on the analytical center of feasible space, which corresponds to a simple quadratic constrained linear optimization, is proposed. At the same time, in order to validate its generalization performance theoretically, its generalization error upper bound is formulated and proved. By the experiments on wine recognition and glass identification dataset, it is shown that the multiclass analytical center classifier outperforms multiclass support vector machine in generalization error.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant nos. 61370096 and 61173012 and the Key Project of Natural Science Foundation of Hunan Province under Grant no. 12JJA005.