The Estimate for Approximation Error of Neural Network with Two Weights

Abstract

The neural network with two weights is constructed and its approximation ability to any continuous functions is proved. For this neural network, the activation function is not confined to the odd functions. We prove that it can limitlessly approach any continuous function from limited close subset of 𝑅^𝑚 to 𝑅^𝑛 and any continuous function, which has limit at infinite place, from limitless close subset of 𝑅^𝑚 to 𝑅^𝑛. This extends the nonlinear approximation ability of traditional BP neural network and RBF neural network.

1. Introduction

So far, many neural networks models have been presented and applied in pattern recognition, automatic control, signal processing, and aided decision. Among these models, BP (feedforward) neural networks and RBF (radial basis function) neural networks are widely used because of their nonlinear ability of approximation to any continuous function. Up to now, these two classes of neural networks have successfully been applied to approach any nonlinear continuous function defined on bounded close subsets [1–7]. However, the approximation result to any continuous function defined on unbounded set has been very rare [8]. This inspires us to look for a new network which is of more wide approximation ability and can approach any continuous function both on a bounded set and on an unbounded set.

Wang et al. [9] presented a new neural network called neural networks with two weights on the base of combining the advantage of BP neural network with that of RBF neural network. This model can not only simulate BP neural network and RBF neural network, but also simulate neural networks with higher order. This neural network not only contains direction weight value with respect to BP network but also contains core weight value with respect to RBF network. The function of its neurons is of the following form: 𝑦=𝜙^𝑚_𝑗=1𝜔_𝑗𝑥_𝑗−𝑧_𝑗𝜔_𝑗𝑥_𝑗−𝑧_𝑗^𝑠𝜔_𝑗𝑥_𝑗−𝑧_𝑗^𝑝+𝜃, (1) where 𝑦 is the output of neurons, 𝜙 is the activation function, 𝜃 is the threshold value, 𝜔_𝑗 is the direction weight value, 𝑧_𝑗 is the core weight value, 𝑥_𝑗 is the input, and 𝑝 and 𝑠 are two parameters.

In (1), when 𝑧_𝑗=0, 𝑝=0, and 𝑠=1, then (1) reduces to mathematical model BP network neurons; when 𝜔_𝑗 takes a fixed value, 𝑝=2 and 𝑠=0, then (1) becomes mathematical model RBF network neurons.

In [10], the approximation operators with logarithmic sigmoidal function of a class of neural networks with two weights and a class of quasi-interpolation operators were investigated. Using these operators as approximation tools, the upper bounds of estimate errors were estimated.

In [2], the authors considered single hidden layer neural networks: 𝑁_𝑛(𝑥)=𝑐₀+^𝑛_𝑖=1𝑐_𝑖𝜙𝜔_𝑖𝑥+𝜃_𝑖,𝑁_𝑛∶𝑅⟶𝑅, (2) where 𝑐_𝑖 represents output weights, 𝜙 is the activaion function, 𝜃_𝑖 is threshold value, and 𝜔_𝑗 is direction weight value.

The main result in [2] is as follows.

Theorem 1. Suppose 𝜙 is a bounded, strictly monotonously increasing, and odd function defined on 𝑅,𝑓∈𝐶[𝑎,𝑏], and 𝑛∈𝑁. Then there exists a feedforward neural network (BP) with one hidden layer 𝑁_𝑛→𝑅 defined by 𝑁_𝑛(𝑥)=𝑐₀+^𝑛_𝑖=1𝑐_𝑖𝜙𝜔_𝑖𝑥+𝜃_𝑖,𝑁_𝑛∶𝑅⟶𝑅, (3) such that sup_{[]𝑥∈𝑎,𝑏}𝑁_𝑛(𝑥)(𝑥≤5−𝑓)2𝜔𝑓,𝑏−𝑎𝑛, (4) where for 𝑖=1,2,…,𝑛𝑥_𝑖=𝑎+𝑖𝑏−𝑎𝑛,𝑥₀=𝑎,𝑚=sup_𝑥∈𝑅𝜙(𝑥),𝑐_𝑖=1𝑓𝑥2𝑚^𝑖−𝑓𝑥_𝑖−1,𝑑_𝑛=𝜙⁻¹𝑚𝑚−,𝑐2𝑛₀=𝑓(𝑎)−^𝑛_𝑖=1𝑐_𝑖𝜙𝜔_𝑖𝑥+𝜃_𝑖,𝜃_𝑖𝑑=−^𝑛𝑛()𝑏−𝑎2𝑎+2𝑖−1𝑏−𝑎𝑛,𝜔_𝑖=2𝑑_𝑛𝑛.𝑏−𝑎 (5)

In this paper, one is concerned with the neural networks with two weights and a single hidden layer: 𝑁_𝑛(𝑥)=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖(𝑥−𝑧)^𝑠|(𝑥−𝑧)|^𝑠|𝑥−𝑧|^𝑝−𝜃_𝑖. (6)

One objective is to construct a new BP neural network which is different from the BP neural network in [2] and prove this network has the approximation ability to any nonlinear continuous function. Another objective is to prove that, by adjusting the values of two parameters 𝑠 and 𝑝, the neural network with two weights can not only approximate any continuous functions defined on a bounded close subset, namely, it has the same approximation ability as BP neural network in [2], but also approximate any continuous function defined on an unbounded set. Namely, the neural network with two weights has a better approximation ability than BP neural network and RBF neural network.

2. Theoretical Result

We use the following notations: the symbols 𝑁,𝑁₀, and 𝑅 represent the set of positive integers, nonnegative integers, and real number, respectively. Using 𝐶[𝑎,𝑏] to denote the space of continuous real-valued functions defined on [𝑎,𝑏], it is equalled by the supremum norm ‖⋅‖. Let 𝑓 be a real-valued function defined on [𝑎,𝑏]. 𝜂(𝑓,𝛿), the modulus of continuity of 𝑓, is defined as 𝜂(𝑓,𝛿)=sup_0<ℎ≤𝛿‖𝑓(⋅+ℎ)−𝑓(⋅)‖. Then we have lim_𝛿→0𝜂(𝑓,𝛿)=0, and for any real number 𝜆≥0,𝜂(𝑓,𝜆𝛿)≤(1+𝜆)𝜂(𝑓,𝛿).

The modulus of continuity is usually considered as the measure of the smoothness of function and the approximation error in approximation theory. The function 𝑓 is called Lipschitz 𝛼(0<𝛼≤1) continuous and is written as 𝑓∈Lip_𝐶(𝑓)𝛼, if there exists a constant 𝐶(𝑓) such that 𝜂(𝑓,𝛿)≤𝐶(𝑓)𝛿^𝛼. 𝐶(ℎ) denotes the positive constant dependents only on ℎ and its value may be different at different occurrence.

Our main result is as follows.

Theorem 2. Suppose 𝜙 is a bounded, strictly monotonously increasing, and odd function defined on 𝑅,𝑓∈𝐶[𝑎,𝑏], and 𝑛∈𝑁. Then there exists a feedforward neural network (BP) with one hidden layer 𝑁_𝑛→𝑅 defined by 𝑁_𝑛(𝑥)=^𝑛_𝑖=1𝑐_𝑖𝜙𝜔_𝑖𝑥+𝜃_𝑖, (7) such that sup_{[]𝑥∈𝑎,𝑏}𝑁_𝑛(𝑥)−𝑓(𝑥)≤1+𝑛+1𝜔4𝑛𝑓,𝑏−𝑎𝑛, (8) where, for 𝑖=1,2,…,𝑛, 𝑥_𝑖=𝑎+𝑖𝑏−𝑎𝑛,𝑑_𝑛=𝜙⁻¹𝑚2𝑛−𝑚,𝑥₀=𝑎,𝑚=sup_𝑥∈𝑅𝜙(𝑥),𝑐_𝑖=1𝑓𝑥2𝑚^𝑖−𝑓𝑥_𝑖−1,𝜔_𝑖=𝑑_𝑛−𝑑^∗_𝑛𝑛,𝜃𝑏−𝑎_𝑖=𝑑_𝑛−𝑑_𝑛−𝑑^∗_𝑛𝑛()𝑏−𝑎𝑎+𝑖−1𝑏−𝑎𝑛,𝑑^∗_𝑛=𝜙⁻¹𝑚𝑚−.2𝑛 (9)

Proof. Divide [𝑎,𝑏] into 𝑛 equal intervals; each has length of (𝑏−𝑎)/𝑛 and let 𝑎=𝑥₀<𝑥₁<⋯<𝑥_𝑛=𝑏. For 𝑖=1,2,…,𝑛, we set 𝑐_𝑖=(1/2𝑚)(𝑓(𝑥_𝑖)−𝑓(𝑥_𝑖−1)), 𝑑_𝑛=𝜙⁻¹((𝑚/2𝑛)−𝑚), 𝑚=sup_𝑥∈𝑅|𝜙(𝑥)|, 𝜃_𝑖=𝑑_𝑛−((𝑑_𝑛−𝑑^∗_𝑛)𝑛/(𝑏−𝑎))[𝑎+(𝑖−1)((𝑏−𝑎)/𝑛)], 𝑑^∗_𝑛=𝜙⁻¹(𝑚−(𝑚/2𝑛)),𝜔_𝑖=(𝑑_𝑛−𝑑^∗_𝑛)𝑛/(𝑏−𝑎).
For any 𝑥∈[𝑎,𝑏], given 𝑛, we assume that 𝑥∈[𝑥_𝑖−1,𝑥_𝑖] and note that 𝑁_𝑛(𝑥)=^𝑛_𝑖=11𝑓𝑥2𝑚^𝑖−𝑓𝑥_𝑖−1𝜙𝜔_𝑖𝑥+𝜃_𝑖. (10) Fix 𝑥^∗∈[𝑥_𝑖−1,𝑥_𝑖] and choose 𝑓, so that 𝑁_𝑛(𝑥^∗_𝑖)=𝑓(𝑥^∗_𝑖). Let 𝐺_𝑖(𝑥^∗_𝑖)=𝜙(𝜔_𝑖𝑥+𝜃_𝑖); it follows that 𝑁_𝑛(𝑥)−𝑓𝑥^∗_𝑖=𝑁_𝑛(𝑥)−^𝑛_𝑖=11𝑓𝑥2𝑚^𝑖−𝑓𝑥_𝑖−1𝐺_𝑖𝑥^∗_𝑖=^𝑛_𝑖=01𝑓𝑥2𝑚^𝑖−𝑓𝑥_𝑖−1𝐺_𝑖(𝑥)−𝐺_𝑖𝑥^∗_𝑖. (11) There are two possible cases to consider: (i) 𝑥>𝑥^∗_𝑖; (ii) 𝑥≤𝑥^∗_𝑖.
Case (i). When 𝑥>𝑥^∗_𝑖, we have, for 𝑖=1,2,…,𝑛, 0<𝐺_𝑖(𝑥)−𝐺_𝑖𝑥^∗_𝑖<𝐺_𝑖𝑥_𝑖−𝐺_𝑖𝑥^∗_𝑖<𝜙𝜔_𝑖𝑎+𝑖𝑏−𝑎𝑛+𝜃_𝑖𝑑+𝑚=𝜙_𝑛=𝑚+𝑚.2𝑛 (12)
Case (ii). When 𝑥≤𝑥^∗_𝑖, we have, for 𝑖=1,2,…,𝑛, 0>𝐺_𝑖(𝑥)−𝐺_𝑖𝑥^∗_𝑖>𝐺_𝑖𝑥_𝑖−1𝜔−𝑚=𝜙_𝑖𝑎+(𝑖−1)𝑏−𝑎𝑛+𝜃_𝑖𝑑−𝑚=𝜙^∗_𝑛𝑚−𝑚=−.2𝑛 (13) From (12) and (13), it follows that 𝐺_𝑖(𝑥)−𝐺_𝑖𝑥^∗_𝑖≤𝑚2𝑛,𝑖=1,2,…,𝑛. (14) Thus from (14), we have 𝑁_𝑛(𝑥)−𝑓𝑥^∗_𝑖≤^𝑛_𝑖=01𝑓𝑥2𝑚^𝑖−𝑓𝑥_𝑖−1𝐺_𝑖(𝑥)−𝐺_𝑖𝑥^∗_𝑖≤𝑛+1𝜂2𝑚𝑓,𝑏−𝑎𝑛×𝑚=2𝑛𝑛+1𝜂4𝑛𝑓,𝑏−𝑎𝑛. (15) Therefore, we have 𝑁_𝑛(𝑥)−𝑓(𝑥)≤𝑁_𝑛(𝑥)−𝑓𝑥^∗_𝑖+𝑓(𝑥)−𝑓𝑥^∗_𝑖=1+𝑛+1𝜂4𝑛𝑓,𝑏−𝑎𝑛. (16) This ends the proof of Theorem 2.

Remark 3. The activation functions are assumed to be odd functions in [2], while in this paper, this assumption is removed. On the other hand, in the construction of neural networks, threshold values and direction weight values are different from those in [2].

Theorem 4. Suppose 𝜙 is a bounded, strictly monotonously increasing, and odd function defined on 𝑅,𝑓∈𝐶[𝑎,𝑏], and 𝑛∈𝑁, 𝑧∉[𝑎,𝑏]. Then there exists a neural network with two weights and one hidden layer 𝑁_𝑛∶[𝑎,𝑏]→𝑅 defined by 𝑁_𝑛(𝑥)=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖(𝑥−𝑧)^𝑠|(𝑥−𝑧)|^𝑠|𝑥−𝑧|^𝑝−𝜃_𝑖 (17) with 𝑝<𝑠,𝑝<0 such that sup_{𝑥∈[𝑎,𝑏]}𝑁_𝑛(𝑥)−𝑓(𝑥)≤1+𝑛+1𝜂4𝑛𝑓,𝑏−𝑎𝑛, (18) where 𝑐_𝑖 is defined in Theorem 2 and 𝑙^𝑠_𝑖 is equal to 𝜔_𝑖 in Theorem 2.

Proof. We make fractional transformation 𝐻∶𝑥→𝑢, where 𝑢=(𝑥−𝑧)^𝑠/|𝑥−𝑧|^𝑝−𝑠, 𝑥∈[𝑎,𝑏]. Since 𝑧∉[𝑎,𝑏], thus 𝐻 is a continuous transformation. Hence 𝑓([𝑎,𝑏])=𝑈 is also a bounded close set; moreover, 0∉𝑈.
There exists an inverse transformation for fractional transformation 𝐻⁻∶𝑢→𝑥, where 𝑥=𝑧+𝑢^1/𝑠|𝑢|^{(𝑠−𝑝)/𝑠𝑝},𝑢∈𝑈. So 𝑔(𝑢)=𝑓(𝑥)=𝑓(𝑧+𝑢^1/𝑠|𝑢|^{(𝑠−𝑝)/𝑠𝑝}) is a continuous real-valued function defined on bounded close set 𝑈. For 𝑔(𝑢), from Theorem 2, there exists a neural network with two weights and one hidden layer 𝑁_𝑛∶[𝑎,𝑏]→𝑅 defined by 𝑁_𝑛(𝑢)=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖𝑢−𝜃_𝑖 (19) such that 𝑔(𝑢)=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖𝑢−𝜃_𝑖=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖(𝑥−𝑧)^𝑠|(𝑥−𝑧)|^𝑠|𝑥−𝑧|^𝑝−𝜃_𝑖=𝑓(𝑥) (20) which satisfies sup𝑔(𝑢)−𝑔(𝑢)≤1+𝑛+1𝜂4𝑛𝑓,𝑏−𝑎𝑛. (21) This ends the proof of Theorem 4.

Remark 5. When 𝑝<𝑠,𝑝<0, the network with two weights is not BP and RBF neural network; thus, Theorem 4 validates that the neural network with two weights has the same approximation ability as BP and RBF neural network on any bounded close set. When 𝑛→∞, from Theorem 4, this neural network can approximate any continuous function defined on a bounded close set.

Theorem 6. Suppose 𝜙 is a bounded, strictly monotonously increasing, and odd function defined on 𝑅; 𝐷 is an unbounded subset of 𝑅 (all accumulation points are in 𝐷 beside ∞) and𝑓∈𝐶(𝐷), 𝑧∉𝐷, lim_𝑥→∞𝑓(𝑥)=𝑏(𝑐𝑜𝑛𝑠𝑡𝑎𝑛𝑡) and then there exists a neural network with two weights and one hidden layer: 𝑁_𝑛(𝑥)=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖(𝑥−𝑧)^𝑠|(𝑥−𝑧)|^𝑠|𝑥−𝑧|^𝑝−𝜃_𝑖 (22) with 𝑝<𝑠,𝑝<0 such that 𝑓(𝑥)=^𝑛_𝑖=1𝑐_𝑖𝜙𝑙^𝑠_𝑖(𝑥−𝑧)^𝑠|(𝑥−𝑧)|^𝑠|𝑥−𝑧|^𝑝−𝜃_𝑖 (23) which satisfies sup_{𝑥∈[𝑎,𝑏]}𝑓(𝑥)−𝑓(𝑥)<1+𝑛+1𝜂4𝑛𝑓,𝑏−𝑎𝑛. (24)

Proof. We make fractional transformation 𝐻∶𝑥→𝑢, where 𝑢=(𝑥−𝑧)^𝑠/|𝑥−𝑧|^𝑝−𝑠, 𝑥∈𝐷. Since 𝑧∉𝐷, thus 𝐻 is a contimuous transformation on 𝐷 and as 𝑥→∞,𝑢→0. Let 𝐻(∞)=0; 𝐻(𝑥) is a continuous function on 𝐷∪{∞}. We prove that 𝐻(𝐷∪{∞})=𝐻(𝐷)∪{0} is a close set. If 𝑣 is an arbitrary accumulation point of 𝐻(𝐷)∪{0}, then 𝑣=0 or 𝑣 is an accumulation point of 𝐻(𝐷). When 𝑣=0, 𝑧∈𝐻(𝐷)∪{0}. When 𝑣 is an accumulation point of 𝐻(𝐷), then there exists a sequence 𝑢^𝑘∈𝐻(𝐷) and another sequence 𝑥^𝑘∈𝐷, such that 𝑥^𝑘→𝑢^𝑘,𝑢^𝑘→𝑣≠0(𝑘→∞). From fractional transformation, there exists 𝑢,𝑧=𝑢,𝑥^𝑘→𝑥 such that 𝑢^𝑘→𝑢. From 𝑢^𝑘→𝑣≠0(𝑘→∞) and inverse transformation 𝐻⁻¹∶𝑥=𝑧+𝑢^1/𝑠|𝑢|^{(𝑠−𝑝)/𝑠𝑝},𝑢∈𝑈, it follows that 𝑥→∞ does not hold. Hence 𝑥 is a finite point of 𝐷. Therefore 𝑥∈𝐷, 𝑣=𝑢∈𝐻(𝐷). So, 𝐻(𝐷∪{∞})=𝐻(𝐷)∪{0} is a close set. Let 𝑔(𝑢)=𝑓(𝑥)=𝑓𝑧+𝑢^1/𝑠|𝑢|^{(𝑠−𝑝)/𝑠𝑝}. (25) Then 𝑔(𝑢) is a continuous function on 𝐻(𝐷). Let 𝑔(0)=𝑏=lim_𝑥→∞𝑓(𝑥); it is easy to prove that 𝑔(𝑢) is a continuous function on 𝐻(𝐷)∪{0}. For a continuous function 𝑔(𝑢) defined on 𝐻(𝐷)∪{0}, by means of Theorem 4, there exists a neural network with two weights ∑^𝑛_𝑖=1𝑐_𝑖𝜙(𝑙^𝑠_𝑖((𝑥−𝑧)^𝑠/|(𝑥−𝑧)|^𝑠)|𝑥−𝑧|^𝑝−𝜃_𝑖) such that 𝑔(𝑢)=^𝑛_𝑖=1𝑐_𝑖𝜙𝜔_𝑖(𝑥−𝑧)^𝑠|(𝑥−𝑧)|^𝑠|𝑥−𝑧|^𝑝−𝜃_𝑖=𝑓(𝑥) (26) which satisfies sup𝑔(𝑢)−𝑔(𝑢)≤1+𝑛+1𝜂4𝑛𝑓,𝑏−𝑎𝑛. (27) This ends the proof of Theorem 6.

Remark 7. When 𝑝<𝑠,𝑝<0, the network with two weights is not BP and RBF neural network; thus, Theorem 6 validates that the neural network with two weights has the approximation ability that BP and RBF neural network do not have on any unbounded set. Namely, the network with two weights has a better approximation ability than BP and RBF neural network on any unbounded set. When 𝑛→∞, from Theorem 6, this neural network can approximate any continuous function defined on unbounded set.

3. Conclusions

In this paper, we construct a new BP neural network and prove that the network has the approximation ability to any nonlinear continuous function. In our result, the threshold values and direction weight values in [2] are different from those in our paper. Second, we prove that neural network with two weights has the same approximation ability as BP neural network and RBF neural network to any continuous function defined on any bounded close set; furthermore, we prove that neural network with two weights has a better approximation ability than BP neural network and RBF neural network to any continuous function defined on any unbounded set.