[Retracted] Cross-Domain End-To-End Aspect-Based Sentiment Analysis with Domain-Dependent Embeddings

Tian, Yingjie; Yang, Linrui; Sun, Yunchuan; Liu, Dalian.

doi:https://doi.org/10.1155/2021/5529312

Complexity

On this page

Abstract Introduction Related Work Discussion Conclusions Data Availability Conflicts of Interest Authors’ Contributions Acknowledgments References Copyright Related Articles

Research Article Retraction

!

This article has been Retracted. To view the article details, please click the ‘Retraction’ tab above.

Special Issue

Complexity Problems Handled by Advanced Computer Simulation Technology in Smart Cities 2021

View this Special Issue

Research Article | Open Access

Volume 2021 | Article ID 5529312 | https://doi.org/10.1155/2021/5529312

[Retracted] Cross-Domain End-To-End Aspect-Based Sentiment Analysis with Domain-Dependent Embeddings

Yingjie Tian,^1,2,3Linrui Yang,²Yunchuan Sun,⁴and Dalian. Liu^5,6

Academic Editor: Zhihan Lv

Received01 Feb 2021

Revised20 Feb 2021

Accepted02 Mar 2021

Published15 Mar 2021

Abstract

With the development of sentiment analysis, studies have been gradually classified based on different researched candidates. Among them, aspect-based sentiment analysis plays an important role in subtle opinion mining for online reviews. It used to be treated as a group of pipeline tasks but has been proved to be analysed well in an end-to-end model recently. Due to less labelled resources, the need for cross-domain aspect-based sentiment analysis has started to get attention. However, challenges exist when seeking domain-invariant features and keeping domain-dependent features to achieve domain adaptation within a fine-grained task. This paper utilizes the domain-dependent embeddings and designs the model CD-E2EABSA to achieve cross-domain aspect-based sentiment analysis in an end-to-end fashion. The proposed model utilizes the domain-dependent embeddings with a multitask learning strategy to capture both domain-invariant and domain-dependent knowledge. Various experiments are conducted and show the effectiveness of all components on two public datasets. Also, it is also proved that as a cross-domain model, CD-E2EABSA can perform better than most of the in-domain ABSA methods.

1. Introduction

Sentiment analysis, as one of the most popular tools in the natural language processing (NLP) domain, has attracted enormous explorations [1]. Through the process of sentiment analysis, which is also named as opinion mining, the sentiment tendency of texts is extracted. Based on different research objects, the field of sentiment analysis can be classified into three levels, including document level, sentence level, and aspect level. Document-level and sentence-level sentiment analyses are based on the assumption that there is only one valuable topic in the document or sentence. However, when considering the practical opinion mining in user feedback related to products or services, one or more entities, the evaluation of which might be different, are mentioned in one sentence, which demands a more fine-grained analysis method. Aspect-level sentiment analysis, which focuses on mining and differentiating the sentiment polarity of various entities or aspects in one sentence, gradually ascends the stage [2]. This paper will focus on the aspect-based sentiment analysis and extracting the author’s feelings about certain entities or targets [3]. There might be one or more entities mentioned in one sentence. Aspect-based sentiment analysis (ABSA) was used to be treated as a task that composed of several steps or subtasks, including aspect extraction that consists of opinion target extraction (OTE) and aspect categories detection (ASD), as well as sentiment polarity clarification (SPC) [4]. Therefore, models were mainly designed, focusing on one or two subtasks. Methods that can identify entities where emotions are expressed in were designed for aspect term extraction (OTE) [5–8]. Moreover, some works aimed at distinguishing aspects with defined aspect categories [9, 10]. Furthermore, the research studies pay more attention on classifying known aspects’ sentiment polarity [11–13]. However, when considering the purpose of practicability, it is more convenient and user friendly to build an end-to-end model regarding aspect-level sentiment analysis as a whole task. But, the challenge exits when designing one model for two or three subtasks at the same time, especially when these tasks are usually solved diversely from the perspective of model designing. ASD and SPC are widely regarded as binary or multiclassification tasks, while OTE is solved with models designed for sequence-labelling tasks. The typical way of combining these subtasks was accomplishing them one by one with a pipeline solution, which resulted in a complex model structure. With the rise of deep learning, neural networks and pretrained frameworks have been designed and developed for NLP tasks including sentiment analysis [14, 15]. Neural networks were then designed for pipeline works that can achieve these subtasks in sequence [16]. Recently, End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA) has also been proposed and become popular [17, 18]. Fine-grained sentiment analysis is considered as one sequence-labelling task with multiple tags representing both position information and sentiment polarity of targets, which achieves the complete task in an end-to-end fashion. Various measures have been taken to define the strategy and the set of labels for sequence labelling, including joint training [19], span-based extraction [20], and the unified model [17]. This paper will employ the strategy of unified model as a tagging scheme for ABSA, which only needs reviews as input and avoids the limitation of mined knowledge brought by defined aspects’ set.

But, there are still issues for utilizing a deep-learning model in aspect-based sentiment analysis. First of all, although the volume of reviews has accumulated rapidly, thanks to the continuous development of computer technology, there is insufficient labelled data for machine learning. Moreover, contents of reviews have certain domain discrepancy, which leads to the challenge of cross-domain knowledge transfer. For instance, models which have trained based on comments for products such as mobile phones might get a bad result with comments for restaurants as the test data. In addition, from the perspective of seeking sentiment polarity towards aspects, same opinion words might bring opposite emotions towards aspects in different domains. As shown in Figure 1, “long” time results in positive sentiment for a review of a restaurant, but different in laptop domain. As a consequence, transfer learning has been introduced to aspect-based sentiment analysis in order to achieve the knowledge transmission across fine-grained opinion mining in various fields. In the beginning, the analysis was also carried out based on a certain subtask model. Recently, transferable end-to-end aspect-based sentiment analysis [21] was also proposed combining the E2E-ABSA and selective transfer learning methods.

This paper will follow the trend of cross-domain E2E-ABSA and design a new strategy by introducing Bert and domain-dependent embeddings. The proposed framework CD-E2EABSA contains the main network of the word embedding layer and the aspect-based sentiment analysis layer with a parameter generation module. Bert is utilized as the word embedding layer in the model, which learned contextual embeddings, and, rather than glove or word2vec, can better facilitate the following downstream task with specialized contextual information. Also, it can avoid the out-of-vocabulary (OOV), especially when reviews of one domain contain a considerable number of proper nouns, which used to result in the introduction of dictionaries and manual features. Besides, such a design ensures the recognition of each aspect with its sentiment polarity for two domains. Previous work of cross-domain ABSA mainly pursued mining domain-invariant features to minimize the distance of two domains and achieve transfer learning. A few attempted to take advantage of domain-dependent features for the task. But, their models were built with two or three subtasks [22], while CD-E2EABSA utilized it in a unified model inspired by a cross-domain NER model [23]. The main contributions of this paper are as follows:(1)Model-based transfer learning is introduced to cross-domain end-to-end aspect-based sentiment analysis in this paper.(2)In order to benefit the cross-domain ABSA without complex structure, Bert, as the pretrained embedding layer, is firstly introduced into transferable E2E-ABSA in this paper. Also, CD-ABSA makes use of domain-related knowledge as features to ensure the model’s cognition of different domains.(3)In the experiments, components are shown effective, and the cross-domain model performance can finally be similar to that of the one-domain model in aspect-based sentiment analysis.

2.1. Aspect-Based Sentiment Analysis

Aspect-based sentiment analysis is traditionally divided into three or two subtasks, in which rule-based [6], topic modeling [24], and traditional sequence labelling models (such as CRF) [25] were used for OTE and ACD, and classifiers [16] or neural networks with attentions [11, 12] were used for SPC. For instance, ILWAANet [26] extracted features from both data and lexicon by multiple attentions to support the more comprehensive capture of aspects.

After the approval that using a more integrated solution can achieve a more effective result in fine-grained sentiment analysis [19], a unified model for target-based opinion mining was proposed in [17]. It promotes the development of ABSA and stimulates further research from different dimensions. First of all, Li et al. [18] combined the unified model with pretrained model Bert for E2E-ABSA. Furthermore, there were works to explore different labelling and modeling strategies. DOER [5] solves the E2E-ABSA by introducing the cross-shared unit to connect aspect term extraction and aspect sentiment classification and use the joint labels instead of collapsed labels. Also, Zeng et al. [27] trained a joint learning end-to-end neural network for ASD and SPC tasks, while Hu et al. [20] introduced the span-based labelling scheme to targeted opinion mining within an open-domain condition and argued the pipeline strategy can perform better. As for modeling strategies, Peng et al. [28] improved the unified model to accomplish the novel subtask they present named aspect sentiment triplet extraction (ASTE). Besides, other machine learning approaches have been employed to enrich the content of E2E-ABSA. Open-domain learning is also introduced to the unified model [29] recently. The main difference between CD-E2EABSA and the abovementioned models is the introduction of the domain adaptation and achieving cross-domain fine-grained sentiment analysis based on the development of end-to-end sentiment analysis.

2.2. Transfer Learning in ABSA

The incorporation of transfer learning and ABSA can result in the knowledge transfer among different subtasks of ABSA or cross-domain aspect-based sentiment analysis. Transfer learning is a mode that encourages the machine to autonomously acquire knowledge from data and apply it to new tasks with new data. According to Pan and Yang [30], the basic methods of transfer learning can be divided into 4 types, namely, sample-based, model-based, feature-based, and relation-based transfer learning. Among them, feature-based and model-based transfer learning are more often combined with fine-grained sentiment analysis. Also, Xu et al. [31] utilized Bert as the language model to learn domain knowledge and accomplish the ABSA task. Feature-based transfer learning refers to the method of transferring features in order to reduce the gap between the source and target domains and then feed features to classifiers. Hu et al. [22] proposed a model achieving the aspect-based sentiment analysis by focusing on distilling domain-invariant features. On the other hand, model-based transfer learning is based on the assumption that the data in the source domain and target domain can share some model parameters. In this field, in addition to directly using pretrained models [18, 32], researchers mainly tried to absorb task transfer to enhance the learning ability of the ABSA model. For instance, Li et al. [18] employed coarse-to-fine task transfer in order to leverage knowledge from the ASD with rich resources to OTE. Recently, using IKTN, Liang et al. [33] achieved the domain-specific and sentiment-related knowledge transfer among subtasks of aspect-level sentiment analysis. Also, Li et al [21] introduced selective adversarial learning (SAL) as the domain adaptation method to the unified model and achieved the cross-domain ABSA. However, most of them did not regard aspect-level sentiment analysis as an end-to-end problem solving, and they did not use contextual information in the feature generation to assist subsequent tasks. CD-E2EABSA will treat cross-domain ABSA as an end-to-end task and explore the model-based transfer learning that employs domain-specific knowledge learnt from the network.

2.3. Domain Embeddings for Sentiment Analysis

Most existing research studies focused on extracting domain-invariant features to assistant domain knowledge transfer in cross-domain ABSA [34]. Liang et al. [33] separated the domain-invariant embeddings from domain-dependent embeddings by using an adversarial multitask learning framework, while Hu et al. [22] used a simpler way to distilling domain-invariant features by regarding domain-independent and domain-dependent features as orthogonal information. On the other hand, some models focused on domain-dependent embeddings in multidomain sentence-level sentiment analysis, which is introduced to multidomain ABSA in this paper. For instance, domain-aware embeddings [35], which combine word embeddings and domain-dependent features, were generalized and enhance the efficiency of sentence-level sentiment analysis. Liu et al [36] introduced the domain embeddings for sentence-level multidomain sentiment analysis in order to address the issue of different sentiment words for different domains. The main difference between it and this paper is that CD-E2EABSA focused on aspect-level sentiment analysis and learned domain representation with a parameter generation network through training instead of using an attention mechanism. Furthermore, the multitask learning strategy is widely used when a model aims at extracting domain-related representations for sentiment analysis [36]. The multitask learning strategy can learn multiple related tasks at the same time and use sharing knowledge in the learning process to improve the performance and generalization ability of the model in each task. An interactive multitask learning network [37] has also been applied to the unified model for E2E-ABSA, which simultaneously trains for both aspect-level and document-level sentiment analysis. CD-E2EABSA will employ the multitask strategy to utilize domain-dependent embeddings in the cross-domain E2E-ABSA task.

3. Methodology

3.1. Problem Statement

This paper deals with the cross-domain end-to-end aspect-based sentiment analysis and treats it as a sequence-labelling task while solving the cross-domain aspect extraction and sentiment classification at the same time. The cross-domain end-to-end aspect-based sentiment analysis task is formulated mathematically as follows, given with words from source domain and with words from target domain . In order to achieve recognizing aspect-level sentiment through the sequence labelling, a unified tagging scheme [17] is adopted for every word. Tags for every word contains {B-POS, B-NEG, B-NEU, I-POS, I-NEG, I-NEU, E-POS, E-NEG, E-NEU, S-POS, S-NEG, S-NEU, O}, where O represents other words and others are designed for aspects in the sentence. Among them, B-, I-, E-, and S- indicate beginning, inside, ending, and single words for one aspect, respectively, while POS, NEG, and NEU represent positive, negative, and neural polarity categories. The Figure 2 shows an example sentence and the unified tag of each word in the sentence from the laptop domain. Domain embeddings are then used to accomplish the cross-domain task. , where s and t represent the source domain and the target domain. The final goal of CD-E2EABSA is to get aspect-level sentiment labels for two domains’ texts with the help of domain embeddings in an end-to-end fashion.

3.2. Methods

For the fine-grained sentiment analysis problem that is regarded as a sequence labelling task, this paper adopts a more advanced word vector generation layer and transforms the traditional BiLSTM-CRF network by introducing domain vectors to achieve cross-domain aspect-based sentiment analysis. This section will illustrate the structure of the model, which is named as CD-E2EABSA and shown in Figure 3. The CD-E2EABSA mainly consists of a word embedding layer, shared-ABSA layer with a parameter generation network, and private CRF layer for each domain. Given an input sequence X, the network utilized Bert as the word embedding layer, which learns the contextual information and forms the embeddings for the following network. Embeddings are then fed into a Bi-LSTM layer with domain-dependent knowledge based on a parameter generation network. Finally, private CRFs are used for each domain and achieve the sequence labelling task for them. This section includes three sections depicting each layer of the model in turns and the multitask strategy we employ.

3.2.1. Word Embedding Layer

The word embedding layer is the first layer in CD-E2EABSA that accepts input sequence X and outputs embeddings for the following neural network, which is basic and essential as it vectorizes words from texts and makes the aspect-based sentiment analysis feasible. Most existing solutions for end-to-end ABSA used traditional word vector sets such as glove as model inputs, which are complicated to operate with data reprocessing for lexicon. Also, some words cannot be found in glove pretrained embeddings and need to be initialized immediately. It has been confirmed that when using Bert [14], generating token-level representations with sentences as input in the ABSA task can take contextual information into account and improve the effect of text mining models on the basis [18]. In this paper, Bert is also introduced as the word embedding layer and produces features that contain each tokens’ contextual information. L transformer layers are employed to refine the token-level features which combine token embeddings, position embeddings, and segment embeddings at first, and is finally generated and transferred as contextualized representation to downstream tasks as equation (1). The word embedding layer is shared between source- and target-domain texts.

3.2.2. Cross-Domain ABSA Layer

Word embeddings are then fed into the cross-domain ABSA layer, where features are distilled and tags are acquired word by word for both domains. The Bidirectional Long Short-Term Memory network (Bi-LSTM) and Conditional Random Field (CRF) [38] are used as the main model structure of the cross-domain ABSA layer in CD-E2EABSA, among which Bi-LSTM can capture majority information from the past and future in one sequence and CRF can take the input sequence and output the target tag sequence. In detail, Bi-LSTM is introduced to accept word embeddings from the word embedding layer and output hidden representations considering the sequence labelling formulation of E2E-ABSA task. Furthermore, traditional Bi-LSTM is transformed in CD-E2EABSA to combining with domain-dependent embeddings to realize the cross-domain task. Traditional hidden outputs of Bi-LSTM can be denoted as in equation (2) with bidirectional hidden states.where and , respectively, represent the forward and backward hidden states and represents the present moment. indicates all parameters used in the Bi-LSTM network. Also, represents concatenation.

Following the work in [23], a parameter generation network is used to reform the Bi-LSTM by utilizing domain embeddings which can learn domain-dependent knowledge through training both domains. Parameters for Bi-LSTM is generated with learnt domain embeddings and . The resulting parameters are calculated as follows:where symbolizes the set of parameters for traditional Bi-LSTM. Also, represents the domain embeddings for each domain, where means the embedding size for the feature and contains source domain and target domain. refers to tensor contraction. Domain embeddings are gradually trained during the process of training and test during the process of dev and test for each domain.

After the generation of parameters, the module of Bi-LSTM is modified to accept the input of parameters for Bi-LSTM. Also, with the input for Bi-LSTM and parameters , bidirectional hidden outputs of Bi-LSTM for one domain can finally be denoted as equation (4), where and , respectively, represent the forward and backward hidden states of Bi-LSTM, is finally calculated by concatenating and . By combining the parameter generation module and Bi-LSTM, CD-E2EABSA learns more contextual knowledge while completing the domain knowledge learning.

In addition, in order to enhance the results of the sequence labelling task, CRF is employed as the output layer for ABSA. CRF pursues the globally most probable tag sequence and is widely used as the output layer in models designed for sequence labelling. Two standard CRFs, including CRF (S) and CRF (T) are used for the source and target domains (Figure 2), respectively. Under this setting, each CRF is trained for one domain and can handle the special case that the source domain and target domain have different label sets. Given are the hidden outputs of Bi-LSTM , and the label of the sequence is . The probability of the label sequence of an input sequence X can be calculated as follows:where and are parameters in CRF. refers to an arbitrary label sequence. All possible tag sequences get through a softmax formulation to get the final likelihood . Also, Viterbi algorithm is used to find the highest scored label sequence as output.

3.2.3. Multitask Learning Strategy

The multitask learning strategy is used to learn domain embeddings and assist the cross-domain ABSA task. The multitask learning strategy ensures the alternate use and learning of domain vectors, which is a prerequisite for realizing independent learning of domain knowledge and sharing of nondomain knowledge. As shown in detail in Algorithm 1, mini batches of two domains’ training set take turns to be fed into the network CD-E2EABSA through the training process. The domain embedding E_D, which contains for the source domain and for the target domain, and parameters of the network are firstly initialized randomly. Through training, each domain generates parameters with W and within Bi-LSTM of the cross-domain ABSA layer, and gradients are then computed for in equation (3). At the end of each training step, all the parameters are updated together by minimizing the loss. Also, the sentence-level negative log likelihood loss equation (6) is used for training one dataset .

	Input: training data
	Parameters: W, E_D, θ_CRF
	Output: Target model
(1)	While training steps not end do:
(2)	Generating minibatches for each domain data: B^s, B^t
(3)	θ_L^s_STM ⟵ f(W, e_s) (Equation (3))
(4)	∆W, ∆d_s, ∆θ_CRFs ⟵ train(B^s) (Equation (7))
(5)	θ_L^t_STM ⟵ f(W, e_t) (Equation (3))
(6)	∆W, ∆d_t, ∆θ_CRFt ⟵ train(B^t) (Equation (7))
(7)	Update W, d_D
(8)	End While

But, in terms of the whole loss function for the training of CD-E2EABSA, both the source domain and target domain need to be considered. Also, the final loss function is denoted in equation (7), where indicates the weight of one domain and regularization is added to reduce overfitting with parameters and , the parameter set of the model. Also, indicates the source and target domain in the loss function.

4. Experiments

In this section, the experiments that were conducted to show the performance of CD-E2EABSA and their results will be illustrated and analysed.

4.1. Datasets and Settings

Two widely used review datasets for laptops and restaurants are included in the experiments, the number of sentences and aspects for two datasets is shown in Table 1. There are customers’ reviews related to restaurants and laptops, and both of them are from SemEval [39] and modified by Li et al. [17]. We utilize the two datasets as two cross-domain ABSA tasks, which are knowledge transferring from the restaurant dataset to the laptop dataset and from the laptop dataset to the restaurant dataset.

As for the model implementation, the transformers2.0 of Pytorch is chosen as the framework to generate the CD-E2EABSA. The pretrained “bert-base-uncased” model is utilized for the word embedding layer illustrated before, where the number of transformers is 12 and the hidden size for each is 768. The embedding size of domain-dependent features is set as 8 in the experiments for parameter generation. As for the cross-domain ABSA layer, single-layer Bi-LSTM is used, and the batch size is set as 8 to fit the GPU memory. Also, the learning rate is set as 2e − 5. The model is trained up to 1500 steps at first. After training 1000 steps, the model selection will be started for every 100 steps according to the microaveraged F1 score. Following these settings, bidirectional transfer learning among two datasets with different random seeds is recorded and the average results are reported.

4.2. Main Results

In this section, an ablation study, case study, and development study are used to present and discuss the performance of CD-E2EABSA with its components. Besides, a comparison study is conducted to show that CD-E2EABSA can be trained in two domains at the same time and has the same effect as the existing models which are trained in a single domain.

4.2.1. Ablation Study

An ablation study is conducted to investigate the effectiveness of the main components we used in CD-E2EABSA. The main results are shown in Table 2, and experiments are conducted on two cross-domain datasets. R-L and L-R indicate the knowledge transfer from the restaurant domain to the laptop domain and from the laptop domain to the restaurant domain, respectively. Dev and Tes represent the scores calculated from validation and test sets in the corresponding domain’s dataset. CD-E2EABSA is compared with four baselines, including CD-E2EABSA (glove), CD-E2E-ABSA (noPG), CD-E2EABSA (noCRF), and CD-E2EABSA (noPG + noCRF).

CD-E2EABSA (glove) uses the 100-sized glove embeddings as the initialized word representations instead of using Bert as the embedding layer. Also, the following network contains the BiLSTM-CRF and parameter generation module, the same as the CD-E2EABSA.

CD-E2EABSA (noPG) does not comprise the parameter generation module introduced before. Also, it uses Bert as the embedding layer, following by BiLSTM-CRF as the aspect-based sentiment layer. Although the multitask learning strategy is also employed, CD-E2EABSA (noPG) is trained without domain-dependent embeddings supporting the distinction between two domains.

CD-E2EABSA (noCRF) does not have private CRF for each domain as the output layer. A softmax layer is followed to produce predictions after input sequence get through the word embedding layer and Bi-LSTM with a parameter generation module.

CD-E2EABSA (noPG + noCRF) is also conducted, which has the word embedding layer and the ABSA layer including Bi-LSTM and a softmax layer.

The main results of the ablation study are depicted in Table 2, and CD-E2EABSA performs best in both cross-domain aspect-based sentiments. Micro-F1 scores are used to represents the performance of models. The comparison among all models indicates the effectiveness of all components in CD-E2EABSA. The worse result from CD-E2EABSA (glove) indicates that contextual information extracted by the word embedding layer can contribute to the result from the perspective of feature extraction in the model. Moreover, the result of CD-E2EABSA (noPG) shows that directly using the same LSTM parameters without domain-dependent vectors is not good, which implies that the parameter generation module can bring domain-related information to the network through training. Besides, it is shown that CRF, as a classical tool of sequence labelling, still contributes well in the E2E-ABSA task, as it seeks the best labelling sequence from global considerations.

4.2.2. Case Study

In order to show the effect of the ablation study more intuitively, a case study is presented in this section to show the significance of CD-E2EABSA through tagging results. Sentences from the laptop-14 test dataset are used and presented in Figures 4 and 5. Figure 4 shows the tags we get from those models, and Figure 5 transforms the tags to final recognition of aspects and their tagged sentiment polarity. It is obvious that CD-E2EABSA performs the best among them and recognizes both aspect and its sentiment polarity correctly. When there is no CRF, the model CD-E2EABSA (noCRF) cannot recognize the aspect in the sentence, nor its sentiment polarity. CD-E2EABSA can handle both issues with CRF. Besides, we found that domain embeddings can ensure the recognition of aspects’ boundary. CD-E2EABSA (noPG) fails to recognize the boundary of the aspect in the sentence. For instance, it treats “Tech support” as two aspects and tagged with diverse sentiment tags. Also, CD-E2EABSA (noPG + noCRF) can only identify a single word as an aspect, while CD-E2EABSA solves this with the parameter generation module and distinguishes more integrated aspects.

4.2.3. Development Study

A development study is conducted to further present the effectiveness of each component in CD-E2EABSA. First of all, it is found through the experiment that the OOV of CD-E2EABSA (glove) is higher than 0.14, which means there are 1279 of 8980 words that cannot be found in the file of glove embeddings and need to be initialized randomly at the start. How would it influence the efficiency of the model? After comparing the CD-E2EABSA and CD-E2EABSA (glove) in the ablation study, we explain this with Figure 6, which shows the curve of micro-F1 scores for the test results of CD-E2EABSA and CD-E2EABSA (glove) in the cross-domain ABSA task from restaurant to laptop through 5-epoch training. CD-E2EABSA (S) and CD-E2EABSA (T) represent the results of CD-E2EABSA in the source (restaurant) domain and target (laptop) domain, respectively, while CD-E2EABSA (glove) (S) and CD-E2EABSA (Glove) (T) represent the results of CD-E2EABSA (glove) in two domains. It is obvious that the word embeddings produced by the word embedding layer in CD-E2EABSA work better than glove embeddings at the beginning and keep their superiority through the training process. This suggests that contextual knowledge learned in the word embedding layer for each token can generate more knowledge for the downstream tasks and improve model performance.

After discussing the significance of the word embedding layer, we also analyse the parameter generator centred on the domain vector. Two domain-dependent embeddings are output and visualized. As it is shown in Figure 7, the two 8-dimensional domain vectors learned though the training process of a cross-domain ABSA task are extracted and reduced to two-dimensional vectors through the t-SNE algorithm [40]. It is obvious that domain-dependent embeddings are totally different for the laptop and restaurant domain, suggesting CD-E2EABSA can learn and distil an embedding for every domain. They contain domain knowledge, which can help CD-E2EABSA to recognize texts from one domain and accomplish the cross-domain ABSA task from the perspective of parameters in Bi-LSTM. In the example in Figure 1, domain knowledge means long time represents different sentiment polarities in restaurant and laptop domains. Moreover, from Figure 5 of the ablation study, it is found that CD-E2EABSA can recognize boot time as a whole target in laptop reviews with the domain embedding.

The impact of steps for training the model are also investigated. As it is demonstrated in Figure 8, taking the result of CD-E2EABSA for cross-domain aspect-based sentiment analysis from the restaurant domain to laptop domain as an example, we found that convergence is still achieved after 1500 steps that we initialized before until 5000 steps and further steps do not provide considerable improvement.

4.2.4. Comparison Study

Comparison experiments are performed to compare CD-E2EABSA and existing models, the results in Table 3 suggesting that CD-E2EABSA can perform nearly the same in two domains simultaneously as the models trained in one domain. We compare the proposed model with ATAE-LSTM [12], IAN [11], IMN [37], and UM [17]. Also, results are shown in Table 3 where F1-R and F1-L represent F1 scores of different models in the restaurant and laptop dataset from [39]. The proposed method is firstly compared with two LSTM-based ABSA models designed for the SPC task, including ATAE-LSTM [12] and IAN [11]. They get input sequences and their aspects and combine the attention mechanism with the LSTM network to identify aspect-level sentiment in the restaurant domain. CD-E2EABSA does not need the aspects as input and achieves the end-to-end ABSA task and performs better than them. CD-E2EABSA is also compared with IMN, an interactive multitask E2E-ABSA model which utilized document-level knowledge for one-domain aspect-based sentiment analysis. It is shown that IMN was conducted in a divided restaurant-domain dataset [37], while CD-E2EABSA is trained on the combination of both datasets. We found that, after 5000 steps of training, the F1 score for the targeted restaurant domain can exceed one of the performances of IMN, which is designed for document-aspect knowledge transfer in one domain. Although CD-E2EABSA cannot get better F1 score than IMN, it is essential to point out that IMN is trained in one domain for one time while CD-E2EABSA is trained with two domains’ datasets at the same time. Moreover, from the perspective of the dataset, IMN uses the document-level dataset additionally to assist the extraction of fine-grained emotions. Finally, the proposed method is compared with UM by Li et al. [17], who introduced the unified tagging scheme for a single-domain E2E-ABSA task. It is depicted that CD-E2EABSA can be trained in both domains and, at the same time, performs better than UM which is trained in a single domain. In other words, CD-E2EABSA can get a better result by training once, while UM needs to be trained twice.

5. Discussion

This section will discuss the limitations of CD-E2EABSA and future research prospects for the solutions proposed in this article based on main results. As it was shown before, this paper designs an end-to-end deep-learning network structure for cross-domain fine-grained sentiment analysis tasks. CD-E2EABSA regards the aspect-level sentiment analysis as a sequence labelling task and links it with the classic sequence labelling task of named entity recognition which draws inspiration from cross-domain named entity recognition. Also, the result shows that CD-E2EABSA can perform better for both domains than most methods trained and test in one domain. In the future, we will discuss whether task-level migration can also be introduced into fine-grained sentiment analysis to solve the problem of insufficient knowledge in this field. However, this requires the addition of other sufficient experimental data as a prerequisite, such as multidomain datasets, or other related sequence annotation datasets in exist domain such as the laptop field. In addition, in the training process of this model, especially the optimizer, there are many excellent optimization [41–43] methods that deserve to be introduced to improve the efficiency of model training, including the Swarm Optimizer [44–46].

6. Conclusions

A new end-to-end model is provided for cross-domain aspect-based sentiment analysis by learning of domain knowledge and sharing of nondomain knowledge. Bert is utilized as the embedding layer and generates contextual representations for words. Bi-LSTM and CRF are used as in the cross-domain ABSA layer for the output fine-grained sentiment analysis. Also, domain embeddings are introduced with a parameter generation network to achieve the mode-based transfer learning through a multitask learning strategy. From the perspective of experiments, we demonstrated the importance and effectiveness of all components in the cross-domain ABSA task. The case study and development study show the recognition of domain knowledge by the proposed model. Also, CD-E2EABSA can get the second-best results after the cross-domain training when compared with exited in-domain aspect-based sentiment analysis models.

Data Availability

Previously reported reviews’ data were used to support this study and are available at DOI: https://doi.org/10.1609/aaai.v33i01.33016714. These prior studies (and datasets) are cited at relevant places within the text as references.

Conflicts of Interest

The authors declare that there are no conflicts of interest regarding the publication of this paper.

Authors’ Contributions

Y.T. and L.Y. were responsible for the methodology; Y.T. and Y.S. collected resources; L.Y. wrote the original draft; Y.T., Y.S., L.Y., and D.L. reviewed and edited the manuscript; and Y.T. supervised the work.

Acknowledgments

This work was partially supported by grants from the National Natural Science Foundation of China (Nos. 12071458, 71731009).

References

M. V. Mäntylä, D. Graziotin, and M. Kuutila, “The evolution of sentiment analysis-A review of research topics, venues, and top cited papers,” Computer Science Review, vol. 27, pp. 16–32, 2018.
View at: Publisher Site | Google Scholar
M. Hu and B. Liu, “Mining and summarizing customer reviews,” in Paper Presented at the Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, July 2004.
View at: Google Scholar
A. Nazir, R. Yuan, L. Wu, and L. Sun, “Issues and challenges of aspect-based sentiment analysis: a comprehensive survey,” IEEE Transactions on Affective Computing, 2020.
View at: Google Scholar
H. H. Do, P. Prasad, A. Maag, and A. Alsadoon, “Deep learning for aspect-based sentiment analysis: a comparative review,” Expert Systems with Applications, vol. 118, pp. 272–299, 2019.
View at: Publisher Site | Google Scholar
H. Luo, T. Li, B. Liu, and J. Zhang, “Doer: dual cross-shared Rnn for aspect term-polarity Co-extraction,” 2019, https://arxiv.org/abs/1906.01794.
View at: Google Scholar
S. Poria, E. Cambria, and A. Gelbukh, “Aspect extraction for opinion mining with a deep convolutional neural network,” Knowledge-Based Systems, vol. 108, pp. 42–49, 2016.
View at: Publisher Site | Google Scholar
R. He, W. Sun Lee, Ng Hwee Tou, and D. Dahlmeier, “An unsupervised neural attention model for aspect extraction,” in Paper presented at the Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, July 2017.
View at: Google Scholar
R. M. Marcacini, R. G. Rossi, Ivone Penque Matsuno, I. P. Matsuno, and S. O. Rezende, “Cross-domain aspect extraction for sentiment analysis: a transductive learning approach,” Decision Support Systems, vol. 114, pp. 70–80, 2018.
View at: Publisher Site | Google Scholar
W. Xue and T. Li, “Aspect based sentiment analysis with gated convolutional networks,” 2018, https://arxiv.org/abs/1805.07043.
View at: Google Scholar
S. Angelidis and M. Lapata, “Summarizing opinions: aspect extraction meets sentiment prediction and they are both weakly supervised,” 2018, https://arxiv.org/abs/1808.08858.
View at: Google Scholar
D. Ma, S. Li, X. Zhang, and H. Wang, “Interactive attention networks for aspect-level sentiment classification,” 2017, https://arxiv.org/abs/1709.00893.
View at: Google Scholar
Y. Wang, M. Huang, X. Zhu, and Li Zhao, “Attention-based lstm for aspect-level sentiment classification,” in Paper Presented at the Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA, November 2016.
View at: Google Scholar
P. Zhao, L. Hou, and O. Wu, “Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification,” Knowledge-Based Systems, vol. 193, Article ID 105443, 2020.
View at: Publisher Site | Google Scholar
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: pre-training of deep bidirectional transformers for language understanding,” 2018, https://arxiv.org/abs/1810.04805.
View at: Google Scholar
Z. Yang, Z. Dai, and Y. Yang, “Xlnet: generalized autoregressive pretraining for language understanding,” 2019, https://arxiv.org/abs/1906.08237.
View at: Google Scholar
M. S. Akhtar, D. Gupta, A. Ekbal, and P. Bhattacharyya, “Feature selection and ensemble construction: a two-step method for aspect based sentiment analysis,” Knowledge-Based Systems, vol. 125, pp. 116–135, 2017.
View at: Publisher Site | Google Scholar
X. Li, L. Bing, P. Li, and W. Lam, “A unified model for opinion target extraction and target sentiment prediction,” in Paper presented at the Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, February 2019.
View at: Google Scholar
X. Li, L. Bing, W. Zhang, and W. Lam, “Exploiting bert for end-to-end aspect-based sentiment analysis,” 2019, https://arxiv.org/abs/1910.00883.
View at: Google Scholar
M. Zhang, Y. Zhang, and D.-T. Vo, “Neural networks for open domain targeted sentiment,” in Paper presented at the Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, September 2015.
View at: Google Scholar
M. Hu, Y. Peng, Z. Huang, D. Li, and Y. Lv, “Open-domain targeted sentiment analysis via span-based extraction and classification,” 2019, https://arxiv.org/abs/1906.03820.
View at: Google Scholar
Z. Li, X. Li, and W. Ying, “Transferable end-to-end aspect-based sentiment analysis with selective adversarial learning,” 2019, https://arxiv.org/abs/1910.14192.
View at: Google Scholar
M. Hu, Y. Wu, and S. Zhao, “Domain-invariant feature distillation for cross-domain sentiment classification,” 2019, https://arxiv.org/abs/1908.09122.
View at: Google Scholar
C. Jia, X. Liang, and Y. Zhang, “Cross-domain ner using cross-domain language modeling,” in Paper Presented at the Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019.
View at: Google Scholar
A. Weichselbraun, S. Gindl, F. Fischer, S. Vakulenko, and A. Scharl, “Aspect-based extraction and analysis of affective knowledge from social media streams,” IEEE Intelligent Systems, vol. 32, no. 3, pp. 80–88, 2017.
View at: Publisher Site | Google Scholar
C. Brun, J. Perez, and C. Roux, “Xrce at semeval-2016 task 5: feedbacked ensemble modeling on syntactico-semantic knowledge for aspect based sentiment analysis,” in Paper presented at the Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), San Diego, CA, USA, June 2016.
View at: Google Scholar
H.-T. Nguyen, L.-M. Nguyen, and Nguyen Le-Minh, “Ilwaanet: an interactive lexicon-aware word-aspect attention network for aspect-level sentiment classification on social networking,” Expert Systems with Applications, vol. 146, Article ID 113065, 2020.
View at: Publisher Site | Google Scholar
Z. Zeng, J. Ma, M. Chen, and Li Xia, “Joint learning for aspect category detection and sentiment analysis in Chinese reviews,” in Paper presented at the China Conference on Information Retrieval, Fuzhou, China, September 2019.
View at: Google Scholar
H. Peng, Lu Xu, and L. Bing, “Knowing what, how and why: a near complete solution for aspect-based sentiment analysis,” in Paper presented at the Proceedings of the AAAI Conference on Artificial Intelligence, San Jose, CA, USA, June 2020.
View at: Google Scholar
Q. Ji, L. Xiang, Y. Ma, G. Liu, and S. Wang, “A unified labeling model for open-domain aspect-based sentiment analysis,” in Paper presented at the 2020 IEEE Fifth International Conference on Data Science in Cyberspace (DSC), Hong Kong, China, July 2020.
View at: Google Scholar
S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345–1359, 2009.
View at: Google Scholar
Hu Xu, B. Liu, L. Shu, and S. Yu Philip, “Dombert: domain-oriented language model for aspect-based sentiment analysis,” 2020, https://arxiv.org/abs/2004.13816.
View at: Google Scholar
A. Rietzler, S. Stabinger, O. Paul, and S. Engl, “Adapt or get left behind: domain adaptation through bert language model finetuning for aspect-target sentiment classification,” 2019, https://arxiv.org/abs/1908.11860.
View at: Google Scholar
Y. Liang, F. Meng, and J. Zhang, “An iterative knowledge transfer network with routing for aspect-based sentiment analysis,” 2020, https://arxiv.org/abs/2004.01935.
View at: Google Scholar
Z. Li, Y. Zhang, W. Ying, Y. Wu, and Q. Yang, “End-to-End adversarial memory network for cross-domain sentiment classification,” in Paper presented at the IJCAI, Sydney, Australia, August 2017.
View at: Google Scholar
Y. Cai and X. Wan, “Multi-domain sentiment classification based on domain-aware embedding and attention,” in Paper presented at the IJCAI, Macao, China, August 2019.
View at: Google Scholar
Qi Liu, Y. Zhang, and J. Liu, “Learning domain representation for multi-domain sentiment classification,” in Paper presented at the Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, LA, USA, June 2018.
View at: Google Scholar
R. He, W. Sun Lee, Ng Hwee Tou, and D. Dahlmeier, “An interactive multi-task learning network for end-to-end aspect-based sentiment analysis,” 2019, https://arxiv.org/abs/1906.06906.
View at: Google Scholar
J. Lafferty, A. McCallum, and F. C. N. Pereira, “Conditional random fields: probabilistic models for segmenting and labeling sequence data,” in Proceedings of the 18th International Conf. on Machine Learning, New York, NY, USA, June 2001.
View at: Google Scholar
M. Pontiki, D. Galanis, and H. Papageorgiou, “Semeval-2016 task 5: aspect based sentiment analysis,” in Paper presented at the International Workshop on Semantic Evaluation, San Diego, CA, USA, June 2016.
View at: Google Scholar
L. Van der Maaten and G. Hinton, “Visualizing data using T-sne,” Journal of Machine Learning Research, vol. 9, no. 11, 2008.
View at: Google Scholar
W. Liu, Z. Wang, and Y. Yuan, “A novel sigmoid-function-based adaptive weighted particle Swarm optimizer,” IEEE Transactions on Cybernetics, vol. 51, no. 2, pp. 1085–1093, 2019.
View at: Publisher Site | Google Scholar
W. Liu, Z. Wang, X. Liu, N. Zeng, and D. Bell, “A novel particle Swarm optimization approach for patient clustering from emergency departments,” IEEE Transactions on Evolutionary Computation, vol. 23, no. 4, pp. 632–644, 2018.
View at: Google Scholar
N. Zeng, Z. Wang, H. Zhang, K.-E. Kim, Y. Li, and X. Liu, “An improved particle filter with a novel hybrid proposal distribution for quantitative analysis of gold immunochromatographic strips,” IEEE Transactions on Nanotechnology, vol. 18, pp. 819–829, 2019.
View at: Publisher Site | Google Scholar
W. Liu, Z. Wang, N. Zeng, Y. Yuan, F. E. Alsaadi, and X. Liu, “A novel randomised particle Swarm optimizer,” International Journal of Machine Learning and Cybernetics, vol. 12, no. 2, pp. 529–540, 2021.
View at: Publisher Site | Google Scholar
I. Ur Rahman, Z. Wang, and W. Liu, “An N-state markovian jumping particle Swarm optimization algorithm,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2020.
View at: Publisher Site | Google Scholar
N. Zeng, Z. Wang, W. Liu et al., “A dynamic neighborhood-based switching particle Swarm optimization algorithm,” IEEE Transactions on Cybernetics, 2020.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2021 Yingjie Tian et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

1801

Downloads

1374

Citations

Complexity

Complexity Problems Handled by Advanced Computer Simulation Technology in Smart Cities 2021

[Retracted] Cross-Domain End-To-End Aspect-Based Sentiment Analysis with Domain-Dependent Embeddings

Abstract

1. Introduction

2. Related Work

2.1. Aspect-Based Sentiment Analysis

2.2. Transfer Learning in ABSA

2.3. Domain Embeddings for Sentiment Analysis

3. Methodology

3.1. Problem Statement

3.2. Methods

3.2.1. Word Embedding Layer

3.2.2. Cross-Domain ABSA Layer

3.2.3. Multitask Learning Strategy

4. Experiments

4.1. Datasets and Settings

4.2. Main Results

4.2.1. Ablation Study

4.2.2. Case Study

4.2.3. Development Study

4.2.4. Comparison Study

5. Discussion

6. Conclusions

Data Availability

Conflicts of Interest

Authors’ Contributions

Acknowledgments

References

Copyright