An Aspect-Sentiment Pair Extraction Approach Based on Latent Dirichlet Allocation for Turkish

: Online user reviews have a great influence on decision-making process of customers and product sales of companies. However, it is very difficult to obtain user sentiments among huge volume of data on the web consequently; sentiment analysis has gained great importance in terms of analyzing data automatically. On the other hand, sentiment analysis divides itself into branches and can be performed better with aspect level analysis. In this paper, we proposed to extract aspect-sentiment pairs from a Turkish reviews dataset. The proposed task is the fundamental and indeed the critical step of the aspect level sentiment analysis. While extracting aspect-sentiment pairs, an unsupervised topic model Latent Dirichlet Allocation (LDA) is used. With LDA, aspect-sentiment pairs from user reviews are extracted with 0.86 average precision based on ranked list. The aspect-sentiment pair extraction problem is first time realized with LDA on a real-world Turkish user reviews dataset. The experimental results show that LDA is effective and robust in aspect-sentiment pair extraction from user reviews.


Introduction
The online review websites are emerging as an important source with personal feelings, opinions, emotions, views and so on of millions of users about products, individuals, services, and more. Analysing opinions of users from these huge volume of reviews manually is very challenge task. As a result of this, an automatic analysis is needed and sentiment analysis has become an important and popular research area. Sentiment analysis, which has been studied since 1990s, is a type of subjective text analysis in the area of natural language processing (NLP), text mining and computational intelligence [1]. In the literature, sentiment analysis can be done in three levels of granularity; document level, sentence level and aspect level [2,3]. In document and sentence level analysis a general analysis as performed. Both of these analysis is the simplest way to do sentiment analysis. These approaches assume that document or sentence contains only one entity on which the sentiment is expressed on. On the other hand, to make a finer-grained sentiment analysis, aspect level analysis has gained great importance in recent years. Aspect is defined as the main entity which is commented on qualified by sentiment positively or negatively in the text. In the review sentence "Oda çok küçüktü. (The room is very small.)" "oda (room)" is the aspect and "küçük (small)" is the sentiment. And compared with the aspect level analysis both document level and sentence level, for instance, when the hotel reviews are analysed, instead of learning general opinions (good or bad) about the hotel, determining opinions about variety of aspects such as room or main course is more valuable and accurate. Consequently, three main tasks in aspect level sentiment analysis; aspect extraction, sentiment word extraction and aspect-sentiment pair extraction [4].
In this study, we aim to extract aspect-sentiment pairs from Turkish hotel reviews by using unsupervised method LDA for the purpose of carrying out aspect level sentiment analysis system. The proposed model is domain independent. For Turkish, LDA is applied for the first time to extract aspect-sentiment pairs. And also, in this study in addition to word aspects multi-word aspects (MWAs) are extracted by using Babelfy. The precision and recall values, which determine the coverage between the humangenerated and automatically generated aspect-sentiment pairs, are used as performance evaluator. The rest of the paper is organized as follows: related works are summarized in the Section 2, the theoretical background of LDA is given in Section 3The dataset description, feature engineering process, evaluation metrics and evaluation results are presented in Section 4. Finally, we conclude with a summary.

Related Works
In the literature, there is a limited study about aspect-sentiment pair extraction. However, there is not such a study, which uses LDA to extract aspect-sentiment pairs for Turkish. The model provides superiority of no need prior knowledge, and be applicable to all languages and domains. Both studies of Hu and Liu, association rule mining is used to extract aspect-sentiment pairs [5,6]. In a given sentence, for an aspect, the nearest sentiment word was accepted as specific to this aspect and they constituted a pair. Popescu and Etzioni benefited from syntactic dependencies computed by MINIPAR and composed ten extraction rules for aspect-sentiment pairs [7]. Chan and King devised corpus-based Feature-Opinion Association algorithm [8]. Algorithm intended to maximize relevance score between aspects and sentiments to match aspect and sentiment pairs. Huang et al. used Bootstrapping to extract aspect-sentiment pairs from Chinese customer reviews [9]. Brody and Elhadad utilized mutual information to determine representative sentiment words of the aspects [10]. Jo  Unification model, which is based on LDA, takes domainindependent sentiment seed list and finds sentiments are specific to aspects [11]. Kim et al. developed two-level tree based hierarchical aspect sentiment model by using Bayesian nonparametric model and recursive Chinese Restaurant Process to extract aspect-sentiment pairs [12]. Yin et al. enacted ontologybased linguistic model by constructing domain ontology automatically to learn semantic relation between aspects and sentiments from Chinese product reviews [13]. Klinger and Cimiano handled aspect-sentiment pair extraction as a joint inference problem and they used imperatively defined factor graphs for extracting pairs [14]. Zhou et al. incorporated domain independent language patterns and domain knowledge as a lexical base for aspect-sentiment pair extraction from Chinese restaurant reviews [15]. Quan and Ren extracted aspect-sentiment pairs based on dependency distance between aspects and sentiment calculated with dependency parser [16]. Wang et al. proposed LDA based Lifelong Aspect based Sentiment Topic (LAST) to determine pairs [17]. In this model, frequent itemset mining is used for finding frequently co-occurred pairs. Türkmen et al. developed pushdown-automata based aspect-sentiment pair extraction model, which used Turkish linguistic rules, for Turkish user reviews [4]. Amplayo and Hwang implemented Micro Aspect Sentiment Model (MicroASM), which is an LDA model, to generate aspectsentiment pairs for short reviews [18].

Latent Dirichlet Allocation
In recent years, topic models have become an active research area in machine learning and text mining applications. Topic models are the algorithms that discover the hidden thematic structure in the unstructured document collections by converting these collections to low dimensional space [19]. In this model, with the hidden thematic structure, main themes of the documents are implied. Such as, in the hotel reviews, the main themes of the reviews can be specific characteristics of the hotel that customers like or don't like.
In the literature, there are many topic models which were implemented by researchers in the past. Among them, LDA is the most popular and complete model. LDA is a generative graphical topic model for collections of discrete data such as text corpora [20]. The simplest basic idea behind LDA is topics have probability distribution over a fixed vocabulary and documents are composed of random mixture of latent topics. Based on this idea, LDA learns the followings; the set of topics, word probabilities related to these topics, the topic of each word, and the mixture of topics for each document [21]. LDA is a fully unsupervised and does not require any prior knowledge. LDA is based on the "bag-of-words" assumption. While the order of words in the document is ignored, LDA utilizes the co-occurrence of words in the same document. The generative model for LDA is given in Fig. 1. The generative process begins with sampling words from fixed vocabulary under topics. Then, distribution over topic proportions is sampled for every document. Distribution over words and topic proportions are obtained with Dirichlet distribution. For each word in the document, a topic from the topic distribution is randomly chosen. Finally, a word is sampled for related topic. For sampling steps in (i) and (ii) in Fig. 1., multinomial distributions are used.
For the graphical representation of LDA plate notation is used in Fig. 2. Plate notation is used to represent replicated variables in the graphical model. In Fig. 2., random variables are represented with nodes and directed edges are used to explain how these random variables are generated along with these edges [22]. Shaded node is reflected the words in the document which are observed and hidden variables are unshaded. In the graphical model for LDA, is the number of total topics to be extracted. While is the total number of documents in the collection, is total words in the ℎ document.
, is the ℎ word in the ℎ document. is the size of fixed vocabulary. and are Dirichlet hyperparameters.
is topic proportions in the documents and is the distribution over words for topics. Based on graphical model, the joint probability of observed and hidden variables is written as in the equation (1).
In the step of obtaining model parameters from equation (2) Collapsed Gibbs Sampling (CGS) is used. With LDA, we actually aim to learn s for each document so estimating and is trivial task and these two parameters are integrated out. Consequently, CGS is performed based on equation (3).

Preprocessing
User reviews as a form of electronic word-of-mouth includes emoticons, non-text characters, HTML tags, letters in capital, and misspelled words so preprocessing is very crucial step for those kind of documents to obtain accurate results. In this study, emoticons, non-text characters, HTML tags are removed from reviews. Letters in capital are converted to lowercase. For misspelled words, Turkish NLP library Zemberek [23] is used.

Multi-word Aspect Extraction
To make a better aspect level analysis multi-word aspects besides word aspects should be taken into account [24]. In this respect, Babelfy is used to extract multi-word aspects. Babelfy is a unified graph based approach, which realizes entity linking (EL) and word sense disambiguation (WSD) by selecting high-coherence semantic interpretations from densest subgraph heuristic with a loose identification of candidate meanings coupled [25]. Some of the extracted multi-words and their types are given in Table 1.
As it can be seen in Table 1, user reviews include two types of multi-words. While multi-words such as "tatil köyü (holiday village)" and "genel müdür (general manager)" are in the Turkish dictionary, "ana restoran (main restaurant)" and "hizmet kalitesi (service quality)" are not in the Turkish dictionary, they constitute multi-words depending on hotel domain.
After multi-word extraction, some additionally preprocessing steps are applied to reviews. Firstly, punctuations and digits are removed them stemming is applied by using Zemberek.

Aspect-Sentiment Pair Extraction
The basic assumption behind the proposed model is; LDA put together words into same topic that co-occurred in the same document.
In this study, the co-occurrence of aspects and sentiments in the same topic, the set of topics, word probabilities related to these topics, and the mixture of topics for each document obtained from LDA are used to extract aspect-sentiment pairs from user reviews as in the equation (6).
In equation (6), ( | ) is the number of topics that includes both sentiment and aspect . ( | ) probability of aspect under topic . ( | ) probability of sentiment under topic . is the number of total topics and is the number of total documents in the collection. Based on the equation (6), for each aspect, probabilities of aspect-sentiment pairs are sorted largest to smallest.

Dataset
To perform aspect-sentiment pair extraction, we employ hotel reviews in Turkish. Dataset contains 1517 user reviews from a Turkish tourism website www.otelpuan.com. The summary of dataset is depicted in Table 2.

Evaluation Measure
In this study, aspect-sentiment pair extraction model is accepted as an Information Retrieval (IR) system and to evaluate the effectiveness and efficiency of the model ranked lists of each aspect-sentiment pairs are established. Our study examines not only the acceptable aspect-sentiment pairs, but also the ranked list of the aspect-sentiment pairs according to their co-occurrence frequency in documents. The ranked list for aspect-sentiment pairs also makes us the see coverage between the human-generated and automatically generated aspect-sentiment pairs. Given a list of aspect-sentiment pairs = { 1 , 2 , … , } which is sorted by most probable ( 1 ) to least probable ( ) based on equation (6). The precision and recall values at each in the ranking are computed by using Table 3.
Precision at position i is denoted by ( ) is calculated with equation (10).
In the equation (10), is the total number of aspectsentiment pairs, which human agree on, up to level i. Recall at position i is denoted by ( ) is calculated with equation (11).
There is no need to learn all agreed aspect-sentiment pairs and we can cut off ranked list at predetermined rank such as . Consequently, average precision is calculated with equation (12).
The quality of extracted aspect-sentiment pairs by proposed model is evaluated according to equation (12).

Experiments and Results
In LDA iteration counts, number of total topics and Dirichlet hyperparameters and are user defined. For the experiments, LDA runs for 1000 iterations of Collapsed Gibbs Sampler. Number of topic is set to 100. is 50⁄ and is 0.01. Each extracted topic is represented with its first twenty words. Example of one of the extracted topic is given in Table 4. In Table 4, while "mükemmel, düşük, super, başarılı, yakışmayan, and çamlık" are sentiments, "bar, sezon, ets, mevsim, güleryüz, fiyat kalite oranı, grup, havuz etkinliği, meyhane, öğle servisi, yedirme, otel, tatil, and resepsyion" are aspects. After obtaining topics, by using predefined aspect and sentiment lists, aspect-sentiment pairs are extracted from these topics. The predefined aspect and sentiment lists are determined automatically by detecting nouns and adjectives. For a specific aspect "bar", the extracted aspect-sentiment pairs and their probabilities based on equation (6) are given in Table 5. To obtain average precision we cut off ranked lists of each aspectsentiment pairs at rank 5. This ranking level decision can be change from user to user. However, if more aspect-sentiment pairs are included into the ranked list, the more rare seen sentiments for an aspect are arose. Examples of some of the ranked lists of extracted aspect-sentiment pairs are given in Table 6. When the proposed model is realized, 153 different aspects are obtained. For experiments 765 aspect-sentiment pairs are evaluated (for each aspect 5 top sentiments are evaluated) and average precision our model is calculated as 0.86. Quantitative results show that model correctly and effectively identifies the aspect-sentiment pairs for hotel domain. Besides, when the results are evaluated qualitatively, it has seen that interesting pairs such as "personel-yeterli", "personeldüzgün", "kahvaltı-çeşitli" are extracted with LDA. Also, this is the strength of the LDA that does not depend on the language and domain and does not need any prior knowledge. So it can be applicable to various domains with any language.

Conclusion
In this paper, we proposed aspect-sentiment pair extraction model for Turkish, which is very important step for successful aspect level sentiment analysis. For this, we applied LDA, which has never used before for Turkish for this task. The obtained result shows that LDA is capable to capture aspect-sentiment pairs from reviews of hotel with high precision value.