Environmental Science and Pollution Research https://doi.org/10.1007/s11356-023-26780-1 APPLICATIONS OF EMERGING GREEN TECHNOLOGIES FOR EFFICIENT VALORIZATION OF AGRO-INDUSTRIAL WASTE: A ROADMAP TOWARDS SUSTAINABLE ENVIRONMENT AND CIRCULAR ECONOMY Entropy-based grid approach for handling outliers: a case study to environmental monitoring data Anwar Shah1 · Bahar Ali1 · Fazal Wahab2 · Inam Ullah3 · Kassian T.T. Amesho4,5,6,7,8,9 · Muhammad Shafiq10,11 Received: 5 November 2022 / Accepted: 29 March 2023 © The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2023 Abstract Grid-based approaches render an efficient framework for data clustering in the presence of incomplete, inexplicit, and uncertain data. This paper proposes an entropy-based grid approach (EGO) for outlier detection in clustered data. The given hard clusters obtained from a hard clustering algorithm, EGO uses entropy on the dataset as a whole or on an individual cluster to detect outliers. EGO works in two steps: explicit outlier detection and implicit outlier detection. Explicit outlier detection is concerned with those data points that are isolated in the grid cells. They are either far from the dense region or maybe a nearby isolated data point and therefore declared as an explicit outlier. Implicit outlier detection is associated with the detection of outliers that are perplexedly deviated from the normal pattern. The determination of such outliers is achieved using entropy change of the dataset or a specific cluster for each deviation. The elbow based on the trade-off between entropy and object geometries optimizes the outlier detection process. Experimental results on CHAMELEON datasets and other similar datasets suggested that the proposed approach(es) detect the outliers more precisely and extend the capability of outliers detection to an additional 4.5% to 8.6%. Moreover, the resultant clusters became more precise and compact when the entropy-based gridding approach is applied on top of hard clustering algorithms. The performance of the proposed algorithms is compared with well-known outlier detection algorithms, including DBSCAN, HDBSCAN, RE3WC, LOF, LoOP, ABOD, CBLOF and HBOS. Finally, a case study for detecting outliers in environmental data has been carried out using the proposed approach and results are generated on our synthetically prepared datasets. The performance shows that the proposed approach may be an industrial-oriented solution to outlier detection in environmental monitoring data. Keywords Environmental monitoring data · Entropy · Grid · Hard clusters · Industrial · Outliers Introduction An outlier is a data point or group of points that deviated from the normal pattern of a given dataset. In general, outliers can be divided into three classes based on their behavior: global, conditional or contextual and collective outliers. The global outliers deviate significantly from the entire dataset (Liu et al. 2010; Yang et al. 2020). The conditional outliers deviate from the rest of the dataset based on some selected context (Bharti et al. 2019; Wang and Davidson 2009). In contrast, collective outliers are a group of instances collectively deviated from Responsible editor: Marcus Schulz B Muhammad Shafiq Srsshafiq@gmail.com Extended author information available on the last page of the article the rest of the dataset (Rajeswari et al. 2018; Sitanggang and Baehaki 2015). Outlier occurs due to several reasons, such as inappropriate scaling, improper data collection, measurement errors, sampling errors, and human and experimental errors (Campello et al. 2013; Campos et al. 2016b; Chen et al. 2017; Mia Hubert and Segaert 2015). Outlier detection plays a major important role in almost every quantitative discipline, such as cybersecurity, finance, and machine learning (Shah et al. 2021, 2022; Shafiq et al. 2020b, a). It has many applications, including fault diagnosis, web analytics, medical diagnosis, fraud detection, criminal activities detection, and malware detection (Andersson et al. 2016; Jabez and Muthukumar 2015; Lin and Brown 2006; Lucas et al. 2020; Malini and Pushpa 2017; Sandosh et al. 2020; Shafiq et al. 2020c). Outlier detection is an essential task in data processing that may cause severe effects in the final evaluation of results. 123 Environmental Science and Pollution Research Therefore, several approaches have been devised to handle outlier detection. These approaches include statisticalbased approaches, distance-based approaches, density-based approaches, deviation-based approaches, subspace-based approaches, depth-based approaches, and clustering-based approaches. We are handling outlier detection in the context of clustering. The main focus of cluster-based outlier detection methods is to define and observe compactness in clusters to find outliers. The outlier may be an independent object or appears as an individual cluster (Christy et al. 2015). The clustering methods implicitly define an outlier or group of outliers being background noise where the clusters are embedded in the foreground (Ester et al. 1996; Qiu et al. 2003; Luo et al. 2007). Density-based spatial clustering is a well-known method to discover clusters of applications and the surrounding noise in the spatial database (Ester et al. 1996). This method defines outliers based on densities or the collection of objects. An object with a low density and the same parameters is considered an outlier. Several extensions and variants to this method are proposed to handle various problems (Birant and Kut 2007; Borah and Bhattacharyya 2004; Duan et al. 2007; He et al. 2014; Louhichi et al. 2014; McInnes et al. 2017). Many researchers work in the same direction by modifying, extending, improving or refining the basic approach to handle different applications (Birant and Kut 2007; Borah and Bhattacharyya 2004; Duan et al. 2007; He et al. 2014; McInnes et al. 2017; Tran et al. 2013). Some approaches determine an object as an outlier based on its farness from the centroid of the cluster (Hautamäki et al. 2005; Zhang and Leung 2003). An approach towards group outliers detection has been devised in Jiang et al. (2001), which considers small-size clusters as group outliers. Another approach is to detect outliers based on a separate cluster called noise or outlier (Gan and Ng 2017; Rehm et al. 2007). The potential outliers in the dataset have a high association level with that cluster. Besides these approaches, fuzzy and graph-based methods have also been devised to handle outliers (Xu et al. 2007; Yang et al. 2010). These existing approaches employ some threshold or constraint to isolate outliers from inliers. There are some key issues with these approaches. The first issue is the determination of a suitable threshold. The second issue: they determine a single criterion for a multi-class dataset. In a multi-class environment, each class may be of different density, structure and geometry, where a single criterion may fail to determine all the outliers in the dataset. We introduce an entropy-based gridding approach, EGO and examine its efficiency and application in outlier detection. The EGO is motivated by the fact that entropy is related to the randomness of a system (Guseva and Kuznetsov 2017). More specifically, entropy measures the separation of an object from the centroid of its corresponding cluster. Removing an object can lead to a drop in the entropy of a 123 particular cluster or dataset. The drop of an object(s) from a family of objects having the same distribution (sparse or dense) can affect the entropy drop in a systematic way which determines that the instances are the members of the same cluster and not outliers. In case of removal of an object(s) where the entropy drop is deviated from regular drops, determine the presence of outlier(s). The EGO can be applied on the whole dataset at once or individual cluster(s) as EGOC B . The EGOC B works on cluster basis where it determines the outliers corresponding to each given hard cluster. The refinement capability of EGOC B is comparatively more than EGO at the cost of increasing time complexity. The stopping criteria of EGO and EGOC B can be optimized using the entropy-geometry elbow. These approaches are evaluated against the inverse relation of objects geometry and entropy drop for each specific geometry of the objects in the underlying grid. Experimental results on two-dimensional real and multi-dimensional textual datasets suggest that entropy-based outlier detection detects and removes an additional 4.5% to 8.6% outliers that were undetected with DBSCAN or HDBSCAN. Moreover, it leads to more cluster compactness and better cluster quality compared to state-of-the-art outlier detection methods. Furthermore, the proposed approaches have been employed for environmental monitoring data taken as a case study. Environmental monitoring data is essential to analyze and detect harmful residuals impacting human health. Usually, industrial plants have monitoring systems and meteorological stations that enumerate the key variables of air quality in the presence of the residuals released by these complexes. These analyzed measurements may be contaminated with outliers that must be discarded to obtain a consistent dataset. To evaluate the proposed approach’s performance, synthetic outliers have been added to the environmental monitoring data. The correlation for fresh air attributes components is carried out using Pearson’s Correlation Coefficient (PCC) (Benesty et al. 2009). Similarly, the correlation of the attributes is computed in the presence of outliers. The attributes are coordinated based on the correlation that separates abnormal patterns from regular patterns. The proposed approach then detects the outliers based on the entropy-based gridding approach. The detection accuracy of the proposed approach for environmental monitoring data is 96.67% which suggests the industry orientation of our methods. The remaining paper has been structured as follows: Background is discussed in the “Background” section, where the “Entropy-based gridding approach using elbow method” and “Experimental results and evaluations” sections describe the proposed approaches and the conducted experiments based on these approaches, respectively. The “Conclusion” section is the conclusion, which is an inference to the paper. Environmental Science and Pollution Research Background This section introduces the background required to explain the proposed approach(es). In particular, we discuss data scaling, gridding, and entropy. Data scaling The density-based clustering is capable of identifying arbitrary size or shaped data clusters while observing difficulty in obtaining clusters with varying densities. A number of efforts have been made to cope with this challenge (Güngör and Özmen 2017; Zhu et al. 2018, 2016). For instance, Rescale is a one-dimensional density-ratio scaling approach where DScale is a multi-dimensional distance-based scaling preprocessing technique (Zhu et al. 2018, 2016). DScale scales all the computed distances between the instances of a dataset. Consider a Universal set U = {x1 , x2 , x3 , …, xn } with n objects. Each object xi has A attributes, i.e., xi = (xi1 , …, xiA ), where xia represents the ath attribute of object xi . To rescale the computed distance d(x, y) to d (x, y), DScale define the scaling function r(x) as, r (x) = |Nη (x, d)| × m A n × ηA A1 (1) Entropy where Nη (x, d) = {y ∈ U| d(x,y) ≤ η} is the η-neighborhood of x, η ∈ (0, 1) is the radius parameter for neighborhood, m = max x,y∈U d(x, y) is the maximum distance between x and y and n is the number of objects inside U . The scaling function r(x) scale the computed distance matrix DM = [d(x,y)] to scaled distance matrix D M = [d (x, y)] as, d (x, y) = d(x, y) × r (x) (d(x, y) − η) × m−η×r (x) m−η (Amini et al. 2014; Erskine et al. 2006; Kotsiantis and Pintelas 2004; Rai and Singh 2010; Rokach 2009; Xu and Tian 2015). These algorithms are used to formulate solutions for many problems such as outlier detection, clustering, task scheduling and pathfinding (Agrawal et al. 1998; Bai et al. 2016; Blythe et al. 2005; Lee and Cho 2016; Liao et al 2004; Ma and Chow 2004; Mahmoud et al. 2016; Ohadi et al. 2020; Park and Lee 2004; Pilevar and Sukumar 2005; Sheikholeslami et al. 2002; Wang et al. 2009, 1997; Yap 2002). The performance of these algorithms depends upon the granularity and mesh refinment (Liao et al 2004; Ohadi et al. 2020). Some notable granularity and refinement approaches operate on a static grid, dynamic grid, local grid and adaptive grid (Batra and Ko 1992; Berger et al. 1989; Berger and Oliger 1984; Eiseman 1987; Fakhari and Lee 2014; Fuchs 1986; Osekowska et al. 2014; Rencis and Mullen 1986). Grid-based approaches are employed in two steps. First, collect the statistical data corresponding to each cell, using precisely selected grid (Batra and Ko 1992; Berger et al. 1989; Berger and Oliger 1984; Eiseman 1987; Fakhari and Lee 2014; Fuchs 1986; Rencis and Mullen 1986). Finally, perform the specific operation like outlier detection or clustering on the deployed grid without accessing the data inside the database. i f y ∈ Nη + η × r (x) i f y ∈ U \ Nη In the area of machine learning, entropy defines the randomness or impurity of a given dataset. The entropy model of Shanon uses a base 2 logarithmic function to measure the rate of entropy, i.e., log2 (P(x)), because an increase in the probability P(x) of an event tends the result towards binary logarithm one as depicted in Fig. 2. In the case of more than one event or element, accumulative entropy can be calculated using the following equation, (2) The above equation defines two cases, one is to rescale distance between x and the elements inside its η-neighborhood using linear scaling where the number of elements remains same. The second case is to rescale the distance between x and y ∈ U \ Nη using min-max normalization to keep the same object rank. The effect of rescale can be simulated as in Fig. 1. Griding and grid-based algorithms A grid is a collection of uniformly arranged straight lines, intersecting each other at equal intervals, representing a series of connected rectangles or squares. Grid-based approaches add value to the performance of algorithms due to their scalability, lower time complexity and parallel processing nature H (x) = − n−1 P(x) log2 (P(x)) (3) i=0 where H(x) is the Shannon entropy, P(x) is the probability of occurrence of the random variable x and P(x) is the contained information. The above equation is the formal definition of entropy which is the baseline to calculate the information gain of a system. The alternative approach to randomness is distance where the elements are more similar to each other as they became close to each other in some feature space. Therefore, we argue that distant or remoteness can be replaced with randomness in those situations where the elements are weighed in same feature space holding different positions. The volume of the system varies with distance among the objects. let us consider Fig. 3 to completely understand the underlying approach. 123 Environmental Science and Pollution Research Fig. 1 Data scaling of with DScale In Fig. 3(a), a circle representing cluster c1 with a centroid co visualized by a black point in the center. The red points represent the objects of the cluster c1 . The Euclidean-based distance (d f ar ) between the object o1 and centroid co is d1 . It is inversely proportional to the probability of similarity (Psim ) of object o1 to centroid co , which is given as follow, Psim ∝ 1 d f ar Psim = C × the center and red points are the objects belonging to cluster c1 . The Euclidean-based distance between the object o1 and centroid co is d2 . The entropy of object o1 will be increase, because d2 > d1 , which leads to increase in the accumulative entropy from E 1 to E 2 . Mathematically, we have E1 = − 1 d f ar n−1 P(x) log2 (P(x)) i=0 (4) =− n−2 P(x) log2 (P(x)) + P(x1 ) log2 (P(x1 )) i=0 where C is the constant of proportionality which depends upon the weight or importance of objects, generally its value is equal to 1. Furthermore, E 1 is the accumulative entropy for Fig. 3(a). Similarly, in Fig. 3(b), the circle represents the cluster c1 with a centroid co visualized by black point in Fig. 2 A graphical representation of entropy of an event 123 =− n−2 Psim (x) log2 (Psim (x)) + Psim (x1 ) log2 (Psim (x1 )) i=0 =− n−2 i=0 Psim (x) log2 (Psim (x)) + 1 1 (5) log2 (d1 ) (d1 ) Environmental Science and Pollution Research Fig. 3 Visual demonstration for distantial entropy of a cluster Similarly, ⎡ n−2 E2 = − ⎣ Psim (x) log2 (Psim (x)) + i=0 ⎤ 1 ⎦ 1 log2 (d2 ) (d2 ) (6) from Eqs. 5–6, E 2 > E 1 as d2 > d1 . Environmental monitoring data In order to have a pollution-free environment, the process of environmental data monitoring is a must-to-do task. Pollution is directly associated with industrial plants and their operations. The analysis is carried out to observe the range of contaminants like ozone O3 , nitrogen dioxide N O2 , lead, excessive carbon and its byproducts and other dangerous-tohealth particulate materials measured in the units of mg/m 3 or parts-per-billion (ppb) by chromatographs. The excessiveness of these substances in the air may cause dangerous impacts on human health; for instance, excessiveness of N O2 may cause skin irritation. High levels of S O2 may cause bronchitis, asthma and even heart attack. The air quality stations measure the contaminated substances, whereas the meteorological stations measure the wind speed, direction, humidity, and temperature. The environmental monitoring data is obtained using chromatographs and stored in the environmental database to be used for analysis. Unfortunately, the data contain abnormalities due to different possibilities, such as instrumentation faults, communication channel faults, electrical faults, and problems with emission sources. This leads to masking and swamping effects. Masking means to consider an abnormal measurement as a normal measurement, while swamping is to consider a normal measurement as an abnormal measurement. That can lead to severe effects on the precautionary measures taken based on these analyses. Many authors formulate solutions to cope with these types of outliers. Pearson works on the identification of systemgenerated outliers. He discussed that an outlier is not always a byproduct of wrong measurements, but it can be caused due to faulty system operations (Pearson 2002). The severity of these outliers has also been discussed in Kadlec et al. (2009); Warne et al. (2004). Hugo and Daniel presented a nonlinear approach to detect outliers (Garces and Sbarbaro 2009). Brahim Alameddine et al. comparatively analyze three outlier detection mechanisms on environmental monitoring data, particularly water quality in different lakes (Alameddine et al. 2010). Petr Visilik et al. automate the process of outlier detection based on statistical analysis (Veselík et al. 2020). Entropy-based gridding approach using elbow method In this section, we present a gridding approach that makes use of the elbow method based on the entropy measurement of each cluster. The entropy is used to measure the randomness in the clusters while the elbow determines the feasibility for objects to decide them as outlier or inlier. In particular, for improved gridded hard cluster(s), we calculate an initial entropy for the system and keep on evaluating the entropy for implicit outliers with different geometries. The elbow method finds the outliers by establishing a trade-off between the entropy and suspected outliers. Data scaling It is not always true that a dataset contains clusters or classes with the same densities. Therefore, a conventional machine- 123 Environmental Science and Pollution Research learning algorithm behaves unexpectedly in the case of varied densities clusters. A data scaling is therefore required to balance the densities of all the clusters to meet the optimum results. There are two scaling methods namely, data-ratio and DScale (Zhu et al. 2018, 2016). The data-ratio and DScale scaling method are used for two-dimensional and n-dimensional data, respectively. We use DScale for data scaling using Eq. 2. The distance matrix D M is computed based on Euclidean distance in the first step, which is transformed to a scaled matrix D M using Eqs. 1–2. Data scaling may improve the clustering process (Zhu et al. 2018, 2016). Hard clustering Let D = { x1 , x2 , . . ., xn } is the dataset containing finite number (n) of objects. The scaling of distances between the objects results in a computed scaled matrix D M . A conventional machine learning algorithm cluster the scaled data into groups or clusters based on the similarities between objects such that C = {c1 , c2 , . . ., c K }. The conventional machinelearning algorithms used in this paper are HDBSCAN (for initial hard clusters with noise removal) or k-means (for oval shaped initial hard clusters). Explicit outliers detection Grid representation The next step is to represent the dataset in a grid. The proposed approach(es) work on normalized attributes and division of the unit square (in case of two attributes) or unit hypercube (in case of more than two attributes) into an equally distributed grid (Gu et al. 2017). This approach uses the Euclidean distance metric to measure the influence of each attribute. Therefore, first, we normalize the attributes A in the range [0,1] to balance the effect of each attribute during distance analysis (Goldstein 2014). It scales the whole problem space to a unit hypercube. There are O(n.A) operations required to complete this step. Further, we divide the unit square (in case of two attributes) or unit hypercube (in case of more than two attributes) into p number of equidistant parts such that p ∈ N. Hence, the total number of grid cells became p A where p is the number of grid cells in one dimension and A is the number of attributes in the data space. Each grid cell can be located through A-tuple ( j0 , j1 , . . ., j A−1 ) where jk ∈ {0, 1, . . ., p -1}. Furthermore, we can map this multidimensional grid to a single value index, i.e., I = {0, 1, 2, . . ., p A }. Algorithm 1 demonstrates the grid-based representation of data space as shown in Fig. 4 and supported by Table 1. The clustered data is logically gridded using Algorithm 1. The objects in the dataset are thereby tested for a possible outlier. The grid plays a key role in the process of explicit outliers detection. An object having all its immediate neighboring cells empty is an exceptional case, which is an obvious outlier. The rest of objects are arranged in sets based on neighboring empty cells. Each object in these sets is examined for possible outliers based on their acquired geometry. Entropy measurement The process of explicit outlier detection is followed by implicit outlier detection mechanism. The initial entropy of the remaining dataset is determined say E 1 . The elements are divided in eight sets with respect to their immediate empty neighboring cells, i.e., G e = {e1 , e2 , . . ., e8 }, where ei represents elements with i number of immediate empty cells. Now, to inquire the effect in entropy by declaring a set of objects as outlier, we compute a power set of G e . The power set P(G e ) has all possible combinations of objects that exist in various geometries, arranged in 28 subsets of set G e . It may be noted that we do not consider ∅ in our analysis such that G e = P(G e ) - 1. Each time a subset of the G e set is examined for outliers verification, the entropy is calculated for the dataset excluding that subset. The graph created an elbow at a specific point declaring the possible boundary between outliers and inliers. Elbow Fig. 4 Data points representation on a grid 123 In machine learning, we often use the sum of the squared distance-based elbow method for determining the number of clusters. Similarly, an entropy-based graph plotted against the objects with various geometries. An elbow evolved at the point of the absurd jump, which determines the boundary between outliers and inliers. Moreover, the process may potentially be repeated using various settings of empty cells around an object such as 3 by 3 or 5 by 5 cells. The increase in processing area leads to more outliers detection. Environmental Science and Pollution Research Table 1 Hash table to map data points in a grid Object Grid Position(I) Index Cells x y x1 I1 4 3.3 18.5 x2 I2 4 3.8 18.2 x3 I3 10, 11 1 x4 I4 27, 28 x5 I5 38, 39 x6 I6 43, 44 x7 I7 15, 16, 21, 22 A special case (group outliers) Group outliers are a special case of outliers. These outliers may exist in groups and be located away from normal clusters. The outliers detected in the above steps be verified against their neighbors as follows, – Case 1: The immediate neighboring cells (cells ) containing outliers. – Case 2: An immediate neighbor is not an outlier; we check all of its neighborhood elements. If all the elements in its neighborhood match the threshold for most of their neighbors, that is acquired for other outliers then decide them as group outliers. – Case 3: An immediate neighbor which is not an outlier; we check all the immediate neighbors of that non-outlier object. The object decided as inlier if most of its immediate neighbors are inliers. I denti f ier I d Location I d 1 I d1 2 Cells (I3 ) 2 4 I d2 2 Cells (I4 ) 3 9 I d3 2 Cells (I5 ) 4 16 I d4 2 Cells (I6 ) 3 12.5 I d5 4 Cells (I7 ) Here oci is the centroid of cluster ci and oi is the object which belongs to the cluster ci . The attributes for each object are denoted by A. The next step is to scale the distances between zero and one. The objects are categorized based on their geometry. Each category of objects is removed and the entropy of the complete dataset is calculated. The computed entropy is plotted against the geometry. The algorithm iterated for each specific geometry of objects and its effect on the entropy of the dataset is recorded. The maximum change in the entropy compared to the initial entropy is considered as the boundary between inliers and outliers. Similarly, the process can be repeated for cluster-based entropy analysis which is more robust. The entropy of each cluster is calculated and plotted against objects geometries. We get an entropy-geometry elbow for each cluster. Finally, we get an optimum cleaned dataset. Algorithm to grid the space A demonstration to entropy measurement in the proposed approach(es) The entropy measurement is the backbone of the proposed method(s). To understand the underlying mechanism, we consider a dataset as shown in Fig. 5. There are two clusters c1 and c2 in the dataset, represented by blue and green points, respectively. The hard clusters are obtained using a clustering algorithm, i.e., k-means for oval-shaped clusters and HDBSAN for non-oval shaped cluster dataset. The geometry of different objects is grasped and stored in a hash table similar to Table 1. The objects with one or more nonempty nearest neighbor cells are considered inlier; however, objects with all empty nearest neighboring cells are considered as outliers in the first place. Next, the centroid for each cluster is computed. The distance between the centroid and each of the object belonging to that cluster is computed using Euclidean distance metric as follow, d(oci , oi ) = A (ocai − oia )2 a=1 (7) This section introduces an algorithm to logically grid the space. The algorithm Grid-Map is shown as Algorithm 1. It is used to grid and map the objects of a dataset to the constructed grid. A scaled matrix D M is used as the only input to the algorithm. The outputs include a grid-based representation of the data objects, number of objects in each cell n[ p] and mapping of the dataset coordinates to the grid index (I ) = {0, 1, . . ., p A }. The algorithm receives a universal set U and a scaled matrix D M of the distances between the objects of U . The first for loop is used to normalize the attributes A in range [0, 1] for each object in the universal set U . The second for loop is used to divide the normalized attributes into p parts or equivalently the unit hypercube (in case of multi attributes) into p A grid cells. Grid cells are represented as gridcell. The collection of all grid cells produces a grid. The third for loop is to count the number of objects in each cell and store them in the array n[i] corresponding to the index (I ). The final loop in the algorithm is used to map the coordinates of each instance to its position on the grid as shown in Table 1. The object located in more than one cell is given an identifier which 123 Environmental Science and Pollution Research Fig. 5 Grid-based representation of a dataset removes the redundancy in the process of reverse mapping in a later stage. This algorithm returns a grid G, the number of objects in each cell of the grid n[i] and their mapping to the coordinate system. Algorithm for complete dataset-based EGO This section elaborates the working mechanism of EGO Algorithm 2 for a complete dataset. The input to this algorithm is a universal set with a finite number of instances. The output of the algorithm is a set of outliers OU T and a set of non-outliers N on OU T which represents compact clusters. The algorithm starts by computing the scaled matrix D M from the distance matrix D M of objects in universal set U . Line 2 is a function call, it takes a single argument and returns the three values, i.e., initial grid G to represent these objects, the number of objects in each cell n[i] of grid G and mapping of objects in a coordinate system to the cells in the initial grid G. In line 3, the algorithm partitions the data using a hard clustering algorithm in case of unlabeled data or creates partitions based on classification labels. Line 4 calculates the entropy EU for the complete dataset using Eq. 3. There are two for loops in this algorithm. The first for loop in lines 5–10, each cell in grid G is examined to be empty or with empty neighbors. These cells be excluded and the objects in these cells are considered as explicit outliers. Next in lines 11–16, the algorithm enters in a nested for loop. The loop goes through power set G e , excludes the cells and calculates the entropy. Line 17 determines the optimum entropy by considering a comparatively abrupt elbow. Line 18 cumulates the objects in the cells, which causes an elbow. Lines 19 and 20 separate the outliers and inliers into two sets OU T and N on OU T . Algorithm 2 An algorithm for complete dataset-based EGO Algorithm 1 Grid-Map algorithm Input A scaled matrix D M from Algorithm (2 or 3) , p > 1 Output A Grid G, Number of objects in each cell n[i], Mapping of data into grid 1: function Grid-MapD M 2: for each xi ∈ U do 3: Normalize A attributes between 0 and 1 4: end for 5: for each a in A do 6: Divide A unit intervals ∈ [0,1] in p parts (or equivalently) unit 7: hypercube [0, 1] A into p A grid cells (gridcell) to obtain G 8: end for 9: for each i in p A do 10: n[i] = Count number of objects 11: end for 12: for each xi in U do 13: Map coordinates of xi to gridcell ∈ G 14: Check for duplicate entry 15: if xi ∈ more than one cell then 16: mark it with identifier 17: end if 18: Get Mapping 19: end for 20: return (G, n[i], Mapping) 21: end function 123 Input A universal set U = {x1 , x2 , x3 , . . . xn }. Output The OU T and N on OU T representing the sets of outliers and inliers. 1: Compute a distance matrix D M from U 2: (Grid(G), n[ind], Mapping) = Grid-MapD M it is a function call 3: Obtain initial partitions C = {c1 , c2 , c3 , ..., c K } using hard clustering algorithm or data labels. 4: Calculate entropy EU for the dataset using Eq. 3 5: for each cell ∈ G do 6: Examine the neighboring cells 7: if All neighbor cells = ∅ ∨ cell = ∅ then 8: cell is empty or otherwise having outliers. 9: end if 10: end for 11: for i in G e do 12: for each non-empty cell ∈ G do 13: Exclude all the cells having i empty neighbors 14: end for 15: Calculate entropy E i for the dataset using Eq. 3 16: end for 17: Compare E i for optimum entropy drop E using elbow 18: Detached = cells causing optimum entropy E 19: OU T = Mapping(Detached → objects) 20: N on OU T = Mapping(U − Detached → objects) Environmental Science and Pollution Research Algorithm for cluster-based EGO (EGOCB ) Algorithm for group outlier-based EGO (EGOclique ) Algorithm 3 works for individual cluster and the result is computed at the end. A universal set with a finite number of instances is an input to the algorithm. There are two outputs to this algorithm, i.e., a set of outliers OU T and a set of nonoutliers N on OU T which represents the compact clusters. The algorithm computes a scaled matrix D M for the objects in the universal set U . In line 2, a function with a single argument is called which returns three values, i.e., initial grid G to represent these objects, the number of objects in each cell n[i] of grid G and mapping of objects in a coordinate system to the cells in the initial grid G. Line 3 creates initial partition using a hard clustering algorithm or data labels. Line 4 computes the entropy EU for the complete dataset using Eq. 3. There are two for loops used in this algorithm. The first for loop in lines 5–10, each cell in grid G is examined to be empty or with empty neighbors. These cells and the objects inside them are considered to be explicit outliers. Lines 11–18, algorithm enters in a nested for loop. The algorithm loops for each cluster ck , calculate initial entropy of cluster ck , loops in through power set G e , exclude the cells, calculate the entropy of ck again and ends with finding an optimum entropy E i . Line 19 cumulates the objects in the cells, which cause an elbow for each ck . Lines 20 and 21 separate the outliers and inliers into two sets OU T and N on OU T . This section presents the EGO Algorithm 4 for detecting group outliers. The set of inliers N on OU T and the desired number of clusters n d are the two inputs to this algorithm. This algorithm gives a family of group outliers OU Tclique . The algorithm computes a scaled matrix D M for the objects in a universal set U . Line 1 maps the given inliers to the coordinate system using Algorithm 1. Lines 2–9 loop for each cell in the grid G followed by an if-else statement, that checks possible group outliers. The if-else statement suggests group outliers, if all immediate neighbors contain outliers or a maximum number of immediate neighbors of the immediate neighbors contain outliers otherwise inliers. Line 10 checks for optimum entropy drop for each group and a combination of different groups to obtain the desired number of clusters. Line 11 removes the non-desired regions. Lines 12–14 decide the group of outliers based on the optimum entropy. Algorithm 3 Algorithm for cluster-based EGO Input A universal set U = {x1 , x2 , x3 , . . . xn }. Output The OU T and N on OU T representing the sets of outliers and inliers. 1: Compute a distance matrix D M from U 2: (Grid(G), n[ind], Mapping) = Grid-MapD M it is a function call 3: Obtain initial partitions C = {c1 , c2 , c3 , ..., c K } using hard clustering algorithm or data labels. 4: Calculate entropy EU for the dataset using Eq. 3 5: for each cell ∈ G 6: Examine the neighboring cells 7: if All neighbor cells = ∅ ∨ cell = ∅ then 8: cell is empty or otherwise having outliers. 9: end if 10: end for 11: for each ck ∈ C 12: Calculate entropy EU for ck using Eq. 3 13: for i in G e do 14: Exclude all the cells having i empty neighbors 15: Calculate entropy E i for the ck using Eq. 3 16: end for 17: Compare E i for optimum entropy drop E using elbow 18: end for 19: Detached = cells causing optimum entropy E for each ck 20: OU T = Mapping(Detached → objects) 21: N on OU T = Mapping(U − Detached → objects) Algorithm 4 An algorithm for group outlier-based EGO Input Set of inliers N on OU T and set of inliers from Algorithm 2 Algorithm 3. Output The set OU Tclique representing the group outliers. 1: Get Mapping for U using Algorithm 1. 2: for each cell ∈ G 3: if cells of cell contain outliers by elbow ∨ ∃cells | Max(cells (cell)) contain outliers by elbow then 4: A group of outliers 5: else if ∃cells | Max(cells (cell)) contain inliers then 6: Inliers 7: Calculate the entropy E i using Eq. 3 8: end if 9: end for 10: Check optimum entropy E drop for each group(s) 11: Detached = Remove the cells 12: if E is optimum then 13: OU Tclique = Mapping(Detached → objects) 14: end if or Experimental results and evaluations This section reports the results of our approach and its comparative analysis with some of the best outlier detection approaches. We measure the performance of our new algorithms EGO and EGOC B in an unsupervised scenario. We use the traditional clustering algorithms that perform cluster generation along with outlier detection. Then, the application of the proposed approaches on the initially purified clusters investigates its capability to further outlier detection and cluster purification. These experiments are performed using five unlabeled and three labeled datasets. The unlabeled datasets include t4.8k, t5.8k, t8.8k, t7.10k and A3 123 Environmental Science and Pollution Research datasets where unlabeled datasets are D31, 20newsgroup and 50 class amazon reviews. The t4.8k, t5.8k and t8.8k are bi-featured datasets containing 8,000 objects each and t7.10k dataset has 10,000 instances including many noise points (Karypis et al. 1999; McInnes et al. 2017). The datasets A3 and D31 are also bi-featured, containing 7, 500 and 3,100 objects, respectively (Krkkinen and Frnti 2002; Veenman et al. 2002). The 20newsgroup (20NG) and 50 class amazon review (50CR) are textual dataset (Chen and Liu 2014; Fei and Liu 2016). The 20NG dataset contains the news on 20 topics (Lang 1995). Each topic contains around one thousand news. The 50CR dataset contains amazon reviews for 50 electronic products. There are one thousand reviews for each electronic product. The initial clusters can be obtained using the most recent clustering algorithm HDBSCAN. Then, we apply EGO and EGOC B on these initial clustering results to evaluate its performance. Experiments for EGO and EGOCB We now evaluate the performance of EGO and EGOC B algorithms for cluster compactness in the presence of noise. Tables 2 and 3 show the detailed analysis of the proposed approaches based on internal validation measures, i.e., Table 2 Comparison of BD index, Dunn index and silhouette for EGO Dataset t4.8k t5.8k t7.10k t8.8k A3 D31 20NG 50CR 123 Davies Boudlin (DB) index, Dunn index and silhouette. DB index is a ratio between within-cluster scatteredness to the separation between clusters in a dataset. Dunn index compares within-cluster variance to the separation between the mean of different clusters. The silhouette compares the cohesion or consistency within-cluster to the clusters separation. Overall, the internal validation measures are used to reflect within-cluster consistency, connectedness and distance or separation between clusters. The unlabeled datasets are evaluated using DBSCAN and HDBSCAN for metric evaluation and then the algorithm (2) and (3) are applied on the top of HDBSCAN algorithm, which made a significant improvement in their compactness, connectedness and separation between different clusters by detecting and removing outliers. In the case of labeled datasets, the initial partitions are obtained using data labels. The proposed algorithms are then applied on the top of labeled based partitions, that improve the cluster or class compactness, connectivity among objects and separation between clusters while detecting outliers. The results for DBSCAN, HDBSCAN and EGO are listed in Table 2. It may be noted that initial partitions are obtained using HDBSCAN or data labels. The EGO algorithm adds further cohesion within-cluster and separation Approaches DB index Dunn index Silhouette DBSCAN 1.030 ± 0.004 0.567 ± 0.001 0.309 ± 0.001 HDBSCAN 1.020 ± 0.002 0.601 ± 0.001 0.316 ± 0.002 EGO 1.010 ± 0.002 0.667 ± 0.002 0.341 ± 0.017 DBSCAN 0.637 ± 0.012 0.663 ± 0.001 0.457 ± 0.012 HDBSCAN 0.578 ± 0.003 0.696 ± 0.004 0.573 ± 0.014 EGO 0.539 ± 0.002 0.741 ± 0.003 0.589 ± 0.013 DBSCAN 0.631 ± 0.003 0.587 ± 0.002 0.424 ± 0.011 HDBSCAN 0.619 ± 0.013 0.617 ± 0.012 0.452 ± 0.011 EGO 0.588 ± 0.012 0.676 ± 0.011 0.499 ± 0.017 DBSCAN 0.626 ± 0.010 0.587 ± 0.012 0.431 ± 0.011 HDBSCAN 0.603 ± 0.010 0.631 ± 0.002 0.471 ± 0.003 EGO 0.573 ± 0.015 0.689 ± 0.020 0.501 ± 0.023 DBSCAN 0.611 ± 0.004 0.589 ± 0.005 0.510 ± 0.004 HDBSCAN 0.593 ± 0.043 0.641 ± 0.027 0.527 ± 0.022 EGO 0.525 ± 0.022 0.703 ± 0.011 0.632 ± 0.003 DBSCAN 0.581 ± 0.030 0.614 ± 0.003 0.545 ± 0.034 Labeled 0.548 ± 0.006 0.638 ± 0.002 0.576 ± 0.003 EGO 0.410 ± 0.020 0.687 ± 0.003 0.671 ± 0.004 DBSCAN 0.898 ± 0.041 0.510 ± 0.022 0.318 ± 0.021 Labeled 0.861 ± 0.007 0.545 ± 0.003 0.339 ± 0.005 EGO 0.723 ± 0.005 0.593 ± 0.002 0.401 ± 0.020 DBSCAN 0.917 ± 0.013 0.503 ± 0.032 0.300 ± 0.001 Labeled 0.894 ± 0.003 0.539 ± 0.002 0.302 ± 0.003 EGO 0.782 ± 0.002 0.601 ± 0.001 0.396 ± 0.007 Environmental Science and Pollution Research Table 3 Comparison of BD index, Dunn index and silhouette for EGOC B Dataset t4.8k t5.8k t7.10k t8.8k A3 D31 20NG 50CR Approaches DB index Dunn index Silhouette 0.309 ± 0.001 DBSCAN 1.030 ± 0.004 0.567 ± 0.001 HDBSCAN 1.020 ± 0.002 0.601 ± 0.001 0.316 ± 0.002 EGOC B 0.998 ± 0.002 0.693 ± 0.002 0.355 ± 0.019 0.457 ± 0.012 DBSCAN 0.637 ± 0.012 0.663 ± 0.001 HDBSCAN 0.578 ± 0.003 0.696 ± 0.004 0.573 ± 0.014 EGOC B 0.522 ± 0.002 0.756 ± 0.031 0.600 ± 0.015 DBSCAN 0.631 ± 0.003 0.587 ± 0.002 0.424 ± 0.011 HDBSCAN 0.619 ± 0.013 0.617 ± 0.012 0.452 ± 0.011 EGOC B 0.540 ± 0.004 0.693 ± 0.012 0.510 ± 0.016 DBSCAN 0.626 ± 0.010 0.587 ± 0.012 0.431 ± 0.011 HDBSCAN 0.603 ± 0.010 0.631 ± 0.002 0.471 ± 0.003 EGOC B 0.569 ± 0.017 0.694 ± 0.003 0.504 ± 0.024 DBSCAN 0.611 ± 0.004 0.589 ± 0.005 0.510 ± 0.004 HDBSCAN 0.593 ± 0.043 0.641 ± 0.027 0.527 ± 0.022 EGOC B 0.500 ± 0.024 0.723 ± 0.017 0.641 ± 0.003 DBSCAN 0.581 ± 0.030 0.614 ± 0.003 0.545 ± 0.034 Labeled 0.548 ± 0.006 0.638 ± 0.002 0.576 ± 0.003 EGOC B 0.400 ± 0.021 0.691 ± 0.004 0.677 ± 0.004 DBSCAN 0.898 ± 0.041 0.510 ± 0.022 0.318 ± 0.021 Labeled 0.861 ± 0.007 0.545 ± 0.003 0.339 ± 0.005 EGOC B 0.710 ± 0.003 0.599 ± 0.003 0.405 ± 0.012 DBSCAN 0.917 ± 0.013 0.503 ± 0.032 0.300 ± 0.001 Labeled 0.894 ± 0.003 0.539 ± 0.002 0.302 ± 0.003 EGOC B 0.771 ± 0.002 0.617 ± 0.002 0.404 ± 0.001 between clusters. The loose instances are detached and similar instances became a separate cluster. In CHAMELEON datasets, the t5.8k got more affected where t4.8k has a small improvement after the application of EGO. EGO detected more outliers for A3 dataset, enhancing their compactness and improving the result at a maximum among the unlabeled datasets. The labeled datasets include D31 and two textual datasets. The performance of EGO for labeled datasets is comparatively higher among the considered datasets. The textual datasets are highly congested and therefore, require a careful observation for outlier detection. The proposed approaches detect the outlier in a unique way, avoiding the space loss and complexity reduction. Table 3 reports the results for EGOC B in terms of three evaluation metrics namely, DB index, Dunn index and silhouette coefficient. It may be noted that for all datasets the results are improved by eliminating more outliers. The compactness of initial clusters and outlier detection performance EGOC B is a bit high in comparison to EGO. The outliers are more precisely detected and clusters are further cleaned and more compactness is added to each cluster. The EGOC B performance for labeled datasets is more realized due to their highly congested nature. Visual demonstration of the proposed approach This section visually demonstrates the performance of the proposed approach EGOC B . In particular, the cluster representation is elaborated in the presence of noise using HDBSCAN and EGOC B . The visuals show the superiority of the performance of EGOC B over the highly reputable algorithm ‘HDBSCAN’. For a visual demonstration, we consider two unlabeled and a labeled dataset. In the case of unlabeled datasets t5.8k and t7.10k, the initial clusters are obtained using HDBSCAN while the primary partitions are obtained using data labels for labeled dataset D31. Figure 6 shows the visuals for unlabeled dataset t5.8k. The initial labels are obtained for a supervised setting. The clusters obtained after HDBSCAN are shown in Fig. 6(b)– (c). There are six clusters in this dataset (represented by different colors) with noise all around them represented by black points. We may observe that the compactness of these clusters is remarkably improved by EGOC B by eliminating more outliers as shown in Fig. 6(d). The last Fig. 6(e) shows the most related instances of all the six clusters. These instances are more likely to increase the cluster consistency. 123 Environmental Science and Pollution Research Fig. 6 Visual results of cluster-based EGO for t5.8k dataset Next, we get into another unlabeled dataset t7.10k as shown in Fig. 7(a). Figure 7(b) shows the HDBSCAN process to detect outliers for the sake of compact cluster formation. Figure 7(c) visualizes the processing of EGC B to detect outliers from scratch that shows its ability as a standalone method. There are nine clusters in this dataset (represented by different colors) with noise all around them represented by black points. We may note that the EGOC B may, however, detect more outliers and improve cluster quality as shown in Fig. 7(c). Figure 7(d) shows the most compacted nine clusters. These clusters are much cohesive with maximum distance from each other. Now we look into the performance of EGOC B algorithm for a labeled dataset D31 as shown in Fig. 8(a). This labeled dataset arranged 3,100 instances in thirty-one classes as shown in Fig. 8(b). In the case of labeled dataset, the partitions are obtained using data labels without applying any clustering algorithm. The EGOC B is then applied on the top of labeled separated dataset. The classes are con- 123 densed and more apart due to the application of EGOC B as shown in Fig. 8(c). The loosely attached instances especially at the boundaries are detached from the cluster. Figure 8(d) shows the instances that belong to each cluster or class. The compactness of these clusters can be observed visually and mathematically in Tables (2 and 3). Comparative analysis of the proposed approach(es) The comparative analysis of outlier detection algorithms is a tangled and chaotic task. This is because of some basic issues that include, the unavailability of suitable datasets with proper ground truth, the consideration of optimal parameters for outlier determination and the choice of outlier discrimination from the rest of the data (Campos et al 2016a; Xu et al. 2018). Despite all the discussed difficulties, we did compare the proposed method with the well-known approaches to realize its effectiveness under the same set of conditions. Environmental Science and Pollution Research Fig. 7 Visual results of cluster-based EGO for t7.10k Dataset Fig. 8 Visual results for cluster-based EGO for D31 Dataset 123 Environmental Science and Pollution Research It may be noted that the proposed approach has the advantage of additional outlier detection because of its application on top of improved hard clusters. Besides this, for a fair comparison, we, however, consider the self-contained nature of the proposed approach. The datasets used in this set of experiments are cardiotocography and Pima Indians Diabetes (Campos et al 2016a). These datasets are semantically meaningful, i.e., realizing the real world data including instances that are deviating from the normal pattern. For example, a dataset of players with a particular class of badminton players. The Cardiotocography dataset containing three classes of individually collected data from people in search of heart disease. These classes contain normal, suspected and pathological patients with heart disease. The normal people are inliers where the other two classes of suspected and pathological patients are considered as outliers. The Pima Indian Diabetes dataset has two classes containing normal and patients with diabetes. The normal people are inliers where the diabetic patients are considered as outliers. To realize the rare nature of outliers in the considered datasets, we randomly downsampled the classes containing outliers at 5%, 10%, 15% and 20% Campos et al (2016a). The experiments are repeated ten times and taking the mean of all results. The following measures evaluate the performance of the proposed approach(es) (Ali et al. 2021; Campos et al 2016a; Xu et al. 2018), Correctly detected outliers and inliers , (8) Total objects Correctly detected outliers , (9) Pr ecision = Total identified outliers Correctly detected outliers Recall = , (10) Total actual outliers 2 × Pr ecision × Recall , (11) F1 = Pr ecision + Recall Accuracy = The above equation means that how accurately the EGO and EGOC B classify a given instance as outlier or inlier. In particular, the precision shows the proportion of predicted outliers that are actually outliers, i.e., how many of the predicted outliers (true outliers plus false outliers) are actually outliers (true outliers). The recall shows the proportion of correctly identified outliers out of actual outliers. The accuracy shows the proportion of correctly classified instances, both outliers and inliers. The F1 score summarizes the performance into a single metric and therefore, computed as a harmonic mean of precision and recall. The detailed results are reported in Tables 4 and 5. The performance of the proposed approach(es) is compared with six well-known outlier detection approaches including RE3WC (reduction- and elevation-based threeway clustering (Ali et al. 2021)) LOF (local outlier factor (Breunig et al. 2000)), LoOP (local outlier probabilities 123 (Kriegel et al. 2009)), ABOD (angle-based outlier detection (Kriegel et al. 2008)), CBLOF (cluster-based local outlier factor (He et al. 2003)) and HBOS (histogram-based outlier score (Goldstein and Dengel 2012)). These approaches based on criterion-based ranking of each object to detect outliers. Table 4 shows the results for 5% and 10% downsampled outlier class in cardiotocography and Indian Pima datasets. The results for both EGO and EGOC B are listed on the top of each comparison. The listed values for different metrics are the mean of ten times repeated experiments for each approach. In the case of Indian Pima Diabetes dataset, the accuracy of EGO for 5% and 10% is the same as RE3WC. The proportion of classification of actual outliers is a bit low for 10% downsampled outliers as compared to RE3WC. The EGOC B work on cluster basis and therefore, has a maximum accuracy for both 5% and 10% downsampled outliers. The precision values for LoOP are high for 5% and very close for 10% downsampled outliers on the Indian Pima Diabetes dataset. In the case of cardiotocography dataset, the proposed approaches perform better in comparison. The recall values of ABOD for 5%, 10% and CBLOF for 10% downsampled outliers are high as compared to the proposed approaches. The F1 score of the proposed approaches is always better in comparison to the participated outlier detection algorithms for 5% and 10% downsampled outliers. Table 5 shows the results for 15% and 20% downsampled outlier class in cardiotocography and Indian Pima datasets. The computed metric values for EGO and EGOC B are listed on the top of each comparison. The tabulated values are the computed mean of ten times repeated experiments for each approach. In the case of Indian Pima Diabetes dataset, the proposed approaches perform better except for a beat in precision. The precision value of LOF for 15% is a bit high as compared to EGO, however, RE3WC, LOF and ABOD beat both EGO and EGOC B for 20% downsampled outliers. In the case of cardiotocography dataset, the EGOC B outperforms all the compared approaches while EGO has a beat in precision. The F1 score result shows that the proposed approaches consistently produce better results as compared to the participated approaches and therefore can be considered as a better solution towards outlier detection. Experiments on environmental monitoring data This section shows the experiments related to the detecting capability of the proposed approach in environmental monitoring data. These experiments are conducted for both real and synthetically prepared datasets. First, the attributes in the dataset have been analyzed based on their correlation. The correlation is computed using the Pearson Correlation Coefficient (PCC) (Benesty et al. 2009). In order to compute the dependencies of various variables, a correlation matrix is Environmental Science and Pollution Research Table 4 Results on Pima Indian Diabetes and Cardiotocography datasets with 5% and 10% downsampled outliers 5% Outliers Approach EGOC B Pima Cardiotography Accuracy Recall Precision F1-Score Accuracy Recall Precision F1-Score 0.91 ± 0.02 0.53 ± 0.03 0.55 ± 0.01 0.54 ± 0.02 0.94 ± 0.04 0.57 ± 0.03 0.58 ± 0.03 0.57 ± 0.02 EGO 0.87 ± 0.01 0.47 ± 0.02 0.51 ± 0.02 0.49 ± 0.02 0.91 ± 0.03 0.51 ± 0.03 0.49 ± 0.02 0.50 ± 0.02 RE3WC 0.87 ± 0.01 0.43 ± 0.04 0.50 ± 0.03 0.46 ± 0.04 0.88 ± 0.01 0.51 ± 0.01 0.29 ± 0.01 0.37 ± 0.13 LOF 0.86 ± 0.02 0.08 ± 0.05 0.31 ± 0.08 0.13 ± 0.07 0.88 ± 0.01 0.51 ± 0.01 0.29 ± 0.01 0.37 ± 0.13 LoOP 0.88 ± 0.09 0.10 ± 0.03 0.70 ± 0.31 0.17 ± 0.09 0.93 ± 0.03 0.22 ± 0.02 0.53 ± 0.04 0.31 ± 0.02 ABOD 0.85 ± 0.02 0.29 ± 0.08 0.37 ± 0.05 0.32 ± 0.04 0.89 ± 0.03 0.62 ± 0.02 0.34 ± 0.02 0.44 ± 0.02 CBLOF 0.75 ± 0.07 0.13 ± 0.04 0.11 ± 0.04 0.12 ± 0.04 0.86 ± 0.02 0.42 ± 0.02 0.21 ± 0.01 0.28 ± 0.07 HBOS 0.71 ± 0.08 0.14 ± 0.09 0.09 ± 0.03 0.11 ± 0.07 0.85 ± 0.03 0.22 ± 0.07 0.14 ± 0.02 0.17 ± 0.02 Accuracy Recall Precision F1-Score Accuracy Recall Precision F1-Score EGOC B 0.87 ± 0.07 0.54 ± 0.06 0.73 ± 0.07 0.62 ± 0.08 0.91 ± 0.06 0.51 ± 0.05 0.66 ± 0.06 0.58 ± 0.05 EGO 0.84 ± 0.08 0.44 ± 0.05 0.69 ± 0.06 0.54 ± 0.05 0.89 ± 0.06 0.51 ± 0.05 0.55 ± 0.07 0.53 ± 0.06 RE3WC 0.84 ± 0.09 0.45 ± 0.09 0.69 ± 0.03 0.54 ± 0.02 0.89 ± 0.02 0.48 ± 0.01 0.55 ± 0.01 0.51 ± 0.01 10% Outliers Approach Pima Cardiotography LOF 0.80 ± 0.08 0.16 ± 0.02 0.68 ± 0.11 0.26 ± 0.04 0.88 ± 0.03 0.24 ± 0.03 0.59 ± 0.03 0.34 ± 0.03 LoOP 0.81 ± 0.05 0.19 ± 0.02 0.71 ± 0.07 0.30 ± 0.09 0.88 ± 0.04 0.25 ± 0.03 0.56 ± 0.04 0.34 ± 0.02 ABOD 0.80 ± 0.09 0.33 ± 0.02 0.53 ± 0.03 0.41 ± 0.07 0.87 ± 0.02 0.65 ± 0.08 0.50 ± 0.03 0.56 ± 0.07 CBLOF 0.73 ± 0.08 0.15 ± 0.02 0.26 ± 0.04 0.19 ± 0.03 0.86 ± 0.03 0.58 ± 0.03 0.47 ± 0.02 0.52 ± 0.09 HBOS 0.74 ± 0.09 0.26 ± 0.05 0.35 ± 0.03 0.30 ± 0.02 0.84 ± 0.03 0.26 ± 0.08 0.37 ± 0.02 0.31 ± 0.02 F1-Score Table 5 Results on Pima Indian Diabetes and Cardiotocography datasets with 15% and 20% downsampled outliers 15% Outliers Approach Pima Cardiotography Accuracy Recall Precision F1-Score Accuracy Recall Precision EGOC B 0.74 ± 0.07 0.37 ± 0.03 0.59 ± 0.05 0.45 ± 0.05 0.89 ± 0.07 0.61 ± 0.04 0.76 ± 0.03 0.68 ± 0.05 EGO 0.72 ± 0.08 0.31 ± 0.04 0.55 ± 0.03 0.40 ± 0.04 0.85 ± 0.05 0.50 ± 0.05 0.66 ± 0.04 0.57 ± 0.06 RE3WC 0.71 ± 0.08 0.29 ± 0.04 0.50 ± 0.02 0.36 ± 0.04 0.84 ± 0.08 0.45 ± 0.04 0.57 ± 0.02 0.51 ± 0.07 LOF 0.72 ± 0.08 0.14 ± 0.05 0.58 ± 0.04 0.23 ± 0.02 0.85 ± 0.04 0.25 ± 0.01 0.73 ± 0.12 0.37 ± 0.08 LoOP 0.70 ± 0.07 0.19 ± 0.02 0.48 ± 0.09 0.27 ± 0.07 0.84 ± 0.02 0.27 ± 0.01 0.61 ± 0.08 0.37 ± 0.09 ABOD 0.68 ± 0.07 0.32 ± 0.04 0.45 ± 0.04 0.37 ± 0.05 0.85 ± 0.09 0.59 ± 0.06 0.57 ± 0.05 0.58 ± 0.05 CBLOF 0.64 ± 0.09 0.25 ± 0.06 0.34 ± 0.05 0.28 ± 0.01 0.82 ± 0.05 0.42 ± 0.06 0.51 ± 0.07 0.47 ± 0.09 HBOS 0.70 ± 0.06 0.26 ± 0.08 0.46 ± 0.04 0.33 ± 0.03 0.79 ± 0.03 0.26 ± 0.04 0.38 ± 0.07 0.31 ± 0.07 Recall Precision F1-Score Accuracy Recall Precision F1-Score 20% Outliers Approach Pima Accuracy Cardiotography EGOC B 0.72 ± 0.04 0.31 ± 0.08 0.40 ± 0.04 0.35 ± 0.05 0.88 ± 0.05 0.58 ± 0.03 0.83 ± 0.04 0.68 ± 0.05 EGO 0.69 ± 0.05 0.29 ± 0.04 0.38 ± 0.05 0.33 ± 0.04 0.84 ± 0.07 0.52 ± 0.03 0.72 ± 0.04 0.60 ± 0.03 RE3WC 0.69 ± 0.04 0.26 ± 0.09 0.64 ± 0.05 0.37 ± 0.06 0.84 ± 0.09 0.43 ± 0.06 0.75 ± 0.08 0.55 ± 0.05 LOF 0.65 ± 0.09 0.06 ± 0.01 0.53 ± 0.08 0.11 ± 0.09 0.79 ± 0.03 0.10 ± 0.08 0.82 ± 0.12 0.17 ± 0.07 LoOP 0.67 ± 0.09 0.23 ± 0.04 0.58 ± 0.05 0.33 ± 0.06 0.81 ± 0.03 0.26 ± 0.07 0.73 ± 0.12 0.39 ± 0.07 ABOD 0.68 ± 0.04 0.24 ± 0.01 0.63 ± 0.05 0.35 ± 0.09 0.82 ± 0.03 0.52 ± 0.05 0.63 ± 0.06 0.57 ± 0.03 CBLOF 0.61 ± 0.04 0.17 ± 0.05 0.34 ± 0.08 0.23 ± 0.01 0.79 ± 0.09 0.38 ± 0.05 0.56 ± 0.03 0.45 ± 0.07 HBOS 0.61 ± 0.07 0.17 ± 0.05 0.34 ± 0.02 0.23 ± 0.04 0.71 ± 0.03 0.10 ± 0.06 0.22 ± 0.07 0.14 ± 0.05 123 Environmental Science and Pollution Research Table 6 Results on environmental monitoring data for a sample size 2000 with 10% contamination Table 8 Results on environmental monitoring data for a sample size 2000 with 20% contamination Contamination Number of Outlier Number of detected Number of False Detected Contamination Number of Outlier Number of detected Number of False Detected Carbon 20 18 5 Carbon 50 49 5 Carbon monoxide 24 23 6 Carbon monoxide 40 38 4 Carbon dioxide 22 21 7 Carbon dioxide 40 38 4 Lead 25 23 6 Lead 40 39 6 Nitrogen monoxide 24 22 7 Nitrogen monoxide 40 37 3 Nitrogen dioxide 22 21 8 Nitrogen dioxide 40 39 6 Ground-level ozone 21 19 4 Ground-level ozone 40 40 8 Sulfur monoxide 22 20 3 Sulfur monoxide 55 53 5 Sulfur dioxide 20 18 3 Sulfur dioxide 55 54 4 computed based on the correlation coefficient. The correlation matrix defines the linear relationship between each pair of variables. Mathematically, it is defined as follows: P(a, a) P(a, b) P(b, a) P(b, b) Cov(a, b) ρ= σa σb Matri x = (12) (13) The above Eqs. 12–13 show the correlation between two attributes, a and b. Here a denotes the standard deviation for a and b where Cov(a, b) represents the covariance. The correlation between various components determines the balanced components in the fresh air. Secondly, it is also observed that some contamination is sometimes allowed until its severity. Therefore, this rule has been strictly followed while computing correlations between various air components. The contamination has been added synthetically to observe the outline detection performance of the proposed approach. Next, the proposed approach has been employed Table 7 Results on environmental monitoring data for a sample size 2000 with 15% contamination Contamination Number of Outlier Number of detected Number of False Detected Carbon 30 28 5 Carbon monoxide 30 28 5 Carbon dioxide 40 39 6 Lead 36 35 6 Nitrogen monoxide 25 23 5 Nitrogen dioxide 35 33 4 Ground-level ozone 35 32 2 Sulfur monoxide 36 33 1 Sulfur dioxide 33 31 2 123 on this contaminated dataset. The results are tabulated in Tables 6, 7, and 8. Table 6 represents the results for EGO on environmental monitoring data using a sample size of 2000 with 10% contamination. The results for carbon, carbon monoxide, carbon dioxide, lead, nitrogen monoxide, nitrogen dioxide, ground-level ozone, sulfur monoxide and sulfur dioxide are 90.00%, 95.83%, 95.45%, 92.00%, 91.67%, 95.45%, 90.48%, 90.91% and 90.00%, respectively. The overall performance for 10% added contamination is 92.42%. There is a relation between the observed outliers and false detections. The high outlier detection rate increases the error rate. It means that these outliers are very much sparsed and connected boundaries with non-outlier components of the air. Table 7 shows the results of the proposed approach employed at environmental monitoring data using a sample size of 2000 with 15% contamination. The results for the contaminations: carbon, carbon monoxide, carbon dioxide, lead, nitrogen monoxide, nitrogen dioxide, ground-level ozone, sulfur monoxide and sulfur dioxide are 93.33%, 93.33%, 97.50%, 97.22%, 92.00%, 94.29%, 91.43%, 91.67%, and 93.94%, respectively. The overall performance for 15% added contamination is 93.86%. The results became improved with the addition of more contaminations. The same relationship can be observed between detection and error rate. The number of true positives increases; meanwhile, false positives also increase. It means that these outliers are far from each other but near with overlapping boundaries of non-outlier components in the air. Table 8 tabulates the performance of the proposed approach for environmental monitoring data using a sample size of 2000 with 20% contamination. The performance of the proposed approach for the contaminations: carbon, carbon monoxide, carbon dioxide, lead, nitrogen monoxide, nitrogen dioxide, ground-level ozone, sulfur monoxide and sulfur dioxide are 98.00%, 95.00%, 95.00%, 97.50%, 92.50%, 97.50%, 100.00%, 96.36%, and 98.18%, respectively. The Environmental Science and Pollution Research overall performance for 15% added contamination is 96.67%. The results keep improving with the addition of more contamination. The observations are the same, i.e., there exists a relationship between detection and error rate. An increase in the detection rate increases the error rate. An increase in the number of true positives may increase false positives. That statement suggests that the outliers with high detection are close to each other, with overlapping boundaries to the non-contaminated components. Ali. Supervision: Anwar Shah, Bahar Ali, and Muhammad Shafiq. Project administration: Anwar Shah, Inam Ullah, Kassian T.T. Amesho, and Muhammad Shafiq. Funding acquisition: Kassian T.T. Amesho, Muhammad Shafiq, Shahid Anwar, and Ahyoung Choi. Funding This work is supported by the National Natural Science Foundation of China, Grant No. 62250410365. Availability of data and materials The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation. Declarations Conclusion Entropy in machine learning is used in many applications. The concept of randomness can be applied where uncertainty exists. We introduce EGO, which combines the two concepts of grid and entropy. The grid effectively distributes the data and facilitates easy analysis. Entropy plays a vital role in measuring randomness. The entropy in this study is considered as a distance function, the value of which became dropped due to the separation of a member from its respective cluster. The EGO is applied on the top of hard clusters obtained using a hard clustering algorithm or data labels. The explicit outliers are removed initially. The entropy is applied to the whole dataset to know the initial randomness. Then the remaining sparse instances are examined for the implicit outliers. The method is flexible and can be applied to clusterbased approach. This has a more time complexity at the cost of good and improved results. The approach is clusterbased EGO or EGOC B . The proposed approach(es) are comparatively analyzed against well-known outlier detection algorithms, including DBSCAN, HDBSCAN, RE3WC, LOF, LoOP, ABOD, CBLOF and HBOS. Experimental results indicate that the EGO and EGOC B effectively detect outliers while producing compact clusters. The results of other algorithms used to cluster in noise-polluted datasets can be further refined by EGO and EGOC B . In particular, our approach(es) detect an additional 4.5% to 8.6% outliers that were undetected by well-known approaches. Moreover, outlier detection is carried out in the environmental data analysis as a case study of the proposed approach. The experiments suggest that the proposed approach(es) may be considered a suitable choice for obtaining compact clusters in the presence of noise, whereas the robustness of environmental data analysis encourages its industry orientation. Author contribution Conceptualization: Anwar Shah and Bahar Ali. Methodology: Anwar Shah, Bahar Ali, and Kassian T.T. Amesho. Software: Anwar Shah, Bahar Ali, Fazal Wahab, and Inam Ullah. Validation: Anwar Shah, Bahar Ali, and Muhammad Shafiq. Formal analysis: Anwar Shah. Investigation: Anwar Shah, Bahar Ali, Inam Ullah, and Muhammad Shafiq. Resources: Anwar Shah and Inam Ullah. Data curation: Anwar Shah, Fazal Wahab, and Inam Ullah. Writing— original draft preparation: Anwar Shah. Writing—review and editing: Anwar Shah and Bahar Ali. Visualization: Anwar Shah and Bahar Ethics statement The studies involving human participants were reviewed and approved by the National Natural Science Foundation of China, Grant No. 62250410365. The ethics committee waived the requirement of written informed consent for participation. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article. Consent to participate Informed consent was obtained from all individual participants included in the study. Consent for publication The authors of the article “Entropy Based Grid Approach for Handling Outliers: A Case Study to Environmental Monitoring Data” give consent for the publication of all the identifiable details, including text, images, and materials, to be published in the journal Environmental Science and Pollution Research. Conflict of interest The authors declare no competing interests. References Agrawal R, Gehrke J, Gunopulos D, et al (1998) Automatic subspace clustering of high dimensional data for data mining applications. In: Proceedings of the international conference on Management of data. pp 94–105 Alameddine I, Kenney MA, Gosnell RJ et al (2010) Robust multivariate outlier detection methods for environmental data. J Environ Eng 136(11):1299–1304 Ali B, Azam N, Shah A et al (2021) A spatial filtering inspired threeway clustering approach with application to outlier detection. Int J Approx Reason 130:1–21 Amini A, Wah TY, Saboohi H (2014) On density-based data streams clustering algorithms: A survey. J Comput Sci Technol 29(1):116– 141 Andersson JL, Graham MS, Zsoldos E et al (2016) Incorporating outlier detection and replacement into a non-parametric framework for movement and distortion correction of diffusion mr images. NeuroImage 141:556–572 Bai M, Wang X, Xin J et al (2016) An efficient algorithm for distributed density-based outlier detection on big data. Neurocomputing 181:19–28 Batra R, Ko KI (1992) An adaptive mesh refinement technique for the analysis of shear bands in plane strain compression of a thermoviscoplastic solid. Comput Mech 10(6):369–379 Benesty J, Chen J, Huang Y, et al (2009) Pearson correlation coefficient. In: Noise reduction in speech processing. Springer, p 1–4 Berger MJ, Oliger J (1984) Adaptive mesh refinement for hyperbolic partial differential equations. J Comput Phys 53(3):484–512 123 Environmental Science and Pollution Research Berger MJ, Colella P et al (1989) Local adaptive mesh refinement for shock hydrodynamics. J Comput Phys 82(1):64–84 Bharti S, Pattanaik K, Pandey A (2019) Contextual outlier detection for wireless sensor networks. J Ambient Intell Humanized Comput 1–20 Birant D, Kut A (2007) St-dbscan: An algorithm for clustering spatialtemporal data. Data Knowl Eng 60(1):208–221 Blythe J, Jain S, Deelman E et al (2005) Task scheduling strategies for workflow-based applications in grids. In: IEEE International Symposium on Cluster Computing and the Grid, vol 2005. pp 759– 767 Borah B, Bhattacharyya D (2004) An improved sampling-based dbscan for large spatial databases. In: Proceedings of the International conference on intelligent sensing and information processing. pp 92–96 Breunig MM, Kriegel HP, Ng RT, et al (2000) Lof: identifying densitybased local outliers. In: Proceedings of the international conference on Management of data. pp 93–104 Campello RJ, Moulavi D, Sander J (2013) Density-based clustering based on hierarchical density estimates. In: Proceedings of the Pacific-Asia conference on knowledge discovery and data mining. pp 160–172 Campos GO, Zimek A, Sander J et al (2016) On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min Knowl Discov 30(4):891–927 Campos GO, Zimek A, Sander J et al (2016) On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min Knowl Discov 30(4):891–927 Chen J, Sathe S, Aggarwal C, et al (2017) Outlier detection with autoencoder ensembles. In: Proceedings of the international conference on data mining. pp 90–98 Chen Z, Liu B (2014) Mining topics in documents: standing on the shoulders of big data. In: Proceedings of the international conference on Knowledge discovery and data mining. pp 1116–1125 Christy A, Gandhi GM, Vaithyasubramanian S (2015) Cluster based outlier detection algorithm for healthcare data. Procedia Comput Sci 50:209–215 Duan L, Xu L, Guo F et al (2007) A local-density based spatial clustering algorithm with noise. Inf Syst 32(7):978–986 Eiseman PR (1987) Adaptive grid generation. Comput Methods Appl Mech Eng 64(1–3):321–376 Erskine RH, Green TR, Ramirez JA, et al (2006) Comparison of gridbased algorithms for computing upslope contributing area. Water Resour Res 42(9) Ester M, Kriegel HP, Sander J, et al (1996) A density-based algorithm for discovering clusters in large spatial databases with noise. In: Knowledge Discovery and Data Mining. pp 226–231 Fakhari A, Lee T (2014) Finite-difference lattice boltzmann method with a block-structured adaptive-mesh-refinement technique. Phys Rev E 89(3):033310 Fei G, Liu B (2016) Breaking the closed world assumption in text classification. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp 506–514 Fuchs L (1986) A local mesh-refinement technique for incompressible flows. Comput Fluids 14(1):69–81 Gan G, Ng MKP (2017) K-means clustering with outlier removal. Pattern Recog Lett 90:8–14 Garces H, Sbarbaro D (2009) Outliers detection in environmental monitoring data. IFAC Proc 42(23):330–335 Goldstein M, Dengel A (2012) Histogram-based outlier score (hbos): A fast unsupervised anomaly detection algorithm. Poster Demo Track 59–63 Goldstein MB (2014) Anomaly detection in large datasets. Verlag Dr. Hut 123 Gu Y, Ganesan RK, Bischke B, et al (2017) Grid-based outlier detection in large data sets for combine harvesters. In: Proceedings of the International Conference on Industrial Informatics. pp 811–818 Güngör E, Özmen A (2017) Distance and density based clustering algorithm using gaussian kernel. Expert Syst Appl 69:10–20 Guseva AI, Kuznetsov IA (2017) The use of entropy measure for higher quality machine learning algorithms in text data processing. In: Proceedings of the International Conference on Future Internet of Things and Cloud Workshops. pp 47–52 Hautamäki V, Cherednichenko S, Kärkkäinen I, et al (2005) Improving k-means by outlier removal. In: Scandinavian Conference on Image Analysis. Springer, pp 978–987 He Y, Tan H, Luo W et al (2014) Mr-dbscan: a scalable mapreducebased dbscan algorithm for heavily skewed data. Front Comput Sci 8(1):83–99 He Z, Xu X, Deng S (2003) Discovering cluster-based local outliers. Pattern Recogn Lett 24(9–10):1641–1650 Jabez J, Muthukumar B (2015) Intrusion detection system (ids): anomaly detection using outlier detection approach. Procedia Comput Sci 48:338–346 Jiang MF, Tseng SS, Su CM (2001) Two-phase clustering process for outliers detection. Pattern Pattern Recognit 22(6–7):691–700 Kadlec P, Gabrys B, Strandt S (2009) Data-driven soft sensors in the process industry. Comput Chem Eng 33(4):795–814 Karypis G, Han EH, Kumar V (1999) Chameleon: Hierarchical clustering using dynamic modeling. Computer 32(8):68–75 Kotsiantis S, Pintelas P (2004) Recent advances in clustering: A brief survey. Trans Inf Sci Appl 1(1):73–81 Kriegel HP, Schubert M, Zimek A (2008) Angle-based outlier detection in high-dimensional data. In: Proceedings of the international conference on Knowledge discovery and data mining. pp 444– 452 Kriegel HP, Kröger P, Schubert E, et al (2009) Loop: local outlier probabilities. In: Proceedings of the conference on Information and knowledge management. pp 1649–1652 Krkkinen I, Frnti P (2002) Dynamic local search algorithm for the clustering problem. Department of Computer Science, University of Joensuu, Tech Rep A-2002-6 Lang K (1995) Newsweeder: Learning to filter netnews. In: Machine Learning Proceedings 1995. Elsevier, p 331–339 Lee J, Cho NW (2016) Fast outlier detection using a grid-based algorithm. PLoS ONE 11(11):e0165972 Liao Wk, Liu Y, Choudhary A (2004) A grid-based clustering algorithm using adaptive mesh refinement. In: Proceedings of the international conference on data mining. pp 61–69 Lin S, Brown DE (2006) An outlier-based data association method for linking criminal incidents. Decis Support Syst 41(3):604–615 Liu B, Yin J, Xiao Y, et al (2010) Exploiting local data uncertainty to boost global outlier detection. In: Proceedings of the International Conference on Data Mining, pp 304–313 Louhichi S, Gzara M, Abdallah HB (2014) A density based algorithm for discovering clusters with varied density. In: Proceedings of World Congress on Computer Applications and Information Systems). pp 1–6 Lucas Y, Portier PE, Laporte L et al (2020) Towards automated feature engineering for credit card fraud detection using multi-perspective hmms. Futur Gener Comput Syst 102:393–402 Luo J, Xu L, Jamont JP et al (2007) Flood decision support system on agent grid: method and implementation. Enterp Inf Syst 1(1):49– 68 Ma EW, Chow TW (2004) A new shifting grid clustering algorithm. Pattern Recogn 37(3):503–514 Mahmoud E, Elmogy AM, Sarhan A (2016) Enhancing grid local outlier factor algorithm for better outlier detection. Artif Intell Mach Learn J 16(1):13–21 Environmental Science and Pollution Research Malini N, Pushpa M (2017) Analysis on credit card fraud identification techniques based on knn and outlier detection. In: Proceedings of the third International Conference on Advances in Electrical, Electronics, Information, Communication and Bio-Informatics. pp 255–258 McInnes L, Healy J, Astels S (2017) hdbscan: Hierarchical density based clustering. J Open Source Softw 2(11):205 Mia Hubert PR, Segaert P (2015) Discussion of multivariate functional outlier detection. Stat Methods Appl 24(2):177–202 Ohadi N, Kamandi A, Shabankhah M, et al (2020) Sw-dbscan: A gridbased dbscan algorithm for large datasets. In: Proceddings of the International Conference on Web Research (ICWR). pp 139–145 Osekowska E, Johnson H, Carlsson B (2014) Grid size optimization for potential field based maritime anomaly detection. Transp Res Procedia 3:720–729 Park NH, Lee WS (2004) Statistical grid-based clustering over data streams. ACM Sigmod Rec 33(1):32–37 Pearson RK (2002) Outliers in process modeling and identification. IEEE Trans Control Syst Technol 10(1):55–63 Pilevar AH, Sukumar M (2005) Gchl: A grid-clustering algorithm for high-dimensional very large spatial data bases. Pattern Recogn Lett 26(7):999–1010 Qiu GF, Li HZ, Xu LD et al (2003) A knowledge processing method for intelligent systems based on inclusion degree. Expert Syst 20(4):187–195 Rai P, Singh S (2010) A survey of clustering techniques. Int J Comput Appl 7(12):1–5 Rajeswari A, Yalini S, Janani R, et al (2018) A comparative evaluation of supervised and unsupervised methods for detecting outliers. In: Proceedings of the Second International Conference on Inventive Communication and Computational Technologies. pp 1068–1073 Rehm F, Klawonn F, Kruse R (2007) A novel approach to noise clustering for outlier detection. Soft Comput 11(5):489–494 Rencis JJ, Mullen RL (1986) Solution of elasticity problems by a self-adaptive mesh refinement technique for boundary element computation. Int J Numer Methods Eng 23(8):1509–1527 Rokach L (2009) A survey of clustering algorithms. In: Data mining and knowledge discovery handbook. p 269–298 Sandosh S, Govindasamy V, Akila G (2020) Enhanced intrusion detection system via agent clustering and classification based on outlier detection. Peer-to-Peer Netw Appl 1–8 Shafiq M, Tian Z, Bashir AK et al (2020) Corrauc: a malicious botiot traffic detection method in iot network using machine-learning techniques. IEEE Internet Things J 8(5):3242–3254 Shafiq M, Tian Z, Bashir AK et al (2020) Iot malicious traffic identification using wrapper-based feature selection mechanisms. Comput Secur 94:101863 Shafiq M, Tian Z, Sun Y et al (2020) Selection of effective machine learning algorithm and bot-iot attacks traffic identification for internet of things in smart city. Futur Gener Comput Syst 107:433– 442 Shah A, Azam N, Ali B et al (2021) A three-way clustering approach for novelty detection. Inf Sci 569:650–668 Shah A, Azam N, Alanazi E, et al (2022) Image blurring and sharpening inspired three-way clustering approach. Appl Intell 1–25 Sheikholeslami S, Chatterjee S, Zhang A (2002) A multi-resolution clustering approach for very large spatial databases. In: Proceedings of the International Conference on Formal Ontology in Information Systems. pp 622–630 Sitanggang IS, Baehaki DAM (2015) Global and collective outliers detection on hotspot data as forest fires indicator in riau province, indonesia. In: Proceedings of the International Conference on Spatial Data Mining and Geographical Knowledge Services. pp 66–70 Tran TN, Drab K, Daszykowski M (2013) Revised dbscan algorithm to cluster data with dense adjacent clusters. Chemometr Intell Lab Syst 120:92–96 Veenman CJ, Reinders MJT, Backer E (2002) A maximum variance cluster algorithm. IEEE Trans Pattern Anal Mach Intell 24(9):1273–1280 Veselík P, Sejkorová M, Nieoczym A, et al (2020) Outlier identification of concentrations of pollutants in environmental data using modern statistical methods. Pol J Environ Stud 29(1) Wang B, Xiao G, Yu H, et al (2009) Distance-based outlier detection on uncertain data. In: Proceddings of the International Conference on Computer and Information Technology. pp 293–298 Wang W, Yang J, Muntz R, et al (1997) Sting: A statistical information grid approach to spatial data mining. In: Proceeding of the conference very large data bases. pp 186–195 Wang X, Davidson I (2009) Discovering contexts and contextual outliers using random walks in graphs. In: Proceedings of the International Conference on Data Mining. pp 1034–1039 Warne K, Prasad G, Rezvani S et al (2004) Statistical and computational intelligence techniques for inferential model development: a comparative evaluation and a novel proposition for fusion. Eng Appl Artif Intell 17(8):871–885 Xu D, Tian Y (2015) A comprehensive survey of clustering algorithms. Ann Data Sci 2(2):165–193 Xu X, Yuruk N, Feng Z, et al (2007) Scan: a structural clustering algorithm for networks. In: Proceedings of the international conference on Knowledge discovery and data mining. pp 824–833 Xu X, Liu H, Li L et al (2018) A comparison of outlier detection techniques for high-dimensional data. Int J Comput Intell Syst 11(1):652–662 Yang H, Antonante P, Tzoumas V et al (2020) Graduated non-convexity for robust spatial perception: From non-minimal solvers to global outlier rejection. IEEE Robot Autom Lett 5(2):1127–1134 Yang X, Zhang G, Lu J et al (2010) A kernel fuzzy c-means clustering-based fuzzy support vector machine algorithm for classification problems with outliers or noises. IEEE Trans Fuzzy Syst 19(1):105–115 Yap P (2002) Grid-based path-finding. In: Conference of the Canadian Society for Computational Studies of Intelligence. pp 44–55 Zhang JS, Leung YW (2003) Robust clustering by pruning outliers. IEEE Trans Syst Man Cybern 33(6):983–998 Zhu Y, Ting KM, Carman MJ (2016) Density-ratio based clustering for discovering clusters with varying densities. Patt Recogn 60:983– 997 Zhu Y, Ting KM, Angelova M (2018) A distance scaling method to improve density-based clustering. In: Proceedings of the PacificAsia Conference on Knowledge Discovery and Data Mining. pp 389–400 Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law. 123 Environmental Science and Pollution Research Authors and Affiliations Anwar Shah1 · Bahar Ali1 · Fazal Wahab2 · Inam Ullah3 · Kassian T.T. Amesho4,5,6,7,8,9 · Muhammad Shafiq10,11 Anwar Shah anwar.shah@nu.edu.pk 4 Institute of Environmental Engineering, National Sun Yat-Sen University, Kaohsiung 804, Taiwan Bahar Ali bahar.ali@nu.edu.pk 5 Center for Emerging Contaminants Research, National Sun Yat-Sen University, Kaohsiung 804, Taiwan Fazal Wahab 1728039@stu.edu.cn 6 Tshwane School for Business and Society, Faculty of Management of Sciences, Tshwane University of Technology, Pretoria, South Africa 7 Centre for Environmental Studies, The International University of Management, Main Campus, Dorado Park Ext 1, Windhoek, Namibia Inam Ullah inam.fragrance@gmail.com Kassian T.T. Amesho kassian.amesho@gmail.com 1 2 3 8 Regent Business School, Durban 4001, South Africa National University of Computer and Emerging Sciences, Karachi, Pakistan 9 Destinies Biomass Energy and Farming Pty Ltd, P.O. Box 7387, Swakopmund, Namibia College of Computer Science and Technology, Northeastern University Shenyang, Shenyang, China 10 Cyberspace Institute of Advanced Technology, Guangzhou University, Guangzhou, China 11 Shenyang Normal University, Shenyang, China BK21 Chungbuk Information Technology Education and Research Center, Chungbuk National University, Cheongju, South Korea 123