composits
library(composits)
library(ggplot2)
library(forecast)
library(dplyr)
#>
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#>
#> filter, lag
#> The following objects are masked from 'package:base':
#>
#> intersect, setdiff, setequal, union
library(tidyr)
library(stringr)
library(broom)
library(maptools)
#> Loading required package: sp
#> Checking rgeos availability: FALSE
#> Please note that 'maptools' will be retired by the end of 2023,
#> plan transition at your earliest convenience;
#> some functionality will be moved to 'sp'.
#> Note: when rgeos is not available, polygon geometry computations in maptools depend on gpclib,
#> which has a restricted licence. It is disabled by default;
#> to enable gpclib, type gpclibPermit()
The goal of composits is to find outliers in compositional,
multivariate and univariate time series. It is an outlier ensemble
method that uses the outlier detection methods from R packages
forecast, tsoutliers, anomalize
and otsad. All the options provided in those packages can
be included in the calls for each of the methods used in the ensemble
function so that the user can create a more customize ensemble.
As describe in the paper four dimension reduction methods are used in
the multivariate ensemble PCA, DOBIN, ICS and ICA. It is recomended that
users verified the scaling and centering options of these methods for
each particular example.
Univariate time series outliers
This example uses a univariate time series containing the daily gold
prices from the R package forecast.
gold2 <- forecast::na.interp(gold)
out <- uv_tsout_ens(gold2)
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
inds <- names(which(table(out$outliers) > 2))
ts_gold <- dplyr::as_tibble(gold2) %>% mutate(t = 1:length(gold2)) %>% rename(value = x)
ggplot(ts_gold, aes(x=t, y=value)) +
geom_line() +
geom_vline(xintercept =as.numeric(inds), color="red", alpha=0.8, size=0.5, linetype ='dashed') + ylab("Gold prices") +
theme_bw()
#> Don't know how to automatically pick scale for object of type ts. Defaulting to continuous.
The red dashed vertical lines show the time points that have been
picked by 3 or more outlier detection methods.
Next we look at the quarterly production of woollen yarn in
Australia. This time series is also taken from the R package
forecast.
out <- uv_tsout_ens(woolyrnq)
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 59.5 days
inds <- names(which(table(out$outliers) > 2))
ts_wool <- dplyr::as_tibble(woolyrnq) %>% mutate(t = 1:length(woolyrnq)) %>% rename(value = x)
ggplot(ts_wool, aes(x=t, y=value)) +
geom_line() +
geom_vline(xintercept =as.numeric(inds), color="red", alpha=0.8, size=0.5, linetype ='dashed') + ylab("Woollen Yarn Production") +
theme_bw()
#> Don't know how to automatically pick scale for object of type ts. Defaulting to continuous.
Multivariate time series outliers
This example includes EU stock market data from the package
datasets. It contains the daily closing prices of 4 European
stock indices: Germany DAX, Switzerland SMI, France CAC, and UK FTSE.
First we plot the data.
stpart <- EuStockMarkets[1:600, ]
stpart <- EuStockMarkets[1:600, ]
as_tibble(stpart) %>% mutate(t = 1:n()) %>%
pivot_longer(cols=1:4) %>%
ggplot2::ggplot( ggplot2::aes(x = t, y = value, color = name)) + ggplot2::geom_line() + ggplot2::theme_bw()
Then we find multivariate outliers. To find the multivariate
outliers, first we decompose the time series to univariate outliers by
using Principle Component Analysis (PCA), Independent Component Analysis
(ICA), DOBIN (Distance based Outlier BasIs using Neighbours) and ICS
(Invariant Coordinate Selection) decomposition methods. The
fast=TRUE option leaves out ICS decomposition. Then for a
selected number of components (default being 2) we find outliers using
the univariate time series ensemble. The outliers are given in the table
below.
out <- mv_tsout_ens(stpart, fast=TRUE)
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
#> Converting from tbl_df to tbl_time.
#> Auto-index message: index = date
#> frequency = 7 days
#> trend = 91 days
out$outliers
#> Indices Total_Score Num_Coords Num_Methods DOBIN PCA ICA
#> 11 36 1.6928 3 3 0.5460645 0.6552774 0.4914581
#> 18 127 0.3120 1 1 0.3120000 0.0000000 0.0000000
#> 20 205 0.8160 3 2 0.2733508 0.4059739 0.1366754
#> 32 316 1.4400 3 2 0.4710592 0.6156465 0.3532944
#> 34 319 0.6880 2 2 0.1720000 0.0000000 0.5160000
#> 41 331 1.5008 3 3 0.2728727 0.4093091 0.8186182
#> 42 366 0.3120 1 1 0.0000000 0.3120000 0.0000000
#> 48 546 0.3120 1 1 0.3120000 0.0000000 0.0000000
#> forecast tsoutliers otsad anomalize Gap_Score_2
#> 11 0 1.248 0.320 0.1248 72
#> 18 0 0.312 0.000 0.0000 5
#> 20 0 0.624 0.192 0.0000 30
#> 32 0 1.248 0.192 0.0000 60
#> 34 0 0.624 0.064 0.0000 23
#> 41 0 1.248 0.128 0.1248 63
#> 42 0 0.312 0.000 0.0000 5
#> 48 0 0.312 0.000 0.0000 5
draw_table_html(out)
|
|
Indices
|
DOBIN
|
PCA
|
ICA
|
Num_Coords
|
forecast
|
tsoutliers
|
otsad
|
anomalize
|
Num_Methods
|
Gap_Score_2
|
Total_Score
|
|
11
|
36
|
0.55
|
0.66
|
0.49
|
3
|
0
|
1.25
|
0.32
|
0.12
|
3
|
72
|
1.69
|
|
41
|
331
|
0.27
|
0.41
|
0.82
|
3
|
0
|
1.25
|
0.13
|
0.12
|
3
|
63
|
1.5
|
|
32
|
316
|
0.47
|
0.62
|
0.35
|
3
|
0
|
1.25
|
0.19
|
0
|
2
|
60
|
1.44
|
|
20
|
205
|
0.27
|
0.41
|
0.14
|
3
|
0
|
0.62
|
0.19
|
0
|
2
|
30
|
0.82
|
|
34
|
319
|
0.17
|
0
|
0.52
|
2
|
0
|
0.62
|
0.06
|
0
|
2
|
23
|
0.69
|
|
18
|
127
|
0.31
|
0
|
0
|
1
|
0
|
0.31
|
0
|
0
|
1
|
5
|
0.31
|
|
42
|
366
|
0
|
0.31
|
0
|
1
|
0
|
0.31
|
0
|
0
|
1
|
5
|
0.31
|
|
48
|
546
|
0.31
|
0
|
0
|
1
|
0
|
0.31
|
0
|
0
|
1
|
5
|
0.31
|
The decomposed time series is shown in the figure below.
plot_decomposed_all(obj=out, X = stpart)
animate_ts_ensemble(out, X= stpart, max_frames = 100)