Method

Time-series classification algorithms are typically benchmarked using the Time Series Classification Repository, which contains 128 univariate (i.e., values sampled uniformly in time) time-series classification datasets/problems and 30 multivariate datasets. Here, we focus on the univariate setting. Many algorithms developed across the sciences have been evaluated across the standardised univariate datasets provided in the repository, with black-box algorithms typically outperforming competitors (see this paper for a review). Given the growing need for algorithmic transparency in the sciences and industry, the present research aims to explore the feasibility of a new approach using one of the 128 datasets as a test case: the "Wafer" dataset. Specifically, this work aims to determine if using informative summary statistics (known as "features"), can be used to construct a high performing classification procedure that rivals or outperforms existing approaches (see this paper for more). Examples of time-series features include properties of the time-series distribution of values, autocorrelation structure, entropy, model-fit statistics, nonlinear time-series analysis, stationarity, and many others (see this paper). At its core, feature-based time-series analysis reduces a time series x time matrix to a time series x feature matrix, which can then be used for statistical learning (such as classification), using the values of time-series features as inputs.

Dataset

The Wafer dataset in the Time Series Classification Repository contains a collection of inline process control measurements that were recorded from a range of sensors during the manufacturing and processing of silicon wafers for semiconductor fabrication. Each unique time series in the Wafer dataset represents the measurements of one sensor during the processing of one wafer by one tool. Labels for two classes are provided in the data: Normal and Abnormal. The goal of this problem is to predict class membership from the time-series values. The dataset contains a pre-designated train-test split, with 1000 samples in the train set (each of length T = 152), and 6164 samples in the test set (each of length T = 152).

Algorithmic Approach

The approach in this work contains three stages: (i) extraction of time-series features for each unique time series; (ii) dimensionality reduction using principal components analysis (PCA) to obtain a reduced set of informative vectors; and (iii) classification using the top principal components (quantified in terms of cumulative variance explained) as inputs into a random forest classifier. Features from two open-source feature sets---catch22 (see here for the native R implementation Rcatch22) and Kats --- will be extracted using the R package theft. catch22 extracts 24 features (as mean and standard deviation are added to the standard 22 features) and Kats extracts 40 features. This will produce a resulting time series x feature matrix of size 7164 x 64. Given the existence of within-set redundancy (i.e., high absolute correlations between features in a set) observed particularly for Kats relative to catch22, as well as between-set redundancy identified in previous work, dimensionality reduction through PCA will be applied to substantially reduce the input matrix size for the classification algorithm into a time series x principal component matrix. A threshold of 80% cumulative variance explained will be used to determine the number of principal components to retain. Following the procedure of previous work, the present approach then trains and evaluates the accuracy of the classifier over 30 resamples of train-test splits, where each is seeded for reproducibility, and the first is always the pre-designated train-test split in the data as it comes from the Time Series Classification Repository. This will enable scientific inference of algorithmic performance with uncertainty, and facilitate a direct comparison with the performance of existing algorithms and benchmarks.

Given the existence of the Time Series Classification Repository which holds the Wafer dataset (whose sole purpose is to facilitate benchmarking of time-series classification algorithms), it is expected that meaningful class separation will be possible. The present research takes a novel approach of chaining time-series feature extraction, dimensionality reduction techniques, and a classification algorithm --- an approach that has seen almost no research attention to-date. As such, a primary goal of this work is to understand the performance of this approach relative to current benchmarks.

Given the high within-set redundancy (i.e., high absolute correlations between features in a set) observed in previous work, it is hypothesised that dimensionality reduction techniques will substantially reduce the input matrix size for the classification algorithm from the original time series x feature matrix to a much smaller time series x principal component matrix. Further, it is also hypothesised that using the time series x principal component matrix as input to a classification algorithm (i.e., random forest classifier) will not result in a substantial reduction in classification performance compared to using the time series x feature matrix nor to existing benchmarks due to the highly informative nature of time-series features in understanding temporal dynamics.

Results

Prior to substantative analysis, exploratory data analysis was performed. A sample of three time series from each class (Normal and Abnormal) is displayed in the figure below. To the eye, some small, but noticeable differences in temporal dynamics and shape are visible between the classes, which suggests that classification based on temporal properties (i.e, "features") is a feasible approach.

Dimensionality reduction

Time-series features were computed from two sets (catch22 and Kats) on all time series, which produced a time series x feature matrix of size 7164 x 64. This matrix is large, which increases computation time for classification algorithms and increases complexity of model interpretation and evaluation. To reduce this complexity, a principal components analysis was performed. The dataset meets the assumptions of PCA, as multicollinearity between time-series features was a reason for conducting the analysis, and there are sufficient samples to adequately perform PCA as the 7164:64 samples:variables ratio exceeds the 20:1 recommendation.

The figure below plots a visual summary of the PCA. Panel (A) plots the percentage of variance explained for the top eight principal components (PC) which together explain 80% of the variance. PC1 explains 37.5% of the variance, but there is a steep drop off following this component, with PC2 explaining 15.6% of the variance. Panel (B) plots cumulative variance explained for the eight retained PCs. The first four PCs explain two-thirds of the variance in the dataset. Panel (C) plots the eigenvalues of the eight PCs. All eight PCs exceed the eigenvalue > 1 threshold for the Kaiser criterion of PC retention.

Loadings for each of the 64 time-series features (variables) on the eight PCs is presented in the figure below. Patterns are evident across the PCs, such as the strong loading of time-series features associated with properties of the autocorrelation and partial autocorrelation function onto PC1, and histogram-based statistics onto PC3. This plot gives us interpretable and informative insight into the relative behaviour of the time-series features extracted from the Wafer dataset.

Prior to modelling, the PCA was distilled into an even lower dimensional space of just two dimensions to understand if class differences could be ascertained with just the two PCs which explain the most variance in the data (a collective 53.1%). This is displayed in the figure below). There is considerable overlap between the "Normal" and "Abnormal" classes in the two-dimensional space, suggesting the additional six principal components are likely needed for accurate classification.

Time-series classification

The time series x principal component matrix (7164 x 8) was passed as input into a random forest classifier using the caret package in R over 30 resamples, where the first sample was the pre-designated train-test split from the Time Series Classification Repository. Mean classification accuracy across the resamples was 99.08% (SD = 0.19%) and is compared to previous benchmarks in the table below. The benchmark algorithms include collective of transformation-based ensembles (COTE), shapelet transform (ST), bag of SFA symbols (BOSS), elastic ensemble (EE), dynamic time warping (DTW), time series forest (TSF), time series bag of features (TSBF), learned pattern (LPS), and move-split-merge (MSM). The current approach, while marginally outperformed by the other algorithms, demonstrates classification performance that can be considered to be on-par with more complex, more black-box algorithms, despite only using eight principal components as an input to a random forest classifier. Importantly, no manual hyperparameter tuning was performed beyond caret's basic parameter grid search that it performs over the $k$-fold cross-validation procedure. It is likely that even stronger performance could be obtained through more detailed hyperparameter tuning and optimisation. Further, relative to the benchmark algorithms presented in the table below, the current approach is fast --- executing the entire classification stage (including all 30 resamples, each with 10-fold cross-validation) in under four minutes locally on a laptop with no parallel processing.

To better understand the inner mechanics of the current approach, we examine the machinery of the random forest models. Understanding relative variable importance can shed a deeper light into random forest models to aid interpretability. Variable importance ranks for each variable (principal components) across the 30 resamples are plotted in the figure below. PC4 (comprised of mostly symbolic features, such as those associated with entropy of small-set probabilities, flat spots, and proportion of magnitudes that exceed a threshold over the standard deviation) is the most important variable across all 30 resamples for predicting class ("Normal" versus "Abnormal").

The performance of PC4 is further demonstrated in the figure below which plots the mean +/- 1SD of variable importance values across the 30 resamples. On average, PC4 exhibits variable importance of a factor of 5.5 higher than any other variable.