ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels

Abstract

Most methods for time series classification that attain state-of-the-art accuracy have high computational complexity, requiring significant training time even for smaller datasets, and are intractable for larger datasets. Additionally, many existing methods focus on a single type of feature such as shape or frequency. Building on the recent success of convolutional neural networks for time series classification, we show that simple linear classifiers using random convolutional kernels achieve state-of-the-art accuracy with a fraction of the computational expense of existing methods. Using this method, it is possible to train and test a classifier on all 85 ‘bake off’ datasets in the UCR archive in $<\,2\,\hbox {h}$, and it is possible to train a classifier on a large dataset of more than one million time series in approximately 1 h.

Appendices

Appendices

Relative accuracy

1.1 ‘Bake Off’ datasets

See Fig. 13.

Fig. 13

Relative accuracy of Rocket versus state-of-the-art classifiers on the ‘bake off’ datasets

1.2 Additional 2018 datasets

See Fig. 14.

Fig. 14

Relative accuracy of Rocket versus state-of-the-art classifiers, additional 2018 datasets

‘Development’ and ‘Holdout’ datasets

See Figs. 15 and 16.

Fig. 15

Mean rank of Rocket versus state-of-the-art classifiers on the ‘holdout’ datasets

Fig. 16

Mean rank of Rocket versus state-of-the-art classifiers on the ‘development’ datasets

Additional plots for the sensitivity analysis

See Figs. 17, 18, 19, 20, 21, 22, 23 and 24.

Fig. 17

Relative accuracy of $k=10{,}000$ versus $k=5000$ on the ‘development’ datasets

Fig. 18

Relative accuracy of $l \in \{7, 9, 11\}$ versus $l \in \{5, 7, 9\}$ on the ‘development’ datasets

Fig. 19

Relative accuracy, normally-distributed versus integer weights, ‘development’ datasets

Fig. 20

Relative accuracy of always versus random centering on the ‘development’ datasets

Fig. 21

Relative accuracy, uniformly versus normally-distributed bias, ‘development’ datasets

Fig. 22

Relative accuracy of exponential versus uniform dilation on the ‘development’ datasets

Fig. 23

Relative accuracy of random versus always padding on the ‘development’ datasets

Fig. 24

Relative accuracy of ppv and max versus only ppv on the ‘development’ datasets

Resamples

We also evaluate Rocket on 10 resamples of both the ‘bake off’ and additional 2018 datasets, using the same first 10 resamples (not including the original training/test split) as in Bagnall et al. (2017). Figure 25 shows the mean rank of Rocket versus HIVE-COTE, TS-CHIEF, Shapelet Transform, Proximity Forest and BOSS on the resampled ‘bake off’ datasets. Figure 27 shows the relative accuracy of Rocket and each of the other methods on the resampled ‘bake off’ datasets. The results for HIVE-COTE, Shapelet Transform, and BOSS are taken from Bagnall et al. (2019). Figures 26 and 28 show the mean rank and relative accuracy of Rocket versus Proximity Forest and TS-CHIEF for 10 resamples of the additional 2018 datasets (published results are not available for other methods for these resamples).

The results for the resamples and the original training/test splits are very similar for both the ‘bake off’ and additional 2018 datasets. In fact, while HIVE-COTE ranks ahead of Rocket, Rocket appears to be ‘stronger’ against both HIVE-COTE and TS-CHIEF on the resamples of the ‘bake off’ datasets than on the original training/test splits. For the resampled ‘bake off’ datasets, Rocket is ahead of HIVE-COTE in terms of win/draw/loss (47/2/36), as it is for the original training/test split (45/7/33). These results confirm that the results for the original training/test split are sound, and representative of the expected performance of Rocket relative to the other methods included in the comparison.

Fig. 25

Mean rank of Rocket versus other classifiers on the resampled ‘bake off’ datasets

Fig. 26

Mean rank of Rocket versus other classifiers, resampled additional 2018 datasets

Fig. 27

Relative accuracy of Rocket versus other classifiers on the resampled ‘bake off’ datasets

Fig. 28

Relative accuracy of Rocket versus other classifiers, resampled additional 2018 datasets

Other methods

We also compare Rocket against four recently-proposed scalable methods for time series classification (see Sect. 2.2), namely, MrSEQL, cBOSS, MiSTiCl, and catch22. We have run each of these four methods on the ‘bake off’ datasets, the additional 2018 datasets, and the scalability experiments in terms of both training set size and time series length. We have run each method with its recommended settings per the relevant papers, and using the same experimental conditions as for Rocket in each case. We have used the Python wrapper for MrSEQL (https://github.com/alan-turing-institute/sktime), the Java version of cBOSS (https://github.com/uea-machine-learning/tsml), the Java version of MiSTiCl (https://github.com/atifraza/MiSTiCl), and the Python wrapper for catch22 (https://github.com/chlubba/catch22).

1.1 ‘Bake Off’ and additional 2018 datasets

Figures 29 and 32 show the mean rank and relative accuracy of Rocket versus MrSEQL, cBOSS, MiSTiCl, and catch 22 for the 85 ‘bake off’ datasets. Figures 30 and 33 show the same for the additional 2018 datasets. Figure 31 shows total compute time.

The results show that Rocket is significantly more accurate and, with one exception, more scalable than these methods. Rocket is considerably ahead of the most accurate of these methods, MrSEQL, in terms of win/draw/loss (54/8/23) on the ‘bake off’ datasets, and Rocket is approximately an order of magnitude faster in terms of total compute time than MrSEQL, cBOSS, and MiSTiCl. While catch22 is very fast, it is the least accurate method.

As for the ‘bake off’ datasets, Rocket is significantly more accurate than MrSEQL, cBOSS, MiSTiCl, or catch22 on the additional 2018 datasets and, with the exception of catch22, considerably faster. Again, Rocket is substantially ahead of the most accurate of these methods in terms of win/draw/loss (32/0/11). Rocket is 4 times faster than cBOSS, 16 times faster than MrSEQL, and almost 22 times faster than MiSTiCl on these datasets. Again, catch22 is the fastest but least accurate method.

Note that while catch22 was originally used in conjunction with a single decision tree, we found that this produced very low accuracy and, of several ‘off the shelf’ classifiers, random forest produced the highest accuracy. Accordingly, we have used catch22 in conjunction with a random forest classifier. MiSTiCl would not run on the ElectricDevices dataset in its published configuration. For this dataset we used MiSTiCl in conjunction with AdaBoost, rather than the default extremely randomised trees. MiSTiCl would not run at all on the Chinatown dataset, so it has been ranked behind the other methods for this dataset, and this dataset has been removed from the relevant plot in Fig. 33.

Fig. 29

Mean rank of Rocket versus other classifiers on the ‘bake off’ datasets

Fig. 30

Mean rank of Rocket versus other classifiers, additional 2018 datasets

Fig. 31

Total compute time for ‘bake off’ datasets (left) and additional 2018 datasets (right)

Fig. 32

Relative accuracy of Rocket versus other classifiers on the ‘bake off’ datasets

Fig. 33

Relative accuracy of Rocket versus other classifiers, additional 2018 datasets

1.2 Scalability

1.2.1 Training set size

Figure 34 shows accuracy and training time versus training set size for Rocket—with 10,000 (default), 1000 and 100 kernels—and the other four methods for the Satellite Image Time Series Dataset. Figure 34 shows that MrSEQL, cBOSS, and MiSTiCl are all fundamentally less scalable than Rocket in terms of training set size. By approximately 32,000 training examples, MrSEQL is approximately 75 times slower than Rocket, MiSTiCl is approximately 200 times slower than Rocket, and cBOSS is more than 300 times slower than Rocket. Additionally, all four methods are noticeably less accurate than Rocket for the same training set size. We note that there appears to be a problem with MrSEQL with more than approximately 8000 training examples. While slower than catch22 with its default settings (i.e., 10,000 kernels), in contexts where this speed difference is important, restricted to 100 kernels Rocket is an order of magnitude faster than catch22 and still significantly more accurate (see Fig. 34).

1.2.2 Time series length

Figure 35 shows training time versus time series length for Rocket versus the other four methods for the InlineSkate dataset. Figure 35 shows that, in practice, the scalability of Rocket, cBOSS, and catch22 in terms of time series length appears to be similar—that is, approximately linear in time series length. Both MrSEQL and MiSTiCl are less scalable. MrSEQL, cBOSS, and MiSTiCl are all slower than Rocket for a given time series length.

Fig. 34

Accuracy (left) and training time (right) versus training set size

Fig. 35

Training time versus time series length

Results for ‘Bake Off’ datasets

See Table 1.

Table 1 Accuracy—‘Bake Off’ datasets

Results for additional 2018 datasets

See Table 2.

Table 2 Accuracy—additional 2018 datasets