Multivariate long-time series traffic passenger flow prediction using causal convolutional sparse self-attention MTS-Informer

Abstract

As an important part of the operation preparation process of the intelligent transportation system, the passenger flow distribution law and forecast can guide the urban rail transit to formulate a reasonable operation scheduling plan. Due to the complexity, multi-variables, and instability of traffic passenger flow data, accurate passenger flow prediction takes a lot of work. Based on a convolutional neural network, a causal convolution self-attention traffic passenger flow prediction model MTS-Informer framework is proposed. This method follows the changing law of auxiliary variables, adopts the stabilization method to reduce the instability of the original sequence, and uses the causal convolution feature to improve the ability of the model’s self-attention mechanism to extract local information from the input sequence. The weakening effect of the self-attention mechanism ensures that it can learn similarly to the differential features in the original sequence data. In addition, the stationarity detection of the original sequence data is added. The experimental results show that the fitting degree of the sample data is significantly improved, and the standard error decreases between 10 and 40%, which verifies the effectiveness of the proposed modeling technique. It has higher prediction accuracy and operating efficiency and can provide a basis for urban traffic passenger flow prediction.

Appendices

Appendix A

To reveal the causal convolutional deep neural network structure in Sect. 3.2, the built-in detailed description is also introduced:

A causal convolutional network consists of an input, hidden, and output layer, respectively, as shown in Fig. 8. Each layer uses the same type of neurons and uses a non-full connection between each layer, which is realized through a mask.

Fig. 8

The input layer is used to input feature vectors, the hidden layer is used to extract raw data information, and the output layer is responsible for outputting results. The self-attention mechanism of the built-in causal convolutional neural network in this work mainly plays a crucial role in obtaining the local characteristics of the input time series

Supplementary explanation for the sparse probabilistic self-attention algorithm mentioned in Sect. 3.2:

$$\begin{aligned}{} & {} A(q_i,k,v)=\sum _{j}^{L_k}{\frac{k(q_i,k_j)}{\sum _{l}k(q_i,k_i)}}=E_{p(k_j\vert q_i)}[v_j] \end{aligned}$$

(1)

$$\begin{aligned}{} & {} p(k_j,q_i)=\frac{k(p_i,k_j)}{\sum _{l}(q_i,k_i)} \end{aligned}$$

(2)

$$\begin{aligned}{} & {} q(k_j,q_j)=\frac{1}{L_k} \end{aligned}$$

(3)

Among them, let $q_i$, $k_i$ and $v_i$ denote the i-th row in Q, K, and V, respectively, $p(k_j,q_i)$ represents the attention probability distribution of the i-th query to all keys. $q(k_j,q_j)$ represents the uniform distribution of query.

Use the KL divergence formula to calculate the distance between the two distributions of P and Q to measure the sparsity of the query. The discrete KL divergence is defined as follows:

$$\begin{aligned}{} & {} D(P\vert \vert Q)=\sum _{i\in x}P(i)*[\log {(\frac{P(i)}{Q(i)})}] \end{aligned}$$

(4)

$$\begin{aligned}{} & {} D(P\mid \mid Q)=\int _x P(x)*[\log {(\frac{P(i)}{Q(i)})}] dx \end{aligned}$$

(5)

Add the attention rate distribution and balanced distribution of the query to the KL divergence, and finally determine the approximate expression of the sparsity of the query as:

$$\begin{aligned} KL(q\vert \vert p)={max}_j\{q_ik_j^T \cdot d^*\}-\frac{1}{L_k}\sum _{j=1}^{L_k}(q_i k_j^T\cdot d^*) \end{aligned}$$

(6)

In the formula, the first item calculates the inner product of the ith query and all keys and selects the maximum value. Compared with the arithmetic mean of the second item, the more significant the result difference, the greater the difference between p and q. In the set selection interval, select the query with a higher difference ranking, and the sampling parameters determine the size of this interval. Therefore, the time series L uses the sparse probabilistic self-attention mechanism to calculate the similarity between Q and K only requires calculating the $O(L\log L)$ dot product operation. The dot-product manipulation of traditional self-attention means it makes the time complexity and memory usage $O(L^2)$ per layer, and stacking n encoder–decoders for long-term sequence inputs has a total usage of $O(n \cdot L^2)$. The probabilistic sparse self-attention mechanism can achieve $O(L\log L)$ regarding time complexity and memory usage by improving network components through the above methods.

Appendix B

In the experiment, the visualization results of the sequence normal distribution test are supplemented as follows:

Fig. 9

Normality test P-P diagram

Fig. 10

Normality test Q-Q plot

The cumulative probability graphic method (Fig. 9) is a scatter diagram drawn according to the cumulative probability of the predictor variable corresponding to the cumulative probability of the specified theoretical distribution. The x-axis is the cumulative probability of the sample and the corresponding y-axis ranges from (0, 1) between. The quantile graphical method (Fig. 10) uses quantiles to express the x-axis as the quantile of the expected overall distribution, and the y-axis as the quantile of the empirical distribution of the predicted sample. We use the above two graphical methods for normality testing, the expected distribution is set to a standard normal distribution, and it is necessary to examine whether the point falls on $y=x$, the intercept of the straight line is the mean, and the slope is the standard deviation. Suppose the overall distribution corresponding to the sample is a normal distribution. In that case, the scatter points corresponding to the samples in the P-P and Q-Q diagram should fall near the $45^{\circ }$ line from the origin.

The optimization operator is particularly critical in improving the self-attention performance of this paper. This group of control experiments can intuitively and clearly express its advantages. Whether to introduce the optimization operator has a particular impact on the self-attention mechanism, and its prediction effect is generally better than that of the basic. The effect of the model has been well proven on various data sets (Fig. 11).

Fig. 11

Control optimization operator experimental group