Background

The working principle of the qPCR is to add fluorophore into the qPCR system, and use the fluorescence signal accumulation to detect the whole qPCR process [1]. The accumulated amount of DNA reaction products after fluorescent labeling is used as amplification data (expressed as amplification curves) can be used to determine the initial target quantity (called N0 at the concentration level and called F0 at the fluorescence level). An amplification reaction is generally displayed by an amplification curve, while the y-axis represents the fluorescence signal accumulation and the x-axis represents the number of cycles. During the process, the product fluorescence can not rise above the background at the beginning and almost tending to a straight line; as the reaction progresses, the fluorescence accumulates until the product is consumed and the fluorescence ceases to increase [2, 3]. The reason for this process is that, initially, the product quantity is very small, caused a weak fluorescence signal to be detected at baseline phase. The exponential increase of the product starts in cycle 1. It becomes visible when its associated fluorescence can be observed above baseline noise. During the transitional phase products continue to accumulate, but reagents become limiting and the reaction efficiency begins to fall. Until the product is no longer produced, so the reaction reaches to plateau phase [4]. Therefore, the baseline phase, exponential phase, transitional phase, and plateau phase of the amplification curve are generated based on the quantitative relationship between the fluorescence signal accumulation and cycles in Fig. 1A.

Fig. 1
figure 1

Amplification process and determination of parameter in CqMAN method. (OriginPro 2020b https://www.originlab.com/)

Full size image

In Fig. 1A, the initial fluorescence of the reaction is at the background level with high noise, almost no fluorescence signal can be detected, then the product fluorescence rises above the background in the exponential phase within a few cycles and begins to saturate in the approach to the final plateau phase. Figure 1B shows the locations of relevant parameters determined by CqMAN method.

For the relevant parameters of the amplification curve, the amplification process determines a quantitative threshold (called Fq in most methods) indicates a detectable fluorescent signal produced by the accumulation of sufficient amplification products which is generally set in the exponential phase. The x-axis of this quantitative threshold corresponds to a cycle called Cq in most methods, which is called CqMAN in our method.

The amplification efficiency(E) is another important parameter for checking qPCR data analysis. Under ideal conditions, the number of DNA sequences will double in each cycle, the percentage of E-1 is 100% (at this time E is 2) [5]. However, due to factors such as reaction inhibitors, enzyme, primer and probes differences, PCR efficiency rarely reaches 100%. Therefore, E is any number between 1 and 2 [6]. Previously published studies have been suggested that PCR efficiencies mostly range between 65 and 90% [7].

After determining the quantitative cycle, the quantitative threshold, and estimating the amplification efficiency, the kinetics of qPCR exponential phase are described by eq. (1) to indicate the initial target quantity of the reaction.

$$ {N}_n={N}_0\times {E}^n $$
(1)

in which N0 and Nn, are the initial target amount of DNA and the DNA target amount after n cycles, respectively. Fn, the fluorescent signal after n cycles and F0, the fluorescence signal represents starting amount of the target DNA are the performance of Nn and N0 at the fluorescence level [2]. Therefore, eq. (1) can be described as eq. (2)

$$ {F}_n={F}_0\times {E}^n $$
(2)

using the relevant parameters estimated by the curve analysis algorithm method can be expressed as eq. (3)

$$ {F}_q={F}_0\times {E}^{C_q} $$
(3)

then the observed initial target quantity(F0) is calculated.

In the past two decades, the rapid development of qPCR technology has led to the production of multiple protocols, reagents, analytical methods and reporting formats. The original standard-Cq method [8, 9] fits a standard curve by preparing multiple sets of replicable experiments of the samples of known concentration, and estimates the concentration of unknown samples from the standard curve. This approach assumes that all standard samples have the same efficiency and is only effective if thresholds are measured from the exponential phase of the PCR reaction, some authors have reported that this assumption may be questionable [10]. Later, an approach proposed by Liu and Saint [11] assumes an efficiency can be obtained by fitting PCR amplification curve with a sigmoid function without preparing standard samples. Since then, the pioneering methods of estimating the target quantity of the initial reaction by calculating the reaction efficiency from the dynamics of a single PCR reaction has been widely used for improvement, and these methods differ in determining the fluorescence baseline, exponential phase, Fq, Cq, E to estimate initial target quantities. Baseline estimation is considered a constant baseline in some methods, including the observed minimum fluorescence, the mean value of the three lowest observations, the mean value of a set of fixed cycles [8, 12,13,14], and the baseline may also be determined by means of a set of dynamically determined baseline phase periods [15, 16] and taking-difference linear regression method [17]. However, the true value of background fluorescence is unknown, and errors in baseline estimation can lead to significant distortion of the results [8, 18, 19]. The difference defined by the exponential phase can easily lead to different results [20]. The residual algorithm estimation with the maximum value of the second derivative as the end point is commonly used [11, 21], or three periods are selected within the midpoint of the fluorescence signal [22]. Estimation of efficiency includes fitting the entire exponential cycle [21, 23], calculating the slope of the points within a certain defined range after linear regression [22, 24], and obtaining the ratio of the threshold fluorescence to the fluorescence value of the previous cycle [25]. Fq is generally defined in the exponential phase and then the value of Cq is determined, but in some methods, Fq and Cq are not involved [15, 26, 27]. And the definition process of all parameters may be combined with the fitting of the amplification curve to better obtain [23, 25, 26].

In order to provide reference for further develo** and evaluating the qPCR curve analysis method and promoting the research of quantitative fluorescence PCR in gene expression, the new curve analysis method and other methods were evaluated on the biomarker dataset and 94-replicates-4-dilutions set in this paper from the aspects of expression level and statistical significance. The goal of this paper is to make our new method a comparison of other methods, at the same time provide users with an alternative curve analysis scheme. In order to evaluate the new method, some evaluation performance indicators were proposed.

Methods

qPCR dataset

Biomarker dataset

Data comes from a previously published study [28] that developed and validated the expression profile of a 59-mRNA gene to improve prognosis in children with neuroblastoma. This dataset measured 59 biomarkers and 5 reference genes in a sample maximization experimental design, using the LightCycler480 SYBR Green Master (Roche) in a 384-well plate with 8 μl reaction. These genes have been reported in at least two independent studies as prognostic genes for neuroblastoma. Three hundred sixty-six cDNA samples from the primary tumor biopsy and a 5-point 10-fold serial dilution series based on an external oligonucleotide standards (from 150,000 to 15 copies, n = 3), and no template control (NTC, n = 3) are included in each plate [28, 29]. This dataset will be referred to as ‘biomarker dataset’ in this study. Since there was no obvious specificity of 63 genes in this dataset, 20 of them (AHCY,AKR1C1,ARHGEF7,BIRC5,CAMTA1,CAMTA2,CD44,CDCA5,CDH5,CDKN3,CLSTN1,CPSG3,DDC,ECEL1,ELAVL4,EPB41L3,EPHA5,EPN2,FYN,HIVEP2) were randomly selected and then 300 (5 × 3 × 20) amplification curve data of 20 genes with concentration of 150,000, 15,000, 1500, 150, 15(3 replicated experiments for each group) were used for subsequent analysis.

94-replicates-4-dilutions set

This data set created a dilution series consisting of four 10-fold serial dilution points from 15,000 to 15 molecules, using 10 ng / μl yeast tRNA as a carrier (Roche) and created NTC samples of the same dilution. qPCR was done on a CFX 384 instrument (Bio-Rad). QPCR was performed on a CFX 384 instrument (Bio-Rad) using a 96-well pipetting robot (Tecan Freedom Evo 150). Amplification reactions were performed in 8 μl samples containing 0.4 μl forward and 0.4 μl reverse primer (5 μM each), 0.2 μl nuclease-free water, 4 μl iQ SYBR Green Supermix (Bio-Rad) and 3 μl of standard oligonucleotide. In 384-well plates (Hard-Shell 384-well microplate and Microseal B clear using an adhesive seal (Bio-Rad)), for each of the 4 dilution points, a total of 94 replicate reactions were distributed. In addition, the NTC reaction was repeated 8 times [28]. This dataset will be referred to as ‘94-replicates-4-dilutions set’. And 44 (4 × 11) amplification curves of the MYCN gene with a diluted concentration of 15, 150, 1500,15,000(11 replicated experiments for each group) were used for subsequent analysis.

qPCR curve analysis method

Previously published curve analysis method

We provide general descriptions of the 7 methods previously published. In this study, these methods will be referred to with their preferred abbreviations LinRegPCR, DART, FPLM, FPK-PCR, 5PSM, PCR-Miner and Cy0. The LinRegPCR program [16] starts with import of raw fluorescence data. A constant baseline fluorescence is determined per reaction with an iterative algorithm that aims at the longest set of data points on a straight line going down from the second derivative maximum cycle. After subtraction of the baseline fluorescence, LinRegPCR sets a window-of-linearity (W-o-L) that includes 4 points in the exponential phase of each sample and calculates the individual PCR efficiency from the slope of the regression line through these points. For each amplicon group, a quantification threshold Fq is set at 1 cycle below the top border of the W-o-L and the Cq is determined for each reaction. DART [22] constructs a model based on the maximum fluorescence value (Rmax) and the baseline fluorescence noise (Rnoise) to determine a central point M, and fits the cycle within a 10-fold range around M to estimate E, Fq, Cq obtain by 10-fold the standard deviation of 1–10 cycles. FPLM [21] uses four-parameter logistic model to fit the fluorescence curve and estimate the exponential phase, and the same as DART in determining Fq, Cq. The bilinear model and the six-parameter logistic model are used in the FPK-PCR [26] to estimate the E and initial target quantity without determining fluorescence threshold.5PSM [25] uses the ratio of the fluorescence value at the second derivative maximum (SDM) after fitting the curve with the five-parameter model to the fluorescence value of the previous cycle as the amplification efficiency and the cycle of SDM is used as the Cq. The principle of PCR-Miner [30] is based on the four-parameter logistic model to fit the raw fluorescence data as a function of PCR cycles to identify the exponential phase of the reaction. The method chooses the first positive second derivative maximum from the logistic model to calculate the dynamic fluorescence threshold and corresponding Cq. A three-parameter simple exponent model is fitted to this exponential phase using an iterative non-linear regression algorithm to compute the individual efficiency. Cy0 [31] obtains the intersection point (Cy0) between the abscissa axis of the curve inflection point and the tangent line based on the nonlinear regression of the Richards equation to the fluorescence value. The efficiency is estimated by the parameters in the post-fitting equation, and then the initial target quantity is obtained.

CqMAN method

CqMAN (Cq Management And Analysis System) is an adaptive analysis system that summarizes the methods and experiences of previous methods and provides a robust, objective, and noise-resistant method for quantification of qPCR results. Since researches have shown that smoothing can at best lead to erroneous accuracy of results, and usually also bias the results [32], the improved adaptive Savitzky-Golay filter in the CqMAN system is only used for visual display of data. The detailed process is shown in Additional file 1. The CqMAN method has been implemented in the system. We provide the URL of the system (http://122.193.29.190:9913/xMAN/en-us/index), and readers can reproduce our experimental results by combining with Additional files 1 and 2.

CqMAN method relies on the modified gompertz model, is fitted to the raw fluorescence data by means of a non-linear fitting routine the Levenberg-Marquardt algorithm that minimizes the residual sum-of-squares to obtain parameters baseline fluorescence (y0) and maximum fluorescence (ymax), exp. is the natural logarithm base, Ln is the natural logarithm, x is the actual cycle number, b and x0 determine the shape of each model.

$$ y={y}_0+\left({y}_{max}-{y}_0\right){\lambda}^{{-\mathit{\exp}}^{-\left(x-{x}_0\right)/b}} $$
(4)
$$ {F}_{SDM}= bLn\left(\left(\left(\sqrt{5}-3\right){\lambda}^{\frac{X_0}{b}}\right)/2\right) $$
(5)

The maximum value of the second derivative are obtained by fitting the second derivative of the gompertz curve to estimate the end of the exponential phase (eq. (5)). xSDM is the cycle at the maximum of the second derivative (SDM) which is applied as the end point of the exponential phase and the fluorescence value corresponding to this cycle is FSDM in CqMAN method. Take the intermediate value of y0 and FSDM as the “midpoint” Fq (eq. (6)), then substitute this value into eq. (4) to obtain the quantitative cycle (CqMAN) (see Fig. 2B).

$$ {F}_q=\left({y}_0+{F}_{SDM}\right)/2 $$
(6)
Fig. 2
figure 2

Performance indicators per method. (OriginPro 2020b https://www.originlab.com/)

Full size image

For efficiency estimation, a three-parameter simple exponent model is fitted to this exponential phase (from CqMAN to xSDM) using an non-linear regression algorithm to estimate the single reaction’s individual efficiency in eq. (7). The individual efficiency of multiple reactions of the same gene is averaged, then the observed target quantity (F0) can be calculated by eq. (8).

$$ {F}_n={F}_q+\alpha \times {E}^n $$
(7)
$$ {F}_0=1/{E_{\boldsymbol{mean}}}^{C\mathrm{q} MAN} $$
(8)

The logistic model used in Cy0, PCR Miner are generally susceptible to the influence of the number of amplified data in the plateau phase, resulting in inaccurate fitting [33]. At the same time, 5PSM adds a parameter to the logistic model to maintain the symmetry of the s-shaped curve structure, which will affect the calculation of parameters such as the maximum of the second derivative, resulting in larger errors. The gompertz model in CqMAN is not easily affected by the data in the plateau phase, and it fits well in all the phase. At the same time, this method can ensure that the CqMAN value is within the exponential phase without judging the starting point of the phase (in the first 2–3 cycles of the cycle where the SDM is located). It does not rely on baseline estimation of the noise larger phase of the fluorescence signal, and avoids the problem of deviation caused by the assumption in the DART and FPLM method that a constant baseline can be determined from the baseline phase. By using nonlinear regression fitting to estimate the average efficiency of all reactions of each gene, CqMAN method further averaged the amplification reaction noise between each gene, more effectively resisting the noise while reducing the estimation error [31]. However, the shortcoming is that this method is prone to error under the influence of dynamic outliers (inhibition), in which aspect FPK-PCR performs better.

Results

Performance indicators

To eliminate the different measurement scales used by the analytical method based on concentration levels and fluorescence levels [34], we divided the data of all concentrations by the highest concentration data and all fluorescence data by the average value of the maximum observed target quantity (F0), so that the average value of the maximum concentration and the maximum observed target quantity is 1. This process is called normalization. Then data sets were used to establish 6 performance indicators to measure the degree of compliance between the observed initial target quantity (F0) calculated by the algorithm and the true value from different angles. Among them, the bias and relative error are used to compare the difference between the observed initial target quantity and the true value; coefficient of variation and precision are used to compare the difference between the observed initial target quantity (F0) of the same group. The smaller the difference, the more reliable the method. Performance indicators as follows.

(1) Bias. The ratio between the average of the observed initial target quantity F0 corresponding to the highest and lowest concentrations is calculated. In biomarker, the expected value of this ratio is 10,000 (because the ratio of the concentration of 150,000 and 15 is 10,000), and in 94-replicates-4-dilutions set, the expected value of this ratio is 0.001 (because the ratio of the diluted concentration of 15 and 15,000 is 0.001) and any value deviating from 10,000 or 0.001 is expressed as a bias. The log-transformed (base 10) between the true value and the initial target quantity F0. After the data is normalized, the linear regression analysis makes the log (F0) and log (NC) (NC, normalized concentration) slopes of the unbiased method 1 and any slope deviates from the value of 1 also expressed as a bias.

(2) Relative error (RE).

$$ RE=\frac{F_0- NC}{NC} $$
(9)

RE is the deviation after F0 and NC are normalized to the same measurement scale.

(3) Coefficient of variation (CV).

$$ CV=\frac{SD_{\boldsymbol{group}}}{\mu_{\boldsymbol{group}}}\times 100\% $$
(10)

CV represents the ratio of the standard deviation (SD) to the average value(μ) of the same group (replicated experiments) of observed initial target quantity (F0).

(4) Precision. Precision represents the within-triplicate variance of the observed initial target quantity (F0) in the same group.

(5) Resolution. A linear regression analysis of log (true) on log(F0) was performed and the 95% CI around the regression line was constructed. The width of this interval was converted into a fold deviation from the regression line and the geometric mean for the 5 groups was calculated as a measure of resolution.

Indicator evaluation

In the supplementary information, the original amplification experiment data of the two data sets used in this study were obtained from Reference [28] after being processed into the readable format of the CqMAN system. We imported the data of these two data setsinto the CqMAN system to obtain the F0, Cq, and E calculated by the CqMAN, integrated the results with the three parameter values of the other 7 methods provided in reference [28] (see biomarker_performance _indicators and 94_replicates_4_dilutions_set_results). In the bias_and_deviat_from_regres of biomarker_performance_indicators, the process of CqMAN calculating 4 performance indicators is shown and it is the same as the calculation process of other 7 methods. Therefore, the calculation process of the other 7 methods is no longer provided. The performance indicators’ calculation results of the 8 methods are provided in biomarker_analysis_dilutoin_series in and 94_replicates_4_dilutions_set_results.

Except that the efficiency analysis results of the other 7 methods (see Fig. 3) directly used the data provided in reference [28] in the subsequent performance indicator analysis, the analysis results of other performance indicators are all reanalysis results.

Fig. 3
figure 3

PCR efficiency per gene. (OriginPro 2020b https://www.originlab.com/)

Full size image

Biomarker dataset analysis

The performance indicator values determined from the concentration series included in the measurement of the 20 genes are summarized in box-and-whisker plots. The boxes range from the 25th to the 75th percentile and are divided by the median; the whiskers are set at the 5th and 95th percentile (A) Bias in the slope level, which is based on the degree of deviation from 1.(B) The box-and-whisker plot of relative errors shows the difference between the observed initial target quantity and the true value.(C) Coefficient of variation is an objective indicator of the effects of measurement scales and dimensions that eliminate fluorescence levels and concentration levels.(D) Precision is determined as the within-triplicate variance and should have the same, low, value in all methods.(E) Resolution defined as the fold-chance that would result in the detection of a difference at a 5% significance level.

The mean value of the efficiencies of each gene per method.

(1) Bias. We expect the ratio between the observed initial target quantity and the true value to be 10,000 or 0.001 in two different datasets. After the data is normalized, the linear regression analysis makes the log (F0) and log (SQ) slopes of the unbiased method 1, which will be unbiased. Cy0 has an advantage in the deviation index because the method calculate the efficiency value based on the slope of the relationship between Cy0 and log (input), and then use this efficiency value and the Cy0 value to calculate F0. Therefore, Cy0 is unbiased and are the result of circular reasoning, but this also ensures that the observed initial target quantity F0 is more accurate. Other methods are positively or negatively biased, and the observed values deviate significantly from the true values in Fig. 2A. Among them, CqMAN performs better in the bias, with an average deviation of 2469.0003(for 10,000) and 0.0182(for 0.001).

(2) Relative error. The relative error was originally used to compare the difference between the measured value and the true value, and the degree of confidence in the response measurement. Here we can use the relative error response to calculate the difference between the observed value and the true value, reflecting the credibility of the algorithm. More intuitive response measurement accuracy than absolute error. We use relative error as one of the indicators to determine the difference between the observed initial target quantity F0 and the true value. Cy0 performed best, average relative error was 0.1050. The average relative error of the rank after the second PCR-Miner was 0.2287, CqMAN was 0.2416, and the highest 5PSM was as high as 0.6939 in Table 1 and Fig. 2B.

Table 1 Analysis of the average of 20 genes in 4 indicators per method
Full size table

(3) Coefficient of variation. The coefficient of variation reflects the degree of dispersion of the data, and at the same time overcomes the effects of large differences in measurement scales or different data sizes. We use the coefficient of variation coefficient to calculate the degree of dispersion of the observed initial target quantities of the three groups at each concentration, and average the five groups of coefficients of variation. The smaller the coefficient of variation, the lower the degree of dispersion. Result showed that CqMAN showed the best performance of 7.20%, Cy0, LinRegPCR, PCR-Miner also stabilized at about 9.60%, and FPK-PCR’s coefficient of variation was as high as 25.12% in Table 1 and Fig. 2C.

(4) Precision. The five concentration sequences were measured three times and the fluorescence data were analyzed. Therefore, the variance of each set of 3 measurements should be small, reflecting only random changes in laboratory procedures and fluorescence measurements, and such changes should always be the same. The resulting three internal variances can be considered as a measure of the accuracy of the analytical method. CqMAN, 5PSM, Cy0, LinRegPCR have lower variability in Fig. 2D.

(5) Resolution. Data points outside the 95% CI of the regression line fitted to the concentration sequence after linear regression will be judged to be significantly different from the true value and expressed in resolution. LinRegPCR has the lowest resolution; lower is better. Cy0, PCR-Miner and CqMAN also perform well in Fig. 2E. With these 4 methods, the observed 2-fold difference is significant for approximately 85% of genes. For 5PSM, DART, FPLM, the resolution lies between the 2 and 3-fold-difference. In FPK-PCR, 40% of genes are over 5-fold-difference.

(6) Efficiency. The range of differences in efficiency values for each method indicates that this variability is the sum of the difference in efficiency between genes and the difference in estimation methods. Therefore, the difference between the methods cannot be explained. Except that DART and FPLM share a method of finding E, other methods get different median values of E. FPK-PCR and PCR-Miner have a large number of efficiency values above 2, which is obviously too high and the median value of CqMAN, Cy0, LinRegPCR, 5PSM is between 1.7 and 1.9. We calculated the standard deviation of the amplification efficiency of the 20 genes, in which LinRegPCR, DART, FPLM calculated E value is relatively stable in Fig. 3.

94-replicates-4-dilutions set analysis

The highest dilute concentration is set to 1, the y-axis is set to log (dilution) (base 10).

(1) Target quantity. For data with dilute concentrations of 15,000, 1500, 150, and 15, respectively, the observed target quantity should be as close as possible to the expected value −3, −2, −1, 0 obtained after calculating the log (F0) (base 10) in Fig. 4. The systematic negative or positive deviation of each analysis method is shown by the deviation of the average F0 from the expected value (Fig. 4: horizontal line). CqMAN, Cy0, PCR-Miner and LinRegPCR have the least bias. DART and FPLM show a higher bias, 5PSM displays a strong overestimation whereas FPK-PCR shows a strong underestimation of F0 values.

Fig. 4
figure 4

Mean observed F0 value per concentration and method. (OriginPro 2020b https://www.originlab.com/)

Full size image

(2) Bias, RE, CV, precision, resolution and E. CqMAN and Cy0 keep lower variance in bias. CqMAN perform best in RE, CV and precision. CqMAN, Cy0, LinRegPCR and PCR-Miner does not vary much between the values in CV and precision. LinRegPCR has the lowest resolution, the average resolution of Cy0, PCR-Miner and CqMAN is around 2-fold. Table 2 clearly illustrate the differences in 6 indicators of 8 methods and the average PCR efficiency of these methods is provided. The efficiency of Cy0 was not provided in the previously published data analysis.

Table 2 Analysis of MYCN gene in 4 indicators and the mean of PCR efficiency per method
Full size table

Discussions

For each of the evaluation indexes of the concentration sequence analysis of each gene, the rank synthesis method was used, and the Friedman test determined that these methods were not significantly different and comparable. Table 3 shows the results of each gene and method. The lower average rank indicates that the method which estimates the initial target quantity is closer to the true value in the performance evaluation of the four indicators we selected.

Table 3 Analysis of performance parameters per method in biomarker dataset (left) and 94-replicates-4-dilutions set (right). For each method, the mean rank is given for each of the performance indicators bias, RE, CV, precision and resolution. The methods are sorted based on the average of these ranks
Full size table

In the average rank sorting of 20 genes in the biomarker data set, the lowest rank average of CqMAN and Cy0 are 2.08. The rank averages of the 5PSM, DART, FPLM, and FPK-PCR are all above 6, and the overall performance of F0 estimation is lower in Table 3. For the 94-replicates-4-dilutions set, the performance of CqMAN is 1.58, the average rank of Cy0 is 1.92, and the performance of LinRegPCR and PCR-Miner are also good; the rank average of 5PSM, DART, FPLM, and FPK-PCR is much higher.

Conclusions

Based on PCR kinetics and exponential model simulations, this study combines the real-time quantitative PCR curve analysis method proposed by the predecessors, and proposes a reliable gene expression level quantification method, CqMAN. To prove the reliability of the method, two data sets from different instruments, different PCR mixtures, and a testable hypothesis were used to evaluate the performance of multiple qPCR curve analysis methods. The fluorescence data of the other 7 methods in the performance analysis process were taken from a previously published research by Ruijter et al. in 2013 [28]. Since the supplemental information from this research provided an excel template for calculating bias and precision, we can directly import the amplification curve data from two data sets analyzed by the CqMAN system into the excel template to obtain the calculated values of the two indicators. The relative error and coefficient of variation are the two statistical indicators proposed by the author of this study for evaluation and analysis. Therefore, due to the difference in indicator settings and the difference in data sets selection, our analysis results are different from the results previously published by Ruijter et.al.

The limitation of this study is that two datasets have limited evaluation of the general applicability of the CqMAN method, so future researches should include more instances and more verification indicators to better verify the robustness and representativeness of the method. However, it is undeniable that the analysis templates, datasets, and analysis results (see supporting information) in this research will definitely help further evaluation of research and make the results comparable with our results.

The aim of this study is not to promote a particular curve analysis method with the best overall performance, because the choice of methods by the experimenters may depend on the different research goals of experimental instruments, reagents, protocols, etc. It is our intention to help users choose the ideal method for their own studies and developers to modify and improve their methods [35].