1 Introduction

Current manufacturing industries must comply with international quality standards [18]. These standards define the quality the products must have in many different aspects such as surface quality, material behavior or product dimensions. The production of defective products can generate large economic losses and damage the reputation of the manufacturer. Depending on the purpose of the product, critical safety issues may also arise. Therefore, modern industries need quality control systems that can inspect the products automatically, in real time, without slowing down production [3].

Long products can be defined by their transversal section. They are manufactured in a rolling mill in which a bar is moved between rolls that press it into the desired shape. In fast production lines, a whole product must be inspected and diagnosed fast enough that the quality inspector can act when a defect appears. This is a definition of a real-time restriction which must be taken into account in all the stages of the processing.

Two very important features of long products are surface and dimensional quality. Surface and dimensional inspection can be carried out manually by quality inspectors to some extent, but generally it is performed using automatic systems [15, 21]. However, using this approach the detailed information that the dense point clouds give is lost as the whole cloud is smoothed to be approximated to surfaces or primitives.

The second approach consists of approximating the point cloud directly to the product without approximating it to primitives or surfaces. Using this method, all the detail stored in each point is used to create a precise comparison between the scanned product and the desired surface, generating a topographic map of the surface [3]. Using these images, a surface inspection system capable of detecting small anomalies in the surface of the products can be developed [2]. In order to calculate these differences, the point clouds must be aligned with a mathematical representation of the desired shape of the product, called a model.

The third approach does not use a model but only the information given by the point clouds. The distance between neighboring points in the point cloud is calculated and analyzed to detect abnormalities [16]. This method cannot be used when the desired shape is not flat or smooth so it may not fit in some cases.

The point clouds must be aligned to the model of the products if one of the two first approaches is used [10]. This alignment is called the registration method. Registration is usually done with a two-step method [19, 23]. In the first step, the cloud is moved and/or rotated in order to be almost aligned with the model using the centroids to calculate the transformation. Then, a second transformation is calculated to align the point cloud and the model perfectly. This second registration step is performed with some kind of iterative algorithm such as iterative closest point (ICP) [25] or globally optimal ICP (Go-ICP) [22] which only converges if it starts from a close solution, given by the first step.

The kind of application should lead to a registration process that uses all the points at once as 3DMatch [7, 24] or a registration that is made profile by profile for reducing or even eliminating the effect of vibrations, movements and flatness problems during the inspection process. This paper is thought for an industrial environment, so the second approach is used.

The time spent on processing the point cloud is critical in processes such as product inspection in fast production lines or augmented reality applications. In these situations, the processing must be done as fast as possible in order to meet strict real-time restrictions. This speed can be achieved by implementing the method in hardware using GPUs [5] or FPGAs [6]. The cost of this kind of hardware is high, so in order to run the registration process in computers with no hardware accelerator, it must be optimized using real-time strategies such as reducing the amount of data or saving information from previous iterations [9].

3 Proposed method

The registration process of a point cloud to a mathematical model is performed in two steps. The first step is usually sufficient for many purposes. However, to better align the whole surface and avoid errors produced by dimensional problems in the product, a second step of registration is performed using the knowledge obtained from the first step.

First, the product must be defined. This method is designed for long products, which are products whose shape can be well defined by a transversal section. This transversal section is usually defined mathematically using arcs and segments, called primitives. The set of arcs and segments that defines the transversal section of a product is called a model.

The point cloud obtained from scanning a transversal section of a long product is called a profile. The set of all the profiles scanned from a product can be used to reconstruct the complete product. In the case of the long products, a set of sensors must be placed around the product and the profiles must be obtained while the product is moving. Usually, these sensors, called profilometers, are devices composed of a laser source and a camera.

The case of study for this paper is a long product that requires strict quality standards: a rail. In Fig. 1, the placement of the sensors is shown as well as the real piece of rail used for the study. As shown in Fig. 1a, the profiles of the product are acquired with four sensors placed around it. Thus, the scan is made up of four different point clouds that must be aligned to the model.

Fig. 1
figure 1

Study case definition

Full size image

The proposed method is thought to be used just after the product is manufactured in an industrial facility. Therefore, the vibrations that are produced in such environment must be taken into account. The set of profiles cannot be aligned with a complete representation of the product at once, they must be aligned one by one in order to get rid of the deviations produced by vibrations. In the case of surface inspection, aligning the profiles one by one also mitigates the effect of flatness defects in the product.

3.1 First registration step

In the first step of the registration process, there is no knowledge of the positioning of the points of the point cloud with respect to the model. Therefore, the whole model must be used in order to obtain the optimal alignment. In this case, four different partial models are obtained in order to perform the registration, as shown in Fig. 1c. These four partial models are defined according to the sensor placement and international standards used for rails [18] which define four zones of interest called A, B, C and D in [17]. Each of the point clouds shown in Fig. 1a is aligned to one of the partial models shown in Fig. 1c, which are zone A, zone BL, zone BR and zone D. The zone C is combined with the zone B in both sides because the sensor that acquires the profile from the side of the product acquires both zones B and C of the same side.

First, a coarse registration is performed to approximately align the point cloud with the model. This registration is based on the alignment of the centroids and orientations between the model and the point cloud. First, the centroid of the model and the centroid of the point cloud are aligned by shifting the centroid of the point cloud to the centroid of the model. The centroids are calculated as the average positions of all the points. Then, the point cloud is rotated to align its orientation with that of the model. The orientation is calculated based on principal component analysis (PCA) to estimate the normal of the surface.

The coarse registration approximates the point cloud to the model so a more accurate registration procedure can be applied in order to get the best alignment. This fine registration is performed using ICP algorithm [1]. This algorithm estimates the correspondences between the point cloud and the model by searching for the closest point in the model to each point in the cloud, and it computes a rigid transformation between them. This process is repeated until converging on a local solution. As the model can be divided in primitives, arcs and segments, the method only looks for the closest point in the closest primitive.

After this first registration step, the point cloud and the data are aligned, but several problems may appear related to the dimensions of the scanned product. For this reason, a second step, which incorporates the information provided by the first step, is necessary.

Fig. 2
figure 2

Partial models for second registration step

Full size image

3.2 Second registration step

After the first step of registration is performed, each point in the point cloud has been assigned (or aligned) to a primitive in the model. This information can be used for a second step of registration using the same ICP algorithm in order to better align the points with each primitive or set of primitives in such a way that the dimensional problems do not affect the surface inspection.

In this second registration, the model must be split into several partial models in order to fit the different zones in the contour. Figure 2 shows three options according to the definition of the sets of primitives.

In Fig. 2a, the partial models used are actually the primitives in the model. These primitives are labeled using initials to refer to their class (A: arc, S: segment), their zone (H: head, W: web, F: foot), their position (T: top, B: bottom) and their side (R: right, L: left). Using these partial models, the points corresponding to a primitive will be aligned only with that primitive, so a perfect match between the points and the model will only be achieved if there is not any volumetric anomaly on the surface.

In Fig. 2b, the partial models are defined as fitting zones, areas in which the human operators can use their physical gauges to check the dimensions and the quality of the rails. They are labeled according to international quality standard zones [17]. Note that most of them are a set made up of a segment and its adjacent arcs and many of these partial models share some arcs with others. The shared arcs add a non-deterministic condition to the assignment of points to partial models: A single point could be assigned to two partial models that share the arc in which the point has its corresponding point. In order to eliminate this difficulty, a third option is presented.

In Fig. 2c, the partial models of Fig. 2b are merged to get partial models with no shared primitives while maintaining clear divisions between the conceptual zones to be checked. Following this idea, the point clouds acquired by the side sensors can be divided into three partial zones that include one wide angle and the adjacent flat zones. These zones are the side and bottom of the head; the web and part of the top of the foot; and the other part of the foot and the side of the foot. The point clouds acquired by the top and bottom sensors are not divided as the conceptual shape they must fit is so simple.

There is only one overlap left in these partial models, located in the side arcs of the head of the rail. This overlap, or shared arc, can be easily solved by using only the points from each acquired point cloud to fit one set of partial models. In this way, the points acquired by the top sensor will be aligned to partial model A, while the points from the right sensor will be aligned to partial model BR in the first step and to partial models BR1, BR2 and BR3 in the second step.

Taking into account the problems presented by the second approach shown in Fig. 2b, only two sets of partial models are proposed, shown in Fig. 2a and c.

3.3 Improvements

The first step of the registration process can be divided into four different registrations: one per partial model or sensor. These four executions are run in parallel to speed up the whole process. However, the time spent to calculate the best transformation for all the points in the point cloud and their corresponding point in the model is excessive. Thus, the algorithm must include a subsampling technique to decrease the number of points used. A cache that stores the primitive associated with each point in the point cloud is included in each registration process to speed up the registration of the next profile of the product.

Subsampling is implemented with two different techniques. The first, 1/N, takes one point of each N points in the point cloud. This technique does not take into account the distance between the points, and therefore, the zones in which more points are acquired will have more weight in the registration process. The second technique, Nmm, samples the model and creates a point cloud with one point for each N millimeters of profile. For each point in this sampled cloud, the closest point in the model is calculated and added to a new point cloud to be used in the registration process, giving the same weight to all parts of the profile.

The second registration step follows the same approach as the first: Only a subset of the points associated with each partial model is used to calculate the transformation. While it does include a cache for each partial model that stores the primitive associated to each point in the point cloud, this is only needed when the first approach of partial models shown in Fig. 2a is not used.

The second registration step has an additional feature: a rejection factor. The points corresponding to a defective surface are further from the model than the rest. If they are used for the registration, they introduce an error as the process tries to minimize the distance between the points and the model. These points can be detected as outlier values of the distance. Thus, the registration process is improved with a rejection factor that only takes into account the closest points to the model to calculate the transformation.

4 Experimental protocol

The piece used for the experimentation, shown in Fig. 1b, has several defects which should be preserved during the registration process. However, this piece was selected for this test as it has a dimensional defect. This defect is located in the web of the rail which is shorter than the desired one, so the distance between the head and the foot does not fit the model. The dimensional defect should not appear as a surface defect in the final diagnosis of the product, so it should be eliminated in the registration process or in a later step of processing. Dimensional defects can also affect the way in which the surface defect are measured. In Fig. 3, an example of how dimensional defects can affect in other measures is shown. In this paper, the second step of the registration process is designed to eliminate the influence of these dimensional defects to obtain better results while inspecting the surface of a product.

Fig. 3
figure 3

Surface defect measuring in a dimensional defect

Full size image

If only the first registration step is performed, the point clouds of the sides of the product can be aligned with either the bottom of the head or the top of the foot of the rail. The ICP algorithm converges into a solution near the coarse registration output. The coarse registration can be near the top or the bottom, so it will directly influence the final ICP solution. This adds some uncertainty to the final registration. In Fig. 4, the dimensional defect of the test piece can be seen in the final registration (coarse and fine registration of the first step): The alignment is different on the left and the right sides. Therefore, the process result is not deterministic in this kind of situations. In the figure, the point cloud on the left side aligns with the bottom of the model, while on the left, it aligns with the top of the model. Therefore, the result is different in both sides, giving the impression that a defect exists in the top left side and in the bottom right side. In Fig. 4, the model is shown in green and the points in red.

Fig. 4
figure 4

Dimensional defect in the test piece

Full size image

The uncertainty given by the local solution obtained using the ICP algorithm can be mitigated using derived methods such as globally optimal ICP (Go-ICP) [22]. However, this options will not provide a solution to the local registration that is needed in order to eliminate the problem produced by the dimensional problems shown in Fig. 3. The Go-ICP algorithm will be tested and compared with the ICP method used in the first registration step to check which of them provides a closer solution from which the second registration step can start.

The dimensional defect of test piece is in the web of the rail, the middle section, which a bit shorter than the models. In a previous work [2], the information of the registration process is used to create topographic maps of the surface of rails. In this previous work, the dimensional defects were eliminated using image filtering.

The experimentation with the second registration step will be carried out using three test cases that appear in the piece shown in Fig. 1b. The first test case is in the bottom part of the head on the rail in which both types of defects appear, a surface defect and a dimensional defect. The second test case is given by a big protrusion that appear in the web of the rail (a product identifier) in order to test the viability of the method whenever a defect of the same size that the primitives in which it is located appears. The third test case is located in the upper side of foot of the rail in which the density of the points is low providing information about the influence of the density of the points in the method.

The test cases defined before will be studied in using four different registration options:

  • Using only the first registration step that provides registration between the whole point cloud obtained by a sensor and the model.

  • Using two-step registration method based on each primitive in the model as defined in Fig. 2a.

  • Using two-step partial registration based on conceptual zones as defined in Fig. 2c with a sampling of 1 in 20 points.

  • Using two-step partial registration based on conceptual zones as defined in Fig. 2c with a sampling of 1 point per 2 mm.

5 Results

First, the comparison between the ICP and the Go-ICP registration is shown in Fig. 5. The comparison is made using the three test cases defined in the previous section. In the figure, the points corresponding to ICP registration are shown in red, while the ones from Go-ICP registration are shown in blue. It can be seen that the results are very similar; the blue points almost are exactly at the same position that the red ones. These experiments were carried out using one-step registration. As this experiment shows that both methods provide very similar results, only ICP method will be used from now on to perform the rest of the test as the Go-ICP method has more complexity and therefore it needs more computational resources.

Fig. 5
figure 5

Registration using ICP (in red) and Go-ICP (in blue). Model is shown in green

Full size image
Fig. 6
figure 6

Examples of results

Full size image
Fig. 7
figure 7

3D Visualization of the results

Full size image
Table 1 Metrics of the experiments
Full size table

Figure 6 shows examples of the results in three problematic zones: a dimensional problem (Fig. 6a–d), a product identifier (Fig. 6e–h) and a zone with low point density (Fig. 6i–l). The state-of-the-art method with no second step is included as well as a measurable surface defect in the head of the rail (see Fig. 6a–d) used to test the results against surface quality inspection.

In order to give a better visualization of the results, the piece is represented in color. The color represents the distance between the points and the model. When the points are inside the model, the distance is represented as a negative number while if it is outside the model the distance is represented as a positive number. The color scale used goes from blue to green to red representing the interval [-4,4] millimeters. This means that the protrusions should be represented in red and the holes should be represented in blue. This visualizations are shown in Fig. 7.

These methods have been tested with more that 10,000 profiles in order to obtain relevant performance metrics. The response time in this kind of application is critical because of the real-time restrictions of the manufacturing process. The depth of the surface defect is also measured from the model to the deepest point acquired in the defect. Its nominal value is 1.5 mm. These measurements are shown in Table 1 proving that the hierarchical method proposed improves the measurement of the surface defects compared to the current state-of-the-art registration. These tests are carried out in a Intel Core i5-7400 CPU at 3.00 GHz with 16 GB RAM memory.

The metrics of the performance test are shown in Table 1. In this table, the average response time is shown for the profiles acquired by each sensor as well as the average distance from the points to the model.

In order to give more information about the registration time depending on the method used, the histograms of the registration times measured are shown in Figs. 8, 9, 10 and 11. As long as the top and bottom zones are not problematic, their histograms are not shown to focus in the problematic zones belonging to the sides of the rail, zones BL and BR.

As the tests were carried out in a specific computer described above, it is also interesting to measure the usage of the resources of the computer during the experiment. To do so, during the tests the CPU usage was measured and it is shown in Fig. 12.

Fig. 8
figure 8

Registration time histograms for one-step procedure

Full size image
Fig. 9
figure 9

Registration time histograms for two-step primitives procedure

Full size image
Fig. 10
figure 10

Registration time histograms for two-step partial 1/N procedure

Full size image
Fig. 11
figure 11

Registration time histograms for two-step partial 1 per Nmm

Full size image
Fig. 12
figure 12

CPU usage during the test

Full size image

6 Discussion

In this section, the results will be discussed. First, we will discuss the effectiveness of the proposed methods in the registration process. Then, the performance in terms of response time will be discussed.

Using the partial models from the first approach, shown in Fig. 2a, the adjustment at all the primitives is improved by eliminating the dimensional defect completely along with the noise it may produce in further processing. It is important to note that the elimination of the defect is done symmetrically, on both sides of the rail. However, this first approach proved to be unsuitable for rails as this type of product has known volumetric anomalies that causes the registration to fail as it looks for the minimum average distance between the points and the partial model. In Fig. 13, one protrusion, a product identifier, is shown as an example of a common volumetric anomaly in rails. The protrusion is almost as long as the primitive in which it is placed, so the registration aligns the point cloud in such a way that the result is inaccurate and noisy although it minimizes all the distances. This situation produces anomalies as it is shown in Fig. 7b, where it can be seen that near the product identifier some blue zones appear as the protrusion is understood as the surface of the product.

Using the partial models from Fig. 2c, the dimensional defect is also eliminated. The problem caused by the protrusion shown in Fig. 13 is still an issue even when the partial models are larger. The effect of these volumetric anomalies can be reduced using a rejection factor as defined in Sect. 3.3. The rejector can eliminate the outliers and not use them in order to calculate the transformation in the registration process. The rejection factor can also be used in the first approach but does not solve the problem with the product identifiers or other large anomalies. Using the primitives as partial models when a large anomaly appears, the outliers are the points which represent the surface of the rail because of the size of the anomaly.

In order to speed up the process, a subsampling is also performed. This means that the registration process for each partial model does not take into account all the points in the point cloud assigned to that partial model, but rather a subset of them. Thus, the probability of a point from a defect or anomaly being used in the registration process is also reduced. This probability is reduced even more by using the subsampling in addition to the rejection factor.

The results show that the partial models method successfully aligned each part of the product and was not affected by surface anomalies such as product identifiers. However, the subsampling method that uses 1 in N points for the registration is heavily affected by low density zones (as shown in Fig. 6k). Thus, the best method for the second step hierarchical registration in this case is the partial model method using equidistant point sampling for registration.

Fig. 13
figure 13

Problem when aligning to primitives with a protrusion

Full size image

Although using the partial models the dimensional anomaly is eliminated from the surfaces, there are some areas in the results that still shows that a dimensional problem exists. In Fig. 7g and h, these areas are shown in the arcs that join the web and the head of the rail.

The primitive registration method shows the best results when there are no volumetric anomalies. When these anomalies appear, the method shows errors that can be classified later as defects or as expected anomalies like product identifiers.

Depending on the type of product and the requirements of the quality inspection, the method chosen may vary. While the primitives method provides the best alignment, it is influenced by volumetric anomalies. The partial method provides slightly inferior alignment but it is much less influenced by low density zones or volumetric anomalies when using an equidistant subsampling for the registration process.

The times obtained from the methods give important information regarding the kind of application in which they can be used. Whereas the primitive registration gives the best results in the registration process (when no big anomalies appear), it is also the slowest one as shown in Fig. 9. The difference between both sides of the product in this method is explained by the influence that the product identifier has in the registration process.

In terms of real-time processing, the most suitable method is the one that uses partial models. Using this method, the variation seen between the different types of subsampling is important, see Figs. 10 and 11. Using 1/N subsampling is not faster but the measures are more aggregated at a certain value, which gives a more deterministic result in terms of response time. Using this method, the results show that the hierarchical method is fast enough to fulfill the real-time requirements of some quality inspection lines.

One important feature to care about is the usage of the CPU during the tests. The tests were carried out generating the profiles at 500 Hz and using four threads to process each of the four sides of the product. In Fig. 12, it can be seen that the CPU usage is distributed near the 30%, which means that the cores could be also used to do some other operations that the application need during the registration. It is also seen that the usage is similar to the one-step registration method, meaning that these methods could be applied to current applications with no hardware change required.

7 Conclusions

This paper proposes a novel two-step hierarchical registration method. The first step of the registration method is well known but has some deficiencies when used by itself in the presence of dimensional defects. The methods proposed for the second step of the registration uses the information obtained in the first step to improve the alignment of the points to the desired model.

In order to fit any kind of inspection application, two methods have been designed. The first one aligns the points to the model according to their correspondence to each primitive in the model. The second one fits conceptual zones of the product that human quality inspectors use to inspect the product. Two subsampling strategies are also tested in order to speed up the process using only a subset of the point cloud for the registration. This reduces the number of operations.

The methods have been executed over more than 10,000 profiles in order to test their performance. These tests show that they are suitable for almost any quality inspection process in real time by simply tuning the subsampling strategy. Testing has been done successfully using a real piece of product which has several problematic zones: a dimensional defect, a protrusion and a low point density zone. In the piece, there is also a surface defect that has been measured, showing that the proposed methods outperform the state-of-the-art registration method in surface inspection applications, based on a coarse registration and ICP.

Depending on the type of product, the best method may vary. If the final product definition always has larger primitives that the size of the potential anomalies, the primitives method will give the best results. Otherwise, in order to eliminate the influence of the anomalies, the partial method should be chosen. Using the partial method, the density of the points should be studied. With a homogeneous method, the subsampling “1 in N” is faster as it requires fewer operations than the equidistant sampling. However, if the density is not homogeneous, the equidistant method should be used.

The real-time constructions of the final application must be taken into account being the best method for real-time scenarios the one that uses partial models with 1 per N points as a subsampling method. This option gives good results in terms of both registration and response time.

This paper provides a set of well-tested methods that can be selected by quality engineers in order to fit the requirements of their product quality inspection line. Using these methods, the registration process can be significantly improved, while meeting the real-time restrictions thanks to improvements such as caching and sampling. The provided methods are also suitable for applications in which surface anomalies and dimensional problems are expected, maintaining good results in the registration process.