High speed neuromorphic vision-based inspection of countersinks in automated manufacturing processes

Abstract

Countersink inspection is crucial in various automated assembly lines, especially in the aerospace and automotive sectors. Advancements in machine vision introduced automated robotic inspection of countersinks using laser scanners and monocular cameras. Nevertheless, the aforementioned sensing pipelines require the robot to pause on each hole for inspection due to high latency and measurement uncertainties with motion, leading to prolonged execution times of the inspection task. The neuromorphic vision sensor, on the other hand, has the potential to expedite the countersink inspection process, but the unorthodox output of the neuromorphic technology prohibits utilizing traditional image processing techniques. Therefore, novel event-based perception algorithms need to be introduced. We propose a countersink detection approach on the basis of event-based motion compensation and the mean-shift clustering principle. In addition, our framework presents a robust event-based circle detection algorithm to precisely estimate the depth of the countersink specimens. The proposed approach expedites the inspection process by a factor of 10$\times$ compared to conventional countersink inspection methods. The work in this paper was validated for over 50 trials on three countersink workpiece variants. The experimental results show that our method provides a standard deviation of 0.025 mm and an accuracy of 0.026 mm for countersink depth inspection despite the low resolution of commercially available neuromorphic cameras. Video Link: https://www.dropbox.com/s/pateqqwh4d605t3/final_video_new.mp4?dl=0.

Introduction

Automation of manufacturing processes has been transforming the industrial landscape for over a century, beginning with Ford’s introduction of the assembly line. While automated manufacturing was initially aimed at producing large quantities, low-value, and standardized items (Yan & Melkote, 2023), it is now being applied to producing high-value, specialized items. This shift is driven by the need to achieve and sustain competitiveness in an operating environment with a growing number of competitors and declining highly specialized labor (Jayasekara et al., 2022). In addition, automated manufacturing is becoming increasingly feasible due to the combination of low-cost computational power and the declining costs of sensors and actuators. This allows for more specialized applications to be realized across various industries, including aerospace and automotive assembly lines (Jayasekara et al., 2022). These technologies are making it possible to create flexible, efficient, and lean automated manufacturing processes.

Automated industrial inspection is one of the essential operations in automated assembly lines, with computer vision being the driving force of the aforementioned process (Feldhausen et al., 2022; Truong et al., 2019). The main challenge with the contact method is high susceptibility to loss of accuracy resulting from debris and cutting lube buildup (Long et al., 2019; Nsengiyumva et al., 2021). Non-contact methods include laser sensors and machine vision, which are increasingly adopted with current industrial automation trends. Haldiman (2015) performed countersink inspection using digital light processing (DLP) projector and a monocular camera achieving a precision of 0.03 mm. In addition, Luker and Stansbury (2020) proposed using high-accuracy laser scanners to inspect countersinks, drop** the precision up to 6 $\mu m$. While these works provide the requirements of the current industrial requirements, they remain costly active methods and require extensive calibration and instrumentation. Therefore, Long et al. (2019) utilized a high-resolution monocular camera for countersink inspection, while abiding by the industry’s requirements and attaining a precision of 0.02 mm. Nevertheless, such precision is conditional, given the imaging sensor is normal to the countersink specimens. To circumvent this limitation, Wang and Wang (2016) utilized spectral domain optical coherence tomography (SD-OCT) for non-contact countersink inspection to deliver accurate countersink measurements without requiring precise scan center alignment.

Despite the growing adoption of computer vision for automated countersink inspection, the execution time of the inspection task was never taken into consideration. In all of the aforementioned methods, sensor motion during inspection is prohibited as it introduces measurement errors, degrading the inspection precision. This restriction rises from the fact that laser-based measurements suffer from high latency while conventional industrial cameras are associated with motion blur due to their low temporal resolution, leading to prolonged inspection time. Neuromorphic vision sensors (NVS) address the limitations of the aforementioned sensing pipelines with a high potential for highly dynamic applications. Neuromorphic vision sensors are bio-inspired sensors whose pixels are smart and independent of each other: when a pixel detects motion, it triggers an event defined by the pixel location and the time of the event. The output of the NVS is therefore not frames at constant time intervals but rather an asynchronous stream of events in space and time, delineating neuromorphic vision with three key advantages: 1- Low latency ($1 \mu s$), 2- high temporal resolution (1ms), and 3- robustness to motion blur (Siqi, 2022). This unprecedented nature of the NVS triggered a paradigm shift in the computer vision community, where neuromorphic vision unveiled its impact in space robotics (Mahlknecht et al., 2022; McMahon-Crabtree & Monet, 2021; Salah et al., 2022), robotic manufacturing (Ayyad et al., 2023), robotic gras** (Huang et al., 2022; Li et al., 2020), autonomous navigation (Bertrand et al., 2020; Liu et al., 2021), tactile sensing (Naeini et al., 2020; Rigi et al., 2018), and visual servoing (Abdul et al., 2021).

Contributions

With the unprecedented capabilities of neuromorphic cameras in hand, this paper investigates the feasibility of high-speed neuromorphic vision-based inspection of countersinks. Nevertheless, the NVS is an emmerging technology and its asynchronous, event-based nature imposes a challenge since traditional image processing techniques are not applicable for feature extraction and event-based algorithms are required for neuromorphic perception. Accordingly, this article comprises the following contributions for neuromorphic inspection of countersinks:

• 1. For the first time, a high-speed countersink inspection approach is proposed using a neuromorphic vision sensor. The proposed inspection pipeline reduces the inspection time to few seconds while providing the precision and accuracy requirements of various automated industries.

• 2. A novel event-based countersink detection algorithm is proposed based on motion compensation and the mean-shift clustering principle that overcomes the speed limitations of conventional countersink inspection methods.

• 3. A robust event-based circle detector is devised on the basis of Huber loss minimization to cope with the asynchronous, event-based nature of the NVS and to provide the required inspection precision and accuracy.

We performed rigorous experiments and validated the proposed inspection framework on over 50 trials of an NVS swee** three countersink workpiece variants with different hole sizes at speeds ranging from 0.05 to 0.5 m/s. The results show that our fast inspection approach provides comparable results to state-of-the-art inspection methods and outperforming them in terms of inspection time by a factor of 10$\times$, while providing a standard deviation of 0.025 mm and an accuracy of 0.026 mm albeit the low resolution of the NVS.

Structure of the article

The rest of the article is organized as follows. “Preliminaries” section provides an overview of the neuromorphic vision working principle and an overview of the inspection setup. “Methodology” section discusses the theory behind the event-based motion compensation and event-based circle detection for countersink depth estimation. Experimental validation of the proposed inspection pipeline is reported in “Experimental validation and results” section, and “Conclusions and future work” section presents conclusions and future works.

Fig. 1

The robotic inspection setup illustrates the different components and defines the coordinate frames used for navigation and control

Preliminaries

Robotic inspection setup

The overall setup of the neuromorphic vision-based inspection system can be seen in Fig. 1. The presented system consists of an industrial robotic manipulator, and a customized end-effector holding a frame-based camera and a neuromorphic camera.

For the purpose of robot navigation and control, we have defined the following frames of reference:

• $\mathcal {F}_{Rb}$: The robot base coordinate frame.

• $\mathcal {F}_{Rt}$: The robot linear track coordinate frame.

• $\mathcal {F}_E$: The robot end-effector coordinate frame.

• $\mathcal {F}_C$: The frame-based camera coordinate frame.

• $\mathcal {F}_N$: The neuromorphic camera coordinate frame.

• $\mathcal {F}_W$: The workpiece coordinate frame.

• $\mathcal {F}_{Ar}$: The fiducial / ArUco coordinate frame.

• $\mathcal {F}_{h^i}$: The coordinate frame of the ith reference hole.

The rotation matrix map** a source frame $\mathcal {F}_S$ to a target frame $\mathcal {F}_T$ can be defined as $_{T}R_{S} \in \mathbb {R}^{3\times 3}$. The position vector that gives the relative position of point b to point a described in $\mathcal {F}_T$ can be defined as $^{T} _{a}\vec {P}_{b} \in \mathbb {R}^3$. Then, we can define the affine transformation $_{T}T_{S} \in \mathbb {R}^{4\times 4}$ as follows:

$$\begin{aligned} _{T}T_{S} = \begin{bmatrix} _{T}R_{S} &{} ^{T}_{T}\vec {P}_{s} \\ \textbf{0}^T &{} 1 \end{bmatrix} \end{aligned}$$

(1)

Using the robot’s forward kinematics relations, the affine transformation matrix $_{Rt}T_{E}$ can be identified and considered known for the remainder of this paper. The robot’s forward kinematics can be denoted as follows:

$$\begin{aligned} _{Rt}T_{E} = g(\theta ), \hspace{5mm} \theta \in \mathbb {C} \end{aligned}$$

(2)

where $\theta$ is the monitored angles of the robotic manipulator and linear track, $g(\theta )$ is a nonlinear function representing the robot’s kinematics, and $\mathbb {C}$ is the robot’s configuration space. Additionally, the constants $_{E}T_{C}$ and $_{E}T_{N}$ can be computed using the geometrical calibration method explained in Ayyad et al. (2023). Hence, $_{B}T_{E}$ and the calibrated transformations can be utilized to solve for $_{B}T_{C}$ and $_{B}T_{N}$. Finally, we define the neuromorphic camera’s twist vector $\vec {V}_{N} = [v_x, v_y, v_z, \omega _x, \omega _y, \omega _z]^T$ that represents the neuromorphic camera’s velocity in its own frame $\mathcal {F}_N$, where $[v_x, v_y, v_z]$ and $[\omega _x, \omega _y, \omega _z]$ are the linear and angular velocities respectively. The velocity components are computed from the forward kinematics of the robot manipulator as follows:

$$\begin{aligned} \vec {\mathcal {V}}_N = J(\theta )\dot{\theta } \end{aligned}$$

(3)

$$\begin{aligned} J(\theta ) \triangleq \dfrac{\partial g_{N}}{\partial \theta } \in \mathbb {R}^{6 \times N_j} \end{aligned}$$

(4)

where $J(\theta )$ is the Jacobian matrix of the camera’s forward kinematics function $g_N$, and $N_j$ is the number of robot joints.

As the major reference frames of the robotic setup are defined, the inspection routine is performed, with the neuromorphic camera swee** the workpiece at a constant high speed. Prior to initiating the inspection process, the robot must first localize the workpiece in 6-DoF and align the neuromorphic camera accordingly. The robot starts with a priori estimate of the workpiece’s pose $_{Rt}\hat{T}_{W}$, and the pose of the fiducial marker $_{Rt}\hat{T}_{Ar}$. The robot then places the frame-based camera with a specific standoff away from the fiducial marker and utilizes the ArUco pose estimation method of the OpenCV library (Tocci et al., 2021) to refine $_{Rt}\hat{T}_{Ar}$, similar to the approach used in Halwani et al. (2023). Subsequently, the pose of each $i^{th}$ hole relative to the fiducial marker is known from the CAD model of the workpiece and is used to estimate the hole’s pose relative to the robot’s base frame:

$$\begin{aligned} _{Rt}\hat{T}_{h^i} = \hspace{1pt} _{Rt}\hat{T}_{Ar} \hspace{1pt} _{Ar}T_{h^i} , \hspace{0.5cm} i \in [1, N_h] \end{aligned}$$

(5)

Once the pose of each hole is estimated, the robot places the neuromorphic camera normal to the first hole with a specific relative pose and initiates the swee** process. The swee** process is performed by moving the neuromorphic camera with a constant velocity $\vec {V}_N^*$ across all the holes that lie in a straight line and have identical orientation. Given the position of the first hole $^{N} _{N}\vec {P}_{h^1}$ and last hole $^{N} _{N}\vec {P}_{h^{N_h}}$ described in $\mathcal {F}_N$, the linear components of $\vec {V}_N^*$ are computed as follows:

$$\begin{aligned} \begin{bmatrix} v^*_x \\ v^*_y \\ v^*_z \\ \end{bmatrix} = v^* \hspace{0.2cm} \frac{^{N} _{N}\vec {P}_{h^{N_h}} -\ ^{N} _{N}\vec {P}_{h^{1}}}{|| ^{N} _{N}\vec {P}_{h^{N_h}} -\ ^{N} _{N}\vec {P}_{h^{1}} ||} \end{aligned}$$

(6)

where $v^*$ is the desired magnitude of the swee** speed. The rotational components of the swee** motion are set to zero $[\omega ^*_x, \omega ^*_y, \omega ^*_z]^T = \vec {0}$. The commanded camera velocity $\vec {V}_N^*$ is then transformed to command joint velocities $\theta ^*$ by inverting the expression in (3), which are tracked using low-level PID controllers.

Neuromorphic vision sensor

The NVS is a bio-inspired technology that comprises an array of photodiode pixels triggered by light intensity variations. Upon activation, the pixels generate asynchronous events, where they are designated with spatial and temporal stamps as

$$\begin{aligned} e_k\doteq (\vec {x}_k,t_k,p_k), \end{aligned}$$

(7)

where $(x, y)_{k} \in W \times H$ is the spatial stamp defining the pixel coordinates, $W\in \{1,...,w\}$ and $H\in \{1,...,h\}$ represent the sensor resolution, $t$ is the timestamp, and $p\in \{-1,+1\}$ is the polarity corresponding to a decrease and increase in light intensity, respectively.

While the NVS output is unorthodox in contrast with traditional imaging sensors, it is fundamentally important to point out that the NVS utilizes identical optics to conventional cameras. Thus, the neuromorphic sensor follows the standard perspective projection model expressed by

$$\begin{aligned} s\begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K \begin{bmatrix} R \text { } T \end{bmatrix} \begin{bmatrix} x \\ y \\ z \\ 1 \end{bmatrix}, \end{aligned}$$

(8)

where u and v are the image coordinates of a mapped point in 3D space with world coordinates x, y, z, whereas K is the intrinsic matrix comprising of the focal lengths $f_{x}$ and $f_{y}$, skew $\gamma$, and principal point $[u_{0}, v_{0}]$. On the other hand, R|t are the extrinsic parameters representing the transformation from world coordinates to the sensor frame.

Methodology

This section presents the event-based inspection pipeline of the countersink holes. As the NVS performs a quick sweep of the countersinks workpiece, the unconventional nature of the neuromorphic sensor permits utilizing conventional image processing techniques for inspection. Therefore, “Event-based detection of countersinks” section presents an event-based method for detecting and identifying the countersinks on the basis of motion compensation and mean-shift clustering for inspection. Finally, we propose a robust circle detection algorithm to estimate the depths of the countersink specimens with high precision, as described in “Depth estimation of countersink holes” section.

Event-based detection of countersinks

As the neuromorphic vision sensor sweeps the workpiece, let the $N_{e}$ events in $e_{k}$ $\in$ $\textbf{E}_{i}$ be the generated events when the sensor is normal to a designated countersink set. Prior to countersink depth estimation, the observed designated countersinks for inspection need to detected and identified from the raw event stream. However, dealing with $\textbf{E}_{i}$ in the space-time domain introduces undesired computational complexities, the events need to be transformed to a 2D space for lightweight feature extraction of the observed holes. A naive approach is to accumulate the events to 2D images directly from their spatial stamps $\vec {x}_{k}$. Nevertheless, large accumulation times are correlated with motion blur, while short accumulation periods lead to degraded features of the countersink specimens. To counteract these limitations, we rather adopt event-based motion compensation (Wang et al., 2022) and leverage the asynchronous nature of neuromorphic events to create sharp images of warped events (IWE) defined by

$$\begin{aligned} H_{i}(\vec {x};\vec {\dot{U}}) = \sum ^{N_{e}}_{k}p_{k}\delta (\textbf{x} - \vec {x}'_{k}), \end{aligned}$$

(9)

where

$$\begin{aligned} \vec {x}'_{k} = \vec {x}_{k} - \vec {\dot{U}} (t_{k} - t_{i}). \end{aligned}$$

(10)

Each pixel $\textbf{x}$ in the IWE $H_{i}$ sums the polarities $p_{k}$ of the warped events $\vec {x}'_{k}$ to a reference time $t_{i}$ given a candidate optical flow $\vec {\dot{U}} = [\dot{u}, \dot{v}]^{T}$, where $t_{i}$ is the initial timestamp of $\textbf{E}_{i}$ and $\delta$ is the dirac delta function. As discussed in “Robotic inspection setup” section, the inspection involves a fixed countersinks workpiece, while the NVS twist vector $\vec {V}_{N}$ and the workpiece depth Z are known as a priori. Therefore, $\vec {\dot{U}}$ can be computed using the image jacobian $\textbf{J}$ as

Fig. 2

a The raw event stream and the constructed $H_{i}$ at 0.05 m/s, and b presents the raw event stream and $H_{i}$ at 0.5 m/s. Both IWEs are textured regardless of the NVS velocity

Fig. 3

a Motion compensated image $H_{i}$, b denoised image $H'_{i}$ using opening morphological transformation, and c identified countersinks for nut-plate installation holes during inspection using the mean-shift clustering principle. The left and right countersinks are highlighted in blue and green, respectively. The red points represent the pilot hole

$$\begin{aligned} \vec {\dot{U}} = \underbrace{\begin{bmatrix} \frac{-F}{Z} &{} 0 &{} u/Z &{} uv/F &{} -(F+u^2/F) &{} v \\ 0 &{} \frac{-F}{Z} &{} v/Z &{} F+v^2/F &{} -uv/F &{} -u \end{bmatrix}}_{\textbf{J}} \vec {V}_{N}, \end{aligned}$$

(11)

where $F$ is the focal length and [u, v] are the pixel coordinates. We constrain the NVS motion during the sweep to 2D linear motion reducing the expression in (11) to $\dot{u} = \frac{-F}{Z}v_{x}$ and $\dot{v} = \frac{-F}{Z}v_{y}$. Having $\vec {\dot{U}}$ expressed in closed form, $H_{i}$ is simply constructed at each timestep i. It is worth mentioning that in our approach, $\textbf{E}_{i}$ comprises of at least $N_{e} = 10000$ events to create a textured $H_{i}$ regardless of its time step size. Figure 2 shows the raw event stream along with the obtained $H_{i}$ at speeds of 0.05 and 0.5 m/s. Notice that the quality of the IWEs is maintained regardless of the sensor speed.

After constructing $H_{i}$, an opening morphological transformation (Sreedhar, 2012) provided by OpenCV library is applied to remove background noise, see Fig. 3. Opening morphology involves an erosion followed by a dilation applied to $H_{i}$ as

$$\begin{aligned} H'_{i} = H_{i} \circ B = (H_{i} \ominus B) \oplus B \end{aligned}$$

(12)

where B is a sliding kernel of size $3\times 3$, $\circ$ is the opening operator, $\ominus$, and $\oplus$ are the erosion and dilation operators, respectively. To illustrate, as the kernel B slides over the image $H_{i}$, the maximum value of all the pixels within the local neighborhood is assigned to the center pixel in the case of dilation and the minimum in the case of erosion. Consequently, the activated pixels of warped events in $e'$ in $H'_{i}$ are what we call the surface of active events (SAE). To identify the normal countersinks set present in $H'_{i}$, $e'$ is fed to unsupervised mean-shift clustering algorithm (Cariou et al., 2022) with a tunable bandwidth $\beta$ and a flat kernel $\mathcal {K}(e')$ defined by

$$\begin{aligned} \mathcal {K}(e') = \frac{1}{2} {\left\{ \begin{array}{ll} 1,&{} \text {if } -1 \le |G| \le 1\\ 0, &{} \text {otherwise,} \end{array}\right. } \end{aligned}$$

(13)

and

$$\begin{aligned} G = |\frac{e'-e_{k}}{\beta }|\text { }, \text {where } i = 1, 2, \cdots , N_{e}. \end{aligned}$$

(14)

Given $\mathcal {K}(e')$, $l_{1} \cup l_{2} \cup \cdots \cup l_{L}$ mutually exclusive clusters are obtained by optimizing the clustering parameters, as illustrated in Cariou et al. (2022). Since multiple countersinks are observed at time step i, the normal countersinks to be inspected correspond to the cluster $\mathbf {L^{*}}$ closest to the principal point expressed by

$$\begin{aligned} \mathbf {L^{*}} = {{\,\mathrm{arg\,min}\,}}_{L}\left| \left| \vec {W}_{l} - \begin{bmatrix} u_{0} \\ v_{0} \end{bmatrix} \right| \right| _{2}, \end{aligned}$$

(15)

where $\vec {W}_{l}$ are the centroids of the clustered activated events $e'_{l}$. Figure 3 shows the mean-shift clustering performance on a pilot hole and two nut-plate installation holes.

Depth estimation of countersink holes

After identifying the $l^{*}$ countersinks, their inner and outer radii need to be evaluated for depth estimation. Nevertheless, conventional circle detectors such as the circle Hough transform (CHT) and edge drawing-circles (ED-Circles) (Topal et al., 2011; Akinlar et al., 2012) are not applicable as an edge image is obtained with the absence of the gradient. Therefore, we devise a nonlinear robust least squares problem, described in Algorithm 1, to jointly cluster the set of events $e'_{l}$ to either edge of the countersink and optimize the fitting parameters $\Gamma$, comprising of the circle inner r and outer R radii, and its image point center h and k. Note that the fitting parameters are iteratively optimized given the same batch of events $e'_{l}$ until convergence as highlighted in Algorithm 1.

Given initial fitting parameters $\mathbf {\gamma }^{*} = [r, R, h, k]_{0}^{T}$, $e'_{l}$ is first transformed to polar coordinates $\vec {x}_{r} = || \vec {x'}_{l} - [h, k]_{0}^{T}||_{2}$, where $\vec {x'}_{l}$ are the spatial stamps of the countersink events set $e'_{l}$. Consequently, $\vec {x'}_{l}$ is clustered with hard data association by

$$\begin{aligned} \vec {x}^{i^{*}}_{r} = {{\,\mathrm{arg\,min}\,}}_{\vec {x}_{r}}|| \vec {x}_{r} - r ||_{2} \end{aligned}$$

(16)

Algorithm 1

Depth Estimation of Countersink Holes

where $\vec {x}^{i}_{r}$ are the activated events corresponding to the inner edge of the, while the rest of the events are otherwise. It is fundamentally important to highlight that the clustering in expression 16 is done at each iteration in optimization given the set of events $e'_{l}$. By jointly optimizing the clustering along with the fitting parameters, the expected likelihood of $e'_{l}$ probability density function is maximized, ensuring robust data association.

Finally, the residual vector $\vec {\xi }$ is formulated with the euclidean distance between the clustered $\vec {x}_{r}$ in 16 and $\gamma ^{*}$ by

$$\begin{aligned} \vec {\xi } = \begin{bmatrix} (\vec {X}^{i} - h_{0})^{2} + (\vec {Y}^{i} - k_{0})^{2} - r^{2} \\ (\vec {X}^{o} - h_{0})^{2} + (\vec {Y}^{o} - k_{0})^{2} - R^{2} \end{bmatrix}, \end{aligned}$$

(17)

where $\vec {X}^{i}$ and $\vec {Y}^{i}$ are the spatial stamps of $\vec {x}^{i}_{r}$, while $\vec {X}^{o}$ and $\vec {Y}^{o}$ are the spatial stamps of $\vec {x}^{o}_{r}$.

Utilizing ordinary least squares (OLS) to optimize $\Gamma$ can be affected by the presence of outliers. It is evident in Figs. 2 and 3 that the countersink circular edge is accompanied by Gaussian noise. While utilizing weighted least squares with a normally distributed weighting vector can alleviate this restriction, reflections also appear on the inner walls of the holes, degrading the aforementioned approach’s performance. Instead of minimizing the sum of the squared residuals, a Huber loss function (Gokcesu et al., 2021) is utilized to penalize the impact of outliers. The Huber loss $\alpha (\vec {\xi })$ is formulated as

$$\begin{aligned} \alpha (\vec {\xi }) = {\left\{ \begin{array}{ll} \vec {\xi }^{2},&{} \text {if } |\vec {\xi }| \le 1 \\ 2\vec {\xi } - 1, &{} \text {otherwise,} \end{array}\right. } \end{aligned}$$

(18)

and is fed to the Levenberg-Marquardt (LM) (Gavin, 2013) algorithm to optimize $\Gamma$ obtained as

$$\begin{aligned} \Gamma ^{*} = \begin{bmatrix} r \\ R \\ h \\ k \\ \end{bmatrix} = {{\,\mathrm{arg\,min}\,}}_{\Gamma }\sum _{l} \alpha (\vec {\xi }), \end{aligned}$$

(19)

where the optimization is initialized with initial condition $\gamma ^{*}$ parameterized by the mean of the inner $r_{0}$ and outer radii $R_{0}$, and the centroid of $\vec {x}_{l}$. Figure 4 shows the circle fitting using Huber loss and is compared to fitting using OLS on nut-plate installation holes. Notice that the fitting is robust against the noise and reflection outliers, unlike the linear loss of OLS.

Fig. 4

a Motion compensated image $H'_{i}$, b circle fitting using ordinary least squares, and c robust fitting by Algorithm 1. The red and green circles represent the inner and outer edges of the. The blue circle is the pilot hole fitted using OLS

Fig. 5

a Front and b isometric views of the. The depth $Z'$ can be derived by trigonometry, given $r_{m}$, $R_{m}$, and $\phi$

Fig. 6

Experimental setup for validating the proposed inspection framework. A KUKA robotic manipulator is utilized for moving the neuromorphic camera to inspect a countersinks workpiece representing large industrial parts. A fiducial marker is installed on the workpiece for the normal alignment of the sensor with the workpiece

Having the inner and outer radii, a known angle $\phi$, and Z, the depth $Z'$ is estimated by simple trigonometry. Figure 5 presents front and isometric views of the s for a visual illustration of the trigonometric relation. First, the inner and outer radii are converted from pixels to mm by

$$\begin{aligned} \begin{bmatrix} r_{m} \\ R_{m} \end{bmatrix} = \begin{bmatrix} \frac{Z}{F} &{} 0 \\ 0 &{} \frac{Z}{F} \end{bmatrix} \begin{bmatrix} r \\ R \end{bmatrix} \end{aligned}$$

(20)

where $r_{m}$ and $R_{m}$ are the radii expressed in mm. Intuitively, $Z'$ can be inferred from Fig. 5 and is expressed by

$$\begin{aligned} Z' = \frac{R_{m}- r_{m}}{\tan \frac{\phi }{2}}. \end{aligned}$$

(21)

Fig. 7

Inspected workpieces with countersinks of varying depth specifications along with the inspected specimens by our method. a–c are the workpiece IDs ordered in descending order with respect to the nominal countersink depths

Experimental validation and results

Experimental setup

We have tested our framework in a variety of challenging scenarios. Figure 6 shows the designed experimental setup to validate the proposed approach. A 7-DOF KUKA KR30 L16 with KL-1000-2 linear track was utilized to move the NVS during the inspection, and a workpiece resembling large indsutrial structures with inspection requirement. The robot end-effector comprises of IDS UI-5860CP ($1920 \times 1080$) as the frame-based sensor for the normal alignment clarified in “Robotic inspection setup” section, while the DVXplorer ($640 \times 480$) as the NVS. It is worth noting that only the NVS was employed during inspection and the experimental results are reported for neuromorphic vision-based countersink inspection. The frame-based camera was utilized to perform the initial alignment of the end-effector with the workpiece and for benchmarking purposes to demonstrate limitations of conventional frame-based imaging for high-speed inspection. Finally, Intel NUC5i5RYH was also mounted on the end-effector for onboard processing, where both imaging sensors were interfaced with the robot operating system (ROS) using a USB 3.0 terminal. Four important aspects of the experimental protocol need to be taken into consideration. First, the NVS motion during the inspection ranged from 0.05 m/s to 0.5 m/s to justify the purpose of this paper. Second, the NVS moved in 2D linear motion as it sweeps the countersinks workpiece, as discussed in “Event-based detection of countersinks” section. Third, we performed the proposed inspection method on three workpiece variants of different countersink sizes to show that the work of this paper is generalizable to different dimensions of countersink specimens, see Fig. 7. Finally, the standoff between the NVS and the workpiece during the sweep was 90mm to maintain high inspection precision. Videos of the experiments are available through the following link https://www.dropbox.com/s/pateqqwh4d605t3/final_video_new.mp4?dl=0.

Fig. 8

a NVS motion trajectory during the inspection, b and c sharp IWEs and IDS frames suffering from motion blur. The reconstructed IWEs justify the purpose of the NVS for high-speed inspection

Fig. 9

Qualitative comparison of the d proposed detector with a artificial event frames accumulated at $\Delta t=33ms$, b conventional frames from the IDS camera, and c E2VID reconstructed grayscale images. Notice that the features in the artificial event frames are perturbed by motion blur and noise, while features IDS images are degraded by motion blur even. On the other hand, E2VID provides textured image reconstruction of the scene, but ED-Circles fails due to reflections. Our detector, on the other hand, shows improved performance at high speeds with robustness against noise, reflection outliers, and motion blur

The experimental validation of the proposed work revolves around three folds. In “Countersink detection performance” section, we compare the war**-Huber-based circle detection performance in terms of detection success rate and the IWEs image quality. We also compare the aforementioned detector to ED-Circles on frames from IDS camera and E2VID (Rebecq et al., 2019) network for event-based image reconstruction. In “Countersink depth estimation results” section, we validate our inspection pipeline by computing by computing the standard deviation of the estimated depths on 10 runs at each NVS speed. In addition to standard deviation, the countersink depth estimations were compared to the nominal countersink depths provided by a calibrated countersink gauge in terms of the mean absolute error (MAE). Finally, “Benchmarks” section benchmarks the work of this paper to state-of-the-art methods for inspection in terms of precision, accuracy and inspection execution time.

Fig. 10

a–e Shows the precision of the proposed robust circle detection and depth estimation algorithms at 0.5 m/s. First row shows a zoomed in view to the inspected countersinks in the motion compensated image, while the second row shows the estimated depths. Notice the precise measurements even though the NVS resolution is relatively low

Countersink detection performance

The quality of the reconstructed IWEs directly impacts the detection performance. Figure 8 compares the obtained neuromorphic IWEs by war** against conventional images for a sweep speed of 0.5 m/s. If artificial event frames are formulated by simple accumulation at fixed time intervals, motion blur is witnessed, and the detection performance is highly influenced by sensor noise, see Fig. 9. Given the improved quality of IWEs, the mean shift clustering algorithm for detection is evaluated based on its detection success rate, as a detection rate of 100% was obtained for speeds up to 0.5 m/s.

We also provide a qualitative comparison of the IWEs and the proposed circle detector in “Depth estimation of countersink holes” section against ED-Circles applied on IDS conventional frames and reconstructed images from the E2VID (Rebecq et al., 2019) network, illustrated in Fig. 9. At 0.05 m/s, ED-Circles on IDS images fail to detect the circular edges of the countersinks due to motion blur, while detection was successful on E2VID network. This is because the network is characterized by its capability of reconstructing deblurred images of the scene. However, ED-Circles fail to detect the holes due to the inner wall reflections reducing the gradient of the circular edges at 0.5 m/s. In addition, E2VID requires a powerful GPU to run online, which is difficult to install on onboard processors. Contrary to the aforementioned approaches, the proposed detection pipeline runs on a low-end Intel Core i5-5250U CPU, leverages the asynchronous nature of neuromorphic events, and demonstrates robustness against motion blur and outliers.

Table 1 Evaluation of countersink depth estimation in terms of standard deviation and its accuracy validated against the ground truth countersink depths by the MAE

Table 2 Standard deviations $\sigma _{d}$ of the countersink inspection system in mm for 10 trials at each NVS speed for workpiece with ID ’A’, where a standard deviation $\sigma _{a}$ of 0.0275 mm is achieved

Table 3 Standard deviations $\sigma _{d}$ of the inspected depths of countersinks in mm for 10 trials at each NVS speed for workpiece with ID ’B’, where a standard deviation $\sigma _{a}$ of 0.0251 mm is achieved

Table 4 Standard deviations $\sigma _{d}$ of the inspected depths of the countersinks in mm for 10 trials at each NVS speed for workpiece with ID ’C’, where a standard deviation $\sigma _{a}$ of 0.0240 mm is achieved

Countersink depth estimation results

While the proposed approach provides the required detection success rates, the repeatability of the countersink depth estimation needs to be evaluated. We first evaluate the repeatability of the inspection measurements $Z'_{k}$ by computing $\sigma _{d}$, where $\sigma _{d}$ is the standard deviation of the inspected countersink depths of each hole over all N inspection trials at each distinct NVS speed as

$$\begin{aligned} \sigma _{d} = \sqrt{\frac{\sum _{n}(Z'_{n} - \mu _{z})}{N}}, \forall \text { } n \in [1,N] \end{aligned}$$

(22)

where $\mu _{z}$ is the mean of the inspected depths along all the $N=10$ trials. Table 1 shows the calculation $\sigma _{d}$ for a countersink sample from each workpiece at a distinct NVS speed of 0.5 m/s for 10 trials, and Fig. 10 visualizes these estimations. After obtaining $\sigma _{d}$, we evaluate

$$\begin{aligned} \sigma _{r} = \sqrt{\frac{\sum _{k}\sigma _{d}}{K}}, \forall \text { } k \in [1,K] \end{aligned}$$

(23)

as the aggregate standard deviation for each inspected hole at all $K=5$ different NVS speeds ranging from 0.05 m/s to 0.5 m/s. Consequently, the obtained $\sigma _{r}$ are used to compute the aggregate standard deviation $\sigma _{a} = \sqrt{\frac{\sum _{m}\sigma _{r}}{M}}$, where $m \in [1,M]$. Tables 2, 3, 4 show the obtained $\sigma _{r}$ and $\sigma _{a}$ standard deviations on each workpiece. Finally, the standard deviation of our whole inspection pipeline is similarly found by aggregating the inspection standard deviations on each workpiece, where our inspection pipeline provides a standard deviation of 0.025 mm.

We also evaluate the accuracy of the proposed inspection pipeline by comparing the depth estimates to the ground truth provided by the countersink guage in terms of the MAE. Similar to the standard deviation computation, we find the MAE for each hole inspected 10 times at each designated NVS speed, as reported in Table 1. Consequently, all the MAE’s are aggregated for each workpiece, where our inspection pipeline provides an inspection accuracy of 0.026 mm abiding by the industrial requirements of various automated industrial production lines. The inspection accuracy for all inspected countersink workpiece variants is illustrates in Tables 5, 6, and 7.

Notice that the MAE and standard deviation for the estimated countersink depths are lower for specimens with lower depths. This is because the motion compensation performance tends to be improved as the depth of the inner countersink walls is closer to the utilized depth for war** the events, as discussed in “Event-based detection of countersinks” section. In all the cases however, the proposed inspection framework achieves a standard deviation of 0.025 mm and MAE of 0.026 mm, abiding by the industrial requirements of various assembly lines. The reason behind such repeatability of the proposed circle detector, even on a low-resolution sensor, is that it relies on an optimization approach fitting the best candidate circle to the observed hole edges. The high dependence of conventional circle detectors on the image gradient degrades their performance even on high-resolution cameras. Assuming that the conventional imaging sensor is static during the inspection and no motion blur is witnessed, the lighting conditions of the inspection setup have a major influence in controlling the accuracy of these circle detectors. In contrast, the high dynamic range of the NVS makes our method robust against illumination variations.

Benchmarks

To evaluate the impact of our event-driven approach, we compare the obtained results against state-of-the-art vision-based methods for inspection, which comprise of Haldiman (2015) inspection using Digital Light Processing (DLP) projector and Sa et al. (2018). These works are compared quantitatively in terms of the depth estimation standard deviation inspection speed, where the comparison is more elaborated in Table 8.

Table 5 Mean absolute error (MAE) of the inspected depths of the countersinks in mm for 10 trials at each NVS speed for workpiece with ID ’A’, where an accuracy of 0.0274 mm is achieved

Table 6 Mean absolute error (MAE) of the inspected depths of the countersinks in mm for 10 trials at each NVS speed for workpiece with ID ’B’, where an accuracy of 0.0263 mm is achieved

Table 7 Mean absolute error (MAE) of the inspected depths of the countersinks in mm for 10 trials at each NVS speed for workpiece with ID ’C’, where an accuracy of 0.0244 mm is achieved

Table 8 Quantitative comparison of the proposed framework against state-of-the-art works for inspection using monocular vision

Our approach shows comparable results with the same orders of magnitude compared to previous works. Compared to Haldiman (2015) inspection approach, our work provides improved inspection precision with an added advantage of alleviating the need of an extra active light projector, which is currently unfavorable in industries for safety precautions, especially with the introduction of human-robot interaction in assembly lines. Long et al. (2019), on the other hand, slightly outperforms our method by 0.0125 mm. Nevertheless, our inspection method provides high-speed inspection and meets current industrial standards. Furthermore, the resolution of their utilized sensor is much higher than the DVXplorer by a factor of more than 1/16. With the current advancements in neuromorphic sensor models, high-definition neuromorphic cameras easily circumvent this limitation and can provide improved standard deviation. Note that we have not provided a comparison for the accuracy as it was not quantified by the aforementioned methods.

Table 9 Sweep time at different NVS speeds for a 1.5 m countersinks workpiece

Table 10 Execution times for the system blocks along with total execution time of the inspection task on 10 countersinks with a sweep speed of 0.5 m/s

Fig. 11

ED-Circles applied on a IDS images. b Reliable detection of countersinks requires the robot to pause at each hole due to high latency and motion blur

In addition, our work provides the advantage of the swift execution time of the inspection task, with the capability of the NVS swee** the workpiece at a high speed of 0.5 m/s. Table 9 reports the sweep time at different NVS speeds and illustrates that the 60 countersinks workpiece is scanned in nearly 3 s. Previous methods, in contrast, require the robot to pause on each countersink to perform the inspection before proceeding to the next hole. We quantify this by inspecting the holes using the IDS frames, and hole detection was carried out using ED-Circles and brute-force matching for outlier removal, in which these experiments mimic the works of (Long et al., 2019). This is illustrated in Fig. 11. During the inspection, the IDS camera paused on each hole for 0.5 s to carry on the inspection algorithm, and the motion between holes takes nearly 0.2 s. In total, the inspection task takes almost 42 s. It is worth highlighting that such improvement is facilitated by the NVS robustness to motion blur, but such a high-speed inspection framework is only attained by the proposed countersink detection and inspection algorithms, which laverage the high temporal resolution of the neuromorphic camera. Thus, one important aspect of our inspection pipeline is the computational time. Table 10 reports the total algorithm time for the proposed inspection pipeline added to the total sweep time. As 0.033 s are required to inspect each countersink, the total execution time of the neuromorphic vision-based inspection task is 4.98 s, reducing the inspection time by nearly a factor of 10. This also shows that the proposed work can be utilized to run online. It is worth mentioning that our algorithm was written in Python, and real-time capabilities can be achieved if written in C++.

Conclusions and future work

In this paper, and for the first time, high-speed neuromorphic vision-based countersink inspection was proposed for automated assembly lines. The proposed approach leverages the neuromorphic vision sensor’s unprecedented asynchronous operation to create sharp, deblurred images through motion compensation. Due to the NVS unorthodox nature, a robust circle detection algorithm was devised to detect the countersinks in the IWEs. To prove the feasibility of neuromorphic vision for high-speed inspection, we have tested this paper’s work in various experiments for a low-resolution NVS reaching speeds up to 0.5 m/s. The high quality of the IWEs and the proposed circle detector played a major role in determining the repeatability of the proposed system. The experimental results show that the inspection task is accelerated by a factor of 10$\times$ while providing a standard deviation of 0.025 mm and an accuracy of 0.0206 mm. For future work, we aim to develop an event-based fiducial marker detection algorithm to utilize the NVS solely for the whole inspection procedure, starting from the initial alignment of the sensor with the inspected workpiece to its high-speed sweep. On the other hand, while the introduced motion compensation-circle detection frameworks were utilized for inspection, these algorithms will be further deployed for robotic positioning systems and visual servoing in automated manufacturing processes such as precision drilling, deburring, and various other robotic manufacturing opportunities.