Introduction

Rapid development of hand-held mobile mapping systems [53] incorporating recent advances in SLAM [59] [63] allows the creation of both accurate and precise 3D point cloud and the trajectory. This trajectory can be considered as ground truth looking from VO and SLAM applications [61] [41]. An alternative approach could be a ground truth reference system using robust visual encoded targets [35]. The limitation of existing approaches is related either to the high complexity of the mobile mapping system [58] or to the overall cost of the solution. For these reasons, we proposed a novel open-hardware mobile mapping system:

   • hand-held,
   • weighs less than 1kg,
   • LiDAR range up to 40m,
   • affordable (open source, open hardware MIT license),
   • can be used indoor and outdoor,
   • up to 5 hours of continuous scanning with suggested velocity up to 10km/h,
   • equipped with the handler for a smartphone.

The contribution of this paper is also related to open-source software that provides an alternative approach to, e.g. g2o [28], gtsam [27], manif [17], ceres [2]. Our aim was to minimize the effort needed for software installation and interoperability between different operating systems. The contribution is:

   • camera to LiDAR calibration,
   • LiDAR odometry,
   • single session alignment,
   • multiple session alignment with georeferencing.

Benchmarking [22] is very important for providing advantages and disadvantages of the current state of the art. Introducing ATE (Absolute Trajectory Error) and RPE (Relative Pose Error) errors by [49] had a great impact. Recent Hilti-Oxford Dataset [64] provides an interesting approach for LiDAR and Visual SLAM benchmarking. It has been collected on construction sites as well as at the famous Sheldonian Theatre in Oxford, providing a large range of difficult problems for SLAM.

Ground truth data sources are crucial when looking from quantitative and qualitative benchmarking points of view. The dominant one in literature is Visual Odometry-SLAM Evaluation KITTI [24] that directs the mainstream of many types of research. The second great example is the Hilti-Oxford dataset [64]: a millimeter-accurate benchmark for SLAM. Most of the recent papers incorporate KITTI or/and Hilti datasets for evaluation purposes. It is an evident rapid growth of more and more sophisticated benchmarks for SLAM applications [46]. Such an approach, unfortunately, introduces an important gap between benchmark and real-live applications. In our paper, we support this observation by showing that taking state-of-the art Visual SLAM approach and performing mobile mapping in, e.g. a typical office environment, is not straightforward. Chosen algorithms: StellaSLAM [50], ORBSLAM3 [16] and DSO [20] [57] have great performance on widely used and accepted by academia benchmarks [22], [15]. Most of these datasets contain data derived from professional cameras such as high fps global shutter and IMU. Such data is typically tested in the mobile robotics domain. Unfortunately, research in the area of smartphones requires a more flexible approach since new devices appear each year. The SLAM technology transition is rather challenging since most of the approaches are not prepared for it. Especially, large resolution cameras with rolling shutters are still an open research topic [43] [44] [30].

A rationale behind our research is the fact that we are demonstrating the bias of potential use of existing benchmarks. In our case, state-of-the-art algorithms do not solve our real-world task. We have chosen rather easy cases – mapping of small indoor scenes. It means that the proposed methodology can have a positive impact on the research community since the required effort for ground truth data collection can be drastically decreased. The content of this paper is as follows:

   • First, we analyze the existing ground truth systems.
   • Second, we introduce open hardware and methodology for 3D data collection.
   • Third, we show ground truth accuracy assessment.
   • Fourth, we show visual SLAM accuracy assessment using state-of-the-art StellaSLAM, ORBSLAM3, and DSO.
   • Finally, we conclude the paper with suggestions and discuss future direction.

Ground truth systems

Rapid growth of indoor ground truth data interest is evident [62], since it provides useful data for qualitative and quantitative measures for SLAM applications. Cost-effective camera-based ground truth for indoor localization is very helpful [10] in performing preliminary tests. For outdoor, typical approaches are a global positioning system [48] and total stations [54]. Obtaining ground truth is a rather sophisticated procedure than easy technology to use. Moreover, deployment of ground truth technology is not always possible, especially in extreme environments [1], e.g. an analysis of SLAM-based LiDAR data quality metrics for geotechnical underground monitoring [23] is very important from the safety point of view. It is related with the work on affordable low-cost handheld LiDAR-based SLAM systems [52]. Further important limitation of the ground truth data source is its accuracy and precision [60]. In most of SLAM applications [19] we do not need a millimeter accuracy to conduct qualitative and quantitative evaluation. Hence, a centimeter level accuracy is sufficient and it can be obtained with many existing technologies such as Terrestrial Laser Scanner [41], [18]. Such accuracy is also provided by GPS, GNSS receivers with Real Time Kinematics (RTK) [13] which is the most popular technique to collect ground truth data in open-sky outdoor environments [14]. We distinguish following satellite positioning systems: Global Navigation Satellite Systems (GNSS), GPS (Global Positioning System, United States), GLONASS (Global Navigation Satellite System, Russian Federation), Galileo (European Global Navigation Satellite Systems Agency (GSA)), BeiDou (approximately translated to “Northern Dipper”, People’s Republic of China), IRNSS (Indian Regional Navigation Satellite System, India) and QZSS (Quasi-Zenith Satellite System, Japan). It is also possible to process such data with PPP (Precise Point Positioning) [51]. PPP relies on carrier-phase measurements as the primary observable to model or estimate effects for centimeter-level resolution. We can incorporate SLAM techniques to improve this data even assuming continent-scale [11], thus optimizing multiple trajectories decreases overall data uncertainty and increases the accuracy.

Mobile mapping systems incorporates LiDAR technology for 3D measurements [32]. Multi beam technology with non-repetitive scanning patterns [38] is recently investigated since this functionality provides 99% coverage of the surrounding environment even without motion [34]. Affordable non-repetitive scanning pattern mobile mapping systems are currently state-of-the-art for building low-cost ground truth systems. It might be possible that solid-state LiDARs [33] will provide even more affordable applications. There are plenty of commercial ground truth systems on the market since benchmarking is crucial both from academic and industrial points of view. Table 1 shows some of the ground truth systems that are extensively used by many researchers [15], [45], [39], [41], [40]. It can be seen that the cost is multiple times larger than our approach, but those systems provide much better accuracy and precision.

Open-hardware for 3D data collection

Figure 1 shows the open-hardware hand-held mobile mapping system. Technical details are available at (https://github.com/JanuszBedkowski/mandeye_controller). The open source project for off-line 3D data processing is available at (https://github.com/MapsHD/HDMapping). It is compatible with ROS (Robot Operating System). The system is composed of LiDAR Livox Mid-360. It is capable of collecting data on USB flash memory. It is equipped with an onboard RaspberryPi4B computer. It can work for more than five hours and the weight is around 1kg.

Ground truth data processing is composed of four modules:

   • Camera to LiDAR calibration,
   • LiDAR odometry,
   • single session refinement,
   • multiple sessions refinement with georeferencing.

Camera to LiDAR calibration is implemented based on fundamental paradigm in computer vision [25] – reprojection error. LiDAR 3D point - image pixel pairs form an optimization problem that calculates extrinsic parameters ([R, t]LiDAR←camera) of the system. LiDAR odometry is composed of highly coupled multiple view normal distributions transform with pose graph SLAM incorporated for preserving motion model derived from IMU processed with Madgwick Orientation Filter [36], [29]. Each consecutive batch of 20 poses is processed within the assumption of the sliding window. This approach differs from classic pair-wise matching with pose graph SLAM [55]. Thus, in our approach, relative pose constraints (20 consecutive poses) are highly coupled with multi-view normal distributions transformed as a single optimization routine. Once the initial trajectory is calculated, it is possible to perform consistency procedure that makes the trajectory smooth.

Camera LiDAR synchronization

Both devices used in the setup have different clocks which requires synchronization between them. There are at least two ways in which we can address the synchronization problem:

1: Synchronize system clocks of both devices and use those system clocks as basis for time stamping sensor data. This is the simplest solution which can use existing clock synchronization methods and protocols, such as NTP (network time protocol) and PTP (precision time protocol). The former is available out-of-the-box in many platforms, such as Android itself (in fact, Android uses NTP to synchronize system clock over the internet). On Unix-like operating systems, NTP daemons are also easily obtainable and configurable. The precision of NTP synchronization varies and is dependent on network architecture. Generally, one can expect synchronization accuracy below 100 ms (less in small LAN networks) (https://www.ion.org/publications/abstract.cfm?articleID=14186, https://www.ntp.org/ntpfaq/NTP-s-algo/#5131-how-accurate-will-my-clock-be). PTP is a much more accurate solution, easily guaranteeing microsecond accuracy, however, it requires more effort in the setup and puts more requirements on network adapters (such as software or hardware timestamping). As this protocol is not available out-of-the-box on Android, we did not consider it further.

2: Synchronize data timestamps based on IMU data. Most API used for fetching data (such as Android Camera2 or Motion sensors API) do not provide timestamps associated with system clock, but rather some internal monotonic clock (e.g. time since booting up the system). When one uses raw timestamps provided by the device, it is pointless to use time synchronization protocols, as those affect system clock. In such cases, direct data synchronization is required. In our experiments, we found that best results are achieved through IMU data synchronization (especially via acceleration). This method can achieve decent results due to high frequency and low latency of IMU data. It is, however, challenging to automate and it requires good IMU data so that synchronization based on movement patterns is possible. In our setup, we opted to use NTP-based synchronization which allowed us to achieve synchronization accuracy below 50 ms. With average speeds not exceeding 0.6 m/s this means that inaccuracy due to synchronization is below 3 cm, which is enough for datasets presented here. NTP-based synchronization was chosen as it was the simplest method to set up and automate.

Camera to LiDAR calibration

We use built-in Camera2 API to get intrinsic parameters for the camera. It is a deliberate choice over manual calibration. Our goal is to test the algorithms on out-of-the-box devices, such as in commercial applications, where user is not supposed to calibrate the device. Relative camera to LiDAR pose estimation is addressed by solving two sub problems: first, a feature matching problem that seeks to establish putative 2D-3D correspondences, and then a Perspective-n-Point problem that minimizes, with reference to the camera pose, the sum of so-called Reprojection Errors (RE). Feature matching problem is solved by a manual procedure, end-user should find at least 5 pairs 2D (camera image) - 3D (3D point cloud) correspondences. These pairs form the Perspective-n-Point problem [56] that is minimized with Gauss-Newton optimization routine.

Normal Distributions Transform

Normal Distributions Transform [37] is an alternative technique to Iterative Closest Point [12], [26] for point cloud data registration and it is available in a well-known Point Cloud Library [42] open source project. It is limited to the pairwise matching of two-point clouds, thus a contribution of the proposed research is a novel approach to NDT enabling fusing it with pose graph SLAM. The key element of the NDT is the representation of the data as a set of normal distributions organized in the regular grid over 3D space. These distributions describe the probability of finding a 3D point at a certain position. The advantage of the method is that it gives a smooth representation of the point cloud, with continuous first and second-order derivatives. Thus, standard optimization techniques described in this paper can be applied. Another advantage of NDT over ICP is its much less computational complexity since the consumptive nearest neighborhood search procedure is not needed. Authors of [9] also elaborate on this advantage. The 3D space decomposition into the regular grid introduces some minor artefacts, but in a presented experiment it is a negligibly small disadvantage. For each bucket from a regular grid containing a sufficient number of measured points, NDT calculates the mean given by the equation (1) and the covariance given by the equation (2).

The likelihood of having measured point P^g_m is given by equation (3).

Each p(P^g_m) can be seen as an approximation of the local surface within the range of the bucket. It describes the position μ of the surface as well as its orientation and smoothness given by Σ.
Let Ψ([R, t]3×4 W←LiDAR,P^l_m) will be a transformation function of the local measurement point [P^l_m, 1]⊺ via pose [R, t]3×4W←LiDAR expressed as (4).

Thus, the NDT optimization problem is defined as the maximization of the likelihood function given in equation (5).

Furthermore, the optimization problem is equivalent to the minimization of the negative log-likelihood given in equation (6).

NDT implementation similar to ICP uses point-to-point observation equation. The only difference is that an information matrix Ω is calculated as an inverse of the covariance matrix from equation (2). The disadvantage of multi-view NDT is the fact that narrow obstacles such as walls observed from neighboring rooms can converge to a single entity (the width of the wall should not converge to 0). To discriminate obstacles we remove such observations that correspond to different viewpoints. It means that the flat surface of one room is not converging to this flat surface observed from the neighboring room. It is implemented as a normal vector geometric check.

Single session refinement

Classic pose graph SLAM [28] is incorporated to optimize manually chosen pairwise matches. Thus, this semi-automatic process uses loop closure edges chosen by end user to optimize the graph composed of trajectory edges, loop closure edges and motion model. The result is a reduced consecutive error of the LiDAR odometry.

Multiple session refinement with georeferencing

Multiple trajectories can be organized into the project. Only one trajectory can be treated as ground truth (obtained e.g. with geodetic survey). Other trajectories will be aligned together and to ground truth based on loop closure edges.

Ground truth accuracy assessment

We performed a quantitative comparison to ground truth data – underground INDOOR scenario 20x90 [m]. It is shown in Figure 2. Ground truth data was collected with terrestrial laser scanner survey TLS Z+F IMAGER 5010 that provides point cloud with the millimeter accuracy and precision [47]. We observed that the maximum vertical deviation is less than 10cm and the maximum horizontal deviation is 3cm. This is a satisfactory result that is sufficient, moreover, further investigation on global accuracy is not our main focus since our LiDAR provides 2cm accuracy on a 20m distance (documentation is available here https://www.livoxtech.com/mid-360).

Visual SLAM accuracy assessment

For the demonstration of the functionality of the proposed affordable ground truth data system, we performed data collection for several INDOOR scenarios. We evaluated DSO https://github.com/JakobEngel/dso [20] [21], OpenVSLAM https://github.com/stella-cv/stella_vslam [50] that is inspired by ORB SLAM https://github.com/UZ-SLAMLab/ORB_SLAM3 [16]. We selected these three open-source visual SLAM implementations since they can be implemented in smartphones with minimal effort. Real-world datasets were recorded using SAMSUNG GALAXY S23 SM-S911B device with ISOCELL GN3 (S5KGN3) electronic rolling shutter sensor locked at 30 FPS as a sequence of corresponding YUV-420-888 images (separate image per YUV component stream) further combined into 8 bit BGR images which after undistortion create a final dataset. Camera characteristics, like sensor intrinsic parameters and lens distortion coefficients, were retrieved from Android camera API and downscaled from native camera resolution to resolution at which the sequence was recorded and processed using state-of-the-art SLAM frameworks. Before dataset recording started the camera was configured so that chromatic aberration correction, distortion correction, auto white balance, auto-focus, and color effects were disabled. Table 2 collects all ATE (Absolute Trajectory Error) results according to methodology from [49]. This quantitative measure compares trajectory to ground truth. It can be seen that DSO performs best, but there is an issue with this statement since Figure 5 shows the failure of all methods. An interesting observation is that OpenVSLAM (current name StallaVSLAM) performs better than its successor ORB SLAM3, so it is beneficial to refactor existing implementations (authors of [50] mainly refactored the work of [16]). This very simple experiment provides significant observations:

   • state-of-the-art visual SLAM algorithms are not an out-of-the-box solution for smartphone cameras without IMU,
   • ground truth system provides sufficient data for quantitative and qualitative benchmark,
   • simple methodology incorporating ATE is not sufficient for correct research statements (it was also observed in [31]).

Conclusion

This paper shows how to build a system for ground truth data collection for machine vision, robotics and other mobile mapping applications. It can be used for qualitative and quantitative SLAM evaluation. This research drastically reduces the cost of benchmark data generation, thus many researchers, instead of focusing mainly on available datasets, will generate new ones. Such an approach removes the existing limitation related to existing benchmarks typically focused on high-end sensors. For this reason, we proposed a novel and affordable ground-truth system that provides an accurate and precise trajectory with a point cloud. It is based on LiDAR Livox Mid-360 with a non-repetitive scanning pattern with affordable IMU, on-board Raspberry Pi 4B computer, battery and software for off-line calculations (LiDAR odometry, SLAM). The software is based on an alternative approach, e.g. g2o, GTSAM, manif or Ceres, since this lightweight implementation does not require any installation on Linux or Windows. This software is dedicated also to non-programmers. We have shown how this system can be used for the evaluation of various state-of-the-art algorithms (Stella SLAM, ORB SLAM3, DSO). An open-hardware measurement device specification is available at https://github.com/JanuszBedkowski/mandeye_controller. We hope this research will boost machine vision experiments since the proposed solution provides ground truth for almost all scenarios. The project has a rapidly growing community and it addresses the most significant issues with ground truth: cost-effectiveness, scale, ergonomic design, simplicity and interoperability. The accuracy in typical indoor scenarios does not exceed 5cm and the precision 3cm. It is sufficient for qualitative and quantitative SLAM evaluation which was demonstrated in this paper.

Link to the paper

https://attachments.waset.org/24/papers/24CA090515[1].pdf

View More

Close