SENSOR FUSION IN NEURAL NETWORKS FOR OBJECT DETECTION

Object detection is an increasingly popular tool used in many fields, especially in the
development of autonomous vehicles. The task of object detections involves the localization
of objects in an image, constructing a bounding box to determine the presence and loca-
tion of the object, and classifying each object into its appropriate class. Object detection
applications are commonly implemented using convolutional neural networks along with the
construction of feature pyramid networks to extract data.
Another commonly used technique in the automotive industry is sensor fusion. Each
automotive sensor – camera, radar, and lidar – have their own advantages and disadvantages.
Fusing two or more sensors together and using the combined information is a popular method
of balancing the strengths and weakness of each independent sensor. Together, using sensor
fusion within an object detection network has been found to be an effective method of
obtaining accurate models. Accurate detections and classifications of images is a vital step
in the development of autonomous vehicles or self-driving cars.
Many studies have proposed methods to improve neural networks or object detection
networks. Some of these techniques involve data augmentation and hyperparameter opti-
mization. This thesis achieves the goal of improving a camera and radar fusion network by
implementing various techniques within these areas. Additionally, a novel idea of integrating
a third sensor, the lidar, into an existing camera and radar fusion network is explored in this
research work.
The models were trained on the Nuscenes dataset, one of the biggest automotive datasets
available today. Using the concepts of augmentation, hyperparameter optimization, sensor
fusion, and annotation filters, the CRF-Net was trained to achieve an accuracy score that
was 69.13% higher than the baseline