Application of Machine Learning to GPU Optimization, Deep Q-Networks and Computational Fluid Dynamics

Throughout society today, machine learning has been catapulted to a transformative problem solving approach across various domains, ranging from natural language processing to computer vision to engineering optimization. The fundamental principle is the ability of algorithms to learn patterns and make decisions based on data, rather than relying on explicitly programmed instructions. This dissertation addresses the research question: “How can machine learning techniques be applied to improve computational efficiency and prediction accuracy in high-performance scientific computing tasks, including GPU kernel optimization, Deep Q-Networks, and computational fluid dynamics?” To answer the question, we devised three distinct problems, each of which is orthogonal to the next to represent a wide breadth of exploration. The problems focus on the two paradigms of supervised learning and reinforcement learning.

In the first problem, we propose a framework that leverages GPU hardware performance counters and a suite of machine learning models, including a deep neural network, a random forest, and a naive Bayes classifier, to automatically characterize and classify GPU kernels using supervised learning. This framework identifies performance bottlenecks and recommends targeted optimization strategies, thereby reducing the reliance on manual tuning and improving resource utilization. Experimental evaluations on stencil computations and sparse matrix operations demonstrate the effectiveness of our approach in enhancing GPU performance.

The second problem introduces an accelerated Deep Q-Network (DQN) algorithm that reformulates the experience replay mechanism in reinforcement learning. By storing (state, action, target) tuples instead of (state, action, reward) tuples, our method eliminates redundant recomputation of target values and minimizes the use of computationally expensive max() operations. This refinement results in faster convergence, as evidenced by a reduction of up to 35% in training time and a decrease in the number of episodes required for convergence in benchmark environments such as MountainCar, Acrobot and Cartpole. A detailed mathematical analysis further justifies the reduced overestimation error inherent in our approach.

The final problem applies supervised learning to computational fluid dynamics (CFD) by developing MoE-Bolt, a novel architecture that combines three-dimensional convolutional neural networks with a gated mixture-of-experts (MoE) framework. Designed for predicting unsteady flow fields in lattice Boltzmann simulations, MoE-Bolt formulates the problem as a sequence prediction, in which a series of previous time steps is used to forecast future velocity fields. By automatically decomposing the computational domain among multiple expert sub-networks, our approach achieves a one-order-of-magnitude improvement in peak signal-to-noise ratio and enhanced structural similarity compared to single-expert models, even for flows at Reynolds numbers outside the training range.

Collectively, these problems demonstrate that the integration of machine learning techniques into GPU optimization, deep reinforcement learning, and CFD can substantially improve computational performance, reduce manual intervention, and improve prediction accuracy. The findings not only validate the proposed methodologies, but also emphasize the potential of data-driven approaches to address complex challenges across diverse domains in scientific computing.