<b>NEUTRON SCATTERING INVESTIGATIONS OF CRITICAL QUANTUM SENSING MATERIALS</b>

<p dir="ltr">This dissertation presents neutron scattering investigations of three classes of quantum materials: frustrated rare-earth magnets, Kitaev spin liquid systems, and topological insulators. Each system was studied using tailored neutron techniques to characterize magnetic ordering, spin dynamics, and spin-polarized transport interactions, with the aim of revealing emergent phenomena relevant to quantum criticality and materials-enabled sensing.</p><p dir="ltr">The first section presents a bidirectional polarized neutron scattering experiment on the three-dimensional topological insulator Bi₂Se₃. Spherical neutron polarimetry device installed at HB-1 beamline at HFIR, ORNL was used to detect changes in neutron beam polarization in the presence of an applied electrical current across the Bi₂Se₃ crystal. The data shows a measurable rotation in the neutron spin orientation correlated with current direction, which is temperature dependent. On the other hand, Hall measurement in the crystal shows observable change in Hall angle when the polarized neutron beam is transmitted through it. Careful control of thermal, magnetic, and geometric effects was used to isolate the observed interactions. These results demonstrate a proof-of-principle platform in which neutron spin can be influenced by topological surface transport, laying experimental groundwork for neutron-based probes of spintronic and quantum sensing phenomena in topological materials.</p><p dir="ltr">In the second part, the magnetic structure of HoB₄ was investigated using white-beam time-of-flight neutron diffraction on CORELLI at SNS, ORNL. Temperature- and field-dependent scans revealed a complex sequence of commensurate and incommensurate magnetic phases, including 1/3^rd and 5/9^th magnetization plateaus and an in-plane quadruplet-split modulation near a classical critical regime. Simulated annealing was employed to reconstruct real-space spin textures using an extended Shastry-Sutherland model with multiple exchange terms (J₁–J₈). The calculations reproduced key features of the experimental data, including the stabilization of long-period modulations and the effects of geometric frustration. These results demonstrate that HoB₄ serves as a model system for tunable field-induced magnetic textures and provides a platform for studying critical phenomena in anisotropic, frustrated lattices.</p><p dir="ltr">In the last part, high-resolution neutron spin echo (NSE) spectrometer at SNS, ORNL was used to probe the low-energy spin dynamics of α-RuCl₃ near its zigzag magnetic transition. NSE measurements were carried out across the Neel temperature to search for signatures of critical slowing down, which would be indicative of a second-order transition influenced by fractionalized excitations. No depolarization of the neutron beam was observed within the accessible time and energy scales of the instrument, corresponding to fluctuations slower than ~10⁻⁹ eV in energy. This result constrains the dynamical character of the transition and suggests that the onset of zigzag order is rapid and not governed by slow critical dynamics within the accessible range. The work provides new experimental bounds for theoretical models describing the approach to the spin-liquid regime in Kitaev materials.</p><p dir="ltr">Overall, this dissertation demonstrates how neutron scattering can be used not only to characterize static and dynamic magnetic phenomena, but also to explore coupled spin interactions at the interface between quantum materials and spin-polarized probes. The findings contribute to a broader understanding of magnetic frustration, quantum criticality, and neutron-matter coupling in topological regimes.</p>