A MULTI-FIDELITY MODELING AND EXPERIMENTAL TESTBED FOR TESTING & EVALUATION OF LEARNING-BASED SYSTEMS

Learning-based systems (LBS) have become essential in various domains, necessitating the development of testing and evaluation (T&E) procedures specifically tailored to address the unique characteristics and challenges of LBS. However, existing frameworks designed for traditional systems do not adequately capture the intricacies of LBS, including their evolving nature, complexity, and susceptibility to adversarial actions. This study advocates for a paradigm shift in T&E, proposing its integration throughout the entire life cycle of LBS, starting from the early stages of development and extending to operations and sustainment. The research objectives focus on exploring innovative approaches for designing LBS-specific T&E strategies, creating an experimental testbed with multi-fidelity modeling capabilities, investigating the optimal degree of test and evaluation required for LBS, and examining the impact of system knowledge access and the delicate balance between T&E activities and data/model rights. These objectives aim to overcome the challenges associated with LBS and contribute to the development of effective testing approaches that assess their capabilities and limitations throughout the life cycle. The proposed experimental testbed will provide a versatile environment for comprehensive testing and evaluation, enabling researchers and practitioners to assess LBS performance across varying levels of complexity. The findings from this study will contribute the development of efficient testing strategies and practical approaches that strike a balance between thorough evaluation and data/model rights. Ultimately, the integration of continuous T&E insights throughout the life cycle of LBS aims to enhance the effectiveness and efficiency of capability delivery by enabling adjustments and improvements at each stage.