Case Study HPM Agent Performing a Simplified Air Traffic Control Task ___________________________________________________

Challenge

The Air Force Research Laboratory (AFRL) has an ongoing effort to conduct annual comparisons of competitively selected cognitive architectures. The challenge is to advance the state-of-the-art in human behavioral representation and begin to systematically address human model validation issues.

AFRL's Agent-Based Modeling and Behavioral Representation (AMBR) multi-year program compares cognitive architectures across different behavioral challenges within a stylized Air Traffic Control (ATC) task. The COGNET/iGEN cognitive architecture has been selected to participate in all AMBR rounds to date.

Result

In Round 1, the COGNET framework and iGEN executable were modified to provide enhanced multi-tasking capabilities and an ATC model was developed to interact with the example environment. A model fly-off was conducted comparing ATC model runs to human runs, analyzing performance (based on accumulated task penalty points), response time (for task-significant events), and workload assessment (subjective). Of the four models tested (CHI's iGEN, Carnegie Mellon's ACT-R, Soar Technologies' EPIC-Soar, and AFRL's D-Cog), the iGEN model results provided the best overall fit to the human test data.

In Round 2, CHI converted its iGEN model to function as part of the Icarus High Level Architecture (HLA) Federation. The original fly-off was repeated to ascertain the effect of HLA implementation on performance. The iGEN model produced results virtually identical to Round 1, again providing the best overall fit to the test data. In addition, while the other models executed more slowly under HLA (running at or below real time), the iGEN model executed faster, supporting rates well above real time.

In Round 3, CHI Systems extended COGNET/iGEN to incorporate category learning within the ATC task environment. The initial Round 3 and revised Round 4 category learning model results were compared to newly collected human trial results. As reported by the comparison moderator (BBN Technologies) at the 24th Annual Meeting of the Cognitive Science Society, only the iGEN model results were indistinguishable from human experiment results.

Zachary, W., Santarelli, T., Ryder, J., Stokes, J., & Scolaro, D. (2001). Developing a multi-tasking cognitive agent using the COGNET/iGEN integrative architecture. In Proceedings of 10th Conference on Computer Generated Forces and Behavioral Representation, (pp. 79-90). Norfolk, VA: Simulation Interoperability Standards Organization (SISO).

Glenn, F., LeMentec J.C., Ryder, J., Santarelli, T., Stokes, J. & Zachary, W. (in press). Development of a Concept Learning Capability for a Human Performance Model. To be presented at the 2003 Conference on Behavior Representation in Modeling and Simulation (BRIMS), Scottsdale, AZ.