A well-intentioned query and the Halloween Problem
Abstract: The author’s description of the Halloween Problem demonstrates the role of cautionary tales in the history of computing. The Halloween Problem emerged in the context of structured query language optimization in relational database research. Normally, a query optimizer works by measuring system calls and paging requests and applying heuristics to the entire access path tree. Query optimization was one of the most challenging tasks facing System R researchers at IBM. These experiments with query optimization form the milieu in which the Halloween Problem emerged.
The Halloween Problem arose in early database systems when an operation changed the very data being used to determine what should happen next, allowing altered records to re-enter the decision process. The problem offers a useful analogy for increasingly capable artificial intelligence. An autonomous system may observe its environment, act upon it, change the conditions it subsequently observes and then act again. At the frontier lies recursive self-improvement: systems capable of modifying their own software, tools or decision processes. The enduring engineering question is deceptively simple: What happens when a system can change the conditions governing its own decisions?
Relata:
