If someone tells you that AI won’t ever be a threat to us because why would it want to kill us, they really don’t get how brittle and nonlinearly weird this tech can be. A wonderful new paper compiles a bunch of examples from AI software that used evolutionary strategies that through trial and error discovered some… unusual solutions. For example,
when MIT Lincoln Labs evaluated GenProg on a buggy sorting program, researchers created tests that measured whether the numbers output by the sorting algorithm were in sorted order. However, rather than actually repairing the program (which sometimes failed to correctly sort), GenProg found an easier solution: it entirely short-circuited the buggy program, having it always return an empty list, exploiting the technicality that an empty list was scored as not being out of order.
Here’s another example where the AI “solved” a problem that technically fit the rules it was given:
In another project, to avoid runaway computation, the fitness function explicitly limited a program’s CPU usage: in response, GenProg produced programs that slept forever, which did not count toward CPU usage limits, since there were no computations actually performed
We rarely hear about these “solutions” because when AI come up with them, researchers tweak the victory conditions to bar them. That works just fine in a highly comtrolled, highly simplifted lab environment. But as we build more and more systems that are expected to work in ever more complex environments, catching all these novel solutions gets a lot harder. And if an AI thinks the best way to sort a list is to kill the list…
Does this mean we should be paranoid about AI? No. But blithely assuming it’ll never be a threat is equally foolish. We need to assume that Uncle Murphy will be a not infrequent guest of AI systems and act accordingly.