By generating hype and dominating attention, dramatic visions of future AI can have a disproportionate influence on public opinion and present a challenge in managing public expectations. However, they can also play a beneficial role. First, even if there is only a very small chance of them occurring, their potential impacts are so serious that they demand at least some reflection and preparation. Second, in extrapolating the challenges of today's AI through the occasionally dramatic lens of our hopes and fears, they provide an opportunity for broad reflection upon what we want from the technology. 3.2.2 Winter

Utopian and dystopian scenarios appear in opposition, but they share a foundation in the idea that AI will have a substantial impact on society. Their true opposite is a vision of AI fading into obscurity with little impact on society. This spectre also haunts AI in the form of an 'AI winter', a period in which interest in AI is substantially reduced, accompanied by stagnation in investment, development and application. The AI sector has experienced such winters before, in the 1970s and 1990s following periods of reduced enthusiasm in the symbolic AI paradigm that dominated the field at that time, until the resurgence of AI under the ML paradigm in the late 2000s. Some are concerned that the current 'AI summer' is merely the zenith of a hype cycle which could also descend to its nadir. This scenario is often positioned as the consequence of deploying utopian and dystopian visions which spread disappointment and fear. Others potential causes of a speculative winter have been identified

3.2.3 Runaway artificial intelligence One of the most commonly cited speculative challenges of AI is that it could take control of its own development, escape human control and constantly develop itself with disastrous consequences. There are several variants of this vision. In some, the AI develops its own values and aims, transcending those set by its human creators, and hides its true intentions and capabilities from humanity until they are ready to implement them successfully. Indeed, since humans are considered intelligent enough to form and pursue their own objectives, a true AGI or ASI should be able to do the same. In others, the AI sticks to the objectives set by its human creators, but with a level of capability, autonomy and determination that innocent but carelessly defined tasks such as 'create paperclips' could lead to disastrous outcomes such as enslaving humanity in paperclip factories or 29

STOA | Panel for the Future of Science and Technology transforming all earthly matter into stationery.33 Either way, future AI's power and autonomy is presented as an existential threat to humanity. Today's AI presents no such risks, yet utopian visions of runaway AI are conspicuous by their absence. They face the same technical barriers as the dystopian variants but also require an optimistic view of the impacts of such a powerful technology, which appears beyond most people's imagination. 3.2.4 Job losses or making employment obsolete?