Spurred by advancements in scale, large language models (LLMs) have demonstrated theability to perform a variety of natural languageprocessing (NLP) tasks zero-shot—i.e., without adaptation on downstream data. Recently,the debut of ChatGPT 1has drawn a great dealof attention from the natural language processing (NLP) community due to the fact that itcan generate high-quality responses to humaninput and self-correct previous mistakes basedon subsequent conversations. However, it is notyet known whether ChatGPT can serve as a generalist model that can perform many NLP taskszero-shot. In this work, we empirically analyzethe zero-shot learning ability of ChatGPT byevaluating it on 20 popular NLP datasets covering 7 representative task categories. With extensive empirical studies, we demonstrate boththe effectiveness and limitations of the currentversion of ChatGPT. We find that ChatGPT performs well on many tasks favoring reasoningcapabilities (e.g., arithmetic reasoning) whileit still faces challenges when solving specifictasks such as sequence tagging. We additionally provide in-depth analysis through qualitative case studies.