Earlier-stage evaluations of a new AI architecture/system need affordableAI benchmarks. Only using a few AI component benchmarks like MLPerfalone in the other stages may lead to misleading conclusions. Moreover, thelearning dynamics are not well understood, and the benchmarks’ shelf-life isshort. This paper proposes a balanced benchmarking methodology. We usereal-world benchmarks to cover the factors space that impacts the learningdynamics to the most considerable extent. After performing an exhaustivesurvey on Internet service AI domains, we identify and implement nineteenrepresentative AI tasks with state-of-the-art models. For repeatable performance ranking (RPR subset) and workload characterization (WC subset), wekeep two subsets to a minimum for affordability. We contribute by far themost comprehensive AI training benchmark suite.The evaluations show: (1) AIBench Training (v1.1) outperforms MLPerfTraining (v0.7) in terms of diversity and representativeness of model complexity, computational cost, convergent rate, computation, and memory accesspatterns, and hotspot functions; (2) Against the AIBench full benchmarks,its RPR subset shortens the benchmarking cost by 64%, while maintainingthe primary workload characteristics; (3) The performance ranking showsthe single-purpose AI accelerator like TPU with the optimized TensorFlowframework performs better than that of GPUs while losing the latter’s general support for various AI models. The specification, source code, andperformance numbers are available from the AIBench homepage https://www.benchcouncil.org/aibench-training/index.html.