An Architectural Assessment of SPEC CPU Benchmark Relevance

Abstract

SPEC compute intensive benchmarks are often used to evaluate processors in high-performance systems. However, such evaluations are valid only if these benchmarks are representative of more comprehensive real workloads. I present a comparative architectural analysis of SPEC CPU benchmarks and the more realistic SPEC Java Server benchmark. This analysis indicates the integer subset of CPU benchmarks are broadly representative of the branching and cache characteristics of server and business workloads. I also leverage the collected data to perform a statistical regression analysis, estimating an application’s performance as a linear function of its observed architectural characteristics. A linear model derived from simulated SPEC CPU data appears to fit the observed data quite well, with 94 percent of performance variance explained by four predictors that incorporate cache and queue usage. The linear model predicts server benchmark performance to within 2 percent of the observed value. Despite the regression model’s close fit to the observed data, its predictive ability is constrained by the trace collection facility. It is infeasible to use the performance of one trace suite to predict the performance of another trace suite if the underlying trace collection processes differ across suites.