The measurement of systemic risk is at the forefront of economists and policymakers concerns in the wake of the 2008 financial crisis. What exactly are we measuring and do any of the proposed measures perform well outside the context of the recent financial crisis? One way to address these questions is to take backtesting seriously and evaluate how useful the recently proposed measures are when applied to historical crises. Ideally, one would like to look at the pre-FDIC era for a broad enough sample of financial panics to confidently assess the robustness of systemic risk measures but pre-FDIC era balance sheet and bank stock price data were heretofore unavailable. We rectify this data shortcoming by employing a recently collected financial dataset spanning the 60 years before the introduction of deposit insurance. Our data includes many of the most severe financial panics in U.S. history. Overall we find CoVaR and SRisk to be remarkably useful in alerting regulators of systemically risky financial institutions.