Hotter temperature can reduce labor productivity, work hours, and labor income. The effects of heat are likely to be a joint consequence of both exposure and vulnerability. Here we explore the impacts of heat on labor income in the US, using regional wealth as a proxy for vulnerability. We find that one additional day >32 °C (90 °F) lowers annual payroll by 0.04%, equal to 2.1% of average weekly earnings. Accounting for humidity results in slightly more precise estimates. Proxying for wealth with dividend payments we find smaller impacts of heat in counties with higher average wealth. Temperature projections for 2040–50 suggest that earnings impacts may be 95% smaller for US counties in the richest decile relative to the poorest. Considering the within country distribution of vulnerability, in addition to exposure, to climate change could substantially change estimated within-country differences between the rich and poor in income losses from climate change.