Ron Brownstein notes that historically an unpopular incumbent president does drag his party down even if he’s not on the ballot:
Unpopular departing presidents, though, have consistently undercut their party in the next election. Democrats lost the White House in 1952 and 1968 after Harry Truman and Lyndon Johnson saw their approval ratings plummet below 50%. Likewise, in the era before polling, the opposition party won the White House when deeply embattled presidents left office after the elections of 1920 (Woodrow Wilson), 1896 (Grover Cleveland), 1860 (James Buchanan) and 1852 (Millard Fillmore). The White House also changed partisan control when weakened presidents stepped down in 1844 and 1884. Only in 1856 and 1876 did this pattern bend, when the parties of troubled presidents Franklin Pierce and Ulysses S. Grant held the White House upon their departure.
It should be pointed out that 1856 is a not very encouraging precedent for the Republicans. In essence, the opposition Whig Party had collapsed (garnering only 21.5 percent of the vote) but the Republican Party hadn’t yet consolidated its position (garnering only 33 percent of the vote), throwing the election to the Democrats by default. I’m fairly confident that the Democratic Party isn’t in the midst of a Whig-style collapse.