When most people say the Founding Fathers, they mean the big seven.
The sticking point for the Constitution (as early drafts did repudiate slavery) was the Southern delegates - though slavery had not yet fully developed as a core part of the region's identity, a lot of money was still tied up in the disgusting trade. Even so, the assumption was that slavery would die out in the South the same way it had done elsewhere - a trend which was reversed by the invention of the cotton gin.