The Bill of Rights wasn’t a part of the original draft of the constitution and proponents of its ratification faced criticism on this grounds. Alexander Hamilton shot back in Federalist 84 that a Bill of Rights would be a terrible idea:
I go further, and affirm, that Bills of Rights, in the sense and to the extent in which they are contended for, are not only unnecessary in the proposed Constitution, but would even be dangerous. They would contain various exceptions to powers not granted; and on this very account, would afford a colorable pretext to claim more than were granted. For why declare that things shall not be done which there is no power to do? Why, for instance, should it be said, that the liberty of the press shall not be restrained, when no power is given by which restrictions may be imposed? I will not contend that such a provision would confer a regulating power; but it is evident that it would furnish, to men disposed to usurp, a plausible pretence for claiming that power. They might urge with a semblance of reason, that the Constitution ought not to be charged with the absurdity of providing against the abuse of an authority, which was not given, and that the provision against restraining the liberty of the press afforded a clear implication, that a power to prescribe proper regulations concerning it was intended to be vested in the National Government. This may serve as a specimen of the numerous handles which would be given to the doctrine of constructive powers, by the indulgence of an injudicious zeal for Bills of Rights.
Then of course a few years later a Bill of Rights was adopted. The Federalist is often used for its insights into what the Founders had in mind with their constitution, so what is one to make of this?
— Perhaps since Hamilton specifically warned that a Bill of Rights would imply that everything not forbidden is permitted, and then such a bill was adopted, we ought to infer that everything not forbidden is now prohibited.
— Or perhaps since Hamilton specifically warned against this we should flag this as a particularly wrongheaded form of inference.
— Or perhaps since Hamilton was a practical politician trying to get people to vote his way on a particular issue, we should regard him as just making the best case possible for a deeply flawed Bill Of Rights-less constitution and ignore his rhetorical flourishes.
To me it all seems like a good argument that relying on history and exegesis of centuries-old documents (as opposed to more recent precedents) as the basis of our legal system is a deeply problematic concept.