Am assuming your ancestors were slaves, which would make your statements true about the Constitution. There were at the time of the Constitution free blacks who could also vote, they were counted the same as whites. It is not all black and white.
Free speech, with only the limits the Supreme Court has placed upon it, allows for freedom of conscience. the right to think and speak as one pleases. Yes, there should be some social restrictions, that are unwritten, but acknowledged, as to the limits. Yet, were that the case, much that has been achieved, to make this a more perfect union, would never have happened. Those in power would simply have ruled their speech illegal, and put them in jail.
A free mind, with the ability to speak its own thoughts, has led to tremendous changes in society, both good and bad. That all is not one way, is no reason to muzzle those whose speech is not to one's liking.
The founders did not have all the answers, but they put in place a way for the answers to be inserted within the Constitution. Allowing for a safety valve as it were.
Whether one wants to go back to 1750 or 1500, the world was awash in slavery, on every continent, even North and South America, before being rediscovered by Columbus. Across North Africa, in Ethiopia, and other parts of Africa, in India, Europe, Asia. It was so prevalent that it was like the wallpaper in one's house, its there but hardly acknowledge or seen. The amazing thing is that people finally figured out it was wrong and changed things.