In most cases, the home page to a site will be the lead into other pages, so for this page you would obviously want a low bounce rate. But visitors could land on one of the inner pages that they have found in a search.
For example I have a page which is for the weather in my locality.
It is high in SERPs for this and has a bounce rate of 87%. That's because most of the visitors who go to that page directly only want to know about the weather, they don't want to buy a house, so they read the information and leave. That's fine for me because 13% do go elsewhere on the site, and the bouncers may come back another time for different information if they liked my page.
So a high bounce rate is definitely not necessarily a bad sign.
Finally, when you view the bounce rate in Google Analytics, bear in mind that the main display figure is for the whole site. You must look further at the rate for individual pages before deciding whether or not you have a problem.