If you notice these past few months and years, there are a lot of sites about adult, poker, blackjack, casino, pharmacy, and spammers are doing massive comment spam, guesbook spam, and other kinds of spam linking to their sites.

Such sites are then viewed by the search engines in a negative way. Don't have a majority of your backlinks from adult, pharmacy, and poker/casino sites.

Check also if your webhost are allowing such sites and you are in the same webserver. If your site is hosted on an IP where there are lots of sites that spammers use, better move your site elsewhere. Or alternatively, you may ask for a dedicated ip address for your site.