I have about a dozen books that I am getting rid of. Mostly paperback and all related to hooligan stuff and casual culture. Authors include Dougie Brimson, Cass Pennant, etc. The only condition is that you pay for shipping. I have a scale and can estimate costs before they ship. If you want a complete list I ca do that. PM me if interested.
I'm going to guess that it refers to the "casuals," who are often paired with hoolies. From what I can tell, though, the casuals tend to be more into fashion than violence. And lo and behold... some of them even have a website... http://www.footballcasual.com/ I think this says it all... Welcome to FootballCasual.com - the site dedicated to the fashion of football. We are obsessed with the aesthetics and the trends which have been around on the terraces throughout the decades. They also say they don't promote violence... wonder if that means, "hey, but if some aggro happens..."
I would say casual culture is the subculture that goes along with people who commit violent acts at matches. The clothing, music, hairstyles you name it.