This is a little bit like comparing apples to oranges, and will also depend on which country or demographic we're talking about. In the US, I seriously doubt any (straight) women pay much attention to soccer - aside from their kids' practice games - but Ladies' Olympic figure skating is the 4th most watched sporting broadcast after Super Bowl, World Series and NBA finals. In Austria, soccer is also king, but alpine skiing, particularly the men's downhill at Kitzbühl, runs a very. very close second. I don't recall Mexican athletes doing much at the Olympics, it's pretty much soccer 24/7/365, and lucha libre. Norwegians don't get too terribly upset if they don't win the World Cup, but not getting Gold in cross-country skiing smarts bad. One thing that bugs me a little bit about some Olympic sports is that these people train very hard for 4 years (or more) for a performance measured in minutes or even seconds. If a gnat zooms past your nose at precisely the wrong moment, your Olympic dreams go poof!
I always got the impression that Olympics were only "popular/bigger" in North America, Russia, China and Australia. Everywhere else = World Cup, and everywhere where it isn't, don't matter