Many people have said the same, but considering the time, the 1960's, I'd guess playing lacrosse, during baseball season, a lot of sub-par athletes played. To me, once you have ball handling and accurate passing, lacrosse looks like contact-allowed four-corners offense or a really good rondo execution.
This may have been even more true because it was in the 1950s, not the '60s, that Brown was playing lacrosse, and high school lacrosse on Long Island was in its infancy then. And when he was in high school, there may have been as much competition from track as from baseball, since Long Island was track hotbed back then (Brown himself played all three, and was a state champion high jumper).
This is another reason why I played soccer and I encourage my son to stick with it. The foot work skills learned/acquired in soccer is phenomenal. An example of this is Hakeem Olajuwon (speaking of bball). He played soccer growing up in Africa... Watch the 1995 NBA finals where it pitted Hakeem against a raw Shaquille O'neal at the center position. Hakeem schools him up and down the court with the ball fakes and fleat footed legwork. Hakeem later attributed his foot work to playing soccer as a youth in his native country.