You're totally right. The reason it happened was because video game systems were considered "toys" back in the day, and the marketing for toys was generally extremely polarized back then, all part of the social conditioning for patriarchy. Girls got to play with baby dolls, tea sets and toys inspired by domestic labor such as cooking and cleaning, and boys got to play with toys themed around heavy machinery, sports, combat/war and so on, all to prepare, socialize and condition them for the gender roles they were/are expected to perform in society.

Early video games were often sports or combat/war themed, so they became toys for boys, and ended up in a positive feedback loop. I would guess that the reason that sports and war themed video games were developed was because the people in charge of developing the games were also mainly men.

Women were significantly sidelined from computers and technology in most of the world around the time video games were being developed despite being extremely significant in the early history of computers. Whole workforces of women extremely competent with programming were fired and replaced with men who had no idea what they were doing in places, setting technological advancement back significantly.