Ip Man met a woman named Cheung Wing-Sing, and the two got married. Cheung Wing-Sing grew up in Foshan, China as well, per The Cinemaholic. They had two sons, Ip Chun and Ip Ching, and two daughters, Yip Ar Sum and Yip Ar Woon (per Biography).
According to Black Belt Magazine, Ip Man's comfy life was interrupted in 1937 when he was forced to join the army as China fought Japan. For the next four years, Ip Man was away from his family, which took a heavy toll on all of them.
Cheung Wing-Sing got sick while he was away. Ip Man owned a house and a farm, and the farm suffered when he wasn't there to care for it. He had grown up wealthy, but now found himself quite poor as the war went on.
Around 1949, the Communist party won the war — and that was bad news for Ip Man and his kids. Taking only his most valuable possessions, he and his sons split up. Ip Man headed for Hong Kong, while his sons, who were now grown up, went to Foshan.