What impact with CBD have on an unborn baby? News 4’s Alyssa Toomey went to local doctors to find out. When it comes to fertility, doctors said both men and women should stop using cannabis if they are trying to get pregnant. Studies show marijuana use can affect sperm count in men.