Let $E$ be an elliptic curve defined over $\mathbb{Q}$ of conductor $N$, $c$ the Manin constant of $E$, and $m$ the product of Tamagawa numbers of $E$ at prime divisors of $N$. Let $K$ be an imaginary quadratic field where all prime divisors of $N$ split in $K$, $P_K$ the Heegner point in $E(K)$, and ${\rm III}(E/K)$ the Shafarevich-Tate group of $E$ over $K$. Let $2u_K$ be the number of roots of unity contained in $K$. Gross and Zagier conjectured that if $P_K$ has infinite order in $E(K)$, then the integer $ c \cdot m \cdot u_K \cdot |{\rm III}(E/K)|^{\frac{1}{2}}$ is divisible by $|E(\mathbb{Q})_{\rm{tor}} |$. In this paper, we prove that this conjecture is true if $E(\mathbb{Q})_{\rm{tor}} \cong \mathbb{Z}/2\mathbb{Z}$ or $\mathbb{Z}/4\mathbb{Z}$ except for two explicit families of curves. Further, we show these exceptions can be removed under Stein--Watkins conjecture.