barretenberg
barretenberg copied to clipboard
Handle edge case y = 0 in `dbl()` and `quadruple_and_add(...)` in biggroup
In the function dbl() for doubling an EC point $(x, y) \in \mathbb{F}_q$, we compute $-\lambda \in \mathbb{F}_q$ defined as
$$\lambda = \frac{3x^2 + a}{2y}$$
using the function msub_div. The denominator $2y$ is not checked to be $0$ in the non-native field $\mathbb{F}_q$. The $y$-coordinate can indeed be 0 in the case when we try to call dbl() on the point $P := (x', 0)$. In such a case, $2P := \mathcal{O}$ leads to a point at infinity. We do not handle this edge case correctly. The fix is to explicitly handle this case and set the point at infinity flag to true if $y = 0$ and $x \neq 0$.
Similar edge case appears when computing $-\lambda$ in quadruple_and_add(...).