Let there be $R$ red sweets, and $N$ be the total number of sweets in the jar.
The probability of both being red after putting the first sweet back to the jar is given by:
$\dfrac{R}{N} \times \dfrac{R}{N} $
The probability of both being red after eating the first sweet is given by:
$\dfrac{R}{N} \times \dfrac{R-1}{N-1} $
-----------book page break-----------
Therefore,
$\left( \dfrac{R}{N} \times \dfrac{R}{N} \right) \div \left( \dfrac{R}{N} \times \dfrac{R-1}{N-1} \right) = \dfrac{105}{100}$
$\Rightarrow \dfrac{R}{N} \times \dfrac{N-1}{R-1} = \dfrac{21}{20}$
$\Rightarrow 20 \times \dfrac{N-1}{N} = 21 \times \dfrac{R-1}{R}$
$\Rightarrow 20 \times \left( 1 - \dfrac{1}{N} \right) = 21 \times \left( 1 - \dfrac{1}{R} \right)$
$\Rightarrow 20 - \dfrac{20}{N} = 21 - \dfrac{21}{R}$
$\Rightarrow \dfrac{21}{R} = 1 + \dfrac{20}{N}$
As $R$ increases, $\dfrac{21}{R}$ decreases, therefore $1 + \dfrac{20}{N}$ and $\dfrac{20}{N}$ will also decrease, and $N$ will increase. Therefore to maximise $N$ we need to maximise $R$.
Also observe that:
Since $N$ is positive, $1 + 1 + \dfrac{20}{N} \gt 1$
Therefore,
$\dfrac{21}{R} \gt 1$
$\Rightarrow R \lt 21$
-----------book page break-----------
Therefore, the maximum possible value of $R$ is $20$.
Therefore
$\dfrac{21}{20} = 1 + \dfrac{20}{N}$
$\Rightarrow 1 + \dfrac{1}{20} = 1 + \dfrac{20}{N}$
$\Rightarrow N = 400$