Downing Street is trying to ignore the challenge from a former minister, but it has raised eyebrows around Westminster.