"Often when the call is given, the future hero first refuses to heed it. This may be from a sense of duty or obligation, fear, insecurity, a sense of inadequacy, or any of a range of reasons that work to hold the person in his or her current circumstances." (link)
The Numbers are part of “the call,” as are the whispers.
Hugo’s been trying to live a “normal” life at home with his parents, but it all starts to fray around the edges. When he “refuses the call” of the Numbers at the birthday party, the supernatural ups the ante.
Most of Hugo’s story in Season 4 revolves around his refusal to heed his shamanic call, and how the forces which call him keep upping the ante to get him to pay attention.