How long have you had him? He is not upset with you just a nervous. Try just gentley putting your hand inside and letting him get used to it being there. Hand feeding can also help you gain his trust.
If you imagine a giant hand chasing you around your house you might be a little scared.
It's good friends that walk beside you through the good times and carry you through the bad.