Then the Lord said to Jacob, “Go back to the land of your fathers and to your relatives, and I will be with you.”
Genesis 31:3 (NIV)I am the God of Bethel, where you anointed a pillar and where you made a vow to me. Now leave this land at once and go back to your native land.Genesis 31:13 (NIV)
In great fear and distress Jacob divided the people who were with him into two groups, and the flocks and herds and camels as well.Genesis 32:7 (NIV)
It is perfectly ok to be afraid.