The Nephilim were on the earth in those days—and also afterward—when the sons of God went to the daughters of humans and had children by them. They were the heroes of old, men of renown.
Genesis 6:4 (NIV)
If you look very closely, Genesis 6 does not actually say the Nephilim are the offspring of the sons of God and human women. It merely says they were present then and afterwords.
Whether they are the offspring or not is unclear, but other later writings outside the Bible (the book of Enoch, Jubilees) suggest so.
If they are not the offspring of the “sons of God” and the good daughters of humanity, the Haman (who is associated with "falling," like the word "Nephilim) connection is stronger, as Haman is not a son of Xerxes in the book of Esther.