Can I move out of state with my son? The father right now has rights to every other weekend.?

I currently live in Virginia and I want to move to New Jersey this summer. Right now my sons father only has rights to every other weekend for visitation with my son. He has not come to get him at all in the past 9 months. Do I need to go through the courts to get permission to move with my son to NJ? Also my sons father is a deadbeat and pays just about next to nothing on what he owes on child support.