Miguel Cervantes
Jedi Master
Georgia does have laws regarding possession. Court cases in the state have determined that a fence defines a property boundary even if it is outside the documented property line if neighbors do not contest the fence within seven years, according to Coleman, Chambers, Rogers & Williams, LLP, a Georgia law firm. This applies to natural boundaries, so a homeowner planting a tree past his or her property can change the line if uncontested. Homeowners can prevent such occurrences by proactively marking the property with wooden stakes.
https://www.reference.com/government-politics/laws-georgia-property-line-fencing-c9eb462c7d0cead2#
According to some state laws (not clear where Georgia stands on this) if your fence has been in existence for more than 7 years without any congruent property owner contesting it then it appears "Adverse Possession" may be on your side.
To know if this fence of yours is dead on your line or even over a few inches you still need a survey.
Planning and zoning to find out your local laws
Code enforcement to put a cease and desist on his activities.
Survey the line to set up your legal argument
Talk to an attorney
Otherwise your options are extremely limited.
I am not an attorney, but I have stayed at a Holiday Inn Express before.