In Galations Paul spends alot time teaching, and even pleading with the legalist to put away the rules. Yet he knew that there would be many who took that as a licence to take the liberty of faith in Christ too far. To live a worldly life. To sin.
He predicts that not only will the conservatives be sceptical of this liberty, this Faith Preaching Paul is doing, but Paul also knows that our flesh may take this new found liberty a little too far. He warns against that.
5:13 only do not use your freedom as an opportunity for the flesh.
For in Christ neither our most conscientious religion nor disregard of religion amounts to anything. What matters is something far more interior: faith expressed in love.