You definitely need to discuss this with your doctor, both for the weaning off (they will work with you to create a plan that will be appropriate for you) and for the possible revert to your old ways. Yes, it is possible that you will get anxiety again and it might get bad. Some people need medication to help them lead a "normal" life, if that is the case with you and you realize you did better with the medication, that's not a failure.