Long-term use of benzodiazepine receptor agonists (BZDs) may depend on clinicians’ BZD discontinuation strategies. We aimed to explore differences in strategies and difficulties with BZD discontinuation between psychiatrists and non-psychiatrists and to identify factors related to difficulties with BZD discontinuation. Japanese physicians affiliated with the Japan Primary Care Association, All Japan Hospital Association, and Japanese Association of Neuro-Psychiatric Clinics were surveyed on the following items: age group, specialty (psychiatric or otherwise), preferred time to start BZD reduction after improvement in symptoms, methods used to discontinue, difficulties regarding BZD discontinuation, and reasons for the difficulties. We obtained 962 responses from physicians (390 from non-psychiatrists and 572 from psychiatrists), of which 94.0% reported difficulty discontinuing BZDs. Non-psychiatrists had more difficulty with BZD discontinuation strategies, while psychiatrists had more difficulty with symptom recurrence/relapse and withdrawal symptoms. Psychiatrists used more candidate strategies in BZD reduction than non-psychiatrists but initiated BZD discontinuation after symptom improvement. Logistic regression analysis showed that psychosocial therapy was associated with less difficulty in BZD discontinuation (odds ratio, 0.438; 95% confidence interval, 0.204–0.942; p = 0.035). Educating physicians about psychosocial therapy may alleviate physicians’ difficulty in discontinuing BZDs and reduce long-term BZD prescriptions.