Set “No” at Allow Branch access against the branch. So that, you can restrict the branch access for Organization and Zonal users. Please note that the direct users of the branch and branch’s staff will still be able to login to the branch when this setting is made.
Path: Settings/Organisation/Manage Organisation/ Branches