I would need exact technical information to be able to offer you any help. Depending on how you create your custom login page some or all of BPS Login Security settings may not work on your custom login page.
What is the URL to your custom login page?
What WordPress Hooks (Actions and Filters) does your custom login page use?