How to add Captcha types with the custom URL?

If you have your own custom url, you will have to register captcha, add your custom URL and enter the details on the Merchant Facility.   Make sure your register for Version 2 - tickbox.

There are fields called Captcha Private Key and Captcha Public key are on the Merchant Facility.

Please add those fields to the  page layout if that is not visible as there were not placed on the Merchant Facility for customers doing first install prior to version 9.4.

If you see an error "ERROR for site owner: Invalid key type" shown in the reCaptcha, then this indicates your keys are for the wrong version.  Please register for Version 2 of the API/keys.
Also, make sure the "Captcha Type" is set to "Im Not a Robot - Strict" on the Merchant Facility.