When you use a custom domain, you will notice that you effectively have two domains, your AMP subdomain, and then your original domain.

Normally, when you use a subdomain, Google treats the subdomain and the original domain as "Two separate entities", meaning that the SEO juice or traffic attribution gets attributed to the domain which a user landed on.

For example, if you have 20 page views, 10 to your subdomain, and 10 to your original domain, normally this would mean that Google gives each of those domains credit for their 10 page views each, and it doesn't give your original domain credit for 20 page views.

However, when you use AMP, we link your pages with the appropriate <link rel="canonical"> and <link rel="amphtml"> tags, so Google knows that any traffic going to your custom AMP subdomain counts towards the SEO of the original (non-AMP) domain.  This is how "canonical" tags work - Google makes sure that the canonical version of the page receives the SEO benefit.

For this reason, you don't need to worry about splitting your SEO benefit between two domains, it it always your original domain that retains the SEO benefit, even when you use a custom AMP subdomain.  There is no SEO penalty by using a custom domain, your original domain still receives all link juice going to AMP pages.