I run my own physical DNS server.
Great. Then all you need to do is distribute the IP address(es) of your DNS server to your local clients.
If I were to use BRSK as an example IPv4 would be CGNAT. If I could not use my own IPv6 DNS server then I would feel the need to disable IPv6 on my network.
A DNS server may have an IPv4 address, an IPv6 address, or both. It will still give the same answers to the same queries, regardless of whether those are received on its IPv4 interface or its IPv6 interface.
My dns server provides ad, tracking and malware blocking through the use of dns block lists. A router advertising its own IPv6 dns servers would circumvent any protection I provide with my own dns server.
Then the issue is simply how your clients learn the correct DNS server address(es) to use.
I think what you're concerned about is that you don't want the router to be announcing *itself* as the DNS server, via either protocol. Hence you need to make sure that the router configuration for DHCP gives out your DNS server's IPv4 address, and the configuration for RA RDNSS and/or stateless DHCPv6 gives out your DNS server's IPv6 address (assuming it has one).
The clients don't actually need to learn an IPv6 address for DNS at all. They could still query your DNS server over IPv4, and get AAAA records with IPv6 addresses to connect to.
However, if clients learn both an IPv4 address and an IPv6 address for DNS, they should both point to the same DNS server. Otherwise, it's indeterminate which one the clients will use - although typically they will try contacting the DNS server over IPv6 first.