Ok, update.. problem fixed after I decided to have a look this evening remotely... all working now via Exchange 2016 sending to Zen's mail servers on a non-Zen Internet circuit via Authentication with TLS.
Problem was we had a setting enabled on our Exchange Send connector to "Proxy" outbound messages via the Client Access Role on Exchange (this feature was introduced in Exchange 2013). It allows organisations to simplify (as one example) how outbound mail flow is sent out to the internet and force it through a single/group of Client Access servers. This only applies to Exchange 2013/2016-->
Problem seemed to be that Zen's mail server didn't introduce back the AUTH command (telling Exchange to send the zen username/password to authenticate with) during the telnet negotiation after the STARTTLS command was issued (for reasons I'm not yet clear on). Instead I changed this specific Send Connector in Exchange to not proxy outbound email via our Client Access Servers and instead just send it straight out via the Transport "Service" on the mailbox server directly to Zen's 'mailhost.zen.co.uk' and hey presto, it worked. Looking at the protocol logs that I set to verbose to troubleshoot you can then see the session getting stood up, TLS negotiating, authentication credentials getting sent, email being sent and then the session stood down.
This kind of 'proxy' setup would only apply to people using Exchange 2013 onwards and I suspect that other mail server platforms probably dont have the ability to "proxy" outbound email via different parts of their platform.
Fred - Hope this helps, but its not really related to smarthost(s)... they are just a mail server you assign in your send connectors to send mail via.. in this case, I am using Zen's "mailhost.zen.co.uk". If you don't populate anything (which is the default) then Exchange will just send via DNS as explained earlier.
10forcash - Yep, SSL 3 and TLS 1.0 disabled as part of our standard server build. As for UNIX and IIS, what I am referring to is the ability to have a mail server running on UNIX (like Exim or Postfix) or say using the SMTP service built into IIS to send mail.
Hope some, no matter how little of this has been useful.. as suspected it turns out it was a setting/config change on Exchange and nothing to do with BT and/or Zen from a network connection/mail relay perspective.
Cheers
Andy
p.s Here is a snippet from the Exchange logs showing the transaction. I've edited a few names and removed all the certificate CN names etc (too much to word wrap)
The 81.x.x.x is my BT Business FTTP Static IP range.
220 smarthost01b.mail.zen.net.uk ESMTP Exim 4.80 Sun, 30 Jul 2017 20:56:11 +0000",
EHLO mx.mydomain.com,
250 smarthost01b.mail.zen.net.uk Hello mx.mydomain.com [81.x.x.x] SIZE 36700160 8BITMIME ETRN PIPELINING STARTTLS HELP,
STARTTLS,
220 TLS go ahead,
CN=*.mydomain.com, OU=PositiveSSL Wildcard, OU=Domain Control Validated CN=COMODO RSA Domain Validation Secure Server CA,
CN=*.zen.co.uk, OU=COMODO SSL Wildcard, OU=Hosted by Zen Internet LTD, OU=Domain Control Validated
TLS protocol SP_PROT_TLS1_2_CLIENT negotiation succeeded using bulk encryption algorithm CALG_AES_256 with strength 256 bits,
Valid,Chain validation status
CN=*.zen.co.uk, OU=COMODO SSL Wildcard, OU=Hosted by Zen Internet LTD, OU=Domain Control Validated
EHLO mx.mydomain.com,
250 smarthost01b.mail.zen.net.uk Hello mx.mydomain.com [81.x.x.x] SIZE 36700160 8BITMIME ETRN PIPELINING AUTH PLAIN LOGIN HELP,
AUTH LOGIN,
334 <authentication information>,
<Binary Data>,
<,334 <authentication information>,
<Binary Data>,
<,235 Authentication succeeded,
,,sending message with RecordId 43142946488393 and InternetMessageId <
[email protected]>
MAIL FROM:<
[email protected]> SIZE=4739,
RCPT TO:<
[email protected]>,
250 OK,
250 Accepted,
DATA,
354 Enter message, ending with ""."" on a line by itself",
250 OK id=1dbvFz-0000Ih-HH,
QUIT,
221 smarthost01b.mail.zen.net.uk closing connection,
Edited by deleted (Sun 30-Jul-17 22:31:26)