When Network Monitor is detecting on SMTP, which RFC is the detection based on?
What RFC does the traffic need to comply to?

SMTP is outlined in RFC 821 and RFC 5321 which includes ESMTP.

The ESMTP format in itself is outlined in RFC 2821 (superseding RFC 821) and updated to the latest definition in RFC 5321.

Please note: From a product perspective the Microsoft Extensions are not explicitly supported due to its propietary format.
If you run into issues, please try to disable the Exchange extensions ( EXCHX ) or use an inline non-Exchange MTA to enforce RFC compliant traffic.

https://datatracker.ietf.org/doc/html/rfc821

https://datatracker.ietf.org/doc/html/rfc5321

https://www.ietf.org/rfc/rfc2821.txt