Dialback was originally the primary method for server-to-server authentication. It was dropped from the core XMPP RFCs in 2011 in favour of TLS authentication. Documentation of the dialback protocol was moved to XEP-0220.
Historically many servers had self-signed or potentially untrusted (e.g. CAcert) certificates, and dialback provides a mechanism to verify these servers through DNS instead of TLS.
Given that the vast majority of certificates today are themselves issued based on DNS verification, there is not a major difference in security (from an authentication point of view) between the two protocols when TLS is used for confidentiality and integrity of the connections.
However the dialback protocol has a fair number of moving pieces, it's not a trivial protocol in practice. Multiple security vulnerabilities have been reported against dialback in XMPP servers in the past. It can be argued that TLS is also complex, but multiple well-scrutinised libraries are available to abstract that. There are no portable libraries for the protocol documented in XEP-0220.
With the rise of Let's Encrypt and other free TLS certificate offerings, the vast majority of XMPP servers in 2019 run with valid certificates, and dialback is rarely used.
Unused dormant code is a liability, especially when that code implements a 20 year-old authentication protocol that has only seen use within our community, and that use is rapidly declining in favour of TLS, which is widely implemented in standard third-party libraries and all modern internet protocols.
Removing support for dialback would decrease the surface area for security issues to arise in server-to-server authentication, and improve the robustness of the server and the network as a whole.
- Remove mod_dialback from the default configuration in the next major release
- Move mod_dialback to prosody-modules in the next major release, or the following one
As discussed in https://logs.xmpp.org/xsf/2019-12-06?p=h#2019-12-06-936bcf581998facd - I'd not be in favor of phasing out dialback. Quoting parts of the discussion linked above:
I don't think we should remove support for people that for one reason or another don't want or can't set up certificates. Dialback offers better security than no security.
I can think of deployments that are deliberately not internet-facing, or have other reasons to not want to depend on Let's Encrypt
Also, everyone having valid certificates very much is an effect of one single organisation providing a service, I think. What happens if, for whatever reason, Let's Encrypt stops doing their thing (or stops being trustworthy)?
Their certificates are only valid for 3 months - having dialback as a fallback to a service that pretty much hinges on one organisation isn't the worst of ideas, maybe.