#598 net.adns stops working if it runs into NOFILE limit
Reporter
Zash
Owner
Zash
Created
Updated
Stars
(0)
Tags
Priority-Medium
Status-Fixed
Milestone-0.9
Type-Defect
Zash
on
debug Records for _xmpp-server._tcp.remote.example. not in cache, sending query (thread: 0x18802f0)...
error Error sending DNS query: Too many open files
debug remote.example has no SRV records, falling back to A/AAAA
debug Records for remote.example not in cache, sending query (thread: 0x17eb3e0)...
error Error sending DNS query: Too many open files
debug Records for remote.example not in cache, sending query (thread: 0x16fbb70)...
error Error sending DNS query: Too many open files
debug DNS lookup failed to get a response for remote.exampleinfo Out of connection options, can't connect to remote.example
debug No other records to try for remote.example - destroying
debug Destroying outgoing session example.local->remote.example: DNS resolution failed: Too many open files
debug First attempt to connect to remote.example, starting with SRV lookup...
debug Records for _xmpp-server._tcp.remote.example. not in cache, sending query (thread: 0x1817df0)...
... then nothing until s2s_timeout
Zash
on
Changes
tags Milestone-0.9
Zash
on
This appears to be because net.dns adds an entry into the "waiting for response" set before getting a socket and sending the query, which can fail. The next query sees that it is already waiting for an answer and writes itself up there too. But since the query packet was never sent, no answer is ever received.
Changes
tags Status-Started
owner Zash
Zash
on
Fixed in 388281be2bc7
Changes
tags Status-Fixed
s
on
Error still present:
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:47 adns error Error sending DNS query: Too many open files
[...]
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
use_libevent = true
# prosodyctl about
Prosody 0.10.0
# Prosody directories
Data directory: /var/lib/prosody
Config directory: /etc/prosody
Source directory: /usr/lib64/prosody
Plugin directories:
/opt/prosody-modules - prosody-modules rev: 0fb95dc11bc8
/usr/lib64/prosody/modules/
# Lua environment
Lua version: Lua 5.1
Lua module search paths:
/usr/lib64/prosody/?.lua
/usr/share/lua/5.1/?.lua
/usr/share/lua/5.1/?/init.lua
/usr/lib64/lua/5.1/?.lua
/usr/lib64/lua/5.1/?/init.lua
/root/.luarocks/share/lua/5.1/?.lua
/root/.luarocks/share/lua/5.1/?/init.lua
Lua C module search paths:
/usr/lib64/prosody/?.so
/usr/lib64/lua/5.1/?.so
/usr/lib64/lua/5.1/loadall.so
/root/.luarocks/lib64/lua/5.1/?.so
LuaRocks: Installed (2.3.0)
# Lua module versions
lfs: LuaFileSystem 1.6.2
libevent: 2.0.21-stable
luaevent: 0.4.4
lxp: LuaExpat 1.3.0
socket: LuaSocket 3.0-rc1
ssl: 0.6
May 17 05:42:25 c2s8a57410 debug Received[c2s]: <iq id='mLgIs-3703' type='get' to='XXX@gabber-germany.de'>
May 17 05:42:25 stanzarouter debug Routing to remote...
May 17 05:42:25 mod_s2s debug opening a new outgoing connection for this stanza
May 17 05:42:25 mod_s2s debug stanza [iq] queued until connection complete
May 17 05:42:25 mod_s2s debug First attempt to connect to gabber-germany.de, starting with SRV lookup...
May 17 05:42:25 adns debug Records for _xmpp-server._tcp.gabber-germany.de. not in cache, sending query (thread: 0x1e4ff540)...
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 mod_s2s debug gabber-germany.de has no SRV records, falling back to A/AAAA
May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x16c3a2d0)...
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files
May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x1de81300)...
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files
May 17 05:42:25 mod_s2s debug DNS lookup failed to get a response for gabber-germany.de
May 17 05:42:25 s2sout156c8720 info Failed in all attempts to connect to gabber-germany.de
May 17 05:42:25 mod_s2s debug No other records to try for gabber-germany.de - destroying
May 17 05:42:25 s2sout156c8720 debug Destroying outgoing session XXX->gabber-germany.de: DNS resolution failed: Too many open files
[...]
May 17 07:10:06 mod_s2s debug opening a new outgoing connection for this stanza
May 17 07:10:06 mod_s2s debug stanza [db:verify] queued until connection complete
May 17 07:10:06 mod_s2s debug First attempt to connect to tlabs.de, starting with SRV lookup...
May 17 07:10:06 adns debug Records for _xmpp-server._tcp.tlabs.de. not in cache, sending query (thread: 0x6f39b30)...
May 17 07:10:06 adns error Error sending DNS query: Too many open files
May 17 07:10:06 mod_s2s debug tlabs.de has no SRV records, falling back to A/AAAA
May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0xf19aaf0)...
May 17 07:10:06 adns error Error sending DNS query: Too many open files
May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files
May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0x37855430)...
May 17 07:10:06 adns error Error sending DNS query: Too many open files
May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files
May 17 07:10:06 mod_s2s debug DNS lookup failed to get a response for tlabs.de
May 17 07:10:06 s2sout3bdf25e0 info Failed in all attempts to connect to tlabs.de
May 17 07:10:06 mod_s2s debug No other records to try for tlabs.de - destroying
May 17 07:10:06 s2sout3bdf25e0 debug Destroying outgoing session XXX->tlabs.de: DNS resolution failed: Too many open files
s
on
Error still present:
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:47 adns error Error sending DNS query: Too many open files
[...]
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
May 17 07:11:12 adns error Error sending DNS query: Too many open files
use_libevent = true
# prosodyctl about
Prosody 0.10.0
# Prosody directories
Data directory: /var/lib/prosody
Config directory: /etc/prosody
Source directory: /usr/lib64/prosody
Plugin directories:
/opt/prosody-modules - prosody-modules rev: 0fb95dc11bc8
/usr/lib64/prosody/modules/
# Lua environment
Lua version: Lua 5.1
Lua module search paths:
/usr/lib64/prosody/?.lua
/usr/share/lua/5.1/?.lua
/usr/share/lua/5.1/?/init.lua
/usr/lib64/lua/5.1/?.lua
/usr/lib64/lua/5.1/?/init.lua
/root/.luarocks/share/lua/5.1/?.lua
/root/.luarocks/share/lua/5.1/?/init.lua
Lua C module search paths:
/usr/lib64/prosody/?.so
/usr/lib64/lua/5.1/?.so
/usr/lib64/lua/5.1/loadall.so
/root/.luarocks/lib64/lua/5.1/?.so
LuaRocks: Installed (2.3.0)
# Lua module versions
lfs: LuaFileSystem 1.6.2
libevent: 2.0.21-stable
luaevent: 0.4.4
lxp: LuaExpat 1.3.0
socket: LuaSocket 3.0-rc1
ssl: 0.6
May 17 05:42:25 c2s8a57410 debug Received[c2s]: <iq id='mLgIs-3703' type='get' to='XXX@gabber-germany.de'>
May 17 05:42:25 stanzarouter debug Routing to remote...
May 17 05:42:25 mod_s2s debug opening a new outgoing connection for this stanza
May 17 05:42:25 mod_s2s debug stanza [iq] queued until connection complete
May 17 05:42:25 mod_s2s debug First attempt to connect to gabber-germany.de, starting with SRV lookup...
May 17 05:42:25 adns debug Records for _xmpp-server._tcp.gabber-germany.de. not in cache, sending query (thread: 0x1e4ff540)...
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 mod_s2s debug gabber-germany.de has no SRV records, falling back to A/AAAA
May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x16c3a2d0)...
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files
May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x1de81300)...
May 17 05:42:25 adns error Error sending DNS query: Too many open files
May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files
May 17 05:42:25 mod_s2s debug DNS lookup failed to get a response for gabber-germany.de
May 17 05:42:25 s2sout156c8720 info Failed in all attempts to connect to gabber-germany.de
May 17 05:42:25 mod_s2s debug No other records to try for gabber-germany.de - destroying
May 17 05:42:25 s2sout156c8720 debug Destroying outgoing session XXX->gabber-germany.de: DNS resolution failed: Too many open files
[...]
May 17 07:10:06 mod_s2s debug opening a new outgoing connection for this stanza
May 17 07:10:06 mod_s2s debug stanza [db:verify] queued until connection complete
May 17 07:10:06 mod_s2s debug First attempt to connect to tlabs.de, starting with SRV lookup...
May 17 07:10:06 adns debug Records for _xmpp-server._tcp.tlabs.de. not in cache, sending query (thread: 0x6f39b30)...
May 17 07:10:06 adns error Error sending DNS query: Too many open files
May 17 07:10:06 mod_s2s debug tlabs.de has no SRV records, falling back to A/AAAA
May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0xf19aaf0)...
May 17 07:10:06 adns error Error sending DNS query: Too many open files
May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files
May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0x37855430)...
May 17 07:10:06 adns error Error sending DNS query: Too many open files
May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files
May 17 07:10:06 mod_s2s debug DNS lookup failed to get a response for tlabs.de
May 17 07:10:06 s2sout3bdf25e0 info Failed in all attempts to connect to tlabs.de
May 17 07:10:06 mod_s2s debug No other records to try for tlabs.de - destroying
May 17 07:10:06 s2sout3bdf25e0 debug Destroying outgoing session XXX->tlabs.de: DNS resolution failed: Too many open files
debug Records for _xmpp-server._tcp.remote.example. not in cache, sending query (thread: 0x18802f0)... error Error sending DNS query: Too many open files debug remote.example has no SRV records, falling back to A/AAAA debug Records for remote.example not in cache, sending query (thread: 0x17eb3e0)... error Error sending DNS query: Too many open files debug Records for remote.example not in cache, sending query (thread: 0x16fbb70)... error Error sending DNS query: Too many open files debug DNS lookup failed to get a response for remote.exampleinfo Out of connection options, can't connect to remote.example debug No other records to try for remote.example - destroying debug Destroying outgoing session example.local->remote.example: DNS resolution failed: Too many open files debug First attempt to connect to remote.example, starting with SRV lookup... debug Records for _xmpp-server._tcp.remote.example. not in cache, sending query (thread: 0x1817df0)... ... then nothing until s2s_timeout
This appears to be because net.dns adds an entry into the "waiting for response" set before getting a socket and sending the query, which can fail. The next query sees that it is already waiting for an answer and writes itself up there too. But since the query packet was never sent, no answer is ever received.
ChangesFixed in 388281be2bc7
ChangesError still present: May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:47 adns error Error sending DNS query: Too many open files [...] May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files use_libevent = true # prosodyctl about Prosody 0.10.0 # Prosody directories Data directory: /var/lib/prosody Config directory: /etc/prosody Source directory: /usr/lib64/prosody Plugin directories: /opt/prosody-modules - prosody-modules rev: 0fb95dc11bc8 /usr/lib64/prosody/modules/ # Lua environment Lua version: Lua 5.1 Lua module search paths: /usr/lib64/prosody/?.lua /usr/share/lua/5.1/?.lua /usr/share/lua/5.1/?/init.lua /usr/lib64/lua/5.1/?.lua /usr/lib64/lua/5.1/?/init.lua /root/.luarocks/share/lua/5.1/?.lua /root/.luarocks/share/lua/5.1/?/init.lua Lua C module search paths: /usr/lib64/prosody/?.so /usr/lib64/lua/5.1/?.so /usr/lib64/lua/5.1/loadall.so /root/.luarocks/lib64/lua/5.1/?.so LuaRocks: Installed (2.3.0) # Lua module versions lfs: LuaFileSystem 1.6.2 libevent: 2.0.21-stable luaevent: 0.4.4 lxp: LuaExpat 1.3.0 socket: LuaSocket 3.0-rc1 ssl: 0.6 May 17 05:42:25 c2s8a57410 debug Received[c2s]: <iq id='mLgIs-3703' type='get' to='XXX@gabber-germany.de'> May 17 05:42:25 stanzarouter debug Routing to remote... May 17 05:42:25 mod_s2s debug opening a new outgoing connection for this stanza May 17 05:42:25 mod_s2s debug stanza [iq] queued until connection complete May 17 05:42:25 mod_s2s debug First attempt to connect to gabber-germany.de, starting with SRV lookup... May 17 05:42:25 adns debug Records for _xmpp-server._tcp.gabber-germany.de. not in cache, sending query (thread: 0x1e4ff540)... May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 mod_s2s debug gabber-germany.de has no SRV records, falling back to A/AAAA May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x16c3a2d0)... May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x1de81300)... May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files May 17 05:42:25 mod_s2s debug DNS lookup failed to get a response for gabber-germany.de May 17 05:42:25 s2sout156c8720 info Failed in all attempts to connect to gabber-germany.de May 17 05:42:25 mod_s2s debug No other records to try for gabber-germany.de - destroying May 17 05:42:25 s2sout156c8720 debug Destroying outgoing session XXX->gabber-germany.de: DNS resolution failed: Too many open files [...] May 17 07:10:06 mod_s2s debug opening a new outgoing connection for this stanza May 17 07:10:06 mod_s2s debug stanza [db:verify] queued until connection complete May 17 07:10:06 mod_s2s debug First attempt to connect to tlabs.de, starting with SRV lookup... May 17 07:10:06 adns debug Records for _xmpp-server._tcp.tlabs.de. not in cache, sending query (thread: 0x6f39b30)... May 17 07:10:06 adns error Error sending DNS query: Too many open files May 17 07:10:06 mod_s2s debug tlabs.de has no SRV records, falling back to A/AAAA May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0xf19aaf0)... May 17 07:10:06 adns error Error sending DNS query: Too many open files May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0x37855430)... May 17 07:10:06 adns error Error sending DNS query: Too many open files May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files May 17 07:10:06 mod_s2s debug DNS lookup failed to get a response for tlabs.de May 17 07:10:06 s2sout3bdf25e0 info Failed in all attempts to connect to tlabs.de May 17 07:10:06 mod_s2s debug No other records to try for tlabs.de - destroying May 17 07:10:06 s2sout3bdf25e0 debug Destroying outgoing session XXX->tlabs.de: DNS resolution failed: Too many open files
Error still present: May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:47 adns error Error sending DNS query: Too many open files [...] May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files May 17 07:11:12 adns error Error sending DNS query: Too many open files use_libevent = true # prosodyctl about Prosody 0.10.0 # Prosody directories Data directory: /var/lib/prosody Config directory: /etc/prosody Source directory: /usr/lib64/prosody Plugin directories: /opt/prosody-modules - prosody-modules rev: 0fb95dc11bc8 /usr/lib64/prosody/modules/ # Lua environment Lua version: Lua 5.1 Lua module search paths: /usr/lib64/prosody/?.lua /usr/share/lua/5.1/?.lua /usr/share/lua/5.1/?/init.lua /usr/lib64/lua/5.1/?.lua /usr/lib64/lua/5.1/?/init.lua /root/.luarocks/share/lua/5.1/?.lua /root/.luarocks/share/lua/5.1/?/init.lua Lua C module search paths: /usr/lib64/prosody/?.so /usr/lib64/lua/5.1/?.so /usr/lib64/lua/5.1/loadall.so /root/.luarocks/lib64/lua/5.1/?.so LuaRocks: Installed (2.3.0) # Lua module versions lfs: LuaFileSystem 1.6.2 libevent: 2.0.21-stable luaevent: 0.4.4 lxp: LuaExpat 1.3.0 socket: LuaSocket 3.0-rc1 ssl: 0.6 May 17 05:42:25 c2s8a57410 debug Received[c2s]: <iq id='mLgIs-3703' type='get' to='XXX@gabber-germany.de'> May 17 05:42:25 stanzarouter debug Routing to remote... May 17 05:42:25 mod_s2s debug opening a new outgoing connection for this stanza May 17 05:42:25 mod_s2s debug stanza [iq] queued until connection complete May 17 05:42:25 mod_s2s debug First attempt to connect to gabber-germany.de, starting with SRV lookup... May 17 05:42:25 adns debug Records for _xmpp-server._tcp.gabber-germany.de. not in cache, sending query (thread: 0x1e4ff540)... May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 mod_s2s debug gabber-germany.de has no SRV records, falling back to A/AAAA May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x16c3a2d0)... May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files May 17 05:42:25 adns debug Records for gabber-germany.de not in cache, sending query (thread: 0x1de81300)... May 17 05:42:25 adns error Error sending DNS query: Too many open files May 17 05:42:25 mod_s2s debug Error in DNS lookup: Too many open files May 17 05:42:25 mod_s2s debug DNS lookup failed to get a response for gabber-germany.de May 17 05:42:25 s2sout156c8720 info Failed in all attempts to connect to gabber-germany.de May 17 05:42:25 mod_s2s debug No other records to try for gabber-germany.de - destroying May 17 05:42:25 s2sout156c8720 debug Destroying outgoing session XXX->gabber-germany.de: DNS resolution failed: Too many open files [...] May 17 07:10:06 mod_s2s debug opening a new outgoing connection for this stanza May 17 07:10:06 mod_s2s debug stanza [db:verify] queued until connection complete May 17 07:10:06 mod_s2s debug First attempt to connect to tlabs.de, starting with SRV lookup... May 17 07:10:06 adns debug Records for _xmpp-server._tcp.tlabs.de. not in cache, sending query (thread: 0x6f39b30)... May 17 07:10:06 adns error Error sending DNS query: Too many open files May 17 07:10:06 mod_s2s debug tlabs.de has no SRV records, falling back to A/AAAA May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0xf19aaf0)... May 17 07:10:06 adns error Error sending DNS query: Too many open files May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files May 17 07:10:06 adns debug Records for tlabs.de not in cache, sending query (thread: 0x37855430)... May 17 07:10:06 adns error Error sending DNS query: Too many open files May 17 07:10:06 mod_s2s debug Error in DNS lookup: Too many open files May 17 07:10:06 mod_s2s debug DNS lookup failed to get a response for tlabs.de May 17 07:10:06 s2sout3bdf25e0 info Failed in all attempts to connect to tlabs.de May 17 07:10:06 mod_s2s debug No other records to try for tlabs.de - destroying May 17 07:10:06 s2sout3bdf25e0 debug Destroying outgoing session XXX->tlabs.de: DNS resolution failed: Too many open files