Hello, apparently, there's a memory leak bug in Prosody 0.11.2. I don't have much information yet, but our Prosody 0.11.2 instance (running on Debian Stretch) irregularly starts eating memory out of the blue until it crashs. I didn't find a pattern yet. So far, the leak bug just happened ~5 times and not in a repetitve time pattern. Also, I didn't find a way to reproduce the memory leak so far. In the logs, I didn't find anything suspicious, either. According to our monitoring, the latest memory leak started pretty exactly at 6pm, but I didn't find a suspicous cronjob or queue task that started at that particular time. Below, I attach a few prosody tracebacks that happened during the time the memory consumption kept growing. See the following graphs (monitoring of prosody memory usage) which show the memory leaks: * By day (shows one memory leak, Prosody restarted before it crashed): https://imgur.com/DrJpMY9 * By week (also shows just the same one memory leak): https://imgur.com/Uiw2YjK * By month (shows three memory leaks, the second one resulted in crashed and dead prosody): https://imgur.com/TzRYZ5i And here's the prosody logs that I found suspicous in the timeframe when the memory leak happened last time. But please note that none of the logs were written at 6pm (when the leak started according to our monitoring). The first traceback is from more than 90 minutes later (7:35pm). Not sure, whether the logs are related to the memory leak at all: Jan 31 19:35:30 jabber prosody[327]: stanzarouter: Received stanza with invalid destination JID: /dev/null Jan 31 19:35:41 jabber prosody[327]: c2s55647b2f14b0: Traceback[c2s]: /usr/lib/prosody/util/serialization.lua:38: Can't serialize table: table has multiple references#012stack traceback:#12#011[C]: in function 'error'#12#011/usr/lib/prosody/util/serialization.lua:38: in function 'fallback'#12#011/usr/lib/prosody/util/serialization.lua:135: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:220: in function </usr/lib/prosody/util/serialization.lua:218>#12#011(...tail calls...)#12#011/usr/lib/prosody/util/datamanager.lua:206: in function </usr/lib/prosody/util/datamanager.lua:195>#12#011(...tail calls...)#12#011...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:116: in function <...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:114>#12#011(...tail calls...)#12#011...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:240: in function '?'#12#011/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>#12#011(...tail calls...)#12#011/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>#12#011(...tail calls...)#12#011/usr/lib/prosody/core/stanza_router.lua:181: in function 'core_post_stanza'#12#011/usr/lib/prosody/core/stanza_router.lua:127: in function 'core_process_stanza'#12#011/usr/lib/prosody/modules/mod_c2s.lua:276: in function 'func'#12#011/usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125> Jan 31 19:38:40 jabber prosody[327]: stanzarouter: Received stanza with invalid destination JID: /dev/null Jan 31 20:49:01 jabber prosody[327]: general: Top-level error, please report:#12/usr/lib/prosody/util/async.lua:177: stack overflow#012stack traceback:#12#011[C]: in function 'parse'#12#011/usr/lib/prosody/util/xmppstream.lua:282: in function 'feed'#12#011/usr/lib/prosody/modules/mod_c2s.lua:285: in function 'data'#12#011/usr/lib/prosody/modules/mod_c2s.lua:308: in function 'onincoming'#12#011/usr/lib/prosody/net/server_event.lua:620: in function </usr/lib/prosody/net/server_event.lua:569>#12#011[C]: in function 'loop'#12#011/usr/lib/prosody/net/server_event.lua:763: in function </usr/lib/prosody/net/server_event.lua:762>#12#011[C]: in function 'xpcall'#12#011/usr/bin/prosody:80: in function 'loop'#12#011/usr/bin/prosody:90: in main chunk#012#011[C]: in ? Jan 31 20:49:01 jabber prosody[327]: general: Top-level error, please report:#12/usr/lib/prosody/util/format.lua:38: stack overflow Jan 31 20:49:01 jabber prosody[327]: general: #012stack traceback:#12#011/usr/lib/prosody/util/format.lua:38: in function </usr/lib/prosody/util/format.lua:27>#12#011[C]: in function 'gsub'#12#011/usr/lib/prosody/util/format.lua:27: in function 'format'#12#011/usr/lib/prosody/modules/mod_posix.lua:124: in function '?'#12#011/usr/lib/prosody/util/logger.lua:49: in function </usr/lib/prosody/util/logger.lua:47>#12#011(...tail calls...)#12#011/usr/bin/prosody:71: in function </usr/bin/prosody:63>#12#011[C]: in function 'loop'#12#011/usr/lib/prosody/net/server_event.lua:763: in function </usr/lib/prosody/net/server_event.lua:762>#12#011[C]: in function 'xpcall'#12#011/usr/bin/prosody:80: in function 'loop'#12#011/usr/bin/prosody:90: in main chunk#012#011[C]: in ? Jan 31 20:51:04 jabber prosody[327]: c2s55647e57c8b0: Traceback[c2s]: /usr/lib/prosody/util/serialization.lua:38: Can't serialize table: table has multiple references#012stack traceback:#12#011[C]: in function 'error'#12#011/usr/lib/prosody/util/serialization.lua:38: in function 'fallback'#12#011/usr/lib/prosody/util/serialization.lua:135: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:194: in function 'serialize_table'#12#011/usr/lib/prosody/util/serialization.lua:220: in function </usr/lib/prosody/util/serialization.lua:218>#12#011(...tail calls...)#12#011/usr/lib/prosody/util/datamanager.lua:206: in function </usr/lib/prosody/util/datamanager.lua:195>#12#011(...tail calls...)#12#011...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:116: in function <...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:114>#12#011(...tail calls...)#12#011...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:240: in function '?'#12#011/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>#12#011(...tail calls...)#12#011/usr/lib/prosody/util/events.lua:79: in function </usr/lib/prosody/util/events.lua:75>#12#011(...tail calls...)#12#011/usr/lib/prosody/core/stanza_router.lua:181: in function 'core_post_stanza'#12#011/usr/lib/prosody/core/stanza_router.lua:127: in function 'core_process_stanza'#12#011/usr/lib/prosody/modules/mod_c2s.lua:276: in function 'func'#12#011/usr/lib/prosody/util/async.lua:127: in function </usr/lib/prosody/util/async.lua:125>

Thanks for the report. In that unreadable mess mangled by syslog, there are two tracebacks. One looks like (condensed): Jan 31 20:51:04 jabber prosody[327]: c2s55647e57c8b0: Traceback[c2s]: /usr/lib/prosody/util/serialization.lua:38: Can't serialize table: table has multiple references ... ...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:116: in function <...re/prosody-modules/mod_cloud_notify/mod_cloud_notify.lua:114> I thought this problem should have gone away with #711. Please report what 3rd party modules you use, and which versions. The second is the stack overflow from #1267, so please make sure to update that.

• tags Status-NeedInfo

Hey @Zash, thanks for going through our syslog mess. > thought this problem should have gone away with #711. Please report what 3rd party modules you use, and which versions. I just verified that `/usr/lib/prosody/util/stanza.lua` has the patch from #711 applied (we have Prosody 0.11.2 installed from your apt repositories). According to `module:list()`, the following 3rd pardy modules are loaded (filtered the ones from prosody-modules: mod_admin_blocklist mod_admin_message mod_auth_custom_http mod_auto_activate_hosts mod_c2s_conn_throttle mod_c2s_limit_sessions mod_carbons mod_carbons_adhoc mod_cloud_notify mod_compat_dialback mod_component_http mod_csi mod_disable_tls mod_extdisco mod_filter_chatstates mod_http_upload mod_lastlog mod_limit_auth mod_limits mod_list_inactive mod_log_http mod_log_sasl_mech mod_mam mod_mam_adhoc mod_mam_muc mod_measure_client_presence mod_muc_cloud_notify mod_muc_limits mod_muc_log_http mod_muc_ping mod_munin mod_omemo_all_access mod_pep_vcard_avatar mod_private_adhoc mod_register mod_register_redirect mod_register_web mod_reload_modules mod_remote_roster mod_s2s_auth_compat mod_s2s_blacklist mod_saslauth_muc mod_smacks mod_smacks_offline mod_stanza_counter_http mod_storage_memory mod_throttle_presence mod_uptime_presence mod_vcard_muc mod_webpresence > The second is the stack overflow from #1267, so please make sure to update that. We have `lua-event` 0.4.6-0~stretch1 installed from your apt repositories. Apparently, we still run into the stack overflow despite running the supposed-to-be-fixed lua-event code? And yes, we restarted prosody several times in between, so I'm really sure that the newest code is running ;)

What does `prosodyctl about` report as lua-event version? Make sure no stray version is installed from eg luarocks. We are currently investigating memory usage issues and have identified some problems in some community modules that might get fixed soon. In the mean time, try disabling anything you are not absolutely certain that you need.

unrelated comment removed

@Zash: The config doobry is referring to: ``` Prosody 0.11.4 # Prosody directories Data directory: /var/lib/prosody Config directory: /etc/prosody Source directory: /usr/lib/prosody Plugin directories: /usr/local/share/prosody-modules - prosody-modules rev: b5d367798570 /usr/lib/prosody/modules/ # Lua environment Lua version: Lua 5.2 Lua module search paths: /usr/lib/prosody/?.lua /usr/local/share/lua/5.2/?.lua /usr/local/share/lua/5.2/?/init.lua /usr/local/lib/lua/5.2/?.lua /usr/local/lib/lua/5.2/?/init.lua /usr/share/lua/5.2/?.lua /usr/share/lua/5.2/?/init.lua Lua C module search paths: /usr/lib/prosody/?.so /usr/local/lib/lua/5.2/?.so /usr/lib/x86_64-linux-gnu/lua/5.2/?.so /usr/lib/lua/5.2/?.so /usr/local/lib/lua/5.2/loadall.so LuaRocks: Not installed # Network Backend: libevent epoll # Lua module versions lfs: LuaFileSystem 1.6.3 libevent: 2.0.21-stable luaevent: 0.4.6 lxp: LuaExpat 1.3.0 socket: LuaSocket 3.0-rc1 ssl: 0.7.1 ```

Hi, do you still experience this memory issue? Later 0.11.x releases had many fixes for memory usage related issues.

• tags Milestone-0.12