prosody-0.9 1nightly255-1~sid running on Debian amd64
Jan 14 18:06:43 mod_c2s error Traceback[c2s]: constant table overflow
stack traceback:
[C]: in function 'loadfile'
/usr/lib/prosody/util/envload.lua:20: in function 'envloadfile'
/usr/lib/prosody/util/datamanager.lua:264: in function 'list_load'
/usr/lib/prosody/modules/mod_offline.lua:40: in function '?'
/usr/lib/prosody/util/events.lua:67: in function </usr/lib/prosody/util/events.lua:63>
(tail call): ?
/usr/lib/prosody/modules/mod_presence.lua:123: in function 'handle_normal_presence'
/usr/lib/prosody/modules/mod_presence.lua:325: in function '?'
/usr/lib/prosody/util/events.lua:67: in function 'fire_event'
/usr/lib/prosody/core/stanza_router.lua:187: in function 'core_post_stanza'
...
/usr/lib/prosody/util/xmppstream.lua:255: in function 'feed'
/usr/lib/prosody/modules/mod_c2s.lua:230: in function 'data'
/usr/lib/prosody/modules/mod_c2s.lua:252: in function 'onincoming'
/usr/lib/prosody/net/server_event.lua:632: in function </usr/lib/prosody/net/server_event.lua:578>
[C]: in function 'loop'
/usr/lib/prosody/net/server_event.lua:793: in function </usr/lib/prosody/net/server_event.lua:792>
[C]: in function 'xpcall'
/usr/bin/prosody:376: in function 'loop'
/usr/bin/prosody:407: in main chunk
[C]: ?
Zash
on
This is because there are too many offline messages for prosody to parse them back in one go.
Fixing this would involve changing how 'list' stores are read so that they would read in smaller chunks.
It would be helpful to know exactly how large the offline message store for the user who caused this is.
Also see #577
Changes
tagStatus-Accepted
tagDifficulty-Easy
Zash
on
Changes
tagsDifficulty-Easy
Thomas
on
I see the same error right now on my prosody 0.10.3
@Zash How can i find out how large the offline message storage is?
Here is the stack trace with new line numbers according v0.10.3
Mar 18 11:12:56 mod_c2s error Traceback[c2s]: constant table overflow
stack traceback:
[C]: in function 'load'
/usr/lib64/prosody/util/envload.lua:24: in function 'envloadfile'
/usr/lib64/prosody/util/datamanager.lua:290: in function 'list_load'
/usr/lib64/prosody/modules/mod_storage_internal.lua:64: in function 'find'
/usr/lib64/prosody/modules/mod_mam/mod_mam.lua:145: in function '?'
/usr/lib64/prosody/util/events.lua:78: in function </usr/lib64/prosody/util/events.lua:74>
(tail call): ?
(tail call): ?
(tail call): ?
/usr/lib64/prosody/util/events.lua:78: in function </usr/lib64/prosody/util/events.lua:74>
...
[C]: in function 'parse'
/usr/lib64/prosody/util/xmppstream.lua:271: in function 'feed'
/usr/lib64/prosody/modules/mod_c2s.lua:265: in function 'data'
/usr/lib64/prosody/modules/mod_c2s.lua:288: in function </usr/lib64/prosody/modules/mod_c2s.lua:285>
(tail call): ?
/usr/lib64/prosody/net/server_select.lua:879: in function </usr/lib64/prosody/net/server_select.lua:861>
[C]: in function 'xpcall'
/usr/lib64/prosody/../../bin/prosody:400: in function 'loop'
/usr/lib64/prosody/../../bin/prosody:431: in main chunk
[C]: ?
Thomas
on
Just updated to 0.11.2
Stack trace does not appear in log any more, but error remains. The following line appears in prosody.err:
"Mar 19 22:02:45 datamanager error Failed to load archive storage ('constant table overflow') for user: ...."
Zash
on
If you know which user has the problematic offline store then it should reside in (assuming common *nix install location) /var/lib/prosody/encoded hostname/offline/encoded username.list
If you rename this file then the errors should go away. It would be helpful if you could inspect the file and report its size, number of items, If you could try to determine whether this was legitimate messages, a DDoS attack or spam etc.
Thomas
on
@Zash: It is a common linux installation and I could locate the files.
The error occurs for two users with legitimate message (no DDoS).
Only one of the affected users has a file in the "offline" folder, but it is a rather small file (8kB). However, both affected users have a large file (~65 MB) in "archive". 'grep item archive/$USER.list | wc -l' shows about 45000 items.
Would it help to convert the archive storage to SQL, or will I run in the same problem using a different store?
Zash
on
Oh, your message said offline store but the traceback points to the archive. What is your `archive_expires_after` setting?
Thomas
on
The problem seems to occur when either archive or offline storage become large.
Apparently, archive_expires_after is set to "never".
Zash
on
archive_expires_after = "never" with storage = "internal" is not recommended, precisely because it stops working like this. It probably does not perform very well even before it runs into the hard limit.
If you want very large archives then you should use a different storage backend for now. Eg SQL (avoid MySQL) or mod_storage_xmlarchive from
Thanks for the numbers. We are working towards having a hard limit on number of items in archives, which would prevent them from growing too large to read. See #733
Zash
on
mod_storage_xmlarchive from prosody-modules*
Thomas
on
@Zash thank you for your explanation. As i like to have larger archives, I will try to migrate to sql storage. Why should I avoid MySQL (I often seen guides to avoid sqlite an go for MySQL/MariaDB)?
Zash
on
SQL, especially MySQL, has mostly been painful for us, see issues like#268 and #1064
This is going off-topic, you're welcome to hang in the chatroom if you want to discuss further.
Zash
on
Lua 5.2 seems to handle a lot more than Lua 5.1
An archive with 165k items, Lua 5.1 fails, Lua 5.2 and Lua 5.3 parses it.
I observed luac5.1 under strace to find approximately how far it managed, which seemed like ~55k items.
So I wonder if this issue will magically go away when moving towards Lua 5.2.
prosody-0.9 1nightly255-1~sid running on Debian amd64 Jan 14 18:06:43 mod_c2s error Traceback[c2s]: constant table overflow stack traceback: [C]: in function 'loadfile' /usr/lib/prosody/util/envload.lua:20: in function 'envloadfile' /usr/lib/prosody/util/datamanager.lua:264: in function 'list_load' /usr/lib/prosody/modules/mod_offline.lua:40: in function '?' /usr/lib/prosody/util/events.lua:67: in function </usr/lib/prosody/util/events.lua:63> (tail call): ? /usr/lib/prosody/modules/mod_presence.lua:123: in function 'handle_normal_presence' /usr/lib/prosody/modules/mod_presence.lua:325: in function '?' /usr/lib/prosody/util/events.lua:67: in function 'fire_event' /usr/lib/prosody/core/stanza_router.lua:187: in function 'core_post_stanza' ... /usr/lib/prosody/util/xmppstream.lua:255: in function 'feed' /usr/lib/prosody/modules/mod_c2s.lua:230: in function 'data' /usr/lib/prosody/modules/mod_c2s.lua:252: in function 'onincoming' /usr/lib/prosody/net/server_event.lua:632: in function </usr/lib/prosody/net/server_event.lua:578> [C]: in function 'loop' /usr/lib/prosody/net/server_event.lua:793: in function </usr/lib/prosody/net/server_event.lua:792> [C]: in function 'xpcall' /usr/bin/prosody:376: in function 'loop' /usr/bin/prosody:407: in main chunk [C]: ?
This is because there are too many offline messages for prosody to parse them back in one go. Fixing this would involve changing how 'list' stores are read so that they would read in smaller chunks. It would be helpful to know exactly how large the offline message store for the user who caused this is. Also see #577
ChangesDifficulty-EasyI see the same error right now on my prosody 0.10.3 @Zash How can i find out how large the offline message storage is? Here is the stack trace with new line numbers according v0.10.3 Mar 18 11:12:56 mod_c2s error Traceback[c2s]: constant table overflow stack traceback: [C]: in function 'load' /usr/lib64/prosody/util/envload.lua:24: in function 'envloadfile' /usr/lib64/prosody/util/datamanager.lua:290: in function 'list_load' /usr/lib64/prosody/modules/mod_storage_internal.lua:64: in function 'find' /usr/lib64/prosody/modules/mod_mam/mod_mam.lua:145: in function '?' /usr/lib64/prosody/util/events.lua:78: in function </usr/lib64/prosody/util/events.lua:74> (tail call): ? (tail call): ? (tail call): ? /usr/lib64/prosody/util/events.lua:78: in function </usr/lib64/prosody/util/events.lua:74> ... [C]: in function 'parse' /usr/lib64/prosody/util/xmppstream.lua:271: in function 'feed' /usr/lib64/prosody/modules/mod_c2s.lua:265: in function 'data' /usr/lib64/prosody/modules/mod_c2s.lua:288: in function </usr/lib64/prosody/modules/mod_c2s.lua:285> (tail call): ? /usr/lib64/prosody/net/server_select.lua:879: in function </usr/lib64/prosody/net/server_select.lua:861> [C]: in function 'xpcall' /usr/lib64/prosody/../../bin/prosody:400: in function 'loop' /usr/lib64/prosody/../../bin/prosody:431: in main chunk [C]: ?
Just updated to 0.11.2 Stack trace does not appear in log any more, but error remains. The following line appears in prosody.err: "Mar 19 22:02:45 datamanager error Failed to load archive storage ('constant table overflow') for user: ...."
If you know which user has the problematic offline store then it should reside in (assuming common *nix install location) /var/lib/prosody/encoded hostname/offline/encoded username.list If you rename this file then the errors should go away. It would be helpful if you could inspect the file and report its size, number of items, If you could try to determine whether this was legitimate messages, a DDoS attack or spam etc.
@Zash: It is a common linux installation and I could locate the files. The error occurs for two users with legitimate message (no DDoS). Only one of the affected users has a file in the "offline" folder, but it is a rather small file (8kB). However, both affected users have a large file (~65 MB) in "archive". 'grep item archive/$USER.list | wc -l' shows about 45000 items. Would it help to convert the archive storage to SQL, or will I run in the same problem using a different store?
Oh, your message said offline store but the traceback points to the archive. What is your `archive_expires_after` setting?
The problem seems to occur when either archive or offline storage become large. Apparently, archive_expires_after is set to "never".
archive_expires_after = "never" with storage = "internal" is not recommended, precisely because it stops working like this. It probably does not perform very well even before it runs into the hard limit. If you want very large archives then you should use a different storage backend for now. Eg SQL (avoid MySQL) or mod_storage_xmlarchive from Thanks for the numbers. We are working towards having a hard limit on number of items in archives, which would prevent them from growing too large to read. See #733
mod_storage_xmlarchive from prosody-modules*
@Zash thank you for your explanation. As i like to have larger archives, I will try to migrate to sql storage. Why should I avoid MySQL (I often seen guides to avoid sqlite an go for MySQL/MariaDB)?
SQL, especially MySQL, has mostly been painful for us, see issues like#268 and #1064 This is going off-topic, you're welcome to hang in the chatroom if you want to discuss further.
Lua 5.2 seems to handle a lot more than Lua 5.1 An archive with 165k items, Lua 5.1 fails, Lua 5.2 and Lua 5.3 parses it. I observed luac5.1 under strace to find approximately how far it managed, which seemed like ~55k items. So I wonder if this issue will magically go away when moving towards Lua 5.2.