#593 mod_c2s error: constant table overflow

Reporter Roi
Owner Nobody
Created
Updated
Stars ★★ (2)
Tags
  • Status-Accepted
  • Type-Defect
  • Priority-Medium
  1. Roi on

    prosody-0.9 1nightly255-1~sid running on Debian amd64 Jan 14 18:06:43 mod_c2s error Traceback[c2s]: constant table overflow stack traceback: [C]: in function 'loadfile' /usr/lib/prosody/util/envload.lua:20: in function 'envloadfile' /usr/lib/prosody/util/datamanager.lua:264: in function 'list_load' /usr/lib/prosody/modules/mod_offline.lua:40: in function '?' /usr/lib/prosody/util/events.lua:67: in function </usr/lib/prosody/util/events.lua:63> (tail call): ? /usr/lib/prosody/modules/mod_presence.lua:123: in function 'handle_normal_presence' /usr/lib/prosody/modules/mod_presence.lua:325: in function '?' /usr/lib/prosody/util/events.lua:67: in function 'fire_event' /usr/lib/prosody/core/stanza_router.lua:187: in function 'core_post_stanza' ... /usr/lib/prosody/util/xmppstream.lua:255: in function 'feed' /usr/lib/prosody/modules/mod_c2s.lua:230: in function 'data' /usr/lib/prosody/modules/mod_c2s.lua:252: in function 'onincoming' /usr/lib/prosody/net/server_event.lua:632: in function </usr/lib/prosody/net/server_event.lua:578> [C]: in function 'loop' /usr/lib/prosody/net/server_event.lua:793: in function </usr/lib/prosody/net/server_event.lua:792> [C]: in function 'xpcall' /usr/bin/prosody:376: in function 'loop' /usr/bin/prosody:407: in main chunk [C]: ?

  2. Zash on

    This is because there are too many offline messages for prosody to parse them back in one go. Fixing this would involve changing how 'list' stores are read so that they would read in smaller chunks. It would be helpful to know exactly how large the offline message store for the user who caused this is. Also see #577

    Changes
    • tag Status-Accepted
    • tag Difficulty-Easy
  3. Zash on

    Changes
    • tags Difficulty-Easy
  4. Thomas on

    I see the same error right now on my prosody 0.10.3 @Zash How can i find out how large the offline message storage is? Here is the stack trace with new line numbers according v0.10.3 Mar 18 11:12:56 mod_c2s error Traceback[c2s]: constant table overflow stack traceback: [C]: in function 'load' /usr/lib64/prosody/util/envload.lua:24: in function 'envloadfile' /usr/lib64/prosody/util/datamanager.lua:290: in function 'list_load' /usr/lib64/prosody/modules/mod_storage_internal.lua:64: in function 'find' /usr/lib64/prosody/modules/mod_mam/mod_mam.lua:145: in function '?' /usr/lib64/prosody/util/events.lua:78: in function </usr/lib64/prosody/util/events.lua:74> (tail call): ? (tail call): ? (tail call): ? /usr/lib64/prosody/util/events.lua:78: in function </usr/lib64/prosody/util/events.lua:74> ... [C]: in function 'parse' /usr/lib64/prosody/util/xmppstream.lua:271: in function 'feed' /usr/lib64/prosody/modules/mod_c2s.lua:265: in function 'data' /usr/lib64/prosody/modules/mod_c2s.lua:288: in function </usr/lib64/prosody/modules/mod_c2s.lua:285> (tail call): ? /usr/lib64/prosody/net/server_select.lua:879: in function </usr/lib64/prosody/net/server_select.lua:861> [C]: in function 'xpcall' /usr/lib64/prosody/../../bin/prosody:400: in function 'loop' /usr/lib64/prosody/../../bin/prosody:431: in main chunk [C]: ?

  5. Thomas on

    Just updated to 0.11.2 Stack trace does not appear in log any more, but error remains. The following line appears in prosody.err: "Mar 19 22:02:45 datamanager error Failed to load archive storage ('constant table overflow') for user: ...."

  6. Zash on

    If you know which user has the problematic offline store then it should reside in (assuming common *nix install location) /var/lib/prosody/encoded hostname/offline/encoded username.list If you rename this file then the errors should go away. It would be helpful if you could inspect the file and report its size, number of items, If you could try to determine whether this was legitimate messages, a DDoS attack or spam etc.

  7. Thomas on

    @Zash: It is a common linux installation and I could locate the files. The error occurs for two users with legitimate message (no DDoS). Only one of the affected users has a file in the "offline" folder, but it is a rather small file (8kB). However, both affected users have a large file (~65 MB) in "archive". 'grep item archive/$USER.list | wc -l' shows about 45000 items. Would it help to convert the archive storage to SQL, or will I run in the same problem using a different store?

  8. Zash on

    Oh, your message said offline store but the traceback points to the archive. What is your `archive_expires_after` setting?

  9. Thomas on

    The problem seems to occur when either archive or offline storage become large. Apparently, archive_expires_after is set to "never".

  10. Zash on

    archive_expires_after = "never" with storage = "internal" is not recommended, precisely because it stops working like this. It probably does not perform very well even before it runs into the hard limit. If you want very large archives then you should use a different storage backend for now. Eg SQL (avoid MySQL) or mod_storage_xmlarchive from Thanks for the numbers. We are working towards having a hard limit on number of items in archives, which would prevent them from growing too large to read. See #733

  11. Zash on

    mod_storage_xmlarchive from prosody-modules*

  12. Thomas on

    @Zash thank you for your explanation. As i like to have larger archives, I will try to migrate to sql storage. Why should I avoid MySQL (I often seen guides to avoid sqlite an go for MySQL/MariaDB)?

  13. Zash on

    SQL, especially MySQL, has mostly been painful for us, see issues like#268 and #1064 This is going off-topic, you're welcome to hang in the chatroom if you want to discuss further.

  14. Zash on

    Lua 5.2 seems to handle a lot more than Lua 5.1 An archive with 165k items, Lua 5.1 fails, Lua 5.2 and Lua 5.3 parses it. I observed luac5.1 under strace to find approximately how far it managed, which seemed like ~55k items. So I wonder if this issue will magically go away when moving towards Lua 5.2.

New comment

Not published. Used for spam prevention and optional update notifications.