#1212 MUC does not recover from unavailable SQL server during init
Reporter
Dennis Schridde
Owner
Zash
Created
Updated
Stars
★ (1)
Tags
Type-Defect
Priority-Medium
Status-Fixed
Milestone-0.11
Dennis Schridde
on
What steps will reproduce the problem?
1. Configure `sql` as `default_storage`
2. Have the MySQL server be unavailable during Prosody startup
What is the expected output?
All Prosody modules should use the same database connection, which is being reestablished with exponential backoff, if it fails. All modules will be reactivated once the database connection is reestablished. This is also and especially true for initialisation.
What do you see instead?
MUC (and possibly others) become unavailable, if establishing a database connection during initialisation fails.
What version of the product are you using? On what operating system?
0.10.2 on Linux
Please provide any additional information below.
sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
$JVUD_HOSTNAME:storage_sql: Unable to read from database user_index store for <host>: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
mod_log_slow_events: Slow event 'host-activated' took 0.71s: no recognised data
sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
$MUC_HOSTNAME:storage_sql: Unable to read from database persistent store for <host>: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
$MUC_HOSTNAME:muc: Error loading list of persistent rooms from storage. Reload mod_muc or restart to recover.
modulemanager: Error initializing module 'muc' on '$MUC_HOSTNAME': /usr/lib64/prosody/modules/muc/mod_muc.lua:44: Storage error: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
stack traceback:
/usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31>
[C]: in function 'error'
/usr/lib64/prosody/modules/muc/mod_muc.lua:44: in main chunk
[C]: in function 'pcall'
/usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module'
/usr/lib64/prosody/core/modulemanager.lua:255: in function 'load'
/usr/lib64/prosody/core/modulemanager.lua:74: in function </usr/lib64/prosody/core/modulemanager.lua:49>
/usr/lib64/prosody/util/events.lua:78: in function 'handlers'
...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event'
/usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate'
/usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50>
/usr/lib64/prosody/util/events.lua:78: in function 'fire_event'
/usr/bin/prosody:358: in function 'prepare_to_start'
/usr/bin/prosody:427: in main chunk
[C]: at 0x56367c6aa3d0
sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
$MUC_HOSTNAME:storage_sql: Unable to read from database persistent store for <host>: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
$MUC_HOSTNAME:muc: Error loading list of persistent rooms from storage. Reload mod_muc or restart to recover.
modulemanager: Error initializing module 'muc' on '$MUC_HOSTNAME': /usr/lib64/prosody/modules/muc/mod_muc.lua:44: Storage error: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
stack traceback:
/usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31>
[C]: in function 'error'
/usr/lib64/prosody/modules/muc/mod_muc.lua:44: in main chunk
[C]: in function 'pcall'
/usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module'
/usr/lib64/prosody/core/modulemanager.lua:255: in function 'load'
/usr/lib64/prosody/core/moduleapi.lua:170: in function 'depends'
/usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: in main chunk
[C]: in function 'pcall'
/usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module'
/usr/lib64/prosody/core/modulemanager.lua:255: in function 'load'
/usr/lib64/prosody/core/modulemanager.lua:77: in function </usr/lib64/prosody/core/modulemanager.lua:49>
/usr/lib64/prosody/util/events.lua:78: in function 'handlers'
...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event'
/usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate'
/usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50>
/usr/lib64/prosody/util/events.lua:78: in function 'fire_event'
/usr/bin/prosody:358: in function 'prepare_to_start'
/usr/bin/prosody:427: in main chunk
[C]: at 0x56367c6aa3d0
modulemanager: Error initializing module 'mam_muc' on '$MUC_HOSTNAME': /usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: Unable to load required module, mod_muc: /usr/lib64/prosody/modules/muc/mod_muc.lua:44: Storage error: Fail>
stack traceback:
/usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31>
[C]: in function 'error'
/usr/lib64/prosody/modules/muc/mod_muc.lua:44: in main chunk
[C]: in function 'pcall'
/usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module'
/usr/lib64/prosody/core/modulemanager.lua:255: in function 'load'
/usr/lib64/prosody/core/moduleapi.lua:170: in function 'depends'
/usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: in main chunk
[C]: in function 'pcall'
/usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module'
/usr/lib64/prosody/core/modulemanager.lua:255: in function 'load'
/usr/lib64/prosody/core/modulemanager.lua:77: in function </usr/lib64/prosody/core/modulemanager.lua:49>
/usr/lib64/prosody/util/events.lua:78: in function 'handlers'
...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event'
/usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate'
/usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50>
/usr/lib64/prosody/util/events.lua:78: in function 'fire_event'
/usr/bin/prosody:358: in function 'prepare_to_start'
/usr/bin/prosody:427: in main chunk
[C]: at 0x56367c6aa3d0
stack traceback:
/usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31>
[C]: in function 'depends'
/usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: in main chunk
[C]: in function 'pcall'
/usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module'
/usr/lib64/prosody/core/modulemanager.lua:255: in function 'load'
/usr/lib64/prosody/core/modulemanager.lua:77: in function </usr/lib64/prosody/core/modulemanager.lua:49>
/usr/lib64/prosody/util/events.lua:78: in function 'handlers'
...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event'
/usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate'
/usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50>
/usr/lib64/prosody/util/events.lua:78: in function 'fire_event'
/usr/bin/prosody:358: in function 'prepare_to_start'
/usr/bin/prosody:427: in main chunk
[C]: at 0x56367c6aa3d0
[...]
sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
Zash
on
This should already be the case in trunk/0.11, where rooms are loaded on-demand instead of all at once on startup.
What you are seeing here is the fix for #1091. Read and follow the error message:
> Error loading list of persistent rooms from storage. Reload mod_muc or restart to recover.
We decided against backporting the from trunk due to how different that code base has become, instead opting to this hard failure with the above suggestion to reload/restart.
What steps will reproduce the problem? 1. Configure `sql` as `default_storage` 2. Have the MySQL server be unavailable during Prosody startup What is the expected output? All Prosody modules should use the same database connection, which is being reestablished with exponential backoff, if it fails. All modules will be reactivated once the database connection is reestablished. This is also and especially true for initialisation. What do you see instead? MUC (and possibly others) become unavailable, if establishing a database connection during initialisation fails. What version of the product are you using? On what operating system? 0.10.2 on Linux Please provide any additional information below. sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) $JVUD_HOSTNAME:storage_sql: Unable to read from database user_index store for <host>: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) mod_log_slow_events: Slow event 'host-activated' took 0.71s: no recognised data sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) $MUC_HOSTNAME:storage_sql: Unable to read from database persistent store for <host>: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) $MUC_HOSTNAME:muc: Error loading list of persistent rooms from storage. Reload mod_muc or restart to recover. modulemanager: Error initializing module 'muc' on '$MUC_HOSTNAME': /usr/lib64/prosody/modules/muc/mod_muc.lua:44: Storage error: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) stack traceback: /usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31> [C]: in function 'error' /usr/lib64/prosody/modules/muc/mod_muc.lua:44: in main chunk [C]: in function 'pcall' /usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module' /usr/lib64/prosody/core/modulemanager.lua:255: in function 'load' /usr/lib64/prosody/core/modulemanager.lua:74: in function </usr/lib64/prosody/core/modulemanager.lua:49> /usr/lib64/prosody/util/events.lua:78: in function 'handlers' ...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event' /usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate' /usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50> /usr/lib64/prosody/util/events.lua:78: in function 'fire_event' /usr/bin/prosody:358: in function 'prepare_to_start' /usr/bin/prosody:427: in main chunk [C]: at 0x56367c6aa3d0 sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) $MUC_HOSTNAME:storage_sql: Unable to read from database persistent store for <host>: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) $MUC_HOSTNAME:muc: Error loading list of persistent rooms from storage. Reload mod_muc or restart to recover. modulemanager: Error initializing module 'muc' on '$MUC_HOSTNAME': /usr/lib64/prosody/modules/muc/mod_muc.lua:44: Storage error: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111) stack traceback: /usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31> [C]: in function 'error' /usr/lib64/prosody/modules/muc/mod_muc.lua:44: in main chunk [C]: in function 'pcall' /usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module' /usr/lib64/prosody/core/modulemanager.lua:255: in function 'load' /usr/lib64/prosody/core/moduleapi.lua:170: in function 'depends' /usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: in main chunk [C]: in function 'pcall' /usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module' /usr/lib64/prosody/core/modulemanager.lua:255: in function 'load' /usr/lib64/prosody/core/modulemanager.lua:77: in function </usr/lib64/prosody/core/modulemanager.lua:49> /usr/lib64/prosody/util/events.lua:78: in function 'handlers' ...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event' /usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate' /usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50> /usr/lib64/prosody/util/events.lua:78: in function 'fire_event' /usr/bin/prosody:358: in function 'prepare_to_start' /usr/bin/prosody:427: in main chunk [C]: at 0x56367c6aa3d0 modulemanager: Error initializing module 'mam_muc' on '$MUC_HOSTNAME': /usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: Unable to load required module, mod_muc: /usr/lib64/prosody/modules/muc/mod_muc.lua:44: Storage error: Fail> stack traceback: /usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31> [C]: in function 'error' /usr/lib64/prosody/modules/muc/mod_muc.lua:44: in main chunk [C]: in function 'pcall' /usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module' /usr/lib64/prosody/core/modulemanager.lua:255: in function 'load' /usr/lib64/prosody/core/moduleapi.lua:170: in function 'depends' /usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: in main chunk [C]: in function 'pcall' /usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module' /usr/lib64/prosody/core/modulemanager.lua:255: in function 'load' /usr/lib64/prosody/core/modulemanager.lua:77: in function </usr/lib64/prosody/core/modulemanager.lua:49> /usr/lib64/prosody/util/events.lua:78: in function 'handlers' ...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event' /usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate' /usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50> /usr/lib64/prosody/util/events.lua:78: in function 'fire_event' /usr/bin/prosody:358: in function 'prepare_to_start' /usr/bin/prosody:427: in main chunk [C]: at 0x56367c6aa3d0 stack traceback: /usr/lib64/prosody/core/modulemanager.lua:31: in function </usr/lib64/prosody/core/modulemanager.lua:31> [C]: in function 'depends' /usr/lib64/prosody/modules/mod_mam_muc/mod_mam_muc.lua:27: in main chunk [C]: in function 'pcall' /usr/lib64/prosody/core/modulemanager.lua:177: in function 'do_load_module' /usr/lib64/prosody/core/modulemanager.lua:255: in function 'load' /usr/lib64/prosody/core/modulemanager.lua:77: in function </usr/lib64/prosody/core/modulemanager.lua:49> /usr/lib64/prosody/util/events.lua:78: in function 'handlers' ...sody/modules/mod_log_slow_events/mod_log_slow_events.lua:12: in function 'fire_event' /usr/lib64/prosody/core/hostmanager.lua:116: in function 'activate' /usr/lib64/prosody/core/hostmanager.lua:59: in function </usr/lib64/prosody/core/hostmanager.lua:50> /usr/lib64/prosody/util/events.lua:78: in function 'fire_event' /usr/bin/prosody:358: in function 'prepare_to_start' /usr/bin/prosody:427: in main chunk [C]: at 0x56367c6aa3d0 [...] sql: Error in SQL transaction: Failed to connect to database: Can't connect to MySQL server on '$DATABASE_HOSTNAME' (111)
This should already be the case in trunk/0.11, where rooms are loaded on-demand instead of all at once on startup. What you are seeing here is the fix for #1091. Read and follow the error message: > Error loading list of persistent rooms from storage. Reload mod_muc or restart to recover. We decided against backporting the from trunk due to how different that code base has become, instead opting to this hard failure with the above suggestion to reload/restart.
Changes