The setup is as follows:
I run a server (prosody 0.11.2-1+deb10u2, prosody-modules 0.0~hg20190203.b54e98d5c4a1+dfsg-1+deb10u1, Debian buster) on which I have a local account. I also have an account on someone else’s server which uses ejabberd. I use mcabber (Debian bullseye/sid) as client for both.
A contact of mine has one account on one server. I have them as contact for both accounts. This person uses Gajim on Debian buster as client.
When this contact is on the road (losing connectivity intermittently) or sending their laptop to sleep (we’re not yet sure which of the two triggers this problem), there are “ghost sessions” left of that user, but only in my prosody account.
Typing “/info” in mcabber when the user went offline the last time (as seen in my session on the ejabberd server) showed (name redacted):
08-07 20:23 *** jid: <contact.of.mine@jabber.domain.tld>
08-07 20:23 *** Name: ContactName
08-07 20:23 *** Type: user
08-07 20:23 *** Subscription: both
08-07 20:23 *** Resource: [o] (0) 128804408295043009161138
Status timestamp: 2021-07-22 08:20:47
08-07 20:23 *** Resource: [o] (0) 642199390901542646538754
Status timestamp: 2021-07-27 11:00:46
The user went online a bit later, and /info now shows:
08-08 02:46 *** jid: <contact.of.mine@jabber.domain.tld>
08-08 02:46 *** Name: ContactName
08-08 02:46 *** Type: user
08-08 02:46 *** Subscription: both
08-08 02:46 *** Resource: [o] (0) 128804408295043009161138
Status timestamp: 2021-07-22 08:20:47
08-08 02:46 *** Resource: [o] (0) 642199390901542646538754
Status timestamp: 2021-07-27 11:00:46
08-08 02:46 *** Resource: [o] (0) 13754181727725189373119282
Status timestamp: 2021-08-08 00:19:41
This is not the first time the problem occurred. The last time, an /etc/init.d/prosody stop+start fixed the issue, so it’s some kind of transient thing.
I found a related issue on https://community.jitsi.org/t/x-disconnected-user-remains-in-session-prosody-0-11/51770/14 but my configuration isn’t even close to theirs.
My configuration follows (hostname anonymised). I had the csi_simple module enabled before and disabled it because I thought it was the cause of the problem, but the problem persists even with it disabled. The configuration is based on the stock Debian package configuration, with only a few lines changed.
# cat /etc/prosody/migrator.cfg.lua
local data_path = '/var/lib/prosody';
input {
type = "prosody_files";
path = data_path;
}
output {
type = "prosody_sql";
driver = "SQLite3";
database = data_path.."/prosody.sqlite";
}
--[[
input {
type = "prosody_files";
path = data_path;
}
output {
type = "prosody_sql";
driver = "SQLite3";
database = data_path.."/prosody.sqlite";
}
]]
# cat /etc/prosody/prosody.cfg.lua
-- Prosody Example Configuration File
--
-- Information on configuring Prosody can be found on our
-- website at https://prosody.im/doc/configure
--
-- Tip: You can check that the syntax of this file is correct
-- when you have finished by running this command:
-- prosodyctl check config
-- If there are any errors, it will let you know what and where
-- they are, otherwise it will keep quiet.
--
-- The only thing left to do is rename this file to remove the .dist ending, and fill in the
-- blanks. Good luck, and happy Jabbering!
---------- Server-wide settings ----------
-- Settings in this section apply to the whole server and are the default settings
-- for any virtual hosts
-- This is a (by default, empty) list of accounts that are admins
-- for the server. Note that you must create the accounts separately
-- (see https://prosody.im/doc/creating_accounts for info)
-- Example: admins = { "user1@example.com", "user2@example.net" }
admins = { "sysop@myhostname.mydomain.mytld" }
-- Enable use of libevent for better performance under high load
-- For more information see: https://prosody.im/doc/libevent
--use_libevent = true
-- Prosody will always look in its source directory for modules, but
-- this option allows you to specify additional locations where Prosody
-- will look for modules first. For community modules, see https://modules.prosody.im/
-- For a local administrator it's common to place local modifications
-- under /usr/local/ hierarchy:
plugin_paths = { "/usr/local/lib/prosody/modules" }
-- This is the list of modules Prosody will load on startup.
-- It looks for mod_modulename.lua in the plugins folder, so make sure that exists too.
-- Documentation for bundled modules can be found at: https://prosody.im/doc/modules
modules_enabled = {
-- Generally required
"roster"; -- Allow users to have a roster. Recommended ;)
"saslauth"; -- Authentication for clients and servers. Recommended if you want to log in.
"tls"; -- Add support for secure TLS on c2s/s2s connections
"dialback"; -- s2s dialback support
"disco"; -- Service discovery
-- Not essential, but recommended
"carbons"; -- Keep multiple clients in sync
"pep"; -- Enables users to publish their avatar, mood, activity, playing music and more
"private"; -- Private XML storage (for room bookmarks, etc.)
"blocklist"; -- Allow users to block communications with other users
"vcard4"; -- User profiles (stored in PEP)
"vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard
"limits"; -- Enable bandwidth limiting for XMPP connections
-- Nice to have
"version"; -- Replies to server version requests
"uptime"; -- Report how long server has been running
"time"; -- Let others know the time here on this server
"ping"; -- Replies to XMPP pings with pongs
"register"; -- Allow users to register on this server using a client and change passwords
--"mam"; -- Store messages in an archive and allow users to access it
--"csi_simple"; -- Simple Mobile optimizations
-- Admin interfaces
--"admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
--"admin_telnet"; -- Opens telnet console interface on localhost port 5582
-- HTTP modules
--"bosh"; -- Enable BOSH clients, aka "Jabber over HTTP"
--"websocket"; -- XMPP over WebSockets
--"http_files"; -- Serve static files from a directory over HTTP
-- Other specific functionality
"posix"; -- POSIX functionality, sends server to background, enables syslog, etc.
--"groups"; -- Shared roster support
--"server_contact_info"; -- Publish contact information for this service
--"announce"; -- Send announcement to all online users
--"welcome"; -- Welcome users who register accounts
--"watchregistrations"; -- Alert admins of registrations
--"motd"; -- Send a message to users when they log in
"legacyauth"; -- Legacy authentication. Only used by some old clients and bots.
--"proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use
-- meow local additions
"ipcheck";
"lastlog";
"smacks";
}
-- These modules are auto-loaded, but should you want
-- to disable them then uncomment them here:
modules_disabled = {
-- "offline"; -- Store offline messages
-- "c2s"; -- Handle client connections
-- "s2s"; -- Handle server-to-server connections
}
-- Disable account creation by default, for security
-- For more information see https://prosody.im/doc/creating_accounts
allow_registration = false
-- Debian:
-- Do not send the server to background, either systemd or start-stop-daemon take care of that.
--
daemonize = false;
-- Debian:
-- Please, don't change this option since /run/prosody/
-- is one of the few directories Prosody is allowed to write to
--
pidfile = "/run/prosody/prosody.pid";
-- Force clients to use encrypted connections? This option will
-- prevent clients from authenticating unless they are using encryption.
c2s_require_encryption = true
-- Force servers to use encrypted connections? This option will
-- prevent servers from authenticating unless they are using encryption.
s2s_require_encryption = true
-- Force certificate authentication for server-to-server connections?
s2s_secure_auth = true
-- Some servers have invalid or self-signed certificates. You can list
-- remote domains here that will not be required to authenticate using
-- certificates. They will be authenticated using DNS instead, even
-- when s2s_secure_auth is enabled.
--s2s_insecure_domains = { "insecure.example" }
-- Even if you disable s2s_secure_auth, you can still require valid
-- certificates for some domains by specifying a list here.
--s2s_secure_domains = { "jabber.org" }
-- Enable rate limits for incoming client and server connections
limits = {
c2s = {
rate = "10kb/s";
};
s2sin = {
rate = "30kb/s";
};
}
-- Select the authentication backend to use. The 'internal' providers
-- use Prosody's configured data storage to store the authentication data.
authentication = "internal_hashed"
-- Select the storage backend to use. By default Prosody uses flat files
-- in its configured data directory, but it also supports more backends
-- through modules. An "sql" backend is included by default, but requires
-- additional dependencies. See https://prosody.im/doc/storage for more info.
--storage = "sql" -- Default is "internal" (Debian: "sql" requires one of the
-- lua-dbi-sqlite3, lua-dbi-mysql or lua-dbi-postgresql packages to work)
-- For the "sql" backend, you can uncomment *one* of the below to configure:
--sql = { driver = "SQLite3", database = "prosody.sqlite" } -- Default. 'database' is the filename.
--sql = { driver = "MySQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" }
--sql = { driver = "PostgreSQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" }
-- Archiving configuration
-- If mod_mam is enabled, Prosody will store a copy of every message. This
-- is used to synchronize conversations between multiple clients, even if
-- they are offline. This setting controls how long Prosody will keep
-- messages in the archive before removing them.
archive_expires_after = "1w" -- Remove archived messages after 1 week
-- You can also configure messages to be stored in-memory only. For more
-- archiving options, see https://prosody.im/doc/modules/mod_mam
-- Logging configuration
-- For advanced logging see https://prosody.im/doc/logging
--
-- Debian:
-- Logs info and higher to /var/log
-- Logs errors to syslog also
log = {
-- Log files (change 'info' to 'debug' for debug logs):
info = "/var/log/prosody/prosody.log";
error = "/var/log/prosody/prosody.err";
-- Syslog:
{ levels = { "error" }; to = "syslog"; };
}
-- Uncomment to enable statistics
-- For more info see https://prosody.im/doc/statistics
-- statistics = "internal"
-- Certificates
-- Every virtual host and component needs a certificate so that clients and
-- servers can securely verify its identity. Prosody will automatically load
-- certificates/keys from the directory specified here.
-- For more information, including how to use 'prosodyctl' to auto-import certificates
-- (from e.g. Let's Encrypt) see https://prosody.im/doc/certificates
-- Location of directory to find certificates in (relative to main config file):
certificates = "certs"
-- HTTPS currently only supports a single certificate, specify it here:
--https_certificate = "/etc/prosody/certs/localhost.crt"
----------- Virtual hosts -----------
-- You need to add a VirtualHost entry for each domain you wish Prosody to serve.
-- Settings under each VirtualHost entry apply *only* to that host.
-- It's customary to maintain VirtualHost entries in separate config files
-- under /etc/prosody/conf.d/ directory. Examples of such config files can
-- be found in /etc/prosody/conf.avail/ directory.
------ meow local options ------
legacy_ssl_ports = { 5223 }
ignore_presence_priority = true
smacks_max_ack_delay = 3
------ Additional config files ------
-- For organizational purposes you may prefer to add VirtualHost and
-- Component definitions in their own config files. This line includes
-- all config files in /etc/prosody/conf.d/
Include "conf.d/*.cfg.lua"
--VirtualHost "localhost"
--VirtualHost "example.com"
-- certificate = "/path/to/example.crt"
------ Components ------
-- You can specify components to add hosts that provide special services,
-- like multi-user conferences, and transports.
-- For more information on components, see https://prosody.im/doc/components
---Set up a MUC (multi-user chat) room server on conference.example.com:
--Component "conference.example.com" "muc"
--- Store MUC messages in an archive and allow users to access it
--modules_enabled = { "muc_mam" }
---Set up an external component (default component port is 5347)
--
-- External components allow adding various services, such as gateways/
-- transports to other networks like ICQ, MSN and Yahoo. For more info
-- see: https://prosody.im/doc/components#adding_an_external_component
--
--Component "gateway.example.com"
-- component_secret = "password"
# ls -l /etc/prosody/conf.d/
total 0
lrwxrwxrwx 1 root root 37 Jul 30 2019 myhostname.mydomain.mytld.cfg.lua -> ../conf.avail/myhostname.mydomain.mytld.cfg.lua
lrwxrwxrwx 1 root root 31 Jul 30 2019 localhost.cfg.lua -> ../conf.avail/localhost.cfg.lua
# cat /etc/prosody/conf.d/localhost.cfg.lua
-- Section for localhost
-- This allows clients to connect to localhost. No harm in it.
VirtualHost "localhost"
# cat /etc/prosody/conf.d/myhostname.mydomain.mytld.cfg.lua
VirtualHost "myhostname.mydomain.mytld"
Component "conference.myhostname.mydomain.mytld" "muc"
name = "Chaträume"
restrict_room_creation = "admin"
max_history_messages = 500
mirabilos
on
I haven’t restarted the server yet, in case there’s something I can do to help debugging this.
If this is a known issue, upgrading the server to Debian bullseye (prosody 0.11.9-2, prosody-modules 0.0~hg20210130.dd3bfe8f182e+dfsg-2) is an option if it’s fixed there.
mirabilos
on
My SSL certificate hook script restarted it, and the ghost session has gone away. But a proper fix would still be welcome…
mirabilos
on
I have two ghost sessions from that user again, Status timestamp 2nd and 3rd September respectively…
MattJ
on
Hi, thanks for the report, and your patience.
Part of the reason I haven't responded is because I wasn't really sure what to say (and haven't had time to write up in detail exactly why I didn't have anything to say).
Here goes...
Your issue report seems to be based on the premise that Prosody is tracking the sessions of your contacts, and telling you their status. The assumption then is that Prosody is, for some reason, keeping contact sessions online after they have gone offline (the "ghosts").
The problem is, that's not how it works. Prosody only tracks *your* sessions, and has no idea about the status of your contacts. Prosody only does two things related to that:
A) When you start a new session with Prosody, Prosody will notify all your contacts and ask them to send their latest status.
B) If your contacts reply, Prosody will forward their replies to your online client(s). Your contacts may not reply (if offline), and they may send additional updates whenever their status changes.
Again, Prosody is only acting as a router of this information, it is not storing or tracking the status of your contacts in any way.
This means that if you are seeing contacts online who are offline, the possible problems are one of:
1) Your contacts are actually still online
2) Your contact's server did not send out a status change notification when they went offline
3) Your contact's server sent a notification when your contact offline, but it could not be delivered to your server
4) Your contact's server sent a notification when your contact offline, and Prosody received it, but for some reason did not route it to your client
5) Your contact's server sent a notification when your contact offline, and Prosody received it and routed it, but your client did not correctly process it.
Issue (1) seems like it's not an issue, but it can be unintuitive behaviour caused by XEP-0198 support in your contact's server (implemented in mod_smacks for Prosody). XEP-0198 will keep client sessions online even if they lose connectivity, so they can later be resumed. Some people set their XEP-0198 stupidly high, which makes this even more apparent, especially when they are experiencing connectivity issues and do not shut down their connection properly.
Issue (2) would likely be a bug or other problem with your contact's server, or a problem with your roster subscription state being corrupted. Rosters should self-heal, but if not, again, it would be a bug.
Issue (3) could be caused by any intermittent network issues between your contact's server and yours. The potential for such issues could be increased by hosting your server on a dynamic IP address, or behind a NAT router that implements aggressive idle connection timeouts. If you think this issue may be affecting you, look into enabling https://modules.prosody.im/mod_s2s_keepalive
Issue (4) could be caused by a bug in Prosody (this is very unlikely - the presence handling code is a core part of Prosody and the logic has not changed for a long time). It could also be caused by a buggy module, though I don't see any top suspects in the configuration you provided.
Issue (5) could be caused by a bug in mcabber. You could verify this by connecting a second client to the same account for a while, and see if it exhibits the same issue.
Hope this helps identify the cause of the issue. If you get any further identifying the problem, please let us know.
Changes
tags Status-NeedInfo
mirabilos
on
First, thanks for the very verbose answer!
I think I can exclude a number of possibilities already:
ad 1: The contact is not still online (as confirmed by phone and/or physical proximity).
I’m not sure it’s a point of the configuration of my contact’s server given that another Jabber account I have, on a site running ejabberd, does not show these ghost sessions. But I can certainly enquire about my contact’s server’s configuration if you think this helps in debugging (I’m fairly certain they also use ejabberd), I think she’s at least a co-admin.
ad 2: I have no idea when the roster “self-heals”; after a prosody restart, the ghost sessions are gone, but until then, they seem to live on.
ad 3: nope, my server is a rented virtual machine (qemu-kvm, so “always on”) operating as very low-traffic server on multiple protocols, and the other is a VM on a rented dedicated server (bare metal) operating as multiple low-to-medium-traffic servers. Both IP and Legacy IPv4 traffic are known to be stable on both ends.
ad 4: I take your word for it. I included my versions in case this was a known issue in the version Debian ships.
ad 5: The second account I use I also use mcabber for, so I believe this can also be excluded…
… hmm, that being said, the versions differ. Good point. I’ll have to try to keep the same mcabber version up and connected to both accounts for a while.
A first test (connect with a second mcabber session on the laptop I use for the other account, which however switches between networks very often) is inconclusive: the ghost sessions don’t appear for newly-connected clients. Similarily, exiting and restarting the mcabber-in-screen loses them. (I do not have any other way of determining them to be ghost session, as I’ve seen one with a status timestamp of about 2 hours old, though ☹)
If you have any other ideas I could try, be my guest…
mirabilos
on
Update: I found I can do '/request last' in mcabber to find out whether one of the sessions of that user is indeed online, and I indeed see “Received IQ:last result from <user@host/resource>” and a couple 503 User session not found.
Now if Prosody could just use this 503 to kill the session so the fulljid gets dropped from my roster, leaving me with only the one actually-online session (or none, if all are 503ing) shown as online, I’d be happy. I mean, MUCs do that already.
I am also wondering if asking the other party to configure a static resource instead of their Gajim’s default random one would mitigate this…
As for the lifetime, one of the sessions has a status timestamp of 26 days ago, so no to the self-healing.
MattJ
on
Hi, thanks for the update.
As mentioned in my comment on 2021-09-07, Prosody does not track the online status of your contacts. In particular your suggestion to "kill the session" of your contact and "drop the full jid from your roster" is not possible - there are no sessions for remote contacts, and rosters never contain full JIDs.
The "503 User session not found" is an error returned by ejabberd.
When your contact's session went offline, ejabberd should have sent a notification about this to you (specifically an "unavailable" presence stanza). Prosody will forward this to mcabber and mcabber should then update its UI to show that your contact is now offline.
Either ejabberd is not emitting this stanza, or it's getting lost somewhere on the way to mcabber, or mcabber isn't handling it correctly.
I'd say that ejabberd not emitting the stanza at all is an unlikely scenario (this is one of the most simple and basic operations of XMPP).
My suggestions for further debugging:
I don't see that I asked this already, but... have you checked Prosody's error log? (if you have log rotation, do check older files as well)
If there is nothing in the error log, enable debug logging (change "info" to "debug" in your log option). Then wait until it occurs again, and grep the logs for your contact's full JID. Then we will know if Prosody is receiving the unavailable presence.
I'm also currently working on a new module that will make it easier to find server-to-server connection issues (currently that's hard without continuous monitoring of the log file). When that's finished it may be useful to rule out any s2s issues.
Good luck! I'm sure we can get to the bottom of this pesky problem eventually.
mirabilos
on
Huh, so this may be a mcabber problem after all? I guess I’ll have to look into that more.
All 52 of Prosody’s error logs are empty save for one DNS issue last December.
The mcabber version on the “bouncer”, to use IRC terms, is indeed quite old. I’ll investigate on that front. I’ve seen a plugin-thingy for mcabber to kill ghosts (apparently not part of the standard functionality) but trying to get that set up is also going to be icky… we’ll see.
The setup is as follows: I run a server (prosody 0.11.2-1+deb10u2, prosody-modules 0.0~hg20190203.b54e98d5c4a1+dfsg-1+deb10u1, Debian buster) on which I have a local account. I also have an account on someone else’s server which uses ejabberd. I use mcabber (Debian bullseye/sid) as client for both. A contact of mine has one account on one server. I have them as contact for both accounts. This person uses Gajim on Debian buster as client. When this contact is on the road (losing connectivity intermittently) or sending their laptop to sleep (we’re not yet sure which of the two triggers this problem), there are “ghost sessions” left of that user, but only in my prosody account. Typing “/info” in mcabber when the user went offline the last time (as seen in my session on the ejabberd server) showed (name redacted): 08-07 20:23 *** jid: <contact.of.mine@jabber.domain.tld> 08-07 20:23 *** Name: ContactName 08-07 20:23 *** Type: user 08-07 20:23 *** Subscription: both 08-07 20:23 *** Resource: [o] (0) 128804408295043009161138 Status timestamp: 2021-07-22 08:20:47 08-07 20:23 *** Resource: [o] (0) 642199390901542646538754 Status timestamp: 2021-07-27 11:00:46 The user went online a bit later, and /info now shows: 08-08 02:46 *** jid: <contact.of.mine@jabber.domain.tld> 08-08 02:46 *** Name: ContactName 08-08 02:46 *** Type: user 08-08 02:46 *** Subscription: both 08-08 02:46 *** Resource: [o] (0) 128804408295043009161138 Status timestamp: 2021-07-22 08:20:47 08-08 02:46 *** Resource: [o] (0) 642199390901542646538754 Status timestamp: 2021-07-27 11:00:46 08-08 02:46 *** Resource: [o] (0) 13754181727725189373119282 Status timestamp: 2021-08-08 00:19:41 This is not the first time the problem occurred. The last time, an /etc/init.d/prosody stop+start fixed the issue, so it’s some kind of transient thing. I found a related issue on https://community.jitsi.org/t/x-disconnected-user-remains-in-session-prosody-0-11/51770/14 but my configuration isn’t even close to theirs. My configuration follows (hostname anonymised). I had the csi_simple module enabled before and disabled it because I thought it was the cause of the problem, but the problem persists even with it disabled. The configuration is based on the stock Debian package configuration, with only a few lines changed. # cat /etc/prosody/migrator.cfg.lua local data_path = '/var/lib/prosody'; input { type = "prosody_files"; path = data_path; } output { type = "prosody_sql"; driver = "SQLite3"; database = data_path.."/prosody.sqlite"; } --[[ input { type = "prosody_files"; path = data_path; } output { type = "prosody_sql"; driver = "SQLite3"; database = data_path.."/prosody.sqlite"; } ]] # cat /etc/prosody/prosody.cfg.lua -- Prosody Example Configuration File -- -- Information on configuring Prosody can be found on our -- website at https://prosody.im/doc/configure -- -- Tip: You can check that the syntax of this file is correct -- when you have finished by running this command: -- prosodyctl check config -- If there are any errors, it will let you know what and where -- they are, otherwise it will keep quiet. -- -- The only thing left to do is rename this file to remove the .dist ending, and fill in the -- blanks. Good luck, and happy Jabbering! ---------- Server-wide settings ---------- -- Settings in this section apply to the whole server and are the default settings -- for any virtual hosts -- This is a (by default, empty) list of accounts that are admins -- for the server. Note that you must create the accounts separately -- (see https://prosody.im/doc/creating_accounts for info) -- Example: admins = { "user1@example.com", "user2@example.net" } admins = { "sysop@myhostname.mydomain.mytld" } -- Enable use of libevent for better performance under high load -- For more information see: https://prosody.im/doc/libevent --use_libevent = true -- Prosody will always look in its source directory for modules, but -- this option allows you to specify additional locations where Prosody -- will look for modules first. For community modules, see https://modules.prosody.im/ -- For a local administrator it's common to place local modifications -- under /usr/local/ hierarchy: plugin_paths = { "/usr/local/lib/prosody/modules" } -- This is the list of modules Prosody will load on startup. -- It looks for mod_modulename.lua in the plugins folder, so make sure that exists too. -- Documentation for bundled modules can be found at: https://prosody.im/doc/modules modules_enabled = { -- Generally required "roster"; -- Allow users to have a roster. Recommended ;) "saslauth"; -- Authentication for clients and servers. Recommended if you want to log in. "tls"; -- Add support for secure TLS on c2s/s2s connections "dialback"; -- s2s dialback support "disco"; -- Service discovery -- Not essential, but recommended "carbons"; -- Keep multiple clients in sync "pep"; -- Enables users to publish their avatar, mood, activity, playing music and more "private"; -- Private XML storage (for room bookmarks, etc.) "blocklist"; -- Allow users to block communications with other users "vcard4"; -- User profiles (stored in PEP) "vcard_legacy"; -- Conversion between legacy vCard and PEP Avatar, vcard "limits"; -- Enable bandwidth limiting for XMPP connections -- Nice to have "version"; -- Replies to server version requests "uptime"; -- Report how long server has been running "time"; -- Let others know the time here on this server "ping"; -- Replies to XMPP pings with pongs "register"; -- Allow users to register on this server using a client and change passwords --"mam"; -- Store messages in an archive and allow users to access it --"csi_simple"; -- Simple Mobile optimizations -- Admin interfaces --"admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands --"admin_telnet"; -- Opens telnet console interface on localhost port 5582 -- HTTP modules --"bosh"; -- Enable BOSH clients, aka "Jabber over HTTP" --"websocket"; -- XMPP over WebSockets --"http_files"; -- Serve static files from a directory over HTTP -- Other specific functionality "posix"; -- POSIX functionality, sends server to background, enables syslog, etc. --"groups"; -- Shared roster support --"server_contact_info"; -- Publish contact information for this service --"announce"; -- Send announcement to all online users --"welcome"; -- Welcome users who register accounts --"watchregistrations"; -- Alert admins of registrations --"motd"; -- Send a message to users when they log in "legacyauth"; -- Legacy authentication. Only used by some old clients and bots. --"proxy65"; -- Enables a file transfer proxy service which clients behind NAT can use -- meow local additions "ipcheck"; "lastlog"; "smacks"; } -- These modules are auto-loaded, but should you want -- to disable them then uncomment them here: modules_disabled = { -- "offline"; -- Store offline messages -- "c2s"; -- Handle client connections -- "s2s"; -- Handle server-to-server connections } -- Disable account creation by default, for security -- For more information see https://prosody.im/doc/creating_accounts allow_registration = false -- Debian: -- Do not send the server to background, either systemd or start-stop-daemon take care of that. -- daemonize = false; -- Debian: -- Please, don't change this option since /run/prosody/ -- is one of the few directories Prosody is allowed to write to -- pidfile = "/run/prosody/prosody.pid"; -- Force clients to use encrypted connections? This option will -- prevent clients from authenticating unless they are using encryption. c2s_require_encryption = true -- Force servers to use encrypted connections? This option will -- prevent servers from authenticating unless they are using encryption. s2s_require_encryption = true -- Force certificate authentication for server-to-server connections? s2s_secure_auth = true -- Some servers have invalid or self-signed certificates. You can list -- remote domains here that will not be required to authenticate using -- certificates. They will be authenticated using DNS instead, even -- when s2s_secure_auth is enabled. --s2s_insecure_domains = { "insecure.example" } -- Even if you disable s2s_secure_auth, you can still require valid -- certificates for some domains by specifying a list here. --s2s_secure_domains = { "jabber.org" } -- Enable rate limits for incoming client and server connections limits = { c2s = { rate = "10kb/s"; }; s2sin = { rate = "30kb/s"; }; } -- Select the authentication backend to use. The 'internal' providers -- use Prosody's configured data storage to store the authentication data. authentication = "internal_hashed" -- Select the storage backend to use. By default Prosody uses flat files -- in its configured data directory, but it also supports more backends -- through modules. An "sql" backend is included by default, but requires -- additional dependencies. See https://prosody.im/doc/storage for more info. --storage = "sql" -- Default is "internal" (Debian: "sql" requires one of the -- lua-dbi-sqlite3, lua-dbi-mysql or lua-dbi-postgresql packages to work) -- For the "sql" backend, you can uncomment *one* of the below to configure: --sql = { driver = "SQLite3", database = "prosody.sqlite" } -- Default. 'database' is the filename. --sql = { driver = "MySQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" } --sql = { driver = "PostgreSQL", database = "prosody", username = "prosody", password = "secret", host = "localhost" } -- Archiving configuration -- If mod_mam is enabled, Prosody will store a copy of every message. This -- is used to synchronize conversations between multiple clients, even if -- they are offline. This setting controls how long Prosody will keep -- messages in the archive before removing them. archive_expires_after = "1w" -- Remove archived messages after 1 week -- You can also configure messages to be stored in-memory only. For more -- archiving options, see https://prosody.im/doc/modules/mod_mam -- Logging configuration -- For advanced logging see https://prosody.im/doc/logging -- -- Debian: -- Logs info and higher to /var/log -- Logs errors to syslog also log = { -- Log files (change 'info' to 'debug' for debug logs): info = "/var/log/prosody/prosody.log"; error = "/var/log/prosody/prosody.err"; -- Syslog: { levels = { "error" }; to = "syslog"; }; } -- Uncomment to enable statistics -- For more info see https://prosody.im/doc/statistics -- statistics = "internal" -- Certificates -- Every virtual host and component needs a certificate so that clients and -- servers can securely verify its identity. Prosody will automatically load -- certificates/keys from the directory specified here. -- For more information, including how to use 'prosodyctl' to auto-import certificates -- (from e.g. Let's Encrypt) see https://prosody.im/doc/certificates -- Location of directory to find certificates in (relative to main config file): certificates = "certs" -- HTTPS currently only supports a single certificate, specify it here: --https_certificate = "/etc/prosody/certs/localhost.crt" ----------- Virtual hosts ----------- -- You need to add a VirtualHost entry for each domain you wish Prosody to serve. -- Settings under each VirtualHost entry apply *only* to that host. -- It's customary to maintain VirtualHost entries in separate config files -- under /etc/prosody/conf.d/ directory. Examples of such config files can -- be found in /etc/prosody/conf.avail/ directory. ------ meow local options ------ legacy_ssl_ports = { 5223 } ignore_presence_priority = true smacks_max_ack_delay = 3 ------ Additional config files ------ -- For organizational purposes you may prefer to add VirtualHost and -- Component definitions in their own config files. This line includes -- all config files in /etc/prosody/conf.d/ Include "conf.d/*.cfg.lua" --VirtualHost "localhost" --VirtualHost "example.com" -- certificate = "/path/to/example.crt" ------ Components ------ -- You can specify components to add hosts that provide special services, -- like multi-user conferences, and transports. -- For more information on components, see https://prosody.im/doc/components ---Set up a MUC (multi-user chat) room server on conference.example.com: --Component "conference.example.com" "muc" --- Store MUC messages in an archive and allow users to access it --modules_enabled = { "muc_mam" } ---Set up an external component (default component port is 5347) -- -- External components allow adding various services, such as gateways/ -- transports to other networks like ICQ, MSN and Yahoo. For more info -- see: https://prosody.im/doc/components#adding_an_external_component -- --Component "gateway.example.com" -- component_secret = "password" # ls -l /etc/prosody/conf.d/ total 0 lrwxrwxrwx 1 root root 37 Jul 30 2019 myhostname.mydomain.mytld.cfg.lua -> ../conf.avail/myhostname.mydomain.mytld.cfg.lua lrwxrwxrwx 1 root root 31 Jul 30 2019 localhost.cfg.lua -> ../conf.avail/localhost.cfg.lua # cat /etc/prosody/conf.d/localhost.cfg.lua -- Section for localhost -- This allows clients to connect to localhost. No harm in it. VirtualHost "localhost" # cat /etc/prosody/conf.d/myhostname.mydomain.mytld.cfg.lua VirtualHost "myhostname.mydomain.mytld" Component "conference.myhostname.mydomain.mytld" "muc" name = "Chaträume" restrict_room_creation = "admin" max_history_messages = 500
I haven’t restarted the server yet, in case there’s something I can do to help debugging this. If this is a known issue, upgrading the server to Debian bullseye (prosody 0.11.9-2, prosody-modules 0.0~hg20210130.dd3bfe8f182e+dfsg-2) is an option if it’s fixed there.
My SSL certificate hook script restarted it, and the ghost session has gone away. But a proper fix would still be welcome…
I have two ghost sessions from that user again, Status timestamp 2nd and 3rd September respectively…
Hi, thanks for the report, and your patience. Part of the reason I haven't responded is because I wasn't really sure what to say (and haven't had time to write up in detail exactly why I didn't have anything to say). Here goes... Your issue report seems to be based on the premise that Prosody is tracking the sessions of your contacts, and telling you their status. The assumption then is that Prosody is, for some reason, keeping contact sessions online after they have gone offline (the "ghosts"). The problem is, that's not how it works. Prosody only tracks *your* sessions, and has no idea about the status of your contacts. Prosody only does two things related to that: A) When you start a new session with Prosody, Prosody will notify all your contacts and ask them to send their latest status. B) If your contacts reply, Prosody will forward their replies to your online client(s). Your contacts may not reply (if offline), and they may send additional updates whenever their status changes. Again, Prosody is only acting as a router of this information, it is not storing or tracking the status of your contacts in any way. This means that if you are seeing contacts online who are offline, the possible problems are one of: 1) Your contacts are actually still online 2) Your contact's server did not send out a status change notification when they went offline 3) Your contact's server sent a notification when your contact offline, but it could not be delivered to your server 4) Your contact's server sent a notification when your contact offline, and Prosody received it, but for some reason did not route it to your client 5) Your contact's server sent a notification when your contact offline, and Prosody received it and routed it, but your client did not correctly process it. Issue (1) seems like it's not an issue, but it can be unintuitive behaviour caused by XEP-0198 support in your contact's server (implemented in mod_smacks for Prosody). XEP-0198 will keep client sessions online even if they lose connectivity, so they can later be resumed. Some people set their XEP-0198 stupidly high, which makes this even more apparent, especially when they are experiencing connectivity issues and do not shut down their connection properly. Issue (2) would likely be a bug or other problem with your contact's server, or a problem with your roster subscription state being corrupted. Rosters should self-heal, but if not, again, it would be a bug. Issue (3) could be caused by any intermittent network issues between your contact's server and yours. The potential for such issues could be increased by hosting your server on a dynamic IP address, or behind a NAT router that implements aggressive idle connection timeouts. If you think this issue may be affecting you, look into enabling https://modules.prosody.im/mod_s2s_keepalive Issue (4) could be caused by a bug in Prosody (this is very unlikely - the presence handling code is a core part of Prosody and the logic has not changed for a long time). It could also be caused by a buggy module, though I don't see any top suspects in the configuration you provided. Issue (5) could be caused by a bug in mcabber. You could verify this by connecting a second client to the same account for a while, and see if it exhibits the same issue. Hope this helps identify the cause of the issue. If you get any further identifying the problem, please let us know.
ChangesFirst, thanks for the very verbose answer! I think I can exclude a number of possibilities already: ad 1: The contact is not still online (as confirmed by phone and/or physical proximity). I’m not sure it’s a point of the configuration of my contact’s server given that another Jabber account I have, on a site running ejabberd, does not show these ghost sessions. But I can certainly enquire about my contact’s server’s configuration if you think this helps in debugging (I’m fairly certain they also use ejabberd), I think she’s at least a co-admin. ad 2: I have no idea when the roster “self-heals”; after a prosody restart, the ghost sessions are gone, but until then, they seem to live on. ad 3: nope, my server is a rented virtual machine (qemu-kvm, so “always on”) operating as very low-traffic server on multiple protocols, and the other is a VM on a rented dedicated server (bare metal) operating as multiple low-to-medium-traffic servers. Both IP and Legacy IPv4 traffic are known to be stable on both ends. ad 4: I take your word for it. I included my versions in case this was a known issue in the version Debian ships. ad 5: The second account I use I also use mcabber for, so I believe this can also be excluded… … hmm, that being said, the versions differ. Good point. I’ll have to try to keep the same mcabber version up and connected to both accounts for a while. A first test (connect with a second mcabber session on the laptop I use for the other account, which however switches between networks very often) is inconclusive: the ghost sessions don’t appear for newly-connected clients. Similarily, exiting and restarting the mcabber-in-screen loses them. (I do not have any other way of determining them to be ghost session, as I’ve seen one with a status timestamp of about 2 hours old, though ☹) If you have any other ideas I could try, be my guest…
Update: I found I can do '/request last' in mcabber to find out whether one of the sessions of that user is indeed online, and I indeed see “Received IQ:last result from <user@host/resource>” and a couple 503 User session not found. Now if Prosody could just use this 503 to kill the session so the fulljid gets dropped from my roster, leaving me with only the one actually-online session (or none, if all are 503ing) shown as online, I’d be happy. I mean, MUCs do that already. I am also wondering if asking the other party to configure a static resource instead of their Gajim’s default random one would mitigate this… As for the lifetime, one of the sessions has a status timestamp of 26 days ago, so no to the self-healing.
Hi, thanks for the update. As mentioned in my comment on 2021-09-07, Prosody does not track the online status of your contacts. In particular your suggestion to "kill the session" of your contact and "drop the full jid from your roster" is not possible - there are no sessions for remote contacts, and rosters never contain full JIDs. The "503 User session not found" is an error returned by ejabberd. When your contact's session went offline, ejabberd should have sent a notification about this to you (specifically an "unavailable" presence stanza). Prosody will forward this to mcabber and mcabber should then update its UI to show that your contact is now offline. Either ejabberd is not emitting this stanza, or it's getting lost somewhere on the way to mcabber, or mcabber isn't handling it correctly. I'd say that ejabberd not emitting the stanza at all is an unlikely scenario (this is one of the most simple and basic operations of XMPP). My suggestions for further debugging: I don't see that I asked this already, but... have you checked Prosody's error log? (if you have log rotation, do check older files as well) If there is nothing in the error log, enable debug logging (change "info" to "debug" in your log option). Then wait until it occurs again, and grep the logs for your contact's full JID. Then we will know if Prosody is receiving the unavailable presence. I'm also currently working on a new module that will make it easier to find server-to-server connection issues (currently that's hard without continuous monitoring of the log file). When that's finished it may be useful to rule out any s2s issues. Good luck! I'm sure we can get to the bottom of this pesky problem eventually.
Huh, so this may be a mcabber problem after all? I guess I’ll have to look into that more. All 52 of Prosody’s error logs are empty save for one DNS issue last December. The mcabber version on the “bouncer”, to use IRC terms, is indeed quite old. I’ll investigate on that front. I’ve seen a plugin-thingy for mcabber to kill ghosts (apparently not part of the standard functionality) but trying to get that set up is also going to be icky… we’ll see.