#650 Make ejabberdsql2prosody (and likely other migrators) process the input sequentially
Reporter
Link Mauve
Owner
Nobody
Created
Updated
Stars
★ (1)
Tags
Status-New
Type-Defect
Priority-Medium
Link Mauve
on
What steps will reproduce the problem?
1. Have a big (~500 MiB) Ejabberd SQL dump
2. Try to migrate it with ejabberdsql2prosody.lua
What is the expected output? What do you see instead?
The script OOMs without having written any file after having used many gigabytes of memory.
Instead, by interleaving parsing and dumping it should be able to keep memory usage very low and eventually finish.
What version of the product are you using? On what operating system?
Trunk as of 7316:62d533535334, but also 0.10 and 0.9 have the same issue, on Debian OldStable.
prosody-migrator was rewritten in https://hg.prosody.im/trunk/rev/4d702f0c6273
It now works like mod_migrate and can use arbitrary storage modules, including mod_storage_ejabberdsql_readonly
What steps will reproduce the problem? 1. Have a big (~500 MiB) Ejabberd SQL dump 2. Try to migrate it with ejabberdsql2prosody.lua What is the expected output? What do you see instead? The script OOMs without having written any file after having used many gigabytes of memory. Instead, by interleaving parsing and dumping it should be able to keep memory usage very low and eventually finish. What version of the product are you using? On what operating system? Trunk as of 7316:62d533535334, but also 0.10 and 0.9 have the same issue, on Debian OldStable.
prosodyctl’s <a href="https://modules.prosody.im/mod_migrate.html">mod_migrate</a> together with <a href="https://modules.prosody.im/mod_storage_ejabberdsql_readonly.html">mod_storage_ejabberdsql_readonly</a> allowed me to do the migration without issue, as it runs user per user, on the various stores available. This issue can probably be closed, and users having the same problem redirected to those two modules.
prosody-migrator was rewritten in https://hg.prosody.im/trunk/rev/4d702f0c6273 It now works like mod_migrate and can use arbitrary storage modules, including mod_storage_ejabberdsql_readonly