#650 Make ejabberdsql2prosody (and likely other migrators) process the input sequentially

Reporter Link Mauve
Owner Nobody
Created
Updated
Stars ★ (1)
Tags
  • Status-New
  • Type-Defect
  • Priority-Medium
  1. Link Mauve on

    What steps will reproduce the problem? 1. Have a big (~500 MiB) Ejabberd SQL dump 2. Try to migrate it with ejabberdsql2prosody.lua What is the expected output? What do you see instead? The script OOMs without having written any file after having used many gigabytes of memory. Instead, by interleaving parsing and dumping it should be able to keep memory usage very low and eventually finish. What version of the product are you using? On what operating system? Trunk as of 7316:62d533535334, but also 0.10 and 0.9 have the same issue, on Debian OldStable.

  2. Link Mauve on

    prosodyctl’s <a href="https://modules.prosody.im/mod_migrate.html">mod_migrate</a> together with <a href="https://modules.prosody.im/mod_storage_ejabberdsql_readonly.html">mod_storage_ejabberdsql_readonly</a> allowed me to do the migration without issue, as it runs user per user, on the various stores available. This issue can probably be closed, and users having the same problem redirected to those two modules.

  3. Zash on

    prosody-migrator was rewritten in https://hg.prosody.im/trunk/rev/4d702f0c6273 It now works like mod_migrate and can use arbitrary storage modules, including mod_storage_ejabberdsql_readonly

New comment

Not published. Used for spam prevention and optional update notifications.