Currently prosody docs suggest to just tar the data dir which can break if prosody is in the process of writing to it (assuming prosody doesn't write to tmp files and then moves them, which it doesn't according to darkrain from the support channel). As such a command to create a consistent snapshot would be great.

Right, I was looking at datamanger, and it seemed like it opened and truncated a file, then writes the contents directly to it. As apollo13 was suggesting, prosody should be doing a temp file + atomic rename.

@paul.aurich We have a patch for this already under consideration for 0.9. The trickier part is list stores where we append, and creating a copy would be quite inefficient.

• tags Milestone-0.9 Status-Accepted

• owner MattJ

This is done in latest 0.9 and trunk for normal stores. Unfortunately list stores are more of a problem as we always write to those in-place, and the file might be read as that is happening. Going to sleep on it.

There is no realistic way we can do this now without delaying the 0.9 release further. Note that the new SQL changes already due for the release will allow for consistent backups by dumping the database at runtime. I'm removing the 0.9 milestone for now, and amending the title.

• tags Milestone-0.9
• title Provide a way to create consistent backups from the filesystem store

I'm noting this now because I thought of it after a conversation with Zash (and then promptly forgot when I was near a computer), and Matthew (or a clever impostor) nudged my memory: So, reminder: goal is consistent backups I think there are two potential corruption vectors (although I've only given this some thought): 1) Inter-file consistency (snapshotting files, capturing separate files in some manner of inconsistent state) 2) Intra-file consistency (a single file is captured half-written) I don't know how to solve the first, but for the second: let's say, in the worst case, that this is split up across multiple write() calls. The copy ends up with the last entry only half-written out. Can we make the load operation resilient against this? If prosody could guarantee an entry was only a single line, upon detecting corruption, it could try stripping off the last line and re-loading. Which would neatly handle a single line copied midway through being written out. If that's not the case, appending may actually be sufficiently safe as-is

@paul.aurich Making the storage format resilient, and making it one line per entry was exactly what I suggested to Zash and MattJ, and is probably how we'll do it. Given that this will be a significant change, we decided not to do it so near release. Note, the current format is loaded by using the Lua parser. We can in fact do some custom parsing to make it be resilient and stream-able (and even binary searchable!), though the one-entry-per-line format is still more elegant. Inter-file consistency is primarily an issue with just rosters (e.g., local user adding another local user) and with account deletion. Luckily XMPP causes subscriptions to become consistent here, and not much is lost. I don't think we will support cross-store transactions in the main storage sub-system. There can be other cases, but these are the only ones I'm aware of currently in prosody and prosody-modules. Intra-file consistency was solved for *nix systems (os.rename isn't atomic on Windows) on normal stores. List stores are the remaining problem.

http://danluu.com/file-consistency/

Related changes: - https://hg.prosody.im/0.10/rev/49feb0da29e1 - https://hg.prosody.im/0.10/rev/f8ba814fe029

• tags Type-Defect Type-Enhancement