#1299 PEP data does not fit into current storage abstraction
Reporter
ge0rg
Owner
Nobody
Created
Updated
Stars
★★★ (5)
Tags
Status-Blocked
Priority-Medium
Type-Defect
ge0rg
on
With the new mod_pep module, prosody will persist PEP items on disk. The directory structure used for that is not optimal.
The below paths are all relative to `/var/lib/prosody/SERVERNAME/`
The list of a user's PEP nodes is stored in `pep/USER.dat` which is perfect.
Individual nodes, however, are stored under `pep_NODE/USER.dat` leading to a cluttering of the prosody storage directory. This is not only impractical because it will end up with a huge number of different PEP usage cases in the top-level directory, but especially bad with OMEMO, which creates a devicelist node per user and a bundles node *per device*. With just a single account, I have the following files and directories on my server:
```
pep
pep/georg.dat
pep_eu%2esiacs%2econversations%2eaxolotl%2ebundles%3a1399321121
pep_eu%2esiacs%2econversations%2eaxolotl%2ebundles%3a1399321121/georg.list
pep_eu%2esiacs%2econversations%2eaxolotl%2edevicelist
pep_eu%2esiacs%2econversations%2eaxolotl%2edevicelist/georg.list
```
Please change the structure to be `pep/USER/NODE.dat` so that it is cleanly scoped, doesn't clutter the top-level storage directory and can be manually inspected per-user.
Thank you very much.
ge0rg
on
According to https://prosody.im/doc/storage you can store PEP node configuration and explicitly named nodes into SQL, but there is no obvious way to store _all_ PEP nodes regardless of their name into SQL without the big storage="sql" hammer.
Zash
on
I agree that this is awkward, however this is the corner we've backed into in order to have persistent PEP data at all without having do drastic changes to the internal storage API.
Each PEP node needs to be its own logical archive.
The current archive API is modeled as an ordered key-value where the key is (user, host, store, archive-id).
The current file system data manager library does not support more deeply nested directory trees. There is some in-progress work on this, see #881.
ge0rg
on
For future readers of this, #1320 does provide a workaround for me (move all PEP nodes into SQLite).
Zash
on
Changes
tags Status-Blocked
titleprosody 0.11 mod_pep: non-optimal persistent directory structure PEP data does not fit into current storage abstraction
With the new mod_pep module, prosody will persist PEP items on disk. The directory structure used for that is not optimal. The below paths are all relative to `/var/lib/prosody/SERVERNAME/` The list of a user's PEP nodes is stored in `pep/USER.dat` which is perfect. Individual nodes, however, are stored under `pep_NODE/USER.dat` leading to a cluttering of the prosody storage directory. This is not only impractical because it will end up with a huge number of different PEP usage cases in the top-level directory, but especially bad with OMEMO, which creates a devicelist node per user and a bundles node *per device*. With just a single account, I have the following files and directories on my server: ``` pep pep/georg.dat pep_eu%2esiacs%2econversations%2eaxolotl%2ebundles%3a1399321121 pep_eu%2esiacs%2econversations%2eaxolotl%2ebundles%3a1399321121/georg.list pep_eu%2esiacs%2econversations%2eaxolotl%2edevicelist pep_eu%2esiacs%2econversations%2eaxolotl%2edevicelist/georg.list ``` Please change the structure to be `pep/USER/NODE.dat` so that it is cleanly scoped, doesn't clutter the top-level storage directory and can be manually inspected per-user. Thank you very much.
According to https://prosody.im/doc/storage you can store PEP node configuration and explicitly named nodes into SQL, but there is no obvious way to store _all_ PEP nodes regardless of their name into SQL without the big storage="sql" hammer.
I agree that this is awkward, however this is the corner we've backed into in order to have persistent PEP data at all without having do drastic changes to the internal storage API. Each PEP node needs to be its own logical archive. The current archive API is modeled as an ordered key-value where the key is (user, host, store, archive-id). The current file system data manager library does not support more deeply nested directory trees. There is some in-progress work on this, see #881.
For future readers of this, #1320 does provide a workaround for me (move all PEP nodes into SQLite).
prosody 0.11 mod_pep: non-optimal persistent directory structurePEP data does not fit into current storage abstraction