Enhancement: Full Backup

[SecurityJon]

Hi!

It would be useful to have a 'full backup' option for any customisation/data applied to the tool. I see there are exports for some features (findings/domains/server) but I couldn't see an easy way to migrate all of the custom data out of the tool (for example reports/users, either via the UI interface or via the filesystem.

Use-case for this would be moving where the tooling is hosted or to decrease backup size - so starting again can be as easy as git-pull, docker-compose, import data, done!

[Sl3ipnir-Sec]

I think instructions on the best way to backup the database in the Wiki would be a good start on this. You can take a backup by doing: sudo docker exec -t ghostwriter_postgres_1 pg_dump --no-owner -U postgres -d ghostwriter > dump_ghostwriter.sql But I'm not sure if this is the "best" way to do it, and I've not had to test restoring from it in production yet. Obviously it won't do things like the report templates etc.

[er4z0r]

I'm currenlty trying to move a database dump from a dev to prod environment and always get errors upon restore. The farthest I got was having my users and clients back but none of the projects.

Has anyone do a successful db backup and restore?

[chrismaddalena]

As a general update to this thread: I am working on a full backup procedure and will be adding it to Ghostwriter CLI to make it as simple as possible to run or schedule.

In production environments, data volumes are mounted local folders inside the Ghoswriter directory. You may be able to zip your Ghostwriter directory and have a portable backup with your database and any uploaded files. I need to test that to see if it gets everything.

@er4z0r From our chats in Slack, I thought you were able to sort out migrating data from your dev environment to prod. Are you good now or still having trouble?

[er4z0r]

My migrated database has been working fine for a week now.

I am still working on an ansbile playbook that will do backup/restore.

This takes a slightly different approach:

• stop all containers except postgres (not ensure no further database access)
• create dump inside psql container and copy to host
• create archive of docker ghostwriter_production_data
• package the two into a timestamped archive
• download it

It can do the same in reverse for restore and (unless you tell it otherwise) will do all the above every time you try to restore a backup. Currently it has two important drawbacks:

• no backup of the application (e.g. if you changed code for getting allauth to work)
• no backup of the .env file

Not sure about the pros/cons of just handling the database docker volumes like I handle the media volume and just archive the whole thing.

Regarding your approach (packing the whole application directory including mounted volumes): So I get an archive of my application directory in the running state including the contents of the data volume, but how would I restore it?

[marcioalm]

@chrismaddalena do we have any update regarding this enhancement for backups in the ghostwritter_cli?

Thanks

[chrismaddalena]

@marcioalm Not yet. I still plan to add it to Ghostwriter CLI at some point, but I have to prioritize what I work on during the limited time I have for development. For now, you can run the backup commands via Docker. For example:

$ docker compose run --rm postgres backup
Backing up the 'ghostwriter' database...
SUCCESS: 'ghostwriter' database backup 'backup_2023_05_23T15_54_19.sql.gz' has been created and placed in '/backups'.

$ docker compose run --rm postgres backups
These are the backups you have got:
total 504K
-rw-r--r-- 1 root root 502K May 23 15:54 backup_2023_05_23T15_54_19.sql.gz

[chrismaddalena]

I took time today to add backup and restore commands to Ghostwriter CLI. It's in the v0.2.12 release:

https://github.com/GhostManager/Ghostwriter_CLI/releases/tag/v0.2.12

These binaries will be bundled with Ghostwriter v3.2.8.

[DrorDvash]

I took time today to add backup and restore commands to Ghostwriter CLI. It's in the v0.2.12 release:

https://github.com/GhostManager/Ghostwriter_CLI/releases/tag/v0.2.12

These binaries will be bundled with Ghostwriter v3.2.8.

Tried the new backup via cli and got error:

Edit: new issue opened https://github.com/GhostManager/Ghostwriter/issues/322

[chrismaddalena]

The Postgres backup command comes from a script from the Django community. It follows general advice not to use the default postgres user for backups. You can make a different user and update POSGTRES_USER in the configuration or change this line in the command file:

https://github.com/GhostManager/Ghostwriter/blob/master/compose/production/postgres/maintenance/backup#L23

The lines can be commented out or removed.