Replies: 11 comments 15 replies
-
|
Use the postgres pg_dump to backup the database : |
Beta Was this translation helpful? Give feedback.
-
|
I'm also thinking the same way Netbox Devices has Export function which return a CSV file and I'd like to run it weekly to backup all our Devices and put it on S3 bucket. Is there any reference that I could look at in this repo? |
Beta Was this translation helpful? Give feedback.
-
|
That won't work, because you cannot restore Netbox from CSV files. Firstly, in general CSV files don't include unique ID references for one object to refer to another. And secondly, there are circular references which have to be restored simultaneously (in particular: a device's primary_ip points to an IP address, but the IP address points to an interface which points to the device). Therefore, you'll will have to use pg_dump to make a usable Netbox backup. Copying it to an S3 bucket can be done in a script using awscli or s3cmd. |
Beta Was this translation helpful? Give feedback.
-
|
I installed psql v10 and I get error message below when I type "psql" on CLI. psql: could not connect to server: No such file or directory Should I manage psql within the docker container? or on local server? |
Beta Was this translation helpful? Give feedback.
-
|
Easiest is to enter the postgres container and run In principle you can install the postgres client on your server or a remote workstation, and connect to the postgres database over the network. However this may require some setting up, for example adding rules to |
Beta Was this translation helpful? Give feedback.
-
|
I currently can't find where hba.conf file located at. I just got in to the 'netbox' database on postgresql container. |
Beta Was this translation helpful? Give feedback.
-
|
You'll need to switch your userid to the user that postgres runs under, which if the container is based on Ubuntu will probably be user For things like hba.conf, please refer to postgres documentation or a postgres user group. |
Beta Was this translation helpful? Give feedback.
-
|
In postgres.env file located at /env, I got into the postgres container after running below. Also I could get into the netbox database after running below. Do you mean I have to use pg_dump so that I could export whole data on netbox database? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. You'll be able to combine it into a single command, probably some variation of the following (untested as I don't run netbox in docker): |
Beta Was this translation helpful? Give feedback.
-
|
This is my first time using netbox and postgres. After a lot of test try to find the correct data base and restore procedure I created the following script to generate the data base backup. It only work due to permission problem in /var/lib/postgresql/ and the restore should be do using postgres user. I used the as part of the backup the option --clean, because if you restore over the actual database it give you errors "tables already exist" sorry for my English because my firts languaje is spanish. The owner of this script should be postgres. #!/bin/sh **********************************************************************************************************05-jul-2023 DRMScript para correr backup de la base de datos de Netbox. Este script debe estar localizadoy debe correrse en el directorio /var/lib/postgresql. El script debe tener como ownership a postgres-rwxr--r-- 1 postgres postgres db_backup.shLa restauración de la base de datos debe hacerlo bajo el usuario de postgresus - postgrespostgres@netboxU20:~$ psql -e -d netbox < netbox-mysql-FECHA_DEL_BACKUP.sqlEl backup de la base de datos se guardara los ultimos 5 dias.**********************************************************************************************************N="netbox-mysql" BKP="/home/oberon" /usr/bin/su -c 'pg_dump -U postgres -F p --clean netbox > /var/lib/postgresql/db_clean.sql' postgres |
Beta Was this translation helpful? Give feedback.
-
Your colleagues may find that confusing, since this is a dump from postgres not mysql :-) This is my rough-and-ready backup script: called from a cronjob: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, yes you right N="netbox-mysql" but is just a variable...... |
Beta Was this translation helpful? Give feedback.
-
|
I have this : and in the /home/netbox folder I have all ran from the 'netbox' crontab (3 times a day) |
Beta Was this translation helpful? Give feedback.
-
|
Hello, I'm a gravedigger but I'm wondering why you delete 45+ days old sql files in line 10 then all 5+ days old file starting by netbox (so sql as well if hostname start with that word, which may happens for a lot of us) at line 25 (last). (less important but why using "find" then "/usr/bin/find" at the end?) |
Beta Was this translation helpful? Give feedback.
-
|
I rewrote the script to suit me (using @PieterL75 and @candlerb comments). If someone does have similar needs, here you go bellow. I also add, if anyone come here and to the same, that you need to change the permission to 0600 for the .pgpass |
Beta Was this translation helpful? Give feedback.
-
|
my take, please feel free to merge into yours to get something out of it. |
Beta Was this translation helpful? Give feedback.
-
|
Isn't there a way to export all the data in a YAML or JSON format and easily import it back? |
Beta Was this translation helpful? Give feedback.
-
|
I don't think you can have a single big document so you'll need to do one per category. Like one for devices, one for vlans, one for IP subnets etc... I think exportation order doesn't matter but importation order will be important as you'll not be able to import, for example, vlan with associated IP subnet if you didn't import the IP subnet beforehand. You will either need a script that calls the API or a tool like Bruno and a collection to export and another one to import that you can play manually. |
Beta Was this translation helpful? Give feedback.
-
|
The thing is that, when I export a category, the exported default CSV columns do not match the imported fields. For example, this is the result of a sample Tenents export. Netbox shows the Tenent import objects as:
When I try to import the default CSV file, it results in an error.
So, it seems that I need to rename and reorder the CSV columns to match the expected category fields. Doing this for all categories is rather impractical. |
Beta Was this translation helpful? Give feedback.
-
|
you can always create your own export templates that create the thing you need. |
Beta Was this translation helpful? Give feedback.
-
|
With the comments above, we can see how to backup Netbox DB from a Docker installation. Basically, docker exec -t netbox-docker-postgres-1 pg_dump -d netbox -U netbox | gzip > pg_dump.sql.gzHowever, it is not shown how to restore such a backup. When Netbox is started for the first time from a docker compose setup, the database is created automatically, and I assume with all the required tables. So, restoring a PostgreSQL dump, is not that simple. Could anyone share how this process is done? |
Beta Was this translation helpful? Give feedback.
-
|
Right, it had not occurred to me to update the gist with the restore steps.
But I did use the script and do restores during some horror migration. I'll
go check the shell for it and update.
... nothing left in the history because i had used screen and on that host the bash history was at default settings, so it wiped itself.
will never understand how people want defaults like that. ffs.
I wrote notes though, I'll put them here:
### DB Management / Restore
used after major version Upgrades
Explanations regarding DB Drops - see here:
https://stackoverflow.com/questions/22287914/pg-dump-and-pg-restore-drop-if-exists
#### Docker Volume
- To reinitialize and be able to run migrations etc. I had to delete the DB Data volume
If there's no backup, and you're stuck, or whatever,
switch back to the old major version and run your backup
datenvolume loeschen (falls kein Backup, erst eine Version zurueck und sichern!)
#### Bring up and prepare DB container
IIRC this can also be done after a start of everything. You really need to drop the other containers though.
Reason being otherwise the netbox/django expectations and DB contents will be diverged a bit. So just don't start.
Update your env, pull the next version(s)
- Then Only launch DB Container
- Then initialize DB Container aufsetzen
You can use docker cp to drop the backup in the container, here I had put it under /tmp
Might need to consider access rights since the container is 'rootless'.
```
fe3a50890d8f:/# dropdb -U netbox netbox
fe3a50890d8f:/# createdb -U netbox netbox
fe3a50890d8f:/# pg_restore -U netbox -d netbox -W </tmp/db-202503091553
```
You can also only use `psql` but there's a but more crap with passing the user / password, so I didn't do that.
You can likely also use a custom initializer to load the db dump. I have not tried. I was too angry with the process.
You can also (if this is a corp setup with actual operability concerns) use a different postgres container, some can load a .sql dump on start automatically.
#### Start
After this you can start the whole netbox, and you'll see the migrations running automatically
```
netbox-1 | Applying vpn.0006_charfield_null_choices... OK
netbox-1 | Applying vpn.0007_natural_ordering... OK
netbox-1 | Applying wireless.0009_wirelesslink_distance... OK
netbox-1 | Applying wireless.0010_charfield_null_choices... OK
netbox-1 | Applying wireless.0011_wirelesslan__location_wirelesslan__region_and_more... OK
netbox-1 | Applying wireless.0012_alter_wirelesslan__location_and_more... OK
netbox-1 | Applying wireless.0013_natural_ordering... OK
```
Migration issues will occur if:
- you jumped too far in versions (don't, it is not resilient and will destroy itself, but don't worry, you can do this over and over)
- you didn't properly wipe the database and there's a version issue (i.e. you got something in there that already fixed an issue a migration should handle)
- venv vs non-venv plugin installs don't match the expectations, so around the borders of pre 4.0 and after 4.0.
Generally, not all versions ran migrations/initializers automatically and you'll have a ton of issues with those.
Try to disable plugins at that point, see that you get the restore down OK, then worry about the plugins.
Sometimes you even wanna finish updating to latest and then reinstall the accompanying more current plugin versions.
best of luck, make backups and never start to think it won't work at the end!
|
Beta Was this translation helpful? Give feedback.
-
|
Oh and there's one more thing I see in the notes, you have to watch out like mad for the versions + tags. Info on TagsThat info is found here: (But the page has no matrix, no actual examples, and no working links) Then you need to set the git branch git branch (i.e.
Dazu brtaucht man ein passendes
|
Beta Was this translation helpful? Give feedback.
-
|
replying put me in a time hole and now i've missed getting dinner, so this better helps someone! |
Beta Was this translation helpful? Give feedback.
-
|
After a clear head and a bit more research, here is what I did. Backupdocker exec netbox-docker-postgres-1 pg_dump -Fc --clean -d netbox -U netbox | gzip > pg_dump.sql.gzRestore
docker compose stop
docker start netbox-docker-postgres-1
docker exec netbox-docker-postgres-1 dropdb -U netbox netbox
docker exec netbox-docker-postgres-1 createdb -U netbox netbox
zcat pg_dump.sql.gz | docker exec -i netbox-docker-postgres-1 pg_restore -v --clean -d netbox -U netbox
docker exec netbox-docker-netbox-1 python3 manage.py migrateFor the most part, it appears that all objects have been restored, except that the
Instead of clicking it, if I type its URL, an error shows up.
<class 'django.db.utils.ProgrammingError'>
column tenancy_contact.group_id does not exist
LINE 1: ...act"."description", "tenancy_contact"."comments", "tenancy_c...
^
Python version: 3.12.3
NetBox version: 4.2.9-Docker-3.2.1
Plugins: None installedThe I've tried running the "migrate" command, but there are no migrations to apply. I'm not sure what the error is. Any thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
can't remember. make a second install of 4.2.9-Docker-3.2.1 on the side and compare the contact/contactgroup tabels. |
Beta Was this translation helpful? Give feedback.
-
Hi,
I'm operating netbox through running docker-compose.
I'm wondering if there's a way to backup all data (e.g. IP addresses, Devices) on netbox.
I want to backup data every day or every week.
Here's the information when I type "docker ps -a" command on my netbox server.
docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b3379a749353 netboxcommunity/netbox:v3.2-1.6.1 "/sbin/tini -- /opt/…" 2 months ago Up 2 months 0.0.0.0:8000->8080/tcp, :::8000->8080/tcp netboxdocker_netbox_1
bcd810b00e02 netboxcommunity/netbox:v3.2-1.6.1 "/sbin/tini -- /opt/…" 2 months ago Up 2 months netboxdocker_netbox-housekeeping_1
fc17e44536a7 netboxcommunity/netbox:v3.2-1.6.1 "/sbin/tini -- /opt/…" 2 months ago Up 2 months netboxdocker_netbox-worker_1
c3eb6d82b84e postgres:14-alpine "docker-entrypoint.s…" 2 months ago Up 2 months 5432/tcp netboxdocker_postgres_1
b16e3fb413e5 redis:6-alpine "docker-entrypoint.s…" 2 months ago Up 2 months 6379/tcp netboxdocker_redis-cache_1
feb69922effa redis:6-alpine "docker-entrypoint.s…" 2 months ago Up 2 months 6379/tcp netboxdocker_redis_1
Please help.
Beta Was this translation helpful? Give feedback.
All reactions