Category: Linux

How to backup your Ubiquiti NVR/CloudKey+ Protect videos remotely to Backblaze B2

Sadly, Ubiquiti offers no way to backup the videos stored on their NVR and Cloudkey appliances. This sucks if you have a fire, or breakin, and the device is stolen or otherwise compromised.

Checking various reddit and ubiquiti community posts, it doesn’t look like a proper solution is coming either (been asked for for years!)

So, I decided to try and fix this issue myself.

Note this takes a LONG TIME to backup if you have continuous recording, or, lots of alerts, so it might not work for you – for example on Starlink, it takes approximately 4 hours for me to backup a single day of alerts from only 6 cameras. Not optimal, but, at least I have backups twice a day I guess.

Anyway, here’s how to do this for yourself if you want.

  1. Login to Backblaze

2. Create a new bucket for your backups. I turned encryption on, but that is up to you.

3. Retain the Bucket ID, Endpoint ID

Bucket ID: <LONG STRING OF NUMBERS>
Endpoint ID: s3.us-west-002.backblazeb2.com

4. Within backblaze, now click ‘App Keys’ on the left

5. Click ‘Add Application Key’

6. Name your key and set these settings;

access to: the bucket you just created for your NVR backups
access level: read/write (so you can restore if needed) 

7. It will provide some information for you to retain which you need to store.

keyID: <long string of numbers>
keyName: <the_name_of_your_key>
applicationKey: <long string of numbers>

8. Next, ssh to your cloudkey and install backblaze-b2

# apt install backblaze-b2

9. Then, do the following to authorize your account

# backblaze-b2 authorize_account <APP_ID>
# backblaze-b2 authorize_account <KEYID_OF_YOUR_BUCKET> 

when prompted, add the application key of your bucket.

10. Then, list the buckets you have available, it should show one which you configured previously on backblaze.

# backblaze-b2 list_buckets

it should show an error that you’re limited only to one bucket, the one you’ve setup for your NVR. This is fine. Proceed.

11. Now we’re all set to setup the backups. Since this is my personal NVR and it’s only for security purposes I didn’t want to setup duplicity or anything for encryption as Backblaze-B2 uses TLS anyways. However you can do so, and I found a cool tutorial here. I’ve turned on encryption on my bucket which should be sufficient for my needs.

12. Let’s test a dry-run of the videos

# backblaze-b2 sync --dryRun --keepDays 30 --threads 5 --excludeAllSymlinks /srv/unifi-protect/video/ b2://<<BUCKET_NAME>>/

13. This should output something like

upload 2022/01/13/68D79ADF56B9_0_rotating_1642109652208.ubv
upload 2022/01/13/68D79ADF56B9_0_timelapse_1642110047421.ubv
upload 2022/01/13/68D79ADF56B9_2_rotating_1642109651962.ubv
upload 2022/01/13/68D79ADF56B9_2_timelapse_1642110887295.ubv
upload 2022/01/13/E063DA00A047_0_rotating_1642109590189.ubv
upload 2022/01/13/E063DA00A047_0_rotating_1642122166120.ubv

14. Now that it works, modify your command to remove the dryrun command.

# backblaze-b2 sync --keepDays 30 --threads 5 --excludeAllSymlinks /srv/unifi-protect/video/ b2://<<BUCKET_NAME>>/

15. Now you can schedule this command to run at whatever interval you want via Cron on the machine.

Note, I have not tried this through a firmware update, it’s likely you have to reset this all up after each upgrade. Also as mentioned above, be aware these updates can (*will*) take forever, so it’s not really a great solution for up to the second backups, but it does work.

Mainly storing this for my reference 🙂

Automated removal of GOES-16 and GOES-17 images

Automating the deletion of GOES-16 and GOES-17 images received using GOESTools the ghetto way.

I was having trouble with my hard drive filling up while pulling images from GOES-16 and GOES-17 satellites. 32gb filles up really quick on my odroid!

So I wrote a ghetto little script that removes files older than 2 days, every day, except for the full color/ full disk images which are the ones I want to keep. I also leave the national weather service images since they take up very little space.

Here it is;

First, create your script

# nano /home/radio/goes/goes_remove_old.sh 

Copy this in it, modifying the paths if needed.

find /var/www/wx/goes/goes16/m1/ch02/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m1/ch07/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m1/ch07_enhanced/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m1/ch13/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m1/ch13_enhanced/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m2/ch02/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m2/ch07/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m2/ch07_enhanced/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m2/ch13/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes16/m2/ch13_enhanced/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/goes17/fd/ch13/* -mtime +2 -exec rm {} \;
find /var/www/wx/goes/ -type d -empty -delete

Then, depending on permissions either edit the user, or, root crontab file. For my setup, it’s root. Again. Ghetto on a private lan. Whatever.

# sudo /bin/zsh 
$ crontab -e 

Put this line in there, modifying paths if needed.

# everyday at 1am, delete the GOES files older than 48 hours (2 days)
0 1 * * * /home/radio/goes/goes_remove_old.sh >/dev/null 2>/dev/null

Exit and save the file.

That’s it, now at 1am every day, all files older than 48 hours will be deleted, saving space for more of those sweet sweet full color images from space! Hope this helps someone else 🙂

Setting up a Matrix server for Secure Encrypted Communications through almost all the channels you and your friends use.

With all the privacy issues these days I decided to check out alternatives to using standard services.

With that I found Matrix which allows for a secure ‘slack like’ service that also has ‘bridges’ that let you connect to iMessage, Slack, Signal, Facebook even Twitter and Instagram.

I have not fully configured this, this is mainly for my records purposes and I stole most of this from the excellent article I found called Running your own secure communication service with Matrix and Jitsi.

First I setup a Debian VPS over at Vultr (affiliate link). Which this assumes you’ve already done too.

So let’s begin!

First ssh to your server and update, upgrade, and install nginx.

apt-get update && apt -y install nginx lsb-release wget apt-transport-https

cd /etc/nginx/sites-enabled
rm default 
cp ../sites-available/default yourdomain.com
cp ../sites-available/default matrix.yourdomain.com
cp ../sites-available/default riot.yourdomain.com

Now we’re going to configure our settings. You should have a domain picked out already. In our example we’ll use ‘mydomain.com’ but this will obviously have to change.

First, you’re going to want to login to your DNS Registrar and setup these records. These will all have to be A Records.

  • TYPE: A IP: 190.190.190.1 POINTS TO: yourdomain.com
  • TYPE: A IP: 190.190.190.1 POINTS TO: matrix.yourdomain.com
  • TYPE: A IP: 190.190.190.1 POINTS TO: riot.yourdomain.com
  • TYPE: A IP: 190.190.190.1 POINTS TO: jitsi.yourdomain.com

** NOTE YOU WILL HAVE TO CHANGE ‘YOURDOMAIN.COM’ AND IP ADDRESS TO MATCH YOUR SETTINGS.

Once that is done we can continue with the rest of the setup. While you’re doing this the DNS records should propagate.

Now we’re going to need to edit each of our files in /etc/nginx/sites-available/

cd /etc/nginx/sites-available 
nano *

And in each file make it look like this, changing the domain and the path. Note that in the matrix configuration the location is proxy_pass instead of a directory.

matrix.yourdomain.com:

server {
        listen 80;
        listen [::]:80;
        root /var/www/matrix;
        index index.html index.htm;
        server_name matrix.yourdomain.com;
        location / {
                proxy_pass http://localhost:8008;
        }
}

yourdomain.com:

server {
        listen 80;
        listen [::]:80;
        root /var/www/html;
        index index.html index.htm;
        server_name yourdomain.com;
        location / {
                try_files $uri $uri/ =404;
        }
}

riot.yourdomain.com:

server {
        listen 80;
        listen [::]:80;
        root /var/www/riot;
        index index.html index.htm;
        server_name riot.yourdomain.com;
        location / {
                try_files $uri $uri/ =404;
        }
}

Now we go ahead and install certbot, and generate the certificates for your hosts with nginx. This gives you secure SSL connections on every new site you just created.

apt install -y python3-certbot-nginx && certbot --nginx -d yourdomain.com -d riot.yourdomain.com -d matrix.yourdomain.com

Now you can go ahead and create the directories you need.

cd /var/www
mkdir riot
mkdir matrix

Now we create the ‘.well-known’ file for the connector to authenticate your domain when you start everything up. You will have to modify your domain in the last command.

mkdir -p /var/www/html/.well-known/matrix
cd /var/www/html/.well-known/matrix
echo '{ "m.server": "matrix.yourdomain.com:443" }' > server

Now we get element (riot) going.

cd /var/www/riot
wget https://github.com/vector-im/element-web/releases/download/v1.7.22/element-v1.7.22.tar.gz
apt install -y gnupg
tar xzvf element-v1.7.22.tar.gz
ln -s element-v1.7.22 element
chown www-data:www-data -R riot

At this point you should be able to start nginx up and visit your domain to set everything up.

systemctl restart nginx 

Now open a browser and hit this url

https://riot.yourdomain.com

There you will make your account. Next we’ll setup Jitsi for video conferencing capabilities. First get the repository added.

echo 'deb https://download.jitsi.org stable/' >> /etc/apt/sources.list.d/jitsi-stable.list

Now install the gpg key needed.

wget -qO -  https://download.jitsi.org/jitsi-key.gpg.key | sudo apt-key add -

Now go ahead and update, and install Jitsi

apt-get update

During the installer you will want to make sure you give the installer the hostname of ‘jitsi.yourdomain.com‘ and make SURE the DNS is already setup (step 2 way up there) otherwise it _will_ fail on you here.

apt-get -y install jitsi-meet

Then you’re going to want to secure it using the provided script.

/usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh

After this, you should now be able to connect to the Jitsi server running on your domain by visiting it’s URL in your browser.

https://jitsi.yourdomain.com/

Now you’re going to want to set up Riot to use Jitsi. You can do this by performing the following.

nano /var/www/riot/config.json

Then change the ‘preferredDomain’ to your server. Change this block

"jitsi": {
        "preferredDomain": "https://jitsi.riot.im"
    }

To this instead

"jitsi": {
        "preferredDomain": "https://jitsi.yourdomain.com"
    }

That’s it. Just refresh your Riot screen in your browser and you are now ready to use Jitsi from within your server. No need for an integration manager to embed Jitsi!

Under the bridge downtown

Next we’re going to install some bridges.

I have not fully configured these (or any of this!!) again just noting this for future when I do have time to configure it fully and maybe it will help someone else!

Next, we’re going to install some bridge. So first off for the install we’re going to install a bunch of prerequisites.

sudo apt install python3 python3-venv
sudo apt install virtualenv python3-virtualenv
sudo apt install git
sudo apt install build-essential
sudo apt install python3-dev
sudo apt install python-olm 

Now add this to your source.list

nano /etc/apt/sources.list 

Paste this in

deb http://deb.debian.org/debian buster-backports main

Then you can install some more software you need.

sudo apt-get update 
sudo apt-get install libolm3/buster-backports
sudo apt-get install libolm-dev/buster-backports

Now configure your locales

dpkg-reconfigure locales

Configure all locales, and choose the default. Now we are ready to install whichever branch of the bridges we want to use. I’ll outline those below.

MASTER BRANCHES:

pip install --upgrade git+https://github.com/tulir/mautrix-instagram.git#egg=mautrix-instagram[all]
pip install --upgrade git+https://github.com/tulir/mautrix-telegram.git#egg=mautrix-telegram[all]
pip install --upgrade git+https://github.com/tulir/mautrix-facebook.git#egg=mautrix-facebook[all]
pip install --upgrade git+https://github.com/tulir/mautrix-signal.git#egg=mautrix-signal[all]

RELEASE BRANCHES:

pip install --upgrade mautrix-instagram[all]
pip install --upgrade mautrix-telegram[all]
pip install --upgrade mautrix-signal[all]
pip install --upgrade mautrix-facebook[all]

Then for each bridge run through the install

mkdir /var/www/bridge-fb 
cd /var/www/bridge-fb
virtualenv -p /usr/bin/python3 .
source ./bin/activate
pip install --upgrade mautrix-facebook[all]
deactivate
cd ..

and..

mkdir /var/www/bridge-telegram 
cd /var/www/bridge-telegram 
virtualenv -p /usr/bin/python3 .
source ./bin/activate
pip install --upgrade mautrix-telegram[all]
deactivate
cd ..

and..

mkdir bridge-signal 
cd bridge-signal 
virtualenv -p /usr/bin/python3 .
source ./bin/activate
pip install --upgrade mautrix-signal[all]
deactivate
cd ..

and finally…

mkdir bridge-insta
cd bridge-insta
virtualenv -p /usr/bin/python3 .
source ./bin/activate
pip install --upgrade mautrix-instagram[all]
deactivate
cd ..

So there you have it. Matrix. Riot/Elemment and Jitsi is installed and you have 4 bridges installed, but NOT configured so you can now begin experimenting!

Hope this can help someone else get it started on their journey to private communications .