Skip to content

GMVault

Note

2020-05-22 : I'm testing a solution/workaround to the OAuth issue. If it doesn't work, maybe I will move to this:

Original Setup

GMVault is a project on GitHub to back up Gmail

Back in 2016 I followed the instructions here to set it up and it worked smoothly.

Original setup steps replicated here (quasi verbatim) :

A. Install Python and GMVault
B. Configure GMVault for each user
C. Get the Synology to run GMVault each night

A. Install Python and GMVault:

  1. Log into Synology NAS as Admin and go to Main Menu -> Package Center -> Settings and set Trust Level to "Synology Inc. and trusted publishers".

  2. In the "Package Sources" tab, click Add, type "SynoCommunity" as Name and "http://packages.synocommunity.com/" as Location and then press OK to validate.

  3. Go back to the Package Center and install Python 2.7 package ("Python") in the Community tab. (Synology provide Python 3, so if a later version of GMVault is made compatible then you wont need to do steps 1-4)

  4. Remove Package Source and reset Trust Level to "Synology Inc."

  5. Create folder "GMail" in Shared Folder "Shared_Folder_1" <== Programs will be installed in this "GMail" folder.
    (Have thought about this afterwards and I would probably create a shared folder called "GMVault" to store the programs in. Also if you have more than one disc then you will need to know which volume this shared folder is on for step 9 onwards - mine is on "volume1")

  6. Create folder "GMail" in Shared Folder "homes/Richard" <== Need to do this for each user who uses GMail, so I have 3 other users that I also created "GMail" folders for.

  7. In Control Panel -> "Terminal & SNMP" -> Enable SSH service
    (Security Advisor will complain about the SSH service not being secure if you use the default port of 22, which I didn't bother changing as I was only logging in via my local network)

  8. My PC runs Windows 7 so I use PUTTY to connect to the Synology, so open a Command Window on "C:\Users\Richard\PUTTY" and run "putty.exe -ssh admin@mySynNas"
    (Change "mySynNas" with the name of your Synology box, and if prompted about the server host key is not cached, click "Yes")

  9. Install the Python Virtual Environment:

    • In PUTTY, enter the following :
      cd /volume1/Shared_Folder_1/GMail
      wget https://pypi.python.org/packages/source/v/virtualenv/virtualenv-1.10.1.tar.gz
      tar xzf virtualenv-1.10.1.tar.gz

  10. This will create the directory "virtualenv-1.10.1" which I renamed as "virtualenv"

  11. Now install GMVault:

    • In PUTTY, enter the following: cd virtualenv
      python virtualenv.py gmvault_env
      gmvault.env/bin/easy_install gmvault
      sh /volume1/Shared_Folder_1/GMail/virtualenv/gmvault.env/bin/gmvault -h

      Should display the help for GMVault, if installed okay!

  12. Exit PUTTY

B. Configure GMVault for each user:

  1. Add the user into the Administrators group (Can't login under SSH otherwise!) and make sure they have Read/Write access to "homes" and "Shared_Folder_1"

  2. ssh into the Synology as that user:
    In Command Window, run putty.exe -ssh Richard@mySynNas

  3. Need to generate the OAuth key for that user:
    In PUTTY,
    if you don't have too many emails, enter:
    sh /volume1/Shared_Folder_1/GMail/virtualenv/gmvault.env/bin/gmvault sync -t full --db-dir /var/services/homes/Richard/GMail/ --emails-only --no-compression my_email...@gmail.com

    OR:
    (To restrict the number of emails downloaded to begin with - I had nearly 14,000 emails, which took just over 3 hours to download when I did a full download later)
    sh /volume1/Shared_Folder_1/GMail/virtualenv/gmvault.env/bin/gmvault sync -t custom --imap-req 'Since 1-Sep-2010 Before 1-Nov-2010' --db-dir /var/services/homes/Richard/GMail/ --emails-only --no-compression my_email...@gmail.com

    GMVault will then prompt to open a web browser to grant OAuth2 access.

  4. Press Enter to open the browser - which won't happen!

  5. Highlight the web address that GMVault displays - DO NOT PRESS CTRL C! Otherwise you'll stop GMVault and will need to run it again!

  6. Open a new window in your web browser and paste the web address into the address bar.

  7. Login to Google Mail and grant access to GMVault.

  8. Copy the authentication token that Google will display and paste into the SSH session. GMVault should now continue and download the emails.

  9. Once done, exit PUTTY.

  10. Remove user from the Administrators group.
    (The users on my Synology are disabled and have Read Only access to "homes" and "Shared_Folder_1" so I also reset those)

  11. Repeat steps 13 to 22 in this section (part B) for each user

  12. In Control Panel -> "Terminal & SNMP" -> Disable the SSH service

C. Get the Synology to run GMVault each night:

  1. Use Synology's Text Editor to create the script "GMail_Backup.sh" which was saved to "/homes/Richard". Copy the lines inbetween SCRIPT BEGIN: and SCRIPT END: changing path names , username and email address as required.
    #!/bin/sh

    # This runs as root so user environment is not set so need to configure HOME
    HOME="/var/services/homes/Richard"
    cd ~
    NOW=$(date +"%Y-%m-%d")
    LOGFILE="$HOME/GMail_log-$NOW.log"
    CURTIME=$(date +"%r")

    echo "------------------------------------" >> $LOGFILE
    echo "$CURTIME: Starting email sync..." >> $LOGFILE
    echo "------------------------------------" >> $LOGFILE
    printf "\n" >> $LOGFILE

    set >> $LOGFILE
    printf "\n" >> $LOGFILE

    sh /volume1/Shared_Folder_1/GMail/virtualenv/gmvault.env/bin/gmvault sync -t quick --db-dir $HOME/GMail/ --check-db no --emails-only --no-compression my_email...@gmail.com >> $LOGFILE

    CURTIME=$(date +"%r")

    printf "\n" >> $LOGFILE
    echo "------------------------------------" >> $LOGFILE
    echo "$CURTIME: email sync finished." >> $LOGFILE
    echo "------------------------------------" >> $LOGFILE
    printf "\n\n" >> $LOGFILE

  1. I copied this script into each users home directory, editing the script each time to replace /Richard with the correct username to set $HOME correctly, and the correct email address for GMVault.

  2. Open Control Panel -> Task Scheduler -> Create Scheduled Task -> "User-defined script"
    Set Task = "GMail Backup Richard"
    Set User = admin <== Though this is probably superflous as the script runs under root anyway!
    Set the day and time to run. (Mine is set to run Daily at 04:00)
    Set User-defined script = sh /var/services/homes/Richard/GMail_Backup.sh

    (I also set mine to email me when the script terminates abnormally)

  3. Repeat steps 25 to 27 in this section (part C) for each user, setting the path, username and email address for each user.

  4. The end.

Notes on the process

It will then create a log file in the user's "home" directory and put the actual emails in the GMail directory for each user. These log files will need to be manually deleted unless you want to set up a script to tidy them up.

I also found having the output of the Task Scheduler useful in debugging the script, which is how I found out I needed to set the $HOME variable in the script (The script would end but would not connect to the Mail server and download the email - the script log file didn't show any error message but the task output did). You can turn that on in Task Scheduler -> Settings.

If you restricted the number of emails when generating the OAuth key (step 15) then you may want to create another script which you run manually, which has the GMVault option "-t full" instead of "-t quick". That way you get the Synology to download all your emails before the Task Scheduler kicks in and does a quick update each night, or whatever you've set the task as.

My script above does not delete emails when they're deleted from the server - if you want that then replace the GMVault option "--check-db no" with "--check-db yes". I have created another script with that setting to "yes" and I'll run it manually every few months once I've updated my offsite backup.

Problems

Google updated their auth for 3rd party apps in 2019 and GMVault stopped working.

I found a few notes on how to set it up but they didn't work for me.

Solution 3 is about generating an app password. I'm trying that at the moment. It worked!

Solution 2 works by activating less security so I didn't try it out.

Solution 1 seems ok. I set it up in 2019 and it ... didn't work. I tried again in May 2020 and am stuck in a rut :

  • The app needs Outh Consent Screen verification by Google
  • The reason is that it's an "external" app, but since I'm not a GApps user, I cannot set up an "internal" app (which is what I need - I only need to set this OAuth for my instance of GMVault)

Solution 3 steps replicated here

Falling back to generating an app password and using that is an option. i think that should bypass any of the oauth stuff. Example:

$ gmvault sync --store-passwd user@gmail.com
<enter app password>
^C
$ gmvault sync -p user@gmail.com
...

See https://support.google.com/accounts/answer/185833?hl=en for details on app passwords. Not ideal, though.

Solution 1 steps replicated here

Procedure to get yourself a new client ID & secret:

  • Start here: https://console.developers.google.com
  • Accept conditions, if you never used the Google API Console (after reading all the terms and conditions, evidently)
  • Create a new project (at the top); of choose one you already have, if that makes sense
  • Go to https://console.developers.google.com/apis ; click + ENABLED APIS AND SERVICES at the top
  • Search for Gmail; select it; click Enable
  • Go to https://console.developers.google.com/apis/credentials; choose OAuth Consent Screen tab
  • Fill name (enter anything), click Add scope
  • Select the checkbox on the https://mail.google.com/ line; click Add
  • Click Save
  • Click Create Credentials; choose OAuth client ID
  • Application Type: Other; give it a name (anything; I suggest gmvault, since this credential will be used by gmvault)
  • Click Create

You will get a client ID and secret.
Those two values needs to replace the existing ones in
$HOME/.gmvault/gmvault_defaults.conf : gmvault_client_id=... and
gmvault_client_secret=...

Also, you might need to change the value of conf_version in that file; make sure it is conf_version=1.9 and NOT conf_version=1.9.1, otherwise, gmvault will overwrite it every time it runs.

Finally, make sure you delete the previously obtained OAuth2 token; running gmvault as you normally do will show you where the token file, that you need to delete, is. Eg:

Get OAuth2 credential from /home/gb/.gmvault/you@gmail.com.oauth2.