Use SSH and dd to Remotely Backup a Raspberry Pi

Posted: 2016-11-25 in Ham Radio, How To, Unix Geekery
Tags: , , ,
Introduction

I love the Raspberry Pi! I am currently using one as a local DSTAR hotspot. It works great. I have tried several different images (Maryland-DSTAR, Western, and DSTAR Commander). I think I have settled on DSTAR Commander. I think I finally have everything working exactly how I want it to.

When making tweaks to the system, it is very important to make a backup of your current system so that you can easily go back to where you started if you messed something up. Additionally, if your SD Card fails (and it WILL), having a backup will enable you to quickly get your system back online.

I used to have to shut down the pi, pull the SD Card out, find my SD Card adapter, plug it into my Mac, make a backup, and then reverse the process to get my system online again.  Those days are over. You can make a backup of your pi and have the backup written to your regular PC all from the command line. Here is how to do it.

Bottom Line Up Front:

From your local machine, run the remote backup command:

ssh pi@xx.x.x.xx "sudo dd if=/dev/mmcblk0 bs=1M | gzip -" | dd of=~/Desktop/pibackup.gz

If you ever need to use your back up you cannot do it remotely (at least that I know of).  You will need to insert the SD card into your local machine and run the following commands:

diskutil unmountDisk /dev/disk#
gzip -dc ~/Desktop/pibackup.gz | sudo dd of=/dev/rdisk# bs=1m conv=noerror,sync

Continue reading if you need much more detailed instructions on what this command does, what each piece of the command means, and how to get all of the necessary inputs.

Note 1:

My local a machine is a MacBook Pro and therefore uses Darwin Unix.  My Raspberry Pi is running Debian unix.  Some of the commands used on the local machine will not work on the remote machine.  Use the info command on the system you want to run the command in order to get the proper syntax.

Note 2:

Not all SD cards are the same size, even if they both say 32GB.  I have seen variations.  To ensure this technique works all of the time, I recommend that you use “gparted” to create some free space at the end of your SD Cards to ensure they are EXACTLY the same size.

Find Your Device

SSH into your RPi using the following command:
ssh pi@xx.x.x.xx
xx.x.x.xx is the ip address of your RPi.

Use the following command to determine the block device of your SD Card:
lsblk -p

lsblk = the command to “list block devices”
-p = tells lsblk to print the full path of the block devices

The output will look something like this:

pi@raspberrypi:~ $ lsblk -p
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
/dev/mmcblk0 179:0 0 29.9G 0 disk
\u251c\u2500/dev/mmcblk0p1 179:1 0 60M 0 part /boot
\u2514\u2500/dev/mmcblk0p2 179:2 0 3.7G 0 part /

/dev/mmcblk0 is our device

Exit your current SSH session by typing exit

Backup Your Raspberry Pi

Use the following command:
ssh pi@xx.x.x.xx "sudo dd if=/dev/mmcblk0 bs=1M | gzip -" | dd of=~/Desktop/pibackup.gz

Breakdown of the command:
  • ssh = secure shell
  • pi = The name of the user you are logging in as on the REMOTE machine
  • @xx.x.x.xx = the ip address of the remote machine
  • sudo = super user do (elevate your privileges to the super user)
  • dd = Data Description; utility used to copy and convert data.  See my other post here for more information on this powerful unix command.
  • if=/dev/mmcblk0 = input file (the device that you are copying FROM)
  • bs=1M = tells dd to use a block size of 1 million bytes.  If the number ends with a “b”, “k”, “m”, “g”, or “w”, the number is multiplied by 512, 1024 (1K), 1048576 (1M), 1073741824 (1G) or the number of bytes in an integer, respectively.  Macs want a capital M; on debian the command wants a lowercase letter.
  • | = pipe command reads the output of the command on the left and pipes it to the command on the right.
  • gzip = program to compress files
  • - = read standard input (tells gzip to compress standard input which is coming from the “pipe”
  • | = pipe the standard output of gzip to the next command
  • dd = Data Description; utility used to copy and convert data.
  • of=~/Desktop/pibackup.gz = Write the output file of dd to a file called pibackup.gz on the local user’s Desktop folder.

When you enter the command it is going to ask you for the password on the REMOTE machine. In this case it is asking for the password of “pi.”  Enter the password and press enter.  You will be staring at a blinking cursor for a long time.

Getting a Status of dd

If your local machine is a Mac, you can easily get a status by pressing “CTRL + T”

You will get a status that looks like this:

load: 0.98 cmd: dd 8967 running 0.17u 1.94s
317344+0 records in
317344+0 records out
162480128 bytes transferred in 167.835383 secs (968092 bytes/sec)
load: 1.13 cmd: dd 8967 running 0.57u 6.17s

You can do this as many times as you want, to get an idea of how much has been copied.

Completion:

When the process is complete, your command prompt will return.  Your remote backup of your raspberry pi is complete and the backup file is on the desktop of your Mac.  Sweet!

References:
Comments
  1. […] my instructions here to backup your updated sd card  to your […]

    Like

  2. anne says:

    Hi, Thanks for the suggestion.

    looks now as if the backup file pops up at the desktop of my Pi……

    Like

  3. John A says:

    Just used this to fix a reboot loop problem. I changed the timezone from UTC to my local time zone. Apparently this causes a reboot loop and the only way to fix is to re-image the SD card. With my backup saved to my computer, it made getting my system back extremely quick and easy.

    Like

  4. SH says:

    My earlier comment had a few errors and omissions. Here’s a corrected version.

    Here’s how to do it using “Bash on Ubuntu on Windows”:

    ssh pi@xxx.xxx.x.xxx “sudo dd if=/dev/mmcblk0 bs=1M | gzip -” | dd of=/mnt/c/Users/’username’/Desktop/pibackup.gz

    Notice the 1m should be 1M, and replace ‘username’ with your actual Windows username (e.g., /mnt/c/Users/Bob/Desktop/pibackup.gz)

    Liked by 1 person

  5. dave says:

    Can you provide examples or a link to how to do the opposite and restore from an archive via ssh?

    Like

    • John A says:

      It’s not really possible since you are overwriting the machine that you have an SSH connection to. If there is a way, I am unaware of it.

      Like

    • Norm says:

      You would need to have physical access to the SD card you’re going to restore the .gz backup onto. Once you have the SD inserted and visible in /dev/sdX, then run something similar to:
      gunzip -c /path/to/your-backup.img.gz | dd of=/dev/sdX

      Like

  6. Scott says:

    Love the breakdown of the commands. Thanks!

    Like

  7. mauvedeity says:

    This process worked (well, is still working) for me as well. It’s really slow, because the gzip is taking 80-90% of the CPU on the Pi, and since mine’s a Pi 1, it’s going to take a while. I also hit the thing with the bs parameter needing a capital M.

    One potential problem is if the sudo command on the Pi needs a password – mine doesn’t but I don’t know why.

    I might actually set this up to run regularly now, to keep my Pi backed up.

    Liked by 1 person

  8. mauvedeity says:

    Oh, and the lsblk command didn’t like the -p parameter. Without that, it gives you output as you’d expect.

    Like

  9. mauvedeity says:

    …actually, having tried this again, I’d suggest taking out the “| gzip -” part, especially for older Pis. You’d also need to change the “.gz” to “.img”. These changes mean that the Pi doesn’t compress the dump on the way through, but sends it over the network uncompressed. Because the processors on Pis can be quite slow, the compression adds a lot of time to the process. On my Pi 1, with compression on it took about 2.5 hours(!), but without compression it was about 55 minutes, transferring data about 9 times faster. If you want it compressed, compress the file on your local machine. Hope this helps!

    Liked by 1 person

    • Rémi Héneault says:

      You can also compress the data using your local computer’s CPU instead of the RPi’s.
      Just move the `gzip` command out of the quotes :
      `ssh pi@xx.x.x.xx “sudo dd if=/dev/mmcblk0 bs=4M” | gzip – | dd of=~/Desktop/pibackup.gz`
      This doubles the speed for me.

      Liked by 1 person

    • remiheneault says:

      You can also compress the data upon arrival on your local computer, that way you don’t use the RPi CPU.
      Just move the `gzip` part out of the quotes :
      `ssh pi@xx.x.x.xx “sudo dd if=/dev/mmcblk0 bs=4M” | gzip | dd of=~/Desktop/pibackup.gz`
      This doubles the speed for me.

      Like

  10. Jesse Geron says:

    You’re the man! I’m going to work on scheduling this to run nightly!

    Liked by 1 person

  11. jgmachine says:

    Question: What happens if new information is written to the card during backup? I’m currently running my rpi as a web server.

    Like

  12. Angelo says:

    Doesn’t work for me. Guessing you have done something to allow for remote sudo execution or that it prompts you for a password? Mine just returns in less than 1 second without actually copying the device from the Pi to the Mac. I also have passwordless login setup for SSH and that works fine – verified with -v switch. If I add a -t to the ssh command, it just sits there on the Mac and there is a process started found via pgrep but it’s dormant. Guessing it’s sitting waiting for something like a password. Edited the sudoers via visudo and added
    %adm ALL=(ALL) NOPASSWD: /bin/dd if=/dev/mmcblk0
    to attempt to allow pi (default user) to execute dd against that device. Still no good.

    Like

    • John A says:

      It may have something to do with your no-password login. When I run the command it asks me for the Pi’s password. I enter it and the command runs.

      Like

      • Angelo says:

        I think I have it working although not sure why on a Mac it does this as I’ve been doing dd via ssh for a lot of years. Anyway, I used the same basic command WITHOUT the quotes and it worked! Go figure. Maybe it’s an OSX version thing? The passwordless is setup so that I can cron the backup and it works fine. Anyway, thanks for the detailed instructions!

        Liked by 1 person

      • John A says:

        Fantastic! Glad it’s working Angelo.

        Like

    • John A says:

      When you use the “-t” option and it doesn’t do anything, that is the process running properly. Pressing CTRL-T will give you a status as I explained in the post.

      Liked by 1 person

  13. mrbeezer says:

    Worked great except ran out of room on sd card. can this be used to write to a usb stick?

    Like

  14. Joost says:

    I am trying this command but cannot get it to work:

    ssh pi@192.168.178.144 “sudo dd if=/dev/mmcblk0 bs=1M | gzip -” | dd of=~/Desktop/pibackup.gz
    pi@192.168.178.144‘s password:
    sudo: no tty present and no askpass program specified
    0+1 records in
    0+1 records out
    20 bytes transferred in 4.103791 secs (5 bytes/sec)

    Every time the “sudo: no tty present and no askpass program specified” 😦
    On the Pi I edited the sudoers file (/etc/sudoers) via “sudo visudo” with the following:
    pi ALL=(ALL) NOPASSWD: ALL

    Like

  15. […] the way, I thoroughly recommend backing up the SD card. I always clone my Pi project cards but on this particular project the card actually corrupted […]

    Like

  16. This is great.. .can yo help me find a way to backup the user home to a zip file on the Mac?

    Like

  17. John A says:

    Try this:
    zip -r /path/to/target.zip /path/to/source

    Like

  18. John A says:

    In your case:
    zip -r ~/Desktop/homefolder.zip /Users/YourHomeFolder

    Would save the contents of your home folder to a zip file on your desktop.

    Like

  19. Andrea says:

    Doesn’t work for me, at least the restore. I have tried the command, and it seems everything went fine, but the SD card is unreadable on my Mac, and if I try to insert it into the pi and boot it it won’t work. Can you please help me? I have connected the SD card on my Mac and used the command you wrote to restore it…

    Like

  20. mannte says:

    The dd comment can now also display the progress by using status=progress. So the command then changes to:
    ssh pi@xx.xx.xx.xx “sudo dd if=/dev/mmcblk0 bs=1M status=progress | gzip -” | dd of=~/xx

    Liked by 2 people

  21. Joost says:

    Finally I think I have got it working.

    As root:
    “usermod -aG sudo ”
    For my example will be “pi” this will give the pi user the option to make use of sudo commands.

    “visudo”
    command must be run to avoid the password to be asked again for commands. The command will give you a text editor, where at the very end the following command should be placed: “pi ALL=(ALL) NOPASSWD: ALL” and exited out and saved.

    Next, exit root and this can test if your sudo is working

    sudo fdisk -l
    This will give you an overview of the device your SD-card on the Pi is named. In my case it was /dev/mmcblk0 (not the p1/p2 behind it)

    On your local system (Mac / Linux / NAS) you can enter below command:
    ssh pi@ “sudo /bin/dd if=/dev/mmcblk0 bs=4M ” | gzip -c > //$(date +%Y%m%d_%H%M%S)_pibackup.gz

    Above command will ssh to the Pi, and as sudo use the dd command (full path needed!) input file the device of your SD-card, this will be outputted to gzip, which will compress this input and writes it to the path defined with a date and timestamp and in my case add a name _pibackup.gz behind it. Handy, as you can make the command as a script (executable) and put it in cron and run it periodic.

    Like

  22. dragos says:

    this is the ruslt i get. the file is created but zero bytes.
    using the lsblk -p and sd card name is good.

    -bash: sudo dd if=/dev/mmcblk0 bs=1M | img -: No such file or directory
    0+0 records in
    0+0 records out
    0 bytes copied, 0.000523647 s, 0.0 kB/s

    Like

  23. Wolfgang says:

    Thanks much for the great post! I am struggling with only one part: The file is not saved on the Macbook but on the Raspberry Pi itself.

    I understand that this part “of=~/Desktop/pibackup.gz” should pump it to the desktop folder of my Mac. Well, it is still saved on the Raspberry Pi..

    Like

  24. John A says:

    Try it without the status command and without the date variable and see if it copies to your Mac. You will know immediately because the file will appear on your desktop.

    Liked by 1 person

  25. Wolfgang says:

    Problem solved! I had used it when connected to the Pi in Terminal, not when the user was MBP. Thanks, John!

    Liked by 1 person

  26. Wolfgang says:

    There is one thing I don’t fully understand about the compression. I am using the above command on a Pi with a 32GB SD-card. 26GB of which is unused. The size of the backup file is 28GB. Shouldn’t it be less with only 6GB used?

    Like

    • John A says:

      The way I understand how the command “dd” works is that it is OS agnostic. It literally copies the bare metal drive, even if they are random 1’s and 0’s. This way it will copy system partitions etc, regardless of the OS.

      Like

  27. rob says:

    I’m having the same problem as Wolfgang, except I’m using a Windows 10 Pro computer. I have entered this command,

    ssh p1@192.168.1.113 “sudo dd if=/dev/mmcblk0 bs=1M status=progress | gzip -” | dd of=~/Desktop/pi_clone.gz,

    in a “Command Prompt” window. The file shows up on the Pi’s desktop.

    Like

  28. Rob says:

    I’m not logged on to the Pi. I tried to run that command from the Win10 “Run Command Box” (WinKey+R) and also in a TERMINAL window. In the later case it throws an error. In the Run Command Box it runs and completes successfully, but the file is saved on the Pi and not on the PC.

    Maybe I’m not running it correctly on my PC? Can you be more explicit?

    Like

    • John A says:

      I’m not a PC guy. I use Macs and other Unix based operating systems. But when you open a terminal window, the directory tree you are in should be that of your PC, not the Pi. You can check this by issuing the command “pwd” which stands for present working directory. The command in this post basically says: I am on my local (pc), SSH into my Pi (remote computer), copy the disk on the remote machine, gzip the information, and then pipe it back to my local machine.” If you start from the remote PC, it will SSH into ITSELF, and save to the desktop of the remote computer.

      Like

  29. Rob says:

    OK. That is what I thought would happen, but evidently the PC sends everything after the “ssh p1@192.168.1.113” to the remote device. I’ll continue to investigate and if/when I find the answer I will post it back here. Thanks for your time and this post….Rob

    Like

    • jerry says:

      For Windows users you need to download dd tool for win10 http://www.chrysocome.net/downloads/dd-0.5.zip
      (copy .exe file i.e. to Windows/system32), open cmd window and run:

      ssh pi@XXX.XXX.XXX.XXX “sudo dd if=/dev/mmcblk0 bs=1M | gzip -” | dd of=\\.\c:/Users/jerry/Desktop/rPi_backup.gz –progress

      you can check with:

      dd –list (in Win10 cmd terminal)

      And you will get a list of the devices connected to your computer and modify “\\.\c:/Users/” as nescessary

      Like

      • John A says:

        Thanks for the Win10 help Jerry. Will definitely be helpful for others not using a Mac or other Linux machine. .

        Like

      • RDK says:

        OK, it has been several months since I considered this idea for remote Pi backup. I’m now ready to try it again, but have a question before I get started. My Pi SD cards seem to have several partitions which Window sees (and wants to format) when I do a local backup using HDDRawCopy on my Windows box. HDDRawCopy creates a copy of the whole SD card, all partitions.

        I understand that this procedure will copy the /dev/mmcblk0 partition on the Pi to a file/folder on my PC.

        My question is about restoring that saved file to an SD card. What about the other partitions which seem to be on the current, active SD card? Are they necessary?

        Thanks….RDK

        Like

      • John A says:

        This will copy the entire device (entire SD card). All partitions. To a single, compressed file on your remote machine.

        If by other partitions, you are talking about the EFI partition, etc. Yes, they are necessary for your Pi’s operating system to work.

        Like

      • Rob says:

        Jerry….I have gotten this command to run, but after 30+ minutes it seems to have an error:
        c:\Windows\System32>ssh me@192.168.xxx.yyy “sudo dd if=/dev/mmcblk0 bs=1M | gzip -” | dd of=\\.\c:/Users/me/Desktop/rPi_backup.gz –progress

        The output and error message is:

        11,754,655,23215193+1 records in
        15193+1 records out
        15931539456 bytes (16 GB, 15 GiB) copied, 2528.86 s, 6.3 MB/s
        11,755,209,216Error reading file: 109 The pipe has been ended

        22959393+0 records in
        22959393+0 records out

        Have you ever seen this error? I’ve gotten it on two different Pi’s when trying to run this over my local network….RDK

        Like

  30. Miquel Noguer says:

    Hi, I tried to run the command from a mac to backup my raspberry and it creates an empty file.

    The command: ssh merrameu@192.168.1.12 -p 36000 “sudo dd if=/dev/mmcblk0 bs=1M | gzip -” | dd of=/Users/imac/Desktop/backupPi.gz

    And the response: Enter passphrase for key ‘/Users/imac/.ssh/id_rsa’:
    sudo: no tty present and no askpass program specified
    0+2 records in
    0+1 records out
    20 bytes transferred in 3.998514 secs (5 bytes/sec)

    Any clue?

    Thanks

    Like

  31. Lenson says:

    Awesome!! That really helped me.
    Maybe mentioned before, but if you want to run it on a Windows machine there is no dd.
    You can download dd for Windows here http://www.chrysocome.net/downloads/dd-0.5.zip .
    Run your shell and and start it in the directory you put the.exe in.
    And the answer from Miquel Noquer for problems wit hsudo access also worked fine if you’re not the root user.

    Like

  32. RDK says:

    Finally got around to trying this technique and it “seemed” to be working. BUT, for the two cases l tried, backing up two different Pi’s on my local network, both ended after 30+ minutes with this message:
    “Error reading file: 109 The pipe has been ended”

    Is this normal or is there really some problem with both Pi? Neither Pi indicates any corruption of “dirty bits”.

    Here is the fully output from my Win 10 command line window:

    c:\Windows\System32>ssh me@192.168.xxx.yyy “sudo dd if=/dev/mmcblk0 bs=1M | gzip -” | dd of=\\.\c:/Users/me/Desktop/rPi_backup.gz –progress

    rawwrite dd for windows version 0.5.
    Written by John Newbigin
    This program is covered by the GPL. See copying.txt for details
    0me@192.168.xxx.yyy‘s password:

    11,754,655,23215193+1 records in
    15193+1 records out
    15931539456 bytes (16 GB, 15 GiB) copied, 2528.86 s, 6.3 MB/s
    11,755,209,216Error reading file: 109 The pipe has been ended

    22959393+0 records in
    22959393+0 records out

    Like

    • John A says:

      Sorry RDK, I’m not a Windows guy. Don’t even have a Windows box to test this on. Check the comments above. A couple of Windows users were able to get it to work.

      Like

  33. […] 1. Use SSH and dd to Remotely Backup a Raspberry Pi | John … […]

    Like

  34. […] This references details a procedure for backing up the SD card on a running Raspberry Pi from and to a file on another computer (https://johnatilano.com/2016/11/25/use-ssh-and-dd-to-remotely-backup-a-raspberry-pi/) […]

    Like

  35. Lance Haile says:

    I get the following error: dd: ~/Desktop/pibackup.gz: No such file or directory

    Like

  36. John A says:

    Try using the complete address to your desktop.

    Like

Leave a comment