Creating a RAID 1 Array for Data Storage from an Existing External Hard Drive Without Data Loss

I have an external hard drive used to store all my pictures, media and videos. Over the years, I have had to enlarge my storage capacity to make room for new additions. Eventually, I could no longer fit into my network-attached storage device and found a great deal on a much larger external hard drive. Unfortunately, this didn’t offer the RAID 1 backup capacity of the two bay NAS. I have the drive connected to a dedicated Linux computer and eventually purchased a second copy of the identical external hard drive once the funds were available. (For some reason the external drive is cheaper than even the least expensive matching drive without housing)

I started by pulling both drives out of their enclosures. I had one drive connected with data, and one drive connected without data. What to do now?

The typical process for this would be to backup all the data to a 3rd drive, connect both drives to the system, and wipe them both to create a new RAID 1 array. Then I would have to copy all the data back from the 3rd drive into the software RAID 1 drive. That requires a 3rd drive or multiple drives with enough capacity to store all the data. In a massive data center, I’m sure that’s no problem. In my basement media room, I don’t have a bunch of extra TBs of space sitting around.

I needed to find a way to get the data into a RAID 1 array without destroying it in the process. It’s possible to create the RAID 1 array in a degraded state. That means it will only have one device in the array. Typically this state happens when a device fails and a new drive is inserted which then receives a complete copy of the data.

The new drive needs to be formatted. I went with ext4 using gparted. Just install the software and follow the gparted instructions to delete all existing partitions and create a new one. Leave 100Mb at the end of the drive to avoid issues with manufacturing differences between the drives. Be VERY careful not to delete the wrong drive. Remember they are identical, and you can quickly get confused. If you erase all your data, you have no backups and no data which is very bad and the whole point of this exercise.

sudo apt install gparted

Then you can create the RAID array with mdadm. This process is decently advanced, so don’t proceed until you know what you’re doing. Read the manual for mdadm. Be sure to change /dev/abcde to the actual partition you just created. Again if you do this wrong, you’re going to lose all your data and be very sad. The missing keyword lets you create the device in a degraded state.

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/abcde

I got a warning about not being able to boot the device, which was fine with me because this is just for data storage. To confirm that it completed correctly, check the magical file which stores raid information.

sudo tail /proc/mdstat

Once the process completes, you will have a drive available at /dev/md0, but it has no file system. If you get the error below, then you need to create the files system.

mount: /home/ste/mnt: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error.

sudo mkfs.ext4 /dev/md0

Now that you’ve got the file system setup you can mount the “array” which is actually just the one drive.
I changed the owner to my user so that I could use the UI to copy the data, but that didn’t work. I was still required to type my password in a bunch, so I just launched nautilus as root for this copy which creates all the new files as root.

sudo mkdir /mnt/new-raid
sudo mount /dev/md0 /mnt/new-raid
sudo nautilus

You can spend the next 15 hours coping your data onto the new drive using the UI or the command line. When you’re done, you will likely need to change the owner of all the new files back to your user.

sudo chown -R myuser: /mnt/new-raid

I added the array configuration to the mdadm.conf file. /etc/mdadm/mdadm.conf. Use this command to get the configuration line and add it to the end of the config file.

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Next, you need to make the old drive match the new drive. My previous drive was in an enclosure and mounted using the fstab. When I removed this device, Ubuntu wouldn’t boot correctly, so I needed to remove my custom mounting before switching around the device. While there I set up the mount for the new array. The UUID is for md0 which can be found in “lsblk.” Don’t use the ones for the drives which are involved in the array. Use the “mount” command to reload the fstab file and confirm your new entry is correct.

sudo lsblk -f
sudo gedit /etc/fstab
UUID=2d603a18-7ed4-4335-906c-fa334f4cad93 /media/Videos ext4 defaults 0 1
sudo mount -av

At this point, you’ve got your new drive setup and mounted the way you wanted it to show up in the file system. You’ve restarted a couple of times to make sure it’s working, and you’ve copied all your files over as the new working copy of the data. Remember you’re flying without a dependable backup so the files on the new RAID will be the only copy of the files. Be sure everything is there and working.

The next step is to wipe the old drive and add it into the RAID array then copy the files from the new drive onto the old drive to complete the process. Use sgdisk to create a backup of the partition table from the old drive. Move that backup to the new drive to ensure identical partitions. Updating the partition table is a destructive process and will wipe out all data on the target drive. The data on the drive which performed the backup drive should be unaffected.

sudo sgdisk --backup=table /dev/sdc
sudo sgdisk --load-backup=table /dev/sdb
sudo sgdisk -G /dev/sdb

Use fdisk or lsblk or blkid to confirm that the partition matches. It should be exactly identical.

fdisk -l

Restart your computer to make the new changes active in the operating system. This drive will need to have it’s RAID and file system setup like the previous drive. At this point, my array is getting set as /dev/md127 instead of the expected /dev/md0 on restart. Stop the array and then have mdadm recreate it to get it back to the md0. You should get a message mdadm: /dev/md/0 has been started with 1 drive (out of 2)

sudo umount /dev/md127
sudo mdadm --stop /dev/md127
sudo mdadm --assemble --scan

To add the newly partitioned drive to the array use mdadm. /dev/md0 is the target array. -a is for adding a new partition and /dev/sdb1 is the partition to add.

sudo mdadm /dev/md0 -a /dev/sdb1
sudo blkid

Looking at the blkid output you should see the 2 partitions sdb1 and sdc1 both of which should be TYPE=”linux_raid_member.” Additionally, you should check the details for your array to make sure everything is functioning correctly.

sudo mdadm --detail /dev/md0

You should now see 2 drives at the bottom of the output. Confirm that the raid level is RAID1, Array Size is what you expected, Raid devices and Total Devices should match at 2 since this is a RAID 1 setup. At the bottom of the screen, the rebuilding process should have already begun when you added the new drive. State spare rebuilding.

If you would like to watch the progress in the terminal, you can use this command to get an updating progress view.

sudo watch cat /proc/mdstat

 

Here are some useful commands for working with the mdadm files and troubleshooting.

sudo watch cat /proc/mdstat
sudo tail /proc/mdstat
sudo nano /etc/mdadm/mdadm.conf
sudo mdadm --detail /dev/md0
sudo mdadm --stop /dev/md0
sudo mdadm --assemble --scan
sudo blkid
sudo lsblk -f
sudo mdadm -Db /dev/md0
sudo mount -av

Sources:
https://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID#Create_the_RAID_device
https://askubuntu.com/questions/973632/unable-to-mount-raid1-md0-wrong-fs-type-bad-option-bad-superblock-on-dev-md
https://unix.stackexchange.com/questions/320103/whats-the-difference-between-creating-mdadm-array-using-partitions-or-the-whole
https://www.digitalocean.com/community/tutorials/how-to-manage-raid-arrays-with-mdadm-on-ubuntu-16-04
https://superuser.com/questions/287462/how-can-i-make-mdadm-auto-assemble-raid-after-each-boot
https://tech.feedyourhead.at/content/copy-partition-table-one-disk-another

How To Set Up a Plex Media Server on Ubuntu 18.04

I did a minimum install for Ubuntu.

Install chrome from the web


Install OpenSSH for convenience

sudo apt install openssh-sever
sudo systemctl status ssh

install atomic toolkit for easy software setup

sudo apt-get install git
sudo git clone https://github.com/htpcBeginner/AtoMiC-ToolKit /opt/AtoMiC-ToolKit
cd /opt/AtoMiC-ToolKit
sudo bash setup.sh
sudo atk

Install sonarr
Transfer settings from old sonarr

  1. Re-install Sonarr
  2. Run Sonarr once to get the AppData directory location
  3. Stop Sonarr
  4. Copy NZBDrone Config Folder to new Install (/home/”username”/.config/NZBDrone)
  5. Extract the backup zip file & restore the files extracted from the zip
  6. Start Sonarr once you have setup the data files
  7. As long as the paths are the same, everything will pickup where it left off

Install Watcher
Create Backup Old Computer
Transfer watcher3.zip to new computer
Install Backup New Computer


Install Sabnzb+
Transfer settings
copy over /usr/share/sabnzbdplus/sabnzbdplus.ini
copy over /usr/share/sabnzbdplus/admin/history1.db
copy over /usr/share/sabnzbdplus/sabnzbd/postprocessing

sudo systemctl restart sabnzbdplus.service

Install Plex
Plex install through ATC fails so download from website
copy data files from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server to new folder
ensure permissions are correct for new plexserver
sudo chown plex:plex


Mount external drive with content to /media/Videos

Plex Running on Ubuntu / Linux / Lubuntu Doesn’t Display My External Drive


Install OpenVPN

https://linuxize.com/post/how-to-enable-ssh-on-ubuntu-18-04/
https://github.com/Sonarr/Sonarr/wiki/Backup-and-Restorehttps://github.com/htpcBeginner/AtoMiC-ToolKit

 

Adding a network share to Ubuntu 18 for Windows 10 access

I’ve previously written some on this topic but I had to go through the process again on a clean install of the latest Ubuntu. Using the build in tools that came with Ubuntu 18 got me close but not all the way to where I wanted to be. I didn’t want other users on my network to have to log in, and I was unable to edit files on the shared drive. The workflow below worked to get everything working from my Windows 10 machine.

At first I installed the samba GUI from the software GUI but I got errors about not having the proper users when I tried to actually use it so I would recommend doing it form the terminal.

sudo apt update
sudo apt upgrade
sudo apt install system-config-samba

For some reason this is the name of samba — just go with it.

sudo system-config-samba

To launch the GUI (this probably will not work on your first try unless they fix the software) To get this to work I have to create the config file that is missing.

sudo touch /etc/libuser.conf

Then start the GUI and get much better control over adding network shares.

sudo system-config-samba

How to Install Backend Theme on Odoo 11 Community

Download the theme from the theme store

https://www.odoo.com/apps/themes/11.0/backend_theme_v11/

This will contain 2 files which you need to add into your addon directory. If you installed with Bitnami then it’s in

/apps/odoo/data/addons

You need to copy the 2 files into that directory. The configuration file by default does not include this directly so you need to add it.

/apps/odoo/conf/odoo-server.conf

The first line should say “addons_path.” At the end of that line add a comma (,) and then the full path to your addon directory.

addons_path = /home/ubuntu/odoo-11.0.20171118-2/apps/odoo/lib/odoo-11.0.post20171118-py3.6.egg/odoo/addons, /home/ubuntu/odoo-11.0.20171118-2/apps/odoo/data/addons/

In Odoo go to the settings page and look for “Activate the developer mode” and click that

Then go to “Browse Apps” and remove the “Apps” from the search bar and search for theme. You may need to click “Update Apps List” if it doesn’t show up automatically.

Update: 06-12-2019 Icons Missing from Menu FileNotFoundException web_responsive

When I installed web_responsive I had some of the permissions wrong for the addon directory. Once I fixed the permissions I was unable to get the icons for the menu to show up. The files were not created in the filestore and no quantity of restarts or reinstalls would resolve the issue. Ultimate I stopped the service, added the update all flag to the service, and restarted the service. Update all took a few minutes to complete. Then I removed the update all command and restarted the service

sudo systemctl stop odoo11.service 
/home/odoo11/odoo11-venv/bin/python3 /home/odoo11/odoo/odoo-bin -c /etc/odoo11.conf --update all
sudo systemctl daemon-reload
sudo systemctl start odoo11.service

Lightweight xfce4 Setup for AWS t2.nano with Ubuntu image and remote tightVNC access

A good place to start reading and understanding what is needed for a GUI

https://help.ubuntu.com/community/Installation/LowMemorySystems

My configuration is below for a minimum system that I consider to be a useable starting point.

sudo apt install xfce4 xfce4-goodies
sudo apt install tightvncserver
tightvncserver :1
sudo apt-get install gnome-icon-theme-full tango-icon-theme
sudo apt install gksu
sudo apt install synaptic
sudo apt install chromium-browser
sudo apt install gedit

open gedit and edit the file /usr/share/applications
change Exec= to “Exec=gksudo synaptic”

Total drive size is 2.144gb when run on the AWS Ubuntu image

Create a partition for the swap so applications have access to more RAM

sudo dd if=/dev/zero of=/mnt/swap.0 bs=1024 count=1048576
sudo mkswap /mnt/swap.0

Add to fstab to automatically mount the partition

sudo su 
echo "/mnt/swap.0 swap swap defaults 0 0" >> /etc/fstab
swapon /mnt/swap.0

Check your work

sudo swapon -s

https://docs.bitnami.com/installer/faq/linux-faq/#how-to-download-and-install-a-bitnami-stack

How to setup lxde tightVNCserver on AWS t2.nano

This will install the most basic desktop environment without any browser or package managers etc. It’s very limited and in my opinion not worth the extra configuration to get it up and running.

# Make sure Debian is the latest and greatest

sudo apt update && sudo apt upgrade

# Install X, LXDE, VPN programs

sudo apt install xorg lxde-core tightvncserver

# Start VNC to create config file and setup your password

tightvncserver :1

# To stop VNC sever if needed

tightvncserver -kill :1

https://www.vandorp.biz/2012/01/installing-a-lightweight-lxdevnc-desktop-environment-on-your-ubuntudebian-vps/

I did not need to follow these steps and tight VNC Server was working

# Edit config file to start session with LXDE: 
nano ~/.vnc/xstartup 

# Add this at the bottom of the file: 
lxterminal & 
/usr/bin/lxsession -s LXDE &

 

 

Mount a SFTP connection to a folder in Ubuntu / Linux

To do this I used a program called SSHFS which has done a great job. First, install if from the repo

sudo apt-get install sshfs

You’ll need to create a directory to use as the location for your files

sudo mkdir /mnt/sshftps

Execute the command to connect the actual SFTP Server. Replace xxx.xxx with the target IP address and use the -p option to specify the connection port. The user parameter is your user name. The :/ at the end of the IP address indicates the end of the IP address. Don’t try to put the port number after the colon

sudo sshfs -o allow_other -p 6789 user@xxx.xx.xxx.xx:/ /mnt/sshftps

This will create a semi-permanent connection which will close if the machine is restarted. You can create a permanent connection which will reopen when the machine starts by editing the fstab file in /etc/fstab. Add a command to the end and restart the machine. Personally, I haven’t tried this because it’s a potential security risk and I didn’t need 100% uptime.

sshfs#username@xxx.xxx.xxx.xxx:/ /mnt/sshftp

Plex Running on Ubuntu / Linux / Lubuntu Doesn’t Display My External Drive

When moving all my files onto an external drive Plex was able to display the drive but was unable to read the content of the folder.

Changing group ownership and permissions did not solve this problem.
The drive was formatted with NTFS before being installing it in a housing and connected to the Linux machine.

The problem is caused by the default mounting parameters when first connecting the drive. In order to fix it you will need to manually setup the drive by editing the fstab file. Use this command

sudo blkid

This will get you the drive UUID of the drive that you need. Look at the labels until one of them looks familiar. You can also use the built in disk utility to get this number. System > Disks > “Gear Icon” > Edit Mount Options but it’s harder than just doing it in the terminal.

Create a folder to use as the path for your drive. I used /media/Videos since that’s what I would be storing.

sudo mkdir /media/Videos

Open the fstab file and add this line to the end using the UUID that you just found.

sudo gedit /etc/fstab
UUID=E12345A1234C1A12345 /media/Videos ntfs-3g defaults,permissions,auto 0 1

It’s easiest just to restart your computer to get these changes applied. You could also unmount the drive and then run the mount command which will pick up the new settings from the fstab file. If you have trouble with this just reboot

sudo mount /media/Videos

Without doing the next couple of steps I didn’t consistently have problems but I did have problems with permissions and file transfer being denied

cd /etc/samba
sudo gedit smb.conf

Scroll to the bottom and add into the section that was just created with the name of your share [videos] in this case

force user = yourUserName

And then restart Samba

sudo restart smbd

This other post helped me get up and running if you want another take on the same process.
http://travelinlibrarian.info/2013/05/how-to-share-an-external-usb-hard-drive-from-ubuntu-to-a-windows-network/

Setting up OctoPi (OctoPrint) with Email Notifier yagmail

I found a few guides to this on the internet but none of them worked for me out of the box using the latest (as of today) Octopi image and a Raspberry Pi 3. What I did eventually get to work was the following

  1. Login to your raspberrypi using SSH. I use Bitvise SSH for all my SSH needs.
  2. Run a few commands to make sure your octoprint is setup correctly. The explanation for the lines is as follows. Activate octoprint environment -> install yagmail (probably already installed) -> install keyrings.alt which is needed by yagmail -> activate python -> run the yagmail registration command
    source ~/oprint/bin/activate
    pip install yagmail
    pip install keyrings.alt
    python
    import yagmail
    yagmail.register('youraccount@gmail.com', 'yourpassword')
  3. While you’re here do a test run to make sure you can connect (this will only work for Gmail SMTP servers. It’s possible to get it working with other servers but you would have to modify some source code to get the ports right. Much easier to just use Gmail)
    yagmail.SMTP(user='youraccount@gmail.com')

    You should get back

    <yagmail.yagmail.SMTP instance at ...>
  4. Install “Email Notifier” through the OctoPrint web interface if you haven’t already.
  5. Restart your RaspberryPi – This may seem unnecessary but I highly recommend it. I ran into lots of problems with emails not sending that were fixed by a simple reboot.