Creating a RAID 1 Array for Data Storage from an Existing External Hard Drive Without Data Loss

I have an external hard drive used to store all my pictures, media and videos. Over the years, I have had to enlarge my storage capacity to make room for new additions. Eventually, I could no longer fit into my network-attached storage device and found a great deal on a much larger external hard drive. Unfortunately, this didn’t offer the RAID 1 backup capacity of the two bay NAS. I have the drive connected to a dedicated Linux computer and eventually purchased a second copy of the identical external hard drive once the funds were available. (For some reason the external drive is cheaper than even the least expensive matching drive without housing)

I started by pulling both drives out of their enclosures. I had one drive connected with data, and one drive connected without data. What to do now?

The typical process for this would be to backup all the data to a 3rd drive, connect both drives to the system, and wipe them both to create a new RAID 1 array. Then I would have to copy all the data back from the 3rd drive into the software RAID 1 drive. That requires a 3rd drive or multiple drives with enough capacity to store all the data. In a massive data center, I’m sure that’s no problem. In my basement media room, I don’t have a bunch of extra TBs of space sitting around.

I needed to find a way to get the data into a RAID 1 array without destroying it in the process. It’s possible to create the RAID 1 array in a degraded state. That means it will only have one device in the array. Typically this state happens when a device fails and a new drive is inserted which then receives a complete copy of the data.

The new drive needs to be formatted. I went with ext4 using gparted. Just install the software and follow the gparted instructions to delete all existing partitions and create a new one. Leave 100Mb at the end of the drive to avoid issues with manufacturing differences between the drives. Be VERY careful not to delete the wrong drive. Remember they are identical, and you can quickly get confused. If you erase all your data, you have no backups and no data which is very bad and the whole point of this exercise.

sudo apt install gparted

Then you can create the RAID array with mdadm. This process is decently advanced, so don’t proceed until you know what you’re doing. Read the manual for mdadm. Be sure to change /dev/abcde to the actual partition you just created. Again if you do this wrong, you’re going to lose all your data and be very sad. The missing keyword lets you create the device in a degraded state.

sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 missing /dev/abcde

I got a warning about not being able to boot the device, which was fine with me because this is just for data storage. To confirm that it completed correctly, check the magical file which stores raid information.

sudo tail /proc/mdstat

Once the process completes, you will have a drive available at /dev/md0, but it has no file system. If you get the error below, then you need to create the files system.

mount: /home/ste/mnt: wrong fs type, bad option, bad superblock on /dev/md0, missing codepage or helper program, or other error.

sudo mkfs.ext4 /dev/md0

Now that you’ve got the file system setup you can mount the “array” which is actually just the one drive.
I changed the owner to my user so that I could use the UI to copy the data, but that didn’t work. I was still required to type my password in a bunch, so I just launched nautilus as root for this copy which creates all the new files as root.

sudo mkdir /mnt/new-raid
sudo mount /dev/md0 /mnt/new-raid
sudo nautilus

You can spend the next 15 hours coping your data onto the new drive using the UI or the command line. When you’re done, you will likely need to change the owner of all the new files back to your user.

sudo chown -R myuser: /mnt/new-raid

I added the array configuration to the mdadm.conf file. /etc/mdadm/mdadm.conf. Use this command to get the configuration line and add it to the end of the config file.

sudo mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Next, you need to make the old drive match the new drive. My previous drive was in an enclosure and mounted using the fstab. When I removed this device, Ubuntu wouldn’t boot correctly, so I needed to remove my custom mounting before switching around the device. While there I set up the mount for the new array. The UUID is for md0 which can be found in “lsblk.” Don’t use the ones for the drives which are involved in the array. Use the “mount” command to reload the fstab file and confirm your new entry is correct.

sudo lsblk -f
sudo gedit /etc/fstab
UUID=2d603a18-7ed4-4335-906c-fa334f4cad93 /media/Videos ext4 defaults 0 1
sudo mount -av

At this point, you’ve got your new drive setup and mounted the way you wanted it to show up in the file system. You’ve restarted a couple of times to make sure it’s working, and you’ve copied all your files over as the new working copy of the data. Remember you’re flying without a dependable backup so the files on the new RAID will be the only copy of the files. Be sure everything is there and working.

The next step is to wipe the old drive and add it into the RAID array then copy the files from the new drive onto the old drive to complete the process. Use sgdisk to create a backup of the partition table from the old drive. Move that backup to the new drive to ensure identical partitions. Updating the partition table is a destructive process and will wipe out all data on the target drive. The data on the drive which performed the backup drive should be unaffected.

sudo sgdisk --backup=table /dev/sdc
sudo sgdisk --load-backup=table /dev/sdb
sudo sgdisk -G /dev/sdb

Use fdisk or lsblk or blkid to confirm that the partition matches. It should be exactly identical.

fdisk -l

Restart your computer to make the new changes active in the operating system. This drive will need to have it’s RAID and file system setup like the previous drive. At this point, my array is getting set as /dev/md127 instead of the expected /dev/md0 on restart. Stop the array and then have mdadm recreate it to get it back to the md0. You should get a message mdadm: /dev/md/0 has been started with 1 drive (out of 2)

sudo umount /dev/md127
sudo mdadm --stop /dev/md127
sudo mdadm --assemble --scan

To add the newly partitioned drive to the array use mdadm. /dev/md0 is the target array. -a is for adding a new partition and /dev/sdb1 is the partition to add.

sudo mdadm /dev/md0 -a /dev/sdb1
sudo blkid

Looking at the blkid output you should see the 2 partitions sdb1 and sdc1 both of which should be TYPE=”linux_raid_member.” Additionally, you should check the details for your array to make sure everything is functioning correctly.

sudo mdadm --detail /dev/md0

You should now see 2 drives at the bottom of the output. Confirm that the raid level is RAID1, Array Size is what you expected, Raid devices and Total Devices should match at 2 since this is a RAID 1 setup. At the bottom of the screen, the rebuilding process should have already begun when you added the new drive. State spare rebuilding.

If you would like to watch the progress in the terminal, you can use this command to get an updating progress view.

sudo watch cat /proc/mdstat

 

Here are some useful commands for working with the mdadm files and troubleshooting.

sudo watch cat /proc/mdstat
sudo tail /proc/mdstat
sudo nano /etc/mdadm/mdadm.conf
sudo mdadm --detail /dev/md0
sudo mdadm --stop /dev/md0
sudo mdadm --assemble --scan
sudo blkid
sudo lsblk -f
sudo mdadm -Db /dev/md0
sudo mount -av

Sources:
https://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID#Create_the_RAID_device
https://askubuntu.com/questions/973632/unable-to-mount-raid1-md0-wrong-fs-type-bad-option-bad-superblock-on-dev-md
https://unix.stackexchange.com/questions/320103/whats-the-difference-between-creating-mdadm-array-using-partitions-or-the-whole
https://www.digitalocean.com/community/tutorials/how-to-manage-raid-arrays-with-mdadm-on-ubuntu-16-04
https://superuser.com/questions/287462/how-can-i-make-mdadm-auto-assemble-raid-after-each-boot
https://tech.feedyourhead.at/content/copy-partition-table-one-disk-another

How To Set Up a Plex Media Server on Ubuntu 18.04

I did a minimum install for Ubuntu.

Install chrome from the web


Install OpenSSH for convenience

sudo apt install openssh-sever
sudo systemctl status ssh

install atomic toolkit for easy software setup

sudo apt-get install git
sudo git clone https://github.com/htpcBeginner/AtoMiC-ToolKit /opt/AtoMiC-ToolKit
cd /opt/AtoMiC-ToolKit
sudo bash setup.sh
sudo atk

Install sonarr
Transfer settings from old sonarr

  1. Re-install Sonarr
  2. Run Sonarr once to get the AppData directory location
  3. Stop Sonarr
  4. Copy NZBDrone Config Folder to new Install (/home/”username”/.config/NZBDrone)
  5. Extract the backup zip file & restore the files extracted from the zip
  6. Start Sonarr once you have setup the data files
  7. As long as the paths are the same, everything will pickup where it left off

Install Watcher
Create Backup Old Computer
Transfer watcher3.zip to new computer
Install Backup New Computer


Install Sabnzb+
Transfer settings
copy over /usr/share/sabnzbdplus/sabnzbdplus.ini
copy over /usr/share/sabnzbdplus/admin/history1.db
copy over /usr/share/sabnzbdplus/sabnzbd/postprocessing

sudo systemctl restart sabnzbdplus.service

Install Plex
Plex install through ATC fails so download from website
copy data files from /var/lib/plexmediaserver/Library/Application Support/Plex Media Server to new folder
ensure permissions are correct for new plexserver
sudo chown plex:plex


Mount external drive with content to /media/Videos

Plex Running on Ubuntu / Linux / Lubuntu Doesn’t Display My External Drive


Install OpenVPN

https://linuxize.com/post/how-to-enable-ssh-on-ubuntu-18-04/
https://github.com/Sonarr/Sonarr/wiki/Backup-and-Restorehttps://github.com/htpcBeginner/AtoMiC-ToolKit

 

Setup Simplify 3D to Print PLA with a PVA Dense Support Layer Palette 2 or Palette+

In this example, I will show how to setup Simplify3D slicing software to produce g-code that can be digested by the Mosaic Manufacturing software to produce dissolvable support prints. A word of warning: this is a very tricky printing technique. You will likely fail ten prints before you get it working correctly, but once you get it dialed in, save your factory file exactly as you had it and use that file for future prints. Don’t try to get all these settings to work with the “Auto-Configure” that’s built into S3D.

The rewards are fantastic for those who are brave enough. In my experience, you don’t need to dissolve the support at all. It will easily peel away from the PLA since PVA and PLA don’t like to stick together much. Support removal time can be 10 seconds for simple models.

If this post helps you then PLEASE support my feature request on the Simplify3D forum. It directly relates to the improvement of these features and so should be helpful for you also!

https://forum.simplify3d.com/viewtopic.php?f=23&t=11487

 


You will need to create a Process file with at least two extruders. One for each mater that you will be working with.

I find it’s convenient to rename the extruder to include the material name to help keep things organized. Remember the Mosaic software will convert these extruders back into a single extruder for your printer.

On the layer tab, this is where you start setting up the materials.

Next is the additions tab. I recommend that you use only a single material for additions at first. If you select All Extruders for the skirt, then you will have extra material changes for no benefit. Prime Pillar will be generated by Mosaic so leave that turned off.

On the Infill tab, you are selecting what material will make up the inside of your part so that should be the same as the Layers tab.

Finally the support tab, which is the critical tab. The support extruder should be your inexpensive material, and the PVA goes on the dense support extruder.


That’s all you need to do to get it working but here are my tips below.

I don’t want any separation between my part and my support material, so I change all those parameters to 0 on the support tab.

PVA doesn’t like to stick to PLA which is the entire point of this process. However, that makes it a little complicated to print the two together. This is especially true for the first layers when you are printing the PVA onto the thin PLA support structures. I highly recommend increasing the Extrusion width by a significant factor. I double my extrusions, but your printer will likely be different.  For the PVA extruder change the Extrusion Width to Manual and set a high value. This ensures that the PVA smashes down into the layer before it which increases the chances of adhesion.

Another thing which will help with adhesion is to print the PVA super slowly. 50% of your other support prints. There’s no way to do this easily in the software so you might have to come up with workarounds. I use OctoPi’s interface to manually adjust the speed during the print and then turn it back up.

PLEASE! Support the feature request post which directly relates to this improvement.

https://forum.simplify3d.com/viewtopic.php?f=23&t=11487

Happy crab hopes you enjoy your dense support structures!

 

Adding a network share to Ubuntu 18 for Windows 10 access

I’ve previously written some on this topic but I had to go through the process again on a clean install of the latest Ubuntu. Using the build in tools that came with Ubuntu 18 got me close but not all the way to where I wanted to be. I didn’t want other users on my network to have to log in, and I was unable to edit files on the shared drive. The workflow below worked to get everything working from my Windows 10 machine.

At first I installed the samba GUI from the software GUI but I got errors about not having the proper users when I tried to actually use it so I would recommend doing it form the terminal.

sudo apt update
sudo apt upgrade
sudo apt install system-config-samba

For some reason this is the name of samba — just go with it.

sudo system-config-samba

To launch the GUI (this probably will not work on your first try unless they fix the software) To get this to work I have to create the config file that is missing.

sudo touch /etc/libuser.conf

Then start the GUI and get much better control over adding network shares.

sudo system-config-samba

Setup a headless Raspberry Pi Zero with Wifi from Windows- Raspbian Stretch

The Raspberry Pi Zero has an unusually sized HDMI port which means it can be hard to hook up to a monitor. If you’re just using it for the terminal you don’t need to see the output anyway. Here’s how to set it up with SSH and Wifi so you’ll just need a power cable to get started.

Download the latest Raspbian Stretch image. Don’t use the noobs installer.

Raspbian

Follow their installation guide to get it onto the SD card. You’ll need to flash the image to the card with Etcher or a similar tool to make sure everything has the correct permissions and boots properly.

https://etcher.io/

Etcher automatically ejects the disk once flashing is completed so remove it from your computer and reinsert it. You’ll probably see a message about needing to format the drive before you can use it. Be sure not to format the drive or you will have to start over.

Open a text editor like Notepad++ and create a file on the drive called “ssh”. The tricky part here is that it need to have no extension. In Windows this is kind of frowned upon but to get it to happen chose save as > change the file type to “all files (*.*)” and then save it as “ssh” with the quotation marks. The quotation marks will be stripped away and no extension will be added. This file can be blank.

Create one more file and save it as “wpa_supplicant.conf” also to the root of the drive so it’s one of the first files that you see. This file needs your wireless network information and be sure it’s correct or you’ll spend a while troubleshooting.

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
network={
    ssid="YOUR_SSID"
    psk="YOUR_WIFI_PASSWORD"
    key_mgmt=WPA-PSK
}

Then eject the drive and put it into your Raspberry Pi. You can power it up but I would recommend pulling up your router admin page now before you do. This should make it easier to spot the new IP address which you’ll need to connect to the Raspberry Pi. By default, it’s called “raspberrypi” on the network but it’s not as reliable in my experience as the IP address.

Default user and password pi and raspberry

Once you have that you’re ready to SSH into the machine with your preferred SSH program. Personally, I use Bitvise SSH which has tons of functionality but still is the easiest to configure correctly. Here’s the website. The download link is “Bitvise SSH Client installer “

How to Install Backend Theme on Odoo 11 Community

Download the theme from the theme store

https://www.odoo.com/apps/themes/11.0/backend_theme_v11/

This will contain 2 files which you need to add into your addon directory. If you installed with Bitnami then it’s in

/apps/odoo/data/addons

You need to copy the 2 files into that directory. The configuration file by default does not include this directly so you need to add it.

/apps/odoo/conf/odoo-server.conf

The first line should say “addons_path.” At the end of that line add a comma (,) and then the full path to your addon directory.

addons_path = /home/ubuntu/odoo-11.0.20171118-2/apps/odoo/lib/odoo-11.0.post20171118-py3.6.egg/odoo/addons, /home/ubuntu/odoo-11.0.20171118-2/apps/odoo/data/addons/

In Odoo go to the settings page and look for “Activate the developer mode” and click that

Then go to “Browse Apps” and remove the “Apps” from the search bar and search for theme. You may need to click “Update Apps List” if it doesn’t show up automatically.

Update: 06-12-2019 Icons Missing from Menu FileNotFoundException web_responsive

When I installed web_responsive I had some of the permissions wrong for the addon directory. Once I fixed the permissions I was unable to get the icons for the menu to show up. The files were not created in the filestore and no quantity of restarts or reinstalls would resolve the issue. Ultimate I stopped the service, added the update all flag to the service, and restarted the service. Update all took a few minutes to complete. Then I removed the update all command and restarted the service

sudo systemctl stop odoo11.service 
/home/odoo11/odoo11-venv/bin/python3 /home/odoo11/odoo/odoo-bin -c /etc/odoo11.conf --update all
sudo systemctl daemon-reload
sudo systemctl start odoo11.service

Error Installing Bitnami Package on AWS t2.Nano – Unable to create symbolic link

I was installing the Bitnami Odoo 11 module on my AWS t2.nano. It’s a fresh instance with a 1gb swap partition and 500mb of hardware RAM. This should be enough to run most if not all Bitnami packages. However when I was installing the package ontop of Bitnami LAMP I got error messages.

Unable to create symbolic link

TLDR: You probably need to restart the machine and install from the command line.

I tried a variety of things to resolve the error

confirm that you have enough memory

free

confirm that you have sufficient disk space

df

confirm that the user can create symbolic links in the location mentioned by bitnami

link -s bitnami_source bitnami_target

If you are able to do all these things a simple restart fixed the problem for me. You can run the installed from the command line without giving resources to the GUI which also seems to help. I actually found it a little nicer than the GUI interface.

cd /user/home/directory_of_download
./bitnami-installer-packge.run --mode text

Lightweight xfce4 Setup for AWS t2.nano with Ubuntu image and remote tightVNC access

A good place to start reading and understanding what is needed for a GUI

https://help.ubuntu.com/community/Installation/LowMemorySystems

My configuration is below for a minimum system that I consider to be a useable starting point.

sudo apt install xfce4 xfce4-goodies
sudo apt install tightvncserver
tightvncserver :1
sudo apt-get install gnome-icon-theme-full tango-icon-theme
sudo apt install gksu
sudo apt install synaptic
sudo apt install chromium-browser
sudo apt install gedit

open gedit and edit the file /usr/share/applications
change Exec= to “Exec=gksudo synaptic”

Total drive size is 2.144gb when run on the AWS Ubuntu image

Create a partition for the swap so applications have access to more RAM

sudo dd if=/dev/zero of=/mnt/swap.0 bs=1024 count=1048576
sudo mkswap /mnt/swap.0

Add to fstab to automatically mount the partition

sudo su 
echo "/mnt/swap.0 swap swap defaults 0 0" >> /etc/fstab
swapon /mnt/swap.0

Check your work

sudo swapon -s

https://docs.bitnami.com/installer/faq/linux-faq/#how-to-download-and-install-a-bitnami-stack

How to setup lxde tightVNCserver on AWS t2.nano

This will install the most basic desktop environment without any browser or package managers etc. It’s very limited and in my opinion not worth the extra configuration to get it up and running.

# Make sure Debian is the latest and greatest

sudo apt update && sudo apt upgrade

# Install X, LXDE, VPN programs

sudo apt install xorg lxde-core tightvncserver

# Start VNC to create config file and setup your password

tightvncserver :1

# To stop VNC sever if needed

tightvncserver -kill :1

https://www.vandorp.biz/2012/01/installing-a-lightweight-lxdevnc-desktop-environment-on-your-ubuntudebian-vps/

I did not need to follow these steps and tight VNC Server was working

# Edit config file to start session with LXDE: 
nano ~/.vnc/xstartup 

# Add this at the bottom of the file: 
lxterminal & 
/usr/bin/lxsession -s LXDE &

 

 

Mount a SFTP connection to a folder in Ubuntu / Linux

To do this I used a program called SSHFS which has done a great job. First, install if from the repo

sudo apt-get install sshfs

You’ll need to create a directory to use as the location for your files

sudo mkdir /mnt/sshftps

Execute the command to connect the actual SFTP Server. Replace xxx.xxx with the target IP address and use the -p option to specify the connection port. The user parameter is your user name. The :/ at the end of the IP address indicates the end of the IP address. Don’t try to put the port number after the colon

sudo sshfs -o allow_other -p 6789 user@xxx.xx.xxx.xx:/ /mnt/sshftps

This will create a semi-permanent connection which will close if the machine is restarted. You can create a permanent connection which will reopen when the machine starts by editing the fstab file in /etc/fstab. Add a command to the end and restart the machine. Personally, I haven’t tried this because it’s a potential security risk and I didn’t need 100% uptime.

sshfs#username@xxx.xxx.xxx.xxx:/ /mnt/sshftp