2018

Building a Raspberry Pi Cluster

I came across a presentation by Ray Tsang et al. that showcased a Raspberry Pi cluster with Docker and Kubernetes support 1. It is very useful to have a personal Raspberry Pi cluster to explore concepts of distributed systems and create proof of concept prototypes. This post will be updated with my experiences on how to create and maintain a Raspberry Pi cluster, so I will add content over time.

1. Parts

My cluster is based on Raspberry Pi 3 Model B. I tried to follow the list of parts presented in 1. Below I list the parts and links to each part in amazon (USA).

Part URL
Raspberry Pi 3 Model B Motherboard link
Anker 60W 6-Port USB Wall Charger link
Samsung 32GB 95MB/s (U1) MicroSD EVO Select Memory Card link
Cat 6 Ethernet Cable 5 ft (5 PACK) Flat Internet Network Cable link
NETGEAR N300 Wi-Fi Router with High Power 5dBi External Antennas (WNR2020v2) link
Kingston Digital USB 3.0 Portable Card Reader for SD, microSD. link

I 3D printed the cluster rack using the files Raspberry Cluster Frame.

setup

2. Operating System

I selected Hypriot OS since it already has support for docker. Also, with HypriotOS 1.7 and up, it is possible to use cloud-init to automatically change some settings on first boot. The version of cloud-init that it seems to support is v0.7.9.

Since my cluster is based on Raspberry PI 3 model B which includes a Quad Core 1.2GHz Broadcom BCM2837 64bit CPU 2, then I use the releases of Hypriot OS provided by the github repo DieterReuter/image-builder-rpi64 (64 bit distribution) instead of the ones provided by the repo hypriot/image-builder-rpi (32-bit distribution) 3.

2.1. Flashing the OS

First, I decided to use the command line script developed by hypriot and hosted on github hypriot/flash 4. The advantage of this over a more simple method such as sudo dd if=image.img of=/dev/rdisk2 bs=1m. After installing the dependencies as described in the hypriot/flash repo, I installed the command line utility in $HOME/bin.

$ # get the release
$ curl -O https://raw.githubusercontent.com/hypriot/flash/master/flash
$ # add executable permissions
$ chmod +x flash
$ # move to ~/bin
$ mv flash $HOME/bin/
$ # export $HOME/bin to the path
$ export PATH=$HOME/bin:$PATH

Second, I got the OS image from DieterReuter/image-builder-rpi645:

$ wget https://github.com/DieterReuter/image-builder-rpi64/releases/download/v20180429-184538/hypriotos-rpi64-v20180429-184538.img.zip
$ wget https://github.com/DieterReuter/image-builder-rpi64/releases/download/v20180429-184538/hypriotos-rpi64-v20180429-184538.img.zip.sha256
$ # Verify the image with sha-256
$ shasum -a 256 hypriotos-rpi64-v20180429-184538.img.zip

Flashing the sd-card:

$ # Setting nodeX and with the user-data.yml
$ flash --hostname nodeX --userdata ./user-data.yml hypriotos-rpi64-v20180429-184538.img

You can find an example of the user-data.yml

2.1. Network configuration

I decided to configure the network interface using static IPs. The idea is to modify the file for eth0 which is located at sudo vim /etc/network/interfaces.d/eth0. As an example this is the setup for node1.

allow-hotplug eth0
iface eth0 inet static
    address 192.168.2.11
    network 192.168.2.0
    netmask 255.255.255.0
    broadcast 192.168.2.255
    gateway 192.168.2.1

I have a small Netgear WNR2020 wireless router dedicated to the Pi cluster, which is connected to another router that provides internet to it. I found useful to modify the DNS routes by adding the google DNS servers. Thus, I modified the file /etc/resolv.conf to look like (you need sudo to write to it):

$ cat /etc/resolv.conf
  nameserver 8.8.8.8
  nameserver 8.8.4.4
  nameserver 192.168.2.1

2.2. Setting up passwordless

I followed the guide by Mathias Kettner 6 to setup ssh login without password. Basically, this is a two part process. First, create the key in your local machine.

$ # generating a key that has not the default filename
$ ssh-keygen -t rsa
 Enter file in which to save the key (<user_path>/.ssh/id_rsa): <user_path>/.ssh/pic_id_rsa
 Enter passphrase (empty for no passphrase):
 Enter same passphrase again:
 Your identification has been saved in <user_path>/.ssh/pic_id_rsa.
 Your public key has been saved in <user_path>/.ssh/pic_id_rsa.pub.
 The key fingerprint is:
 ...

Second, setup the public key on the remote host.

$ # Create .ssh folder in remote machine
$ ssh ruser@remote 'mkdir -p ~/.ssh'
 ruser@remote's password:
$ # Send key.pub to remote machine
$ cat ~/.ssh/pic_id_rsa.pub | ssh ruser@remote 'cat >> ~/.ssh/authorized_keys'
 ruser@remote's password:
$ # Changing permissions in the remote machine of both .ssh and .ssh/authorized_keys
$ ssh ruser@remote 'chmod 700 ~/.ssh'
$ ssh ruser@remote 'chmod 640 ~/.ssh/authorized_keys'

Now, you can ssh to remote without password. I’m using a Mac laptop so I had to add the key to the ssh agent by $ ssh-add $HOME/.ssh/pic_id_rsa.

3. Docker

There are several docker images for arm64v8 in the docker registry.

References

2017

Starting With Clang Development

In this post we’ll setup Clang for out-of-the-tree development (3.9.0).

HOW TO

I assume that clang/llvm is already compiled from source and it is installed as we covered in building llvm with cmake. Lets assume that the path for the llvm installation is $HOME/opt/llvm. Make sure you exported the following env to the shell:

VERSION=3.9.0
export LLVM_HOME=$HOME/opt/llvm
export PATH=$LLVM_HOME/bin:$PATH
export C_INCLUDE_PATH=$LLVM_HOME/include:$LLVM_HOME/lib/clang/$VERSION/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=$LLVM_HOME/include:$LLVM_HOME/lib/clang/$VERSION/include:$LLVM_HOME/include/c++/v1:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=$LLVM_HOME/lib:$LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LLVM_HOME/lib:$DYLD_LIBRARY_PATH  

Clone the Clang repo.

I forked the clang repo hosted in github.com/llvm-mirror. You can either clone the repo or fork it.

Since I intent to modify the source code, I located the cloned folder in my preferred location for programming projects (lets say it is at $HOME/Dev/clang_dev).

Adding script to compile the project using cmake and ninja

I created a simple bash script that runs cmake to prepare for compilation and runs ninja for compiling the files. You need to replace $PATH_TO_INSTALL with the location of your choice.

#!/bin/bash
if [ -z "$1" ]; then
    echo "USAGE: compile.sh <phase> <dry_run>. It can be cmake, ninja_build, ninja_install"
    exit -1
fi
phase=$1
dry_run=""
TARGET_SRC=$PWD
TARGET_BUILD=$TARGET_SRC/../clang_dev_build
TARGET_INSTALL=$PATH_TO_INSTALL
if [ "$#" -ge "2" ]; then
    dry_run="-n"
fi
mkdir $TARGET_BUILD
echo Changing folders to $TARGET_BUILD...
cd $TARGET_BUILD
echo We\'re at $PWD
if [ "$phase" == "cmake" ]; then
  cmake $TARGET_SRC -G "Eclipse CDT4 - Ninja" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE="Release" -DCMAKE_INSTALL_PREFIX=$TARGET_INSTALL 
fi
if [ "$phase" == "ninja_build" ]; then
  ninja $dry_run
fi
if [ "$phase" == "ninja_install" ]; then
  ninja install $dry_run -v
fi

Tested with…

  • OSX 10.11.6
  • Compiled with LLVM version 3.9.0 compiled from source code.
  • cmake 3.6.2
  • ninja 1.7.1

2016

Building OpenMP Support for Clang

In this post we’ll build the OpenMP support for clang/llvm (3.9.0) from the release source code. I assume that clang/llvm is already installed as we covered in building llvm with cmake.

How to…

The following sections use bash scripts to install the OpenMP support for clang. It should work with linux and osx.

Download the source code distribution.

We cover this in building llvm with cmake.

Compile OpenMP support for CLANG/LLVM with CMake and ninja

We use a bash script to compile the OpenMP source code using cmake and ninja. Ninja is similar to gnu make but it claims that it is faster on large code base projects.

#!/bin/bash

if [ -z "$1" ]; then
    echo "USAGE: compile.sh <phase> <dry_run>. It can be cmake, ninja_build, ninja_install"
    exit -1
fi

phase=$1
dry_run=""

TARGET_SRC=${HOME}/local/llvm-out-of-tree/openmp
TARGET_BUILD=$TARGET_SRC/runtime/build
TARGET_INSTALL=${HOME}/local/llvm-oot-install

if [ "$#" -ge "2" ]; then
    dry_run="-n"
fi

mkdir $TARGET_BUILD
echo Changing folders to $TARGET_BUILD...
cd $TARGET_BUILD
echo We\'re at $PWD

if [ "$phase" == "cmake" ]; then
  cmake $TARGET_SRC -G "Eclipse CDT4 - Ninja" -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE="Release" -DCMAKE_INSTALL_PREFIX=$TARGET_INSTALL -DCMAKE_OSX_ARCHITECTURES='i386;x86_64'
fi

if [ "$phase" == "ninja_build" ]; then
  ninja $dry_run
fi

if [ "$phase" == "ninja_install" ]; then
  ninja install $dry_run -v
fi

Setting up the bash environment variables to work with this installation.

To use correctly clang/llvm with the openmp libraries you’ll need to export the following variables. I’m using a simple tool to source user-defined environments sourcing_tool. The following example should work on osx.

VERSION=3.9.0
export LLVM_HOME=$HOME/opt/llvm
export LLVM_BUILD=$HOME/local/llvm-build
export LLVM_OOT=$HOME/opt/llvm-oot
export PATH=$LLVM_HOME/bin:$PATH
#export LLVM_LIB_SEARCH_PATH=$LLVM_HOME/lib
export C_INCLUDE_PATH=$LLVM_HOME/include:$LLVM_HOME/lib/clang/$VERSION/include:$LLVM_OOT/include:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=$LLVM_HOME/include:$LLVM_HOME/lib/clang/$VERSION/include:$LLVM_OOT/include:$LLVM_HOME/include/c++/v1:$CPLUS_INCLUDE_PATH
export LIBRARY_PATH=$LLVM_HOME/lib:$LLVM_OOT/lib:$LIBRARY_PATH
export DYLD_LIBRARY_PATH=$LLVM_HOME/lib:$LLVM_OOT/lib:$DYLD_LIBRARY_PATH

Scripts

I’m hosting the scripts in github at https://github.com/spino327/llvm_clang_install.

Tested with…

  • OSX 10.11.6
  • Compiled with Apple LLVM version 7.3.0
  • cmake 3.6.2
  • ninja 1.7.1

View more posts...