Revision as of 04:55, 12 December 2017

Ethereum Installation

Overview

In the below article i'll be covering how to configure an Ethereum deployment on Ubuntu. Out of the available linux resources out there, Debian based are well supported by Ethereum developers, with some even setting up a repository so software can be easily obtained.
Regarding Redhat based solutions, most everything I've seen requires you to download the source, compile and make yourself, which is not ideal for any solution long term that you plan on keeping up to date.

With the above mentioned, this article will be based on the latest Ubuntu server build, which at the time of this writing is 16.04.3 LTS.
My main reference article.

Initial Hardware Setup

As a personal rule, I always start with the most basic configuration and slowly build upon it, step by step, rather than just installing everything at once. In my experience, this has helped me overcome issues as they are easier to identify this way.
With that said, I would first recommend setting up your system in the most basic functional configuration possible. My beginning config was this:
Installed:

• Power supply - EVGA 1k watt PS
• Intel Mainboard - AS Rock H81 Pro BTC (with onboard video through HDMI)
• Intel Processor - 2.9Ghz Pentium
• CPU heatsink/fan
• ThumbDrive - Toshiba 16GB USB 2.0

OS Installation

I've chosen to use a thumbdrive as my hard disk. I do not think that disk speed will affect performance, so i've gone with the cheapest solution i can find that I think will work, an 8GB Sandisk drive straight from China. Mine cost $2.84 shipped, but I see others just like it for $2.99 and some mini 8GB versions (sit almost flush with the USB port) for $1.49 shipped.

Again, I've used Ubuntu Server x64 16.04.3 for the operating system. Make sure to download the right ISO, mine was almost 900MB.

USB Installation

Assuming you are working from a Debain based machine (Ubuntu, Mint, etc), you can download unetbootin from the repos in order to install Ubuntu onto the USB drive.

~$ sudo apt update && sudo apt install unetbootin -y

If you wanted to check and make sure it was available in your repos, run a quick search

~$ sudo apt-cache search unetbootin

With unetbootin installed, you can use the GUI to write the ISO to the Flash Drive.

Alternatively, if unetbootin fails (which it did for me, the USB was not bootable), you can see if your distribution has its own USB creator. Thankfully Mint had USB Image Writer and I used that to create the bootable USB.

NOTE: you may need to use (2) thumb drives to pull this off. (1) will contain the installation media, the other will be used for the Operating system to be installed to.

BIOS Configuration suggestions

Below are some BIOS configuration suggestions. I'm not going to show you how to change them or what all of the helpful ones might be because there are simply too many different types of BIOS solutions out there, however, the list below is fairly generic.

• Set correct date/time
• Set Boot config so your USB is first.
• disable sound card
• set auto power in if AC is lost
• disable onboard RAID
• disable full screen logo
• change halt on to no errors (you want it to boot without stopping)
• quick boot enabled (no memory checking)
• disable boot up seek floppy

OS Configuration

Setup DHCP

~$ sudo vim /etc/network/interfaces
Add:
#dhcp - enp3s0=nic
auto enp3s0
iface enp3s0 inet dhcp

Update and upgrade all packages

~$ sudo apt update && sudo apt upgrade -y

Install sshd

~$ sudo apt install ssh -y

Configure sshd. This is a basic, low security, setup working over port 22 and password authentication. It is not meant for a device serving ssh connections to the internet.

~$ sudo vim /etc/ssh/ssh_config
Uncomment:
 PasswordAuthentication yes
 Port 22

Video Card Installation

With a base OS configured, I decided to go ahead and install the first video card (test card).
Hardware installed:

• AMD Radeon RX580 4GB
• PCI-e Riser (USB)

NOTE: Depending on your motherboard, you may have to configure the BIOS to use the onboard video instead of the PCIe Graphics card in order to run headless (ie. no monitor plugged it). If you do not, you mainboard may look for a monitor to be plugged in before it will allow the system to boot.

With the video card installed and the system powered on, let's make sure the system sees the video card.

~$ lspci -v | grep -A 10 AMD | grep -A 10 VGA
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 67df (rev e7) (prog-if 00 [VGA controller])
	Subsystem: Tul Corporation / PowerColor Device 2378
	Flags: fast devsel, IRQ 11
	Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
	Memory at f0000000 (64-bit, prefetchable) [disabled] [size=2M]
	I/O ports at e000 [disabled] [size=256]
	Memory at f0200000 (32-bit, non-prefetchable) [disabled] [size=256K]
	Expansion ROM at f0240000 [disabled] [size=128K]
	Capabilities: <access denied>

We can see that the video card is seen and has a location of 02:00.0.

VGA Configuration

At the time of this writing, AMD has released beta drivers specifically made for mining. You can view them here: http://support.amd.com/en-us/kb-articles/Pages/AMDGPU-Pro-Beta-Mining-Driver-for-Linux-Release-Notes.aspx.

I'll be using the latest drivers as of 12/11/2017, which are 17.40-483984.

Prior Versions: 16.40-348864 16.6-37918417.10-401251 17.30-465504

New Versions: 17.40-492261

Download and extract the drivers to /opt/amdgpu-pro-17.40-483984. referer reference

~$ wget -O /tmp/amdgpu-pro-17.40-483984.tar.xz --referer=http://support.amd.com https://www2.ati.com/drivers/linux/beta/ubuntu/amdgpu-pro-17.40-483984.tar.xz
~$ sudo tar -Jxvf /tmp/amdgpu-pro-17.40-483984.tar.xz -C /opt/

Install the drivers. This will take a few minutes, mine took almost 30.

~$ /opt/amdgpu-pro-17.40-483984/amdgpu-pro-install -y 

Set some needed group membersip

~$ sudo usermod -a -G video $LOGNAME

Download and configure ROCm

~$ sudo apt install -y rocm-amdgpu-pro
~$ echo 'export LLVM_BIN=/opt/amdgpu-pro/bin' | sudo tee /etc/profile.d/amdgpu-pro.sh

Enable large page support NOTE: needed for improved mining performance

~$ sudo vim /etc/default/grub
Change:
GRUB_CMDLINE_LINUX_DEFAULT=""
To:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdgpu.vm_fragment_size=9"

Update grub and reboot

~$ sudo update-grub && sudo reboot

Verify the version you have installed

~$ dpkg -l amdgpu-pro
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                              Version               Architecture          Description
+++-=================================-=====================-=====================-========================================================================
ii  amdgpu-pro                        17.40-492261          amd64                 Meta package to install amdgpu Pro components.

Uninstalling AMD Drivers

In case you need to uninstall the AMD drivers to upgrade to a newer version, follow the steps below (reference:

~$ sudo amdgpu-pro-uninstall   

This will take a few minutes.

Ethereum Wallet Creation

If you already have an Ethereum wallet, you can skip this step. Otherwise, if you want to create a wallet (or one specific for mining, follow these instructions.
Add repo, update repo cache, download and install packages.

~$ sudo add-apt-repository -y ppa:ethereum/ethereum
~$ sudo apt update
~$ sudo apt install software-properties-common ethereum geth -y

Create the new wallet

~$ geth account new 

You'll be prompted to specify a password. Remember this, you will need it to access your wallet.
Also, when the wallet is created, the keystore will be deposited here: ~/.ethereum/keystore
NOTE: It is very important that you remember your password and keep a copy of the keystore in a secure location. (but maybe not together). BOTH of these are needed for you to access your wallet. Without either of them, you cannot access your wallet and all the associated coins are lost.

Install and Configure Ethminer

Now that we have everything needed to mine, lets install and configure the miner.
Ethminer available from the repository (shown below), is NOT the latest version and GPU mining is discontinued from it.

~$ ethminer -V
ethminer version 1.2.9
Build: Linux/g++/Interpreter/RelWithDebInfo

Instead, you will want to download the latest version from this github project and then unpack it.

~$ wget -O /tmp/ethminer-0.12.0-Linux.tar.gz https://github.com/ethereum-mining/ethminer/releases/download/v0.12.0/ethminer-0.12.0-Linux.tar.gz
~$ sudo tar -xvf /tmp/ethminer-0.12.0-Linux.tar.gz -C /opt/

Now let's create a symbolic link for ease of access to the application.

~$ sudo ln -s /opt/bin/ethminer /usr/bin/ethminer

List all opencl detected devices. you may need to know the deviceID.

~$ ethminer --list-devices

Listing OpenCL devices.
FORMAT: [platformID] [deviceID] deviceName
[0] [0] Ellesmere
	CL_DEVICE_TYPE: GPU
	CL_DEVICE_GLOBAL_MEM_SIZE: 1751240704
	CL_DEVICE_MAX_MEM_ALLOC_SIZE: 1490870272
	CL_DEVICE_MAX_WORK_GROUP_SIZE: 256
  ✘  23:47:52|ethminer  CUDA error: No CUDA driver found

Run a benchmark to test your card. This will take a little while.

~$ ethminer -M -G

Looking at the results, they should look similar to this:

ethminer -M -G
  ℹ  23:51:29|ethminer  Found suitable OpenCL device [ Ellesmere ] with 4281331712  bytes of GPU memory
Benchmarking on platform: CL
Preparing DAG for block #0
 cl  23:51:29|cl-0      No work. Pause for 3 s.
Warming up...
 cl  23:51:32|cl-0      New work: header #50c856ae… target 0000000000000002000000000000000000000000000000000000000000000000
 cl  23:51:32|cl-0      New seed #00000000…
 cl  23:51:33|cl-0      Platform: AMD Accelerated Parallel Processing
 cl  23:51:33|cl-0      Device:   Ellesmere  / OpenCL 1.2 AMD-APP (2482.3)
 cl  23:51:33|cl-0      Build info:
 cl  23:51:33|cl-0      Creating light cache buffer, size 16776896
 cl  23:51:33|cl-0      Creating DAG buffer, size 1073739904
 cl  23:51:33|cl-0      Loading kernels
 cl  23:51:33|cl-0      Writing light cache buffer
 cl  23:51:33|cl-0      Creating buffer for header.
 cl  23:51:33|cl-0      Creating mining buffer
 cl  23:51:33|cl-0      Generating DAG
 cl  23:51:34|cl-0      DAG 0 %
 cl  23:51:34|cl-0      DAG 6 %
 cl  23:51:35|cl-0      DAG 12 %
 cl  23:51:35|cl-0      DAG 18 %
 cl  23:51:36|cl-0      DAG 25 %
 cl  23:51:36|cl-0      DAG 31 %
 cl  23:51:37|cl-0      DAG 37 %
 cl  23:51:37|cl-0      DAG 43 %
 cl  23:51:38|cl-0      DAG 50 %
 cl  23:51:38|cl-0      DAG 56 %
 cl  23:51:39|cl-0      DAG 62 %
 cl  23:51:39|cl-0      DAG 68 %
 cl  23:51:39|cl-0      DAG 75 %
 cl  23:51:40|cl-0      DAG 81 %
 cl  23:51:40|cl-0      DAG 87 %
 cl  23:51:41|cl-0      DAG 93 %
 cl  23:51:41|cl-0      Switch time 11898 ms / 8898140 us
Trial 1... 18887859
Trial 2... 19624070
Trial 3... 21508713
Trial 4... 21508713
Trial 5... 21508713
  ✘  23:51:59|cl-0      OpenCL Error: clSetKernelArg -49
min/mean/max: 18887859/20607613/21508713 H/s
inner mean: 13710927 H/s

In the above example, my RX580 card is running at 21.5 MH/s, on the factory settings and factory BIOS.

Temperature Monitoring

The easiest way to do this is with lm-sensors

~$ sudo apt install -y lm-sensors 

Once installed, make sure to reboot.
After reboot, verify that sensors can see your VGA card(s).

~$ sensors
amdgpu-pci-0200
Adapter: PCI adapter
fan1:        1112 RPM
temp1:        +30.0°C  (crit =  +0.0°C, hyst =  +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +27.0°C  (high = +84.0°C, crit = +90.0°C)
Core 0:         +21.0°C  (high = +84.0°C, crit = +90.0°C)
Core 1:         +26.0°C  (high = +84.0°C, crit = +90.0°C)

In the above example I can see my (1) GPU, amdgpu-pci-0200 (notice the location reference, 0200).
I can also see my CPU, coretemp-isa-0000.

Actively Monitor GPU Temperature and Clock speeds

Here are (2) ways to actively monitor GPU temperatures and core clock speeds.

Temperatures with lm-sensors

This will query sensors" every second and show you the GPU temp & fan speed, as well as the CPU temperatures.

~$ watch -n 1 "sensors"
Every 1.0s: sensors                                                                      Mon Dec 11 12:43:55 2017

amdgpu-pci-0200
Adapter: PCI adapter
fan1:        1116 RPM
temp1:        +30.0°C  (crit =  +0.0°C, hyst =  +0.0°C)

coretemp-isa-0000
Adapter: ISA adapter
Physical id 0:  +27.0°C  (high = +84.0°C, crit = +90.0°C)
Core 0:         +20.0°C  (high = +84.0°C, crit = +90.0°C)
Core 1:         +27.0°C  (high = +84.0°C, crit = +90.0°C)

GPU Core & Memory clocks, wattage used, GPU temp, and GPU load.

The directory for you may be slightly different. Mine was /sys/kernel/debug/dri/1/ while others have reported /sys/kernel/debug/dri/0

~$ sudo watch -n 1 "cat /sys/kernel/debug/dri/1/amdgpu_pm_info | grep -A 10 'GFX Clocks and Power'"
Every 1.0s: cat /sys/kernel/debug/dri/1/amdgpu_pm_info | grep -A 10 'GFX Clocks and ...  Mon Dec 11 12:48:08 2017

GFX Clocks and Power:
        1750 MHz (MCLK)
        300 MHz (SCLK)
        8.155 W (VDDC)
        16.0 W (VDDCI)
        37.1 W (max GPU)
        37.1 W (average GPU)

GPU Temperature: 30 C
GPU Load: 0 %

Fan Control

Before we get into GPU Overclocking, lets quickly visit how to manually set the GPU fan speed. First we need to locate the directory that the fan values are located in, which we can do by looking for one of them.

~$ sudo find / -type f -name pwm1
/sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/hwmon/hwmon0/pwm1

So we know that the files we want to read/write are located within /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/hwmon/hwmon0/
For now, there are only (2) files that we are concerned with:

pwm1

shows the current value of the GPU fan speed, based on the pwm1_max value.

pwm1_max

shows the max integer value of the GPU fan speed. You'll use this to determine what value will equal your desired fan speed percentage.

So for example, lets find out what it is currently set at and the max:

~$ cat /sys/devices/pci0000\:00/0000\:00\:1c.0/0000\:02\:00.0/hwmon/hwmon0/pwm1
86
~$ cat /sys/devices/pci0000\:00/0000\:00\:1c.0/0000\:02\:00.0/hwmon/hwmon0/pwm1_max
255

So here we can see that the fan percentage is currently set at 34% (86/255). Thats probably not going to be good enough, so we can manually set it if we want to.

NOTE: By default, the AMD drivers will automatically ramp up the GPU fan speed based on load. If you proceed with manually setting the value, this feature will no longer work until you reboot.

To set it manually to 80%, we're going to set the value to 208 ((255*.8)+4). In my experiments, I always had to add (4) to the value that I wanted, otherwise the resulting value would be 4 below. This may not be the case for you, you'll have to test it.

~$ echo -n "208" > /sys/devices/pci0000\:00/0000\:00\:1c.0/0000\:02\:00.0/hwmon/hwmon0/pwm1

And you should be able to see the fans increase in speed noticeably, but if you want to check, run this:

~$ cat /sys/devices/pci0000\:00/0000\:00\:1c.0/0000\:02\:00.0/hwmon/hwmon0/pwm1
204

Now your fans are statically set to 80% until the next reboot.

GPU Performance

In this section we will cover methods for increasing the GPU performance and possibly decreasing power consumption.

Overclocking

With the installed AMD drivers and software, you can adjust the GPU core clock and GPU memory clock fairly easily.
First, determine where the appropriate files live for your card. We'll do a search for the file that controls the memory core clock.

~$ $ sudo find / -type f -name pp_mclk_od
/sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_mclk_od

Notice the hardware position referenced: .../0000:02:00.0/... This allows you to determine which card you are modifying.
With the path discovered, let's see which files are available for us to play with:

~$ sudo ls -la /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp*
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_compute_power_profile
-r--r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_cur_state
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_dpm_mclk
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_dpm_pcie
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_dpm_sclk
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_force_state
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_gfx_power_profile
-rw-r--r-- 1 root root 4096 Dec 11 18:38 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_mclk_od
-r--r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_num_states
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_sclk_od
-rw-r--r-- 1 root root 4096 Dec 11 20:14 /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_table 

Adjusting GPU Core Clock

NOTE: Before you can perform any of the changes below, you must be logged in as root. Elevation through sudo will not work.
The GPU core clock can be adjusted from 0-20% by modifying the value of a single file:

root ~$ echo "1" > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_sclk_od

In the above example you will have overclocked the GPU core clock by 1%. Now don't freak out if you don't see anything change, the core clock only ramps up when it is under load. So far example, the max GPU clock speed for my RX 580 is 1350mhz, but when I set this overclock to 1%, it increases to 1363mhz (but only shows under load, like during a benchmark).
If you wanted to check what the current "overdriven" value is, you can do this easily:

root ~$ cat /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_sclk_od
1

Recommendation: Only overclock by small percentages (1-2 at a time) and throughly test between each change, making sure to keep a close eye on the temperatures in real time.

Adjusting Memory Core Clock

NOTE: Before you can perform any of the changes below, you must be logged in as root. Elevation through sudo will not work.
The Memory core clock can be adjusted from 0-20% by modifying the value of a single file:

root ~$ echo "1" > /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_mclk_od

In the above example you will have overclocked the Memory core clock by 1%. Unlike the GPU core clock though, you should see this change immediately reflected, not just when it is under load.
If you wanted to check what the current "overdriven" value is, you can do this easily:

root ~$ cat /sys/devices/pci0000:00/0000:00:1c.0/0000:02:00.0/pp_mclk_od
1

Recommendation: Only overclock by small percentages (1-2 at a time) and throughly test between each change, making sure to keep a close eye on the temperatures in real time.

Adjustability

So i'm not sure what all of these mean, but i'll take a stab at some of them based on their values and some google searching of the file names. reference

pp_compute_power_profile

unknown

pp_cur_state (read-only)

This appears to be either a 0 or 1, indicating whether the device is powered on.

pp_dpm_mclk (read-write)

This appears to be the dpm available levels for the memory core clock. It shows which one is selected with an asterisk next to it.

pp_dpm_pcie (read-write)

This appears to show the dpm available levels for the pci-e interface and which speed is currently used; the selection indicated with an asterisk next to it.

pp_dpm_sclk (read-write)

This appears to show the dpm available levels (profiles) for the GPU core clock. it shows which one is selected with an asterisk next to it.

pp_force_state

unknown

pp_gfx_power_profile

unknown

pp_mclk_od (read-write)

This appears to show the percent the memory clock has been "overdriven", by percentage. You can adjust this from 0 (no overclock) all the way up to 20 (you've just created a paperweight).

pp_num_states

unknown

pp_sclk_od (read-write)

This appears to show the percent the gpu core clock has been "overdriven", by percentage. You can adjust this from 0 (no overclock) all the way up to 20 (holy shit its on fire).

pp_table

unknown

Ethereum Pool

You will want to use a pool to mine ethereum unless you have a datacenter sized operation, in which case you won't be using this guide, so if you're reading this, pick a pool.
Here is a quick way to see the distribution of Ethereum pools: https://etherchain.org/statistics/miners. After looking at this and reviewing the pool differences, I went with ethermine.org as they allow for anonymous mining and you can set a very low payout 0.05. Payout minimum is important if you have a low power setup (i.e anything that can't mine 1 ETH in a day) as you want your shares deposited as soon as possible into your wallet.

To configure the payout, you first need to be mining (generally for at least 10-15 minutes). Once you've started, you can input your Ethereum wallet address in the search engine and it will pull up your miner(s) statistic(s). Click on Settings and adjust your payout (I went with the current minimum, 0.05).

Mining

So with the pool picked, you are ready to start mining! You'll need to get some information from your pool, like what their servers are, which ports to use, etc. They will generally give you an example to work off of. In my instance, this is the command that I ended up with:

~$ /opt/bin/ethminer --farm-recheck 200 -RH -G -S us1.ethermine.org:4444 -FS us2.ethermine.org:4444 -O 28122ca4c6dd16c1dc113f6c4965fa36da329817.ncwp-miner02

Breaking it down:

--farm-recheck 200

this was a suggestion from my pool.

-RH

this tells ethminer to report the hashrate to the pool (not supported by all pools)

-G

very important, tells ethminer to use OpenCL when mining. Without, your hash rates will be terrible.

-S

sets the primary server for your pool

-FS

sets the secondary server for your pool (failover)

-O

sets the eth wallet address you want eth to be delivered to, along with the name of your miner (prepended by a dot (.))

Final Thoughts

Well, the guide is far from done.

• I still have to do tweaking to the GPU to extract a few more MH/s as it should be possible to reach 27MH/s or so. So far GPU/Memory tweaks have not had any positive affects on mining rates, only increased temps. maybe new drivers?
• There are still (5) more cards to install into this system. Hopefully they go pretty easily, but i'm sure there will be things to figure out, which i'll document.
• I'd like to create some scripts or find existing tools that dump temp/hashrate/wattage data into influxdb so it can be graphed with grafana or chronograf.
• it would be awesome to manage this entire thing with puppet.