DevOps Tools/CI/Jenkins

Overview | Continuous Integration (CI) | Source Control Management (SCM) | Containerization | Configuration | Integration

Jenkins

Installation (Docker - OSX)

Reference
Download, Install, and Start Docker CE edition (.dmg)
Install Jenkins image from repo using blueocean image.
~$ sudo docker run \

-u root

 [Expand] 

--rm \

( Optional ) Automatically removes the Docker container (which is the instantiation of the jenkinsci/blueocean image below) when it is shut down. This keeps things tidy if you need to quit Jenkins.

 [Expand] 

-d \

( Optional ) Runs the jenkinsci/blueocean container in the background (i.e. "detached" mode) and outputs the container ID. If you do not specify this option, then the running Docker log for this container is output in the terminal window.

 [Expand] 

-p 8080:8080 \

Maps (i.e. "publishes") port 8080 of the jenkinsci/blueocean container to port 8080 on the host machine. The first number represents the port on the host while the last represents the container’s port. Therefore, if you specified -p 49000:8080 for this option, you would be accessing Jenkins on your host machine through port 49000.

 [Expand] 

-p 50000:50000 \

( Optional ) Maps port 50000 of the jenkinsci/blueocean container to port 50000 on the host machine. This is only necessary if you have set up one or more JNLP-based Jenkins agents on other machines, which in turn interact with the jenkinsci/blueocean container (acting as the "master" Jenkins server, or simply "Jenkins master"). JNLP-based Jenkins agents communicate with the Jenkins master through TCP port 50000 by default. You can change this port number on your Jenkins master through the Configure Global Security page. If you were to change your Jenkins master’s TCP port for JNLP agents value to 51000 (for example), then you would need to re-run Jenkins (via this docker run …​ command) and specify this "publish" option with something like -p 52000:51000, where the last value matches this changed value on the Jenkins master and the first value is the port number on the Jenkins master’s host machine through which the JNLP-based Jenkins agents communicate (to the Jenkins master) - i.e. 52000.

 [Expand] 

-v jenkins-data:/var/jenkins_home \

( Optional but highly recommended ) Maps the /var/jenkins_home directory in the container to the Docker volume with the name jenkins-data. If this volume does not exist, then this docker run command will automatically create the volume for you. This option is required if you want your Jenkins state to persist each time you restart Jenkins (via this docker run …​ command). If you do not specify this option, then Jenkins will effectively reset to a new instance after each restart.

Notes: The jenkins-data volume could also be created independently using the docker volume create command:

docker volume create jenkins-data

Instead of mapping the /var/jenkins_home directory to a Docker volume, you could also map this directory to one on your machine’s local file system. For example, specifying the option -v $HOME/jenkins:/var/jenkins_home would map the container’s /var/jenkins_home directory to the jenkins subdirectory within the $HOME directory on your local machine, which would typically be /Users/<your-username>/jenkins or /home/<your-username>/jenkins.

 [Expand] 

-v /var/run/docker.sock:/var/run/docker.sock \

( Optional ) /var/run/docker.sock represents the Unix-based socket through which the Docker daemon listens on. This mapping allows the jenkinsci/blueocean container to communicate with the Docker daemon, which is required if the jenkinsci/blueocean container needs to instantiate other Docker containers. This option is necessary if you run declarative Pipelines whose syntax contains the agent section with the docker parameter -

i.e.

agent { docker { …​ } }. Read more about this on the Pipeline Syntax page.

 [Expand] 

jenkinsci/blueocean

The jenkinsci/blueocean Docker image itself. If this image has not already been downloaded, then this docker run command will automtically download the image for you. Furthermore, if any updates to this image were published since you last ran this command, then running this command again will automatically download these published image updates for you.

Note: This Docker image could also be downloaded (or updated) independently using the docker pull command:

docker pull jenkinsci/blueocean

No markup
~$ sudo docker run \
  -u root \
  --rm \
  -d \
  -p 8080:8080 \
  -p 50000:50000 \
  -v jenkins-data:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  jenkinsci/blueocean

Configuration

You should now be able to access your local jenkins instance by opening a webpage from http://127.0.0.1:8080
Jenkins will initially want you to provide a secret key which you can get from the container logs.

~$ sudo docker container ls -a                                                                                                                                                                                  ~
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                              NAMES
441bad1bf979        jenkinsci/blueocean   "/sbin/tini -- /usr/…"   About an hour ago   Up About an hour    0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   thirsty_goldwasser

~$ docker logs 441bad1bf979 2>&1 | grep -B2 -i initialAdminPassword                                                                                                                                             ~
b30a84793e9b41e3b4d044b7e2584643

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

Installation (CentOS 7 - using Repo)

Download and install the latest version of CentOS 7. I went with the Minimal install and below are the initial common packages.

~$ ~$ sudo yum install telnet net-tools vim tcpdump bind-utils redhat-lsb-core wget nfs-utils -y

Install some of the dependencies and repo info

~$ sudo yum install java-1.8.0-openjdk-devel -y
~$ curl --silent --location http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo | sudo tee /etc/yum.repos.d/jenkins.repo
~$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

Install Jenkins

~$ sudo yum install jenkins

Start and enable on boot

~$ sudo systemctl start jenkins
~$ sudo systemctl enable jenkins

Adjust Firewall

~$ sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp
~$ sudo firewall-cmd --reload

Installation using Apache/Tomcat (CentOS 7)

Download and install the latest version of CentOS 7. I went with the Minimal install and below are the initial common packages.

~$ sudo yum install telnet net-tools vim tcpdump bind-utils redhat-lsb-core wget nfs-utils -y

Download and Install

Install httpd, tomcat, and some of the dependencies

~$ sudo yum install java-1.8.0-openjdk-devel httpd tomcat tomcat-webapps -y

enable startup

~$ sudo systemctl enable httpd tomcat
~$ sudo systemctl start httpd tomcat

Verify proxy modules are setup

~$ apachectl -M | grep ajp
 proxy_ajp_module (shared)

Apache configuration

Create the virtual host file. The example below is super basic and is missing a lot, but is functional.
Also, I've already added the proxy paths for Jenkins and will set it up later. This will allow for the redirect of traffic hitting port 80 to go to the proper location. Servername would be important if you wanted apache to proxy based on the hostname in the http headers.

~$ sudo vim /etc/httpd/conf.d/localhost.conf

<VirtualHost *:80>

  ProxyRequests Off
  ProxyPass /examples ajp://localhost:8009/examples
  ProxyPassReverse /examples ajp://localhost:8009/examples

  ProxyPass /jenkins ajp://localhost:8009/jenkins
  ProxyPassReverse /jenkins ajp://localhost:8009/jenkins

</VirtualHost>

Test the configuration. Ignore the FQDN issue for this example, Syntax is OK, so we are good.

~$  apachectl configtest
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
Syntax OK

Tomcat Configuration

Verify that tomcat is listening for ajp requests. By default it should be.

$ cat /usr/share/tomcat/conf/server.xml | grep -C 2 '<Connector port="8009"'

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

Setting JENKINS_HOME

It is needed to set JENKINS_HOME as the Tomcat user will need a directory to read/write/execute in. There are a few ways to do this, one other being to define the variable within context.xml in the Tomcat configuration, but I prefer to use environment variables.
Thankfully Tomcat already has a couple of built-in EnvironmentFile declarations within /usr/lib/systemd/system/tomcat.service

EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME="
EnvironmentFile=-/etc/sysconfig/tomcat 

Knowing this, we can define our JENKINS_HOME within /etc/sysconfig/tomcat as it is meant for specific service values, /etc/tomcat/tomcat.conf is meant for all services.

~$ sudo printf '\n#Define JENKINS_HOME \nJENKINS_HOME="/usr/share/tomcat/webapps/jenkins/.jenkins"\n' | sudo tee -a /etc/sysconfig/tomcat > /dev/null

Again, I already know where I want the JENKINS_HOME to be, so i've defined it above even though the directory may not exist yet.

Download and Install Jenkins

We are finally at the point where we can download and install Jenkins. Since we only need the .war, we can download via the latest stable release and put in the tomcat webapps folder.

~$ sudo wget -O /usr/share/tomcat/webapps/jenkins.war http://mirrors.jenkins.io/war-stable/latest/jenkins.war

Finally we are ready to restart Apache and Tomcat

~$ sudo systemctl restart httpd tomcat

(Optional) Firewalld

Depending on your setup, you may need to add some firewalld rules to all traffic. Really only http is needed, but I like to start from a working point with additional test endpoints and then start taking away.

~$ sudo firewall-cmd --permanent --zone=public --add-service=http
~$ sudo firewall-cmd --permanent --zone=public --add-port="8080/tcp"
~$ sudo firewall-cmd --permanent --zone=public --add-port="8009/tcp"
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: ssh dhcpv6-client http
  ports: 8080/tcp 8009/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Installation using Nginx/Tomcat (CentOS 7) Corrected Home Dir and updated

Download and install the latest version of CentOS 7. I went with the Minimal install and below are the initial common packages.

~$ ~$ sudo yum install telnet net-tools vim tcpdump bind-utils redhat-lsb-core wget nfs-utils -y

Download and Install

Install nginx, tomcat, and some of the dependencies

~$ sudo yum install epel-release -y
~$ sudo yum install java-1.8.0-openjdk-devel nginx tomcat tomcat-webapps git-y

enable startup

~$ sudo systemctl enable nginx tomcat
~$ sudo systemctl start nginx tomcat

Nginx configuration

Create the new site configuration file. The example below is super basic and is missing a lot, but is functional.
Also, I've already added the proxy paths and JENKINS_HOME for Jenkins and will set it up later. This will allow for the redirect of traffic hitting port 80 to go to the proper location.

~$ sudo vim /etc/nginx/conf.d/jenkins.conf

upstream jenkins {
  keepalive 32; # keepalive connections
  server 127.0.0.1:8080; # jenkins ip and port
}

server {
  listen          80;       # Listen on port 80 for IPv4 requests

  server_name     jenkins01.r00tedvw.com;

  #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
  root            /usr/share/tomcat/webapps/;

  access_log      /var/log/nginx/jenkins.access.log;
  error_log       /var/log/nginx/jenkins.error.log;
  ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.

  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    #rewrite all static files into requests to the root
    #E.g /static/12345678/css/something.css will become /css/something.css
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }

  location /userContent {
    #have nginx handle all the static requests to the userContent folder files
    #note : This is the $JENKINS_HOME dir
        root /opt/jenkins/;
    if (!-f $request_filename){
      #this file does not exist, might be a directory or a /**view** url
      rewrite (.*) /$1 last;
          break;
    }
        sendfile on;
  }

  location / {
      sendfile off;
      proxy_pass         http://jenkins;
      proxy_redirect     default;
      proxy_http_version 1.1;

      proxy_set_header   Host              $host;
      proxy_set_header   X-Real-IP         $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   X-Forwarded-Proto $scheme;
      proxy_max_temp_file_size 0;

      #this is the maximum upload size
      client_max_body_size       10m;
      client_body_buffer_size    128k;

      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      proxy_buffering            off;
      proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      proxy_set_header Connection ""; # Clear for keepalive
  }

}

NOTE: At this step I spent too much time trying to figure out why nginx was not passing the root I specified in the site configuration file. Turns out that the include statement within nginx.conf very specifically looks for files ending in *.conf within /etc/nginx/conf.d/.

Configure/Disable selinux

In my case, when nginx tried to connect to tomcat on 8080, selinux blocked the connection.

2018/12/20 16:04:27 [crit] 27795#0: *1 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 10.0.2.2, server: localhost, request: "GET /jenkins HTTP/1.1", upstream: "http://127.0.0.1:8080/jenkins/", host: "localhost:8890"

To check if selinux is your problem, you can temporarily disable it.

~$ sudo setenforce Permissive

Now to configure it to allow permanently:

~$ sudo setsebool -P httpd_can_network_connect 1

Or if you like to live on the wild side, disable selinux altogether.

~$ /etc/selinux/config
 change SELinux=enforcing to SELinux=disabled

Setting JENKINS_HOME

It is needed to set JENKINS_HOME as the Tomcat user will need a directory to read/write/execute in. There are a few ways to do this, one other being to define the variable within context.xml in the Tomcat configuration, but I prefer to use environment variables.
Thankfully Tomcat already has a couple of built-in EnvironmentFile declarations within /usr/lib/systemd/system/tomcat.service

EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME="
EnvironmentFile=-/etc/sysconfig/tomcat 

Knowing this, we can define our JENKINS_HOME within /etc/sysconfig/tomcat as it is meant for specific service values, /etc/tomcat/tomcat.conf is meant for all services.

~$ sudo printf '\n#Define JENKINS_HOME \nJENKINS_HOME="/opt/jenkins"\n' | sudo tee -a /etc/sysconfig/tomcat > /dev/null

Again, I already know where I want the JENKINS_HOME to be, so i've defined it above even though the directory may not exist yet.

Create JENKINS_HOME

Finally we can create the JENKINS_HOME directory. I have opted to place it within /opt/jenkins, though you can place it anywhere really, but keep in mind if you are running selinux in an enforcing mode, you may have to make some additional adjustments.

~$ sudo mkdir /opt/jenkins
~$ sudo chown tomcat:tomcat /opt/jenkins
~$ sudo chmod 755 /opt/jenkins

Now, if you've created the above directory and are using selinux in an enforcing mode, you'll run into the same problem I did where selinux blocked tomcat from creating any folders or files within /opt/jenkins. After a few days of head pounding, the easiest way I found to resolve the issue is to "clone" the security context of a working directory/file. Since /usr/share/tomcat/webapps is where Tomcat expands WAR files, this should work.

~$ sudo chcon --reference /var/lib/tomcat/webapps/ /opt/jenkins/
~$ sudo ls -laZ /opt/
drwxr-xr-x. root   root   system_u:object_r:usr_t:s0       .
dr-xr-xr-x. root   root   system_u:object_r:root_t:s0      ..
drwxr-xr-x. tomcat tomcat system_u:object_r:tomcat_var_lib_t:s0 jenkins

UPDATE: its been brought to my attention that using chcon may not be the best choice for setting a new security context as, while it should be persistent, it does not tell selinux about the change and if the filesystem is relabeled then the security context would be lost. Instead, we can tell selinux about the change:

~$ sudo semanage fcontext -a -e /usr/share/tomcat/webapps /opt/jenkins 

Now if you were to list the system labels that selinux has recorded, you would see a new entry confirming our change.

~$ sudo semanage fcontext -l | grep tomcat
...
/opt/jenkins = /usr/share/tomcat/webapps

Create Jenkins User

We will need a jenkins user for integration with Gitlab/Github and probably other things.

~$ sudo useradd jenkins -d /home/jenkins -G wheel

Let's also create the SSH Keys

~$ sudo su jenkins
~$ cd ~
~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jenkins/.ssh/id_rsa):
Created directory '/home/jenkins/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jenkins/.ssh/id_rsa.
Your public key has been saved in /home/jenkins/.ssh/id_rsa.pub.

~$ exit

Download and Install Jenkins

We are finally at the point where we can download and install Jenkins. Since we only need the .war, we can download via the latest stable release and put in the tomcat webapps folder.

~$ sudo wget -O /usr/share/tomcat/webapps/jenkins.war http://mirrors.jenkins.io/war-stable/latest/jenkins.war

Finally we are ready to restart nginx and Tomcat

~$ sudo systemctl restart nginx tomcat

(Optional) Firewalld

Depending on your setup, you may need to add some firewalld rules to all traffic. Really only http is needed, but I like to start from a working point with additional test endpoints and then start taking away.

~$ sudo firewall-cmd --permanent --zone=public --add-service=http
~$ sudo firewall-cmd --permanent --zone=public --add-port="8080/tcp"
~$ sudo firewall-cmd --permanent --zone=public --add-port="8009/tcp"
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: ssh dhcpv6-client http
  ports: 8080/tcp 8009/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

Backup Jenkins

NOTE: This does NOT retain file permissions or ownership. Git only supports retention of the execution bit, nothing else. However, in a production environment, file permissions and ownership should not be handled by Git, they should be handled by something such as Puppet or Ansible.

The easiest way I've found to backup Jenkins is to copy the entire contents of /var/lib/jenkins to a SCM, such as GitLab. This folder contains all the updates, plugins, users, and all other files.
However, this sounds easier than it is. Jenkins has a handful of files and directories which only have root permissions, so if you need to rsync them to another host (which you may use Git from, like your laptop), you can do so with something like this. Just make sure to do it from the Jenkins box as there are certain files that only root has permissions to.

~$ $ sudo rsync -r -a -vv -e ssh /var/lib/jenkins/ user@10.80.16.126:"/Users/user/Git/jenkins01/"

From there, a normal git add . should add all the hidden files and folders to the git staging area.
NOTE: There may be some hidden folders that do not get added because they are empty. This is expected behavior with Git. You would need to add a file within the directory for it to be picked up, such as this:

~$ sudo vim ./groovy/grapes/.gitignore
# Ignore everything in this directory
# *
# Except this file
!.gitignore

Force kill job

Sometimes you need to force kill a job when the abort options do not work.
Using the Script Console you can execute arbitrary jenkins commands (Manage > Script Console)

Jenkins.instance.getItemByFullName("JobName/FullProjectName")
                .getBuildByNumber(JobNumber)
                .finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting build")
                );

ie.
Jenkins.instance.getItemByFullName("IE/TestProject")
				.getBuildByNumber(45)
				.finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting build")
                );