Latest revision as of 13:58, 18 August 2020

Overview | Continuous Integration (CI) | Source Control Management (SCM) | Containerization | Configuration | Integration

[edit] Jenkins

[edit] Installation (Docker - OSX)

Reference
Download, Install, and Start Docker CE edition (.dmg)
Install Jenkins image from repo using blueocean image.
~$ sudo docker run \

-u root

No markup
~$ sudo docker run \
  -u root \
  --rm \
  -d \
  -p 8080:8080 \
  -p 50000:50000 \
  -v jenkins-data:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  jenkinsci/blueocean

[edit] Configuration

You should now be able to access your local jenkins instance by opening a webpage from http://127.0.0.1:8080
Jenkins will initially want you to provide a secret key which you can get from the container logs.

~$ sudo docker container ls -a                                                                                                                                                                                  ~
CONTAINER ID        IMAGE                 COMMAND                  CREATED             STATUS              PORTS                                              NAMES
441bad1bf979        jenkinsci/blueocean   "/sbin/tini -- /usr/…"   About an hour ago   Up About an hour    0.0.0.0:8080->8080/tcp, 0.0.0.0:50000->50000/tcp   thirsty_goldwasser

~$ docker logs 441bad1bf979 2>&1 | grep -B2 -i initialAdminPassword                                                                                                                                             ~
b30a84793e9b41e3b4d044b7e2584643

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

[edit] Installation (CentOS 7 - using Repo)

Download and install the latest version of CentOS 7. I went with the Minimal install and below are the initial common packages.

~$ ~$ sudo yum install telnet net-tools vim tcpdump bind-utils redhat-lsb-core wget nfs-utils -y

Install some of the dependencies and repo info

~$ sudo yum install java-1.8.0-openjdk-devel -y
~$ curl --silent --location http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo | sudo tee /etc/yum.repos.d/jenkins.repo
~$ sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key

[edit] Install Jenkins

~$ sudo yum install jenkins

Start and enable on boot

~$ sudo systemctl start jenkins
~$ sudo systemctl enable jenkins

[edit] Adjust Firewall

~$ sudo firewall-cmd --permanent --zone=public --add-port=8080/tcp
~$ sudo firewall-cmd --reload

[edit] Installation using Apache/Tomcat (CentOS 7)

Download and install the latest version of CentOS 7. I went with the Minimal install and below are the initial common packages.

~$ sudo yum install telnet net-tools vim tcpdump bind-utils redhat-lsb-core wget nfs-utils -y

[edit] Download and Install

Install httpd, tomcat, and some of the dependencies

~$ sudo yum install java-1.8.0-openjdk-devel httpd tomcat tomcat-webapps -y

[edit] enable startup

~$ sudo systemctl enable httpd tomcat
~$ sudo systemctl start httpd tomcat

Verify proxy modules are setup

~$ apachectl -M | grep ajp
 proxy_ajp_module (shared)

[edit] Apache configuration

Create the virtual host file. The example below is super basic and is missing a lot, but is functional.
Also, I've already added the proxy paths for Jenkins and will set it up later. This will allow for the redirect of traffic hitting port 80 to go to the proper location. Servername would be important if you wanted apache to proxy based on the hostname in the http headers.

~$ sudo vim /etc/httpd/conf.d/localhost.conf

<VirtualHost *:80>

  ProxyRequests Off
  ProxyPass /examples ajp://localhost:8009/examples
  ProxyPassReverse /examples ajp://localhost:8009/examples

  ProxyPass /jenkins ajp://localhost:8009/jenkins
  ProxyPassReverse /jenkins ajp://localhost:8009/jenkins

</VirtualHost>

Test the configuration. Ignore the FQDN issue for this example, Syntax is OK, so we are good.

~$  apachectl configtest
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message
Syntax OK

[edit] Tomcat Configuration

Verify that tomcat is listening for ajp requests. By default it should be.

$ cat /usr/share/tomcat/conf/server.xml | grep -C 2 '<Connector port="8009"'

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

[edit] Setting JENKINS_HOME

It is needed to set JENKINS_HOME as the Tomcat user will need a directory to read/write/execute in. There are a few ways to do this, one other being to define the variable within context.xml in the Tomcat configuration, but I prefer to use environment variables.
Thankfully Tomcat already has a couple of built-in EnvironmentFile declarations within /usr/lib/systemd/system/tomcat.service

EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME="
EnvironmentFile=-/etc/sysconfig/tomcat 

Knowing this, we can define our JENKINS_HOME within /etc/sysconfig/tomcat as it is meant for specific service values, /etc/tomcat/tomcat.conf is meant for all services.

~$ sudo printf '\n#Define JENKINS_HOME \nJENKINS_HOME="/usr/share/tomcat/webapps/jenkins/.jenkins"\n' | sudo tee -a /etc/sysconfig/tomcat > /dev/null

Again, I already know where I want the JENKINS_HOME to be, so i've defined it above even though the directory may not exist yet.

[edit] Download and Install Jenkins

We are finally at the point where we can download and install Jenkins. Since we only need the .war, we can download via the latest stable release and put in the tomcat webapps folder.

~$ sudo wget -O /usr/share/tomcat/webapps/jenkins.war http://mirrors.jenkins.io/war-stable/latest/jenkins.war

Finally we are ready to restart Apache and Tomcat

~$ sudo systemctl restart httpd tomcat

[edit] (Optional) Firewalld

Depending on your setup, you may need to add some firewalld rules to all traffic. Really only http is needed, but I like to start from a working point with additional test endpoints and then start taking away.

~$ sudo firewall-cmd --permanent --zone=public --add-service=http
~$ sudo firewall-cmd --permanent --zone=public --add-port="8080/tcp"
~$ sudo firewall-cmd --permanent --zone=public --add-port="8009/tcp"
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: ssh dhcpv6-client http
  ports: 8080/tcp 8009/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

[edit] Installation using Nginx/Tomcat (CentOS 7) Corrected Home Dir and updated

Download and install the latest version of CentOS 7. I went with the Minimal install and below are the initial common packages.

~$ ~$ sudo yum install telnet net-tools vim tcpdump bind-utils redhat-lsb-core wget nfs-utils -y

[edit] Download and Install

Install nginx, tomcat, and some of the dependencies

~$ sudo yum install epel-release -y
~$ sudo yum install java-1.8.0-openjdk-devel nginx tomcat tomcat-webapps git-y

[edit] enable startup

~$ sudo systemctl enable nginx tomcat
~$ sudo systemctl start nginx tomcat

[edit] Nginx configuration

Create the new site configuration file. The example below is super basic and is missing a lot, but is functional.
Also, I've already added the proxy paths and JENKINS_HOME for Jenkins and will set it up later. This will allow for the redirect of traffic hitting port 80 to go to the proper location.

~$ sudo vim /etc/nginx/conf.d/jenkins.conf

upstream jenkins {
  keepalive 32; # keepalive connections
  server 127.0.0.1:8080; # jenkins ip and port
}

server {
  listen          80;       # Listen on port 80 for IPv4 requests

  server_name     jenkins01.r00tedvw.com;

  #this is the jenkins web root directory (mentioned in the /etc/default/jenkins file)
  root            /usr/share/tomcat/webapps/;

  access_log      /var/log/nginx/jenkins.access.log;
  error_log       /var/log/nginx/jenkins.error.log;
  ignore_invalid_headers off; #pass through headers from Jenkins which are considered invalid by Nginx server.

  location ~ "^/static/[0-9a-fA-F]{8}\/(.*)$" {
    #rewrite all static files into requests to the root
    #E.g /static/12345678/css/something.css will become /css/something.css
    rewrite "^/static/[0-9a-fA-F]{8}\/(.*)" /$1 last;
  }

  location /userContent {
    #have nginx handle all the static requests to the userContent folder files
    #note : This is the $JENKINS_HOME dir
        root /opt/jenkins/;
    if (!-f $request_filename){
      #this file does not exist, might be a directory or a /**view** url
      rewrite (.*) /$1 last;
          break;
    }
        sendfile on;
  }

  location / {
      sendfile off;
      proxy_pass         http://jenkins;
      proxy_redirect     default;
      proxy_http_version 1.1;

      proxy_set_header   Host              $host;
      proxy_set_header   X-Real-IP         $remote_addr;
      proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
      proxy_set_header   X-Forwarded-Proto $scheme;
      proxy_max_temp_file_size 0;

      #this is the maximum upload size
      client_max_body_size       10m;
      client_body_buffer_size    128k;

      proxy_connect_timeout      90;
      proxy_send_timeout         90;
      proxy_read_timeout         90;
      proxy_buffering            off;
      proxy_request_buffering    off; # Required for HTTP CLI commands in Jenkins > 2.54
      proxy_set_header Connection ""; # Clear for keepalive
  }

}

NOTE: At this step I spent too much time trying to figure out why nginx was not passing the root I specified in the site configuration file. Turns out that the include statement within nginx.conf very specifically looks for files ending in *.conf within /etc/nginx/conf.d/.

[edit] Configure/Disable selinux

In my case, when nginx tried to connect to tomcat on 8080, selinux blocked the connection.

2018/12/20 16:04:27 [crit] 27795#0: *1 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 10.0.2.2, server: localhost, request: "GET /jenkins HTTP/1.1", upstream: "http://127.0.0.1:8080/jenkins/", host: "localhost:8890"

To check if selinux is your problem, you can temporarily disable it.

~$ sudo setenforce Permissive

Now to configure it to allow permanently:

~$ sudo setsebool -P httpd_can_network_connect 1

Or if you like to live on the wild side, disable selinux altogether.

~$ /etc/selinux/config
 change SELinux=enforcing to SELinux=disabled

[edit] Setting JENKINS_HOME

It is needed to set JENKINS_HOME as the Tomcat user will need a directory to read/write/execute in. There are a few ways to do this, one other being to define the variable within context.xml in the Tomcat configuration, but I prefer to use environment variables.
Thankfully Tomcat already has a couple of built-in EnvironmentFile declarations within /usr/lib/systemd/system/tomcat.service

EnvironmentFile=/etc/tomcat/tomcat.conf
Environment="NAME="
EnvironmentFile=-/etc/sysconfig/tomcat 

Knowing this, we can define our JENKINS_HOME within /etc/sysconfig/tomcat as it is meant for specific service values, /etc/tomcat/tomcat.conf is meant for all services.

~$ sudo printf '\n#Define JENKINS_HOME \nJENKINS_HOME="/opt/jenkins"\n' | sudo tee -a /etc/sysconfig/tomcat > /dev/null

Again, I already know where I want the JENKINS_HOME to be, so i've defined it above even though the directory may not exist yet.

[edit] Create JENKINS_HOME

Finally we can create the JENKINS_HOME directory. I have opted to place it within /opt/jenkins, though you can place it anywhere really, but keep in mind if you are running selinux in an enforcing mode, you may have to make some additional adjustments.

~$ sudo mkdir /opt/jenkins
~$ sudo chown tomcat:tomcat /opt/jenkins
~$ sudo chmod 755 /opt/jenkins

Now, if you've created the above directory and are using selinux in an enforcing mode, you'll run into the same problem I did where selinux blocked tomcat from creating any folders or files within /opt/jenkins. After a few days of head pounding, the easiest way I found to resolve the issue is to "clone" the security context of a working directory/file. Since /usr/share/tomcat/webapps is where Tomcat expands WAR files, this should work.

~$ sudo chcon --reference /var/lib/tomcat/webapps/ /opt/jenkins/
~$ sudo ls -laZ /opt/
drwxr-xr-x. root   root   system_u:object_r:usr_t:s0       .
dr-xr-xr-x. root   root   system_u:object_r:root_t:s0      ..
drwxr-xr-x. tomcat tomcat system_u:object_r:tomcat_var_lib_t:s0 jenkins

UPDATE: its been brought to my attention that using chcon may not be the best choice for setting a new security context as, while it should be persistent, it does not tell selinux about the change and if the filesystem is relabeled then the security context would be lost. Instead, we can tell selinux about the change:

~$ sudo semanage fcontext -a -e /usr/share/tomcat/webapps /opt/jenkins 

Now if you were to list the system labels that selinux has recorded, you would see a new entry confirming our change.

~$ sudo semanage fcontext -l | grep tomcat
...
/opt/jenkins = /usr/share/tomcat/webapps

[edit] Create Jenkins User

We will need a jenkins user for integration with Gitlab/Github and probably other things.

~$ sudo useradd jenkins -d /home/jenkins -G wheel

Let's also create the SSH Keys

~$ sudo su jenkins
~$ cd ~
~$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jenkins/.ssh/id_rsa):
Created directory '/home/jenkins/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/jenkins/.ssh/id_rsa.
Your public key has been saved in /home/jenkins/.ssh/id_rsa.pub.

~$ exit

[edit] Download and Install Jenkins

We are finally at the point where we can download and install Jenkins. Since we only need the .war, we can download via the latest stable release and put in the tomcat webapps folder.

~$ sudo wget -O /usr/share/tomcat/webapps/jenkins.war http://mirrors.jenkins.io/war-stable/latest/jenkins.war

Finally we are ready to restart nginx and Tomcat

~$ sudo systemctl restart nginx tomcat

[edit] (Optional) Firewalld

Depending on your setup, you may need to add some firewalld rules to all traffic. Really only http is needed, but I like to start from a working point with additional test endpoints and then start taking away.

~$ sudo firewall-cmd --permanent --zone=public --add-service=http
~$ sudo firewall-cmd --permanent --zone=public --add-port="8080/tcp"
~$ sudo firewall-cmd --permanent --zone=public --add-port="8009/tcp"
~$ sudo firewall-cmd --reload
~$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp0s3
  sources:
  services: ssh dhcpv6-client http
  ports: 8080/tcp 8009/tcp
  protocols:
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:

[edit] Backup Jenkins

NOTE: This does NOT retain file permissions or ownership. Git only supports retention of the execution bit, nothing else. However, in a production environment, file permissions and ownership should not be handled by Git, they should be handled by something such as Puppet or Ansible.

The easiest way I've found to backup Jenkins is to copy the entire contents of /var/lib/jenkins to a SCM, such as GitLab. This folder contains all the updates, plugins, users, and all other files.
However, this sounds easier than it is. Jenkins has a handful of files and directories which only have root permissions, so if you need to rsync them to another host (which you may use Git from, like your laptop), you can do so with something like this. Just make sure to do it from the Jenkins box as there are certain files that only root has permissions to.

~$ $ sudo rsync -r -a -vv -e ssh /var/lib/jenkins/ [email protected]:"/Users/user/Git/jenkins01/"

From there, a normal git add . should add all the hidden files and folders to the git staging area.
NOTE: There may be some hidden folders that do not get added because they are empty. This is expected behavior with Git. You would need to add a file within the directory for it to be picked up, such as this:

~$ sudo vim ./groovy/grapes/.gitignore
# Ignore everything in this directory
# *
# Except this file
!.gitignore

[edit] Force kill job

Sometimes you need to force kill a job when the abort options do not work.
Using the Script Console you can execute arbitrary jenkins commands (Manage > Script Console)

Jenkins.instance.getItemByFullName("JobName/FullProjectName")
                .getBuildByNumber(JobNumber)
                .finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting build")
                );

ie.
Jenkins.instance.getItemByFullName("IE/TestProject")
				.getBuildByNumber(45)
				.finish(
                        hudson.model.Result.ABORTED,
                        new java.io.IOException("Aborting build")
                );