DevOps Tools/Repository Management/Artifactory

Artifactory

Installation

Docker

Using docker you can pull the container from jfrog and then mount a local volume to house the configuration files and data.
https://www.jfrog.com/confluence/display/RTF/Installing+with+Docker

~$ sudo mkdir -p /opt/jfrog/artifactory
~$ sudo chown -R 1030:1030 /opt/jfrog/artifactory
~$ docker pull docker.bintray.io/jfrog/artifactory-oss:latest
~$ docker run --name artifactory-oss -d -v /opt/jfrog/artifactory:/var/opt/jfrog/artifactory -p 8081:8081 docker.bintray.io/jfrog/artifactory-oss:latest

Repository

Done on CentOS7

Dependencies

~$ sudo yum install -y wget java-1.8.0-openjdk  java-1.8.0-openjdk-devel

Set the Java variables.

~$ sudo vim /etc/profile.d/java.sh
#!/bin/bash
export JAVA_HOME=$(dirname $(dirname $(readlink $(readlink $(which javac)))))
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

~$ source /etc/profile.d/java.sh
~$ echo "JAVA_HOME: $JAVA_HOME"; echo "PATH: $PATH"; echo "CLASSPATH: $CLASSPATH"
JAVA_HOME: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64
PATH: /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/r00t/.local/bin:/home/r00t/bin:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/bin
CLASSPATH: .:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/lib:/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/lib/tools.jar

Add artifactory repo and install

~$ wget https://bintray.com/jfrog/artifactory-rpms/rpm -O bintray-jfrog-artifactory-oss-rpms.repo
~$ sudo mv bintray-jfrog-artifactory-oss-rpms.repo /etc/yum.repos.d/
~$ sudo yum install -y jfrog-artifactory-oss

Add the Artifactory home environment variable

~$ sudo vim /etc/profile.d/artifactory.sh
#!/bin/bash
export ARTIFACTORY_HOME=/opt/jfrog/artifactory
export JFROG_HOME=/opt/jfrog

Install Database

Artifactory requires Mariadb 10.2.9 or greater.

~$ sudo vim /etc/yum.repos.d/MariaDB.repo
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.5/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

~$ sudo yum install -y mariadb-server
~$ sudo systemctl start mariadb.service
~$ sudo systemctl enable mariadb.service

Configure the Database

~$ mysql_secure_installation
Enter current password for root (enter for none):
Set root password? [Y/n] y
New password:
Re-enter new password:
Remove anonymous users? [Y/n] y
Disallow root login remotely? [Y/n] y
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y/n] y

Configure Database

Artifactory provides a series of example sql scripts for setting up the needed resources: /opt/jfrog/artifactory/app/misc/db/.
For mariadb, we can modify the mariadb sql script appropriately.

~$ sudo vim /opt/jfrog/artifactory/app/misc/db/createdb_mariadb.sql
CREATE DATABASE artdb CHARACTER SET utf8 COLLATE utf8_bin;
CREATE USER 'artifactory'@'localhost' IDENTIFIED BY 'Password';
GRANT ALL ON artdb.* TO 'artifactory'@'localhost' IDENTIFIED BY 'Password';
FLUSH PRIVILEGES;

Now implement

~$ sudo mysql -u root -p < /opt/jfrog/artifactory/app/misc/db/createdb_mariadb.sql

Install & Setup Connector/J

Download the latest version of the mariadb-java-client Jar from here https://downloads.mariadb.org/connector-java/ or https://downloads.mariadb.com/Connectors/java/
In my case, the latest was 2.6.2

~$ wget https://downloads.mariadb.com/Connectors/java/connector-java-2.6.2/mariadb-java-client-2.6.2.jar

Move it to the right directory and make sure the permissions are correct and match $JFROG_HOME/artifactory/var

~$ sudo mv mariadb-java-client-2.6.2.jar $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib
~$ ls -la $JFROG_HOME/artifactory
lrwxrwxrwx.  1 artifactory artifactory  26 Aug 22 17:29 var -> /var/opt/jfrog/artifactory
~$ ls -la /var/opt/jfrog/
drwxr-xr-x. 6 artifactory artifactory 57 Aug 22 17:29 artifactory
~$ sudo chown artifactory:artifactory $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib/mariadb-java-client-2.6.2.jar
~$ sudo chmod 755 $JFROG_HOME/artifactory/var/bootstrap/artifactory/tomcat/lib/mariadb-java-client-2.6.2.jar

Update system.yaml with the correct changes.

~$ sudo rm $JFROG_HOME/artifactory/var/etc/system.yaml
~$ sudo touch $JFROG_HOME/artifactory/var/etc/system.yaml
~$ sudo chmod 755 $JFROG_HOME/artifactory/var/etc/system.yaml
~$ sudo vim $JFROG_HOME/artifactory/var/etc/system.yaml
configVersion: 1
shared:
  database:
    type: mariadb
    driver: org.mariadb.jdbc.Driver
    url: jdbc:mariadb://localhost:3306/artdb?characterEncoding=UTF-8&elideSetAutoCommits=true&useSSL=false&useMysqlMetadata=true
    username: artifactory
    password: password

Increase File descriptor and open file limits

The defaults that come with CentOS 7 will not be sufficient for artifactory, so we need to increase both.

~$ sudo vim /etc/sysctl.conf
...
fs.file-max = 100000
~$ sudo vim /etc/security/limits.conf
...
* soft nproc 65535
 * hard nproc 65535
 * soft nofile 65535
 * hard nofile 65535
#End

Update

~$ sudo sysctl -p

Check to make sure the artifactory user has the updated limits.

~$ sudo su artifactory --shell /bin/bash --command "ulimit -n"
65535
~$ sudo su artifactory --shell /bin/bash --command "cat /proc/sys/fs/file-max"
100000

Open Firewall

~$ sudo firewall-cmd --permanent --zone=public --add-port=8081/tcp
~$ sudo firewall-cmd --permanent --zone=public --add-port=8082/tcp
~$ sudo firewall-cmd --reload

Start and Enable

~$ sudo systemctl enable artifactory.service
~$ sudo systemctl start artifactory.service

Uploading bulk

quick script to upload in bulk and add the sha1 checksum.

#!/bin/bash
FILES=/Users/r00t/Downloads/*
username=r00t
password=FDOdohfsohf347fsHDJDokhf89
artifactoryurl=http://usa-rnc-dockerhost:8081/artifactory/generic-local/
for f in $FILES
do
	echo "Uploading $f to artifactory"
	checksum=$(shasum -a 1 $f | awk '{ print $1 }')
	file=$(printf $f | awk -F "/" '{ print $NF}')
	curl --header "X-Checksum-Sha1:${checksum}" -u $username:$password -T $f "$artifactoryurl/$file"
done

Using API token

also calculating all the hashes with openssl

#!/bin/bash

url=http://dockerhost:8081/artifactory/generic-local/SubFolder
apikey=697929fdb66e46e1db0b4ec1fd4292028a853c7879a969b3e0188b14ee24d55d98d7e8110c2579c0
localfile=/Users/r00t/test.html
FILES=/Users/r00t/Downloads/crap/artifactoryfiles/*
search='partialfilename'

for f in $FILES
do

	if [[ $f =~ "$search" ]]; then
		file=$(basename $f)
		sha1checksum=$(openssl sha1 $f | awk '{ print $2 }')
		md5checksum=$(openssl md5 $f | awk '{ print $2 }')
		sha256checksum=$(openssl sha256 $f | awk '{ print $2 }')

		echo "uploading file: $f"
		curl --header "X-JFrog-Art-Api:${apikey}" \
     	     	--header "X-Checksum-Sha1:${sha1checksum}" \
     	     	--header "X-Checksum-MD5:${md5checksum}" \
     	     	--header "X-Checksum-Sha256:${sha256checksum}" \
     	     	--upload-file $f \
     	     	-X PUT "$url/$file"
	fi
done

Downloading Bulk

Done on a mac

#!/bin/bash

url=http://dockerhost:8081
repo=generic-local
path=/SubFolder
apikey=697929fdb66e46e1db0b4ec1fd4292028a853c7879a969b3e0188b14ee24d55d98d7e8110c2579c0

for f in $(curl $url/artifactory/api/storage/$repo$path | jq '.children[].uri' | sed -E -e 's/"|"\///g')
do
	echo "downloading file: $f"
	curl --header "X-JFrog-Art-Api:${apikey}" --remote-name $url/artifactory/$repo$path/$f
done 

Deleting Bulk

If you need to delete a series of files in bulk, using a loose matching search term (regex)

#!/bin/bash

url=http://dockerhost:8081
repo=generic-local
path=/SubFolder
apikey=697929fdb66e46e1db0b4ec1fd4292028a853c7879a969b3e0188b14ee24d55d98d7e8110c2579c0
search='partialfilename'

for f in $(curl $url/artifactory/api/storage/$repo$path | jq '.children[].uri' | sed -E -e 's/"|"\///g')
do
	if [[ $f =~ "$search" ]]; then
		echo "deleting file: $f"
		curl --header "X-JFrog-Art-Api:${apikey}" -X DELETE $url/artifactory/$repo$path/$f
	fi
done