AWS/CLI

From r00tedvw.com wiki
(Difference between revisions)
Jump to: navigation, search
(→‎Installation (MacOSX))
 
(18 intermediate revisions by one user not shown)
Line 1: Line 1:
−
[[AWS/Quick_Reference|Quick Reference]] | [[AWS/CLI|AWS CLI]]
+
[[AWS/Quick_Reference|Quick Reference]] | [[AWS/CLI|AWS CLI]] | [[AWS/CloudFormation|CloudFormation]]
−
=Installation (CentOS7)=
+
=Installation (CentOS7) with AWS zip=
 +
[https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html]
 +
<nowiki>~$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
 +
~$ unzip awscliv2.zip
 +
~$ sudo ./aws/install
 +
~$ aws --version
 +
aws-cli/2.2.4 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.centos.7 prompt/off</nowiki>
 +
=Installation (CentOS7) with pip=
 
Install EPEL
 
Install EPEL
 
  <nowiki>~$ sudo yum install -y epel-release
 
  <nowiki>~$ sudo yum install -y epel-release
 
sudo yum makecache</nowiki>
 
sudo yum makecache</nowiki>
 
Find and install PIP
 
Find and install PIP
−
  <nowiki>~$ sudo yum search pip | grep python3
+
  <nowiki>~$ sudo yum install -y python3 python3-libs python3-setuptools python3-pip
 +
~$ sudo yum search python3 | grep "pip\|setuptools\|libs"
 +
python3-pip.noarch : A tool for installing and managing Python3 packages
 
python34-pip.noarch : A tool for installing and managing Python3 packages
 
python34-pip.noarch : A tool for installing and managing Python3 packages
−
python36-pip.noarch : A tool for installing and managing Python3 packages
+
libselinux-python3.x86_64 : SELinux python 3 bindings for libselinux
−
~$ sudo yum install -y python36-pip</nowiki>
+
python3-apipkg.noarch : A Python namespace control and lazy-import mechanism
 +
python3-libs.i686 : Python runtime libraries
 +
python3-libs.x86_64 : Python runtime libraries
 +
python3-setuptools.noarch : Easily build and distribute Python 3 packages
 +
python34-libs.x86_64 : Python 3 runtime libraries
 +
python34-setuptools.noarch : Easily build and distribute Python 3.4 packages
 +
~$ sudo yum install -y python3 python3-libs python3-setuptools python3-pip
 +
</nowiki>
 
Verify PIP
 
Verify PIP
−
  <nowiki>~$ pip3 -V
+
  <nowiki>~$ python3 -m pip -V
 
pip 8.1.2 from/usr/lib/python3.6/site-packages (python 3.6)</nowiki>
 
pip 8.1.2 from/usr/lib/python3.6/site-packages (python 3.6)</nowiki>
 
Upgrade PIP
 
Upgrade PIP
−
  <nowiki>~$ sudo pip3 install --upgrade pip
+
  <nowiki>~$ python3 -m pip install --upgrade pip --user
−
~$ pip -V
+
Collecting pip
−
pip 19.1 from /usr/local/lib/python3.6/site-packages/pip (python 3.6)</nowiki>
+
  Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB)
−
Install AWS CLI
+
    100% |████████████████████████████████| 1.7MB 404kB/s
−
  <nowiki>~$ $ pip search awscli | grep "awscli ("
+
Installing collected packages: pip
 +
Successfully installed pip-21.3.1
 +
~$ python3 -m pip -V
 +
pip 21.3.1 from /home/fgiuliani/.local/lib/python3.6/site-packages/pip (python 3.6)</nowiki>
 +
Grep AWS CLI
 +
  <nowiki>~$ pip search awscli | grep "awscli ("
 
okta-awscli (0.4.0)                          - Provides a wrapper for Okta authentication to awscli
 
okta-awscli (0.4.0)                          - Provides a wrapper for Okta authentication to awscli
−
awscli (1.16.145)                            - Universal Command Line Environment for AWS.
+
awscli (1.16.145)                            - Universal Command Line Environment for AWS.</nowiki>
−
~$ pip install awscli --upgrade --user</nowiki>
+
UPDATE: pip search no longer works and there is no replacement because the python devs are trash.
 +
<nowiki>~$ python3 -m pip search awscli
 +
ERROR: XMLRPC request failed [code: -32500]
 +
RuntimeError: PyPI's XMLRPC API is currently disabled due to unmanageable load and will be deprecated in the near future. See https://status.python.org/ for more information.
 +
</nowiki>
 +
Install aws cli
 +
<nowiki>~$ python3 -m pip install awscli --upgrade --user</nowiki>
 
Verify AWS CLI
 
Verify AWS CLI
 
  <nowiki>~$ aws --version
 
  <nowiki>~$ aws --version
−
aws-cli/1.16.145 Python/3.6.6 Linux/3.10.0-957.10.1.el7.x86_64 botocore/1.12.135</nowiki>
+
aws-cli/2.2.4 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.centos.7 prompt/off</nowiki>
 +
 
 +
=Installation (MacOSX)=
 +
Make sure you have python 2.65+
 +
<nowiki>~$ python --version                                                                                                   
 +
Python 2.7.10</nowiki>
 +
If you don't, install it with brew and then make the proper alias.
 +
<nowiki>~$ brew install python
 +
~$ echo "alias python=/usr/local/bin/python3" >> ~/.zshrc
 +
~$ source ~/.zshrc
 +
~$ python --version
 +
Python 3.12.2</nowiki>
 +
Download, unzip, and install via script.
 +
<nowiki>~$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
 +
~$ unzip awscli-bundle.zip
 +
~$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws</nowiki>
 +
==Updated method with brew==
 +
<nowiki>~$ brew install awscli
 +
~$ aws --version
 +
aws-cli/2.15.25 Python/3.11.8 Darwin/23.1.0 source/x86_64 prompt/off</nowiki>
 +
 
 +
=Credentials=
 +
Get your access and secret keys from:
 +
*AWS Management Console > [click] username > [click] My Security Credentials
 +
*[click] Create access key > [copy] Access Key ID > [click] Show secret access key > [copy] secret access key
 +
**'''NOTE:''' you will never be able to access the secret access key again. Make sure you copy it, otherwise you'll have to create a new key.
 +
==Credentials for role user==
 +
Say you want to get an access and secret key for a role user that you can switch to:
 +
*AWS Management Console > [click] services > [click] IAM under Security, Identity, & Compliance > [click] Users
 +
*Add User > give them a user name > select "Programmatic access" for the Access type > for permissions I assigned "AdministratorAccess".
 +
*[click] on the new user under Users > [click] Security Credentials tab > create access key > copy access and secret keys.
 +
**'''NOTE:''' you will never be able to access the secret access key again. Make sure you copy it, otherwise you'll have to create a new key.
 +
==Setup Credentials AWS CLI==
 +
reference: [https://blog.gruntwork.io/authenticating-to-aws-with-the-credentials-file-d16c0fbcbf9e https://blog.gruntwork.io/authenticating-to-aws-with-the-credentials-file-d16c0fbcbf9e]<br>
 +
reference: [https://blog.gruntwork.io/authenticating-to-aws-with-environment-variables-e793d6f6d02e https://blog.gruntwork.io/authenticating-to-aws-with-environment-variables-e793d6f6d02e]
 +
<nowiki>~$ aws configure
 +
AWS Access Key ID [none]:  {access key}
 +
AWS Secret Access Key [none]: {secret key}
 +
Default region name [none]: {preferred region}
 +
Default output format [none]: {preferred output}
 +
 
 +
ie.
 +
~$ aws configure
 +
AWS Access Key ID [none]:  ODH1748RKHR48892743
 +
AWS Secret Access Key [none]: 48RJHJHD39739DHJHDUOHOI3719457DVMBNH
 +
Default region name [none]: us-east-1
 +
Default output format [none]: json</nowiki>
 +
 
 +
This will save the info in:
 +
* Credentials - <code> ~/.aws/credentials </code>
 +
* Configuration - <code> ~/.aws/config </code>
 +
<br>
 +
Test your credentials by using a simple query to look at users in IAM.
 +
<nowiki>~$ aws iam list-users</nowiki>
 +
 
 +
=Assume Role=
 +
You will probably need to assume a role through the AWS CLI.  Below is a script I wrote to help ease this process.
 +
==MAC OSX==
 +
<nowiki>
 +
#!/bin/bash
 +
#NOTE: on mac-osx use "source" to execute script instead of directly
 +
 
 +
#make sure you configure $HOME/.aws/credentials with all of the aws_roles you list in the array below
 +
 
 +
#ref: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-mfa
 +
 
 +
#EXAMPLE:
 +
#[profile role-with-mfa]
 +
#region = us-west-2
 +
#role_arn= arn:aws:iam::128716708097:role/cli-role
 +
#source_profile = cli-user
 +
#mfa_serial = arn:aws:iam::128716708097:mfa/cli-user
 +
 
 +
tempjson="$HOME/assume-role-output.json"
 +
 
 +
#aws_roles could be parsed from $HOME/.aws/credentials rather than hardcoded like is below.
 +
aws_roles=(
 +
roledisplayname\|arn:aws:iam::############:role/role-permission \
 +
sandbox\|arn:aws:iam::123456789123:role/power-user
 +
)
 +
 
 +
function assume_role {
 +
echo ""
 +
echo "Select the role to impersonate, followed by [ENTER]:"
 +
for ((i=0;i<=(${#aws_roles[@]});i++))
 +
do
 +
if (($i >= 1)); then
 +
echo $(echo -n "$i) "${aws_roles[i]} | awk -F '|' '{ print$1 }')
 +
fi
 +
done
 +
echo -n "Role #: "
 +
read role
 +
description=$(printf ${aws_roles[$role]} | awk -F '|' '{ print$1 }')
 +
arn=$(printf ${aws_roles[$role]} | awk -F '|' '{ print$2 }')
 +
 
 +
echo "Backing up aws config and credential files"
 +
cp -f $HOME/.aws/config $HOME/.aws/config_backup && cp -f $HOME/.aws/credentials $HOME/.aws/credentials_backup
 +
 
 +
#a temp file is used because each time you query aws it resets the keys.  So when we tried to query aws for each variable, authentication failed because by the last variable, the first variable value had changed.
 +
echo "Assuming role: $description and getting the session"
 +
touch $tempjson
 +
aws sts assume-role --role-arn "$arn" --role-session-name "$description-session" --profile $description > $tempjson
 +
 
 +
echo "Setting the AWS_ACCESS_KEY_ID to env var"
 +
export AWS_ACCESS_KEY_ID=$(jq -r '.Credentials.AccessKeyId' $tempjson)
 +
 
 +
echo "Setting the AWS_SECRET_ACCESS_KEY to env var"
 +
export AWS_SECRET_ACCESS_KEY=$(jq -r '.Credentials.SecretAccessKey' $tempjson)
 +
 
 +
echo "Setting the AWS_SESSION_TOKEN to env var"
 +
export AWS_SESSION_TOKEN=$(jq -r '.Credentials.SessionToken' $tempjson)
 +
 
 +
export AWS_PROFILE="$description"
 +
}
 +
 
 +
function remove_role {
 +
echo ""
 +
printf "Currently to remove the assumed role, you must exit the shell session.\nConfirm if you want to close your shell session [y/n]: "
 +
read close
 +
if [ $close = "y" ]; then
 +
echo "Unsetting environment variables to remove assumed role"
 +
unset AWS_ACCESS_KEY_ID && unset AWS_SECRET_ACCESS_KEY && unset AWS_SESSION_TOKEN
 +
echo "Assumed role removed"
 +
echo "Exitting shell"
 +
exit
 +
elif [ $close = "n" ]; then
 +
echo "Cancelling, current role will be maintained."
 +
else
 +
echo "Invalid selection.  Try again."
 +
remove_role
 +
fi
 +
}
 +
 
 +
function start {
 +
echo ""
 +
printf "1) Assume Role\n2) Remove Assumed Role\nSelect your choice, followed by [ENTER]: "
 +
read choice
 +
if (($choice == 1)); then
 +
assume_role
 +
elif (($choice == 2)); then
 +
remove_role
 +
else
 +
echo "Not a valid selection, try again"
 +
start
 +
fi
 +
}
 +
 
 +
start</nowiki>

Latest revision as of 00:14, 6 March 2024

Quick Reference | AWS CLI | CloudFormation

Contents

[edit] Installation (CentOS7) with AWS zip

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html 

~$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
~$ unzip awscliv2.zip
~$ sudo ./aws/install
~$ aws --version
aws-cli/2.2.4 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.centos.7 prompt/off

[edit] Installation (CentOS7) with pip

Install EPEL 

~$ sudo yum install -y epel-release
sudo yum makecache

Find and install PIP 

~$ sudo yum install -y python3 python3-libs python3-setuptools python3-pip
~$ sudo yum search python3 | grep "pip\|setuptools\|libs"
python3-pip.noarch : A tool for installing and managing Python3 packages
python34-pip.noarch : A tool for installing and managing Python3 packages
libselinux-python3.x86_64 : SELinux python 3 bindings for libselinux
python3-apipkg.noarch : A Python namespace control and lazy-import mechanism
python3-libs.i686 : Python runtime libraries
python3-libs.x86_64 : Python runtime libraries
python3-setuptools.noarch : Easily build and distribute Python 3 packages
python34-libs.x86_64 : Python 3 runtime libraries
python34-setuptools.noarch : Easily build and distribute Python 3.4 packages
~$ sudo yum install -y python3 python3-libs python3-setuptools python3-pip

Verify PIP 

~$ python3 -m pip -V
pip 8.1.2 from/usr/lib/python3.6/site-packages (python 3.6)

Upgrade PIP 

~$ python3 -m pip install --upgrade pip --user
Collecting pip
  Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB)
    100% |████████████████████████████████| 1.7MB 404kB/s 
Installing collected packages: pip
Successfully installed pip-21.3.1
~$ python3 -m pip -V
pip 21.3.1 from /home/fgiuliani/.local/lib/python3.6/site-packages/pip (python 3.6)

Grep AWS CLI 

~$ pip search awscli | grep "awscli ("
okta-awscli (0.4.0)                          - Provides a wrapper for Okta authentication to awscli
awscli (1.16.145)                            - Universal Command Line Environment for AWS.

UPDATE: pip search no longer works and there is no replacement because the python devs are trash. 

~$ python3 -m pip search awscli
ERROR: XMLRPC request failed [code: -32500]
RuntimeError: PyPI's XMLRPC API is currently disabled due to unmanageable load and will be deprecated in the near future. See https://status.python.org/ for more information.

Install aws cli 

~$ python3 -m pip install awscli --upgrade --user

Verify AWS CLI 

~$ aws --version
aws-cli/2.2.4 Python/3.8.8 Linux/3.10.0-957.el7.x86_64 exe/x86_64.centos.7 prompt/off

[edit] Installation (MacOSX)

Make sure you have python 2.65+ 

~$ python --version                                                                                                     
Python 2.7.10

If you don't, install it with brew and then make the proper alias. 

~$ brew install python
~$ echo "alias python=/usr/local/bin/python3" >> ~/.zshrc
~$ source ~/.zshrc
~$ python --version
Python 3.12.2

Download, unzip, and install via script. 

~$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
~$ unzip awscli-bundle.zip
~$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws

[edit] Updated method with brew

~$ brew install awscli
~$ aws --version
aws-cli/2.15.25 Python/3.11.8 Darwin/23.1.0 source/x86_64 prompt/off

[edit] Credentials

Get your access and secret keys from:

  • AWS Management Console > [click] username > [click] My Security Credentials
  • [click] Create access key > [copy] Access Key ID > [click] Show secret access key > [copy] secret access key
    • NOTE: you will never be able to access the secret access key again. Make sure you copy it, otherwise you'll have to create a new key.

[edit] Credentials for role user

Say you want to get an access and secret key for a role user that you can switch to:

  • AWS Management Console > [click] services > [click] IAM under Security, Identity, & Compliance > [click] Users
  • Add User > give them a user name > select "Programmatic access" for the Access type > for permissions I assigned "AdministratorAccess".
  • [click] on the new user under Users > [click] Security Credentials tab > create access key > copy access and secret keys.
    • NOTE: you will never be able to access the secret access key again. Make sure you copy it, otherwise you'll have to create a new key.

[edit] Setup Credentials AWS CLI

reference: https://blog.gruntwork.io/authenticating-to-aws-with-the-credentials-file-d16c0fbcbf9e
reference: https://blog.gruntwork.io/authenticating-to-aws-with-environment-variables-e793d6f6d02e 

~$ aws configure
AWS Access Key ID [none]:  {access key}
AWS Secret Access Key [none]: {secret key}
Default region name [none]: {preferred region}
Default output format [none]: {preferred output}

ie.
~$ aws configure
AWS Access Key ID [none]:  ODH1748RKHR48892743
AWS Secret Access Key [none]: 48RJHJHD39739DHJHDUOHOI3719457DVMBNH
Default region name [none]: us-east-1
Default output format [none]: json

This will save the info in:

  • Credentials - ~/.aws/credentials
  • Configuration - ~/.aws/config


Test your credentials by using a simple query to look at users in IAM. 

~$ aws iam list-users

[edit] Assume Role

You will probably need to assume a role through the AWS CLI. Below is a script I wrote to help ease this process.

[edit] MAC OSX

#!/bin/bash
#NOTE: on mac-osx use "source" to execute script instead of directly

#make sure you configure $HOME/.aws/credentials with all of the aws_roles you list in the array below

#ref: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-mfa

#EXAMPLE:
#[profile role-with-mfa]
#region = us-west-2
#role_arn= arn:aws:iam::128716708097:role/cli-role
#source_profile = cli-user
#mfa_serial = arn:aws:iam::128716708097:mfa/cli-user

tempjson="$HOME/assume-role-output.json"

#aws_roles could be parsed from $HOME/.aws/credentials rather than hardcoded like is below.
aws_roles=(
	roledisplayname\|arn:aws:iam::############:role/role-permission \
	sandbox\|arn:aws:iam::123456789123:role/power-user
	)

function assume_role {
	echo ""
	echo "Select the role to impersonate, followed by [ENTER]:"
	for ((i=0;i<=(${#aws_roles[@]});i++))
	do
		if (($i >= 1)); then
			echo $(echo -n "$i) "${aws_roles[i]} | awk -F '|' '{ print$1 }')
		fi
	done
	echo -n "Role #: "
	read role
	description=$(printf ${aws_roles[$role]} | awk -F '|' '{ print$1 }')
	arn=$(printf ${aws_roles[$role]} | awk -F '|' '{ print$2 }')

	echo "Backing up aws config and credential files"
	cp -f $HOME/.aws/config $HOME/.aws/config_backup && cp -f $HOME/.aws/credentials $HOME/.aws/credentials_backup

	#a temp file is used because each time you query aws it resets the keys.  So when we tried to query aws for each variable, authentication failed because by the last variable, the first variable value had changed.
	echo "Assuming role: $description and getting the session"
	touch $tempjson
	aws sts assume-role --role-arn "$arn" --role-session-name "$description-session" --profile $description > $tempjson

	echo "Setting the AWS_ACCESS_KEY_ID to env var"
	export AWS_ACCESS_KEY_ID=$(jq -r '.Credentials.AccessKeyId' $tempjson)

	echo "Setting the AWS_SECRET_ACCESS_KEY to env var"
	export AWS_SECRET_ACCESS_KEY=$(jq -r '.Credentials.SecretAccessKey' $tempjson)

	echo "Setting the AWS_SESSION_TOKEN to env var"
	export AWS_SESSION_TOKEN=$(jq -r '.Credentials.SessionToken' $tempjson)

	export AWS_PROFILE="$description"
}

function remove_role {
	echo ""
	printf "Currently to remove the assumed role, you must exit the shell session.\nConfirm if you want to close your shell session [y/n]: "
	read close
	if [ $close = "y" ]; then
		echo "Unsetting environment variables to remove assumed role"
		unset AWS_ACCESS_KEY_ID && unset AWS_SECRET_ACCESS_KEY && unset AWS_SESSION_TOKEN
		echo "Assumed role removed"
		echo "Exitting shell"
		exit
	elif [ $close = "n" ]; then
		echo "Cancelling, current role will be maintained."
	else
		echo "Invalid selection.  Try again."
		remove_role
	fi
}

function start {
	echo ""
	printf "1) Assume Role\n2) Remove Assumed Role\nSelect your choice, followed by [ENTER]: "
	read choice
	if (($choice == 1)); then
		assume_role
	elif (($choice == 2)); then
		remove_role
	else
		echo "Not a valid selection, try again"
		start
	fi
}

start
Retrieved from "https://wiki.r00tedvw.com/index.php?title=AWS/CLI&oldid=3446"
Personal tools
Namespaces

Variants
Actions
Navigation
Mediawiki
Confluence
DevOps Tools
Open Source Products
Ubuntu
Ubuntu 22
Mac OSX
Oracle Linux
AWS
Windows
OpenVPN
Grafana
InfluxDB2
TrueNas
MagicMirror
OwnCloud
Pivotal
osTicket
OTRS
phpBB
WordPress
VmWare ESXI 5.1
Crypto currencies
HTML
CSS
Python
Java Script
PHP
Raspberry Pi
Canvas LMS
Kaltura Media Server
Plex Media Server
MetaSploit
Zoneminder
ShinobiCE
Photoshop CS2
Fortinet
Uploaded
Certifications
General Info
Games
Meal Plans
NC Statutes
Politics
Volkswagen
Covid
NCDMV
Toolbox