Pivotal/Cloud Foundry/OpsManager

Cloud Foundry | Cloud Foundry CLI | Apps | Tasks | Logs | OpsManager

install packages via cli with om tools

It is best to always do this from within the CloudFoundry network as some of these packages can be rather large, a SSH session in Ops Manager is a great choice. However, if you have a fast connection which can upload GBs relatively quickly, you can do it from your local machine.

install om-tools

Start by installing om-tools, a cli tool for interacting with pivnet. Download the latest release from github (https://github.com/pivotal-cf/om/)

~$ wget https://github.com/pivotal-cf/om/releases/download/0.39.0/om-linux
~$ chmod +x ~/om-linux
~$ ln -s /users/$USER/om-linux /usr/local/bin/om
~$ source ~/.bash_profile
~$ om -version

install pivnet cli

Next we need to install the pivnet cli. Download the latest release from github (https://github.com/pivotal-cf/pivnet-cli/)

~$ wget https://github.com/pivotal-cf/pivnet-cli/releases/download/v0.0.53/pivnet-darwin-amd64-0.0.53
~$ chmod +x ~/pivnet-darwin-amd64-0.0.53
~$ ln -s ~/Git/pivnet-cli/pivnet-darwin-amd64-0.0.53 /usr/local/bin/pivnet
~$ source ~/.bash_profile
~$ pivnet -version

get pivnet token

Get your pivnet legacy token from here: https://login.run.pivotal.io/login
Click Username > Edit Profile > Look for LEGACY API TOKEN [DEPRECATED] > copy token.
Apply the API token and test the pivnet CLI

~$ pivnet login --api-token=h9482hd929dh2998hg
~$ pivnet products

Download Product tile

While you can very easily do this by simply going to https://network.pivotal.io/, i'm going to show how to do it through solely CLI.
Using pivnet cli we need to discover the following items before we can download the product file:

• product-slug (generally the product name)
• release version (can include spaces)
• product file ID

Let's start by getting the product-slug for the mysqlv2 tile

[~] pivnet products | grep -E 'SLUG|mysql'
| ID  |                    SLUG                    |              NAME              |
|  41 | p-mysql                                    | MySQL for PCF v1               |
| 209 | a9s-mysql                                  | a9s MySQL for PCF              |
| 180 | pivotal-mysql                              | MySQL for PCF                  |

Now lets get the most current release vesion

[~] pivnet releases -p pivotal-mysql
+--------+---------+--------------------------------+--------------------------+
|   ID   | VERSION |          DESCRIPTION           |        UPDATED AT        |
+--------+---------+--------------------------------+--------------------------+
| 122966 | 2.3.1   | MySQL for PCF v2.3.1. It       | 2018-07-12T20:27:34.759Z |
|        |         | provides dedicated single node |                          |
|        |         | instances with configurable    |                          |
|        |         | plans, full backups, and       |                          |
|        |         | metrics.                       |                          |

Finally lets get the product file ID:

[~] pivnet product-files -p pivotal-mysql -r 2.3.1
+--------+--------------------------------+--------------+---------------------+------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
|   ID   |              NAME              | FILE VERSION |      FILE TYPE      |                              SHA256                              |                                            AWS OBJECT KEY                                            |
+--------+--------------------------------+--------------+---------------------+------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| 149560 | Open Source License Disclosure |            1 | Open Source License | 2a3c887a92d299e19db179ed12d93aa5dc715c4c0665100a493428f9da4ac228 | product-files/pivotal-mysql/open_source_license_pivotal-mysql-2.3.0-build.172-f1cb31e-1526618752.txt |
|        | for MySQL for PCF 2.3.0        |              |                     |                                                                  |                                                                                                      |
| 164030 | MySQL for PCF v2               | 2.3.1        | Software            | 370d15b112965c3fddb01ff9f921534d6b906d86e1ca3dcc22c451096292fda3 | product-files/pivotal-mysql/pivotal-mysql-2.3.1-build.11.pivotal                                     |
+--------+--------------------------------+--------------+---------------------+------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+

With all (3) we can now download the tile:

[~] pivnet download-product-files -p pivotal-mysql -r 2.3.1 -i 164030 -d ~/ --accept-eula

Setup alias or token variable

Depending on how you want to communicate through om, you can either use a username/password combo or a token from ops manager.

Username/Password

alias oml="om -k -t https://opsmgr-10.haas-59.pez.pivotal.io -u admin -p password"

If you want the om output to be verbose and list the raw http response, use the -tr flag

alias oml="om -k -t https://opsmgr-10.haas-59.pez.pivotal.io -u admin -p password -tr"

Uploading tile to ops manager

As stated initially, uploading should be done from a location that has a fast connection to Ops Manager. You can even do this from Ops Manager itself using SSH.
Setup the $FILE variable using the file that we just downloaded.

~$ export FILE=pivotal-mysql-2.3.1-build.11.pivotal

Using the Alias and FILE variables that we set, we can upload the package to Ops Manager.

~$ ubuntu@opsmgr-10-haas-59-pez-pivotal-io:~$ oml upload-product --product $FILE &
[1] 5654
ubuntu@opsmgr-10-haas-59-pez-pivotal-io:~$ processing product
beginning product upload to Ops Manager
 1.80 GiB / 1.80 GiB [===========================================] 100.00% 1m17s
54s elapsed, waiting for response from Ops Manager...
finished upload

[1]+  Done                    om -k -t https://opsmgr-10.haas-59.pez.pivotal.io -u admin -p password upload-product --product $FILE

Staging Product

Using OM tools and our oml alias we can check the available products.

~$ oml available-products
+---------------+----------------+
|     NAME      |    VERSION     |
+---------------+----------------+
| pivotal-mysql | 2.3.1-build.11 |
| cf            | 2.1.7          |
| aws-services  | 1.4.8          |
+---------------+----------------+

With that information, we can stage the tile.

~$ oml stage-product --product-name pivotal-mysql --product-version "2.3.1-build.11"
staging pivotal-mysql 2.3.1-build.11
finished staging

Download JSON config

In order to utilize om configure-product we need a JSON file to pass with the settings. Rather than building one from scratch, the easiest way is to pull it using om curl.
NOTE:The product must be staged or deployed for this to work.

Obtain GUID

First we need to obtain a product identification string that uniquely identifies the product. There are (2) ways, one using om curl and another using curl against the API endpoint.

om curl method

Using the same oml alias that we setup earlier, we can hit the API endpoint and obtain the GUID. I've also piped the output to jq just to make the JSON data pretty.

~$ oml curl --path /api/v0/staged/products | jq '.'
Status: 200 OK
Cache-Control: no-cache, no-store
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 28 Aug 2018 19:18:33 GMT
Expires: Fri, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Server: nginx/1.4.6 (Ubuntu)
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Request-Id: f49e7ef1-07c8-49f8-add8-a1352c6bafcc
X-Runtime: 3.460875
X-Xss-Protection: 1; mode=block
[
  {
    "installation_name": "p-bosh",
    "guid": "p-bosh-910e4c6a7d025dfd298c",
    "type": "p-bosh",
    "product_version": "2.1-build.326"
  },
  {
    "installation_name": "cf-582ff16096af938ffc2d",
    "guid": "cf-582ff16096af938ffc2d",
    "type": "cf",
    "product_version": "2.1.7"
  },
  {
    "installation_name": "pivotal-mysql-0f6e3fc68d8a5d2e1b93",
    "guid": "pivotal-mysql-0f6e3fc68d8a5d2e1b93",
    "type": "pivotal-mysql",
    "product_version": "2.3.1-build.11"
  }
]

curl method

Get Access Token

Let's get the access token so that we can authenticate.
If you don't have uaac CLI installed, run this from a machine with Ruby on it.

~$ gem install cf-uaac

Target your Ops Manager IP:

~$ uaac target https://YOUR_OPSMAN_IP/uaa 

Log in to your Ops Manager with the Client name “opsman” and empty Client secret:

~$ uaac token owner get

Client name: opsman
Client secret: JUST_PRESS_ENTER
User name: YOUR_USERNAME_HERE
Password: YOUR_PASSWORD_HERE

Retrieve your Ops Manager access token:

~$ uaac context 
[1]*[https://opsmgr-10.haas-59.pez.pivotal.io/uaa]
  skip_ssl_validation: true

  [0]*[admin]
      user_id: c077f79e-bce3-4fa7-b3d4-f8297ccecbe1
      client_id: opsman
      access_token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiJiYWI0ODIxOWQ0NjBjODM5YzY2YzE2MiIsInN1YiI6ImMwNzdmNzllLWJjZTMtNGZhNy1iM2Q0LWY4Mjk3Y2NlY2JlMSIsInNjb3BlIjpbIm9wc21hbi5hZG1pbiIsInNjaW0ubWUiLCJjbGllbnRzLmFkbWluIiwidWFhLmFkbWluIl0sImNsaWVudF9pZCI6Im9wc21hbiIsImNpZCI6Im9wc21hbiIsImF6cCI6Im9wc21hbiIsImdyYW50X3R5cGUiOiJwYXNzd29yZCIsInVzZXJfaWQiOiJjMDc6Ly9sb2NhbGhvc3Q6ODA4MC91YWEvb2F1dGgvdG9riZTEiLCJvcmlnaW4iOiJ1YWEiLCJ1c2VyX25hbWUiOiJhZG1pbiIsImVtYWlsIjoiYWRtaW5AdGVzdC5vcmciLCJhdXRoX3RpbWUiOjE1MzU0NjYwNjksInJldl9zaWciOiI5NjMzOWNkNyIsImlhdCI6MTUzNTQ2NjA2OSwiZXhwIjoxNTM1NTA5MjY5LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvdWFhL29hdXRoL3Rva2VuIiwiemlkIjoidWFhIiwiYXVkIjpbInNjaW0iLCJvcHNtYW4iLCJjbGllbnRzIiwidWFhIl19.icaWDw_9ngRBVUpTuBfJNw21lSWrhqG56qcEe-0aVXH8UyPzqKOaeLcJgtPekZT_bxgX-WHzLP0pQuosKFtWcnuuaFyR8i3ZeIJPrw8Y07e5Hbqv5yq5wc82rLOI9aKj2QfjsYFrRtRWtdZYXlA4lmc8O6CJjaFYb6RywnOnyo1EzMO6o-F5OCK3-XVeyyRdn6uHzyoAcouJSB8QjMfI7Zu8nI-QoI8hDT0-j-UjGec3qLCQ1iXHX0HxW6fjxcrkWVZFvG4GiVCoc-Zj_B57OSfmplzptxdJ6ISiZGHs9khuWXOCXeNvbY3PbG7WYeQBL02Lafyom2u00FxDWpIi1g
      token_type: bearer
      refresh_token: eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiJiYWI2OTdhZGY5M2I0MDkyYmM4NDUzNWY4NTk2NzFlMS1yIiwic3ViIjoiYzA3N2Y3OWUtYmNlMy00ZmE3LWIzZDQtZjgyOTdjY2VjYmUxIiwic2NvcGUiOlsib3BzbWFuLmFkbWluIiwic2NpbS5tZSIsImNsaWVudHMuYWRtaW4iLCJI6Im9wc21hbiIsImNpZCI6Im9wc21hbiIsImF6cCI6IHAiOjE1MzU2Mzg4NjksImNpZCI6Im9wc21hbiIsImNsaWVudF9pZCI6Im9wc21hbiIsImlzcyI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MC91YWEvb2F1dGgvdG9rZW4iLCJ6aWQiOiJ1YWEiLCJncmFudF90eXBlIjoicGFzc3dvcmQiLCJ1c2VyX25hbWUiOiJhZG1pbiIsIm9yaWdpbiI6InVhYSIsInVzZXJfaWQiOiJjMDc3Zjc5ZS1iY2UzLTRmYTctYjNkNC1mODI5N2NjZWNiZTEiLCJyZXZfc2lnIjoiOTYzMzljZDciLCJhdWQiOlsic2NpbSIsIm9wc21hbiIsImNsaWVudHMiLCJ1YWEiXX0.IQ8iIu3OH2GM7FhXNoS5QhuMxL2Bg4k_hZNY57B1tZPXu_LBX_MYtouHte4rIgJf6dBaImMoeu89k4Bb3feLEOvVopdT8pCBwDUAUCecc15C_m_kilP-wwXbavrJB9Lvc4MVRx6QQBo9OpPTOmBNuMlyg8FE0b4OM8HTfiyJ9EB8Q8KzCJKLCRJyn0NLqe2apebLGUt1RzqW9pNjXfWRtaW5AdGVzdC5vcmciLCJhdXRoX3RpbWUiOjE1MzU0NtT6jgK56eY85Rh2jWLM9hBBsdxRutzFEaPTlVB3P-9Leqa-gFzWfBfZQh07hNagqm0JggehiiBZoikXLgwSXUg
      expires_in: 43199
      scope: opsman.admin scim.me clients.admin uaa.admin
      jti: 5a5861d454a548219d460c839c66c162

Set Access Token and submit CURL

Now we can set the access token as a variable and submit our curl to the ops man API.

~$ access_token=eyJhbGciOiJSUzI1NiIsImtpZCI6ImtleSUzI1NiIsImtpZCI6ImtleS0xIiwidHlwIjoiSldUIn0.eyJqdGkiOiJiYWI0ODIxOWQ0NjBjODM5YzY2YzE2MiIsInN1YiI6ImMwNzdmNzllLWJjZTMtNGZhNy1iM2Q0LWY4Mjk3Y2NlY2JlMSIsInNjb3BlIjpbIm9wc21hbi5hZG1pbiIsInNjaW0ubWUiLCJjbGllbnRzLmFkbWluIiwidWFhLmFkbWluIl0sImNsaWVudF9pZCI6Im9wc21hbiIsImNpZCI6Im9wc21hbiIsImF6cCI6Im9wc21hbiIsImdyYW50X3R5cGUiOiJwYXNzd29yZCIsInVzZXJfaWQiOiJjMDc6Ly9sb2NhbGhvc3Q6ODA4MC91YWEvb2F1dGgvdG9riZTEiLCJvcmlnaW4iOiJ1YWEiLCJ1c2VyX25hbWUiOiJhZG1pbiIsImVtYWlsIjoiYWRtaW5AdGVzdC5vcmciLCJhdXRoX3RpbWUiOjE1MzU0NjYwNjksInJldl9zaWciOiI5NjMzOWNkNyIsImlhdCI6MTUzNTQ2NjA2OSwiZXhwIjoxNTM1NTA5MjY5LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0OjgwODAvdWFhL29hdXRoL3Rva2VuIiwiemlkIjoidWFhIiwiYXVkIjpbInNjaW0iLCJvcHNtYW4iLCJjbGllbnRzIiwidWFhIl19.icaWDw_9ngRBVUpTuBfJNw21lSWrhqG56qcEe-0aVXH8UyPzqKOaeLcJgtPekZT_bxgX-WHzLP0pQuosKFtWcnuuaFyR8i3ZeIJPrw8Y07e5Hbqv5yq5wc82rLOI9aKj2QfjsYFrRtRWtdZYXlA4lmc8O6CJjaFYb6RywnOnyo1EzMO6o-F5OCK3-XVeyyRdn6uHzyoAcouJSB8QjMfI7Zu8nI-QoI8hDT0-j-UjGec3qLCQ1iXHX0HxW6fjxcrkWVZFvG4GiVCoc-Zj_B57OSfmplzptxdJ6ISiZGHs9khuWXOCXeNvbY3PbG7WYeQBL02Lafyom2u00FxDWpIi1g

~$ curl "https://opsmgr-10.haas-59.pez.pivotal.io/api/v0/staged/products" --insecure\
    -X GET \
    -H "Authorization: Bearer $access_token"

[{"installation_name":"p-bosh","guid":"p-bosh-910e4c6a7d025dfd298c","type":"p-bosh","product_version":"2.1-build.326"},{"installation_name":"cf-582ff16096af938ffc2d","guid":"cf-582ff16096af938ffc2d","type":"cf","product_version":"2.1.7"},{"installation_name":"pivotal-mysql-0f6e3fc68d8a5d2e1b93","guid":"pivotal-mysql-0f6e3fc68d8a5d2e1b93","type":"pivotal-mysql","product_version":"2.3.1-build.11"}]%

And the GUID we are looking for is: pivotal-mysql-0f6e3fc68d8a5d2e1b93

Pretty JSON

If you want to print out the JSON in a parse format to make it more easily readable, you can do this with jq.
Start by installing jq, a cli tool for parsing JSON in line. Download the latest release from github (https://github.com/stedolan/jq/releases)

~$ wget https://github.com/stedolan/jq/releases/download/jq-1.5/jq-osx-amd64
~$ chmod +x ~/jq-osx-amd64
~$ ln -s /users/$USER/jq-osx-amd64 /usr/local/bin/jq
~$ source ~/.bash_profile
~$ jq --version

Now you can submit the curl request and have the output very readable.

~$ curl "https://opsmgr-10.haas-59.pez.pivotal.io/api/v0/staged/products" --insecure\
    -X GET \
    -H "Authorization: Bearer $access_token" | jq '.'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   399    0   399    0     0    176      0 --:--:--  0:00:02 --:--:--   176
[
  {
    "installation_name": "p-bosh",
    "guid": "p-bosh-910e4c6a7d025dfd298c",
    "type": "p-bosh",
    "product_version": "2.1-build.326"
  },
  {
    "installation_name": "cf-582ff16096af938ffc2d",
    "guid": "cf-582ff16096af938ffc2d",
    "type": "cf",
    "product_version": "2.1.7"
  },
  {
    "installation_name": "pivotal-mysql-0f6e3fc68d8a5d2e1b93",
    "guid": "pivotal-mysql-0f6e3fc68d8a5d2e1b93",
    "type": "pivotal-mysql",
    "product_version": "2.3.1-build.11"
  }
]

Obtain JSON

Now let's combine the GUID with the rest of the URL and curl the Ops Man API endpoint

~$ oml curl --path /api/v0/staged/products/pivotal-mysql-0f6e3fc68d8a5d2e1b93/properties > mysql.json

With the created JSON file, you can now adjust the product settings and deploy it.

quick reference

query opsmanager api and print pretty json

~$ testjson=$(curl "https://opsmgr-10.haas-59.pez.pivotal.io/api/v0/stemcell_assignments" --insecure \
    -X GET \
    -H "Authorization: Bearer $access_token") | echo $testjson | python -m json.tool | more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0{
    "products": [
        {
            "available_stemcell_versions": [
                "3468.51"
            ],
            "deployed_stemcell_version": "3468.51",
            "guid": "p-rabbitmq-49407616425a3f96dd5b",
            "identifier": "p-rabbitmq",
            "is_staged_for_deletion": false,
            "label": "RabbitMQ",
            "product_version": "1.12.7",
            "required_stemcell_os": "ubuntu-trusty",
            "required_stemcell_version": "3468.51",
            "staged_stemcell_version": "3468.51"
        },
        {
            "available_stemcell_versions": [
                "3541.37"
            ],
            "deployed_stemcell_version": "3541.37",
            "guid": "pivotal-mysql-f1cb955f294464dfaffd",
            "identifier": "pivotal-mysql",
            "is_staged_for_deletion": false,
            "label": "MySQL for Pivotal Cloud Foundry v2",
            "product_version": "2.3.1-build.11",
            "required_stemcell_os": "ubuntu-trusty",
            "required_stemcell_version": "3541.34",
            "staged_stemcell_version": "3541.37"
        },
        {
            "available_stemcell_versions": [
                "3541.37"
            ],
            "deployed_stemcell_version": "3541.37",
            "guid": "p-bosh-404ef12b578f0da0977e",
            "identifier": "p-bosh",
            "is_staged_for_deletion": false,
            "label": "BOSH Director",
            "product_version": "2.1-build.348",
            "required_stemcell_os": "ubuntu-trusty",
            "required_stemcell_version": "3541.37",
            "staged_stemcell_version": "3541.37"
        },
        {
            "available_stemcell_versions": [
                "3541.37"
            ],
            "deployed_stemcell_version": "3541.37",
            "guid": "cf-5a3f75999090a1afbfdb",
            "identifier": "cf",
            "is_staged_for_deletion": false,
            "label": "Pivotal Application Service",
            "product_version": "2.1.10",
            "required_stemcell_os": "ubuntu-trusty",
            "required_stemcell_version": "3541.36",
            "staged_stemcell_version": "3541.37"
        },
        {
            "available_stemcell_versions": [
                "3468.51"
            ],
            "deployed_stemcell_version": "3468.51",
            "guid": "p-spring-cloud-services-a033fcb589ee1a7435fa",
            "identifier": "p-spring-cloud-services",
            "is_staged_for_deletion": false,
            "label": "Spring Cloud Services",
            "product_version": "1.5.6",
            "required_stemcell_os": "ubuntu-trusty",
            "required_stemcell_version": "3468",
            "staged_stemcell_version": "3468.51"
        }
    ],
    "stemcell_library": [
        {
            "hypervisor": "esxi",
            "infrastructure": "vsphere",
            "light": false,
            "os": "ubuntu-trusty",
            "version": "3468.51"
        },
        {
            "hypervisor": "esxi",
            "infrastructure": "vsphere",
            "light": false,
            "os": "ubuntu-trusty",
            "version": "3541.37"
        }
    ]
}
100  2020    0  2020    0     0   1020      0 --:--:--  0:00:01 --:--:--  1020

query opsman api and filter value needed with jq

~$ oml curl --path /api/v0/staged/products/cf-f2bd84b70904b9457b3d/properties | jq '.properties[".tcp_router.static_ips"].value'