Google Cloud Platform Compute

Table of Contents

• Getting Started
  • Helpful Background Information
  • Prerequisites
  • How to Install
  • Testing
• Configuration
  • Example Properties
  • Connection Properties
  • Authentication Properties
  • Healthcheck Properties
  • Request Properties
  • SSL Properties
  • Throttle Properties
  • Proxy Properties
  • Mongo Properties
  • Device Broker Properties
• Using this Adapter
  • Generic Adapter Calls
  • Adapter Cache Calls
  • Adapter Broker Calls
  • Specific Adapter Calls
  • Authentication
• Additional Information
  • Enhancements
  • Contributing
  • Helpful Links
  • Node Scripts
• Troubleshoot
  • Connectivity Issues
  • Functional Issues

Getting Started

These instructions will help you get a copy of the project on your local machine for development and testing. Reading this section is also helpful for deployments as it provides you with pertinent information on prerequisites and properties.

Helpful Background Information

There is Adapter documentation available on the Itential Documentation Site. This documentation includes information and examples that are helpful for:

Authentication
IAP Service Instance Configuration
Code Files
Endpoint Configuration (Action & Schema)
Mock Data
Adapter Generic Methods
Headers
Security
Linting and Testing
Build an Adapter
Troubleshooting an Adapter

Others will be added over time. Want to build a new adapter? Use the Itential Adapter Builder

Prerequisites

The following is a list of required packages for installation on the system the adapter will run on:

Node.js
npm
Git

The following list of packages are required for Itential opensource adapters or custom adapters that have been built utilizing the Itential Adapter Builder. You can install these packages by running npm install inside the adapter directory.

If you are developing and testing a custom adapter, or have testing capabilities on an Itential opensource adapter, you will need to install these packages as well.

chai
eslint
eslint-config-airbnb-base
eslint-plugin-import
eslint-plugin-json
testdouble

How to Install

1. Set up the name space location in your IAP node_modules.

cd /opt/pronghorn/current/node_modules (* could be in a different place)
if the @itentialopensource directory does not exist, create it:
    mkdir @itentialopensource

2. Clone/unzip/tar the adapter into your IAP environment.

cd \@itentialopensource
git clone git@gitlab.com:\@itentialopensource/adapters/adapter-gcp_compute
or
unzip adapter-gcp_compute.zip
or
tar -xvf adapter-gcp_compute.tar

3. Run the adapter install script.

cd adapter-gcp_compute
npm install
npm run lint:errors
npm run test

4. Restart IAP

systemctl restart pronghorn

5. Create an adapter service instance configuration in IAP Admin Essentials GUI

6. Copy the properties from the sampleProperties.json and paste them into the service instance configuration in the inner/second properties field.

7. Change the adapter service instance configuration (host, port, credentials, etc) in IAP Admin Essentials GUI

For an easier install of the adapter use npm run adapter:install, it will install the adapter in IAP. Please note that it can be dependent on where the adapter is installed and on the version of IAP so it is subject to fail. If using this, you can replace step 3-5 above with these:

3. Install adapter dependencies and check the adapter.

cd adapter-gcp_compute
npm run adapter:install

4. Restart IAP

systemctl restart pronghorn

5. Change the adapter service instance configuration (host, port, credentials, etc) in IAP Admin Essentials GUI

Testing

Mocha is generally used to test all Itential Opensource Adapters. There are unit tests as well as integration tests performed. Integration tests can generally be run as standalone using mock data and running the adapter in stub mode, or as integrated. When running integrated, every effort is made to prevent environmental failures, however there is still a possibility.

Unit Testing

Unit Testing includes testing basic adapter functionality as well as error conditions that are triggered in the adapter prior to any integration. There are two ways to run unit tests. The prefered method is to use the testRunner script; however, both methods are provided here.

node utils/testRunner --unit

npm run test:unit
npm run test:baseunit

To add new unit tests, edit the test/unit/adapterTestUnit.js file. The tests that are already in this file should provide guidance for adding additional tests.

Integration Testing - Standalone

Standalone Integration Testing requires mock data to be provided with the entities. If this data is not provided, standalone integration testing will fail. When the adapter is set to run in stub mode (setting the stub property to true), the adapter will run through its code up to the point of making the request. It will then retrieve the mock data and return that as if it had received that data as the response from GcpCompute. It will then translate the data so that the adapter can return the expected response to the rest of the Itential software. Standalone is the default integration test.

Similar to unit testing, there are two ways to run integration tests. Using the testRunner script is better because it prevents you from having to edit the test script; it will also resets information after testing is complete so that credentials are not saved in the file.

node utils/testRunner
  answer no at the first prompt

npm run test:integration

To add new integration tests, edit the test/integration/adapterTestIntegration.js file. The tests that are already in this file should provide guidance for adding additional tests.

Integration Testing

Integration Testing requires connectivity to GcpCompute. By using the testRunner script it prevents you from having to edit the integration test. It also resets the integration test after the test is complete so that credentials are not saved in the file.

Note: These tests have been written as a best effort to make them work in most environments. However, the Adapter Builder often does not have the necessary information that is required to set up valid integration tests. For example, the order of the requests can be very important and data is often required for creates and updates. Hence, integration tests may have to be enhanced before they will work (integrate) with GcpCompute. Even after tests have been set up properly, it is possible there are environmental constraints that could result in test failures. Some examples of possible environmental issues are customizations that have been made within GcpCompute which change order dependencies or required data.

node utils/testRunner
answer yes at the first prompt
answer all other questions on connectivity and credentials

Test should also be written to clean up after themselves. However, it is important to understand that in some cases this may not be possible. In addition, whenever exceptions occur, test execution may be stopped, which will prevent cleanup actions from running. It is recommended that tests be utilized in dev and test labs only.

Reminder: Do not check in code with actual credentials to systems.

Configuration

This section defines all the properties that are available for the adapter, including detailed information on what each property is for. If you are not using certain capabilities with this adapter, you do not need to define all of the properties. An example of how the properties for this adapter can be used with tests or IAP are provided in the sampleProperties.

Example Properties

  "properties": {
    "host": "compute.googleapis.com",
    "port": 443,
    "choosepath": "",
    "base_path": "/compute",
    "version": "v1",
    "cache_location": "none",
    "encode_pathvars": true,
    "encode_queryvars": true,
    "save_metric": false,
    "stub": false,
    "protocol": "https",
    "authentication": {
      "auth_method": "no_authentication",
      "username": "username",
      "password": "password",
      "token": "token",
      "invalid_token_error": 401,
      "token_timeout": 3600000,
      "token_cache": "local",
      "auth_field": "header.headers.Authorization",
      "auth_field_format": "Bearer {token}",
      "auth_logging": false,
      "client_id": "",
      "client_secret": "",
      "grant_type": "",
      "sensitive": [],
      "sso": {
        "protocol": "",
        "host": "",
        "port": 0
      },
      "multiStepAuthCalls": [
        {
          "name": "",
          "requestFields": {},
          "responseFields": {},
          "successfullResponseCode": 200
        }
      ],
      "configureGtoken": {
        "email": "",
        "scope": "https://www.googleapis.com/auth/cloud-platform",
        "sub": "",
        "keyFile": "/opt/pronghorn/RENAME_TO_YOUR_KEYFILE.json",
        "key": "",
        "additionalClaims": "",
        "eagerRefreshThresholdMillis": 300000
      }
    },
    "healthcheck": {
      "type": "startup",
      "frequency": 600000,
      "query_object": {},
      "addlHeaders": {}
    },
    "throttle": {
      "throttle_enabled": false,
      "number_pronghorns": 1,
      "sync_async": "sync",
      "max_in_queue": 1000,
      "concurrent_max": 1,
      "expire_timeout": 0,
      "avg_runtime": 200,
      "priorities": [
        {
          "value": 0,
          "percent": 100
        }
      ]
    },
    "request": {
      "number_redirects": 0,
      "number_retries": 3,
      "limit_retry_error": [
        0
      ],
      "failover_codes": [],
      "attempt_timeout": 5000,
      "global_request": {
        "payload": {},
        "uriOptions": {},
        "addlHeaders": {},
        "authData": {}
      },
      "healthcheck_on_timeout": true,
      "return_raw": false,
      "archiving": false,
      "return_request": false
    },
    "proxy": {
      "enabled": false,
      "host": "",
      "port": 1,
      "protocol": "http",
      "username": "",
      "password": ""
    },
    "ssl": {
      "ecdhCurve": "",
      "enabled": false,
      "accept_invalid_cert": false,
      "ca_file": "",
      "key_file": "",
      "cert_file": "",
      "secure_protocol": "",
      "ciphers": ""
    },
    "mongo": {
      "host": "",
      "port": 0,
      "database": "",
      "username": "",
      "password": "",
      "replSet": "",
      "db_ssl": {
        "enabled": false,
        "accept_invalid_cert": false,
        "ca_file": "",
        "key_file": "",
        "cert_file": ""
      }
    },
    "devicebroker": {
      "enabled": true,
      "getDevice": [
        {
          "path": "/{project}/global/networks",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "project": "{project}"
          },
          "responseDatakey": "items",
          "responseFields": {
            "name": "{name}",
            "ostype": "{kind}",
            "ostypePrefix": "VPC Network-",
            "ipaddress": "{id}",
            "port": "{selfLinkWithId}"
          }
        }
      ],
      "getDevicesFiltered": [
        {
          "path": "/{project}/global/networks",
          "method": "GET",
          "pagination": {
            "offsetVar": "",
            "limitVar": "",
            "incrementBy": "limit",
            "requestLocation": "query"
          },
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "project": "helical-crowbar-382719"
          },
          "responseDatakey": "items",
          "responseFields": {
            "name": "{name}",
            "ostype": "{kind}",
            "ostypePrefix": "VPC Network-",
            "ipaddress": "{id}",
            "port": "{selfLinkWithId}",
            "project": "{project}"
          }
        }
      ],
      "isAlive": [
        {
          "path": "/{project}/global/networks/{network}",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "project": "{project}",
            "network": ""
          },
          "responseDatakey": "",
          "responseFields": {}
        }
      ],
      "getConfig": [
        {
          "path": "/{project}/global/networks/{network}",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "project": "{project}",
            "network": ""
          },
          "responseDatakey": "",
          "responseFields": {}
        }
      ],
      "getCount": [
        {
          "path": "/{project}/global/networks",
          "method": "GET",
          "query": {},
          "body": {},
          "headers": {},
          "handleFailure": "ignore",
          "requestFields": {
            "project": "helical-crowbar-382719"
          },
          "responseDatakey": "items",
          "responseFields": {}
        }
      ]
    },
    "cache": {
      "enabled": false,
      "entities": [
        {
          "entityType": "device",
          "frequency": 3600,
          "flushOnFail": false,
          "limit": 1000,
          "retryAttempts": 5,
          "sort": true,
          "populate": [
            {
              "path": "/{project}/global/networks",
              "method": "GET",
              "pagination": {
                "offsetVar": "",
                "limitVar": "",
                "incrementBy": "limit",
                "requestLocation": "query"
              },
              "query": {},
              "body": {},
              "headers": {},
              "handleFailure": "ignore",
              "requestFields": {
                "project": "helical-crowbar-382719"
              },
              "responseDatakey": "items",
              "responseFields": {
                "name": "{name}",
                "ostype": "{kind}",
                "ostypePrefix": "VPC Network-",
                "ipaddress": "{id}",
                "port": "{selfLinkWithId}",
                "project": "{project}"
              }
            }
          ],
          "cachedTasks": [
            {
              "name": "",
              "filterField": "",
              "filterLoc": ""
            }
          ]
        }
      ]
    }
  }

Connection Properties

These base properties are used to connect to GcpCompute upon the adapter initially coming up. It is important to set these properties appropriately.

A connectivity check tells IAP the adapter has loaded successfully.

Authentication Properties

The following properties are used to define the authentication process to GcpCompute.

Note: Depending on the method that is used to authenticate with GcpCompute, you may not need to set all of the authentication properties.

Examples of authentication field format

"{token}"
"Token {token}"
"{username}:{password}"
"Basic {b64}{username}:{password}{/b64}"

Healthcheck Properties

The healthcheck properties defines the API that runs the healthcheck to tell the adapter that it can reach GcpCompute. There are currently three types of healthchecks.

• None - Not recommended. Adapter will not run a healthcheck. Consequently, unable to determine before making a request if the adapter can reach GcpCompute.
• Startup - Adapter will check for connectivity when the adapter initially comes up, but it will not check afterwards.
• Intermittent - Adapter will check connectivity to GcpCompute at a frequency defined in the frequency property.

Request Properties

The request section defines properties to help handle requests.

SSL Properties

The SSL section defines the properties utilized for ssl authentication with GcpCompute. SSL can work two different ways: set the accept\_invalid\_certs flag to true (only recommended for lab environments), or provide a ca\_file.

Throttle Properties

The throttle section is used when requests to GcpCompute must be queued (throttled). All of the properties in this section are optional.

Proxy Properties

The proxy section defines the properties to utilize when GcpCompute is behind a proxy server.

Mongo Properties

The mongo section defines the properties used to connect to a Mongo database. Mongo can be used for throttling as well as to persist metric data. If not provided, metrics will be stored in the file system.

Device Broker Properties

The device broker section defines the properties used integrate GcpCompute to the device broker. Each broker call is represented and has an array of calls that can be used to build the response. This describes the calls and then the fields which are available in the calls.

Using this Adapter

The adapter.js file contains the calls the adapter makes available to the rest of the Itential Platform. The API detailed for these calls should be available through JSDOC. The following is a brief summary of the calls.

Generic Adapter Calls

These are adapter methods that IAP or you might use. There are some other methods not shown here that might be used for internal adapter functionality.

Adapter Cache Calls

These are adapter methods that are used for adapter caching. If configured, the adapter will cache based on the interval provided. However, you can force a population of the cache manually as well.

Adapter Broker Calls

These are adapter methods that are used to integrate to IAP Brokers. This adapter currently supports the following broker calls.

Specific Adapter Calls

Specific adapter calls are built based on the API of the GcpCompute. The Adapter Builder creates the proper method comments for generating JS-DOC for the adapter. This is the best way to get information on the calls.

Authentication

This document will go through the steps for authenticating the Google Cloud Platform Compute adapter with Google Token Security. Properly configuring the properties for an adapter in IAP is critical for getting the adapter online. You can read more about adapter authentication HERE.

Companies periodically change authentication methods to provide better security. As this happens this section should be updated and contributed/merge back into the adapter repository.

Google Token Authentication

The Google Cloud Platform Compute adapter authenticates with a Google token.

STEPS

1. Ensure you have access to a Google Cloud Platform Compute server and that it is running
2. Follow the steps in the README.md to import the adapter into IAP if you have not already done so
3. Use the properties below for the properties.authentication field

Using key in service config (Needed for Cloud Instance)

"authentication": {
  "auth_method": "no_authentication",
  "configureGtoken": {
    "email": "sample@email.com",
    "key": "-----BEGIN PRIVATE KEY-----\nMIIEv********--\n",
    "scope": "https://www.********",
    "eagerRefreshThresholdMillis": 300000,
    "project_id": "myprojectid"
  }
}

Using keyfile in service config

"authentication": {
  "auth_method": "no_authentication",
  "configureGtoken": {
    "email": "",
    "scope": "https://www.googleapis.com/auth/cloud-platform",
    "sub": "",
    "keyFile": "/opt/pronghorn/RENAME_TO_YOUR_KEYFILE.json",
    "key": "",
    "additionalClaims": "",
    "eagerRefreshThresholdMillis": 300000
  }
}

you can leave all of the other properties in the authentication section, they will not be used for Google Cloud Platform Compute Google token authentication. 4. Restart the adapter. If your properties were set correctly, the adapter should go online.

Troubleshooting

• Make sure you copied over the correct Googgle properties as these are used to retrieve the token.
• Turn on debug level logs for the adapter in IAP Admin Essentials.
• Turn on auth_logging for the adapter in IAP Admin Essentials (adapter properties).
• Investigate the logs - in particular:
  • The FULL REQUEST log to make sure the proper headers are being sent with the request.
  • The FULL BODY log to make sure the payload is accurate.
  • The CALL RETURN log to see what the other system is telling us.
• Credentials should be masked by the adapter so make sure you verify the username and password - including that there are erroneous spaces at the front or end.
• Remember when you are done to turn auth_logging off as you do not want to log credentials.

Additional Information

Enhancements

Adding a Second Instance of an Adapter

You can add a second instance of this adapter without adding new code on the file system. To do this go into the IAP Admin Essentials and add a new service config for this adapter. The two instances of the adapter should have unique ids. In addition, they should point to different instances (unique host and port) of the other system.

Adding Adapter Calls

There are multiple ways to add calls to an existing adapter.

The easiest way would be to use the Adapter Builder update process. This process takes in a Swagger or OpenAPI document, allows you to select the calls you want to add and then generates a zip file that can be used to update the adapter. Once you have the zip file simply put it in the adapter directory and execute npm run adapter:update.

mv updatePackage.zip adapter-gcp_compute
cd adapter-gcp_compute
npm run adapter:update

If you do not have a Swagger or OpenAPI document, you can use a Postman Collection and convert that to an OpenAPI document using APIMatic and then follow the first process.

If you want to manually update the adapter that can also be done the key thing is to make sure you update all of the right files. Within the entities directory you will find 1 or more entities. You can create a new entity or add to an existing entity. Each entity has an action.json file, any new call will need to be put in the action.json file. It will also need to be added to the enum for the ph_request_type in the appropriate schema files. Once this configuration is complete you will need to add the call to the adapter.js file and, in order to make it available as a workflow task in IAP, it should also be added to the pronghorn.json file. You can optionally add it to the unit and integration test files. There is more information on how to work on each of these files in the Adapter Technical Resources on our Documentation Site.

Files to update
* entities/<entity>/action.json: add an action
* entities/<entity>/schema.json (or the schema defined on the action): add action to the enum for ph_request_type
* adapter.js: add the new method and make sure it calls the proper entity and action
* pronghorn.json: add the new method
* test/unit/adapterTestUnit.js (optional but best practice): add unit test(s) - function is there, any required parameters error when not passed in
* test/integration/adapterTestIntegration.js (optional but best practice): add integration test

Adding Adapter Properties

While changing adapter properties is done in the service instance configuration section of IAP, adding properties has to be done in the adapter. To add a property you should edit the propertiesSchema.json with the proper information for the property. In addition, you should modify the sampleProperties to have the new property in it.

Files to update
* propertiesSchema.json: add the new property and how it is defined
* sampleProperties: add the new property with a default value
* test/unit/adapterTestUnit.js (optional but best practice): add the property to the global properties
* test/integration/adapterTestIntegration.js (optional but best practice): add the property to the global properties

Changing Adapter Authentication

Often an adapter is built before knowing the authentication and authentication processes can also change over time. The adapter supports many different kinds of authentication but it does require configuration. Some forms of authentication can be defined entirely with the adapter properties but others require configuration.

Files to update
* entities/.system/action.json: change the getToken action as needed
* entities/.system/schemaTokenReq.json: add input parameters (external name is name in other system)
* entities/.system/schemaTokenResp.json: add response parameters (external name is name in other system)
* propertiesSchema.json: add any new property and how it is defined
* sampleProperties: add any new property with a default value
* test/unit/adapterTestUnit.js (optional but best practice): add the property to the global properties
* test/integration/adapterTestIntegration.js (optional but best practice): add the property to the global properties

Enhancing Adapter Integration Tests

The adapter integration tests are written to be able to test in either stub (standalone) mode or integrated to the other system. However, if integrating to the other system, you may need to provide better data than what the adapter provides by default as that data is likely to fail for create and update. To provide better data, edit the adapter integration test file. Make sure you do not remove the marker and keep custom code below the marker so you do not impact future migrations. Once the edits are complete, run the integration test as it instructs you to above. When you run integrated to the other system, you can also save mockdata for future use by changing the isSaveMockData flag to true.

Files to update
* test/integration/adapterTestIntegration.js: add better data for the create and update calls so that they will not fail.

As mentioned previously, for most of these changes as well as other possible changes, there is more information on how to work on an adapter in the Adapter Technical Resources on our Documentation Site.

Contributing

First off, thanks for taking the time to contribute!

The following is a set of rules for contributing.

Code of Conduct

This project and everyone participating in it is governed by the Code of Conduct. By participating, you are expected to uphold this code. Please report unacceptable behavior to support@itential.com.

How to Contribute

Follow the contributing guide (here)[https://gitlab.com/itentialopensource/adapters/contributing-guide]

Helpful Links

Adapter Technical Resources

Node Scripts

There are several node scripts that now accompany the adapter. These scripts are provided to make several activities easier. Many of these scripts can have issues with different versions of IAP as they have dependencies on IAP and Mongo. If you have issues with the scripts please report them to the Itential Adapter Team. Each of these scripts are described below.

Troubleshoot

Run npm run troubleshoot to start the interactive troubleshooting process. The command allows you to verify and update connection, authentication as well as healthcheck configuration. After that it will test these properties by sending HTTP request to the endpoint. If the tests pass, it will persist these changes into IAP.

You also have the option to run individual commands to perform specific test:

• npm run healthcheck will perform a healthcheck request of with current setting.
• npm run basicget will perform some non-parameter GET request with current setting.
• npm run connectivity will perform networking diagnostics of the adatper endpoint.

Connectivity Issues

1. You can run the adapter troubleshooting script which will check connectivity, run the healthcheck and run basic get calls.

npm run troubleshoot

2. Verify the adapter properties are set up correctly.

Go into the Itential Platform GUI and verify/update the properties

3. Verify there is connectivity between the Itential Platform Server and GcpCompute Server.

ping the ip address of GcpCompute server
try telnet to the ip address port of GcpCompute
execute a curl command to the other system

4. Verify the credentials provided for GcpCompute.

login to GcpCompute using the provided credentials

5. Verify the API of the call utilized for GcpCompute Healthcheck.

Go into the Itential Platform GUI and verify/update the properties

Functional Issues

Adapter logs are located in /var/log/pronghorn. In older releases of the Itential Platform, there is a pronghorn.log file which contains logs for all of the Itential Platform. In newer versions, adapters can be configured to log into their own files.

Itential Documents:

Documentation

Contributing Guide

FAQ

Related Items:

Adapters