Strengthen Operational Security with IBM WebSphere Automation

banner

Author(s): Tien-Thanh Le, Ajay Reddy, Brian Hanczaryk, Jagadeeswar R Gangaraju, Yee-Kang Chang, Rafael Osorio

Last updated: July 2021

Duration: 45 mins

Need support? Contact Rafael Osorio and Kevin Postreich

Introduction to WebSphere Automation

IBM WebSphere Automation is focused on delivering value into existing WebSphere Application Server (WAS) environments, helping administrators reduce the cost, effort, and risk of addressing common vulnerabilities, automating tasks, and remediating capacity incidents.

It removes manual toil so that your team can spend more time innovating while minimizing the cost of extending the life and maximizing the ROI of your WebSphere investments.

In conversations with customers, the same three concerns come up repeatedly. Organizations need to keep their IT estate secure and compliant, resilient to disruption and running optimally while reducing costs and maximizing ROI.

WebSphere Automation helps organizations gain visibility, operational efficiencies, and cost savings quickly by extending the life of WebSphere investments and giving teams time back to focus on unlocking new value and fixing the imbalance of pure maintenance versus innovation tasks.

WebSphere operators and administrators save time and embrace DevSecOps by implementing patches more efficiently on virtual and container environments to keep operations compliant and secure.
Enhance remediation capabilities with insights and recommendations to improve the speed and depth of understanding of outages and anomalies as they occur.
Augment the operational experience with access to simplified and consolidated information that enables teams to act.

With WebSphere Automation, security, business efficiency and resiliency become standard. IBM can meet you wherever you are in your optimization and automation journeys to help you quickly deliver value and increase ROI, all while laying a solid automation foundation to support future growth.

IBM WebSphere Automation is available as a stand-alone offering or as an addition to IBM Cloud Pak® for Watson AIOps. As part of IBM Automation platform, IBM WebSphere Automation includes containerized components and common software services on top of a common automation layer, to manage WebSphere’s incidents, hybrid applications, and cost with complete observability, governance, and compliance.

Deploy virtually anywhere through containers supported by Red Hat® OpenShift® software, on IBM Cloud®, on essentially any existing infrastructure on-premises, or through private and public clouds. Use only the capabilities you need with a fully modular approach that’s designed to be easy to consume.

Business Context

You are a WebSphere Administrator, part of a WebSphere Operations Team responsible for maintaining security compliance of the WebSphere estate in the enterprise. A typical “as-is” process for maintaining security compliance for WebSphere environments is depicted below.

Today (as-is):

IBM sends generic “FLASH” to indicate a new WAS security bulletin.
You subscribe and receive IBM Security Bulletins to be aware about vulnerabilities, its potential impact, severity, and recommended solutions.
Generally, WAS inventory is maintained in spreadsheets.
Based on that, you check if this CVE applies to your managed inventory (Spreadsheet)
You determine if an APAR / Fix Pack upgrade should be applied to existing environment
You deploy the fix to the impacted environments
You update the WAS inventory (Spreadsheet) and provide up-to-date reports to audit and compliance teams

As is, your inventory is a spreadsheet, containing all information about your servers, such as the versions of the installed servers, which operating system they're installed on, and iFixes which have been applied, etc

Currently, this is a very manual, time-consuming process, and you'd like to automate this process to direct valuable time and resource elsewhere. This is where IBM WebSphere Automation will help!

You would like to have:

Management dashboard: Consolidated dashboard increases awareness and response time to common vulnerabilities and exposures (CVEs).
Automated vulnerability tracking: Let WebSphere Automation track new security bulletins across your existing traditional WebSphere and Liberty environments, on virtual machines or containers.
Contextual notifications: Receive security bulletin notifications only when new vulnerabilities affect the environment you manage, reducing noise and interruptions to the WebSphere operations team.
Shared, live visibility to key stakeholders: WebSphere operators and security compliance teams can see the real-time security posture of the WebSphere estate, accelerating action and minimizing the risk of miscommunication.

In this lab, you use the IBM WebSphere Automation to secure operations to reduce risk and meet compliance.

At the end of this lab, you will be able to connect teams with the most relevant information through a single dashboard. This enables you to discover, analyze and remediate common vulnerabilities and exposures across instances. Furthermore, this information can be exported to a CSV file to be shared amongst the broader team.

Accessing and starting the environment

When the demo environment is provisioned, use the provided username and password to access and start the environment. You should see the following screen:

WebSphere Automation is pre-installed in the OCP cluster hosted on the VMs.
If the environment is NOT already started, go ahead, and Start the environment by clicking on the Play button. It takes about 10-15 minutes for the environment to start and stabilize.
You will be working on the STUDENT VM only.

All the WebSphere images and iFixes required for the lab are pre-installed:
- /opt/IBM/WebSphere/Liberty200012/
- /opt/IBM/WebSphere/Liberty20009/
- /opt/IBM/WebSphere/AppServer9057/
- /opt/IBM/WebSphere/AppServer9056/
Click the screen representing the STUDENT VM
Once you log in to the Student VM you will see the Desktop, which contains all the programs that you will be using (browsers, terminal, etc.)

The login credentials for the STUDENT” image is:
- User ID: ibmuser
- Password: engageibm!
You can resize the virtual desktop with the Fit to window button, located at the top of the Skytap environment window.

Receiving vulnerability notifications

Accessing the WebSphere Automation UI

A WebSphere administrator sets up WebSphere Automation by registering and configuring WebSphere Application Servers and WebSphere Liberty servers for vulnerability tracking and by configuring email notifications.

WebSphere administrators can also view the results of vulnerability assessment in WebSphere Automation to plan their response for the WebSphere Application Server and WebSphere Liberty servers that they manage.

For this lab, WebSphere Automation is pre-installed on an OCP cluster. You have your individual WebSphere Automation installation. Let’s access your environment.

On the Student VM, open a browser and enter the following URL

Note: It takes about 10-15 minutes for the environment to start and stabilize once it has started. If you encounter "502 Gateway Error" accessing the WebSphere Automation URL, please wait a few minutes and retry.

https://cpd-websphere-automation.apps.ocp.ibm.edu
On the login page, select the OpenShift authentication as the authentication type.

Note: If using Firefox, enlarge the browser window until you see the graphics, as illustrated below.

Note: If necessary, accept all the warnings and certificates. Depending on your browser, you might have to scroll down to permit access.
Enter ocadmin as username and ibmocp46 as password. And click Log In.
The IBM Automation Welcome page appears.
On the Welcome page, open the Menu, click Operate, and then click Application runtimes.

Application Runtimes represent the Traditional WebSphere and WebSphere Liberty servers that have been registered with IBM Automation.
The Application runtimes page appears. There should be no data since no WebSphere servers are registered/configured yet.
Before you start to register servers to the Dashboard, you need to configure an email to received notifications about CVEs.

a. Open the Notification configuration tab.
The Email server configuration is pre-configured for this lab. You only need to add your personal email to receive notifications of new security vulnerabilities.

a. Click Add email address (1)

b. Enter your email and click Save (2).

Great! Your email is configured to receive security notifications.

In the next section, you will register servers to WebSphere Automation.

Getting configuration parameters

Add each of your WebSphere Application Server and WebSphere Liberty servers to WebSphere Automation by registering them with the usage metering service.

To register your application servers with the usage metering service in WebSphere Automation, you must configure the usage metering feature in each application server. To configure the usage metering feature in each of your application servers, you must obtain the following usage metering details:

URL: The URL of the usage metering service in WebSphere Automation. This service registers WebSphere Application Server and Liberty servers with WebSphere Automation so that you can track security vulnerabilities.
API Key: The token used to authenticate the WebSphere Application Server and Liberty servers during the registration process.
Usage metering certificate: The certificate that contains the public key. This key allows an application server that is registering with WebSphere Automation to do an SSL handshake with the metering service.

In this section, you will get these configuration parameters. Let’s start it!

Return to the desktop and open a new terminal window.
Log in as root user
```
su root
```
a. Enter the following password when prompted

Note: The first character in the password is numeric One (1)
```
1bm2021rhjb
```

Next, log in to the OCP Cluster using the command below:

oc login --username=ocadmin --password=ibmocp46 --insecure-skip-tls-verify=true --server=https://api.ocp.ibm.edu:6443

oc login

Make sure that the project you are working on is websphere-automation:
```
oc project websphere-automation
```
Use the oc command to get the URL of the usage metering service in WebSphere Automation and save it to a file “/opt/IBM/WebSphere/metering-url.txt”
```
echo https://$(oc get route cpd -n websphere-automation -o jsonpath='{.spec.host}')/websphereauto/meteringapi > /opt/IBM/WebSphere/metering-url.txt
```
View the contents of the saved file to ensure the URL was captured.
```
cat /opt/IBM/WebSphere/metering-url.txt
```

Get the api-key that will be used to authenticate the WebSphere Application Server and Liberty servers during the registration process. Save it to a file named “/opt/IBM/WebSphere/api-key.txt”

oc -n websphere-automation get secret automation-secure-metering-apis-encrypted-tokens -o jsonpath='{.data.automation-secure-metering-apis-sa}' | base64 -d > /opt/IBM/WebSphere/api-key.txt; echo >> /opt/IBM/WebSphere/api-key.txt

View the contents of the saved file to ensure the api-key (token) was captured.
```
cat /opt/IBM/WebSphere/api-key.txt
```
Finally, get the Server certificate that is used for SSL handshake between the servers and IBM Automation, and save it to a file named “/opt/IBM/WebSphere/cacert.pem”
```
oc get secret external-tls-secret -n websphere-automation -o jsonpath='{.data.cert\.crt}' | base64 -d > /opt/IBM/WebSphere/cacert.pem
```
View the contents of the saved file to ensure the api-key (token) was captured.
```
cat /opt/IBM/WebSphere/cacert.pem
```

Great! Now you have all the configuration parameters necessary to register the application servers with the usage metering service in WebSphere Automation.

In the next section, you register your first server in WebSphere Automation.

Configuring Liberty Server v20.0.0.12

In this section, you configure Liberty Server version 20.0.0.12 to register to WebSphere Automation. Since Liberty servers are easily created, you will first create a new Liberty server and start it.

Create a new Liberty server, version 20.0.0.12, using the command below:

/opt/IBM/WebSphere/Liberty200012/bin/server create Liberty_200012_server

2012 server created

Now you need to configure the server to use TLS/SSL using the provided server_tls.xml file:

cp -f /home/ibmuser/Desktop/lab_backup/liberty200012/server_tls.xml /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/server.xml

a. If prompted to overwite the file, enter "y" for yes.

Start the Liberty server:
```
/opt/IBM/WebSphere/Liberty200012/bin/server start Liberty_200012_server
```
4. Now, you need to update the DefaultKeyStore with the server certificate to trust the server for https connection:

The command below will import the “cacert.pem” file, that you saved in a previous step, into the Liberty trust store, thereby trusting the server for secure HTTPS connections from IBM Automation.
```
keytool -import -trustcacerts -file /opt/IBM/WebSphere/cacert.pem -keystore /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/resources/security/key.p12 -storetype PKCS12 -storepass th1nkpassword -noprompt
```
Configure the usage metering in the new server. This is configured in the Liberty server.xml file.

a. Open the server.xml file using the “gedit” editor
```
gedit /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/server.xml
```
b. Add the usageMetering-1.0 feature to the existing features in the <featureManager> element, as illustrated below:
```
<feature>usageMetering-1.0</feature>
```
c. In the server.xml file, add the usageMetering element below:
```
<usageMetering url="${metering-url}" apiKey="${api-key}" sslRef="defaultSSL"/>
```
Save and Close the server.xml file.

Create a bootstrap.properties file for the Liberty server that contains the variable values for metering_url and api_key that are defined in the server.xml file

echo "metering-url=$(cat /opt/IBM/WebSphere/metering-url.txt)" >> /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/bootstrap.properties

echo "api-key=$(cat /opt/IBM/WebSphere/api-key.txt)" >> /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/bootstrap.properties

Use the cat command to view the contents of the bootstrap.properties file to ensure it has proper values assigned to the variables
```
cat /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/bootstrap.properties
```
The metering-url and api-key variables should have values as illustrated below.
Restart the Liberty server so that it can be initialized with the bootstrap.properties file.

Note: Use the --clean option when starting the Liberty server, to clear any cached data.
```
/opt/IBM/WebSphere/Liberty200012/bin/server stop Liberty_200012_server

/opt/IBM/WebSphere/Liberty200012/bin/server start Liberty_200012_server --clean
```
View the Liberty server “messages.log” file and find the message indicating that the server was registered to the metering service.
```
tail /opt/IBM/WebSphere/Liberty200012/usr/servers/Liberty_200012_server/logs/messages.log
```
In the WebSphere Automation UI, navigate to the Servers tab, to view the list of registered servers. The new Liberty server should be registered

a. Confirm that the Liberty server is registered in the WebSphere Automation Application runtimes page, as illustrated below.

IMPORTANT:

If the IBM Automation UI does not automatically detect the Liberty server that you registered, then it is extremely likely that the server.xml file or the bootstrap.properties file has not been configured correctly.

1. Review the server.xml configuration.
    - Ensure the usageMetering-1.0 feature is included in the list of features.

2. Review the bootstrap.properties file
    - Ensure the usageMetering url is CORRECT. No additional characters or spaces.
    - Ensure the usageMetering apiKey is CORRECT. No additional characters or spaces.

If the Liberty server was successfully registered, it is displayed in the Application Runtimes in IBM automation UI.

dashboard server 1


	TIP: Note: New vulnerabilities are discovered constantly, so the number of CVEs discovered may be different than illustrated.

Configuring Liberty Server v20.0.0.9

In this section, you configure another Liberty Server to your WebSphere Automation dashboard. The process is the same (using the usage metering service in WebSphere Automation). However, to simplify the lab, we have pre-created the configuration files to make the configuration simpler.

Return to the Terminal window where you are logged in as “root”.

Tip: You can run the “whoami” command in the terminal window to determine the currently logged in user.
First, you need to create the Liberty server version 20.0.0.9, using the command below:
```
/opt/IBM/WebSphere/Liberty20009/bin/server create Liberty_20009_server
```
Since you have already configured usageMetering feature for Liberty version 20.0.0.12, we have a provided couple of server.xml files to make the configuration simpler.

a. Copy the tls configuration that we provided, to the new Liberty server configuration.
```
cp -f /home/ibmuser/Desktop/lab_backup/liberty20009/server_tls.xml /opt/IBM/WebSphere/Liberty20009/usr/servers/Liberty_20009_server/server.xml
```
b. If prompted to overwite the file, enter"y" for yes.

Start the Liberty server:

/opt/IBM/WebSphere/Liberty20009/bin/server start Liberty\_20009\_server

Update the DefaultKeyStore with the server certificate to trust the server for https connections:

keytool -import -trustcacerts -file /opt/IBM/WebSphere/cacert.pem -keystore /opt/IBM/WebSphere/Liberty20009/usr/servers/Liberty_20009_server/resources/security/key.p12 -storetype PKCS12 -storepass th1nkpassword -noprompt

Copy the server.xml that contains the usageMetering feature and properties (url, api-key):

cp -f /home/ibmuser/Desktop/lab_backup/liberty20009/server_configured.xml /opt/IBM/WebSphere/Liberty20009/usr/servers/Liberty_20009_server/server.xml

a. If prompted to overwite the file, enter"y" for yes.

Back to the WebSphere Automation dashboard, confirm that the 2^nd Liberty server is registered.

The dashboard shows that this server is vulnerable for some CVEs
Click on the “+1 more” under the Unresolved CVEs for the Liberty_20009_server, to see the list of the unresolved CVEs, including the CVE-2020-10693.

Note that the +1 could be different if additional CVEs have been discovered since the time of this writing.

a. The list of Unresolved CVEs for Liberty version 20.0.0.9 should include CVE-2020-10693

b. Check your email that you registered with IBM automation. A mail notification was sent indicating the vulnerability.


	Information: In the email message, there will be a link that would redirect to the console to show more details on the vulnerability. However, Because of network restrictions in the lab environment, this link will not work.

Configuring traditional WebSphere (tWAS) v9.0.5.6

In this section, you configure a traditional WebSphere Application Server to your WebSphere Automation dashboard. With traditional WebSphere, you use the wsadmin script to configure the usage metering service.

First, let’s start the traditional WebSphere (tWAS) 9.0.5.6 server. Return to the terminal window and execute the command below:
```
/opt/IBM/WebSphere/AppServer9056/bin/startServer.sh tWAS_9056_server
```
Configure usage-metering using the wsadmin script below:

The wsadmin command invokes a Python script named configuretWasUsageMetering.py. The script requires the same url and apiKey that you gathered earlier from the IBM automation environment. These parameters are captured from the text files that you saved earlier in the lab and inserted into the url and apiKey parameters.
```
/opt/IBM/WebSphere/AppServer9056/bin/wsadmin.sh -f /api-usagemetering/scripts/configuretWasUsageMetering.py url=$(cat /opt/IBM/WebSphere/metering-url.txt) apiKey=$(cat /opt/IBM/WebSphere/api-key.txt) trustStorePassword=th1nkpassword
```
The script should run successfully as illustrated below.

Great, you first traditional WAS server is configured. Let’s check the WebSphere Automation dashboard.
Go back to the browser, navigate to the Servers dashboard, and confirm that the tWAS v9.0.5.6 server was registered.

You should see that this server is vulnerable to 14 or more CVEs.

TIP:
Note: New vulnerabilities are discovered constantly, so the number of CVEs discovered may be different than illustrated.
Check your email. A mail notification was sent showing the vulnerability.

Configuring traditional WebSphere (tWAS) v9.0.5.7

In this section, you configure another version of a traditional WebSphere Application Server to your WebSphere Automation dashboard. The steps here are the same of the last section.

First, you just need to start the tWAS 9.0.5.7 server. On the terminal window, execute the command below:
```
/opt/IBM/WebSphere/AppServer9057/bin/startServer.sh tWAS_9057_server
```

Again, you need to configure the usage-metering.

/opt/IBM/WebSphere/AppServer9057/bin/wsadmin.sh -f /api-usagemetering/scripts/configuretWasUsageMetering.py url=$(cat /opt/IBM/WebSphere/metering-url.txt) apiKey=$(cat /opt/IBM/WebSphere/api-key.txt) trustStorePassword=th1nkpassword

Great, your second traditional WAS server is configured. Check our WebSphere Automation dashboard. Back to the browser, dashboard, confirm that the tWAS v9.0.5.7 is registered.

You should see that this server is vulnerable to CVE-2021-26296.

As explained before, maybe your server has more vulnerability than illustrated above, since new vulnerabilities are constantly discovered.
Again, check your email. A mail notification was sent showing the vulnerability.

Update Liberty server v20.0.0.9 to fix the vulnerability

In this section, you will fix one vulnerability in our Liberty Server v20.0.0.9 using two different approaches. First, you fix by only updating the configuration. Later, you apply an iFix to solve the vulnerability.

Updating configurations

Liberty 20.0.0.9 is impacted by CVE-2020-10693. This is because it configures the beanValidation-2.0 feature. See CVE-2020-10693 for more info.

We should resolve this by applying the iFix PH29942 as documented in the CVE-2020-10693. However, for the lab, we can also do a quick test by unconfiguring this feature, to illustrate that IBM Automation does interrogate the Liberty Server configuration, to determine if a Liberty server is impacted by a CVE that is the result of a specific configuration feature. This is valuable because IBM Automation can pinpoint specific servers that are impacted not ONLY based on the version of server, but also the specific configuration that is impacted by the CVE.

Edit the server.xml, using the command below:

gedit /opt/IBM/WebSphere/Liberty20009/usr/servers/Liberty_20009_server/server.xml

Comment out the beanValidation-2.0 feature:

<!-- <feature>beanValidation-2.0</feature> -->

Save and close the server.xml file.
Back to your browser, check that the Liberty 20.0.0.9 server does NOT show the CVE-2020-10693 vulnerability. The update is picked up automatically.

Great, you removed the vulnerability by updating the configuration.

Resolve the issue By Applying an iFix

However, instead of removing the beanValidationFeature-2.0, the correct process is to apply the appropriate iFix to get rid of the vulnerability.

First, add back the beanValidation feature:

gedit /opt/IBM/WebSphere/Liberty20009/usr/servers/Liberty_20009_server/server.xml

Uncomment the beanValidation-2.0 feature:
```
<feature>beanValidation-2.0</feature>
```
Save and Close the server.xml file.
Back to the browser, make sure the vulnerability shows up again.
You need to stop the server before the iFix can be applied. Return to the terminal window and run the command below to stop the Liberty2009 server.
```
/opt/IBM/WebSphere/Liberty20009/bin/server stop Liberty_20009_server
```
Now, apply the iFix, by running the following command.

Note: We have already downloaded the iFixes used for this lab. They are stored in /iFix directory on the STUDENT VM.
```
/iFix/PH29942/imcl_ifix_install.sh 20009
```

Great, the iFix was applied. Now, start the server again:

/opt/IBM/WebSphere/Liberty20009/bin/server start Liberty_20009_server

Return to the WebSphere Automation Dashboard. You should notice that the CVE-2020-10693 was removed from the Liberty_20009_server.

Update tWAS server v9.0.5.7 to fix the vulnerability

In this section, you will apply an iFix to the traditional WebSphere 9.0.5.7 server to remove the CVE-2021-26296 vulnerability.

Notice in the IBM Automation console that tWAS 9.0.5.7 is vulnerable to CVE-2021-26296.

Now, you will fix it by applying the appropriate iFix.

First, stop the server, wait until it has stopped:

/opt/IBM/WebSphere/AppServer9057/bin/stopServer.sh tWAS_9057_server

Run the following script to install the iFix:
```
/iFix/PH34711/imcl_ifix_install.sh 9057
```

Start the server once the iFix installation has completed:

/opt/IBM/WebSphere/AppServer9057/bin/startServer.sh tWAS_9057_server

Go back to the WebSphere Automation dashboard. Notice that the CVE-2021-26296 was removed from tWAS 9.0.5.7. Also, you can see on the right under “Applied iFixes” there are 4 applied fixes.

Update tWAS server v9.0.5.7 to introduce the vulnerability back (optional)

You can remove the iFix that was applied to confirm that the CVE-2021-26296 shows up in the unresolved CVE list

Stop the server:

/opt/IBM/WebSphere/AppServer9057/bin/stopServer.sh tWAS_9057_server

Uninstall the iFix:

/iFix/PH34711/imcl_ifix_uninstall.sh 9057

Start the server:

/opt/IBM/WebSphere/AppServer9057/bin/startServer.sh tWAS_9057_server

Go back to the WebSphere Automation Dashboard on your browser. Notice that CVE-2021-26296 is now again in the tWAS 9.0.5.7 server:

Summary

Congratulations! You have completed the WebSphere Automation lab.

With automated tooling and insights, IBM WebSphere Automation enables teams to modernize and secure IT estates, adapt and respond to incidents efficiently, and optimize WebSphere operations. WebSphere system operators and administrators can reduce the cost, effort, and risk of addressing vulnerabilities, automate critical activities, and preserve uptime with early detection, notification, and remediation of incidents.

IBM WebSphere Automation helps teams remove manual toil to work less on maintenance tasks and more on strategic activities, while unlocking new value, extending the life, and increasing ROI of WebSphere investments.

IBM WebSphere Automation is part of IBM Automation, a set of shared automation services that help you get insight into how your processes run, visualize hotspots and bottlenecks, and use financial impact information to prioritize which issues to address first.

To learn more about IBM WebSphere Automation, visit ibm.com/cloud/websphere-automation.