Category: OpenVMS

OpenVMS x86 Initial Installation on VMWare ESXi vSphere Host

I had a lot of trouble trying to get piped serial connector working with ESXi but was finally able to get everything working with the steps below. I should note that the install manual was actually pretty good and helped me figure out the below.

For me I’m running ESXi 7.x with vCenter 7.x

I had issues with getting the piped method to work, and had a lot of issues.

I think a version of these instructions below also exist in the PDF manual but here are my notes as well in case I’m mistaken.

On the server that will be hosting the VM, you need to SSH into it’s shell and run the command:

esxcli network firewall ruleset rule list -r remoteSerialPort

In my case, the inbound/outbound ranges were:

Ruleset           Direction  Protocol  Port Type  Port Begin  Port End
----------------  ---------  --------  ---------  ----------  --------
remoteSerialPort  Outbound   TCP       Dst                 0     65535
remoteSerialPort  Inbound    TCP       Dst                23        23
remoteSerialPort  Inbound    TCP       Dst              1024     65535

So I picked a port like 2020 that falls both in the outbound range, and the inbound range.

But I had to check it was enabled with:

esxcli network firewall ruleset list -r remoteSerialPort

And it said:

Name              Enabled
----------------  -------
remoteSerialPort    false

Because it wasn’t enabled, I had to run:

esxcli network firewall ruleset set -r remoteSerialPort -e true

That took care of the firewall issue for me on the blade hosting the x86 VM.

On the VM, create a new serial but instead of selecting “Use named pipe” select “Use Network
The direction is set to “Server
The IP you want to use is the IP of the ESXi host which is hosting the VM, and you can use whatever port falls in the range from the previous step, so in my case the host is 192.168.1.27 and the port used is [b]2020[/b]

So the URI value is: tcp://192.168.1.27:2020

Make sure the check boxes for Connect At Power On is checked, and Yield CPU on poll is checked, but you can uncheckUse Virtual Serial Port Concentrator

After that’s all set, you gotta move onto setting up your SSH terminal, in my case I used PuTTY
The putty details are from a PDF manual regarding how to set it up, but it’s a lot less forgiving then usual

From Putty, go to Settings, then

“Terminal”
Then “Local echo”
“Force Off”

AND:

In Terminal settings, check the following boxes:

  • TURN OFF Implicit CR in every LF
  • TURN OFF Implicit LF in every CR
  • TURN OFF Local Echo
  • TURN OFF Local line editing
    1. In Window settings:
      set your Columns and Rows to match the Boot Manager display (typically 120 Columns, 40 Rows)
    2. If you are using a RAW session (recommended):

    • Host Name: 192.168.1.27

    • Port: 2020
    • Connection Type: Raw
    • Terminal → Implicit CR in every LF: UNCHECK
    • Terminal → Implicit LF in every CR: UNCHECK
    • Terminal → Local echo: Force off
    • Terminal → Local line editing: Force off
    • Telnet → Telnet Negotiation Mode: Passive
    • Telnet → Return key sends Telnet New Line instead of ^M: UNCHECK

Upgrading OpenVMS TCP to MultiNet/VSI TCPIP

A few VERY IMPORTANT notes.

  • This guide applies to OpenVMS 8.4.2U1.
  • From what I understand as of this post date, VSI has given up on their purchased Process software Multinet variation and have gone BACK to the original DEC/HPE TCPIP stack instead.
  • This guide doesn’t really have much value at this point but might later if VSI decides to use MultiNet later in the future.
  • This guide walks through how to install it, but does not include steps for “massaging it” to actually work once installed – in fact I had to use the added “switcher” to switch back to the old TCPIP…
    • VSI MultiNet isn’t finished yet so it’s not surprising it doesn’t work, or at least not without extensive continued work.

Source: https://vmssoftware.com/…VSI_TCPIP_INSTALL_QS.PDF

Step 1: Get the KITS files…

First, you’ll need to find the files for the TCPIP2 from the KITS folder under I64LPS. Have this ready to go for later.

Step 2: Update Your MODPARAMS file

Go ahead and edx/edit your SYS$SYSTEM:MODPARAMS.DAT file and make sure the following is set within it:

! Setup for VSI TCPIP 11-SEPT-2020 WM
MIN_GH_EXEC_DATA = 1500
MIN_GH_EXEC_CODE = 4200

Then run the below to place the new parameters into action for now and later:

$ @SYS$UPDATE:AUTOGEN SAVPARAMS SETPARAMS FEEDBACK

Step 3: Setting up the Directory

Create a TCPIP2 directory to upload the patches/kits

$ set default sys$common:[000000]
$ create/dir [.tcpip2]

For SFTP/SCP purposes, it’ll be located at:

/$30$dkb3/SYS0/SYSCOMMON/TCPIP2/

Step 4: Pre-Installation

VERY CAREFULLY follow the notes for wbem within it’s own .TXT file.

VERY IMPORTANT: For shared system disks, you’ll need to perform the install on ALL nodes one at a time, not just once!

These notes are for a single node system disk, non-clustered.

Install the pre-patch:

$ @sys$startup:wbem_services$shutdown
$ set default SYS$COMMON:[TCPIP2.pre]
$ unzip *.zipexe
$ PRODUCT INSTALL * /SOURCE=SYS$COMMON:[TCPIP2.pre]

When prompted:

  • Select install All Patches
  • Select defaults to prompts

After it’s done, run:

$ run SYS$COMMON:[WBEM_SERVICES]WBEM_Services$Config.exe

Follow the defaults, do not change the location and refresh the repository. (Not sure what this actually entails behind the scenes, but guessing it’s a localhost wbem thing) This will take roughly 10-20 minutes.

Step 5: Patch Installation

You can now run the TCPIP Install.

$ set default SYS$COMMON:[TCPIP2.patch]
$ unzip *.zipexe
$ PRODUCT INSTALL * /SOURCE=SYS$COMMON:[TCPIP2.patch...]

When prompted with the below…

   1 - VSI I64VMS VMS842L1I_CLUCONFIG V1.0 Patch (remedial update)
   2 - VSI I64VMS VSI_TCPIP V10.6          Layered Product
   ? - Help
   E - Exit

Choose one or more items from the menu:

You should type:

1,2

and press enter.

You should then see:

The following products have been selected:
    VSI I64VMS VMS842L1I_CLUCONFIG V1.0    Patch (remedial update)
    VSI I64VMS VSI_TCPIP V10.6             Layered Product
    
Do you want to continue? [YES]

Step 6: Before Reboot

Prior to rebooting, ensure that you comment out all commands in your system startup procedure that start the TCP/IP stacks within:  SYS$MANAGER:SYSTARTUP_VMS.COM 

Such as:

$ @TCPIP$STARTUP
$ @START_MULTINET
$ @TCPWARE:STARTNET

You must also comment out any corresponding START/NETWORK commands from your IP network stack.

In this example, the startup file: SYS$MANAGER:SYSTARTUP_VMS.COM

Had the following line:

$ @TCPIP$STARTUP

Changed to:

$ @SYS$STARTUP:IP$STARTUP

Step 7: Shutting it down…

TREAD LIGHTLY HERE! Don’t forget, if you’re not already working from the MP, you’ll need to connect to that instead of directly to the node via SSH.

Once you’re connected to the MP you can run the following shutdown command to shut it down…

$ @SYS$SYSTEM:SHUTDOWN

This is because you may not be able to connect with TCIP through SSH directly to the node if something goes wrong.

After rebooting, in my personal case, I had issues with the network having configuration errors still.

So I had to run:

@SYS$MANAGER:IP$SET_STACK

It then gave me a menu that allowed me to set the TCPIP stack back to how it previously was, and it gave an option to reboot now which I selected yes to perform.

End.

The lesson is, don’t use VSI Multinet aka VSI TCPIP.

OpenVMS: How to Search Intrusions List Easily

Sometimes listing out the show/intrusions command can be tricky and lengthy to try and see if a user has been set as an intruder.

To overcome this easily, simply PIPE a SEARCH command through the show/intrusions command.

So for example you can use the following command to make analyzing the intrusions a lot easier.

$ PIPE show/intrusions | search sys$input "your search string"