Troubleshooting using the serial console (original) (raw)

Linux Windows

This page describes how to enable interactive access to an instance's serial console to debug boot and networking issues, troubleshoot malfunctioning instances, interact with the GRand Unified Bootloader (GRUB), and perform other troubleshooting tasks.

A virtual machine (VM) instance has four virtual serial ports. Interacting with a serial port is similar to using a terminal window, in that input and output is entirely in text mode and there is no graphical interface or mouse support. The instance's operating system, BIOS, and other system-level entities often write output to the serial ports, and can accept input such as commands or answers to prompts. Typically, these system-level entities use the first serial port (port 1) and serial port 1 is often referred to as the serial console.

If you only need to view serial port output without issuing any commands to the serial console, you can call thegetSerialPortOutputmethod or use Cloud Logging to read information that your instance has written to its serial port; seeViewing serial port logs. However, if you run into problems accessing your instance through SSH or need to troubleshoot an instance that is not fully booted, you can enable interactive access to the serial console, which lets you connect to and interact with any of your instance's serial ports. For example, you can directly run commands and respond to prompts in the serial port.

When you enable or disable the serial port, you can use any Boolean value that is accepted by the metadata server. For more information, seeBoolean values.

Before you begin

Permissions required for this task

To perform this task, you must have the followingpermissions:

Enabling interactive access on the serial console

Enable interactive serial console access for individual VM instances or for an entire project.

Enabling access for a project

Enabling interactive serial console access on a project enables access for all VM instances that are part of that project.

By default, interactive serial port access is disabled. You can also explicitly disable it by setting the serial-port-enable key to FALSE. In either case, any per-instance setting overrides the project-level setting or the default setting.

Console

  1. In the Google Cloud console, go to the Metadata page.
    Go to Metadata
  2. Click Edit to edit metadata entries.
  3. Add a new entry that uses the key serial-port-enable and valueTRUE.
  4. Save your changes.

gcloud

Using the Google Cloud CLI, enter theproject-info add-metadatacommand as follows:

gcloud compute project-info add-metadata
--metadata serial-port-enable=TRUE

REST

In the API, make a request to theprojects().setCommonInstanceMetadatamethod, providing the serial-port-enable key with a value of TRUE:

{ "fingerprint": "FikclA7UBC0=", "items": [ { "key": "serial-port-enable", "value": "TRUE" } ] }

Enabling access for a VM instance

Enable interactive serial console access for a specific instance. A per-instance setting, if it exists, overrides any project-level setting. You can also disable access for a specific instance, even if access is enabled on the project level, by setting serial-port-enable to FALSE, instead of TRUE. Similarly, you can enable access for one or more instances even if it is disabled for the project, explicitly or by default.

Console

  1. In the Google Cloud console, go to the VM instances page.
    Go to the VM instances page
  2. Click the instance you want to enable access for.
  3. Click Edit.
  4. Under the Remote access section, toggle the Enable connecting to serial ports checkbox.
  5. Save your changes.

gcloud

Using the Google Cloud CLI, enter theinstances add-metadatacommand, replacing instance-name with the name of your instance.

gcloud compute instances add-metadata instance-name
--metadata serial-port-enable=TRUE

REST

In the API, make a request to theinstances().setMetadatamethod with the serial-port-enable key and a value of TRUE:

POST https://compute.googleapis.com/compute/v1/projects/myproject/zones/us-central1-a/instances/example-instance/setMetadata { "fingerprint": "zhma6O1w2l8=", "items": [ { "key": "serial-port-enable", "value": "TRUE" } ] }

For bare metal instances, increase the bit rate, also known as baud rate, for the serial console to 115,200 bps (~11.5kB/sec). Using a slower speed results in garbled or missing console output.

Bootloader configuration varies between operating systems and OS versions. Refer to the OS distributor's documentation for instructions.

If modifying the bit rate on the command line for the current session, use a command similar to the following:

console=ttyS0,115200

If modifying the GRUB configuration, use a command similar to the following:

serial --speed=115200

Make sure that you update the actual bootloader configuration. This can be done with update-grub, grub2-mkconfig, or a similar command.

Connecting to a serial console

Compute Engine offers regional serial console gateways for each Google Cloud region. After enabling interactive access for a VM's serial console, you can connect to a regional serial console.

The serial console authenticates users withSSH keys. Specifically, you must add your public SSH key to the project or instance metadata and store your private key on the local machine from which you want to connect. The gcloud CLI and the Google Cloud console automatically add SSH keys to the project for you. If you are using a third-party client, you might need to add SSH keys manually.

If you are using a third-party client, you can additionallyvalidate the connection using the serial console's host keys. When you use the Google Cloud CLI to connect, host key authentication is done automatically on your behalf.

Console

To connect to a VM's regional serial console, do the following:

  1. In the Google Cloud console, go to the VM instances page.
    Go to the VM instances page
  2. Click the instance you want to connect to.
  3. Under Details, click Connect to serial console to connect on the default port (port 1).
  4. If you want to connect to another serial port, click the down arrow next to the Connect to serial console button and change the port number accordingly.
  5. For Windows instances, open the drop-down menu next to the button and connect to Port 2 to access the serial console.

gcloud

To connect to a VM's regional serial console, use thegcloud compute connect-to-serial-port command:

gcloud compute connect-to-serial-port VM_NAME 

  --port=PORT_NUMBER

Replace the following:

Other SSH clients

You can connect to an instance's serial console using other third-party SSH clients, as long as the client lets you connect to TCP port 9600. Before you connect, you can optionallyvalidate the connection using the serial console's host keys.

To connect to a VM's regional serial console, run one of the following commands, depending on your VM's OS:

Replace the following:

If you are having trouble connecting using a third-party SSH client, you can run the gcloud compute connect-to-serial-port command with the --dry-runcommand-line option to see the SSH command that it would have run on your behalf. Then you can compare the options with the command you are using.

Validate third-party SSH client connections

When you use a third-party SSH client that isn't the Google Cloud CLI, you can ensure that you're protected against impersonation or man-in-the-middle attacks by checking Google's Serial Port SSH host key. To set up your system to check the SSH host key, complete the following steps:

  1. Download the SSH host key for the serial console you will be using:
    • For regional connections, the SSH host key for a region can be found athttps://www.gstatic.com/vm_serial_port_public_keys/REGION/REGION.pub
    • For global connections, download Google's Serial Port SSH host key
  2. Open your known hosts file, generally located at ~/.ssh/known_hosts.
  3. Add the contents of the SSH host key, with the server's hostname prepended to the key. For example, if the us-central1 server key contains the linessh-rsa AAAAB3NzaC1yc..., then ~/.ssh/known_hosts should have a line like this:
    [us-central1-ssh-serialport.googleapis.com]:9600 ssh-rsa AAAAB3NzaC1yc...

For security reasons, Google might occasionally change the Google Serial Port SSH host key. If your client fails to authenticate the server key, immediately end the connection attempt and complete the earlier steps to download a new Google Serial Port SSH host key.

If, after updating the host key, you continue to receive a host authentication error from your client, stop attempts to connect to the serial port and contact Google support. Don't provide any credentials over a connection wherehost authenticationhas failed.

Disconnecting from the serial console

To disconnect from the serial console, follow the instructions for the method you used to connect.

Console

In the Google Cloud console, disconnect from the serial console by doing the following:

  1. Close the browser window or tab that contains the serial console connection.

gcloud

In the Google Cloud CLI, disconnect from the serial console by doing the following:

  1. Press the ENTER key.
  2. Type ~. (tilde, followed by a period).

Other SSH clients

In other SSH clients, disconnect from the serial console by doing the following:

  1. Press the ENTER key.
  2. Type ~. (tilde, followed by a period).

In the Google Cloud CLI, or using SSH, you can discover other commands by typing ~?. You can also examine the man page for SSH with the following command:

man ssh

Don't try to disconnect using any of the following methods:

Connecting to a serial console with a login prompt

If you are trying to troubleshoot an issue with a VM that has booted completely or trying to troubleshoot an issue that occurs after VM has booted past single user mode, you might be prompted for login information when trying to access the serial console.

By default, Google-supplied Linux system images are not configured to allow password-based logins for local users. However, Google-supplied Windows images are configured to allow password-based logins for local users.

If your VM is running an image that is preconfigured with serial port logins, you need to set up a local password on the VM so that you can sign in to the serial console, if prompted. You can set up a local password after connecting to the VM or by using a start-up script.

Setting up a local password using a startup script

You can use a startup script to set up a local password that lets you connect to the serial console during or after VM creation.

To set up a local password in an existing VM, select one of the following options:

Linux

  1. In the Google Cloud console, go to the VM instances page.
    Go to VM instances
  2. In the Name column, click the name of the VM for which you want to add a local password.
    The details page of the VM opens.
  3. Click Edit.
    The page to edit the details of the VM opens.
  4. In the Metadata > Automationsection, do the following:
    1. If the VM has an existing startup script, then remove it and store the script somewhere safe.
    2. Add the following startup script:
    #!/bin/bash  
useradd USERNAME  
echo 'USERNAME:PASSWORD' | chpasswd  
usermod -aG google-sudoers USERNAME

    Replace the following:
    * USERNAME: the username that you want to add.
    * PASSWORD: the password for the username. As some operating systems require minimal password length and complexity, specify a password as follows:
    * Use at least 12 characters.
    * Use a mix of upper and lower case letters, numbers, and symbols.

  5. Click Save.
    The details page of the VM opens.
  6. Click Reset.
  7. Connect to the serial console.
  8. When prompted, enter your login information.

Windows

  1. In the Google Cloud console, go to the VM instances page.
    Go to VM instances
  2. In the Name column, click the name of the VM for which you want to add a local password.
    The details page of the VM opens.
  3. Click Edit.
    The page to edit the details of the VM opens.
  4. In the Metadata section, do the following:
    1. If the VM has an existing startup script, then store the script somewhere safe, and then, to delete the script, clickDelete item.
    2. Click Add item.
    3. In the Key field, enter windows-startup-script-cmd.
    4. In the Value field, enter the following script:
    net user USERNAME PASSWORD /ADD /Y  
net localgroup administrators USERNAME /ADD

    Replace the following:
    * USERNAME: the username that you want to add.
    * PASSWORD: the password for the username. As some operating systems require minimal password length and complexity, specify a password as follows:
    * Use at least 12 characters.
    * Use a mix of upper and lower case letters, numbers, and symbols.

  5. Click Save.
    The details page of the VM opens.
  6. Click Reset.
  7. Connect to the serial console.
  8. When prompted, enter your login information.

After the user has been created, replace the startup script with the startup script that you stored in this section.

Setting up a local password using passwd on the VM

The following instructions describe how to set up a local password for a user on a VM so that the user can log on to the serial console of that VM using the specified password.

  1. Connect to the VM. Replace instance-namewith the name of your instance.
    gcloud compute ssh instance-name
  2. On the VM, create a local password with the following command. This sets a password for the user that you are currently logged in as.
    sudo passwd $(whoami)
  3. Follow the prompts to create a password.
  4. Next, log out of the instance andconnect to the serial console.
  5. Enter your login information when prompted.

Setting up a login on other serial ports

Login prompts are enabled on port 1 by default on most Linux operating systems. However, port 1 can often be overwhelmed by logging data and other information being printed to the port. Instead, you can choose to enable a login prompt on another port, such as port 2 (ttyS1), by executing one of the following commands on your VM. You can see a list of available ports for an VM inUnderstanding serial port numbering.

The following table lists images preconfigured with a serial console login and the default ports.

Operating system Ports with a login prompt by default Service management
CentOS 6 1 upstart
CentOS 7 1 systemd
CoreOS 1 systemd
COS 1 systemd
Debian 8 1 systemd
Debian 9 1 systemd
OpenSUSE 13 1 systemd
OpenSUSE Leap 1 systemd
RHEL 6 1 upstart
RHEL 7 1 systemd
SLES 11 1 sysvinit
SLES 12 1 systemd
Ubuntu 14.04 1 upstart
Ubuntu 16.04 1 systemd
Ubuntu 17.04 1 systemd
Ubuntu 17.10 1 systemd
Windows COM2 N/A

To enable login prompts on additional serial ports, complete the following instructions.

systemd

For Linux operating systems using systemd:

upstart

For Linux operating systems using upstart:

  1. Create a new /etc/init/ttyS1.conf file to reflect ttyS1 by copying and modifying an existing ttyS0.conf file. For example:
    • On Ubuntu 14.04:
      sudo sh -c "sed -e s/ttyS0/ttyS1/g < /etc/init/ttyS0.conf > /etc/init/ttyS1.conf"
    • On RHEL 6.8 and CentOS 6.8
      sudo sh -c "sed -ne '/^# # ttyS0/,/^# exec/p' < /etc/init/serial.conf | sed -e 's/ttyS0/ttyS1/g' -e 's/^# *//' > /etc/init/ttyS1.conf"
  2. Start on a login prompt on ttyS1 without restarting:
    sudo start ttyS1

sysvinit

For Linux operating systems using sysvinit, run the following command:

sudo sed -i~ -e 's/^#T([01])/T\1/' /etc/inittab sudo telinit q

Understanding serial port numbering

Each virtual machine instance has four serial ports. For consistency with thegetSerialPortOutputAPI, each port is numbered 1 through 4. Linux and other similar systems number their serial ports 0 through 3. For example, on many operating system images, the corresponding devices are /dev/ttyS0 through /dev/ttyS3. Windows refers to serial ports as COM1 through COM4. To connect to what Windows considersCOM3 and Linux considers ttyS2, you would specify port 3. Use the following table to help you figure out which port you want to connect to.

Virtual machine instance serial ports Standard Linux serial ports Windows COM ports
1 /dev/ttyS0 COM1
2 /dev/ttyS1 COM2
3 /dev/ttyS2 COM3
4 /dev/ttyS3 COM4

Note that many Linux images use port 1 (/dev/ttyS0) for logging messages from the kernel and system programs.

Sending a serial break

TheMagic SysRq keyfeature lets you perform low-level tasks regardless of the system's state. For example, you can sync file systems, reboot the instance, end processes, and unmount file systems using the Magic SysRq key feature.

To send a Magic SysRq command using a simulated serial break:

  1. Press the ENTER key.
  2. Type ~B (tilde, followed by uppercase B).
  3. Type the Magic SysRq command.

Viewing serial console audit logs

Compute Engine provides audit logs to track who has connected and disconnected from an instance's serial console. To view logs, you must havepermissions for the Logs Vieweror be a project viewer or editor.

  1. In the Google Cloud console, go to the Logs Explorer page.
    Go to Logs Explorer
  2. Expand the drop-down menu and select GCE VM Instance.
  3. In the search bar, type ssh-serialport.googleapis.com and press Enter.
  4. A list of audit logs appears. The logs describe connections and disconnections from a serial console. Expand any of the entries to get more information.

For any of the audit logs, you can:

  1. Expand the protoPayload property.
  2. Look for methodName to see activity this log applies to (either a connection or disconnection request). For example, if this log tracks a disconnection from the serial console, the method name would say"google.ssh-serialport.v1.disconnect". Similarly, a connection log would say "google.ssh-serialport.v1.connect". An audit log entry is recorded at the beginning and end of each session on the serial console.

There are different audit log properties for different log types. For example, audit logs relating to connections have properties that are specific to connection logs, while audit logs for disconnections have their own set of properties. There are certain audit log properties that are also shared between both log types.

All serial console logs

The following table provides audit log properties and their values for all serial console logs:

Property Value
requestMetadata.callerIp The IP address and port number from which the connection originated.
serviceName ssh-serialport.googleapis.com
resourceName A string containing the project ID, zone, instance name, and serial port number to indicate which serial console this pertains to. For example, projects/myproject/zones/us-east1-a/instances/example-instance/SerialPort/2 is port number 2, also known as COM2 or /dev/ttyS1, for the instanceexample-instance.
resource.labels Properties identifying the instance ID, zone, and project ID.
timestamp A timestamp indicating when the session began or ended.
severity NOTICE
operation.id An ID string uniquely identifying the session; you can use this to associate a disconnect entry with the corresponding connection entry.
operation.producer ssh-serialport.googleapis.com

Connection logs

The following table provides audit log properties and their values specific for connection logs:

Property Value
methodName google.ssh-serialport.v1.connect
status.message Connection succeeded.
request.serialConsoleOptions Any optionsthat were specified with the request, including the serial port number.
request.@type type.googleapis.com/google.compute.SerialConsoleSessionBegin
request.username The username specified for this request. This is used to select the public key to match.
operation.first TRUE
status.code For successful connection requests, a status.code value ofgoogle.rpc.Code.OK indicates that the operation completed successfully without any errors. Because the enum value for this property is0, the status.code property is not displayed. However, any code that checks for a status.code value ofgoogle.rpc.Code.OK will work as expected.

Disconnection logs

The following table provides audit log properties and their values specific for disconnection logs:

Property Value
methodName google.ssh-serialport.v1.disconnect
response.duration The amount of time, in seconds, that the session lasted.
response.@type type.googleapis.com/google.compute.SerialConsoleSessionEnd
operation.last TRUE

Failed connection logs

When a connection fails, Compute Engine creates an audit log entry. A failed connection log looks very similar to a successful connection entry, but has the following properties to indicate a failed connection.

Property Value
severity ERROR
status.code Thecanonical Google API error codethat best describes the error. The following are possible error codes that might appear: google.rpc.Code.INVALID_ARGUMENT: The connection failed because the client provided an invalid port number or tried to reach an unknown channel. See the list ofvalid port numbers. google.rpc.Code.PERMISSION_DENIED: You have not enabled interactive serial console in the metadata server. For more information, seeEnabling interactive access on the serial console. google.rpcCode.UNAUTHENTICATED: No SSH keys found or no matching SSH key found for this instance. Check that you areauthenticated to the VM instance. google.rpc.Code.UNKNOWN: There was an unknown error with your request. You can reach out to Google on thegce-discussion groupor file a bug report.
status.message The human-readable message for this entry.

Disabling interactive serial console access

You can disable interactive serial console access by changing metadata on the specific instance or project, or by setting anOrganization Policy that disables interactive serial console access to all VM instances for one or more projects that are part of the organization.

Disabling interactive serial console on a particular instance or project

Project owners and editors, as well as users who have been granted thecompute.instanceAdmin.v1 role, can disable access to the serial console by changing the metadata on the particular instance or project. Similar toenabling serial console access, set the serial-port-enable metadata to FALSE:

serial-port-enable=FALSE

For example, using the Google Cloud CLI, you can apply this metadata to a specific instance like so:

gcloud compute instances add-metadata instance-name
--metadata=serial-port-enable=FALSE

To apply the metadata to the project:

gcloud compute project-info add-metadata
--metadata=serial-port-enable=FALSE

Disabling interactive serial console access through Organization Policy

If you have been granted the orgpolicy.policyAdmin role on the organization, you can set anorganization policythat prevents interactive access to the serial console, regardless of whether interactive serial console access is enabled on the metadata server. After the organization policy is set, the policy effectively overrides theserial-port-enable metadata key, and no users of the organization or project can enable interactive serial console access. By default, this constraint is set to FALSE.

The constraint for disabling interactive serial console access is as follows:

compute.disableSerialPortAccess

Complete the following instructions to set this policy on the organization. After setting up a policy, you can grant exemptions on a per-project basis.

gcloud

To set the policy using the Google Cloud CLI, run theresource-manager enable-enforce command. Replaceorganization-id with yourorganization ID. For example, 1759840282.

gcloud resource-manager org-policies enable-enforce
--organization organization-id compute.disableSerialPortAccess

REST

To set a policy in the API, make a POST request to the following URL. Replace organization-name with yourorganization name. For example, organizations/1759840282.

POST https://cloudresourcemanager.googleapis.com/v1/organization-name:setOrgPolicy

The request body should contain a policy object with the following constraint:

"constraint": "constraints/compute.disableSerialPortAccess"

For example:

{ "policy": { "booleanPolicy": { "enforced": TRUE }, "constraint": "constraints/compute.disableSerialPortAccess" } }

The policy is immediately effective, so any projects under the organization immediately stop allowing interactive access to the serial console.

To temporarily disable the policy, use the disable-enforce command:

gcloud resource-manager org-policies disable-enforce
--organization organization-id compute.disableSerialPortAccess

Alternatively, you can make an API request where the request body sets theenforced parameter to FALSE:

{ "policy": { "booleanPolicy": { "enforced": FALSE }, "constraint": "constraints/compute.disableSerialPortAccess" } }

Setting the organization policy at the project level

You can set the same organizational policy on a per-project basis. This overrides the setting at the organization level.

gcloud

To turn off enforcement of this policy for a specific project. Replaceproject-id with your project ID.

gcloud resource-manager org-policies disable-enforce
--project project-id compute.disableSerialPortAccess

You can turn on enforcement of this policy by using the enable-enforcecommand with the same values.

REST

In the API, make a POST request to the following URL to enable interactive serial console access for the project, replacingproject-id with the project ID:

POST https://cloudresourcemanager.googleapis.com/v1/projects/project-id:setOrgPolicy

The request body should contain a policy object with the following constraint:

"constraint": "constraints/compute.disableSerialPortAccess"

For example:

{ "policy": { "booleanPolicy": { "enforced": FALSE }, "constraint": "constraints/compute.disableSerialPortAccess" } }

Tips and tricks

Advanced options

You can also use the following advanced options with the serial port.

Controlling max connections

You can set the max-connections property to control how many concurrent connections can be made to this serial port at a time. The default and maximum number of connections is 5. For example:

gcloud compute connect-to-serial-port instance-name
--port port-number
--extra-args max-connections=3

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.max-connections=3@ssh-serialport.googleapis.com

Setting replay options

By default, each time you connect to the serial console, you will receive a replay of the last 10 lines of data, regardless of whether the last 10 lines have been seen by another SSH client. You can change this setting and control how many and which lines are returned by setting the following options:

With the Google Cloud CLI, append the following to yourconnect-to-serial-port command, where N is the specified number of lines (or bytes or absolute byte index, depending on which replay option you are selecting):

--extra-args replay-lines=N

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i private-ssh-key-file -p 9600 myproject.us-central1-f.example-instance.jane.port=3.replay-lines=N@ssh-serialport.googleapis.com

You can also use a combination of these options as well. For example:

replay-lines=N and replay-bytes=new

Replay the specified number of lines OR replay all output not previously sent to any client, whichever is larger. The first client to connect with this flag combination will see all the output that has been sent to the serial port, and clients that connect subsquently will only see the lastN lines. Examples:

gcloud compute connect-to-serial-port instance-name--port port-number --extra-args replay-lines=N,replay-bytes=new

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.replay-lines=N.replay-bytes=new@ssh-serialport.googleapis.com

replay-lines=N and replay-bytes=M

Replay lines up to, but not more than, the number of lines or bytes described by these flags, whichever is less. This option won't replay more than N or M bytes.

gcloud compute connect-to-serial-port instance-name--port port-number --extra-args replay-lines=N,replay-bytes=M

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.replay-lines=N.replay-bytes=M@ssh-serialport.googleapis.com

Handling dropped output

The most recent 1 MiB of output for each serial port is always available and generally, your SSH client shouldn't miss any output from the serial port. If, for some reason, your SSH client stops accepting output for a period of time but does not disconnect, and more than 1 MiB of new data is produced, your SSH client might miss some output. When your SSH client is not accepting data fast enough to keep up with the output on the serial console port, you can set the on-dropped-output property to determine how the console behaves.

Set any of the following applicable options with this property:

For example:

gcloud compute connect-to-serial-port instance-name
--port port-number
--extra-args on-dropped-output=ignore

ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.on-dropped-output=ignore@ssh-serialport.googleapis.com

Enabling disconnect using exit or logout commands

You can enable disconnecting on exit or logout commands by setting theon-dtr-low property to disconnect when you connect to the serial console.

On the Google Cloud CLI, append the following flag to yourconnect-to-serial-port command:

--extra-args on-dtr-low=disconnect

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i private-ssh-key-file -p 9600 myproject.us-central1-f.example-instance.jane.port=3.on-dtr-low=disconnect@ssh-serialport.googleapis.com

Enabling the disconnect option might cause your instance to disconnect one or more times when you are rebooting the instance because the operating system resets the serial ports while booting up.

The default setting for the on-dtr-low option is none. If you use the default setting none, you can reboot your instance without being disconnected from the serial console, but the console won't disconnect through normal means such as exit or logout commands, or normal key combinations like Ctrl+D.

What's next