Safe shell (SSH) is a regular method of accessing server infrastructure within the IT business. In contrast, no single commonplace resolution exists for managing consumer SSH entry in an automatic and safe method. Allow us to discover how we tackled this problem at GoodData.

Our Necessities

Given the various and globally distributed platform infrastructure we function at GoodData, safe shell (SSH) is a core method of accessing our techniques in an effort to carry out obligatory operations and debugging. It’s subsequently very important for us to have an SSH authorization technique that’s:

• safe in accordance with the most recent business requirements;
• sturdy in its efficiency; and
• straightforward to make use of for our staff.

Having utilized FreeIPA to offer SSH entry administration for a number of years, a strategic determination was made in late 2020 to modify to Okta because the company-wide identification administration resolution. This transition introduced us with a chance to refresh our entry management setup and guarantee its consistency with present safety requirements.

Trying to find a Resolution

We recognized inadequate scalability as the first shortcoming of our earlier SSH setup. With FreeIPA, each SSH login try requires the consumer’s public SSH key to be verified towards the server, rendering FreeIPA a single level of failure and stopping any consumer logins to focus on machines in case our FreeIPA servers skilled an outage or upkeep.

We got down to discover and implement an answer that will keep away from this pitfall; one that will be sturdy sufficient to help day by day operations on the dimensions we require, i.e. managing entry to a whole bunch of servers distributed throughout the globe.

Moreover, deprecating long-lived SSH credentials from the ecosystem was one other objective we deemed vital. There was an inherent safety danger stemming from static SSH keys being current on staff’ units, as they might probably be stolen and used for entry to our infrastructure. Whereas we had partially mitigated this danger utilizing multi-factor authentication, we have been striving for a perfect resolution that will solely depend on short-lived, dynamic credentials. This fashion, any leaked credential wouldn’t pose a safety danger, as a result of it could expire in a short time.

After intensive analysis, we chosen HashiCorp Vault to offer SSH entry administration by way of its dynamic SSH certificates engine.

Technical design

Since Vault had already been current in our infrastructure as a retailer for static key-value credentials, extending its function by including the SSH certificates signing backend was pretty simple.

Consumer Authentication

For consumer entry, Vault has been built-in with Okta utilizing the OIDC authentication technique. We outlined a number of consumer login roles similar to various ranges of entry granted to distinct teams of customers; every function is tied to a particular OIDC declare worth offered by Okta. Each Vault login function, in flip, permits signing SSH certificates by a corresponding SSH backend function. The SSH certificates key ID configuration ensures that consumer function info is preserved within the ensuing signed certificates.

For instance, if a consumer first.final belongs to group1, they’ll login to Vault utilizing login function group1. Vault coverage will even enable them to signal a certificates utilizing SSH function group1, and the ensuing certificates’s key ID shall be okta-first.final:group1.

SSH Consumer Provisioning

In absence of a centralized SSH authorization server, a problem stays: the best way to decide which customers are allowed to hook up with a particular server? To resolve this, we’ve outlined two distinct SSH login eventualities we wanted to help:

1. login to externally uncovered soar stations; and
2. additional hops to internally accessible servers.

On soar stations, we require the customers to log in below a private non-privileged account, in an effort to present a secure atmosphere to entry techniques utilizing private credentials (similar to AWS IAM Id Middle logins or studying secrets and techniques from Vault itself). Subsequently, a easy Python script is periodically run on the soar station servers to fetch a listing of related customers from Okta API and create a system account for every of them.

On all different servers, a single shared consumer account is adequate for all customers logging in by way of SSH; subsequently, no periodic consumer synchronization from Okta is required.

By this splitting of use circumstances, the necessity to ship Okta credentials is proscribed to a handful of soar station nodes whereas we nonetheless hold the extent of consumer entry separation we require.

Consumer Authorization

The method of authorizing customers makes use of the sshd AuthorizedPrincipalsCommand directive; three items of setup are pre-delivered by our configuration administration device to facilitate this:

• Vault SSH certificates authority public key;
• checklist of Vault SSH roles allowed to log in to a given server; and
• an authorization script to be referred to as upon consumer login.

When a consumer makes an attempt to log in to a server, AuthorizedPrincipalsCommand known as, with the %i (certificates’s key ID) and %u (username) parameters offered to our authorization script. The script parses the consumer’s Vault function from the important thing ID (making use of its static format described above) and compares it towards the checklist of roles allowed to log in. On soar stations, it moreover verifies whether or not the consumer is logging in below their very own private account identify. A second-factor verification can be hooked into the login course of for the soar stations case.

Advantages and Classes

Utilizing Vault’s SSH certificates engine, we’ve been capable of transition to a way more sturdy and safe login system than earlier than. Customers throughout the corporate respect the system’s stability; since authorization is finished with info statically accessible to SSH servers and no reside communication with Vault is required at login time, the consumer expertise is far smoother than beforehand. Safety has additionally been improved by eliminating static, long-lived SSH keys in favor of shortly expiring SSH certificates.

We’ve got encountered minor challenges with the brand new setup. Having a shared system account for all customers on a majority of servers implies that auditing consumer actions is barely more durable; we’ve labored round this by parsing the certificates key ID into an atmosphere variable upon login. Instructions similar to who should not helpful since introducing the brand new setup. Retraining customers to the brand new Vault-based SSH login workflow was an preliminary hurdle we’ve been capable of overcome with time.

Total, although, we conclude that the brand new method of doing SSH at GoodData gives extra safety and a greater consumer expertise than the previous resolution based mostly on static SSH keys.