Windows 11/10 and WSL 2 DevOps environment - Windows Server

In this post I describe some of the settings and changes I made to make my Windows 10 and Ubuntu WSL into a fully fledged development environment. Here are the settings and tools I use for DevOps/SysOps on Windows.

Table of Contents show

The settings I describe here make that I can use tools like git, Ansible, Visual Studio Code, Testinfra and OpenSSH connections as good as possible in my workflow. Please share your thoughts, configurations and changes below in the comments, perhaps (likely) I and others can learn from them.

OpenSSH

I have installed Microsoft Win32-OpenSSH (Win32 port of OpenSSH) Client for use with my Yubico YubiKey, as described in “How do I get my YubiKey to work with SSH in Windows 11 and Windows 10?“. Before Win32-OpenSSH version 8.9.1.0p1-Beta I had to rely on Git Bash, which is a great tool, but made things more complicated for me (yet another ssh client installed, upgrade broke OpenSSH FIDO/U2F helper, and so on).

All ssh connections into the network need to go through a bastion host. I added a ProxyJump configuration to my Windows .ssh/config file:

Host bastionhost.example.com
  HostbasedKeyTypes sk-ecdsa-sha2-nistp256@openssh.com,sk-ssh-ed25519@openssh.com
  Compression yes
  User username
  ForwardAgent yes

Host *.example.com
  ProxyJump bastionhost.example.com
  IdentityFile c:/users/username/.ssh/id_ed25519
  IdentityFile /mnt/c/Users/username/.ssh/id_ed25519
  User username
  ForwardAgent yes
Code language: CSS (css)

Once the connection with the bastionhost is created, I can continue using regular ED25519 keys. Here you also see an IdentityFile directive pointing to my Windows .ssh/id_ed25519 file using a Linux path. That is because I share OpenSSH keys with WSL in Windows 10. Doing so there is no need for having separate key files for Windows and Linux/WSL. Or is there?

Did you know you can tunnel RDP connections through SSH? This comes in very handy if your Windows Servers are behind a firewall or bastionhost.

Linux/WSL OpenSSH config

I found that, while working with Visuals Studio Code, git and Ansible/testinfra, it sometimes is more convenient to have a separate ssh config file for Linux/WSL. For example some actions I perform have to go through a SOCKS5 proxy, and I need a ProxyCommand without breaking my current ssh configuration. So I created an additional file: .ssh/wsl_config.

The munged ProxyCommand is:

Host *.example.net
  ProxyCommand ncat.exe --proxy-type socks5 --proxy [::1]:4001 %h %p
  User username
  ForwardAgent yes
Code language: CSS (css)

(note the different TLD “.net” here).

ncat.exe is the Windows netcat variant as found in nmap. You need to install this in Windows, not Linux/WSL. “Why no ProxyJump for this as well?”, I hear you ask. Well, when I run Testinfra tests with ProxyJump, I have to log on (pad the YubiKey) multiple times per test. That sucks :)

To support YubiKey OTP authentication, I added the following to my .bashrc file to create an environment variable:

export SSH_SK_HELPER="/mnt/c/Program Files/OpenSSH/ssh-sk-helper.exe"
Code language: JavaScript (javascript)

And the IdentityFile directive for my private keys:

Host bastionhost.example.com
  IdentityFile /mnt/c/Users/username/.ssh/id_ecdsa_sk_1
  IdentityFile /mnt/c/Users/username/.ssh/id_ecdsa_sk_2
  Compression yes
  User username
  ForwardAgent yes

If you’ve added this and try to log onto bastionhost.example.com from within WSL, you notice the WinHello pop-up asking you to tap your YubiKey.

Sweet! :)

Reuse SSH connections

If you connect and disconnect a lot, for example you’re git pull‘ing various repos, or running Testinfra tests, might benefit from keeping a connection open and reusing it. Unfortunately OpenSSH directives like ControlMaster, ControlPath and ControlPersist are not supported in Windows. But they are in WSL.

Add to your .ssl/wsl_config file, under the Host directive it concerns:

Host bastionhost.example.com
  ControlMaster    auto
  ControlPath   ~/tmp/%r@%h:%p.sock
  ControlPersist 60s
Code language: JavaScript (javascript)

Don’t use ~/.ssh/ as your ControlPath if that folder is symlinked to Windows. For example if you share OpenSSH keys with WSL and Windows. Make sure ~/tmp exists.

ControlPersist 60s makes sure your connection will be kept open for at most 1 minute. Use ControlPersist Yes to keep it open until server will close the connection on its timeout.

Using this little piece of configuration prevents you having to touch / tap your YubiKey twenty times in a row while pulling in 20 repos or running multiple Testinfra tests, because you are reusing a single ssh connection. Neat :)

Keychain ssh-agent – Using SSH-Agent correctly in WSL 2

Partially taken from esc.sh’s post Using SSH-Agent the right way in Windows 10 WSL2.

Unfortunately you cannot integrate your Windows ssh-agent into WSL, and in WSL you’ll notice an agent session does not persist when you open a new terminal window. Long story short, install and use the package keychain.

sudo apt-get install keychain
Code language: JavaScript (javascript)

If necessary, you can edit your ~/.bashrc and add:

/usr/bin/keychain -q --nogui $HOME/.ssh/id_ecdsa_sk_1
/usr/bin/keychain -q --nogui $HOME/.ssh/id_ecdsa_sk_2
source $HOME/.keychain/$HOSTNAME-sh
Code language: PHP (php)

When using -q there is no keychain greeting when you open a new shell.

Git

My local code repos are in WSL, it’s just easier. Why? Well, I started doing that since we had double colon (::) in file- and folder names in Puppet, something NTFS just won’t understand. And running Testinfra tests from within Visual Studio Code requires having your repo there.

Git needs some TLC, because it has to work with your newly created WSL OpenSSH config. If you don’t, git/WSL tries to use your Windows .ssh/config file, which lacks the ProxyCommand.

Add to your .bashrc file:

export GIT_SSH_COMMAND="ssh -F ~/.ssh/wsl_config"
Code language: JavaScript (javascript)

This tells git to always use this specific ssh command, and the -F argument tells ssh to use this specific config file. Neat!

Python

For Python I didn’t have to make a lot of adjustments, only that I didn’t want __pycache__ files everywhere / everytime. For this I added to .bashrc:

export PYTHONDONTWRITEBYTECODE=1
Code language: JavaScript (javascript)

Testinfra

With Testinfra you can write unit tests in Python to test actual state of your servers configured by management tools like Salt, Ansible, Puppet, Chef and so on. For use in Visual Studio Code, the path to py.test and pytest need to be known (e.g in the path environment variable):

export PATH="$PATH:~/.local/bin"
Code language: JavaScript (javascript)

If you want to run a Testinfra test from within Visual Studio Code Remote, I found I have to provide the full path to my ssh config (--ssh-config=/home/username/.ssh/wsl_config) and prefix the hosts argument with ssh:// as protocol:

py.test --ssh-config=/home/username/.ssh/wsl_config tests.py --hosts="ssh://dbserver-01.dc03.example.net" -v --sudo 
Code language: JavaScript (javascript)

This took some trial and error.

CloudMonkey

cloudmonkey ☁️? is a command line interface (CLI) for Apache CloudStack. CloudMonkey can be use both as an interactive shell and as a command line tool which simplifies Apache CloudStack configuration and management.

In (regular) Linux/Ubuntu you can install CloudMonkey using snap (snap is available by default in Ubuntu 16.04 and up):

$ sudo snap install cloudmonkey

But unfortunately, this doesn’t work with WSL, as snap install fails. Luckily you can download standalone binaries and use CloudMonkey in Windows and Linux:

and this results in:

# WSL 2
$ /bin/cmk version
Apache CloudStack ? CloudMonkey 6.2.0

# Windows
PS C:\Users\janreilink> cmk.windows.x86-64.exe
Apache CloudStack ? CloudMonkey 6.2.0
Report issues: https://github.com/apache/cloudstack-cloudmonkey/issues

(localcloud) cmk > version
Apache CloudStack ? CloudMonkey 6.2.0
(localcloud) cmk > exit
Code language: PHP (php)

Packer

Packer, by HashiCorp, lets you create identical machine images for multiple platforms from a single source configuration. Just download the required .zip file and launch packer from your command line.

Visual Studio Code

Visual Studio Code (“VS Code”) is my editor / IDE of choice. It supports remote connections to Windows Subsystem for Linux (WSL). You can easiliy open a Linux/WSL folder using the command: code . (this opens up the current folder in Visual Studio Code).

You can use \\wsl$ as UNC path in Windows to access your Linux/WSL files from Windows Explorer.

VS Code Extensions

In VS Code, I have the following extensions installed:

Remote – WSL, or I can’t connect with WSL
Ansible
docs-yaml
GitLens – Git supercharged
IIS Express (I do some dotnet/IIS stuff too sometimes – Umbraco, IIS)
JSON Tools
Puppet
Pylance
Python
ShellCheck
XML Tools
YAML

More OpenSSH in Windows Server and Windows 11 / Windows 10, the series

Here on Sysadmins of the North are more posts in a series of posts about OpenSSH in Windows. Whether it’s Windows Server or Windows 11 / 10. You may find these posts interesting:

I hope you like it, let me know.

Conclusion and giveaway

In this post I showed you how you can turn Windows 11 and Windows 10 into a fully fledged development environment. It may take some getting used to, but if you’re as shortcut-Ninja as I’m (I alt-tab, ctrl-c/ctrl-v a lot), you can work pretty fast in this set up. Just don’t forget to run sudo apt update and sudo apt upgrade to keep Ubuntu updated.