QA Regression Testing

Merge ~ahasenack/qa-regression-testing:qat-openssh-fixes-and-fido-support into qa-regression-testing:master

  1. Git
  2. lp:~ahasenack/qa-regression-testing
  3. qat-openssh-fixes-and-fido-support
  4. Merge into master
Proposed by Andreas Hasenack
Status: Merged
Merged at revision: 00699043c116b02e6f60ecef2e5289c7caff907d
Proposed branch: ~ahasenack/qa-regression-testing:qat-openssh-fixes-and-fido-support
Merge into: qa-regression-testing:master
Diff against target: 396 lines (+309/-8)
2 files modified
notes_testing/openssh/README.txt (+165/-0)
scripts/test-openssh.py (+144/-8)
Reviewer Review Type Date Requested Status
Steve Beattie Approve
Review via email: mp+381582@code.launchpad.net

Description of the change

Tests for openssh with U2F support.

This is the final requirement for the libfido/libcbor MIR[1], as agreed to in [2].

Instructions were added to notes_testing/openssh/README.txt

I was just able to add basically 2 automated tests. One generates the key (and is not a unit test, because I need that key for the next test), and the other uses it for a ssh localhost check. I would have loved to add tests to verify ssh fails if the device isn't present, but all this interactivity doesn't work well with the unittest framework. I would have to keep asking the user stuff like "Please remove the hw key now", then check that it was removed, then move on to trying ssh, and then ask for the key to be put back it for the next test in the suite. I accept suggestions here if you think this should be improved.

That being said, manual testing isn't that hard, and I also contemplated that in the README.txt document.

Please let me know what you think, or if something is too confusing and needs clarification, or could be done in a better way.

Note I also had to fix the openssh tests, as a few were expecting a password-based authentication. These changes are in separate commits.

1. https://bugs.launchpad.net/ubuntu/+source/libfido2/+bug/1864439
2. https://bugs.launchpad.net/ubuntu/+source/libfido2/+bug/1864439/comments/24

To post a comment you must log in.
Revision history for this message
Seth Arnold (seth-arnold) wrote :

Thanks Andreas, looks good to me!

Revision history for this message
Steve Beattie (sbeattie) wrote :

Hey Andreas, looked good to me, the only change I made was to move the added install dependency of yubikey-manager to the alternates section, so that the install wouldn't fail on older releases where that package does not exist.

Thanks!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/notes_testing/openssh/README.txt b/notes_testing/openssh/README.txt
2new file mode 100644
3index 0000000..37acc80
4--- /dev/null
5+++ b/notes_testing/openssh/README.txt
6@@ -0,0 +1,165 @@
7+OpenSSH U2F tests
8+=================
9+
10+Since focal and openssh 8.2p1, yubikeys can be used as 2fa hardware devices for
11+ssh key authentication. Two new key types have been added for this purpose:
12+ecdsa-sk and ed25519-sk. Not all hardware devices support both, with ecdsa-sk
13+being more common.
14+
15+The test-openssh.py script has been modified to also test openssh with u2f
16+devices, but not by default, since some preparation is required. Thankfully the
17+yubikeys are basically plug and play, the only quirk being in how to pass it
18+through to the test environment. The instructions here are about using a lxd
19+container for these tests.
20+
21+Locating and passing the hidraw devices
22+---------------------------------------
23+When you insert a yubikey in your machine (host), two new hidraw devices will appear.
24+
25+First list the devices you already have, before inserting the key. In this example, I have none:
26+$ ls -la /dev/hidraw*
27+ls: cannot access '/dev/hidraw*': No such file or directory
28+
29+Then insert the device, and repeat:
30+$ ls -la /dev/hidraw*
31+crw-rw----+ 1 root plugdev 241, 0 abr 2 09:54 /dev/hidraw0
32+crw-rw----+ 1 root plugdev 241, 1 abr 2 09:54 /dev/hidraw1
33+
34+These are the devices we need to pass to the lxd container.
35+
36+Spawn a container as usual. For example:
37+$ lxc launch ubuntu-daily:focal f1
38+Creating f1
39+Starting f1
40+
41+Now pass the devices to the container:
42+$ lxc config device add f1 u2fhi0 unix-char path=/dev/hidraw0
43+Device u2fhi0 added to f1
44+$ lxc config device add f1 u2fhi1 unix-char path=/dev/hidraw1
45+Device u2fhi1 added to f1
46+
47+The names "u2fhi0" and "u2fhi1" do not really matter, as long as they are unique.
48+
49+Finally, enter the container (ssh if you have setup keys already, or lxc exec):
50+$ lxc exec f1 bash
51+root@f1:~# sudo -u ubuntu -i
52+To run a command as administrator (user "root"), use "sudo <command>".
53+See "man sudo_root" for details.
54+
55+Check if the devices are there:
56+$ ls -la /dev/hidraw*
57+crw-rw---- 1 root root 241, 0 Apr 2 12:56 /dev/hidraw0
58+crw-rw---- 1 root root 241, 1 Apr 2 12:56 /dev/hidraw1
59+
60+Note their permissions are slightly different. On the host, acls are used to grant the console user rw access to them. In the lxd, only root has access.
61+
62+To quickly check if the device is really visible, install yubikey-manager and run "sudo ykman info":
63+$ sudo ykman info
64+Device type: YubiKey 4
65+Serial number: XXXXXXX
66+Firmware version: 4.3.7
67+Enabled USB interfaces: OTP+FIDO+CCID
68+
69+Applications
70+OTP Enabled
71+FIDO U2F Enabled
72+OpenPGP Enabled
73+PIV Enabled
74+OATH Enabled
75+FIDO2 Not available
76+
77+
78+Running the U2F tests
79+---------------------
80+Since the U2F tests are interactive, as they require the device to be touched
81+when it's to be used, they have been moved to their own test class, and need to
82+be explicitly requested with the "--u2f" command line option.
83+
84+Without the "--u2f" option, a warning will be printed reminding the user of
85+their existence, and the normal openssh test suite will run.
86+
87+$ cd scripts
88+$ sudo ./install-packages test-openssh.py
89+...
90+$ sudo ./test-openssh.py --u2f
91+Skipping private tests
92+
93+Running U2F tests *ONLY*
94+
95+Please make sure you have plugged in your yubikey
96+Verifying if yubikey is visible...
97+Yubikey found! Starting the tests.
98+
99+Generating a u2f keypair for the tests
100+Please touch the device after it starts blinking
101+Done
102+test_ssh_with_hw_key_present (__main__.OpenSSHTestU2F)
103+Test ssh with the hardware present, touch when blinking ... ok
104+
105+----------------------------------------------------------------------
106+Ran 1 test in 14.314s
107+
108+OK
109+
110+
111+Manual testing
112+--------------
113+If you want, you can also manually test the U2F support quite easily:
114+
115+a) Create a key suitable for U2F (optionally use a passphrase if you want):
116+ubuntu@focal-openssh-client:~$ ssh-keygen -t ecdsa-sk
117+Generating public/private ecdsa-sk key pair.
118+You may need to touch your authenticator to authorize key generation.
119+Enter file in which to save the key (/home/andreas/.ssh/id_ecdsa_sk):
120+Enter passphrase (empty for no passphrase):
121+Enter same passphrase again:
122+Your identification has been saved in /home/andreas/.ssh/id_ecdsa_sk
123+Your public key has been saved in /home/andreas/.ssh/id_ecdsa_sk.pub
124+The key fingerprint is:
125+SHA256:bS6vX6b+Bp8Xu/LF4Gw10dV0Y6AkjFPjPoO5q0A546M andreas@nsnx
126+…
127+
128+This produces the normal two files representing the keypair:
129+ubuntu@focal-openssh-client:~$ l .ssh/id_ecdsa_sk*
130+-rw------- 1 andreas andreas 610 mar 30 17:58 .ssh/id_ecdsa_sk
131+-rw-r--r-- 1 andreas andreas 221 mar 30 17:58 .ssh/id_ecdsa_sk.pub
132+
133+
134+b) Copy the public key part ~/.ssh/id_ecdsa_sk.pub over to the server you want
135+to login, and store it in ~/.ssh/authorized_keys as usual
136+
137+On the server you end up with something like this:
138+ubuntu@focal-openssh-server:~$ cat ~/.ssh/authorized_keys
139+sk-ecdsa-sha2-nistp256@openssh.com AAAAInNrLWVjZHNhLXNoYTItbmlzdHAyNTZAb3BlbnNzaC5jb20AAAAIbmlzdHAyNTYAAABBBE4OP/pzi7CtVQg+ACCa3VGJ+ap2pOzBAekN1nz+febebdioKSsFCaAca1LrlLg++++9FKuIDcdOTdrXLaJupgYAAAAEc3NoOg== andreas@nsnx
140+
141+
142+c) Use the key to login. Depending if you have ssh-agent setup via gnome, or if
143+bug LP: #1869897 is not yet fixed, it may look like it's stalled:
144+
145+ubuntu@focal-openssh-client:~$ ssh -i .ssh/id_ecdsa_sk 10.0.100.75
146+<hangs> ← physically touch the hardware key at this point
147+(...)
148+Welcome to Ubuntu Focal Fossa (development branch) (GNU/Linux 5.4.0-18-generic x86_64)
149+(...)
150+Last login: Mon Mar 30 20:29:05 2020 from 10.0.100.1
151+ubuntu@focal-openssh-server:~$
152+
153+If no gnome ssh-agent is involved, of if that bug was fixed, you will get a
154+prompt to touch the device:
155+
156+ubuntu@focal-openssh-client:~$ SSH_AUTH_SOCK= ssh -i .ssh/id_ecdsa_sk 10.0.100.75
157+Confirm user presence for key ECDSA-SK SHA256:bS6vX6b+Bp8Xu/LF4Gw10dV0Y6AkjFPjPoO5q0A546M
158+(...)
159+Welcome to Ubuntu Focal Fossa (development branch) (GNU/Linux 5.4.0-18-generic x86_64)
160+(...)
161+Last login: Mon Mar 30 20:39:05 2020 from 10.0.100.1
162+ubuntu@focal-openssh-server:~$
163+
164+
165+d) Try again, without the yubikey inserted, and it will fail immediately
166+ubuntu@focal-openssh-client:~$ ssh -i .ssh/id_ecdsa_sk 10.0.100.75
167+Confirm user presence for key ECDSA-SK SHA256:bS6vX6b+Bp8Xu/LF4Gw10dV0Y6AkjFPjPoO5q0A546M
168+sign_and_send_pubkey: signing failed for ECDSA-SK ".ssh/id_ecdsa_sk": invalid format
169+ubuntu@focal-openssh-server: Permission denied (publickey).
170+
171+This test (d) is not done in the test suite because it would have to be the last test.
172diff --git a/scripts/test-openssh.py b/scripts/test-openssh.py
173index d8b5ea1..748a2f4 100755
174--- a/scripts/test-openssh.py
175+++ b/scripts/test-openssh.py
176@@ -1,4 +1,4 @@
177-#!/usr/bin/python
178+#!/usr/bin/python2
179 #
180 # test-openssh.py quality assurance test script for PKG
181 # Copyright (C) 2010-2018 Canonical Ltd.
182@@ -18,7 +18,7 @@
183 # along with this program. If not, see <http://www.gnu.org/licenses/>.
184 #
185 # packages required for test to run:
186-# QRT-Packages: openssh-server openssh-client python-pexpect
187+# QRT-Packages: openssh-server openssh-client python-pexpect yubikey-manager
188 # packages where more than one package can satisfy a runtime requirement:
189 # QRT-Alternates:
190 # files and directories required for the test to run:
191@@ -32,8 +32,17 @@
192 these tests non-destructive, there is no guarantee this script will not
193 alter the machine. You have been warned.
194
195+ To test U2F authentication, passthrough the hidraw devices to the lxd container.
196+ Find out which hidraw* devices show up when you plug the key in, and pass those
197+ through like this (here I used 0 and 1):
198+
199+ lxc config device add <container-name> u2fhi0 unix-char path=/dev/hidraw0
200+ lxc config device add <container-name> u2fhi1 unix-char path=/dev/hidraw1
201+
202 '''
203 from __future__ import print_function
204+from shutil import rmtree
205+from tempfile import mkdtemp
206
207 import unittest, sys, os
208 import testlib
209@@ -49,8 +58,9 @@ except ImportError:
210 use_private = False
211 print("Skipping private tests", file=sys.stdout)
212
213-class OpenSSHTest(testlib.TestlibCase):
214- '''Test both openssh server and client.'''
215+
216+class BaseOpenSSHTest(testlib.TestlibCase):
217+ '''Base openssh test class'''
218
219 def setUp(self):
220 '''Set up prior to each test_* function'''
221@@ -92,7 +102,8 @@ ITfOx9KgntLukRe860E+CbkBxEhPD+2+GhtXL0d21o4JoS/YQb80
222 testlib.config_replace(self.sshd_rsa_private_keyfile, self.sshd_rsa_private_key)
223 testlib.config_replace(self.sshd_rsa_private_keyfile + ".pub", self.sshd_rsa_public_key)
224
225- self._restart_daemon()
226+ # already restarts the daemon
227+ self.modify_sshd_config({"PasswordAuthentication": "yes"})
228
229 # create a user to log in via ssh
230 self.user = testlib.TestUser()
231@@ -100,6 +111,7 @@ ITfOx9KgntLukRe860E+CbkBxEhPD+2+GhtXL0d21o4JoS/YQb80
232 self._create_ssh_dir(self.user)
233 self._create_ssh_dir(self.other_user)
234
235+
236 def _create_ssh_dir(self, user):
237 '''Creates a .ssh dir'''
238 os.mkdir(user.home + "/.ssh", 0o700)
239@@ -188,7 +200,6 @@ ITfOx9KgntLukRe860E+CbkBxEhPD+2+GhtXL0d21o4JoS/YQb80
240 self.assertEqual(expected, rc, out)
241 self.assertEqual(cmp_out.splitlines(), out, out)
242
243-
244 def tearDown(self):
245 '''Clean up after each test_* function'''
246
247@@ -200,6 +211,10 @@ ITfOx9KgntLukRe860E+CbkBxEhPD+2+GhtXL0d21o4JoS/YQb80
248 self.user = None
249 self.other_user = None
250
251+
252+class OpenSSHTest(BaseOpenSSHTest):
253+ '''Test both openssh server and client.'''
254+
255 def test_00_sshd_listening(self):
256 '''Test to ensure ssh is running'''
257 self.assertTrue(testlib.check_port(22, 'tcp'))
258@@ -445,15 +460,136 @@ ITfOx9KgntLukRe860E+CbkBxEhPD+2+GhtXL0d21o4JoS/YQb80
259 self.assertFalse("Roaming not allowed by server" in out,
260 "Roaming isn't disabled in '%s'" % out)
261
262+
263+class OpenSSHTestU2F(BaseOpenSSHTest):
264+ '''U2F interactive tests for OpenSSH'''
265+ # This requires:
266+ # - openssh 8.2 or higher
267+ # - a u2f device plugged in and passed through the test env (lxc or vm)
268+ # - it's an interactive test: the user will have to touch the device
269+
270+ @classmethod
271+ def setUpClass(cls):
272+ '''Generate the u2f keypair used in the tests.'''
273+ cls.u2f_keypair_path = mkdtemp()
274+ cls.u2f_keypair = generate_u2f_key(cls.u2f_keypair_path)
275+
276+ @classmethod
277+ def tearDownClass(cls):
278+ # remove temporary directory for keys
279+ rmtree(cls.u2f_keypair_path)
280+
281+ def test_ssh_with_hw_key_present(self):
282+ '''Test ssh with the hardware present, touch when blinking'''
283+ # copy public key over to the target user
284+ authorized_keys_file = self.user.home + "/.ssh/authorized_keys"
285+ shutil.copy2(self.u2f_keypair["public_key_file"], authorized_keys_file)
286+ os.chown(authorized_keys_file, self.user.uid, self.user.gid)
287+ # Can't easily use the random user created in setUp(), because that user
288+ # needs to have rw access to the /dev/hidraw* devices.
289+ # Figuring out which hidraw* devices belong to the yubikey, so we
290+ # could adjust their permissions with ACLs, got complicated/
291+ # So, let's just call ssh as root, and ignore the host fingerprint
292+ # that setUp(), via _create_ssh_dir(), so nicely created for us
293+ command = ["ssh", "-i", self.u2f_keypair["private_key_file"],
294+ "-o", "StrictHostKeyChecking=no",
295+ "-o", "UserKnownHostsFile=/dev/null",
296+ "{}@localhost".format(self.user.login),
297+ "cat", "/etc/lsb-release"]
298+ child = pexpect.spawn(command.pop(0), command, timeout=5)
299+ expected = 'Confirm user presence for key ECDSA-SK {}'.format(
300+ self.u2f_keypair["fingerprint"])
301+ try:
302+ child.expect(expected)
303+ except pexpect.TIMEOUT:
304+ self.fail("Didn't get expected '{}' message".format(expected))
305+ ssh_out = [line.strip('\r\n') for line in child.readlines()]
306+ ssh_out.pop(0)
307+ child.close()
308+ rc, local_out = self.shell_cmd(['cat', '/etc/lsb-release'])
309+ self.assertEqual(0, rc, local_out)
310+ # if local_out differs from what we got via ssh, show the ssh
311+ # output
312+ self.assertEqual(local_out.splitlines(), ssh_out)
313+
314+
315+def generate_u2f_key(path):
316+ """
317+ Interactively generates a u2f key with ssk-keygen.
318+ This also serves to test key generation with the hardware device.
319+
320+ Returns a dict:
321+ {"private_key_file": path, "public_key_file": path,
322+ "fingerprint": fingerprint}
323+ """
324+ print("Generating a u2f keypair for the tests")
325+ print("Please touch the device after it starts blinking")
326+ command = ['ssh-keygen', '-t', 'ecdsa-sk', '-N', '', '-f',
327+ '{}/id_ecdsa_sk'.format(path)]
328+ child = pexpect.spawn(command.pop(0), command, timeout=5)
329+ expected = 'You may need to touch your authenticator to authorize key generation.'
330+ try:
331+ ret = child.expect(expected)
332+ except pexpect.TIMEOUT:
333+ sys.exit("Didn't get expected '{}' message".format(expected))
334+ try:
335+ output = [line.strip('\r\n') for line in child.readlines()]
336+ except pexpect.TIMEOUT:
337+ sys.exit("Device wasn't touched")
338+ child.close()
339+ private_key_file = None
340+ public_key_file = None
341+ fingerprint = None
342+ for line in output:
343+ if line.startswith("Your identification has been saved in"):
344+ private_key_file = line.split(" ")[-1]
345+ if line.startswith("Your public key has been saved in"):
346+ public_key_file = line.split(" ")[-1]
347+ if line.startswith("SHA256:"):
348+ fingerprint = line.split(" ")[0]
349+ if not all([private_key_file, public_key_file]):
350+ sys.exit("u2f keypair generation failed")
351+ if not all(
352+ [os.path.exists(private_key_file), os.path.exists(public_key_file)]):
353+ sys.exit("u2f keypair generation failed")
354+ print("Done")
355+ return {"private_key_file": private_key_file,
356+ "public_key_file": public_key_file,
357+ "fingerprint": fingerprint}
358+
359+
360 if __name__ == '__main__':
361- # more configurable
362+ testlib.require_sudo()
363 suite = unittest.TestSuite()
364- suite.addTest(unittest.TestLoader().loadTestsFromTestCase(OpenSSHTest))
365+ u2f = False
366+ if len(sys.argv) > 1 and sys.argv[1] == '--u2f':
367+ u2f = True
368+ if u2f:
369+ print("\nRunning U2F tests *ONLY*\n")
370+ print("Please make sure you have plugged in your yubikey")
371+ print("Verifying if yubikey is visible...")
372+ rc, out = testlib.cmd(['ykman', 'info'])
373+ if rc != 0:
374+ print("\nNo yubikey found!\n")
375+ print("In LXD you need to pass through the hidraw* devices from the host")
376+ print("\nExample:")
377+ print("lxc config device add <container-name> u2fhi0 unix-char path=/dev/hidraw0")
378+ print("lxc config device add <container-name> u2fhi1 unix-char path=/dev/hidraw1")
379+ print("\nRun 'sudo ykman info' to verify\n")
380+ sys.exit(1)
381+ else:
382+ print("Yubikey found! Starting the tests.\n")
383+ suite.addTest(unittest.TestLoader().loadTestsFromTestCase(OpenSSHTestU2F))
384+ else:
385+ print("\nWARNING: Skipping interactive U2F tests, use --u2f to run them\n")
386+ suite.addTest(unittest.TestLoader().loadTestsFromTestCase(OpenSSHTest))
387
388 # Pull in private tests
389 if use_private:
390 suite.addTest(unittest.TestLoader().loadTestsFromTestCase(PrivateOpenSSHTest))
391
392 rc = unittest.TextTestRunner(verbosity=2).run(suite)
393+ if not u2f:
394+ print("\nPlease remember to also run the U2F tests with --u2f\n")
395 if not rc.wasSuccessful():
396 sys.exit(1)

Subscribers

People subscribed via source and target branches