Launchpad itself

Merge lp:~adeuring/launchpad/hwdb-build-udev-device-list into lp:launchpad

  1. hwdb-build-udev-device-list
  2. Merge into devel
Proposed by Abel Deuring
Status: Merged
Approved by: Brad Crittenden
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~adeuring/launchpad/hwdb-build-udev-device-list
Merge into: lp:launchpad
Diff against target: 248 lines
2 files modified
lib/canonical/launchpad/scripts/hwdbsubmissions.py (+51/-0)
lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py (+155/-1)
To merge this branch: bzr merge lp:~adeuring/launchpad/hwdb-build-udev-device-list
Reviewer Review Type Date Requested Status
Brad Crittenden (community) code Approve
Review via email: mp+13644@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

This branch adds a new method SubmissionParser.buildUdevDeviceList() for the HWDB submission processing script.

Hardware devices mentioned in a given submission are represented by instances of class UdevDevice. The new method SubmissionParser.buildUdevDeviceList() gets the data found in a submission in a dictionary parsed_data and uses it it to create UdevDevice instances. parsed_data contains, among other things, the elements parsed_data['hardware']['udev'], parsed_data['hardware']['sysfs-attributes'] and parsed_data['hardware']['dmi'].

The udev data is our main source for imformations about devices; in some cases we need additional data from sysfs-attributes and from dmi. parsed_data['hardware']['dmi'] contains data relevant for the root node of a device; parsed_data['hardware']['sysfs-attributes'] is a dictionary containing more data for some devices mentioned in the udev data.

The udev data describes a hierachy of devices (like "A USB controller and an Ethernet interface are connected to the main system; the USB controller is connected to a USB hub, which in turn is connected to a USB printer") by the path names of the udev nodes. Since we store the device hierachy of the devices in a given submission in the HWDB, UdevDevice has an attribute "children", which should contain, well, the children of a given node.

I wanted to use the method assertIsNone in one or two of the tests; in order to do that, I had to change the import of class TestCase from unittest to lp.testing.

tests: ./bin/test --test=test_hwdb_submission_processing

= Launchpad lint =

Checking for conflicts. and issues in doctests and templates.
Running jslint, xmllint, pyflakes, and pylint.
Using normal rules.

Linting changed files:
  lib/canonical/launchpad/scripts/hwdbsubmissions.py
  lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py

== Pyflakes notices ==

lib/canonical/launchpad/scripts/hwdbsubmissions.py
    22: redefinition of unused 'etree' from line 20

== Pylint notices ==

lib/canonical/launchpad/scripts/hwdbsubmissions.py
    20: [F0401] Unable to import 'xml.etree.cElementTree' (No module named etree)

This complaint is not related to my changes.

Revision history for this message
Brad Crittenden (bac) wrote :

This branch looks good Abel. Thanks.

review: Approve (code)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/launchpad/scripts/hwdbsubmissions.py'
2--- lib/canonical/launchpad/scripts/hwdbsubmissions.py 2009-10-19 13:49:04 +0000
3+++ lib/canonical/launchpad/scripts/hwdbsubmissions.py 2009-10-20 14:10:34 +0000
4@@ -1407,6 +1407,57 @@
5 if parent_udi is not None:
6 hal_devices[parent_udi].addChild(device)
7
8+ def buildUdevDeviceList(self, parsed_data):
9+ """Create a list of devices from the udev data of a submission."""
10+ self.devices = devices = {}
11+ sysfs_data = parsed_data['hardware']['sysfs-attributes']
12+ dmi_data = parsed_data['hardware']['dmi']
13+ for udev_data in parsed_data['hardware']['udev']:
14+ device_path = udev_data['P']
15+ if device_path == UDEV_ROOT_PATH:
16+ device = UdevDevice(
17+ self, udev_data, sysfs_data=sysfs_data.get(device_path),
18+ dmi_data=dmi_data)
19+ else:
20+ device = UdevDevice(
21+ self, udev_data, sysfs_data=sysfs_data.get(device_path))
22+ devices[device_path] = device
23+
24+ # The parent-child relations are derived from the path names of
25+ # the devices. If A and B are the path names of two devices,
26+ # the device with path name A is an ancestor of the device with
27+ # path name B, iff B.startswith(A). If C is the set of the path
28+ # names of all ancestors of A, the element with the longest path
29+ # name belongs to the parent of A.
30+ #
31+ # There is one exception to this rule: The root node has the
32+ # the path name '/devices/LNXSYSTM:00', while the path names
33+ # of PCI devices start with '/devices/pci'. We'll temporarily
34+ # change the path name of the root device so that the rule
35+ # holds for all devices.
36+ if UDEV_ROOT_PATH not in devices:
37+ self._logError('No udev root device defined', self.submission_key)
38+ return False
39+ devices['/devices'] = devices[UDEV_ROOT_PATH]
40+ del devices[UDEV_ROOT_PATH]
41+
42+ path_names = sorted(devices, key=len, reverse=True)
43+ for path_index, path_name in enumerate(path_names[:-1]):
44+ # Ensure that the last ancestor of each device is our
45+ # root node.
46+ if not path_name.startswith('/devices'):
47+ self._logError(
48+ 'Invalid device path name: %r' % path_name,
49+ self.submission_key)
50+ return False
51+ for parent_path in path_names[path_index+1:]:
52+ if path_name.startswith(parent_path):
53+ devices[parent_path].addChild(devices[path_name])
54+ break
55+ devices[UDEV_ROOT_PATH] = devices['/devices']
56+ del devices['/devices']
57+ return True
58+
59 def getKernelPackageName(self):
60 """Return the kernel package name of the submission,"""
61 root_hal_device = self.hal_devices[ROOT_UDI]
62
63=== modified file 'lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py'
64--- lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py 2009-10-19 13:49:04 +0000
65+++ lib/canonical/launchpad/scripts/tests/test_hwdb_submission_processing.py 2009-10-20 14:10:34 +0000
66@@ -10,7 +10,7 @@
67 import logging
68 import os
69 import pytz
70-from unittest import TestCase, TestLoader
71+from unittest import TestLoader
72
73 from zope.component import getUtility
74 from zope.testing.loghandler import Handler
75@@ -32,6 +32,7 @@
76 from canonical.launchpad.webapp.errorlog import ErrorReportingUtility
77 from canonical.testing import BaseLayer, LaunchpadZopelessLayer
78
79+from lp.testing import TestCase
80
81 class TestCaseHWDB(TestCase):
82 """Common base class for HWDB processing tests."""
83@@ -113,6 +114,24 @@
84 return
85 raise AssertionError('No log message found: %s' % expected_message)
86
87+ def assertErrorMessage(self, submission_key, log_message):
88+ """Search for log_message in the log entries for submission_key.
89+
90+ :raise: AssertionError if no log message exists that starts with
91+ "Parsing submission <submission_key>:" and that contains
92+ the text passed as the parameter message.
93+ """
94+ expected_message = 'Parsing submission %s: %s' % (
95+ submission_key, log_message)
96+
97+ for record in self.handler.records:
98+ if record.levelno != logging.ERROR:
99+ continue
100+ candidate = record.getMessage()
101+ if candidate == expected_message:
102+ return
103+ raise AssertionError('No log message found: %s' % expected_message)
104+
105
106 class TestHWDBSubmissionProcessing(TestCaseHWDB):
107 """Tests for processing of HWDB submissions."""
108@@ -169,6 +188,141 @@
109 self.assertEqual(child.parent, parent,
110 'Invalid value of child.parent.')
111
112+ def makeUdevDeviceParsedData(self, paths, sysfs_data=None):
113+ """Build test data that can be passed to buildUdevDevice()."""
114+ def makeUdevDevice(path):
115+ """Make a trivial UdevInstance with the given device path."""
116+ return {
117+ 'P': path,
118+ 'E': {}
119+ }
120+ udev_device_data = [makeUdevDevice(path) for path in paths]
121+ if sysfs_data is None:
122+ sysfs_data = {}
123+ parsed_data = {
124+ 'hardware': {
125+ 'udev': udev_device_data,
126+ 'sysfs-attributes': sysfs_data,
127+ 'dmi': {'/sys/class/dmi/id/sys_vendor': 'FUJITSU SIEMENS'},
128+ }
129+ }
130+ return parsed_data
131+
132+ def test_buildUdevDeviceList(self):
133+ """Test the creation of UdevDevice instances for a submission."""
134+ root_device_path = '/devices/LNXSYSTM:00'
135+ pci_pci_bridge_path = '/devices/pci0000:00/0000:00:1c.0'
136+ pci_ethernet_controller_path = (
137+ '/devices/pci0000:00/0000:00:1c.0/0000:02:00.0')
138+ pci_usb_controller_path = '/devices/pci0000:00/0000:00:1d.7'
139+ pci_usb_controller_usb_hub_path = (
140+ '/devices/pci0000:00/0000:00:1d.7/usb1')
141+ usb_storage_device_path = '/devices/pci0000:00/0000:00:1d.7/usb1/1-1'
142+ udev_paths = (
143+ root_device_path, pci_pci_bridge_path,
144+ pci_ethernet_controller_path, pci_usb_controller_path,
145+ pci_usb_controller_usb_hub_path, usb_storage_device_path,
146+ )
147+
148+ parsed_data = self.makeUdevDeviceParsedData(udev_paths)
149+ parser = SubmissionParser()
150+ self.assertTrue(parser.buildUdevDeviceList(parsed_data))
151+
152+ self.assertEqual(len(udev_paths), len(parser.devices))
153+ for path in udev_paths:
154+ self.assertEqual(path, parser.devices[path].device_id)
155+
156+ devices = parser.devices
157+
158+ root_device = parser.devices[root_device_path]
159+ expected_children = set(
160+ (devices[pci_pci_bridge_path], devices[pci_usb_controller_path]))
161+ self.assertEqual(expected_children, set(root_device.children))
162+
163+ pci_pci_bridge = devices[pci_pci_bridge_path]
164+ self.assertEqual(
165+ [devices[pci_ethernet_controller_path]], pci_pci_bridge.children)
166+
167+ usb_controller = devices[pci_usb_controller_path]
168+ usb_hub = devices[pci_usb_controller_usb_hub_path]
169+ self.assertEqual([usb_hub], usb_controller.children)
170+
171+ usb_storage = devices[usb_storage_device_path]
172+ self.assertEqual([usb_storage], usb_hub.children)
173+
174+ def test_buildUdevDeviceList_root_node_has_dmi_data(self):
175+ """The root node of a udev submissions has DMI data."""
176+ root_device_path = '/devices/LNXSYSTM:00'
177+ pci_pci_bridge_path = '/devices/pci0000:00/0000:00:1c.0'
178+ udev_paths = (root_device_path, pci_pci_bridge_path)
179+
180+ parsed_data = self.makeUdevDeviceParsedData(udev_paths)
181+ parser = SubmissionParser()
182+ parser.buildUdevDeviceList(parsed_data)
183+
184+ self.assertEqual(
185+ {'/sys/class/dmi/id/sys_vendor': 'FUJITSU SIEMENS'},
186+ parser.devices[root_device_path].dmi)
187+
188+ self.assertIs(None, parser.devices[pci_pci_bridge_path].dmi)
189+
190+ def test_buildUdevDeviceList_sysfs_data(self):
191+ """Optional sysfs data is passed to UdevDevice instances."""
192+ root_device_path = '/devices/LNXSYSTM:00'
193+ pci_pci_bridge_path = '/devices/pci0000:00/0000:00:1c.0'
194+ pci_ethernet_controller_path = (
195+ '/devices/pci0000:00/0000:00:1c.0/0000:02:00.0')
196+
197+ udev_paths = (
198+ root_device_path, pci_pci_bridge_path,
199+ pci_ethernet_controller_path)
200+
201+ sysfs_data = {
202+ pci_pci_bridge_path: 'sysfs-data',
203+ }
204+
205+ parsed_data = self.makeUdevDeviceParsedData(udev_paths, sysfs_data)
206+ parser = SubmissionParser()
207+ parser.buildUdevDeviceList(parsed_data)
208+
209+ self.assertEqual(
210+ 'sysfs-data', parser.devices[pci_pci_bridge_path].sysfs)
211+
212+ self.assertIs(
213+ None, parser.devices[pci_ethernet_controller_path].sysfs)
214+
215+ def test_buildUdevDeviceList_invalid_device_path(self):
216+ """Test the creation of UdevDevice instances for a submission.
217+
218+ All device paths must start with '/devices'. Any other
219+ device path makes the submission invalid.
220+ """
221+ root_device_path = '/devices/LNXSYSTM:00'
222+ bad_device_path = '/nonsense'
223+ udev_paths = (root_device_path, bad_device_path)
224+
225+ parsed_data = self.makeUdevDeviceParsedData(udev_paths)
226+ parser = SubmissionParser(self.log)
227+ parser.submission_key = 'udev device with invalid path'
228+ self.assertFalse(parser.buildUdevDeviceList(parsed_data))
229+ self.assertErrorMessage(
230+ parser.submission_key, "Invalid device path name: '/nonsense'")
231+
232+ def test_buildUdevDeviceList_missing_root_device(self):
233+ """Test the creation of UdevDevice instances for a submission.
234+
235+ Each submission must contain a udev node for the root device.
236+ """
237+ pci_pci_bridge_path = '/devices/pci0000:00/0000:00:1c.0'
238+ udev_paths = (pci_pci_bridge_path, )
239+
240+ parsed_data = self.makeUdevDeviceParsedData(udev_paths)
241+ parser = SubmissionParser(self.log)
242+ parser.submission_key = 'no udev root device'
243+ self.assertFalse(parser.buildUdevDeviceList(parsed_data))
244+ self.assertErrorMessage(
245+ parser.submission_key, "No udev root device defined")
246+
247 def testKernelPackageName(self):
248 """Test of SubmissionParser.getKernelPackageName.
249