Launchpad itself

Merge lp:~adeuring/launchpad/hwdb-submissions-with-udev-data into lp:launchpad

  1. hwdb-submissions-with-udev-data
  2. Merge into devel
Proposed by Abel Deuring
Status: Merged
Approved by: Abel Deuring
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~adeuring/launchpad/hwdb-submissions-with-udev-data
Merge into: lp:launchpad
Diff against target: None lines
To merge this branch: bzr merge lp:~adeuring/launchpad/hwdb-submissions-with-udev-data
Reviewer Review Type Date Requested Status
Henning Eggers (community) Approve
Review via email: mp+11683@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

This branch changes the RelaxNG schema hardware-1_0.rng to allow
HWDB submissions without HAL data.

As discussed in bug 399319, both checkbox (the HWDB client) and
Launchpad's hardware database rely up to now on HAL data as the
main source for device data.

HAL will be retired in Karmic++, so we have to switch to other
data sources. These new sources are udev, the attribute files
found in sysfs and some DMI (Desktop management Interface) data.

The XML data submitted by checkbox is validated against a RelaxNG
schema file; this branch extends the schema so that submissions
with the new data will pass the validation.

The changes:

1. The node <hal> is no longer unconcitionally required; instead
the nodes <udev>, <dmi> and <sysfs-attributes> may appear.

2. The submitted data may contain test results, where the tests
can/should refer to existing devices. Each HAL device node
has a unique integer attribute "id"; the test results may have a
node <target id="123">, where "id" should reference HAL node
with the same "id" value.

The new udev data is the plain text output from running "udevadm
info --export-db", which does not contain unique integer IDs for
devices. Instead, devices are identified by their sysfs path.
This path will be used to identify devices in tests; hence the
schema file now allows arbitrary strings for <target id="...">.

3. HAL provided the kernel version; this is replaced by the new
node <kernel-release>.

This branch also adds a sample file containing the new data,
l/c/l/scripts/tests/hardwaretest-udev.xml. This is a modified
variant of l/c/l/scripts/tests/hardwaretest.xml; running diff
for these two files may give a better idea what this change is
about than simply looking at the new file.

If you want to check if the sample file is indeed valid, run

xmllint --noout --relaxng hardware-1_0.rng tests/hardwaretest-udev.xml

And

xmllint --noout --relaxng hardware-1_0.rng tests/hardwaretest.xml

shows that "old" data still validates.

Real tests of the schema changes will follow in another branch.

Revision history for this message
Henning Eggers (henninge) wrote :

Thanks for the nice chat and explanations. ;-) All looks fine!

Danke!
Henning

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'lib/canonical/launchpad/scripts/hardware-1_0.rng'
--- lib/canonical/launchpad/scripts/hardware-1_0.rng 2009-02-24 17:43:37 +0000
+++ lib/canonical/launchpad/scripts/hardware-1_0.rng 2009-09-14 09:16:45 +0000
@@ -98,40 +98,64 @@
98 </element>98 </element>
99 </zeroOrMore>99 </zeroOrMore>
100 </element>100 </element>
101 <optional>
102 <element name="kernel-release">
103 <attribute name="value">
104 <text/>
105 </attribute>
106 </element>
107 </optional>
101 </interleave>108 </interleave>
102 </define>109 </define>
103110
104 <define name="hardwareSection">111 <define name="hardwareSection">
105 <interleave>112 <interleave>
106 <element name="hal">113 <choice>
107 <attribute name="version">114 <element name="hal">
108 <text/>115 <attribute name="version">
109 </attribute>116 <text/>
110 <oneOrMore>117 </attribute>
111 <element name="device">118 <oneOrMore>
112 <attribute name="id">119 <element name="device">
113 <data type="integer">120 <attribute name="id">
114 <except>121 <data type="integer">
115 <value/>122 <except>
116 </except>123 <value/>
117 </data>124 </except>
118 </attribute>125 </data>
119 <attribute name="udi">126 </attribute>
120 <text/>127 <attribute name="udi">
121 </attribute>128 <text/>
122 <optional>129 </attribute>
123 <attribute name="parent">130 <optional>
124 <data type="integer"/>131 <attribute name="parent">
125 </attribute>132 <data type="integer"/>
126 </optional>133 </attribute>
127 <!-- XXX: Abel Deuring 2007-12-07:134 </optional>
128 specify a set of required properties? -->135 <!-- XXX: Abel Deuring 2007-12-07:
129 <oneOrMore>136 specify a set of required properties? -->
130 <ref name="property"/>137 <oneOrMore>
131 </oneOrMore>138 <ref name="property"/>
132 </element>139 </oneOrMore>
133 </oneOrMore>140 </element>
134 </element>141 </oneOrMore>
142 </element>
143 <group>
144 <interleave>
145 <element name="udev">
146 <text/>
147 </element>
148 <element name="dmi">
149 <text/>
150 </element>
151 <element name="sysfs-attributes">
152 <zeroOrMore>
153 <text/>
154 </zeroOrMore>
155 </element>
156 </interleave>
157 </group>
158 </choice>
135 <element name="processors">159 <element name="processors">
136 <oneOrMore>160 <oneOrMore>
137 <element name="processor">161 <element name="processor">
@@ -156,11 +180,7 @@
156 <zeroOrMore>180 <zeroOrMore>
157 <element name="alias">181 <element name="alias">
158 <attribute name="target">182 <attribute name="target">
159 <data type="integer">183 <text/>
160 <except>
161 <value/>
162 </except>
163 </data>
164 </attribute>184 </attribute>
165 <interleave>185 <interleave>
166 <element name="vendor">186 <element name="vendor">
@@ -292,11 +312,7 @@
292 <zeroOrMore>312 <zeroOrMore>
293 <element name="target">313 <element name="target">
294 <attribute name="id">314 <attribute name="id">
295 <data type="integer">315 <text/>
296 <except>
297 <value/>
298 </except>
299 </data>
300 </attribute>316 </attribute>
301 <interleave>317 <interleave>
302 <zeroOrMore>318 <zeroOrMore>
303319
=== added file 'lib/canonical/launchpad/scripts/tests/hardwaretest-udev.xml'
--- lib/canonical/launchpad/scripts/tests/hardwaretest-udev.xml 1970-01-01 00:00:00 +0000
+++ lib/canonical/launchpad/scripts/tests/hardwaretest-udev.xml 2009-09-14 09:16:45 +0000
@@ -0,0 +1,453 @@
1<?xml version="1.0" ?>
2<system version="1.0">
3
4 <!-- summary: generic information about the submission -->
5 <summary>
6
7 <!-- live_cd: Was this submission made on a system running an Ubuntu Live
8 CD or on a regular Ubuntu/Linux installation?
9 -->
10 <live_cd value="False"/>
11
12 <!-- system_id: A hash of the "system identifier". This value is intended
13 to identify the tested computer model; the value should
14 be derived from the properties
15 system.product, system.vendor of the HAL UDI
16 /org/freedesktop/Hal/devices/computer.
17 -->
18 <system_id value="f982bb1ab536469cebfd6eaadcea0ffc"/>
19
20 <!-- distribution, distroseries: These values are retrieved from
21 /etc/lsb-release, parameters DISTRIB_ID and DISTRIB_RELEASE.
22 -->
23 <distribution value="Ubuntu"/>
24 <distroseries value="7.04"/>
25
26 <!-- architecture: The processor architecture of the operating system.
27 -->
28 <architecture value="amd64"/>
29
30 <!-- private: If False, this submission is publicly accessible from
31 Launchpad, else it is only accesible by the submitter, by
32 Launchpad administrators and by scripts running with
33 administrator rights. Submissions marked "private" should
34 only be used to gather statistical data.
35 -->
36 <private value="False"/>
37
38 <!-- contactable: If True, the owner agrees to be contacted by other
39 persons about devices which appear in his submission.
40 Example of a use case: Developers can ask device owners
41 to perform tests.
42 -->
43 <contactable value="False"/>
44
45 <!-- date_created: Date and time (UTC) of the submission.
46 -->
47 <date_created value="2007-09-28T16:09:20.126842"/>
48
49 <!-- client: The name and version of the program that created the
50 submission data.
51 -->
52 <client name="hwtest" version="0.9">
53
54 <!-- plugin: name and version of a plugin used by the client.
55 This tag may appear more than once.
56 -->
57 <plugin name="architecture_info" version="1.1"/>
58 <plugin name="find_network_controllers" version="2.34"/>
59 <plugin name="internet_ping" version="1.1"/>
60 <plugin name="harddisk_speed" version="0.7"/>
61 </client>
62
63 <!-- The kernel name and version, as shown by "uname -r"
64 -->
65 <kernel-release value="2.6.28-14-generic"/>
66 </summary>
67
68 <!-- hardware: data about the hardware the submission was made on.
69 -->
70 <hardware>
71
72 <!-- udev: The output of running "udevadm info - -export-db" -->
73
74 <udev>P: /devices/LNXSYSTM:00
75E: UDEV_LOG=3
76E: DEVPATH=/devices/LNXSYSTM:00
77E: MODALIAS=acpi:LNXSYSTM:
78
79P: /devices/pci0000:00/0000:00:1a.0
80E: UDEV_LOG=3
81E: DEVPATH=/devices/pci0000:00/0000:00:1a.0
82E: DRIVER=uhci_hcd
83E: PCI_CLASS=C0300
84E: PCI_ID=8086:2834
85E: PCI_SUBSYS_ID=17AA:20AA
86E: PCI_SLOT_NAME=0000:00:1a.0
87E: MODALIAS=pci:v00008086d00002834sv000017AAsd000020AAbc0Csc03i00
88
89P: /devices/pci0000:00/0000:00:1a.0/usb3
90N: bus/usb/003/001
91S: char/189:256
92E: UDEV_LOG=3
93E: DEVPATH=/devices/pci0000:00/0000:00:1a.0/usb3
94E: MAJOR=189
95E: MINOR=256
96E: DEVTYPE=usb_device
97E: DRIVER=usb
98E: DEVICE=/proc/bus/usb/003/001
99E: PRODUCT=1d6b/1/206
100E: TYPE=9/0/0
101E: BUSNUM=003
102E: DEVNUM=001
103E: DEVNAME=/dev/bus/usb/003/001
104E: DEVLINKS=/dev/char/189:256
105
106P: /devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0
107E: UDEV_LOG=3
108E: DEVPATH=/devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0
109E: DEVTYPE=usb_interface
110E: DRIVER=hub
111E: DEVICE=/proc/bus/usb/003/001
112E: PRODUCT=1d6b/1/206
113E: TYPE=9/0/0
114E: INTERFACE=9/0/0
115E: MODALIAS=usb:v1D6Bp0001d0206dc09dsc00dp00ic09isc00ip00
116
117P: /devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0/usb_endpoint/usbdev3.1_ep81
118N: usbdev3.1_ep81
119S: char/252:4
120E: UDEV_LOG=3
121E: DEVPATH=/devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0/usb_endpoint/usbdev3.1_ep81
122E: MAJOR=252
123E: MINOR=4
124E: DEVNAME=/dev/usbdev3.1_ep81
125E: DEVLINKS=/dev/char/252:4
126
127P: /devices/pci0000:00/0000:00:1f.1
128E: UDEV_LOG=3
129E: DEVPATH=/devices/pci0000:00/0000:00:1f.1
130E: DRIVER=ata_piix
131E: PCI_CLASS=1018A
132E: PCI_ID=8086:2850
133E: PCI_SUBSYS_ID=17AA:20A6
134E: PCI_SLOT_NAME=0000:00:1f.1
135E: MODALIAS=pci:v00008086d00002850sv000017AAsd000020A6bc01sc01i8a
136
137P: /devices/pci0000:00/0000:00:1f.1/host3
138E: UDEV_LOG=3
139E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3
140E: DEVTYPE=scsi_host
141
142P: /devices/pci0000:00/0000:00:1f.1/host3/scsi_host/host3
143E: UDEV_LOG=3
144E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3/scsi_host/host3
145
146P: /devices/pci0000:00/0000:00:1f.1/host3/target3:0:0
147E: UDEV_LOG=3
148E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0
149E: DEVTYPE=scsi_target
150
151P: /devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0
152E: UDEV_LOG=3
153E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0
154E: DEVTYPE=scsi_device
155E: DRIVER=sr
156E: MODALIAS=scsi:t-0x05
157</udev>
158
159 <!-- The content of publicly accessible files in /sys/class/dmi/id/
160 format: filename:content
161 as for example generated by "grep -r . /sys/class/dmi/id/"
162 -->
163
164 <dmi>/sys/class/dmi/id/bios_vendor:LENOVO
165/sys/class/dmi/id/bios_version:7LETB9WW (2.19 )
166/sys/class/dmi/id/bios_date:06/06/2008
167/sys/class/dmi/id/sys_vendor:LENOVO
168/sys/class/dmi/id/product_name:6457BAG
169/sys/class/dmi/id/product_version:ThinkPad T61
170/sys/class/dmi/id/board_vendor:LENOVO
171/sys/class/dmi/id/board_name:6457BAG
172/sys/class/dmi/id/board_version:Not Available
173/sys/class/dmi/id/chassis_vendor:LENOVO
174/sys/class/dmi/id/chassis_type:10
175/sys/class/dmi/id/chassis_version:Not Available
176/sys/class/dmi/id/chassis_asset_tag:No Asset Information
177/sys/class/dmi/id/modalias:dmi:bvnLENOVO:bvr7LETB9WW(2.19)
178</dmi>
179
180 <!-- Additional data for SCSI devices: vendor, model, type
181
182 For each udev node which has DEVTYPE=scsi_device, we need
183 the content of the sysfs files vendor, model, type. The data
184 is stored in the same format as the DMI data:
185 /path/to/file:filecontent
186 -->
187 <sysfs-attributes>/sys/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0/vendor:HL-DT-ST
188/sys/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0/model:DVDRAM GSA-4083N
189/sys/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0/type:5
190 </sysfs-attributes>
191
192 <!-- processors: Data about processors installed in a system.
193 The data is retrieved from /proc/cpuinfo.
194 -->
195 <processors>
196
197 <!-- processor: Data from /proc/cpuinfo about a single processor.
198 -->
199 <processor id="123" name="0">
200
201 <!-- property: The data of one line of /proc/cpuinfo.
202 attribute name: The name of the property
203 (the text left of the ':' in a line of /proc/cpuinfo)
204 attribute type: A Python type appropriate for the value.
205 -->
206 <property name="wp" type="bool">
207 True
208 </property>
209 <property name="flags" type="list">
210 <value type="str">
211 fpu
212 </value>
213 <value type="str">
214 vme
215 </value>
216 <value type="str">
217 de
218 </value>
219 </property>
220 <property name="cpu_mhz" type="float">
221 1000.0
222 </property>
223 </processor>
224 </processors>
225
226 <!-- aliases: optional data provided by the user:
227 The name of a peripheral, PCI card etc as shown by a label on
228 the device. OEM devices are often sold under different names
229 by different vendors; having a set of alias names for a device
230 allows users of the HWDB to search for information by these
231 "marketing names".
232 -->
233 <aliases>
234 <!-- alias: The "label name" of a device or system.
235 attribute target: The sysfs path of a device as given
236 in <udev>.
237 -->
238 <alias target="/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0">
239
240 <!-- vendor: The vendor name shown on the device label.
241 -->
242 <vendor>Medion</vendor>
243
244 <!-- model: The model name of shown on the label.
245 -->
246 <model>QuickPrint 9876</model>
247 </alias>
248 </aliases>
249 </hardware>
250
251 <!-- software: Data about the software installed on the system.
252 -->
253 <software>
254
255 <!-- lsbrelease: The data from /etc/lsb-release.
256 -->
257 <lsbrelease>
258
259 <!-- property: the data from one line of /etc/lsb-release.
260 attribute type: A Python type appropriate for this
261 property (str).
262 -->
263 <property name="release" type="str">
264 7.04
265 </property>
266 <property name="codename" type="str">
267 feisty
268 </property>
269 <property name="distributor-id" type="str">
270 Ubuntu
271 </property>
272 <property name="description" type="str">
273 Ubuntu 7.04
274 </property>
275 <property name="dict_example" type="dict">
276 <value name="a" type="str">value for key a</value>
277 <value name="b" type="int">1234</value>
278 </property>
279 </lsbrelease>
280
281 <!-- packages: Data about the installed software packages.
282 -->
283 <packages>
284
285 <!-- package: Data about a single package.
286 The <property> sub-tags contain the DEB properties
287 "name", "priority", "section", "source", "version",
288 "installed_size", "size", "summary".
289
290 XXX Abel Deuring 2007-12-12: What about submissions
291 from RPM-based Linux versions? (And "exotic" variants
292 like Gentoo?)
293 -->
294 <package name="metacity" id="200">
295 <property name="installed_size" type="int">
296 868352
297 </property>
298 <property name="section" type="str">
299 x11
300 </property>
301 <property name="summary" type="str">
302 A lightweight GTK2 based Window Manager
303 </property>
304 <property name="priority" type="str">
305 optional
306 </property>
307 <property name="source" type="str">
308 metacity
309 </property>
310 <property name="version" type="str">
311 1:2.18.2-0ubuntu1.1
312 </property>
313 <property name="size" type="int">
314 429128
315 </property>
316 </package>
317 </packages>
318 <!-- Information extracted from Xorg.0.log.
319 HAL does not provide any information about Xorg drivers, so
320 we retrieve that from the xserver's log file.
321 -->
322 <xorg version="1.3.0">
323 <!-- driver: Data about a driver.
324 (optional)
325 attribute name: The name of the driver.
326 attribute version: The version of the driver.
327 attribute class: The module class of the driver
328 attribute device: The ID of a device driven by this driver.
329 -->
330 <driver name="fglrx" version="1.23" class="X.Org Video Driver"
331 device="12"/>
332 </xorg>
333 </software>
334
335 <!-- questions: User's answers to questions asked by the client.
336 -->
337 <questions>
338
339 <!-- question: Data of a question.
340 attribute name: The unique name of the question.
341 attribute plugin: The name of the plugin which asked
342 the question.
343 attribute version: The version of the question.
344 attribute type: Allowed values are "manual" and "automatic".
345 A "manual" question requires user input for the answer;
346 an "automatic" question gets the answer automatically.
347 -->
348 <question name="detected_network_controllers"
349 plugin="find_network_controllers">
350
351 <!-- target: Information about a device or software package the
352 question is about. The attribute "id" is the sysfs path of
353 a device as given in <udev>, or the ID of a software package
354 node.
355 This node may appear multiple times.
356 -->
357 <target id="/devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0/usb_endpoint/usbdev3.1_ep81">
358 <!-- driver: The driver which controls the target device. This tag
359 may appear more than once.
360
361 While we are working on a project called "Hardware Database",
362 we are not that much interested in the question, if a device
363 works "as such", but if their Linux driver(s) work.
364
365 It is not in every case possible to identify the used driver
366 from HAL data, so we need another way to add this information.
367 (example: HAL does not know, which driver is used for the
368 graphics card.)
369
370 Example for multiple drivers: Some scanners have a SCSI _and_
371 a USB interface; if such a scanner is tested, we not only want
372 to know, which Sane backend is used, but also, which interface
373 is used.
374
375 Also, it might be interesting to know for many USB 2.0 devices,
376 if a USB 1 (uhci_hcd or ohci_hcd driver) or the USB 2 driver
377 (ehci_hcd) was used. A USB 1 driver might for example explain
378 latency problems.
379 -->
380 <driver>ipw3945</driver>
381 </target>
382
383 <!-- ID of the 88E8055 PCI-E Gigabit Ethernet Controller -->
384 <target id="/devices/pci0000:00/0000:00:1f.1"/>
385
386 <!-- command: The command line of an external command required to
387 ask this question.
388 -->
389 <command/>
390
391 <!-- answer: The answer to the question. Two types of answers are
392 defined, "multiple_choice" and "measurement". (See below
393 for an example of the latter.)
394 attribute type: Must be "multiple_choice" or "measurement".
395 -->
396 <answer type="multiple_choice">pass</answer>
397
398 <!-- answer_choices: The list of possible choices.
399 The data should only be used for consistency
400 checks and to detect variants of the question.
401 -->
402 <answer_choices>
403 <value type="str">fail</value>
404 <value type="str">pass</value>
405 <value type="str">skip</value>
406 </answer_choices>
407
408 <!-- A user comment about the device or about the test.
409 -->
410 <comment>
411 The WLAN adapter drops the connection very frequently.
412 </comment>
413 </question>
414
415 <question name="internet_ping"
416 plugin="internet_ping">
417 <target id="/devices/pci0000:00/0000:00:1f.1"/>
418 <command/>
419 <answer type="multiple_choice">pass</answer>
420 <answer_choices>
421 <value type="str">fail</value>
422 <value type="str">pass</value>
423 <value type="str">skip</value>
424 </answer_choices>
425 </question>
426
427 <!-- example for a "measurement question"
428 -->
429 <question name="harddisk_speed"
430 plugin="harddisk_speed">
431 <target id="/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0"/>
432 <command>hdparm -t /dev/sda</command>
433 <!-- answer: The answer to a "measurement question".
434 attribute type: See above.
435 attribute unit: The unit of the result of the measurement.
436 XXX Abel Deuring 2007-12-12 bug=175978 We should
437 enumerate a list of allowed units, in order to avoid
438 multiple units for the same dimension. e.g., B/sec,
439 MB/sec or inch, cm, foot.
440
441 For dimensionless values, the attribute unit is omitted.
442
443 "Percentage" and similar "convenience pseudo-units" like
444 ppm are _not_ allowed; instead a dimensionless
445 value must be used, where 0 is equivalent 0% and 1.0 is
446 equivalent to 100%.
447 -->
448 <answer type="measurement" unit="MB/sec">38.4</answer>
449 </question>
450 </questions>
451 <!-- miscellaneous additional text data.
452 -->
453</system>