Launchpad itself

Merge lp:~adeuring/launchpad/hwdb-submissions-with-udev-data into lp:launchpad

  1. hwdb-submissions-with-udev-data
  2. Merge into devel
Proposed by Abel Deuring
Status: Merged
Approved by: Abel Deuring
Approved revision: no longer in the source branch.
Merged at revision: not available
Proposed branch: lp:~adeuring/launchpad/hwdb-submissions-with-udev-data
Merge into: lp:launchpad
Diff against target: None lines
To merge this branch: bzr merge lp:~adeuring/launchpad/hwdb-submissions-with-udev-data
Reviewer Review Type Date Requested Status
Henning Eggers (community) Approve
Review via email: mp+11683@code.launchpad.net
To post a comment you must log in.
Revision history for this message
Abel Deuring (adeuring) wrote :

This branch changes the RelaxNG schema hardware-1_0.rng to allow
HWDB submissions without HAL data.

As discussed in bug 399319, both checkbox (the HWDB client) and
Launchpad's hardware database rely up to now on HAL data as the
main source for device data.

HAL will be retired in Karmic++, so we have to switch to other
data sources. These new sources are udev, the attribute files
found in sysfs and some DMI (Desktop management Interface) data.

The XML data submitted by checkbox is validated against a RelaxNG
schema file; this branch extends the schema so that submissions
with the new data will pass the validation.

The changes:

1. The node <hal> is no longer unconcitionally required; instead
the nodes <udev>, <dmi> and <sysfs-attributes> may appear.

2. The submitted data may contain test results, where the tests
can/should refer to existing devices. Each HAL device node
has a unique integer attribute "id"; the test results may have a
node <target id="123">, where "id" should reference HAL node
with the same "id" value.

The new udev data is the plain text output from running "udevadm
info --export-db", which does not contain unique integer IDs for
devices. Instead, devices are identified by their sysfs path.
This path will be used to identify devices in tests; hence the
schema file now allows arbitrary strings for <target id="...">.

3. HAL provided the kernel version; this is replaced by the new
node <kernel-release>.

This branch also adds a sample file containing the new data,
l/c/l/scripts/tests/hardwaretest-udev.xml. This is a modified
variant of l/c/l/scripts/tests/hardwaretest.xml; running diff
for these two files may give a better idea what this change is
about than simply looking at the new file.

If you want to check if the sample file is indeed valid, run

xmllint --noout --relaxng hardware-1_0.rng tests/hardwaretest-udev.xml

And

xmllint --noout --relaxng hardware-1_0.rng tests/hardwaretest.xml

shows that "old" data still validates.

Real tests of the schema changes will follow in another branch.

Revision history for this message
Henning Eggers (henninge) wrote :

Thanks for the nice chat and explanations. ;-) All looks fine!

Danke!
Henning

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'lib/canonical/launchpad/scripts/hardware-1_0.rng'
2--- lib/canonical/launchpad/scripts/hardware-1_0.rng 2009-02-24 17:43:37 +0000
3+++ lib/canonical/launchpad/scripts/hardware-1_0.rng 2009-09-14 09:16:45 +0000
4@@ -98,40 +98,64 @@
5 </element>
6 </zeroOrMore>
7 </element>
8+ <optional>
9+ <element name="kernel-release">
10+ <attribute name="value">
11+ <text/>
12+ </attribute>
13+ </element>
14+ </optional>
15 </interleave>
16 </define>
17
18 <define name="hardwareSection">
19 <interleave>
20- <element name="hal">
21- <attribute name="version">
22- <text/>
23- </attribute>
24- <oneOrMore>
25- <element name="device">
26- <attribute name="id">
27- <data type="integer">
28- <except>
29- <value/>
30- </except>
31- </data>
32- </attribute>
33- <attribute name="udi">
34- <text/>
35- </attribute>
36- <optional>
37- <attribute name="parent">
38- <data type="integer"/>
39- </attribute>
40- </optional>
41- <!-- XXX: Abel Deuring 2007-12-07:
42- specify a set of required properties? -->
43- <oneOrMore>
44- <ref name="property"/>
45- </oneOrMore>
46- </element>
47- </oneOrMore>
48- </element>
49+ <choice>
50+ <element name="hal">
51+ <attribute name="version">
52+ <text/>
53+ </attribute>
54+ <oneOrMore>
55+ <element name="device">
56+ <attribute name="id">
57+ <data type="integer">
58+ <except>
59+ <value/>
60+ </except>
61+ </data>
62+ </attribute>
63+ <attribute name="udi">
64+ <text/>
65+ </attribute>
66+ <optional>
67+ <attribute name="parent">
68+ <data type="integer"/>
69+ </attribute>
70+ </optional>
71+ <!-- XXX: Abel Deuring 2007-12-07:
72+ specify a set of required properties? -->
73+ <oneOrMore>
74+ <ref name="property"/>
75+ </oneOrMore>
76+ </element>
77+ </oneOrMore>
78+ </element>
79+ <group>
80+ <interleave>
81+ <element name="udev">
82+ <text/>
83+ </element>
84+ <element name="dmi">
85+ <text/>
86+ </element>
87+ <element name="sysfs-attributes">
88+ <zeroOrMore>
89+ <text/>
90+ </zeroOrMore>
91+ </element>
92+ </interleave>
93+ </group>
94+ </choice>
95 <element name="processors">
96 <oneOrMore>
97 <element name="processor">
98@@ -156,11 +180,7 @@
99 <zeroOrMore>
100 <element name="alias">
101 <attribute name="target">
102- <data type="integer">
103- <except>
104- <value/>
105- </except>
106- </data>
107+ <text/>
108 </attribute>
109 <interleave>
110 <element name="vendor">
111@@ -292,11 +312,7 @@
112 <zeroOrMore>
113 <element name="target">
114 <attribute name="id">
115- <data type="integer">
116- <except>
117- <value/>
118- </except>
119- </data>
120+ <text/>
121 </attribute>
122 <interleave>
123 <zeroOrMore>
124
125=== added file 'lib/canonical/launchpad/scripts/tests/hardwaretest-udev.xml'
126--- lib/canonical/launchpad/scripts/tests/hardwaretest-udev.xml 1970-01-01 00:00:00 +0000
127+++ lib/canonical/launchpad/scripts/tests/hardwaretest-udev.xml 2009-09-14 09:16:45 +0000
128@@ -0,0 +1,453 @@
129+<?xml version="1.0" ?>
130+<system version="1.0">
131+
132+ <!-- summary: generic information about the submission -->
133+ <summary>
134+
135+ <!-- live_cd: Was this submission made on a system running an Ubuntu Live
136+ CD or on a regular Ubuntu/Linux installation?
137+ -->
138+ <live_cd value="False"/>
139+
140+ <!-- system_id: A hash of the "system identifier". This value is intended
141+ to identify the tested computer model; the value should
142+ be derived from the properties
143+ system.product, system.vendor of the HAL UDI
144+ /org/freedesktop/Hal/devices/computer.
145+ -->
146+ <system_id value="f982bb1ab536469cebfd6eaadcea0ffc"/>
147+
148+ <!-- distribution, distroseries: These values are retrieved from
149+ /etc/lsb-release, parameters DISTRIB_ID and DISTRIB_RELEASE.
150+ -->
151+ <distribution value="Ubuntu"/>
152+ <distroseries value="7.04"/>
153+
154+ <!-- architecture: The processor architecture of the operating system.
155+ -->
156+ <architecture value="amd64"/>
157+
158+ <!-- private: If False, this submission is publicly accessible from
159+ Launchpad, else it is only accesible by the submitter, by
160+ Launchpad administrators and by scripts running with
161+ administrator rights. Submissions marked "private" should
162+ only be used to gather statistical data.
163+ -->
164+ <private value="False"/>
165+
166+ <!-- contactable: If True, the owner agrees to be contacted by other
167+ persons about devices which appear in his submission.
168+ Example of a use case: Developers can ask device owners
169+ to perform tests.
170+ -->
171+ <contactable value="False"/>
172+
173+ <!-- date_created: Date and time (UTC) of the submission.
174+ -->
175+ <date_created value="2007-09-28T16:09:20.126842"/>
176+
177+ <!-- client: The name and version of the program that created the
178+ submission data.
179+ -->
180+ <client name="hwtest" version="0.9">
181+
182+ <!-- plugin: name and version of a plugin used by the client.
183+ This tag may appear more than once.
184+ -->
185+ <plugin name="architecture_info" version="1.1"/>
186+ <plugin name="find_network_controllers" version="2.34"/>
187+ <plugin name="internet_ping" version="1.1"/>
188+ <plugin name="harddisk_speed" version="0.7"/>
189+ </client>
190+
191+ <!-- The kernel name and version, as shown by "uname -r"
192+ -->
193+ <kernel-release value="2.6.28-14-generic"/>
194+ </summary>
195+
196+ <!-- hardware: data about the hardware the submission was made on.
197+ -->
198+ <hardware>
199+
200+ <!-- udev: The output of running "udevadm info - -export-db" -->
201+
202+ <udev>P: /devices/LNXSYSTM:00
203+E: UDEV_LOG=3
204+E: DEVPATH=/devices/LNXSYSTM:00
205+E: MODALIAS=acpi:LNXSYSTM:
206+
207+P: /devices/pci0000:00/0000:00:1a.0
208+E: UDEV_LOG=3
209+E: DEVPATH=/devices/pci0000:00/0000:00:1a.0
210+E: DRIVER=uhci_hcd
211+E: PCI_CLASS=C0300
212+E: PCI_ID=8086:2834
213+E: PCI_SUBSYS_ID=17AA:20AA
214+E: PCI_SLOT_NAME=0000:00:1a.0
215+E: MODALIAS=pci:v00008086d00002834sv000017AAsd000020AAbc0Csc03i00
216+
217+P: /devices/pci0000:00/0000:00:1a.0/usb3
218+N: bus/usb/003/001
219+S: char/189:256
220+E: UDEV_LOG=3
221+E: DEVPATH=/devices/pci0000:00/0000:00:1a.0/usb3
222+E: MAJOR=189
223+E: MINOR=256
224+E: DEVTYPE=usb_device
225+E: DRIVER=usb
226+E: DEVICE=/proc/bus/usb/003/001
227+E: PRODUCT=1d6b/1/206
228+E: TYPE=9/0/0
229+E: BUSNUM=003
230+E: DEVNUM=001
231+E: DEVNAME=/dev/bus/usb/003/001
232+E: DEVLINKS=/dev/char/189:256
233+
234+P: /devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0
235+E: UDEV_LOG=3
236+E: DEVPATH=/devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0
237+E: DEVTYPE=usb_interface
238+E: DRIVER=hub
239+E: DEVICE=/proc/bus/usb/003/001
240+E: PRODUCT=1d6b/1/206
241+E: TYPE=9/0/0
242+E: INTERFACE=9/0/0
243+E: MODALIAS=usb:v1D6Bp0001d0206dc09dsc00dp00ic09isc00ip00
244+
245+P: /devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0/usb_endpoint/usbdev3.1_ep81
246+N: usbdev3.1_ep81
247+S: char/252:4
248+E: UDEV_LOG=3
249+E: DEVPATH=/devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0/usb_endpoint/usbdev3.1_ep81
250+E: MAJOR=252
251+E: MINOR=4
252+E: DEVNAME=/dev/usbdev3.1_ep81
253+E: DEVLINKS=/dev/char/252:4
254+
255+P: /devices/pci0000:00/0000:00:1f.1
256+E: UDEV_LOG=3
257+E: DEVPATH=/devices/pci0000:00/0000:00:1f.1
258+E: DRIVER=ata_piix
259+E: PCI_CLASS=1018A
260+E: PCI_ID=8086:2850
261+E: PCI_SUBSYS_ID=17AA:20A6
262+E: PCI_SLOT_NAME=0000:00:1f.1
263+E: MODALIAS=pci:v00008086d00002850sv000017AAsd000020A6bc01sc01i8a
264+
265+P: /devices/pci0000:00/0000:00:1f.1/host3
266+E: UDEV_LOG=3
267+E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3
268+E: DEVTYPE=scsi_host
269+
270+P: /devices/pci0000:00/0000:00:1f.1/host3/scsi_host/host3
271+E: UDEV_LOG=3
272+E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3/scsi_host/host3
273+
274+P: /devices/pci0000:00/0000:00:1f.1/host3/target3:0:0
275+E: UDEV_LOG=3
276+E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0
277+E: DEVTYPE=scsi_target
278+
279+P: /devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0
280+E: UDEV_LOG=3
281+E: DEVPATH=/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0
282+E: DEVTYPE=scsi_device
283+E: DRIVER=sr
284+E: MODALIAS=scsi:t-0x05
285+</udev>
286+
287+ <!-- The content of publicly accessible files in /sys/class/dmi/id/
288+ format: filename:content
289+ as for example generated by "grep -r . /sys/class/dmi/id/"
290+ -->
291+
292+ <dmi>/sys/class/dmi/id/bios_vendor:LENOVO
293+/sys/class/dmi/id/bios_version:7LETB9WW (2.19 )
294+/sys/class/dmi/id/bios_date:06/06/2008
295+/sys/class/dmi/id/sys_vendor:LENOVO
296+/sys/class/dmi/id/product_name:6457BAG
297+/sys/class/dmi/id/product_version:ThinkPad T61
298+/sys/class/dmi/id/board_vendor:LENOVO
299+/sys/class/dmi/id/board_name:6457BAG
300+/sys/class/dmi/id/board_version:Not Available
301+/sys/class/dmi/id/chassis_vendor:LENOVO
302+/sys/class/dmi/id/chassis_type:10
303+/sys/class/dmi/id/chassis_version:Not Available
304+/sys/class/dmi/id/chassis_asset_tag:No Asset Information
305+/sys/class/dmi/id/modalias:dmi:bvnLENOVO:bvr7LETB9WW(2.19)
306+</dmi>
307+
308+ <!-- Additional data for SCSI devices: vendor, model, type
309+
310+ For each udev node which has DEVTYPE=scsi_device, we need
311+ the content of the sysfs files vendor, model, type. The data
312+ is stored in the same format as the DMI data:
313+ /path/to/file:filecontent
314+ -->
315+ <sysfs-attributes>/sys/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0/vendor:HL-DT-ST
316+/sys/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0/model:DVDRAM GSA-4083N
317+/sys/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0/type:5
318+ </sysfs-attributes>
319+
320+ <!-- processors: Data about processors installed in a system.
321+ The data is retrieved from /proc/cpuinfo.
322+ -->
323+ <processors>
324+
325+ <!-- processor: Data from /proc/cpuinfo about a single processor.
326+ -->
327+ <processor id="123" name="0">
328+
329+ <!-- property: The data of one line of /proc/cpuinfo.
330+ attribute name: The name of the property
331+ (the text left of the ':' in a line of /proc/cpuinfo)
332+ attribute type: A Python type appropriate for the value.
333+ -->
334+ <property name="wp" type="bool">
335+ True
336+ </property>
337+ <property name="flags" type="list">
338+ <value type="str">
339+ fpu
340+ </value>
341+ <value type="str">
342+ vme
343+ </value>
344+ <value type="str">
345+ de
346+ </value>
347+ </property>
348+ <property name="cpu_mhz" type="float">
349+ 1000.0
350+ </property>
351+ </processor>
352+ </processors>
353+
354+ <!-- aliases: optional data provided by the user:
355+ The name of a peripheral, PCI card etc as shown by a label on
356+ the device. OEM devices are often sold under different names
357+ by different vendors; having a set of alias names for a device
358+ allows users of the HWDB to search for information by these
359+ "marketing names".
360+ -->
361+ <aliases>
362+ <!-- alias: The "label name" of a device or system.
363+ attribute target: The sysfs path of a device as given
364+ in <udev>.
365+ -->
366+ <alias target="/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0">
367+
368+ <!-- vendor: The vendor name shown on the device label.
369+ -->
370+ <vendor>Medion</vendor>
371+
372+ <!-- model: The model name of shown on the label.
373+ -->
374+ <model>QuickPrint 9876</model>
375+ </alias>
376+ </aliases>
377+ </hardware>
378+
379+ <!-- software: Data about the software installed on the system.
380+ -->
381+ <software>
382+
383+ <!-- lsbrelease: The data from /etc/lsb-release.
384+ -->
385+ <lsbrelease>
386+
387+ <!-- property: the data from one line of /etc/lsb-release.
388+ attribute type: A Python type appropriate for this
389+ property (str).
390+ -->
391+ <property name="release" type="str">
392+ 7.04
393+ </property>
394+ <property name="codename" type="str">
395+ feisty
396+ </property>
397+ <property name="distributor-id" type="str">
398+ Ubuntu
399+ </property>
400+ <property name="description" type="str">
401+ Ubuntu 7.04
402+ </property>
403+ <property name="dict_example" type="dict">
404+ <value name="a" type="str">value for key a</value>
405+ <value name="b" type="int">1234</value>
406+ </property>
407+ </lsbrelease>
408+
409+ <!-- packages: Data about the installed software packages.
410+ -->
411+ <packages>
412+
413+ <!-- package: Data about a single package.
414+ The <property> sub-tags contain the DEB properties
415+ "name", "priority", "section", "source", "version",
416+ "installed_size", "size", "summary".
417+
418+ XXX Abel Deuring 2007-12-12: What about submissions
419+ from RPM-based Linux versions? (And "exotic" variants
420+ like Gentoo?)
421+ -->
422+ <package name="metacity" id="200">
423+ <property name="installed_size" type="int">
424+ 868352
425+ </property>
426+ <property name="section" type="str">
427+ x11
428+ </property>
429+ <property name="summary" type="str">
430+ A lightweight GTK2 based Window Manager
431+ </property>
432+ <property name="priority" type="str">
433+ optional
434+ </property>
435+ <property name="source" type="str">
436+ metacity
437+ </property>
438+ <property name="version" type="str">
439+ 1:2.18.2-0ubuntu1.1
440+ </property>
441+ <property name="size" type="int">
442+ 429128
443+ </property>
444+ </package>
445+ </packages>
446+ <!-- Information extracted from Xorg.0.log.
447+ HAL does not provide any information about Xorg drivers, so
448+ we retrieve that from the xserver's log file.
449+ -->
450+ <xorg version="1.3.0">
451+ <!-- driver: Data about a driver.
452+ (optional)
453+ attribute name: The name of the driver.
454+ attribute version: The version of the driver.
455+ attribute class: The module class of the driver
456+ attribute device: The ID of a device driven by this driver.
457+ -->
458+ <driver name="fglrx" version="1.23" class="X.Org Video Driver"
459+ device="12"/>
460+ </xorg>
461+ </software>
462+
463+ <!-- questions: User's answers to questions asked by the client.
464+ -->
465+ <questions>
466+
467+ <!-- question: Data of a question.
468+ attribute name: The unique name of the question.
469+ attribute plugin: The name of the plugin which asked
470+ the question.
471+ attribute version: The version of the question.
472+ attribute type: Allowed values are "manual" and "automatic".
473+ A "manual" question requires user input for the answer;
474+ an "automatic" question gets the answer automatically.
475+ -->
476+ <question name="detected_network_controllers"
477+ plugin="find_network_controllers">
478+
479+ <!-- target: Information about a device or software package the
480+ question is about. The attribute "id" is the sysfs path of
481+ a device as given in <udev>, or the ID of a software package
482+ node.
483+ This node may appear multiple times.
484+ -->
485+ <target id="/devices/pci0000:00/0000:00:1a.0/usb3/3-0:1.0/usb_endpoint/usbdev3.1_ep81">
486+ <!-- driver: The driver which controls the target device. This tag
487+ may appear more than once.
488+
489+ While we are working on a project called "Hardware Database",
490+ we are not that much interested in the question, if a device
491+ works "as such", but if their Linux driver(s) work.
492+
493+ It is not in every case possible to identify the used driver
494+ from HAL data, so we need another way to add this information.
495+ (example: HAL does not know, which driver is used for the
496+ graphics card.)
497+
498+ Example for multiple drivers: Some scanners have a SCSI _and_
499+ a USB interface; if such a scanner is tested, we not only want
500+ to know, which Sane backend is used, but also, which interface
501+ is used.
502+
503+ Also, it might be interesting to know for many USB 2.0 devices,
504+ if a USB 1 (uhci_hcd or ohci_hcd driver) or the USB 2 driver
505+ (ehci_hcd) was used. A USB 1 driver might for example explain
506+ latency problems.
507+ -->
508+ <driver>ipw3945</driver>
509+ </target>
510+
511+ <!-- ID of the 88E8055 PCI-E Gigabit Ethernet Controller -->
512+ <target id="/devices/pci0000:00/0000:00:1f.1"/>
513+
514+ <!-- command: The command line of an external command required to
515+ ask this question.
516+ -->
517+ <command/>
518+
519+ <!-- answer: The answer to the question. Two types of answers are
520+ defined, "multiple_choice" and "measurement". (See below
521+ for an example of the latter.)
522+ attribute type: Must be "multiple_choice" or "measurement".
523+ -->
524+ <answer type="multiple_choice">pass</answer>
525+
526+ <!-- answer_choices: The list of possible choices.
527+ The data should only be used for consistency
528+ checks and to detect variants of the question.
529+ -->
530+ <answer_choices>
531+ <value type="str">fail</value>
532+ <value type="str">pass</value>
533+ <value type="str">skip</value>
534+ </answer_choices>
535+
536+ <!-- A user comment about the device or about the test.
537+ -->
538+ <comment>
539+ The WLAN adapter drops the connection very frequently.
540+ </comment>
541+ </question>
542+
543+ <question name="internet_ping"
544+ plugin="internet_ping">
545+ <target id="/devices/pci0000:00/0000:00:1f.1"/>
546+ <command/>
547+ <answer type="multiple_choice">pass</answer>
548+ <answer_choices>
549+ <value type="str">fail</value>
550+ <value type="str">pass</value>
551+ <value type="str">skip</value>
552+ </answer_choices>
553+ </question>
554+
555+ <!-- example for a "measurement question"
556+ -->
557+ <question name="harddisk_speed"
558+ plugin="harddisk_speed">
559+ <target id="/devices/pci0000:00/0000:00:1f.1/host3/target3:0:0/3:0:0:0"/>
560+ <command>hdparm -t /dev/sda</command>
561+ <!-- answer: The answer to a "measurement question".
562+ attribute type: See above.
563+ attribute unit: The unit of the result of the measurement.
564+ XXX Abel Deuring 2007-12-12 bug=175978 We should
565+ enumerate a list of allowed units, in order to avoid
566+ multiple units for the same dimension. e.g., B/sec,
567+ MB/sec or inch, cm, foot.
568+
569+ For dimensionless values, the attribute unit is omitted.
570+
571+ "Percentage" and similar "convenience pseudo-units" like
572+ ppm are _not_ allowed; instead a dimensionless
573+ value must be used, where 0 is equivalent 0% and 1.0 is
574+ equivalent to 100%.
575+ -->
576+ <answer type="measurement" unit="MB/sec">38.4</answer>
577+ </question>
578+ </questions>
579+ <!-- miscellaneous additional text data.
580+ -->
581+</system>