Merge lp:~soren/nova/raw-disk-images into lp:~hudson-openstack/nova/trunk

Proposed by Soren Hansen
Status: Merged
Approved by: Soren Hansen
Approved revision: 439
Merged at revision: 471
Proposed branch: lp:~soren/nova/raw-disk-images
Merge into: lp:~hudson-openstack/nova/trunk
Diff against target: 832 lines (+264/-268)
15 files modified
.mailmap (+1/-0)
MANIFEST.in (+1/-1)
nova/api/ec2/cloud.py (+1/-1)
nova/compute/api.py (+11/-6)
nova/compute/disk.py (+19/-3)
nova/flags.py (+3/-4)
nova/tests/virt_unittest.py (+108/-24)
nova/virt/libvirt.rescue.qemu.xml.template (+0/-37)
nova/virt/libvirt.rescue.uml.xml.template (+0/-26)
nova/virt/libvirt.rescue.xen.xml.template (+0/-34)
nova/virt/libvirt.uml.xml.template (+0/-26)
nova/virt/libvirt.xen.xml.template (+0/-30)
nova/virt/libvirt.xml.template (+61/-18)
nova/virt/libvirt_conn.py (+58/-58)
tools/pip-requires (+1/-0)
To merge this branch: bzr merge lp:~soren/nova/raw-disk-images
Reviewer Review Type Date Requested Status
Jay Pipes (community) Approve
Vish Ishaya (community) Approve
Salvatore Orlando (community) Needs Information
Rick Clark (community) Approve
Review via email: mp+43712@code.launchpad.net

Commit message

Add raw disk image support.

Description of the change

This branch adds raw disk image support. It was originally implemented by Justin Santa Barbara, then forward-ported by Rick Clark, and now I (believe I) finished it off.

To post a comment you must log in.
Revision history for this message
Rick Clark (dendrobates) wrote :

Like the new templates.

review: Approve
Revision history for this message
Soren Hansen (soren) wrote :

I need to make sure this works after the eventlet merge..

Revision history for this message
Soren Hansen (soren) wrote :

Should be fine now.

Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

Hi Soren,

I checked out your code, and I am not able to launch raw images through the EC2 api.
When I use ' --kernel nokernel', I get the following error:
ERROR:root:NotFound: Image nokernel could not be found

I believe this happens because the image service still tries to retrieve the kernel image.
from compute/api.py lines 86-87:
            if kernel_id:
                self.image_service.show(context, kernel_id)

In the original branch from justinsb, kernel and ramdisk images were not retrieve when the null_kernel 'magic number' was used.

            if kernel_id == str(FLAGS.null_kernel):
                kernel_id = None
                ramdisk_id = None
                logging.debug("Creating a raw instance")
            # Make sure we have access to kernel and ramdisk (if not raw)
            if kernel_id:
                image_service.show(context, kernel_id)
            if ramdisk_id:
                image_service.show(context, ramdisk_id)

Is there something I'm not seeing at the moment, or do we need to fix compute/api.py?

review: Needs Information
Revision history for this message
Rick Clark (dendrobates) wrote :

Salvatore,
I have reproduced your problem. Soren is now out for Christmas holiday. I suggest that we merge this to clear up the queue and then fix the bug. It does not break existing behavior. Waiting for Soren to fix this branch will delay merging an the fix to api.py should be simple.

Revision history for this message
Thierry Carrez (ttx) wrote :

+1, just make sure we have a bug filed to remember to fix that one.

Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

Hi Rick,

I fixed it in my brach which has been proposed for merge.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

Lets get this in and aadd salvatore's fix.

review: Approve
Revision history for this message
Jay Pipes (jaypipes) wrote :

Looks great. One tiny pep8 fix to make, though:

jpipes@serialcoder:~/repos/nova/raw-disk-images$ pep8 bin/* nova
nova/compute/disk.py:108:1: E302 expected 2 blank lines, found 1

review: Needs Fixing
Revision history for this message
Soren Hansen (soren) wrote :

Fixed.

Revision history for this message
Jay Pipes (jaypipes) wrote :

w00t.

review: Approve
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

The attempt to merge lp:~soren/nova/raw-disk-images into lp:nova failed. Below is the output from the failed tests.

Traceback (most recent call last):
  File "run_tests.py", line 71, in <module>
    from nova.tests.virt_unittest import *
  File "/tmp/tmpgcaUjT/nova/tests/virt_unittest.py", line 27, in <module>
    from nova.virt import libvirt_conn
  File "/tmp/tmpgcaUjT/nova/virt/libvirt_conn.py", line 60, in <module>
    from Cheetah.Template import Template
ImportError: No module named Cheetah.Template

Revision history for this message
Soren Hansen (soren) wrote :

soren@hudson:~$ sudo apt-get install python-cheetah
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-cheetah
0 upgraded, 1 newly installed, 0 to remove and 28 not upgraded.
Need to get 0B/199kB of archives.
After this operation, 1073kB of additional disk space will be used.
Selecting previously deselected package python-cheetah.
(Reading database ... 41319 files and directories currently installed.)
Unpacking python-cheetah (from .../python-cheetah_2.0.1-2ubuntu7_amd64.deb) ...
Processing triggers for man-db ...
Setting up python-cheetah (2.0.1-2ubuntu7) ...
Processing triggers for python-support ...
soren@hudson:~$

Retrying.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :
Download full text (12.8 KiB)

The attempt to merge lp:~soren/nova/raw-disk-images into lp:nova failed. Below is the output from the failed tests.

nova.tests.access_unittest
  AccessTestCase
    test_001_allow_all ... [OK]
    test_002_allow_none ... [OK]
    test_003_allow_project_manager ... [OK]
    test_004_allow_sys_and_net ... [OK]
nova.tests.api_unittest
  ApiEc2TestCase
    test_authorize_revoke_security_group_cidr ... [OK]
    test_authorize_revoke_security_group_foreign_group ... [OK]
    test_create_delete_security_group ... [OK]
    test_describe_instances ... [OK]
    test_get_all_key_pairs ... [OK]
    test_get_all_security_groups ... [OK]
  XmlConversionTestCase
    test_number_conversion ... [OK]
nova.tests.auth_unittest
  AuthManagerDbTestCase
    test_004_signature_is_valid ... [OK]
    test_005_can_get_credentials ... [OK]
    test_add_user_role_doesnt_infect_project_roles ... [OK]
    test_adding_role_to_project_is_ignored_unless_added_to_user ... [OK]
    test_can_add_and_remove_user_role ... [OK]
    test_can_add_remove_user_with_role ... [OK]
    test_can_add_user_to_project ... [OK]
    test_can_create_and_get_project ... [OK]
    test_can_create_and_get_project_with_attributes ... [OK]
    test_can_create_project_with_manager ... [OK]
    test_can_delete_project ... [OK]
    test_can_delete_user ... [OK]
    test_can_generate_x509 ... [OK]
    test_can_list_project_roles ... [OK]
    test_can_list_projects ... [OK]
    test_can_list_user_roles ... [OK]
    test_can_list_users ... [OK]
    test_can_modify_project ... [OK]
    test_can_modify_users ... [OK]
    test_can_remove_project_role_but_keep_user_role ... [OK]
    test_can_remove_user_from_project ... [OK]
    test_can_remove_user_roles ... [OK]
    test_can_retrieve_project_by_user ... [OK]
    test_create_and_find_user ... [OK]
    test_create_and_find_with_properties ... [OK]
    test_create_project_assigns_manager_to_me...

lp:~soren/nova/raw-disk-images updated
439. By Soren Hansen

Add my @linux2go.dk address to .mailmap

Revision history for this message
Soren Hansen (soren) wrote :

I suck.

Note to self: Running tests before committing to make sure everything is fine isn't enough. *sigh*

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file '.mailmap'
2--- .mailmap 2010-11-23 21:34:53 +0000
3+++ .mailmap 2010-12-20 21:55:17 +0000
4@@ -19,6 +19,7 @@
5 <mordred@inaugust.com> <mordred@hudson>
6 <paul@openstack.org> <pvoccio@castor.local>
7 <paul@openstack.org> <paul.voccio@rackspace.com>
8+<soren.hansen@rackspace.com> <soren@linux2go.dk>
9 <todd@ansolabs.com> <todd@lapex>
10 <todd@ansolabs.com> <todd@rubidine.com>
11 <vishvananda@gmail.com> <vishvananda@yahoo.com>
12
13=== modified file 'MANIFEST.in'
14--- MANIFEST.in 2010-11-20 21:26:15 +0000
15+++ MANIFEST.in 2010-12-20 21:55:17 +0000
16@@ -13,7 +13,7 @@
17 include nova/compute/fakevirtinstance.xml
18 include nova/compute/interfaces.template
19 include nova/virt/interfaces.template
20-include nova/virt/libvirt.*.xml.template
21+include nova/virt/libvirt*.xml.template
22 include nova/tests/CA/
23 include nova/tests/CA/cacert.pem
24 include nova/tests/CA/private/
25
26=== modified file 'nova/api/ec2/cloud.py'
27--- nova/api/ec2/cloud.py 2010-12-09 17:30:03 +0000
28+++ nova/api/ec2/cloud.py 2010-12-20 21:55:17 +0000
29@@ -751,7 +751,7 @@
30 kwargs['image_id'],
31 min_count=int(kwargs.get('min_count', max_count)),
32 max_count=max_count,
33- kernel_id=kwargs.get('kernel_id'),
34+ kernel_id=kwargs.get('kernel_id', None),
35 ramdisk_id=kwargs.get('ramdisk_id'),
36 display_name=kwargs.get('display_name'),
37 description=kwargs.get('display_description'),
38
39=== modified file 'nova/compute/api.py'
40--- nova/compute/api.py 2010-12-14 21:56:42 +0000
41+++ nova/compute/api.py 2010-12-20 21:55:17 +0000
42@@ -73,14 +73,19 @@
43 is_vpn = image_id == FLAGS.vpn_image_id
44 if not is_vpn:
45 image = self.image_service.show(context, image_id)
46+
47+ # If kernel_id/ramdisk_id isn't explicitly set in API call
48+ # we take the defaults from the image's metadata
49 if kernel_id is None:
50- kernel_id = image.get('kernelId', FLAGS.default_kernel)
51+ kernel_id = image.get('kernelId', None)
52 if ramdisk_id is None:
53- ramdisk_id = image.get('ramdiskId', FLAGS.default_ramdisk)
54+ ramdisk_id = image.get('ramdiskId', None)
55
56 # Make sure we have access to kernel and ramdisk
57- self.image_service.show(context, kernel_id)
58- self.image_service.show(context, ramdisk_id)
59+ if kernel_id:
60+ self.image_service.show(context, kernel_id)
61+ if ramdisk_id:
62+ self.image_service.show(context, ramdisk_id)
63
64 if security_group is None:
65 security_group = ['default']
66@@ -103,8 +108,8 @@
67 base_options = {
68 'reservation_id': utils.generate_uid('r'),
69 'image_id': image_id,
70- 'kernel_id': kernel_id,
71- 'ramdisk_id': ramdisk_id,
72+ 'kernel_id': kernel_id or '',
73+ 'ramdisk_id': ramdisk_id or '',
74 'state_description': 'scheduling',
75 'user_id': context.user_id,
76 'project_id': context.project_id,
77
78=== modified file 'nova/compute/disk.py'
79--- nova/compute/disk.py 2010-12-08 20:59:22 +0000
80+++ nova/compute/disk.py 2010-12-20 21:55:17 +0000
81@@ -106,6 +106,13 @@
82 % (outfile, local_type, local_first, local_last))
83
84
85+def extend(image, size, execute):
86+ file_size = os.path.getsize(image)
87+ if file_size >= size:
88+ return
89+ return execute('truncate -s size %s' % (image,))
90+
91+
92 def inject_data(image, key=None, net=None, partition=None, execute=None):
93 """Injects a ssh key and optionally net data into a disk image.
94
95@@ -115,7 +122,7 @@
96 If partition is not specified it mounts the image as a single partition.
97
98 """
99- out, err = execute('sudo losetup -f --show %s' % image)
100+ out, err = execute('sudo losetup --find --show %s' % image)
101 if err:
102 raise exception.Error('Could not attach image to loopback: %s' % err)
103 device = out.strip()
104@@ -129,6 +136,15 @@
105 partition)
106 else:
107 mapped_device = device
108+
109+ # We can only loopback mount raw images. If the device isn't there,
110+ # it's normally because it's a .vmdk or a .vdi etc
111+ if not os.path.exists(mapped_device):
112+ raise exception.Error('Mapped device was not found (we can'
113+ ' only inject raw disk images): %s' %
114+ mapped_device)
115+
116+ # Configure ext2fs so that it doesn't auto-check every N boots
117 out, err = execute('sudo tune2fs -c 0 -i 0 %s' % mapped_device)
118
119 tmpdir = tempfile.mkdtemp()
120@@ -156,7 +172,7 @@
121 execute('sudo kpartx -d %s' % device)
122 finally:
123 # remove loopback
124- execute('sudo losetup -d %s' % device)
125+ execute('sudo losetup --detach %s' % device)
126
127
128 def _inject_key_into_fs(key, fs, execute=None):
129@@ -165,7 +181,7 @@
130 key is an ssh key string.
131 fs is the path to the base of the filesystem into which to inject the key.
132 """
133- sshdir = os.path.join(os.path.join(fs, 'root'), '.ssh')
134+ sshdir = os.path.join(fs, 'root', '.ssh')
135 execute('sudo mkdir -p %s' % sshdir) # existing dir doesn't matter
136 execute('sudo chown root %s' % sshdir)
137 execute('sudo chmod 700 %s' % sshdir)
138
139=== modified file 'nova/flags.py'
140--- nova/flags.py 2010-12-10 00:05:13 +0000
141+++ nova/flags.py 2010-12-20 21:55:17 +0000
142@@ -235,12 +235,11 @@
143
144 DEFINE_string('default_image', 'ami-11111',
145 'default image to use, testing only')
146-DEFINE_string('default_kernel', 'aki-11111',
147- 'default kernel to use, testing only')
148-DEFINE_string('default_ramdisk', 'ari-11111',
149- 'default ramdisk to use, testing only')
150 DEFINE_string('default_instance_type', 'm1.small',
151 'default instance type to use, testing only')
152+DEFINE_string('null_kernel', 'nokernel',
153+ 'kernel image that indicates not to use a kernel,'
154+ ' but to use a raw disk image instead')
155
156 DEFINE_string('vpn_image_id', 'ami-CLOUDPIPE', 'AMI for cloudpipe vpn server')
157 DEFINE_string('vpn_key_suffix',
158
159=== modified file 'nova/tests/virt_unittest.py'
160--- nova/tests/virt_unittest.py 2010-12-10 00:05:13 +0000
161+++ nova/tests/virt_unittest.py 2010-12-20 21:55:17 +0000
162@@ -40,19 +40,51 @@
163 self.network = utils.import_object(FLAGS.network_manager)
164 FLAGS.instances_path = ''
165
166- def test_get_uri_and_template(self):
167- ip = '10.11.12.13'
168-
169- instance = {'internal_id': 1,
170- 'memory_kb': '1024000',
171- 'basepath': '/some/path',
172- 'bridge_name': 'br100',
173- 'mac_address': '02:12:34:46:56:67',
174- 'vcpus': 2,
175- 'project_id': 'fake',
176- 'bridge': 'br101',
177- 'instance_type': 'm1.small'}
178-
179+ test_ip = '10.11.12.13'
180+ test_instance = {'memory_kb': '1024000',
181+ 'basepath': '/some/path',
182+ 'bridge_name': 'br100',
183+ 'mac_address': '02:12:34:46:56:67',
184+ 'vcpus': 2,
185+ 'project_id': 'fake',
186+ 'bridge': 'br101',
187+ 'instance_type': 'm1.small'}
188+
189+ def test_xml_and_uri_no_ramdisk_no_kernel(self):
190+ instance_data = dict(self.test_instance)
191+ self.do_test_xml_and_uri(instance_data,
192+ expect_kernel=False, expect_ramdisk=False)
193+
194+ def test_xml_and_uri_no_ramdisk(self):
195+ instance_data = dict(self.test_instance)
196+ instance_data['kernel_id'] = 'aki-deadbeef'
197+ self.do_test_xml_and_uri(instance_data,
198+ expect_kernel=True, expect_ramdisk=False)
199+
200+ def test_xml_and_uri_no_kernel(self):
201+ instance_data = dict(self.test_instance)
202+ instance_data['ramdisk_id'] = 'ari-deadbeef'
203+ self.do_test_xml_and_uri(instance_data,
204+ expect_kernel=False, expect_ramdisk=False)
205+
206+ def test_xml_and_uri(self):
207+ instance_data = dict(self.test_instance)
208+ instance_data['ramdisk_id'] = 'ari-deadbeef'
209+ instance_data['kernel_id'] = 'aki-deadbeef'
210+ self.do_test_xml_and_uri(instance_data,
211+ expect_kernel=True, expect_ramdisk=True)
212+
213+ def test_xml_and_uri_rescue(self):
214+ instance_data = dict(self.test_instance)
215+ instance_data['ramdisk_id'] = 'ari-deadbeef'
216+ instance_data['kernel_id'] = 'aki-deadbeef'
217+ self.do_test_xml_and_uri(instance_data,
218+ expect_kernel=True, expect_ramdisk=True,
219+ rescue=True)
220+
221+ def do_test_xml_and_uri(self, instance,
222+ expect_ramdisk, expect_kernel,
223+ rescue=False):
224 user_context = context.RequestContext(project=self.project,
225 user=self.user)
226 instance_ref = db.instance_create(user_context, instance)
227@@ -60,13 +92,14 @@
228 self.network.set_network_host(context.get_admin_context(),
229 network_ref['id'])
230
231- fixed_ip = {'address': ip,
232+ fixed_ip = {'address': self.test_ip,
233 'network_id': network_ref['id']}
234
235 ctxt = context.get_admin_context()
236 fixed_ip_ref = db.fixed_ip_create(ctxt, fixed_ip)
237- db.fixed_ip_update(ctxt, ip, {'allocated': True,
238- 'instance_id': instance_ref['id']})
239+ db.fixed_ip_update(ctxt, self.test_ip,
240+ {'allocated': True,
241+ 'instance_id': instance_ref['id']})
242
243 type_uri_map = {'qemu': ('qemu:///system',
244 [(lambda t: t.find('.').get('type'), 'qemu'),
245@@ -78,23 +111,71 @@
246 (lambda t: t.find('./devices/emulator'), None)]),
247 'uml': ('uml:///system',
248 [(lambda t: t.find('.').get('type'), 'uml'),
249- (lambda t: t.find('./os/type').text, 'uml')])}
250+ (lambda t: t.find('./os/type').text, 'uml')]),
251+ 'xen': ('xen:///',
252+ [(lambda t: t.find('.').get('type'), 'xen'),
253+ (lambda t: t.find('./os/type').text, 'linux')]),
254+ }
255+
256+ for hypervisor_type in ['qemu', 'kvm', 'xen']:
257+ check_list = type_uri_map[hypervisor_type][1]
258+
259+ if rescue:
260+ check = (lambda t: t.find('./os/kernel').text.split('/')[1],
261+ 'rescue-kernel')
262+ check_list.append(check)
263+ check = (lambda t: t.find('./os/initrd').text.split('/')[1],
264+ 'rescue-ramdisk')
265+ check_list.append(check)
266+ else:
267+ if expect_kernel:
268+ check = (lambda t: t.find('./os/kernel').text.split('/'
269+ )[1], 'kernel')
270+ else:
271+ check = (lambda t: t.find('./os/kernel'), None)
272+ check_list.append(check)
273+
274+ if expect_ramdisk:
275+ check = (lambda t: t.find('./os/initrd').text.split('/'
276+ )[1], 'ramdisk')
277+ else:
278+ check = (lambda t: t.find('./os/initrd'), None)
279+ check_list.append(check)
280
281 common_checks = [
282 (lambda t: t.find('.').tag, 'domain'),
283- (lambda t: t.find('./devices/interface/filterref/parameter').\
284- get('name'), 'IP'),
285- (lambda t: t.find('./devices/interface/filterref/parameter').\
286- get('value'), '10.11.12.13')]
287+ (lambda t: t.find('./devices/interface/filterref/parameter'
288+ ).get('name'), 'IP'),
289+ (lambda t: t.find('./devices/interface/filterref/parameter'
290+ ).get('value'), '10.11.12.13'),
291+ (lambda t: t.findall('./devices/interface/filterref/parameter'
292+ )[1].get('name'), 'DHCPSERVER'),
293+ (lambda t: t.findall('./devices/interface/filterref/parameter'
294+ )[1].get('value'), '10.0.0.1'),
295+ (lambda t: t.find('./devices/serial/source').get('path'
296+ ).split('/')[1], 'console.log'),
297+ (lambda t: t.find('./memory').text, '2097152')]
298+
299+ if rescue:
300+ common_checks += [(lambda t: t.findall('./devices/disk/source'
301+ )[0].get('file').split('/')[1],
302+ 'rescue-disk'),
303+ (lambda t: t.findall('./devices/disk/source'
304+ )[1].get('file').split('/')[1],
305+ 'disk')]
306+ else:
307+ common_checks += [(lambda t: t.findall('./devices/disk/source'
308+ )[0].get('file').split('/')[1],
309+ 'disk')]
310
311 for (libvirt_type, (expected_uri, checks)) in type_uri_map.iteritems():
312 FLAGS.libvirt_type = libvirt_type
313 conn = libvirt_conn.LibvirtConnection(True)
314
315- uri, _template, _rescue = conn.get_uri_and_templates()
316+ uri = conn.get_uri()
317 self.assertEquals(uri, expected_uri)
318
319- xml = conn.to_xml(instance_ref)
320+ xml = conn.to_xml(instance_ref, rescue)
321 tree = xml_to_tree(xml)
322 for i, (check, expected_result) in enumerate(checks):
323 self.assertEqual(check(tree),
324@@ -106,6 +187,9 @@
325 expected_result,
326 '%s failed common check %d' % (xml, i))
327
328+ # This test is supposed to make sure we don't override a specifically
329+ # set uri
330+ #
331 # Deliberately not just assigning this string to FLAGS.libvirt_uri and
332 # checking against that later on. This way we make sure the
333 # implementation doesn't fiddle around with the FLAGS.
334@@ -114,7 +198,7 @@
335 for (libvirt_type, (expected_uri, checks)) in type_uri_map.iteritems():
336 FLAGS.libvirt_type = libvirt_type
337 conn = libvirt_conn.LibvirtConnection(True)
338- uri, _template, _rescue = conn.get_uri_and_templates()
339+ uri = conn.get_uri()
340 self.assertEquals(uri, testuri)
341
342 def tearDown(self):
343
344=== removed file 'nova/virt/libvirt.rescue.qemu.xml.template'
345--- nova/virt/libvirt.rescue.qemu.xml.template 2010-10-24 21:57:37 +0000
346+++ nova/virt/libvirt.rescue.qemu.xml.template 1970-01-01 00:00:00 +0000
347@@ -1,37 +0,0 @@
348-<domain type='%(type)s'>
349- <name>%(name)s</name>
350- <os>
351- <type>hvm</type>
352- <kernel>%(basepath)s/rescue-kernel</kernel>
353- <initrd>%(basepath)s/rescue-ramdisk</initrd>
354- <cmdline>root=/dev/vda1 console=ttyS0</cmdline>
355- </os>
356- <features>
357- <acpi/>
358- </features>
359- <memory>%(memory_kb)s</memory>
360- <vcpu>%(vcpus)s</vcpu>
361- <devices>
362- <disk type='file'>
363- <source file='%(basepath)s/rescue-disk'/>
364- <target dev='vda' bus='virtio'/>
365- </disk>
366- <disk type='file'>
367- <source file='%(basepath)s/disk'/>
368- <target dev='vdb' bus='virtio'/>
369- </disk>
370- <interface type='bridge'>
371- <source bridge='%(bridge_name)s'/>
372- <mac address='%(mac_address)s'/>
373- <!-- <model type='virtio'/> CANT RUN virtio network right now -->
374- <filterref filter="nova-instance-%(name)s">
375- <parameter name="IP" value="%(ip_address)s" />
376- <parameter name="DHCPSERVER" value="%(dhcp_server)s" />
377- </filterref>
378- </interface>
379- <serial type="file">
380- <source path='%(basepath)s/console.log'/>
381- <target port='1'/>
382- </serial>
383- </devices>
384-</domain>
385
386=== removed file 'nova/virt/libvirt.rescue.uml.xml.template'
387--- nova/virt/libvirt.rescue.uml.xml.template 2010-09-20 08:50:08 +0000
388+++ nova/virt/libvirt.rescue.uml.xml.template 1970-01-01 00:00:00 +0000
389@@ -1,26 +0,0 @@
390-<domain type='%(type)s'>
391- <name>%(name)s</name>
392- <memory>%(memory_kb)s</memory>
393- <os>
394- <type>%(type)s</type>
395- <kernel>/usr/bin/linux</kernel>
396- <root>/dev/ubda1</root>
397- </os>
398- <devices>
399- <disk type='file'>
400- <source file='%(basepath)s/rescue-disk'/>
401- <target dev='ubd0' bus='uml'/>
402- </disk>
403- <disk type='file'>
404- <source file='%(basepath)s/disk'/>
405- <target dev='ubd1' bus='uml'/>
406- </disk>
407- <interface type='bridge'>
408- <source bridge='%(bridge_name)s'/>
409- <mac address='%(mac_address)s'/>
410- </interface>
411- <console type="file">
412- <source path='%(basepath)s/console.log'/>
413- </console>
414- </devices>
415-</domain>
416
417=== removed file 'nova/virt/libvirt.rescue.xen.xml.template'
418--- nova/virt/libvirt.rescue.xen.xml.template 2010-10-25 00:18:24 +0000
419+++ nova/virt/libvirt.rescue.xen.xml.template 1970-01-01 00:00:00 +0000
420@@ -1,34 +0,0 @@
421-<domain type='%(type)s'>
422- <name>%(name)s</name>
423- <os>
424- <type>linux</type>
425- <kernel>%(basepath)s/kernel</kernel>
426- <initrd>%(basepath)s/ramdisk</initrd>
427- <root>/dev/xvda1</root>
428- <cmdline>ro</cmdline>
429- </os>
430- <features>
431- <acpi/>
432- </features>
433- <memory>%(memory_kb)s</memory>
434- <vcpu>%(vcpus)s</vcpu>
435- <devices>
436- <disk type='file'>
437- <source file='%(basepath)s/rescue-disk'/>
438- <target dev='sda' />
439- </disk>
440- <disk type='file'>
441- <source file='%(basepath)s/disk'/>
442- <target dev='sdb' />
443- </disk>
444- <interface type='bridge'>
445- <source bridge='%(bridge_name)s'/>
446- <mac address='%(mac_address)s'/>
447- </interface>
448- <console type="file">
449- <source path='%(basepath)s/console.log'/>
450- <target port='1'/>
451- </console>
452- </devices>
453-</domain>
454-
455
456=== removed file 'nova/virt/libvirt.uml.xml.template'
457--- nova/virt/libvirt.uml.xml.template 2010-09-27 11:13:29 +0000
458+++ nova/virt/libvirt.uml.xml.template 1970-01-01 00:00:00 +0000
459@@ -1,26 +0,0 @@
460-<domain type='%(type)s'>
461- <name>%(name)s</name>
462- <memory>%(memory_kb)s</memory>
463- <os>
464- <type>%(type)s</type>
465- <kernel>/usr/bin/linux</kernel>
466- <root>/dev/ubda1</root>
467- </os>
468- <devices>
469- <disk type='file'>
470- <source file='%(basepath)s/disk'/>
471- <target dev='ubd0' bus='uml'/>
472- </disk>
473- <interface type='bridge'>
474- <source bridge='%(bridge_name)s'/>
475- <mac address='%(mac_address)s'/>
476- <filterref filter="nova-instance-%(name)s">
477- <parameter name="IP" value="%(ip_address)s" />
478- <parameter name="DHCPSERVER" value="%(dhcp_server)s" />
479- </filterref>
480- </interface>
481- <console type="file">
482- <source path='%(basepath)s/console.log'/>
483- </console>
484- </devices>
485-</domain>
486
487=== removed file 'nova/virt/libvirt.xen.xml.template'
488--- nova/virt/libvirt.xen.xml.template 2010-10-25 00:18:24 +0000
489+++ nova/virt/libvirt.xen.xml.template 1970-01-01 00:00:00 +0000
490@@ -1,30 +0,0 @@
491-<domain type='%(type)s'>
492- <name>%(name)s</name>
493- <os>
494- <type>linux</type>
495- <kernel>%(basepath)s/kernel</kernel>
496- <initrd>%(basepath)s/ramdisk</initrd>
497- <root>/dev/xvda1</root>
498- <cmdline>ro</cmdline>
499- </os>
500- <features>
501- <acpi/>
502- </features>
503- <memory>%(memory_kb)s</memory>
504- <vcpu>%(vcpus)s</vcpu>
505- <devices>
506- <disk type='file'>
507- <source file='%(basepath)s/disk'/>
508- <target dev='sda' />
509- </disk>
510- <interface type='bridge'>
511- <source bridge='%(bridge_name)s'/>
512- <mac address='%(mac_address)s'/>
513- </interface>
514- <console type="file">
515- <source path='%(basepath)s/console.log'/>
516- <target port='1'/>
517- </console>
518- </devices>
519-</domain>
520-
521
522=== renamed file 'nova/virt/libvirt.qemu.xml.template' => 'nova/virt/libvirt.xml.template'
523--- nova/virt/libvirt.qemu.xml.template 2010-09-27 11:13:29 +0000
524+++ nova/virt/libvirt.xml.template 2010-12-20 21:55:17 +0000
525@@ -1,32 +1,75 @@
526-<domain type='%(type)s'>
527- <name>%(name)s</name>
528+<domain type='${type}'>
529+ <name>${name}</name>
530+ <memory>${memory_kb}</memory>
531 <os>
532- <type>hvm</type>
533- <kernel>%(basepath)s/kernel</kernel>
534- <initrd>%(basepath)s/ramdisk</initrd>
535- <cmdline>root=/dev/vda1 console=ttyS0</cmdline>
536+#if $type == 'uml'
537+ #set $disk_prefix = 'ubd'
538+ #set $disk_bus = 'uml'
539+ <type>uml</type>
540+ <kernel>/usr/bin/linux</kernel>
541+ <root>/dev/ubda1</root>
542+#else
543+ #if $type == 'xen'
544+ #set $disk_prefix = 'sd'
545+ #set $disk_bus = 'scsi'
546+ <type>linux</type>
547+ <root>/dev/xvda1</root>
548+ #else
549+ #set $disk_prefix = 'vd'
550+ #set $disk_bus = 'virtio'
551+ <type>hvm</type>
552+ #end if
553+ #if $getVar('rescue', False)
554+ <kernel>${basepath}/rescue-kernel</kernel>
555+ <initrd>${basepath}/rescue-ramdisk</initrd>
556+ #else
557+ #if $getVar('kernel', None)
558+ <kernel>${kernel}</kernel>
559+ #if $type == 'xen'
560+ <cmdline>ro</cmdline>
561+ #else
562+ <cmdline>root=/dev/vda1 console=ttyS0</cmdline>
563+ #end if
564+ #if $getVar('ramdisk', None)
565+ <initrd>${ramdisk}</initrd>
566+ #end if
567+ #else
568+ <boot dev="hd" />
569+ #end if
570+ #end if
571+#end if
572 </os>
573 <features>
574 <acpi/>
575 </features>
576- <memory>%(memory_kb)s</memory>
577- <vcpu>%(vcpus)s</vcpu>
578+ <vcpu>${vcpus}</vcpu>
579 <devices>
580- <disk type='file'>
581- <source file='%(basepath)s/disk'/>
582- <target dev='vda' bus='virtio'/>
583- </disk>
584+#if $getVar('rescue', False)
585+ <disk type='file'>
586+ <source file='${basepath}/rescue-disk'/>
587+ <target dev='${disk_prefix}a' bus='${disk_bus}'/>
588+ </disk>
589+ <disk type='file'>
590+ <source file='${basepath}/disk'/>
591+ <target dev='${disk_prefix}b' bus='${disk_bus}'/>
592+ </disk>
593+#else
594+ <disk type='file'>
595+ <source file='${basepath}/disk'/>
596+ <target dev='${disk_prefix}a' bus='${disk_bus}'/>
597+ </disk>
598+#end if
599 <interface type='bridge'>
600- <source bridge='%(bridge_name)s'/>
601- <mac address='%(mac_address)s'/>
602+ <source bridge='${bridge_name}'/>
603+ <mac address='${mac_address}'/>
604 <!-- <model type='virtio'/> CANT RUN virtio network right now -->
605- <filterref filter="nova-instance-%(name)s">
606- <parameter name="IP" value="%(ip_address)s" />
607- <parameter name="DHCPSERVER" value="%(dhcp_server)s" />
608+ <filterref filter="nova-instance-${name}">
609+ <parameter name="IP" value="${ip_address}" />
610+ <parameter name="DHCPSERVER" value="${dhcp_server}" />
611 </filterref>
612 </interface>
613 <serial type="file">
614- <source path='%(basepath)s/console.log'/>
615+ <source path='${basepath}/console.log'/>
616 <target port='1'/>
617 </serial>
618 </devices>
619
620=== modified file 'nova/virt/libvirt_conn.py'
621--- nova/virt/libvirt_conn.py 2010-12-16 00:31:32 +0000
622+++ nova/virt/libvirt_conn.py 2010-12-20 21:55:17 +0000
623@@ -27,12 +27,7 @@
624 :libvirt_type: Libvirt domain type. Can be kvm, qemu, uml, xen
625 (default: kvm).
626 :libvirt_uri: Override for the default libvirt URI (depends on libvirt_type).
627-:libvirt_xml_template: Libvirt XML Template (QEmu/KVM).
628-:libvirt_xen_xml_template: Libvirt XML Template (Xen).
629-:libvirt_uml_xml_template: Libvirt XML Template (User Mode Linux).
630-:libvirt_rescue_xml_template: XML template for rescue mode (KVM & QEMU).
631-:libvirt_rescue_xen_xml_template: XML templage for rescue mode (XEN).
632-:libvirt_rescue_uml_xml_template: XML template for rescue mode (UML).
633+:libvirt_xml_template: Libvirt XML Template.
634 :rescue_image_id: Rescue ami image (default: ami-rescue).
635 :rescue_kernel_id: Rescue aki image (default: aki-rescue).
636 :rescue_ramdisk_id: Rescue ari image (default: ari-rescue).
637@@ -62,36 +57,20 @@
638 from nova.compute import power_state
639 from nova.virt import images
640
641+from Cheetah.Template import Template
642+
643 libvirt = None
644 libxml2 = None
645
646
647 FLAGS = flags.FLAGS
648-flags.DEFINE_string('libvirt_rescue_xml_template',
649- utils.abspath('virt/libvirt.rescue.qemu.xml.template'),
650- 'Libvirt RESCUE XML Template for QEmu/KVM')
651-flags.DEFINE_string('libvirt_rescue_xen_xml_template',
652- utils.abspath('virt/libvirt.rescue.xen.xml.template'),
653- 'Libvirt RESCUE XML Template for xen')
654-flags.DEFINE_string('libvirt_rescue_uml_xml_template',
655- utils.abspath('virt/libvirt.rescue.uml.xml.template'),
656- 'Libvirt RESCUE XML Template for user-mode-linux')
657 # TODO(vish): These flags should probably go into a shared location
658 flags.DEFINE_string('rescue_image_id', 'ami-rescue', 'Rescue ami image')
659 flags.DEFINE_string('rescue_kernel_id', 'aki-rescue', 'Rescue aki image')
660 flags.DEFINE_string('rescue_ramdisk_id', 'ari-rescue', 'Rescue ari image')
661 flags.DEFINE_string('libvirt_xml_template',
662- utils.abspath('virt/libvirt.qemu.xml.template'),
663- 'Libvirt XML Template for QEmu/KVM')
664-flags.DEFINE_string('libvirt_xen_xml_template',
665- utils.abspath('virt/libvirt.xen.xml.template'),
666- 'Libvirt XML Template for Xen')
667-flags.DEFINE_string('libvirt_uml_xml_template',
668- utils.abspath('virt/libvirt.uml.xml.template'),
669- 'Libvirt XML Template for user-mode-linux')
670-flags.DEFINE_string('injected_network_template',
671- utils.abspath('virt/interfaces.template'),
672- 'Template file for injected network')
673+ utils.abspath('virt/libvirt.xml.template'),
674+ 'Libvirt XML Template')
675 flags.DEFINE_string('libvirt_type',
676 'kvm',
677 'Libvirt domain type (valid options are: '
678@@ -120,12 +99,9 @@
679 class LibvirtConnection(object):
680
681 def __init__(self, read_only):
682- (self.libvirt_uri,
683- template_file,
684- rescue_file) = self.get_uri_and_templates()
685+ self.libvirt_uri = self.get_uri()
686
687- self.libvirt_xml = open(template_file).read()
688- self.rescue_xml = open(rescue_file).read()
689+ self.libvirt_xml = open(FLAGS.libvirt_xml_template).read()
690 self._wrapped_conn = None
691 self.read_only = read_only
692
693@@ -148,20 +124,14 @@
694 return False
695 raise
696
697- def get_uri_and_templates(self):
698+ def get_uri(self):
699 if FLAGS.libvirt_type == 'uml':
700 uri = FLAGS.libvirt_uri or 'uml:///system'
701- template_file = FLAGS.libvirt_uml_xml_template
702- rescue_file = FLAGS.libvirt_rescue_uml_xml_template
703 elif FLAGS.libvirt_type == 'xen':
704 uri = FLAGS.libvirt_uri or 'xen:///'
705- template_file = FLAGS.libvirt_xen_xml_template
706- rescue_file = FLAGS.libvirt_rescue_xen_xml_template
707 else:
708 uri = FLAGS.libvirt_uri or 'qemu:///system'
709- template_file = FLAGS.libvirt_xml_template
710- rescue_file = FLAGS.libvirt_rescue_xml_template
711- return uri, template_file, rescue_file
712+ return uri
713
714 def _connect(self, uri, read_only):
715 auth = [[libvirt.VIR_CRED_AUTHNAME, libvirt.VIR_CRED_NOECHOPROMPT],
716@@ -442,18 +412,28 @@
717 if not os.path.exists(basepath('disk')):
718 images.fetch(inst.image_id, basepath('disk-raw'), user,
719 project)
720- if not os.path.exists(basepath('kernel')):
721- images.fetch(inst.kernel_id, basepath('kernel'), user,
722- project)
723- if not os.path.exists(basepath('ramdisk')):
724- images.fetch(inst.ramdisk_id, basepath('ramdisk'), user,
725- project)
726+
727+ if inst['kernel_id']:
728+ if not os.path.exists(basepath('kernel')):
729+ images.fetch(inst['kernel_id'], basepath('kernel'),
730+ user, project)
731+ if inst['ramdisk_id']:
732+ if not os.path.exists(basepath('ramdisk')):
733+ images.fetch(inst['ramdisk_id'], basepath('ramdisk'),
734+ user, project)
735
736 def execute(cmd, process_input=None, check_exit_code=True):
737 return utils.execute(cmd=cmd,
738 process_input=process_input,
739 check_exit_code=check_exit_code)
740
741+ # For now, we assume that if we're not using a kernel, we're using a
742+ # partitioned disk image where the target partition is the first
743+ # partition
744+ target_partition = None
745+ if not inst['kernel_id']:
746+ target_partition = "1"
747+
748 key = str(inst['key_data'])
749 net = None
750 network_ref = db.network_get_by_instance(context.get_admin_context(),
751@@ -473,12 +453,20 @@
752 inst['name'], inst.image_id)
753 if net:
754 logging.info('instance %s: injecting net into image %s',
755- inst['name'], inst.image_id)
756- disk.inject_data(basepath('disk-raw'), key, net,
757- execute=execute)
758+ inst['name'], inst.image_id)
759+ try:
760+ disk.inject_data(basepath('disk-raw'), key, net,
761+ partition=target_partition,
762+ execute=execute)
763+ except Exception as e:
764+ # This could be a windows image, or a vmdk format disk
765+ logging.warn('instance %s: ignoring error injecting data'
766+ ' into image %s (%s)',
767+ inst['name'], inst.image_id, e)
768
769- if os.path.exists(basepath('disk')):
770- utils.execute('rm -f %s' % basepath('disk'))
771+ if inst['kernel_id']:
772+ if os.path.exists(basepath('disk')):
773+ utils.execute('rm -f %s' % basepath('disk'))
774
775 local_bytes = (instance_types.INSTANCE_TYPES[inst.instance_type]
776 ['local_gb']
777@@ -487,8 +475,13 @@
778 resize = True
779 if inst['instance_type'] == 'm1.tiny' or prefix == 'rescue-':
780 resize = False
781- disk.partition(basepath('disk-raw'), basepath('disk'),
782- local_bytes, resize, execute=execute)
783+
784+ if inst['kernel_id']:
785+ disk.partition(basepath('disk-raw'), basepath('disk'),
786+ local_bytes, resize, execute=execute)
787+ else:
788+ os.rename(basepath('disk-raw'), basepath('disk'))
789+ disk.extend(basepath('disk'), local_bytes, execute=execute)
790
791 if FLAGS.libvirt_type == 'uml':
792 utils.execute('sudo chown root %s' % basepath('disk'))
793@@ -514,14 +507,21 @@
794 'bridge_name': network['bridge'],
795 'mac_address': instance['mac_address'],
796 'ip_address': ip_address,
797- 'dhcp_server': dhcp_server}
798- if rescue:
799- libvirt_xml = self.rescue_xml % xml_info
800- else:
801- libvirt_xml = self.libvirt_xml % xml_info
802+ 'dhcp_server': dhcp_server,
803+ 'rescue': rescue}
804+ if not rescue:
805+ if instance['kernel_id']:
806+ xml_info['kernel'] = xml_info['basepath'] + "/kernel"
807+
808+ if instance['ramdisk_id']:
809+ xml_info['ramdisk'] = xml_info['basepath'] + "/ramdisk"
810+
811+ xml_info['disk'] = xml_info['basepath'] + "/disk"
812+
813+ xml = str(Template(self.libvirt_xml, searchList=[xml_info]))
814 logging.debug('instance %s: finished toXML method', instance['name'])
815
816- return libvirt_xml
817+ return xml
818
819 def get_info(self, instance_name):
820 try:
821
822=== modified file 'tools/pip-requires'
823--- tools/pip-requires 2010-12-13 17:02:27 +0000
824+++ tools/pip-requires 2010-12-20 21:55:17 +0000
825@@ -2,6 +2,7 @@
826 pep8==0.5.0
827 pylint==0.19
828 IPy==0.70
829+Cheetah==2.4.2.1
830 M2Crypto==0.20.2
831 amqplib==0.6.1
832 anyjson==0.2.4