RunInstances uses same ID if previous request failed

Bug #677475 reported by Sandy Walsh
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Ryan Lucio

Bug Description

Ubuntu 10.10-64

% euca-run-instances ami-tiny --kernel aki-lucid --ramdisk ari-lucid -k nova_key

if this fails due to networking problems or insufficient disk space (the two conditions I hit) I cannot re-run the command due to a duplicate instance-id database error (mysql). The nova db must be dropped and recreated to get past this.

Doing euca-terminate-instance does not resolve the problem.

% euca-run-instances ami-tiny --kernel aki-lucid --ramdisk ari-lucid -k nova_key
IntegrityError: (IntegrityError) (1062, "Duplicate entry '2147483647' for key 'internal_id'") 'INSERT INTO instances (created_at, updated_at, deleted_at, deleted, internal_id, admin_pass, user_id, project_id, image_id, kernel_id, ramdisk_id, server_name, launch_index, key_name, key_data, state, state_description, memory_mb, vcpus, local_gb, hostname, host, instance_type, user_data, reservation_id, mac_address, scheduled_at, launched_at, terminated_at, display_name, display_description) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)' (datetime.datetime(2010, 11, 19, 14, 3, 40, 3977), None, None, False, 2874477778, None, 'sandy', 'darksecret', u'ami-tiny', u'aki-lucid', u'ari-lucid', None, 0, u'nova_key', 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDYCSAjjqr3mlEvS7f8tu9GGd+hVOQO2c4U/BvFNeWJmdisXr122u3SWsqTGfzyU7067oqIiGvXCipnEi6AyPE5PKfRK68MQLb52jYMI5fHGWapb4I/hS6LKfbt0NXa8ZGnjCmvM5y6P1eJvs0W1+6/NAU31/MHkn2QcT3IFXahaw== swalsh@novadev\n', None, 'scheduling', 2048, 1, 20, None, None, u'm1.small', '', 'r-ahpmyjce', '02:16:3e:64:05:7b', None, None, None, None, None)

PS> in some situations ./instances/instance#### was not removed either, but I can't consistently repo this.

Related branches

Revision history for this message
Soren Hansen (soren) wrote :

I'm completely baffled why would it try to create another instance with the same instance_id. Those are supposed to be unique (picked at random, and checked for uniqueness).

Anyways, we do need a better way to handle failed RunInstances calls. The end users will expect their virtual machines to eventually start, so just nuking the entry from the DB seems rude.

Revision history for this message
Soren Hansen (soren) wrote :

Thinking about this some more, I firmly believe the only bug here is that it's trying to use the same instance ID. The failed one shouldn't just get cleared out. Something needs to take ownership of the failed request and take an appropriate action (set it to "failed" and move on with life, or retry it at certain intervals, or something), so I'm changing the bug title to reflect this. If you disagree, please don't hesitate to complain :)

summary: - Database not cleaned up when run-instances fails
+ RunInstances uses same ID if previous request failed
Changed in nova:
importance: Undecided → Medium
Changed in nova:
assignee: nobody → Sandy Walsh (sandy-walsh)
status: New → In Progress
Revision history for this message
Ryan Lucio (rlucio) wrote :

This issue is only about duplicates on the surface. The real source of the problem is that the function that generates the internal_id assumes it the field is a uint32 but the database column is defaulting to be the equivalent of int32. So any value for internal_id that is greater than int32_max will be truncated to that value.

The reason you see the duplicate value error is because these truncated values are all the same... but the database column is set to only allow unique values. A big hint to this is that the error is always "Duplicate entry '2147483647' which is int32_max.

Changed in nova:
assignee: Sandy Walsh (sandy-walsh) → Ryan Lucio (rlucio)
Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2011.1
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.