Invalid device name /dev/sdf1 on EC2 spot request using an EBS snapshot

[jldonovan3]

I am requesting spot instances that have an EBS snapshot attached that contains project files to render. I get a bad-parameters failure on the spot request with the following state: failed: Invalid device name /dev/sdf1 The output from the brenda-run command gives me my Snapshot ID, and what appears to be the device mapping:
Project EBS snapshot: [('snap-xxxxxxxx', '/dev/sdf1')]

Would the spot requests be failing because brenda (or boto) is device mapping to something AWS is saying is invalid? brenda is assigning block_device_map to bdm , and in brenda/aws.py, bdm is using boto.ec2.blockdevicemapping.BlockDeviceMapping()

from brenda/aws.py:

def blk_dev_map(opts, conf, itype, snapshots):
    if not int(conf.get('NO_EBS', '0')):
        bdm = boto.ec2.blockdevicemapping.BlockDeviceMapping()
        snap = project_ebs_snapshot(conf)
        snap_id = translate_snapshot_name(conf, snap, snapshots)
        snap_description = []
        if snap_id:
            dev = utils.blkdev(0)
            bdm[dev] = boto.ec2.blockdevicemapping.EBSBlockDeviceType(snapshot_id=snap_id, delete_on_termination=True)
            snap_description.append((snap, snap_id, dev))
        i = 0
        for k in additional_ebs_iterator(conf):
            i += 1
            snap = parse_ebs_url(conf[k].split(',')[0])
            snap_id = translate_snapshot_name(conf, snap, snapshots)
            if snap_id:
                dev = utils.blkdev(i)
                bdm[dev] = boto.ec2.blockdevicemapping.EBSBlockDeviceType(snapshot_id=snap_id, delete_on_termination=True)
                snap_description.append((snap, snap_id, dev))
        istore_dev = add_instance_store(opts, conf, bdm, itype)
        return bdm, snap_description, istore_dev
    else:
        return None, None, None

In the file used to start up the instances, brenda/run.py, bdm is given as the block_device_map

from brenda/run.py:

def spot(opts, conf):
    ami_id = utils.get_opt(opts.ami, conf, 'AMI_ID', default=AMI_ID, must_exist=True)
    price = utils.get_opt(opts.price, conf, 'BID_PRICE', must_exist=True)
    reqtype = 'persistent' if opts.persistent else 'one-time'
    itype = brenda_instance_type(opts, conf)
    snapshots = aws.get_snapshots(conf)
    bdm, snap_description, istore_dev = aws.blk_dev_map(opts, conf, itype, snapshots)
    script = startup_script(opts, conf, istore_dev)
    user_data = None
    if not opts.idle:
        user_data = script
    ssh_key_name = conf.get("SSH_KEY_NAME", "brenda")
    sec_groups = (conf.get("SECURITY_GROUP", "brenda"),)
    run_args = {
        'image_id'      : ami_id,
        'price'         : price,
        'type'          : reqtype,
        'count'         : opts.n_instances,
        'instance_type' : itype,
        'user_data'     : user_data,
        'key_name'      : ssh_key_name,
        'security_groups' : sec_groups,
        'block_device_map' : bdm,
        }

    print "----------------------------"
    print "AMI ID:", ami_id
    print "Max bid price", price
    print "Request type:", reqtype
    print "Instance type:", itype
    print "Instance count:", opts.n_instances
    if snap_description:
        print "Project EBS snapshot:", snap_description
    if istore_dev:
        print "Instance store device:", istore_dev
    print "SSH key name:", ssh_key_name
    print "Security groups:", sec_groups
    print_script(opts, conf, script)
    aws.get_done(opts, conf) # sanity check on DONE var
    if not opts.dry_run:
        ec2 = aws.get_ec2_conn(conf)
        reservation = ec2.request_spot_instances(**run_args)
        print reservation

It seems similar to what's going on in this thread, but I'm not sure how to implement the solution in my case with brenda.

boto version: 2.49.0

Appreciate any help!

[gwhobbs]

@jldonovan3 – did you ever figure this out? I am running into the same issue.

[gwhobbs]

@jldonovAfter a bit more digging, I found this in the AWS docs:

HVM AMIs do not support the use of trailing numbers on device names, except for /dev/sda1, which is reserved for the root device, and /dev/sda2.

Changing the device naming to exclude the trailing number seems to have fixed the problem for me. See details on the commit or check out my fork, which fixes a few other things as well. I'd be happy to submit a pull request but it doesn't appear that this repo is currently being maintained.

[jldonovan3]

@gwhobbs Sorry about the delayed reply, had to dig up some stuff since this was a while ago. But yes, I think you came to the same solution I think I came up with. There's an bunch of device names in brenda/utils.py, and the first one it takes is f1, so it tries to name the device sdf1, etc.

I removed the number from the first device name from a local branch I was messing around with:

def blkdev(index, istore=False, mount_form=False): if istore: # instance store devs = ('b', 'c', 'd', 'e') else: # EBS devs = ( 'f', 'g1', 'h1', 'i1', 'j1', 'k1', 'l1', 'm1', 'n1', 'o1', 'p1', 'f2', 'g2', 'h2', 'i2', 'j2', 'k2', 'l2', 'm2', 'n2', 'o2', 'p2', 'f3', 'g3', 'h3', 'i3', 'j3', 'k3', 'l3', 'm3', 'n3', 'o3', 'p3', 'f4', 'g4', 'h4', 'i4', 'j4', 'k4', 'l4', 'm4', 'n4', 'o4', 'p4', 'f5', 'g5', 'h5', 'i5', 'j5', 'k5', 'l5', 'm5', 'n5', 'o5', 'p5', 'f6', 'g6', 'h6', 'i6', 'j6', 'k6', 'l6', 'm6', 'n6', 'o6', 'p6', ) if mount_form: return '/dev/xvd' + devs[index] else: return '/dev/sd' + devs[index]

Thanks for updating the tool! I'm not a programmer by trade and only tinkered with it to get it to do what I needed. Glad to see somebody's taking it under their wing.

[gwhobbs]

@jldonovan3 – thanks for the reply! I'm glad you were able to find a solution at the time. Yeah, this is a super useful tool but there are a few issues that seem to have crept up over the years.