Partitions not created

[marshall-mcmullen]

Greetings, just found this project and am amazed at what you've built so far. LOVE the idea of being able to partition disks natively inside Go without having to have all the tools installed on a machine and having to shell out. Great stuff.

Problem is that the partitions don't appear to be getting created. Maybe I'm doing something wrong? I'd really love if you can double check how I'm using this and see if anything obvious jumps out at you.

Here's a simplified version of what I'm doing:

numPartitions := 2
drive, err := diskfs.Open("/dev/nvme0n1")
check(err)

table := &gpt.Table{
  Partitions: make([]*gpt.Partition, numPartitions)),
}

table.Partitions[0] = &gpt.Partition{
  Name: "boot",
  Type:   "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
  Size: 104857600
}

table.Partitions[1] = &gpt.Partition{
  Name: "root",
  Type:   "0FC63DAF-8483-4772-8E79-3D69D8477DE4",
  Size: 128027547648,
}

err = drive.Partition(table)
check(err)

log.Info("Drive info POST PARTITIONING", log.Fields{
  "drive": drive,
})

There are NO errors reported. And the log message at the end looks like everything is proper:

{
  "msg": "Drive info POST PARTITIONING",
  "time": "2022-12-02T00:01:07Z"
  "drive": {
    "DefaultBlocks": false,
    "File": {},
    "Info": {},
    "LogicalBlocksize": 512,
    "PhysicalBlocksize": 512,
    "Size": 512110190592,
    "Table": {
      "GUID": "55d819bf-041d-4339-afd5-300e9de6f038",
      "LogicalSectorSize": 512,
      "Partitions": [
        {
          "Attributes": 0,
          "End": 205311,
          "GUID": "CC092F2D-5D89-4962-A63A-9C90BA8B03A1",
          "Name": "boot",
          "Size": 104857600,
          "Start": 512,
          "Type": "C12A7328-F81F-11D2-BA4B-00A0C93EC93B"
        },
        {
          "Attributes": 0,
          "End": 250259115,
          "GUID": "2B421E42-1389-4795-9B98-27525B773CBC",
          "Name": "root",
          "Size": 128027547648,
          "Start": 205312,
          "Type": "0FC63DAF-8483-4772-8E79-3D69D8477DE4"
        }
      ],
      "PhysicalSectorSize": 512,
      "ProtectiveMBR": false
    },
    "Type": 1,
    "Writable": true
  }
}

However, when I exit the Go program above, and try to print the partitions via fdisk they are not there:

/ # fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 477 GB, 512110190592 bytes, 1000215216 sectors
488386 cylinders, 64 heads, 32 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Disk /dev/nvme0n1 doesn't contain a valid partition table

This is running inside an Alpine Install ISO with busybox.

Thank you for any help you can provide!!

[marshall-mcmullen]

I have also tried partprobe to rescan for the partitions but they are not there. I've also looked in /sys/block and they are not there. It really feels like the partitions are not actually getting created as opposed to them just not being visible (yet).

I feel like I must be missing another important step here after calling err = drive.Partition(table) to actually create the partitions. But I looked at the code and don't see any examples of doing anything after that. There's no Close or Flush or Write that I'm missing is there?

[deitch]

Greetings, just found this project and am amazed at what you've built so far. LOVE the idea of being able to partition disks natively inside Go without having to have all the tools installed on a machine and having to shell out. Great stuff

Thank you indeed. It started off as an offshoot of my work on linuxkit, where we not only need to shell out to docker, but, for things like ext4, we need to spin up a VM. Unfortunately, et4 has been half done for a long time. Let's say that filesystem work is challenging.

[deitch]

Type: "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",

Just as a small aside, most of these types are defined in gpt, see gpt.Type

[deitch]

feel like I must be missing another important step here after calling err = drive.Partition(table) to actually create the partitions. But I looked at the code and don't see any examples of doing anything after that. There's no Close or Flush or Write that I'm missing is there?

No, and it is strange that is failing. But the CI (for obvious reasons) runs all of its tests on files rather than block devices, so it is possible we simply broke something at some point and missed it because CI did not catch it.

Can you do the following? Whip up a quick fully-functional test program, or at least the complete main.go, and post it here (or on a gist). Then run it against a file and against the nvme device, see if the results are different, and post here? If it turns out to be both, we have something to work with; if it turns out to be just the block device, we also have something to work with.

[zaolin]

Any progress here? I have the same issues from the 1.2.0 release to the latest master. I think it never worked.

[zaolin]

@deitch Here is an example which doesn't work https://github.com/immune-gmbh/isogen

[zaolin]

If I write the ProtectiveMBR it works with GPT generation

[deitch]

Here is an example which doesn't work https://github.com/immune-gmbh/isogen

@zaolin is that a private repo? I get a 404 for it.

[deitch]

If I write the ProtectiveMBR it works with GPT generation

Can you modify that example to make writing the protective MBR optional (once you give me access to it)?

Did you recreate the problem with block devices only? Or with file as well?

[zaolin]

@deitch It's now public. Yes I did it. It fixed the GPT generation issue. I just tried to generate an iso file. My problem here is that I want to generate a bootable ISO for UEFI systems similar to ubuntu/fedora bootable iso. I gave up at the moment because I would need to dig in even more and I don't understand how they made an eltrorito image on the same disk with your library.

[deitch]

similar to ubuntu/fedora bootable iso

I remember digging into this a few years ago. I do not remember what the solution was. There are websites that describe how to create a UEFI-bootable ISO, but there was more to it than that.

If you do dig into it, and manage to figure it out, do open an issue or (better yet) a PR here? Whether it means changes to the library, or just a good example, very happy to take it.

In the meantime, let's look at the issue you raised.

[deitch]

@zaolin I took your sample repo (you can remove it now, if you want) and simplified it to the below:

package main

import (
	"flag"
	"fmt"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/partition/gpt"
)

func main() {
	out := flag.String("out", "", "output file, required")
	doParts := flag.Bool("parts", false, "whether or not to make partitions")
	flag.Parse()
	if out == nil || *out == "" {
		os.Exit(-1)
	}
	if err := makeGPTDisk(*out, *doParts); err != nil {
		fmt.Printf("failed to make efi bootloader iso: %v", err)
		os.Exit(-1)
	}
}

func makeGPTDisk(out string, makeParts bool) error {
	var (
		padding          int64 = 5 * 1024 * 1024
		diskSize         int64 = 20 * 1024 * 1024
		partitionSectors int64 = 100
		partition1Start  int64 = 2048
		partition1End    int64 = partitionSectors + partition1Start - 1
		partition2Start  int64 = partition1End + 1
		partition2End    int64 = ((diskSize - padding) / int64(diskfs.SectorSize512)) - partition2Start + 1
	)

	_ = os.Remove(out)
	disk, err := diskfs.Create(out, diskSize, diskfs.Raw, diskfs.SectorSize512)
	if err != nil {
		return fmt.Errorf("failed to create disk file: %w", err)
	}
	disk.LogicalBlocksize = 2048
	table := &gpt.Table{}
	if makeParts {
		table.Partitions = []*gpt.Partition{
			{
				Start: uint64(partition1Start),
				End:   uint64(partition1End),
				Type:  gpt.MicrosoftBasicData,
				Name:  "ISO9660",
			},
			{
				Start: uint64(partition2Start),
				End:   uint64(partition2End),
				Type:  gpt.EFISystemPartition,
				Name:  "EFI System",
			},
		}
	}
	err = disk.Partition(table)
	if err != nil {
		return fmt.Errorf("failed to create partitiont table: %w", err)
	}
	return nil
}

I ran it both with and without partitions, and it worked fine. I checked with gdisk, as fdisk, depending on variants, does not always recognize GPT partition tables correctly. For alpine, apk add gptfdisk.

$ go run . --out ./foo.img
$ gdisk -l ./foo.img
GPT fdisk (gdisk) version 1.0.9.1

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with corrupt MBR; using GPT and will write new
protective MBR on save.
Disk ./foo.img: 40960 sectors, 20.0 MiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 9305AEB4-3DAC-43C0-9CA3-03DDE3769B74
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 40927
Partitions will be aligned on 2048-sector boundaries
Total free space is 40894 sectors (20.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name

$ go run . --out ./foo.img --parts true
$ gdisk -l ./foo.img
GPT fdisk (gdisk) version 1.0.9.1

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with corrupt MBR; using GPT and will write new
protective MBR on save.
Disk ./foo.img: 40960 sectors, 20.0 MiB
Sector size (logical): 512 bytes
Disk identifier (GUID): FEE5E1A1-E31D-4398-9E2E-491299C478E1
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 40927
Partitions will be aligned on 4-sector boundaries
Total free space is 14368 sectors (7.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048            2147   50.0 KiB    0700  ISO9660
   2            2148           28573   12.9 MiB    EF00  EFI System

It might be due to using gdisk vs fdisk, or it might be something in the numbers passed (I changed them a bit in my sample).

[antoineco]

I haven't been able to use this library for the same reason. edit: issue solved, read on

---

edit: the issue is related to gdisk vs. fdisk as suggested above.

fdisk does not recognize the partition table, but gdisk does:

GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with corrupt MBR; using GPT and will write new
protective MBR on save.
Disk /dev/sdf: 2097152 sectors, 1024.0 MiB
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 4233EB08-9718-4E0C-9CFB-D307A3D5D936
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 2097119
Partitions will be aligned on 2048-sector boundaries
Total free space is 12222 sectors (6.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         2086911   1018.0 MiB  EF00  EFI System

---

Original problem description with snippet

Here is a snippet of the code I use to create my partition table and partition:

package main

import (
	"fmt"
	"io"
	"os"

	"github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/disk"
	"github.com/diskfs/go-diskfs/filesystem"
	"github.com/diskfs/go-diskfs/partition/gpt"
)

func main() {
	err := PartitionAndFormat("/dev/sdf")
	if err != nil {
		panic(err)
	}
}

func PartitionAndFormat(device string) error {
	d, err := diskfs.Open(device)
	if err != nil {
		return fmt.Errorf("opening device: %w", err)
	}
	defer d.File.Close()

	startSector := uint64(PartitionAlignmentBoundary / int64(diskfs.SectorSize512))
	endSector := uint64(d.Size/int64(diskfs.SectorSize512)) - 1

	pt := &gpt.Table{
		Partitions: []*gpt.Partition{{
			Type:  gpt.EFISystemPartition,
			Name:  "EFI System",
			Start: startSector,
			End:   endSector,
		}},
	}
	if err = d.Partition(pt); err != nil {
		return fmt.Errorf("writing partition table: %w", err)
	}

	fs, err := d.CreateFilesystem(disk.FilesystemSpec{
		Partition:   1,
		FSType:      filesystem.TypeFat32,
		VolumeLabel: "boot",
	})
	if err != nil {
		return fmt.Errorf("creating FAT32 filesystem: %w", err)
	}

	return nil
}

Reported by go-diskfs:

// disk

&{
  File: 0xc0000a4150,
  Info: 0xc0002a6750,
  Type: 1,
  Size: 1073741824,
  LogicalBlocksize: 512,
  PhysicalBlocksize: 512,
  Table: 0xc0002934d0,
  Writable: true,
  DefaultBlocks: false,
}

// partition(s)

gpt
[&{
  Start: 2048,
  End: 2097151,
  Size: 1072693248,
  Type: "C12A7328-F81F-11D2-BA4B-00A0C93EC93B",
  Name: "EFI System",
  GUID: "F49DF9DD-BCC6-4D5F-8499-C8E59A512B56",
  Attributes: 0,
  logicalSectorSize: 512,
  physicalSectorSize: 512,
},&{
  Start: 0,
  End: 0,
  Size: 0,
  Type: "00000000-0000-0000-0000-000000000000",
  Name: "",
  GUID: "00000000-0000-0000-0000-000000000000",
  Attributes: 0,
  logicalSectorSize: 512,
  physicalSectorSize: 512,
},
// ...
]

Reported by fdisk -l:

Disk /dev/sdf: 1 GiB, 1073741824 bytes, 2097152 sectors
Disk model: Amazon Elastic Block Store
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes