This repository was archived by the owner on Feb 12, 2021. It is now read-only.

This repository was archived by the owner on Feb 12, 2021. It is now read-only.

md: document how to use software RAID with CoreOS #222

Open

Open

md: document how to use software RAID with CoreOS#222

Labels

help-wantednew document

opened

Contributor

People may want to software RAID CoreOS. Can we do this today? Can we help with the coreos-install script?

Contributor

No, I would rather not do raid in the current installer script. If we want to start working on a full featured Linux installer we should do that in a language that isn't shell.

Member

What about software raid outside of the installation script?

At least it would be useful to document what needs to be done, if anything, besides building the raid and running the installer against it. I tried this but couldn't get my machine to boot.

Contributor

@robszumski software raid is what we are talking about.

@jsierles sounds like we have bugs to fix because my intent is to make that work.

added

on Jul 28, 2014

Any news on the software raid documentation?

Would be rather useful

Also would see this as very useful functionality.

Yes, it would be great :)
@philips I saw this commit But yeah... Does anybody can tell me where to start if I want raid software? emerge mdadm ??

Contributor

@pierreozoux mdadm is included base images but we haven't played with it at all. Setting up non-root raid volumes should work just the same as any other distro. Same ol' mdadm command for creating and assembling volumes. You may need to enable mdadm.service if you want to assemble volumes on boot via /etc/mdadm.conf as opposed to using the raid-autodetect partition type and letting the kernel do it. It might be possible to move the root filesystem as long as the raid-autodetect partition type is used but for that you are almost certainly better off with using multi-device support in btrfs.

Contributor

What certainly won't work right now is installing all of coreos on top of software raid, the update and boot processes both assume the ESP and /usr partitions are plain disk partitions.

@marineam Would this constraint of CoreOS also apply to btrfs-raids?

Contributor

@brejoc multi-device btrfs for the root filesystem should work

What about migrating after install. Eg to RAID 1 from installed /dev/sda (one partition sda1 for demonstration) should be something like this from a Rescue CD or similar:

sfdisk -d /dev/sda | sfdisk /dev/sdb
sfdisk --id /dev/sdb 1 fd
mdadm --zero-superblock /dev/sdb1
mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb1
mkfs.btrfs /dev/md0
mkdir /mnt/source;mount /dev/sda /mnt/source
mkdir /mnt/target;mount /dev/md0 /target
cp -a /mnt/source/* /mnt/target

Thereafter the disk mount configuration needs to be changed and the kernel root device in the bootloader, as well as the bootloader installed to both disks.

modify /mnt/target/etc/fstab Replace /dev/sda1 with /dev/md0 - but this is non-existent on CoreOS
bootloader since 435 seems to be GRUB which helps but I cannot find a grub binary only config in /usr/boot

Thoughts?

@warwickchapman just in case you finished your exploration into this topic and came up with a complete solution - or if someone else has - I'd appreciate if you shared it. I know too little about setting up and messing with RAID / mounts / boot in order to complete this myself. It's not a hard requirement for my use case but it would help being able to have RAID to be able to use both/all disks in a system. I understand it's also possible to set up a distributed file system like Ceph and let it manage the disks without RAID, and that would work for the use cases I have in mind, but for now I'm happy about any additional complexity I can avoid!

Contributor

As noted on IRC, for btrfs if raid0 or raid1 is all you need then it is easiest to just add devices to btrfs and rebalance: https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

As for md raid if the partition types are the raid-autodetect type then the raid volume will be assembled automatically. But you can only put the ROOT filesystem on raid, we don't currently support putting the other partitions on anything other than plain disk devices.

23 remaining items

I removed sda9 for the sake of md0. But you are true: It's kind of a hack atm and needs to be documented or discouraged.

TBH I think it would be very important to add and/or document a solution for software RAID. Independently of how good coreos can handle failure (together with fleet, docker, ...) I'm not really fancy about "loosing a server" just because of a disc failure (which happens "all the time" btw).

I don't seam to get it to work, I've setup md0 and added the ROOT label, but even if I add rd.auto=1 to grub, it just hangs there on boot. After I type mdadm --assemble --scan in the emergency console, the boot continues. Any idea?

Cannot get CoreOS running with RAID 1 after spending 2 days. This is essential in my opinion. No valid documentation found therefore. Quitting CoreOS.

Contributor

coreos/bugs#1025

I worked on attached ephemeral raid and was able to get it working using the following units in my cloud config. I tested reboots as well and the raid came back and mounted correctly.

This code is specific to creating a software raid 0 on GCE using two local SSD's with the NVME interface. Feedback or suggestions would be appreciated.

  - name: format-ephemeral-raid.service
    command: start
    content: |
      [Unit]
      After=dev-nvme0n1.device dev-nvme0n2.device
      Requires=dev-nvme0n1.device dev-nvme0n2.device
      ConditionPathExists=!/dev/md0
      [Service]
      Type=oneshot
      RemainAfterExit=yes
      ExecStart=/bin/bash -c 'mdadm --create /dev/md0 --level=0 --raid-devices=2 /dev/nvme0n[1-2]'
      ExecStart=/bin/bash -c 'mkfs.ext4 -F /dev/md0'
      ExecStartPost=/bin/bash -c 'mkdir -p /etc/mdadm'
      ExecStartPost=/bin/bash -c 'echo "DEVICE /dev/nvme0n1 /dev/nvme0n2" > /etc/mdadm/mdadm.conf'
      ExecStartPost=/bin/bash -c 'mdadm --detail --scan >> /etc/mdadm/mdadm.conf'

  - name: data.mount
    command: start
    content: |
      [Unit]
      Requires=format-ephemeral-raid.service
      After=format-ephemeral-raid.service
      Before=docker.service
      [Mount]
      What=/dev/md0
      Where=/data
      Type=ext4

I used cmoad's approach on aws running kubernetes with coreos worked like a charm!

Hi!
Has there been any progress on this front? Developing consistent documentation for SWRAID. Would going with HWRAID be an easier option for now?

Member

Check out Ignition's example for software raid: https://coreos.com/ignition/docs/latest/examples.html#create-a-raid-enabled-data-volume

More background on Ignition: https://coreos.com/blog/introducing-ignition.html

can i use that example to setup a software raid for the root partition?

Contributor

@celevra you can use something like this (have not actually tested it):

{
  "ignition": { "version": "2.0.0" },
  "storage": {
    "disks": [
      {
        "device": "/dev/vdb",
        "wipeTable": true,
        "partitions": [{
          "label": "root.1",
          "typeGuid": "A19D880F-05FC-4D3B-A006-743F0F84911E"
        }]
      },
      {
        "device": "/dev/vdc",
        "wipeTable": true,
        "partitions": [{
          "label": "root.2",
          "typeGuid": "A19D880F-05FC-4D3B-A006-743F0F84911E"
        }]
      }
    ],
    "raid": [{
      "devices": [
        "/dev/disk/by-partlabel/root.1",
        "/dev/disk/by-partlabel/root.2"
      ],
      "level": "mirror",
      "name": "root"
    }],
    "filesystems": [
      {
        "name": "data",
        "mount": {
          "device": "/dev/disk/by-label/ROOT",
          "format": "ext4",
          "create": { "force": true }
        }
      },
      {
        "name": "root",
        "mount": {
          "device": "/dev/md/root",
          "format": "ext4",
          "create": {
            "options": [ "-L", "ROOT" ],
            "force": true
          }
        }
      }
    ]
  }
}

This will set up a single RAID partition on vdb and vdc, assemble them into an array, and then create the new ROOT filesystem. It also wipes out the default ROOT partition on vda9 (since you don't want two). When you use this, you'll also need rd.auto to tell the initramfs to automatically assemble the array on every boot.

thank you crawford, will try that out

Dear All,

I'm new to CoreOS and i'd like to test this on my barebone server.

I have following Issue:
I want to have 2x SSD's in RAID1 for root as I always do.

I tried crawfords solution, installed CoreOS with ignition-file, but this resulted in an error, where the boot sequence is waiting on a job with no limit, which means forever due to that job will never be finished. When i kill the server manually and reboot, the drive is not bootable anymore.

When I try to setup a HW-Raid, my CoreOS-LiveUSB does not recognize my HW-RAID and takes the two drives as regular sda and sdb. Additionally, the coreos-install script fails with an error return code 32.

BTW: I have attached another 4 Drives for Storage in the same node.

Does anyone have a solution?

mentioned this

on Aug 26, 2017

os: add docs for having root on raid #1141

mentioned this

Consider supporting /usr partition on RAID coreos/bugs#2143

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

Labels

help-wantednew document

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Participants