vboxwrapper: create the 'virtualbox home directory' in the project dir

[davidpanderson]

The 'VBox home directory' is where VBox writes log files
(which are read by vboxwrapper).
If this is not specified by the env var VBOX_USER_HOME,
we need to create it somewhere and set the env var to point there.

Previously we put it in /projects.
That's no good because it's not a project, and the client erased it.

We also tried putting it in the (real) user's home dir.
That's no good because

1. we shouldn't mess with the home dir
2. in sandboxed configs we're running as user 'boinc_projects',
   and don't have access to the home dir.

According to https://boinc.berkeley.edu/sandbox_design.php,
the only places 'boinc_projects' can write are
projects/, slots/, and their subdirectories.
So the logical places to put .VirtualBox are
this job's slot directory, or its project directory.
I chose the latter.

[Copilot]

Copilot wasn't able to review any files in this pull request.

Files not reviewed (1)

• samples/vboxwrapper/vbox_vboxmanage.cpp: Language not supported

[NenTech]

This is the latest result with the 26209 (compared to the code):
`2025-01-22 22:54:38 (13555):
Command: VBoxManage -q --version
Exit Code: 0
Output:
7.1.4r165100

2025-01-22 22:54:38 (13555):
Command: VBoxManage -q list systemproperties
Exit Code: 0
Output:
VBoxManage: error: Failed to initialize COM because the global settings directory '/var/empty/Library/VirtualBox' is not accessible!

2025-01-22 22:54:38 (13555):
Command: VBoxManage -q list hostinfo
Exit Code: 0
Output:
VBoxManage: error: Failed to initialize COM because the global settings directory '/var/empty/Library/VirtualBox' is not accessible!`

Can someone supply me this file compiled for testing?

[davidpanderson]

I'm not sure the changes I've been making (involving the 'virtualbox home directory')
are relevant to this problem.

On the Mac, BOINC jobs run as a 'hidden user', boinc_project.
This user has minimal privileges.
It looks like it's unable to run VboxManage successfully because it can't access
'/var/empty/Library/VirtualBox'

I'm confused.

• Did vboxwrapper ever work on Mac? What's changed?
• why is it trying to initialize COM?

[NenTech]

I'm not sure the changes I've been making (involving the 'virtualbox home directory') are relevant to this problem.

On the Mac, BOINC jobs run as a 'hidden user', boinc_project. This user has minimal privileges. It looks like it's unable to run VboxManage successfully because it can't access '/var/empty/Library/VirtualBox'

I'm confused.

• Did vboxwrapper ever work on Mac? What's changed?

Yes it did work. When I switch to the 26207b wrapper, tasks are running but failing due to network connection..

• why is it trying to initialize COM?

Here I have the trace and replay of the task:
https://lhcathome.cern.ch/lhcathome/forum_thread.php?id=6277&postid=51445

The task:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419017121

[davidpanderson]

I can't see (in those logs) any problems except the cvmfs connect failure,
and the COM errors which don't seem to matter.
Possibly a CERN issue?

[NenTech]

I can't see (in those logs) any problems except the cvmfs connect failure, and the COM errors which don't seem to matter. Possibly a CERN issue?

That is true. The error I sent earlier here (Command: VBoxManage -q list systemproperties) is with the new wrapper. I'm trying to debug it now on my Mac. I cannot find it. With automatic launch of the wrapper, the project fails immediately after start.

https://lhcathome.cern.ch/lhcathome/result.php?resultid=419017986

New info, the VirtualBox home directory is changed. The folder is made (without the .).
vboxmanage still goes to ~/Virtualbox for the settings.

BOINC is not allowed to create .VirtualBox on Mac since this is system only! (VirtualBox is allowed)

https://lhcathome.cern.ch/lhcathome/result.php?resultid=419018059

The folder for the VirtualBox settings need to be changed since VirtualBox places them in ~ of which is not allowed for other users. I'm tired now. Going to bed and will try to continue tomorrow with testing on this.

[computezrmle]

I just archived a couple of logs from the LHC@home server before they disappear.
Can provide them if necessary.

• all of them valids from Apple computers
• all of them vboxwrapper 26207
• various BOINC client versions
• various VirtualBox versions (even the old v5.2.44)
• various subprojects CMS/Theory

I remember even @NenTech had valids reported by the previous app versions using vboxwrapper 26207 (unfortunately the logs are not available any more).

This recent result from @NenTech failed although vboxwrapper 26207 has been used:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419022990

Very unlikely that it is caused by the vdi file since

• this hasn't changed (except the filename and it's VirtualBox UUID => both a must to avoid conflicts between old/new app versions if differencing images are used)
• it is the same that works for Windows and Linux

Host internal networking appears to work since "Mounting the shared directory" succeeded.
External networking fails, hence ntp and CVMFS connections are affected.

The Hypervisor System Log reports lots of this:
00:00:03.831814 DCon01 ERROR [COM]: aRC=E_ACCESSDENIED (0x80070005) aIID={e36a5081-a82a-40bd-9e4e-42a44d6ce50f} aComponent={MachineWrap} aText={The object functionality is limited}, preserve=false aResultDetail=0
Not sure which object is meant.

Could it be the app version setup?
To verify this @NenTech please extract the app_version section (Theory 30060) from client_state.xml and post it here.

Another log from @NenTech's computer reports this:
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419016867

VBoxManage -q showvminfo "boinc_a0b98d0ab8ba4e7c" --machinereadable 
Output:
VBoxManage: error: Failed to initialize COM because the global settings directory '/Users/nentech/Library/VirtualBox' is not accessible!

Not sure why this points to the user's directory.
I would expect it to point to the BOINC account's directory.
@NenTech Did you run the test from your normal user account or via BOINC.

[davidpanderson]

don't merge for now

[NenTech]

@computezrmle
Hereby the <app_version> of the Theory and CMS app. Both are failing the same way.

<app_version>
    <app_name>Theory</app_name>
    <version_num>30060</version_num>
    <platform>x86_64-apple-darwin</platform>
    <avg_ncpus>1.000000</avg_ncpus>
    <flops>4727558580.521004</flops>
    <plan_class>vbox64_theory</plan_class>
    <api_version>8.1.0</api_version>
    <file_ref>
        <file_name>vboxwrapper_26208_x86_64-apple-darwin</file_name>
        <main_program/>
        <copy_file/>
    </file_ref>
    <file_ref>
        <file_name>Theory_2025_01_16_prod.xml</file_name>
        <open_name>vbox_job.xml</open_name>
        <copy_file/>
    </file_ref>
    <file_ref>
        <file_name>Theory_2025_01_16_prod.vdi</file_name>
    </file_ref>
    <dont_throttle/>
    <needs_network/>
</app_version>
<app_version>
    <app_name>CMS</app_name>
    <version_num>7060</version_num>
    <platform>x86_64-apple-darwin</platform>
    <avg_ncpus>4.000000</avg_ncpus>
    <flops>18910234322.084015</flops>
    <plan_class>vbox64_mt_mcore_cms</plan_class>
    <api_version>8.1.0</api_version>
    <cmdline>--nthreads 4</cmdline>
    <file_ref>
        <file_name>vboxwrapper_26208_x86_64-apple-darwin</file_name>
        <main_program/>
        <copy_file/>
    </file_ref>
    <file_ref>
        <file_name>CMS_2025_01_16_prod.xml</file_name>
        <open_name>vbox_job.xml</open_name>
        <copy_file/>
    </file_ref>
    <file_ref>
        <file_name>CMS_2025_01_16_prod.vdi</file_name>
    </file_ref>
    <dont_throttle/>
    <needs_network/>
</app_version>

I did the testing with normal user account and via BOINC. The results with all versions of the wrapper are the same via normal user account as the 26207b via BOINC.
The result below I got because I changed the wrapper that it would not exit after failing.

VBoxManage -q showvminfo "boinc_a0b98d0ab8ba4e7c" --machinereadable 
Output:
VBoxManage: error: Failed to initialize COM because the global settings directory '/Users/nentech/Library/VirtualBox' is not accessible!

Found the error and I have it fixed in my version 26210!
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419047588

There is still 1 issue left. User group boinc_project is not allowed to create the directory VirtualBox in BOINC_data.

[CharlieFenton]

User group boinc_project is not allowed to create the directory VirtualBox in BOINC_data.

I'm surprised, since I believe all project applications run as user boinc_master and group boinc_project, and are able to traverse the directory tree into the slots and project directories and successfully create files in those subdirectories. I don't see any indication in the referenced result log. How do you know that VBoxWrapper is failing to create the directory due to a permission error?

[CharlieFenton]

believe all project applications run as user boinc_master and group boinc_project,

This is based on my reading of the current setprojectgrp.cpp, which is invoked using switcher in set_to_project_group() in sandbox.cpp which in turn is called from both ACTIVE_TASK::start() and ACTIVE_TASK::setup_file() in app_start.cpp as well as other places.

[davidpanderson]

According to https://boinc.berkeley.edu/sandbox_design.php,
apps run as user boinc_projects and group boinc_projects,
so they can't write to the top-level data directory.

I'm submitting a PR for vboxwrapper where it creates .VirtuaBox
in its project directory, which it's guaranteed to have write access to.

[davidpanderson]

I think this should work, but I can't currently test it.
It's designed to work on
Mac: standard install (sandboxed)
Win: regular or service (sandboxed) install
Linux: any type of install

[computezrmle]

This modification works on Linux with a couple of vbox tasks from LHC@home but I'm not happy writing the vbox logs to /projects/abc/.VirtualBox/ for the following reason.

Bad scenario

Assume you run a BOINC client attached to at least 2 vbox projects /projects/abc/ and /projects/fgh/.
If project abc starts a vbox task it will also start a VBoxSCV instance writing it's logs to /projects/abc/.VirtualBox/.
If project fgh starts a task while the 1st VBoxSCV instance is running it's log entries will also be written to /projects/abc/.VirtualBox/ because all processes run under the same username.
VirtualBox controls that through a socket (e.g. on Linux in /tmp/.vbox-username-ipc/).

Now the bad thing - remove project abc from the BOINC client.
This will either cause an error during cleanup of directory /projects/abc/ since there are open files or it will remove the VBoxSCV logs although that process is running to serve project fgh.

Possible solution (not yet tested)
Write the vbox logs to /projects/.VirtualBox/.
If I correctly understand the sandbox design that directory should be writeable by the boinc_project group.

[computezrmle]

BTW
@NenTech
The snippets from client_state.xml look fine.

[CharlieFenton]

According to https://boinc.berkeley.edu/sandbox_design.php, apps run as user boinc_projects and group boinc_projects, so they can't write to the top-level data directory.

That information is out of date. I changed it in commit aecfe8e on November 8, 2022 as part of a fix to "Show graphics" from Manager for MacOS 13. Please see my comments here and here

[CharlieFenton]

Possible solution (not yet tested)
Write the vbox logs to /projects/.VirtualBox/.

That's where the VirtualBox directory was before these changes. See #5658. The problem with that was that every time the BOINC client is restarted, it cleans the BOINC Data/projects/ directory by removing any subdirectories that are not a currently attached project. So the VirtualBox directory was deleted each time the BOINC client was restarted.

[Crystal-Pellet]

Beyond the show-stopper computezrmle reported, the current vboxwrapper-26209 test version was also working OK on Windows.
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419058695

[davidpanderson]

I didn't realize that different jobs shared the same VBoxSVC instance. In the above example, the more immediate problem is that the fgh job will look for the log in projects/fgh/.Virtualbox/, and it's actually in projects/abc/.Virtualbox.

So - since according to Charlie the data directory is in fact writeable to the boinc_projects user - the best thing is to put .VirtualBox in the data directory. I'll make this change.

[CharlieFenton]

I think I have a simpler solution. The original problem #5658 was that the VirtualBox directory was in the Projects subdirectory, and every time the BOINC client is restarted, it cleans the "BOINC Data/projects/" directory by removing any subdirectories that are not a currently attached project. So the VirtualBox directory was deleted each time the BOINC client was restarted.

The simplest solution might be to modify the BOINC client code which cleans up the "BOINC Data/projects/" directory to recognize the VirtualBox directory as a special case and not delete it. Then we can revert the location of the VirtualBox directly to "/Library/Application Support/BOINC Data/VirtualBox/" (with or without the leading period.)

I recommend that it be VirtualBox as before and not .VirtualBox so it is not a hidden file, for ease of user support and debugging. And this has the added bonus of providing backward compatibility with earlier versions of VBoxWrapper.

What do you all think?

@davidanderson can you make those changes? Unfortunately, due to Apple's security restrictions I'll have to build the installer from this branch and run it through Apple's "Notarization" process before @NenTech can run it to test it. I won;t have time to do that until after the weekend.

[NenTech]

I checked the permission issue in BOINC_data simply by viewing if the folder was made. After I added boinc_project to the BOINC_data folder, the wrapper was be able to make the folder. I think indeed that the best solution would be to put the VirtualBox folder without leading . to the projects folder.
Unfortunatly I don’t have time today or tomorrow to test it. I will do it Sunday. I can test Windows as well btw.

[davidpanderson]

Oops - I just read charlie's earlier comment.
So it looks like vboxwrapper can't write to the data dir.
So it seems to me each possible place for .VirtualBox has a fatal flaw (w/ sandbox, at least on Mac):

• user home dir: boinc_projects can't write there.
  Question: did vboxwrapper ever work on the Mac?
• BOINC data dir: boinc_projects can't write there
• datadir/project: client deletes it on startup
• datadir/project/abc: fails if two different projects use vboxwrapper

Is this correct? If so, what's the lesser of these evils?

[davidpanderson]

BTW, the next version of the client will not remove unrecognized stuff in projects/
(for reasons unrelated to this).
But we're looking for a vboxwrapper solution that will work with current clients.

[computezrmle]

  Question: did vboxwrapper ever work on the Mac?

Yes, it worked fine for LHC@home until last week.

[CharlieFenton]

Any user can traverse the top level BOINC Data directory (user, group and other all have x permission) but only user and group boinc_master can write to it. So neither user nor group boinc_project can create a directory there. But after traversing the top level directory they can write into and create files and directories in the projects and slots directories which have user boinc_master and group boinc_project.

[davidpanderson]

Yeah. But we don't want to put .Virtualbox in either projects/ or slots/,
because the current client will delete it on startup

[computezrmle]

From my point of view the right place should be datadir/VirtualBox/.
This directory should be created by the installer including write permissions for the boinc_projects group.
That way VirtualBox should be able to create it's logs there.

I agree with Charlie Fenton that it shouldn't be a hidden directory (hiding it doesn't make sense).

For older clients it may be fine to tell the volunteers to manually create the directory and set the permissions.
Are there any projects beside LHC@home currently using VirtualBox?

[davidpanderson]

So it worked for LHC before last week. Good.
What version of vboxwrapper did LHC use before last week?
Where did it create .Virtualbox?
Please look at the code and let me know.

---

OK, it put it in projects/.

The path of least resistance is to continue putting it there.
That's will work with current clients as long as they don't restart,
and it will work with future clients regardless.

[computezrmle]

So it worked for LHC before last week. Good. What version of vboxwrapper did LHC use before last week?

They used an artifact from here based on v26207 +PRs up to #5608.

Where did it create .Virtualbox? Please look at the code and let me know.

In projects/

[CharlieFenton]

But we're looking for a vboxwrapper solution that will work with current clients.

That of course would be ideal. But isn't there a way for projects to send certain tasks only to clients with a minimum version number? They could tell clients to upgrade BOINC in Notices.

I've seen messages in the Event Log saying that a project has tasks available but not for my computer or other criteria. Is there such message saying it has tasks for a newer version of BOINC?

[davidpanderson]

Yes, but I'd like to minimize complexity here.

[Crystal-Pellet]

I tested 2 other scenarios with vboxwrapper v26209 on Windows with creating the .VirtualBox directory under e.g. /projects/lhcathome.cern.ch_lhcathome

1. Without running any BOINC VM and getting a new VM-task, the directory .VirtualBox is created when it was not created before,
   but when VBoxSVC.exe was already running the BOINC .VirtualBox directory is not used, but VBox default location usually "C:\Users\User\.VirtualBox" is used.
   VBoxSVC.exe could already run, because the user have a VM of his own running or just is using VirtualBox Manager.

2. 2 clients on one host. Both using the same project. The first VM-task is using BOINC's VirtualBox directory,
   but when the second client gets a VM-task the .VirtualBox directory is created, but the task will crash after 2 x 5 times a VBox manage command like:
   2025-01-25 14:14:10 (1548):
   Command: VBoxManage -q --version
   Exit Code: 0
   Output:
   7.1.6r167084

2025-01-25 14:15:01 (1548):
Command: VBoxManage -q list systemproperties
Exit Code: -182
Output:

2025-01-25 14:15:54 (1548):
Command: VBoxManage -q list systemproperties
Exit Code: -182
Output:

2025-01-25 14:16:47 (1548):
Command: VBoxManage -q list systemproperties
Exit Code: -182
Output:

2025-01-25 14:17:39 (1548):
Command: VBoxManage -q list systemproperties
Exit Code: -182
Output:

2025-01-25 14:18:31 (1548):
Command: VBoxManage -q list systemproperties
Exit Code: -182
Output:

2025-01-25 14:19:19 (1548):
Command: VBoxManage -q list systemproperties
Exit Code: -182
Output:

2025-01-25 14:20:11 (1548):
Command: VBoxManage -q list hostinfo
Exit Code: -182
Output:

2025-01-25 14:21:03 (1548):
Command: VBoxManage -q list hostinfo
Exit Code: -182
Output:

2025-01-25 14:21:56 (1548):
Command: VBoxManage -q list hostinfo
Exit Code: -182
Output:

2025-01-25 14:22:53 (1548):
Command: VBoxManage -q list hostinfo
Exit Code: -182
Output:

2025-01-25 14:23:40 (1548):
Command: VBoxManage -q list hostinfo
Exit Code: -182
Output:

2 clients could be usefull, because of differentiating sub-projects like LHC's Theory. ATLAS and CMS or

3rd scenario: two different projects using VM's. I started testing this but Rosetta does not have VBox-tasks at the moment.

[Crystal-Pellet]

3rd scenario: Running VMs of different projects:

When you have a running VM of project 1 and start a VBox-task of project 2, the VBox-xml and -log of project 1 are used.
When you suspend both tasks of the 2 projects (not keeping in memory, so saved to disk) and resume the task of project 2 first,
new VBox-xml and -log are written in the .VirtualBox of project 2. Resuming task of project 1 later is using that directory too.

Important remark: With VirtualBox Manager the BOINC VMs are not shown!

So suspending both tasks again and open VirtualBox Manager (empty list of VMs) and resuming the BOINC-tasks,
the VMs popup in VirtualBox Manager and not BOINCs log and xml's are used, but the default VBox directory C:\Users\User.VirtualBox.

[NenTech]

The wrapper does work. I still have the issue with the failing tasks. I think this is also part of the folder permissions. But for now, I still can not find the bug.

• The file Heartbeat is not created.
• When I create the file with the wrapper, it's not updated. So there is something wrong between the VM and the folders on Mac.

This was my latest test: (created the heartbeat file with the wrapper, saved it a few times and let it run until it crashes.)
https://lhcathome.cern.ch/lhcathome/result.php?resultid=419123606

[AenBleidd]

What is the status of this PR? Is it tested and ready to be merged?

[CharlieFenton]

What is the status of this PR? Is it tested and ready to be merged?

No, I'm pretty sure it is not ready. The following change has not been done. @davidpanderson wrote here:

OK, it put it in projects/.
The path of least resistance is to continue putting it there. That's will work with current clients as long as they don't restart, and it will work with future clients regardless.

And there seems to be some consensus that if it goes in projects/ the subdirectory should be named VirtualBox without a leading period.

[NenTech]

I am running the tests now on a working VM of LHC. The ATLAS task is running fine for now. In a few hours it should be completed.
Task with wrapper 26206: https://lhcathome.cern.ch/lhcathome/result.php?resultid=419186593
Task with wrapper 26210: https://lhcathome.cern.ch/lhcathome/result.php?resultid=419189576
In my 26210 version I made various diagnostics and manual tests for parts which failed on the CMS task.

[NenTech]

I think the wrapper is working perfectly now on MacOS.

Is it possible to let the wrapper change ownership/permissions from VirtualBox to boinc_master?
boinc_chown did not work for me. (Function not found)

[CharlieFenton]

Is it possible to let the wrapper change ownership/permissions from VirtualBox to boinc_master?

I'm pretty sure the answer is no, because a project application does not have sufficient permissions.

[NenTech]

Is it possible to let the wrapper change ownership/permissions from VirtualBox to boinc_master?

I'm pretty sure the answer is no, because a project application does not have sufficient permissions.

chmod works but does not fix the startup issue. chown does not work.

I would say this change is ready to get merged.

[NenTech]

Change works on MacOS. Until new release of BOINCManager, the Mac_SA_Secure.sh must be executed or the folder VirtualBox need to be removed from BOINC_Data/projects.

[CharlieFenton]

No, I'm pretty sure it is not ready. The following change has not been done.

For some reason I did not see the most recent changes in the "Files changed" diff when I wrote that. I just reviewed the source code and I think it is probably ready to be merged, since @NenTech has tested it and says it works.

[AenBleidd]

Thank you all for the awesome work done!