Valeria: iOS H.264 screen capture service

[renegadelink]

Summary

Adds ValeriaScreenCapture, a unified iOS H.264 screen-capture service, plus a pymobiledevice3 valeria CLI on top of it. Two backends behind one API:

• macOS — valeria_cmio talks to CoreMediaIO directly via pure ctypes. Same kernel pathway QuickTime Player uses for iOS screen recording.

Both backends emit identical H264Frame objects carrying the iDevice's native H.264 video. Decode/render is the consumer's job.

Public API

from pymobiledevice3.services.valeria import ValeriaScreenCapture

cap = ValeriaScreenCapture.create(udid=None, backend="auto")
cap.start()
for frame in cap.frames():
    sys.stdout.buffer.write(frame.to_annex_b())

ValeriaScreenCapture.create() auto-selects cmio on macOS and libusb elsewhere. H264Frame.to_annex_b() produces bytes that decode straight into FFmpeg / a hardware H.264 decoder; frames() / aframes() give sync / async iterators.

CLI

pymobiledevice3 valeria -o capture.h264 --duration 10

Test plan

• macOS: grant Screen Recording permission to the terminal (System Settings → Privacy & Security → Screen Recording).

AI Assistance

Used Claude extensively. Reviewed and verified this PR myself.

[nanoscopic]

Good job. Was wondering when someone else would do this.

Be aware that the CoreMediaIO pathway on MacOS is not stable and it crashes out relatively frequently and it can be difficult to reset it. I originally was using that as a video option years ago and removed it because it was too unstable.

How stable is this implementation? To what extent have you tested it? The main issue I've seen in implementing Valeria myself is that the timing code can lead to it crashing out.

I haven't fiddled with Valeria for maybe 2 months, but last I did I recall that it isn't always mode 5. Sometimes it needs mode 6... So I think that bit of the comment on the commit may be wrong.

[renegadelink]

Good job. Was wondering when someone else would do this.

Be aware that the CoreMediaIO pathway on MacOS is not stable and it crashes out relatively frequently and it can be difficult to reset it. I originally was using that as a video option years ago and removed it because it was too unstable.

How stable is this implementation? To what extent have you tested it? The main issue I've seen in implementing Valeria myself is that the timing code can lead to it crashing out.

Thanks! I only have one device to test on but I did plenty of testing on it and didn't have an issue. I'm hoping others can help test with more devices though.

[nanoscopic]

It's doubtful that you can legally add this code into the project.

As it is an AI conversion of Quicktime Video Hack, which is MIT licensed, it is a derivative and needs to stay under the same license. It cannot be recast into GPL3 and therefore cannot be legally contributed into pymobiledevice3 without Daniel's permission.

Daniel may be willing to allow it as he owns the Copyright for that and could permit it. You should discuss it with him.

[renegadelink]

It's doubtful that you can legally add this code into the project.

As it is an AI conversion of Quicktime Video Hack, which is MIT licensed, it is a derivative and needs to stay under the same license. It cannot be recast into GPL3 and therefore cannot be legally contributed into pymobiledevice3 without Daniel's permission.

Daniel may be willing to allow it as he owns the Copyright for that and could permit it. You should discuss it with him.

Thanks! I'll do just that!

[nanoscopic]

@renegadelink I'll need to check my research to give you a better / clearer answer on the difference between mode 5 and 6. My recollection offhand is that it depends on the iOS version, and that you need mode 6 if you want to be able to simultaneously do xctest on some versions along with the video streaming. Essentially mode 6 is everything mode 5 is plus more things.

As there are now two implementations of Valeria out there now with you adding this, I'm likely to go ahead and release my implementation as open source as well as there is likely no point to hoarding it anymore. ( besides which that I haven't done anything with my implementation yet besides build and test it )

[nanoscopic]

@renegadelink Another option now that I think about it would be to make the derivative code portion a module and keep that MIT and then use that module from glue code in pymobiledevice3.

Probably Daniel is okay with the AI derivative using the new license though, so -shrug-. Options.

[renegadelink]

@renegadelink Another option now that I think about it would be to make the derivative code portion a module and keep that MIT and then use that module from glue code in pymobiledevice3.

Probably Daniel is okay with the AI derivative using the new license though, so -shrug-. Options.

Thanks @nanoscopic! I just pushed an update — Daniel's copyright + full MIT text are now at the top of valeria_libusb.py. I also opened a discussion on his repo to ask him about it.

Yea we could make it a pmd3 dependency that has MIT license if he wants or I don't hear back.

[renegadelink]

@renegadelink I'll need to check my research to give you a better / clearer answer on the difference between mode 5 and 6. My recollection offhand is that it depends on the iOS version, and that you need mode 6 if you want to be able to simultaneously do xctest on some versions along with the video streaming. Essentially mode 6 is everything mode 5 is plus more things.

As there are now two implementations of Valeria out there now with you adding this, I'm likely to go ahead and release my implementation as open source as well as there is likely no point to hoarding it anymore. ( besides which that I haven't done anything with my implementation yet besides build and test it )

Your reading of mode 5 and 6 makes sense to me from my testing. It's also something I would really need more devices to test on to be sure its a good implementation as I don't even know if all devices have mode 6 (which I kinda doubt).

[nanoscopic]

It definitely doesn't have mode 6 on all devices that is what confused me. Before recently I hadn't messed with Valeria in some years, and was only familiar with mode 5. And... it didn't work. And I was confused as to why and then I figured it out. It's fairly obvious when it doesn't work. Between myself and partners on my system we have 50+ phones and nearly as many iOS versions intentionally so we can test this sort of stuff...

I'll let you know when I put up my implementation. My license is different but I have no issue with you looking through it yourself or with AI to learn more options and ways to make it better. I only ask that you not have AI adapt my implementation into something else under a different license.

As an aside it's so irritating that Apple locked their system driver to iPhones so you can't readily attach to the interface for video and are forced to use their poor implementation of it on MacOS. I was attempting to work around that by modifying a cheap Linux device to serve as a MITM to alter the VID and PID, but unfortunately most gadget mode support in chipsets for cheap devices can't handle the number of endpoints on modern iPhones. With too many USB endpoints I couldn't get it to work cleanly.

With more expensive hardware I can make it work, but that defeats the point as I wanted a cheap way to do it.

Also spent a while altering firmware on USB hubs to see if I could manage to alter VID and PID with a standard USB hub with modified firmware. I was able to fully RE many common USB 3 hub firmwares and alter them but still couldn't manage to get them to work as a way to prevent MacOS latching on.

All this is somewhat irrelevant and makes little difference. My only point is I was looking for ways to get a clean Valeria implementation working on MacOS and what I've tried so far didn't work.

[doronz88]

I will review this once all the licensing stuff are settled.

Also, I did not encounter anything about performing it on macOS on anything public at least. Care to share a link where is it?

The macOS implementation is just using the builtin module from the DSC (wrapped using ctypes) to retrieve the frames. There is no protocol implementation actually taken from a different repository, so as far as I understand, there shouldn't be a licensing issue at least for macOS.

[renegadelink]

I will review this once all the licensing stuff are settled.

Also, I did not encounter anything about performing it on macOS on anything public at least. Care to share a link where is it?

The macOS implementation is just using the builtin module from the DSC (wrapped using ctypes) to retrieve the frames. There is no protocol implementation actually taken from a different repository, so as far as I understand, there shouldn't be a licensing issue at least for macOS.

Okay I dropped libusb Valeria from this PR — macOS CoreMediaIO only for now. I'll follow up either as a separate PR or its own package pmd3 imports, depending on Daniel's response.

[doronz88]

@renegadelink do you consider this pr as ready for cr and possibly merging?

[renegadelink]

@renegadelink do you consider this pr as ready for cr and possibly merging?

yup. I did what I could with my one test device and I have no issues.

[doronz88]

After testing, this PR needs much more work as the API supplied is too much unreliable

[nanoscopic]

@doronz88 I literally said that is why Valeria ( or at least the code by Daniel ) was unusable by me. I had to do a significant amount of work to add stuff to reset the interfaces ( on my Github... ) and quickly reset the services when they inevitably crash.

As this is based off AI rewrite of Daniel's code, it will be just as flaky. ( implementation wise )
As Valeria is Valeria, it is also terribly flaky from Apple side generally and that cannot be fixed.

The majority of the flakiness is from bad Apple implementation especially around being able to consistently start the stream and run it for more than a few minutes.

All this is why I removed Valeria use from my product, ControlFloor, because it so bad. Ultimately Upload Broadcast Extension is the only reliable official method.

I wouldn't call Valeria officially supported. More like half-assed by Apple and then abandoned.

Because the problems don't primarily stem from this code but rather Apple, I don't see why holding up contribution of it helps anything. First the code needs to go in, then it has to have all the crap I did to deal with the flakiness re-added.

If you try to demand it be reasonably stable from the start, it will never be added by anyone.

[renegadelink]

After testing, this PR needs much more work as the API supplied is too much unreliable

Understood. Well if anyone wants to help test by running -vv and giving me their logs I can try and help fix any issues. I don't have any issues on my test device so unfortunately there's not much more I can do. Should I close the PR?