Data sync (not a bug)

[user1321]

Good Day!
First of all thank you for sharing your project!
did you tried to use this project with real gyro sensor? Something like MPU6500.

I just thinking about how to sync sensor data with GoPro. In other words how to avoid async of gyro data and video.

To be more clear why I asking, probably you heard about https://www.reelsteady.com project (close source codes) . It able to stabilize video from GoPro 5 black (and up) using gyro data.

So, idea is similar, only difference is video source can be lets say RPi camera.

Did you tried something similar or that project is more about encapsulate GPS data ?

[user1321]

Good Day!

In overall - I failed with that idea, even if MP4 file was imported to ReelSteady. It is better to spend time on something else. (I am talking about Reelsteady)
But, just for anyone how will think to do same here are some notes from my side to just save your time not doing same mistakes. Maybe it will helps somehow.

Long story:

So, unlike DashWar program that work outof the box with generated MP4 file by "mp4-inject-sensor-data", Reelsteady require not only data stream in GoPro format, but also MP4 file generated in same way as GoPro camera generate its files. And that is reall pain.

There are two interesting DLL file:

1. Assembly-CSharp.dll - .Net file that can be easly recovered to C# using "dnSpy". That DLL probably a core of the program. (Called by ReelSteadyGoTrial.exe )
2. RSMovieParser.dll - Probably VC. (So, IDA PRO).

After trying to import to ReelSteady app MP4 files generated by gstreamer, ReelSteady report error:
"Invalid Movie, no metadata found.".

Firstly I was thinking that something wrong with "mp4-inject-sensor-data" , for example missing MP4 meta tag: Firmware, but after it was added - same.
Latter I understood that there is also missing UDTA tags:
\mp4box -info -v hero5.mp4 ..... UDTA types: A978797A (1) FIRM (1) LENS (1) CAME (1) SETT (1) AMBA (1) MUID (1) HMMT (1) BCID (1) GUMI (1) free (1) ...

It is possible to extract them from original GoPro MP4 file and import to generated MP4 file. But, that will do nothing.
Same error: "Invalid Movie, no metadata found.".
Even if there is UDTA tags and same metatags.
So, there something else - and that that is HOW MP4 file generated.
ReelSteady require exactly same file format.

For example:

So, to fix that I was hoping to understand how ReelSteady actually works:

But, sorry, I can't understand from IDA PRO what why it generate error code( Maybe you can explain.

So, to overcome that it is possible to use "mp4extract" and "mp4edit". (Keep in mind that I was failed to use ffmpeg because it generate own MP4 header even while just copy video streams from working MP4 file.)

mp4extract and mp4edit way: (source hero6.mp4 file . ReelSteady must process that file fine without any errors)

`
mp4extract ftyp hero6.mp4 ftyp6.atom
mp4extract moov hero6.mp4 moov6.atom
mp4extract moov/udta hero6.mp4 udta6.atom
mp4extract mdat hero6.mp4 mdat6.atom

mp4edit --remove ftyp --insert :ftyp6.atom:0 --remove free out30.mp4 out_editHero6.mp4
mp4edit --insert moov:udta6.atom:1 out_editHero6.mp4 out_editHero6_2.mp4
mp4edit --insert :mdat6.atom:1 out_editHero6_2.mp4 out_editHero6_3.mp4
`

Funny, but it is not enough to just copy UDTA, mdat also required.
Probably ReelSteady is detecting gopro not by UDTA, but from very beginning of the MP4 file.
Once again if you able to understand result of IDA Pro, probably that can be solved in better way.

After upper commands error message: "Invalid Movie, no metadata found" was gone.
But, other one comes up: "Sorry, this shooting mode is not supported" ( If I remember correctly)

To fix not expected FPS, hardcode value was added:

Also, keep in mind that ReelSteady expecting to see same resolution as in original gopro
MP4. (not sure about audio stream)
So, gstreamer pipeline was changed a bit:
mp4-inject-sensor-data \ main.cpp

`
std::string launch_string =
"videotestsrc is-live=true"
" ! queue"
" ! video/x-raw, width=2704, height=1520, format=UYVY"
" ! videoconvert"
" ! x264enc"
" ! muxer.video_0 audiotestsrc is-live=true"
" ! audioresample"
" ! avenc_aac"
" ! muxer.audio_0 mp4mux name=muxer streamable=true"
" ! queue"
" ! filesink name=filesink location=out.mp4";

g_print("launch_string: %s\n", launch_string.c_str());
//g_object_set(src, "pattern", 2, NULL);

pipeline = gst_parse_launch (launch_string.c_str(), NULL);

mp4mux = gst_bin_get_by_name(GST_BIN(pipeline), "muxer");
`
To build gstreamer was used this repo:
https://gitlab.freedesktop.org/erlend_ne/gst-plugins-good/tree/mp4mux-add-gpmf-track-latest

mp4mux-add-gpmf-track-latest/gst/isomp4/atoms.c
was changed a bit: (\tGoPro AAC)
`
atom_mdia_set_hdlr_type_audio (AtomMDIA * mdia, AtomsContext * context)
{
atom_hdlr_set_type (&mdia->hdlr, context, FOURCC_mhlr, FOURCC_soun);
/* Some players (low-end hardware) check for this name, which is what

• QuickTime itself sets */
  atom_hdlr_set_name (&mdia->hdlr, "\tGoPro AAC");
  }
  `
  Dont know if it makes any difference. (it was done before mp4edit )

So, final result:

As you may see it is also required to adjust Lens FOV.
Probably not a problem.
But biggest disappointment ReelSteady require Internet connection to work.
It is not just license check, it do some video stab calculations on Cloud.
So, in my opinion it not a problem to buy licence for 100$, but, it will work as long as Cloud service will be running. As soon Cloud will be turned off local software will immediately break.
Please, correct me if I am wrong.

For me that mean that there no sense to convert RPi camera to GoPro( At least using this software.

[erlenner]

@user1321 Hi! This seems to have slipped through the cracks. Sorry about that.

I haven't tried doing any kind of sensor fusion based post processing using this project. However, I might do so in the future, so thank you for the notes :). I might have a go at getting ReelSteady to accept the video files directly in the not too distant future.

As you discovered, I did create a gst-plugins-good branch where I set the hdlr atom to \tGoPro AVC . This was needed in order to get GoPro Quik to accept the videos. It is mentioned here, but I should definitely update the readme as well to point to the correct branch.