Skip to content

added cropping of capture/background #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 26 commits into
base: main
Choose a base branch
from
Draft

added cropping of capture/background #153

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
127009b
added cropping of capture/background
Sep 1, 2022
65627e8
Suggestions mostly added, a little bugfix
Sep 1, 2022
7b41672
Suggestions mostly added, a little bugfix
Sep 2, 2022
cb0fe4f
fix typo
Sep 2, 2022
3eeb58b
typo correction
Sep 2, 2022
01e9a75
fix typo
Sep 2, 2022
f81e26e
Syntax modified
Sep 2, 2022
112e979
Syntax modified
Sep 2, 2022
22472ad
correction
Sep 2, 2022
7dc9bcc
coorection as suggested
Sep 2, 2022
6894b3e
coorection as suggested
Sep 2, 2022
1769f7e
suggestion processed
Sep 3, 2022
e9a5242
bug fixed
Sep 4, 2022
0f62a57
bug fixed
Sep 4, 2022
81c1c83
bug fix
Sep 4, 2022
4c5f72c
bug fix
Sep 4, 2022
ffb9883
type
Sep 4, 2022
7353ff8
typo
Sep 4, 2022
5a92b93
typo
Sep 4, 2022
5ee0d4a
bug fixed
Sep 8, 2022
44dbb6a
Fix and suggestion considered
Sep 10, 2022
4c6431e
little cleanup
Sep 10, 2022
975b96f
little corrections
Sep 11, 2022
481cb47
little corrections
Sep 11, 2022
f80a8a6
bettet video/image detection, gstreamer support foranimated gif
Oct 1, 2022
ea1e350
better crop handling
Oct 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions app/background.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#include <opencv2/videoio.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>

#include <lib/libbackscrub.h>
// Internal state of background processing
struct background_t {
int debug;
Expand Down Expand Up @@ -183,11 +183,14 @@ int grab_background(std::shared_ptr<background_t> pbkd, int width, int height, c
if (pbkd->video) {
// grab frame & frame no. under mutex
std::unique_lock<std::mutex> hold(pbkd->rawmux);
cv::resize(pbkd->raw, out, cv::Size(width, height));
cv::Rect_<int> crop = calcCropping(pbkd->raw.cols,pbkd->raw.rows,width, height);
cv::resize(pbkd->raw(crop), out, cv::Size(width, height));
frm = pbkd->frame;
} else {
// resize still image as requested into out
cv::resize(pbkd->raw, out, cv::Size(width, height));
cv::Rect_<int> crop = calcCropping(pbkd->raw.cols,pbkd->raw.rows,width, height);
cv::resize(pbkd->raw(crop), out, cv::Size(width, height));
out = pbkd->raw;
frm = 1;
}
return frm;
Expand Down
54 changes: 49 additions & 5 deletions app/deepseg.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ int main(int argc, char* argv[]) try {
bool flipVertical = false;
int fourcc = 0;
size_t blur_strength = 0;
cv::Rect_<int> crop_region(0,0,0,0);

const char* modelname = "selfiesegmentation_mlkit-256x256-2021_01_19-v1215.f16.tflite";

Expand Down Expand Up @@ -568,6 +569,15 @@ int main(int argc, char* argv[]) try {
if (expWidth != vidGeo.value().first) {
fprintf(stderr, "Warning: virtual camera aspect ratio does not match capture device.\n");
}
// calculate crop region, only if result always smaller
if (expWidth != vidGeo.value().first &&
vidGeo.value().first <= capGeo.value().first &&
vidGeo.value().second <= capGeo.value().second) {
crop_region = calcCropping(capGeo.value().first,
capGeo.value().second,
vidGeo.value().first,
vidGeo.value().second);
}

// dump settings..
printf("debug: %d\n", debug);
Expand Down Expand Up @@ -600,7 +610,12 @@ int main(int argc, char* argv[]) try {
}
}
// default green screen background (at capture true geometry)
cv::Mat bg = cv::Mat(capGeo.value().second, capGeo.value().first, CV_8UC3, cv::Scalar(0, 255, 0));
cv::Mat bg;
if ( crop_region.height == 0) {
bg = cv::Mat(capGeo.value().second, capGeo.value().first, CV_8UC3, cv::Scalar(0, 255, 0));
} else {
bg = cv::Mat(crop_region.width, crop_region.height, CV_8UC3, cv::Scalar(0, 255, 0));
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DRY:

Suggested change
cv::Mat bg;
if ( crop_region.height == 0) {
bg = cv::Mat(capGeo.value().second, capGeo.value().first, CV_8UC3, cv::Scalar(0, 255, 0));
} else {
bg = cv::Mat(crop_region.width, crop_region.height, CV_8UC3, cv::Scalar(0, 255, 0));
}
std::pair<int, int> bg_dim = !crop_region.height ?
{ capGeo.value().second, capGeo.value().first } :
{ crop_region.width, crop_region.height };
cv::Mat bg = cv::Mat(bg_dim.second, bg_dim.first, CV_8UC3, cv::Scalar(0, 255, 0));

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a mistake here! I must check this!

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not okay:

bg_dim = !crop_region.height ? {...} : {...};

gcc 12.2.1 is not happy, and try to assign !crop_region.height and ignore the rest.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I hoped the compiler can resolve the initializer list, but you might need to make it explicit as std::make_pair(a, b).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have tried to solve this, but there is no advantage compared to my first code, only more code.


// Virtual camera (at specified geometry)
int lbfd = loopback_init(s_vcam, vidGeo.value().first, vidGeo.value().second, debug);
Expand All @@ -615,9 +630,26 @@ int main(int argc, char* argv[]) try {


// Processing components, all at capture true geometry
cv::Mat mask(capGeo.value().second, capGeo.value().first, CV_8U);
cv::Mat mask;
if ( crop_region.height ) {
cv::Mat masktmp((int)(crop_region.height), (int)(crop_region.width), CV_8U);
mask = masktmp;
} else {
cv::Mat masktmp(capGeo.value().second, capGeo.value().first, CV_8U);
mask = masktmp;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optimize similar to example shown above. masktmp is unused (except for assignment). Drop unnecessary parens.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See previous not committed suggestion.

Copy link
Author

@jjsarton jjsarton Sep 2, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may possibly be:

	std::optional<std::pair<size_t, size_t>> mask_dim = capGeo;
	if ( crop_region.height ) {
		mask_dim = {crop_region.width, crop_region.height};
	}
	cv::Mat mask(bg_dim.value().second, bg_dim.value().first, CV_8U);

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace std::pair<size_t, size_t> by cv::Size and the overall code will magically shrink quite a bit …


cv::Mat raw;
CalcMask ai(s_model.value(), threads, capGeo.value().first, capGeo.value().second);
int aiw,aih;
if ( crop_region.width == 0) {
aiw=capGeo.value().first;
aih=capGeo.value().second;
} else {
aiw=crop_region.width;
aih=crop_region.height;
}
CalcMask ai(s_model.value(), threads, aiw, aih);

ti.lastns = timestamp();
printf("Startup: %ldns\n", diffnanosecs(ti.lastns,ti.bootns));

Expand All @@ -631,11 +663,15 @@ int main(int argc, char* argv[]) try {
// copy new frame to buffer
cap.retrieve(raw);
ti.retrns = timestamp();
ai.set_input_frame(raw);
ti.copyns = timestamp();

if (raw.rows == 0 || raw.cols == 0) continue; // sanity check

if ( crop_region.height) {
raw((cv::Rect_<int>)crop_region).copyTo(raw);
}
ai.set_input_frame(raw);

if (filterActive) {
// do background detection magic
ai.get_output_mask(mask);
Expand All @@ -646,7 +682,15 @@ int main(int argc, char* argv[]) try {
// - default green (initial value)
bool canBlur = false;
if (pbk) {
if (grab_background(pbk, capGeo.value().first, capGeo.value().second, bg)<0)
int tw,th;
if ( crop_region.height ) {
tw = crop_region.width;
th = crop_region.height;
} else {
tw = capGeo.value().first;
th = capGeo.value().second;
}
if (grab_background(pbk, tw, th, bg)<0)
throw "Failed to read background frame";
canBlur = true;
} else if (blur_strength) {
Expand Down
25 changes: 24 additions & 1 deletion lib/libbackscrub.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -365,7 +365,11 @@ bool bs_maskgen_process(void *context, cv::Mat &frame, cv::Mat &mask) {

// scale up into full-sized mask
cv::Mat tmpbuf;
cv::resize(ctx.ofinal(ctx.in_roidim),tmpbuf,ctx.mroi.size());
// with body-pix-float-050-8.tflite the size of ctx.ofinal is 33x33
// and the wanted roi may be greater as 33x33 so we can crash with
// cv::resize(ctx.ofinal(ctx.in_roidim),tmpbuf,ctx.mroi.size());
ctx.ofinal.copyTo(tmpbuf);
cv::resize(tmpbuf,tmpbuf,ctx.mroi.size());

// blur at full size for maximum smoothness
cv::blur(tmpbuf,ctx.mroi,ctx.blur);
Expand All @@ -375,3 +379,22 @@ bool bs_maskgen_process(void *context, cv::Mat &frame, cv::Mat &mask) {
return true;
}

cv::Rect calcCropping(int cw, int ch, int vw, int vh)
{
// if the input and output aspect reatio are not the same
// we can crop the source image. For example if the
// input image has a 16:9 (1280x720) ratio and the output is 4:3 (960x720)
// we will return the cropRegion set as x=160, width=960, y=0, height=720
// which the centered part ofe the original image
cv::Rect cropRegion = {0,0,0,0};
float sc = (float)vw / cw;
float st = (float)vh / ch;
sc = st > sc ? st : sc;
int sz = (int)(vw / sc) - cw;
cropRegion.x = (sz < 0 ? -sz : sz)/2;
sz = (int)(vh / sc) - ch;
cropRegion.y = (sz < 0 ? -sz : sz)/2;
cropRegion.width = cw - cropRegion.x*2;
cropRegion.height = ch - cropRegion.y*2;
return cropRegion;
}
2 changes: 2 additions & 0 deletions lib/libbackscrub.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,4 +38,6 @@ extern void bs_maskgen_delete(void *context);
// Process a video frame into a mask
extern bool bs_maskgen_process(void *context, cv::Mat& frame, cv::Mat &mask);

extern cv::Rect calcCropping(int inWidth, int inHeight, int targetWidth, int targetHight);

#endif