Very slow OCR, Misclicks, and Misidentifications in Rerollbot

[JosXa]

I just tried out the rerollbot and run into a variety of problems.

1. The Tesseract detection is quite slow, definitely nowhere near the advertised 2 per second.
2. The bot appears to misclick when it's found an item it likes to purchase, it clicks too low on the y axis, or attempts to purchase the wrong item
3. It misidentifies items it's supposed to purchase (I think, this might just be point 2. though).

See this screencast which hopefully has all the information you need to help me debug this:

https://drive.google.com/file/d/1-2QcOV7YIkUTWnuemdOgU4wWkiCEZAjG/view?usp=sharing
(too large for GitHub, thus Google Drive)

[NiHoel]

Hi,

1. In debug mode it is slow. But the main reason is that it often does not recognize the trader (see 2.)
2. Your resolution is very odd (1920x1200) as far as I can tell from the video. My tool is only suited for 16:9 (i.e. 1920x1080, 2560x1440, 3840x2160) or widescreen (3440x1440). It is best to change to one of these resolutions. Your resolution leads to a different positioning of the trade menu such that the crops of the screenshot (see right part of the video) to detect items are off:

[NiHoel]

If you are a bit into programming, you can modify the values in

Anno1800UXEnhancer/cpp/visual studio/CalculatorServer/reader/reader_trading.cpp

Lines 46 to 57 in fba9ca6

	const cv::Rect2f trading_params::pane_menu_offering = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.62303, trading_params::off_y + 0.4048), cv::Point2f(trading_params::off_x + 0.78253, trading_params::off_y + 0.66492));
	const cv::Rect2f trading_params::pane_menu_offering_with_counter = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.62272, trading_params::off_y + 0.45134), cv::Point2f(trading_params::off_x + 0.78265, trading_params::off_y + 0.71442));
	const cv::Rect2f trading_params::pane_menu_name = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.64018, trading_params::off_y + 0.33630), cv::Point2f(trading_params::off_x + 0.75973, trading_params::off_y + 0.36261));
	const cv::Rect2f trading_params::pane_menu_reroll = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.68871, trading_params::off_y + 0.70479), cv::Point2f(trading_params::off_x + 0.71735, trading_params::off_y + 0.75452));
	const cv::Rect2f trading_params::pane_menu_execute = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.43937, trading_params::off_y + 0.76836), cv::Point2f(trading_params::off_x + 0.56126, trading_params::off_y + 0.79713));
	const cv::Rect2f trading_params::pane_menu_title = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.45754, trading_params::off_y + 0.18393), cv::Point2f(trading_params::off_x + 0.528, trading_params::off_y + 0.22357));
	const cv::Rect2f trading_params::pane_menu_ship_cargo = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.22050, trading_params::off_y + 0.43526), cv::Point2f(trading_params::off_x + 0.37741, trading_params::off_y + 0.63224));
	const cv::Rect2f trading_params::pane_menu_ship_sockets = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.22019, trading_params::off_y + 0.51098), cv::Point2f(trading_params::off_x + 0.37766, trading_params::off_y + 0.71486));
	const cv::Rect2f trading_params::pane_menu_available_items = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.65177, trading_params::off_y + 0.40448), cv::Point2f(trading_params::off_x + 0.75311, trading_params::off_y + 0.42691));
	const cv::Rect2f trading_params::pane_tooltip_reroll_heading = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.72351, trading_params::off_y + 0.76533), cv::Point2f(trading_params::off_x + 0.83483, trading_params::off_y + 0.79372));
	const cv::Rect2f trading_params::pane_tooltip_reroll_price = cv::Rect2f(cv::Point2f(trading_params::off_x + 0.79202, trading_params::off_y + 0.82894), cv::Point2f(trading_params::off_x + 0.84363, trading_params::off_y + 0.85168));

and compile the program. I use the following Geogebra sheet to get the values:

https://drive.google.com/file/d/1L4eaConT5wBHj_hIemVw8lHWs4hMXckF/view?usp=sharing

Select points A,B,C simultanously and insert an image. Then, adjust the rectangles such that they fit agian. Finally, copy the new values to the source code (discard the '-' in front of the y-coordinate)

[JosXa]

Right, I figured something along those lines. I'm not well versed in C++, but if I suddenly find myself with a bunch of time maybe I'll open a PR to make this configurable?

[NiHoel]

Presumably, the UI scales with the aspect ratio. So one could code this (even for other aspects). See how I did this for widescreen:

Anno1800UXEnhancer/cpp/visual studio/CalculatorServer/reader/reader_trading.cpp

Line 94 in fba9ca6

recog.crop_widescreen(img).copyTo(this->screenshot);

One could crop the height in your case. I just don't have the time.