Recording and playback initial version: Supports simple recording and basic playback capabilities. #66

lixf6 · 2025-11-22T08:41:27Z

录制回放初版：支持简单的录制和基本的回放能力

Copilot

Pull request overview

This PR introduces an initial version of recording and playback functionality for mobile UI automation. It adds the ability to record user interactions, save them as scripts, generate code in multiple formats (Appium Python/JS, UIAutomator2, XCUITest), and replay recorded actions.

Key Changes:

New /api/record endpoints for saving, listing, retrieving, generating, and executing recorded scripts
Support for recording tap, input, swipe, back, and home actions with element selectors and coordinates
Script generation in multiple automation framework formats
Script execution engine with support for both UIAutomator2 and ADB-based drivers

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 23 comments.

Show a summary per file

File	Description
uiautodev/router/record.py	New recording API router with endpoints for script management and playback, including script generators for multiple frameworks
uiautodev/model.py	Added data models for recording functionality (Selector, RecordEvent, RecordScript, SaveScriptRequest, SaveScriptResponse)
uiautodev/driver/android/adb_driver.py	Enhanced app_launch method with stop_first parameter and monkey command for more reliable app launching
uiautodev/command_types.py	Changed default value of AppLaunchRequest.stop from False to True for cleaner app launches
uiautodev/command_proxy.py	Updated app_launch function to support new stop_first parameter with backward compatibility
uiautodev/app.py	Registered record router and added DELETE/PUT methods to CORS middleware

Comments suppressed due to low confidence (1)

uiautodev/driver/android/adb_driver.py:170

This statement is unreachable.

        try:

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-22T08:48:35Z

uiautodev/router/record.py

+            )
+
+        # Wait a bit between actions
+        time.sleep(0.5)


Magic number detected: The sleep duration 0.5 is a magic number. Consider defining it as a named constant (e.g., ACTION_DELAY = 0.5) to improve code readability and maintainability.

Copilot · 2025-11-22T08:48:35Z

uiautodev/command_types.py

 class AppLaunchRequest(BaseModel):
    package: str
-    stop: bool = False
+    stop: bool = True  # Default to True: stop app before launch for clean start


Breaking change in default behavior: The default value of stop has been changed from False to True. This means that existing code calling AppLaunchRequest without explicitly setting stop will now stop the app before launching, which is a behavioral change that could break existing workflows. Consider documenting this change clearly or using a different parameter name to avoid breaking existing code.

Suggested change

stop: bool = True # Default to True: stop app before launch for clean start

stop: bool = False # Default to False: do not stop app before launch (backward compatible)

Copilot · 2025-11-22T08:48:35Z

uiautodev/router/record.py

+                    lines.append(f'    await driver.$("//*[@text=\'{event.selector.text}\']").click();')
+            elif event.action == "input":
+                if event.selector and event.selector.id:
+                    lines.append(f'    await driver.$("#{event.selector.id}").setValue("{event.value or ""}");')


Potential code injection vulnerability: event.value is directly interpolated into JavaScript code without escaping. Input values containing double quotes or special characters could break the generated code. Consider using proper escaping for the generated string.

Suggested change

lines.append(f' await driver.$("#{event.selector.id}").setValue("{event.value or ""}");')

lines.append(f' await driver.$("#{event.selector.id}").setValue({json.dumps(event.value or "")});')

Copilot · 2025-11-22T08:48:35Z

uiautodev/router/record.py

+            if event.x is not None and event.y is not None:
+                print(f"[execute_script] [COORDINATE] Tapping at recorded coordinates ({event.x}, {event.y})")
+                logger.info(f"[COORDINATE] Tapping at recorded coordinates ({event.x}, {event.y})")
+                driver.tap(int(event.x), int(event.y))
+            else:
+                error_msg = "No selector or coordinates provided for tap action"
+                print(f"[execute_script] {error_msg}")
+                logger.error(error_msg)
+                return ScriptExecutionResult(
+                    step=step_index + 1,
+                    action=event.action,
+                    success=False,
+                    message=error_msg,
+                    error=error_msg
+                )


Duplicate code block detected. Lines 545-559 are identical to lines 528-544. This unreachable code will never execute because all code paths in the preceding block either return early or complete the tap action. This duplicated block should be removed.

Suggested change

if event.x is not None and event.y is not None:

print(f"[execute_script] [COORDINATE] Tapping at recorded coordinates ({event.x}, {event.y})")

logger.info(f"[COORDINATE] Tapping at recorded coordinates ({event.x}, {event.y})")

driver.tap(int(event.x), int(event.y))

else:

error_msg = "No selector or coordinates provided for tap action"

print(f"[execute_script] {error_msg}")

logger.error(error_msg)

return ScriptExecutionResult(

step=step_index + 1,

action=event.action,

success=False,

message=error_msg,

error=error_msg

)

Copilot · 2025-11-22T08:48:36Z

uiautodev/router/record.py

+                    lines.append(f'    await driver.$("#{event.selector.id}").click();')
+                elif event.selector and event.selector.text:
+                    lines.append(f'    await driver.$("//*[@text=\'{event.selector.text}\']").click();')


Potential code injection vulnerability: event.selector.id is directly interpolated into JavaScript selector without escaping. Special characters in the ID could break the generated code. Consider using proper escaping for the generated string.

Copilot · 2025-11-22T08:48:39Z

uiautodev/router/record.py

+                    lines.append(f'        app.buttons["{event.selector.id}"].tap()')
+                elif event.selector and event.selector.text:
+                    lines.append(f'        app.staticTexts["{event.selector.text}"].tap()')
+            elif event.action == "input":
+                if event.selector and event.selector.id:
+                    lines.append(f'        app.textFields["{event.selector.id}"].typeText("{event.value or ""}")')


Potential code injection vulnerability: event.selector.id, event.selector.text, and event.value are directly interpolated into Swift code without escaping. Special characters in these values could break the generated code. Consider using proper escaping for the generated strings.

Copilot · 2025-11-22T08:48:39Z

uiautodev/router/record.py

+
+import json
+import logging
+import os


Import of 'os' is not used.

Suggested change

import os

Copilot · 2025-11-22T08:48:39Z

uiautodev/router/record.py

+
+from uiautodev.model import RecordEvent, RecordScript, SaveScriptRequest, SaveScriptResponse
+from uiautodev.provider import AndroidProvider
+from uiautodev.command_proxy import send_command


Import of 'send_command' is not used.

Suggested change

from uiautodev.command_proxy import send_command

Copilot · 2025-11-22T08:48:39Z

uiautodev/router/record.py

+from uiautodev.model import RecordEvent, RecordScript, SaveScriptRequest, SaveScriptResponse
+from uiautodev.provider import AndroidProvider
+from uiautodev.command_proxy import send_command
+from uiautodev.command_types import Command, TapRequest, SendKeysRequest, By, FindElementRequest


Import of 'By' is not used.
Import of 'Command' is not used.
Import of 'TapRequest' is not used.
Import of 'SendKeysRequest' is not used.
Import of 'FindElementRequest' is not used.

Suggested change

from uiautodev.command_types import Command, TapRequest, SendKeysRequest, By, FindElementRequest

# Removed unused imports: Command, TapRequest, SendKeysRequest, By, FindElementRequest

Copilot · 2025-11-22T08:48:40Z

uiautodev/command_proxy.py

+            # Driver supports stop_first parameter
+            driver.app_launch(params.package, stop_first=stop_first)
+            return
+    except (TypeError, AttributeError):


'except' clause does nothing but pass and there is no explanatory comment.

Suggested change

except (TypeError, AttributeError):

except (TypeError, AttributeError):

# If signature inspection fails, fallback to manual stop/launch below.

codeskyblue · 2025-11-22T09:06:38Z

uiautodev/router/record.py

+    total: int
+
+
+class ScriptGenerator:


代码生成没必要放到后端，放前端就可以。这样后面升级也方便

codeskyblue · 2025-11-22T09:09:49Z

uiautodev/driver/android/adb_driver.py

        )

-    def app_launch(self, package: str):
+    def app_launch(self, package: str, stop_first: bool = True):


stop_first这里不要加，分开成两个操作。 app_stop() 和 app_launch(). 因为除了android驱动还有ios，鸿蒙驱动。修改一个就好把其他的两个也一起改了。不划算

录制回放初版：支持简单的录制和基本的回放能力

2439c78

Copilot AI review requested due to automatic review settings November 22, 2025 08:41

Copilot started reviewing on behalf of lixf6 November 22, 2025 08:41 View session

Copilot finished reviewing on behalf of lixf6 November 22, 2025 08:45

Copilot AI reviewed Nov 22, 2025

View reviewed changes

optimize magic number

f90a0d4

codeskyblue requested changes Nov 22, 2025

View reviewed changes

lixf6 added 2 commits November 23, 2025 15:21

support frontend json post replay execute

1c7e7dd

optimize app_launch

ec7d49b

	stop: bool = True # Default to True: stop app before launch for clean start
	stop: bool = False # Default to False: do not stop app before launch (backward compatible)

	lines.append(f' await driver.$("#{event.selector.id}").setValue("{event.value or ""}");')
	lines.append(f' await driver.$("#{event.selector.id}").setValue({json.dumps(event.value or "")});')

	from uiautodev.command_types import Command, TapRequest, SendKeysRequest, By, FindElementRequest
	# Removed unused imports: Command, TapRequest, SendKeysRequest, By, FindElementRequest

	except (TypeError, AttributeError):
	except (TypeError, AttributeError):
	# If signature inspection fails, fallback to manual stop/launch below.

Recording and playback initial version: Supports simple recording and basic playback capabilities. #66

Are you sure you want to change the base?

Recording and playback initial version: Supports simple recording and basic playback capabilities. #66

Uh oh!

Conversation

lixf6 commented Nov 22, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

lixf6 Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

lixf6 Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

lixf6 Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

codeskyblue Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

codeskyblue Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants