Midscene YAML Generator

Typical Workflow

User Requirement → [Generator] Generate YAML
                → [Generator] Auto dry-run validation
                → Validation failed? → [Generator] Auto-fix
                → [Runner] Execute
                → Execution failed? → [Runner] Analyze + Fix YAML → Re-execute
                → Success → Display report summary

Trigger Conditions

• "Generate a YAML to..."
• "Help me write an automation script..."
• "Create a Midscene test case..."
• "I want to automate the XXX operation..."
• "Convert this requirement to YAML..."
• "Write a Midscene config file..."

• "Generate a YAML for..."
• "Write an automation script to..."
• "Create a test case for..."
• "Automate the login flow"
• "Convert this requirement to YAML"
• "Write a Midscene config file for..."

Workflow

Step 1: Analyze Requirement Complexity

• Open web pages / Launch applications
• Basic interactions like clicking, hovering, inputting, scrolling, keyboard operations
• AI automatic planning and execution (

  ai

  )
• Data extraction (

  aiQuery

  )
• Validation assertions (

  aiAssert

  )
• Wait conditions (

  aiWaitFor

  )
• Tool operations (

  sleep

  ,

  javascript

  ,

  recordToReport

  )
• Platform-specific operations (

  runAdbShell

  ,

  runWdaRequest

  ,

  launch

  )

• Conditional judgment ("If...then...")
• Loop operations ("Repeat", "Traverse", "Pagination")
• Variables and dynamic data ("Define variables", "Parameterization")
• External API calls ("Call API")
• Error handling and retries ("If failed...", "Retry")
• Parallel tasks ("Do...simultaneously")
• Data transformation processing ("Filter", "Sort", "Map")
• Import and reuse sub-flows ("Reuse", "Import")

if

for

Step 2: Determine Target Platform

web: { url: "...", headless: false }

android: { deviceId: "..." }

launch: "package name"

ios: { wdaPort: 8100 }

launch: "bundleId"

computer: { ... }

• headless: true/false

  — Run in headless mode (default false)

• viewportWidth

  /

  viewportHeight

  — Viewport size (default 1280×720)

• userAgent

  — Custom User-Agent

• deviceScaleFactor

  — Device pixel ratio (e.g., set to 2 for Retina screens)

• waitForNetworkIdle

  — Network idle wait configuration, supports

  true

  or object format

  { timeout: 2000, continueOnNetworkIdleError: true }

• cookie

  — Path to Cookie JSON file (enables login-free session recovery)

• bridgeMode

  — Bridge mode:

  false

  (default) |

  'newTabWithUrl'

  |

  'currentTab'

  , reuses logged-in desktop browser

• chromeArgs

  — Array of custom Chrome launch arguments (e.g.,

  ['--disable-gpu', '--proxy-server=...']

  )

• serve

  — Local static file directory, starts built-in server

• acceptInsecureCerts

  — Ignore HTTPS certificate errors (default false)

• forceSameTabNavigation

  — Restrict navigation to current tab (default true)

Step 3: Natural Language → YAML Conversion

Action Selection Priority (Important)

1. Prefer

   ai:

   — Describe the entire intent in natural language, let AI automatically plan and execute multi-step operations. Suitable for most scenarios with the highest success rate
2. When precise control is needed — Use specific actions like

   aiTap

   ,

   aiInput

   (e.g., filling in specific form fields)
3. When data extraction is needed — Must use

   aiQuery

   (

   ai:

   cannot return structured data)
4. When state validation is needed — Use

   aiAssert

   or

   aiWaitFor

ai:

aiInput

aiTap

web:
  url: "https://www.baidu.com"

tasks:
  - name: "Search for Midscene"
    flow:
      - ai: "Enter Midscene in the search box and click search"
      - sleep: 3000
      - aiAssert: "The page displays search results"

Native Mode YAML Format Specification (Important)

- aiInput: "Search box"
  value: "Keyword"
- aiWaitFor: "Page loaded completely"
  timeout: 10000
- aiTap: "Button description"
  deepThink: true
- aiAssert: "Page contains expected content"
  errorMessage: "Content validation failed"

- aiInput:
    locator: "Search box"
    value: "Keyword"
- aiQuery:
    query: "Extract product list"
    name: "products"

Native Action Mapping

web: { url: "XXX" }

ai: "XXX"

fileChooserAccept: "path"

aiTap: "XXX"

aiHover: "XXX"

aiInput: "XXX"

value: "YYY"

mode: "replace"|"clear"|"typeOnly"|"append"

aiKeyboardPress: "XXX"

keyName

aiScroll: "Target area"

direction: "down"

distance

scrollType

aiWaitFor: "XXX"

aiAssert: "XXX"

aiQuery: { query: "XXX", name: "result" }

sleep: N*1000

javascript: "Code content"

recordToReport: "Title"

content: "Description"

aiDoubleClick: "XXX"

deepThink: true

aiRightClick: "XXX"

deepThink: true

aiLocate: "XXX"

name: "elem"

aiBoolean: "XXX"

name: "flag"

domIncluded

screenshotIncluded

aiNumber: "XXX"

name: "count"

domIncluded

screenshotIncluded

aiString: "XXX"

name: "text"

domIncluded

screenshotIncluded

aiAsk: "XXX"

name: "answer"

aiDragAndDrop: "A"

to: "B"

{ from: "A", to: "B" }

aiClearInput: "XXX"

runAdbShell: "Command"

runWdaRequest: { ... }

launch: "Package name"

Extended Control Flow Mapping

variables: { XXX: "YYY" }

${ENV:XXX}

${ENV.XXX}

logic: { if: "XXX", then: [YYY], else: [ZZZ] }

loop: { type: repeat, count: N, steps: [...] }

loop: { type: for, items: "XXX", itemVar: "item", steps: [...] }

itemVar

as

item

loop: { type: while, condition: "XXX", maxIterations: N, steps: [...] }

try: { steps: [A] }, catch: { steps: [B] }

parallel: { branches: [{steps: [A]}, {steps: [B]}], waitAll: true, merge_results: true }

external_call: { type: http, method: POST, url: "XXX", response_as: "varName" }

external_call: { type: shell, command: "XXX" }

import: [{ flow: "XXX.yaml", as: name }]

data_transform: { source, operation, ... }

Step 4: Select Template Starting Point

templates/

• templates/native/web-basic.yaml

  — Basic web operations

• templates/native/web-login.yaml

  — Login flow

• templates/native/web-data-extract.yaml

  — Data extraction

• templates/native/web-search.yaml

  — Web search flow

• templates/native/web-file-upload.yaml

  — File upload form

• templates/native/web-multi-tab.yaml

  — Multi-tab operations

• templates/native/deep-think-locator.yaml

  — Image-assisted location (deepThink/xpath)

• templates/native/android-app.yaml

  — Android testing

• templates/native/ios-app.yaml

  — iOS testing

• templates/native/computer-desktop.yaml

  — Desktop app automation

• templates/extended/web-conditional-flow.yaml

  — Conditional branching

• templates/extended/web-pagination-loop.yaml

  — Pagination loop

• templates/extended/web-data-pipeline.yaml

  — Data pipeline

• templates/extended/multi-step-with-retry.yaml

  — Multi-step with retry

• templates/extended/api-integration-test.yaml

  — API integration

• templates/extended/e2e-workflow.yaml

  — End-to-end complete workflow

• templates/extended/reusable-sub-flows.yaml

  — Sub-flow reuse (import/use)

• templates/extended/responsive-test.yaml

  — Multi-viewport responsive testing

• templates/extended/web-auth-flow.yaml

  — OAuth/login authentication flow (using variables and environment references)

native/web-basic.yaml

native/web-login.yaml

native/web-data-extract.yaml

native/web-search.yaml

native/web-file-upload.yaml

extended/web-auth-flow.yaml

native/computer-desktop.yaml

extended/web-conditional-flow.yaml

extended/web-pagination-loop.yaml

extended/web-data-pipeline.yaml

extended/multi-step-with-retry.yaml

extended/api-integration-test.yaml

extended/e2e-workflow.yaml

extended/reusable-sub-flows.yaml

extended/responsive-test.yaml

native/deep-think-locator.yaml

native/web-multi-tab.yaml

Step 5: Generate YAML

1. File Header: Add comments explaining the requirement source and generation time
2. engine field: Extended mode must explicitly declare

   engine: extended

3. features list: In Extended mode, declare the features used (e.g.,

   features: [logic, variables, loop]

   ), which can be omitted in Native mode
4. agent configuration (optional):

   testId

   is used to identify tests,

   groupName

   /

   groupDescription

   for report classification,

   cache: true

   can cache AI results to speed up repeated runs
5. aiActContext (optional): Provide additional context information for AI Agent (such as language annotation for multilingual websites, special domain terms), set in

   agent: { aiActContext: "Description" }

6. continueOnError (optional): If you need to continue executing subsequent tasks after a task fails, set

   continueOnError: true

7. output export (optional): Export results like

   aiQuery

   to a JSON file for use in subsequent processes

Output Format

# Auto-generated by Midscene YAML Generator
# Requirement Description: [Original user requirement]
# Generation Time: [timestamp]

engine: native|extended
features: [...]  # Extended mode only

# Optional: agent configuration
# agent:
#   testId: "test-001"
#   groupName: "Automation Testing Group"
#   groupDescription: "Description"
#   cache: true

[platform_config]

tasks:
  - name: "[Task Name]"
    # continueOnError: true  # Optional: Continue on failure
    flow:
      [Generated steps]
    # output:                # Optional: Export data
    #   filePath: "./midscene-output/data.json"
    #   dataName: "variableName"

Step 6: Validate and Output

1. Output the file to the

   ./midscene-output/

   directory
2. Call the validator to confirm the YAML is valid:
   bash

   node scripts/midscene-run.js <file> --dry-run

3. If validation fails, analyze the error cause and auto-fix
4. After validation passes, prompt the user to execute using Runner:
   bash

   node scripts/midscene-run.js <file>

Best Practices for Writing AI Instructions

aiTap

aiAssert

Description Precision

• Poor:

  aiTap: "Button"

  — There may be multiple buttons on the page
• Good:

  aiTap: "Blue login button at the top right corner of the page"

  — Position + Color + Function
• Better:

  aiTap: "Button with text 'Login Now' in the navigation bar"

  — Precise to text content

Location Strategy Priority

1. Natural language description (Preferred): High readability, adapts to page changes
2. deepThink mode: Enable when there are multiple similar elements on complex pages, AI will perform deeper analysis with higher accuracy but longer time consumption
3. Image-assisted location (image prompting): When text description is insufficient, screenshot annotations can be used to help AI understand the target element (official

   locate.images

   capability)
4. xpath selector (Last resort): When natural language cannot locate precisely. Note: xpath is only applicable to Web platform, Android/iOS should use natural language description

# Prefer natural language
- aiTap: "Edit button in the third row of the product list"

# Enable deepThink for complex scenarios (when there are many similar elements or location is inaccurate)
- aiTap: "Edit icon in the third row of data"
  deepThink: true

# Last resort: use xpath (Web platform only)
- aiTap: ""
  xpath: "//table/tbody/tr[3]//button[@class='edit']"

Image-assisted Location (locate object)

locate

# Use image to assist AI in identifying target element
- aiTap:
    locate:
      prompt: "Icon button similar to the reference image"
      images:
        - name: "target-icon"
          url: "https://example.com/icon.png"
      convertHttpImage2Base64: true

# Simplified form: directly provide in images option
- aiTap: "Icon button similar to the reference image"
  images:
    - "./images/target-icon.png"

aiQuery Result Formatting

query

- aiQuery:
    query: >
      Extract all product information on the page and return it as an array.
      Each element should contain the following fields:
      - name: Product name (string)
      - price: Price (number)
      - inStock: In stock (boolean)
    name: "productList"

Wait Strategy

aiWaitFor

- aiTap: "Submit button"
- aiWaitFor: "Submit success prompt appears, or page redirects to result page"
  timeout: 10000

Data Transformation Operation Reference

data_transform

filter

condition

item

sort

by

order

map

template

reduce

reducer

initial

unique

distinct

by

slice

start

end

flatten

depth

groupBy

by

field

Two Formats: Flat format

{source, operation, name}

is suitable for single-step operations; nested format

{input, operations:[], output}

supports chained multi-step operations. Both formats support all 8 operations.

Platform-Specific Notes

Web Platform

• url

  must include the full protocol (

  https://

  )
• Use

  aiWaitFor

  to wait for page loading to complete before operations
• Ensure input boxes are interactive before form operations

Android Platform

• Need to configure

  deviceId

  (ADB device ID, e.g.,

  emulator-5554

  )
• Use

  launch: "com.example.app"

  to launch the app (as an action step in flow)
• Can use

  runAdbShell

  to execute ADB commands

iOS Platform

• Need to configure

  wdaPort

  (WebDriverAgent port, default 8100) and

  wdaHost

  (default localhost)
• Use

  launch: "com.example.app"

  to launch the app (as an action step in flow)
• Can use

  runWdaRequest

  to send WebDriverAgent requests

Computer Platform

• For general desktop automation scenarios

Common Anti-patterns

• Unnecessary use of nested object format — Flat format is recommended (

  aiInput: "Search box"

  +

  value: "Keyword"

  ), which is more concise and readable. Nested format (

  aiInput: { locator: "Search box", value: "Keyword" }

  ) is valid in both modes but is usually only used when complex parameters like

  locate

  image positioning are needed
• Missing

  engine: extended

  in Extended mode — Must declare the engine when using any extended features (variables, loops, conditions, etc.)
• Forgetting

  maxIterations

  in loops —

  while

  loops must set a safety upper limit, the count of

  for

  and

  repeat

  loops should not exceed 10000
• Using nested object format for

  aiWaitFor

  — Should use

  aiWaitFor: "Condition"

  +

  timeout: 10000

  instead of

  aiWaitFor: { condition: "Condition" }

• Missing

  features

  declaration — Extended mode should list the features used to facilitate detection and optimization

Pre-output Self-check List

• Does each

  aiInput

  have a corresponding

  value

  parameter?
• Is there

  aiWaitFor

  after key operations to ensure page state is ready?
• Does Extended mode declare

  engine: extended

  and

  features

  list?
• Does the loop have a safety upper limit (

  maxIterations

  or reasonable

  count

  )?
• Are sensitive information (passwords, Tokens) referenced via environment variables using

  ${ENV:XXX}

  ?
• Are AI instruction descriptions precise enough (including features like position, text, color)?

Notes

• Parameters for AI instructions (aiTap, aiAssert, etc.) are described in natural language, no CSS selectors needed
• Both Chinese and English descriptions are acceptable, Midscene's AI engine supports multiple languages
• Results of

  aiQuery

  are stored via the

  name

  field and can be referenced in subsequent steps using

  ${name}

  (Extended mode only)
• It is recommended to set a reasonable

  timeout

  (in milliseconds) for

  aiWaitFor

  , default is usually 15 seconds
• Be sure to set

  maxIterations

  as a safety upper limit in loops to prevent infinite loops

• ${ENV:XXX}

  or

  ${ENV.XXX}

  can be used to reference environment variables, avoiding hardcoding sensitive information in YAML
• Always explicitly declare the

  engine

  field to avoid unexpected behavior from automatic detection
• Variable references are case-sensitive:

  ${userName}

  and

  ${username}

  are different variables
• Avoid circular imports: Importing B.yaml in A.yaml and A.yaml in B.yaml will cause runtime errors
• Be sure to verify syntax and structure via

  --dry-run

  after generation (Note:

  --dry-run

  does not detect model configuration, AI operations require

  MIDSCENE_MODEL_API_KEY

  to be configured for actual execution)
• Prompt users to use the Midscene Runner skill to execute the generated file

Iterative Fix Process

1. Runner can fix it automatically: If the error can be resolved by modifying YAML (e.g., imprecise location description, insufficient wait time), Runner Skill will directly modify and retry
2. When regeneration is needed: If the error involves fundamental design issues (e.g., wrong mode selected, missing key steps), users can describe the failure to Generator, which will regenerate an improved YAML based on the error information
3. Recommended Flow: Generate → dry-run validation → Execute → If failed, describe error for Generator to fix → Re-execute

Collaboration Agreement

1. Generated File Path:

   ./midscene-output/<filename>.yaml

2. Execution Mode: native or extended
3. Recommended Next Command:

   node scripts/midscene-run.js <path> --dry-run

4. If dry-run validation fails, automatically analyze the error, fix the YAML, and re-validate