View Categories

7.2. Working with UIObjects


Stego has 7 types of UIObjects. Drag the UIObject you want from the screen to the UIObject Field to add it.




OD (Object Detection) #


Use buttons, menus, input fields, and other elements that AI recognizes in mobile apps.


Inspector #


View information about UIObjects that AI recognizes through OD (Object Detection) in the Inspector panel.

  • LABEL: UIObject type
  • TEXT: Text contained in the UIObject (if any)
  • BOX: UIObject position information on screen


Attributes #


Click OD in the UIObject Field to change OD attributes.

  • Label: UIObject type
  • Text: Change UIObject text and set comparator
    • Comparator
      • =: Check if UIObject Text exactly matches the value
      • *=: Check if the value is contained in UIObject Text
      • ^=: Check if the value matches the beginning of UIObject Text
      • $=: Check if the value matches the end of UIObject Text
      • search: Check for matches using regular expressions
      • not used: Don’t use Text as a comparator
  • Text Similarity (Threshold): Only available when comparator is ‘=’
    • Set how similar text needs to be for matching
    • Allows similarity adjustment since identical text may be recognized differently due to resolution or environment differences
    • Higher values require more exact text matches (default: 0.8, range: 0~1)
  • Case Sensitivity: Set whether to distinguish between uppercase and lowercase letters
  • Selector: Specify UIObject order to use when multiple matching UIObjects exist
    • Positive numbers: Select in order from top of screen (1, 2, 3…)
    • Negative numbers: Select in order from bottom of screen (-1, -2, -3…)




OCR (Optical Character Recognition) #


Use text information from UIObjects. Currently supports Korean, English, numbers, and some special characters.


OCR recognizes text in word units.

When you need to use a sentence made up of multiple words as one UIObject, you can drag multiple words to use them as one UIObject.


Inspector #


View information about UIObjects that AI recognizes through OCR (Optical Character Recognition) in the Inspector panel.

  • TEXT: Text contained in the UIObject
  • BOX: UIObject position information on screen


Attributes #


Click OCR in the UIObject Field to change OCR attributes.

  • Text: Change UIObject text and set comparator
    • Comparator
      • =: Check if UIObject Text exactly matches
      • *=: Check if the value is contained in UIObject Text
      • ^=: Check if the value matches the beginning of UIObject Text
      • $=: Check if the value matches the end of UIObject Text
      • search: Check for matches using regular expressions
  • Ignore Line Break: Ignore line breaks when determining if Text matches (only available when comparator is ‘=’)
  • Text Similarity (Threshold): Only available when comparator is ‘=’
    • Set how similar text needs to be for matching
    • Allows similarity adjustment since identical text may be recognized slightly differently due to resolution or environment differences
    • Higher values require more exact text matches (default: 0.8, range: 0~1)
  • Case Sensitivity: Set whether to distinguish between uppercase and lowercase letters
  • Font Style Sensitivity: Set whether to distinguish font style similarity
  • Selector: Specify UIObject order to use when multiple matching UIObjects exist
    • Positive numbers: Select in order from top of screen (1, 2, 3…)
    • Negative numbers: Select in order from bottom of screen (-1, -2, -3…)




Crop Image #


Use when finding images that appear on screen.


Drag on the device screen to choose the desired area.

You can find UIObjects more accurately by excluding the background as much as possible.


Attributes #


Click Crop Image in the UIObject Field to change Crop Image coordinates.

  • Left: X coordinate of UIObject’s top-left corner
  • Top: Y coordinate of UIObject’s top-left corner
  • Right: X coordinate of UIObject’s bottom-right corner
  • Bottom: Y coordinate of UIObject’s bottom-right corner




Custom Box #


Use when specifying a particular area on screen.


Drag on the device screen to choose the desired area.

Use this when performing actions over wide areas like scrolling or swiping.


Attributes #


Click Custom Box in the UIObject Field to change Custom Box coordinates.

  • Left: X coordinate of UIObject’s top-left corner
  • Top: Y coordinate of UIObject’s top-left corner
  • Right: X coordinate of UIObject’s bottom-right corner
  • Bottom: Y coordinate of UIObject’s bottom-right corner




Full Screen #


Use the entire screen as a UIObject.


Attributes #


You cannot change Full Screen attributes.

  • Left: X coordinate of UIObject’s top-left corner
  • Top: Y coordinate of UIObject’s top-left corner
  • Right: X coordinate of UIObject’s bottom-right corner
  • Bottom: Y coordinate of UIObject’s bottom-right corner




Accessibility #


Use when the screen doesn’t display normally due to screen security settings (black screen or some areas not visible).


Set by choosing a UIObject in Inspector and dragging.


Inspector #


View information about UIObjects that AI recognizes through Accessibility in the Inspector panel.

  • IDENTIFIER: Text that distinguishes the UIObject (when provided by the app)
  • TYPE: UIObject type
  • TEXT: Text contained in the UIObject (if any)
  • BOX: UIObject position information on screen


Attributes #


Click Accessibility in the UIObject Field to change Accessibility attributes.

  • Identifier: Set UIObject identifier text (when provided by the app) (e.g., ReloadButton)
    • Comparator
      • =: Check if it matches the UIObject’s Identifier
      • not used: Don’t use Identifier as a comparator
  • Type: Set UIObject type (when provided by the app) (e.g., Button)
    • Comparator
      • =: Check if it matches the UIObject’s Type
      • not used: Don’t use Type as a comparator
  • Text: Change UIObject text and set comparator
    • Comparator
      • =: Check if UIObject Text exactly matches
      • *=: Check if the value is contained in UIObject Text
      • ^=: Check if the value matches the beginning of UIObject Text
      • $=: Check if the value matches the end of UIObject Text
      • search: Check for matches using regular expressions
      • not used: Don’t use Text as a comparator
  • Text Similarity (Threshold): Only available when comparator is ‘=’
    • Set how similar text needs to be for matching
    • Allows similarity adjustment since identical text may be recognized slightly differently due to resolution or environment differences
    • Higher values require more exact text matches (default: 0.8, range: 0~1)
  • Case Sensitivity: Set whether to distinguish between uppercase and lowercase letters
  • Font Style Sensitivity: Set whether to distinguish font style similarity
  • Selector: Specify UIObject order to use when multiple matching UIObjects exist
    • Positive numbers: Select in order from top of screen (1, 2, 3…)
    • Negative numbers: Select in order from bottom of screen (-1, -2, -3…)




Relative #


Use positions calculated from a reference UIObject as a UIObject.


Choose another UIObject, then click the relative button to set up relative positioning.

The reference UIObject appears in yellow, and Relative appears in blue.




Attributes #


Click Relative in the UIObject Field to change reference UIObject and Relative attributes.


Reference UIObject (e.g., OCR): Same as the Attributes of the reference UIObject type.

Relative:

  • Left: X coordinate of UIObject’s top-left corner
  • Top: Y coordinate of UIObject’s top-left corner
  • Right: X coordinate of UIObject’s bottom-right corner
  • Bottom: Y coordinate of UIObject’s bottom-right corner
Scroll to Top