UI Automation and Microsoft Active Accessibility
Microsoft Active Accessibility was the earlier solution for making applications accessible. Microsoft UI Automation is the new accessibility model for Microsoft Windows and is intended to address the needs of assistive technology products and automated testing tools. UI Automation offers many improvements over Active Accessibility.
This topic includes the main features of UI Automation and explains how these features differ from Active Accessibility.
【这里的自动化测试专指GUI自动化（不包含Web）】
以前写过一篇跟UI自动化测试有关的技术，谈到了一个自动化测试工具必备的几个功能，而且也提到了Windows平台自动化测试工具所基于的一些技术。下边就说一下这些技术的比较和展望，同时也包含了一些纠结……
- Windows API
识别窗口：需要通过FindWindow和EnumWindows来查找到窗口句柄，然后再调用其它API(GetWindowText，GetWindowRect, GetWindowLong…)来获取窗口属性，以此来找到想要的控件（窗口）
操作窗口和获取属性：通过SetWindowText和GetWindowText来操作控件上显示的文字，通过SetForegroundWindow设置顶层窗口,GetForegroundWindow获取当前的顶层窗口，类似的还有GetActiveWindow和SetActiveWindow。其它操作就不一一列举了……从理论上来说，通过Windows API和Windows Message可以完成对大部分窗口（控件也是窗口，一起都是窗口）的操作，也可以获取部分控件的部分属性。但是缺点和优点也比较明显：
优点：就是看起来很高深很强大很底层，对标准Windows的控件支持还不错
缺点：底层意味着复杂，意味着需要多层的封装，意味着开发效率低下，也意味着对Windows API的完全依赖。如果碰到一个非标准（自定义控件），用这种方式实现的自动化工具基本上完全束手无策，即使有开发团队的支持也很难完成自动化。当然了，算坐标，甚至用全局坐标来做也可以做，但这是野蛮做法，自动化测试的代码非常不稳定，维护和分析结果的成本也很高。
总而言之，言而总之，没有哪个自动化测试工具完全用Windows API来实现……当然了，也不是说Windows API就没有用了，基本上所有的自动化工具都会或多或少，或直接或间接的调到Windows API，也没人敢说他不调一个Windows API就能做自动化测试工具的，最起码鼠标键盘消息的模拟他就没法弄…… - MSAA – Microsoft Active Accessibility
MSAA在1997年在Windows 95中就已经包含的技术，也许你没有听说过，但是在所有的标准控件中，都实现了MSAA的接口，在微软所有的操作系统中都集成了MSAA的组件，在这里，我就不讨论MSAA的历史和相关技术了，就啰嗦一点点感慨：MSAA天生就不是设计给自动化测试的，它存在的意义在于提供一套接口，让开发人员可以方便的给残疾人开发可以使用的软件，比如读屏程序（鼠标移动到按钮的时候，可以发出声音，辅助视力障碍的人操作电脑），从而实现微软将电脑普及到每一个家庭的梦想。
下面进入正题，虽然MSAA不是设计给自动化测试的，但是现有的所有自动化测试工具都是基于或者部分基于MSAA来实现的。MSAA很多时候直接把它叫做IAccessible，它本身是一个COM组件，最主要是实现了IAccessible的接口，它提供了一些方法，通过这些方法，可以获取控件更详细的信息，也可以通过一些方法对控件进行简单的操作（DoDefaultAction）。有兴趣的，可以参考Microsoft Active Accessibility
优点：比起Windows API来说，用户只需要跟IAccessible打交道，通过这个接口能获得的控件信息相对丰富很多，基本操作也不需要通过Windows Message的方式来实现。另外一个比较大的优点就是，自定义控件的支持，当然了，并不是说开发写一个自定义控件，这个控件就可以通过MSAA来识别，而是说当开发人员在实现自定义控件的时候，可以实现IAccessible的接口，并且通过这个接口，把一些的属性和操作暴露出来，测试人员就可以将这个控件当作标准控件，并通过MSAA来自动化。看起来麻烦了点，但是最起码对自动化自定义控件提供了可能。
缺点：天生不足，MSAA从来就不是给自动化测试设计的，所以也不会考虑自动化测试的需求，获取到的控件信息比Windows API多，但是相对自动化测试的需求来说还是远远不够，而且仅仅支持一个基本操作，其它的操作还必须通过Windows Message。另外就是微软推出WPF以后，MSAA的局限性越加明显（这也是因为WPF的控件属性更加丰富，更具定制性，更自由，用MSAA难以描述），这也是微软推出UIAutomation的一个的诱因。 - UIAutomation
伴随着自动化测试的应用越来越广泛，以及WPF的发布，微软在MSAA的基础上，对MSAA进行封装，重新设计并实现了UIAutomation的类库（.Net），微软根据自动化测试的需求，重新实现了一套自动化体系，大家可以看下边的图，这个比较准确。从此自动化测试人员迎来了更广阔的一片蓝天（虽然还飘着点小小的乌云……），随之也有了一些小小的纠结：
a. UIAutomation （后边就简称为UIA） Vs MSAA
在UIA发布的时候，基于MSAA的自动化工具已经发展的非常成熟，比如Silktest和WinRunner… 那么大家在开发一个自己的自动化工具的时候，应该用MSAA呢，还是UIA？ 这篇文章可以给一个两者的大概关系：UI Automation and Microsoft Active Accessibility。 按照微软的想法和设计，UIA是要取代MSAA成为自动化测试的标准类库，并且对WPF来说，UIA才是一等公民。从架构上来讲，UIA在针对标准控件的时候，通过UI Automation Proxy调用了MSAA Server，基本上覆盖了MSAA的功能：
[Issue]
If you are using CheckboxList ASP.NET
using System.Runtime.InteropServices;
public const uint LVM_FIRST = 0×1000;
public const uint LVM_GETITEMCOUNT = LVM_FIRST + 4;
public const uint LVM_GETITEMW = LVM_FIRST + 75;
[DllImport("user32.DLL")]
public static extern int SendMessage(IntPtr hWnd, uint Msg, int wParam, int lParam);
[DllImport("user32.DLL")]
public static extern IntPtr FindWindow(string lpszClass, string lpszWindow);
[DllImport("user32.DLL")]
public static extern IntPtr FindWindowEx(IntPtr hwndParent,
// 首先得到输入框的句柄。通过spy++这类工具分析，聊天窗体的类名为“#32770”// 但当前系统里不只一个类名为“#32770”的窗体，这就需要全体遍历一次。// 类名为“#32770”标题含“聊天”基本能确定。为保险起见还判断一下所在进程是否为“qq.exe”
using
Copy from:
The following values, defined in oleacc.h, describe the roles of objects within an application. Use only the values from the following list; do not add custom roles or roles that are not predfined. Prior to using these object roles, client applications should use Inspect to verify that the object role is used by UI elements.
Clients retrieve an object’s role by calling IAccessible::get_accRole, which returns either a string or one of the following values. Clients call GetRoleText with the role value to retrieve a localized string that describes the object’s role. The localized strings for the role values are in the file oleaccrc.dll.
- ROLE_SYSTEM_ALERT
- The object represents an alert or a condition that a user should be notified about. This role is used only for objects that embody an alert but are not associated with another user interface element such as a message box, graphic, text, or sound.
- ROLE_SYSTEM_ANIMATION
- The object represents an animation control, which contains content that changes over time, such as a control that displays a series of bitmap frames. Animation controls are displayed when files are copied, or when some other time-consuming task is performed.
- ROLE_SYSTEM_APPLICATION
- The object represents a main window for an application.
- ROLE_SYSTEM_BORDER
- The object represents a window border. The entire border is represented by a single object rather than by separate objects for each side.
- ROLE_SYSTEM_BUTTONDROPDOWN
- The object represents a button that drops down a list of items.
- ROLE_SYSTEM_BUTTONDROPDOWNGRID
- The object represents a button that drops down a grid.
- ROLE_SYSTEM_BUTTONMENU
- The object represents a button that drops down a menu.
- ROLE_SYSTEM_CARET
- The object represents the system caret.
- ROLE_SYSTEM_CELL
- The object represents a cell within a table.
- ROLE_SYSTEM_CHARACTER
- The object represents a cartoon-like graphic object, such as Microsoft Office Assistant, which is displayed to provide help to users of an application.
- ROLE_SYSTEM_CHART
- The object represents a graphical image used to represent data.
- ROLE_SYSTEM_CHECKBUTTON
- The object represents a check box control, an option that is turned on or off independently of other options.
- ROLE_SYSTEM_CLIENT
- The object represents a window’s client area. Active Accessibility uses this role as a default if there is a question about the role of a UI element.
- ROLE_SYSTEM_CLOCK
- The object represents a control that displays time.
- ROLE_SYSTEM_COLUMN
- The object represents a column of cells within a table.
- ROLE_SYSTEM_COLUMNHEADER
- The object represents a column header, providing a visual label for a column in a table.
- ROLE_SYSTEM_COMBOBOX
- The object represents a combo box; an edit control with an associated list box that provides a set of predefined choices.
- ROLE_SYSTEM_CURSOR
- The object represents the system mouse pointer.
- ROLE_SYSTEM_DIAGRAM
- The object represents a graphical image used to diagram data.
- ROLE_SYSTEM_DIAL
- The object represents a dial or knob.
- ROLE_SYSTEM_DIALOG
- The object represents a dialog box or message box.
- ROLE_SYSTEM_DOCUMENT
- The object represents a document window. A document window is always contained within an application window. This role applies only to multiple-document interface (MDI) windows and refers to the object that contains the MDI title bar.
- ROLE_SYSTEM_DROPLIST
- The object represents the calendar control, SysDateTimePick32. Oleacc uses this role to indicate that Active Accessibility has found either a date or a calendar control.
- ROLE_SYSTEM_EQUATION
- The object represents a mathematical equation.
- ROLE_SYSTEM_GRAPHIC
- The object represents a picture.
- ROLE_SYSTEM_GRIP
- The object represents a special mouse pointer, which allows a user to manipulate user interface elements such as windows. For example, a user clicks and drags a sizing grip in the lower-right corner of a window to resize it.
- ROLE_SYSTEM_GROUPING
- The object logically groups other objects. There is not always a parent-child relationship between the grouping object and the objects it contains.
- ROLE_SYSTEM_HELPBALLOON
- The object displays a Help topic in the form of a ToolTip or Help balloon.
- ROLE_SYSTEM_HOTKEYFIELD
- The object represents a hot-key field that allows the user to enter a combination or sequence of keystrokes.
- ROLE_SYSTEM_INDICATOR
- The object represents an indicator, such as a pointer graphic, that points to the current item.
- ROLE_SYSTEM_IPADDRESS
- The object represents an edit control designed for an Internet Protocol (IP) address. The edit control is divided into sections for the different parts of the IP address.
- ROLE_SYSTEM_LINK
- The object represents a link to something else. This object might look like text or a graphic, but it acts like a button.
- ROLE_SYSTEM_LIST
- The object represents a list box, allowing the user to select one or more items.
- ROLE_SYSTEM_LISTITEM
- The object represents an item in a list box or the list portion of a combo box, drop-down list box, or drop-down combo box.
- ROLE_SYSTEM_MENUBAR
- The object represents the menu bar (positioned beneath the title bar of a window) from which menus are selected by the user.
- ROLE_SYSTEM_MENUITEM
- The object represents a menu item, which is an entry in a menu that a user can choose to carry out a command, select an option, or display another menu. Functionally, a menu item is equivalent to a push button, radio button, check box, or menu.
- ROLE_SYSTEM_MENUPOPUP
- The object represents a menu, which presents a list of options from which the user can make a selection to perform an action. All menu types must have this role, including drop-down menus that are displayed by selecting from a menu bar, and shortcut menus that are displayed by clicking the right mouse button.
- ROLE_SYSTEM_OUTLINE
- The object represents an outline or tree structure, such as a tree view control, that displays a hierarchical list and allows the user to expand and collapse branches.
- ROLE_SYSTEM_OUTLINEBUTTON
- The object represents items that navigate like an outline item. You can use the up and down arrows to move through the outline. However, instead of expanding and collapsing the menus by using left and right arrow keys, these menus expand or collapse when the space bar or enter key is pressed and the item has focus.
- ROLE_SYSTEM_OUTLINEITEM
- The object represents an item in an outline or tree structure.
- ROLE_SYSTEM_PAGETAB
- The object represents a page tab. The only child of a page tab control is a ROLE_SYSTEM_GROUPING object that contains the contents of the associated page.
- ROLE_SYSTEM_PAGETABLIST
- The object represents a container of page tab controls.
- ROLE_SYSTEM_PANE
- The object represents a pane within a frame or document window. Users can navigate between panes and within the contents of the current pane, but cannot navigate between items in different panes. Thus, panes represent a level of grouping lower than frame windows or documents, but above individual controls. The user navigates between panes by pressing TAB, F6, or CTRL+TAB, depending on the context.
- ROLE_SYSTEM_PROGRESSBAR
- The object represents a progress bar, dynamically showing the user the percent complete of an operation in progress. This control takes no user input.
- ROLE_SYSTEM_PROPERTYPAGE
- The object represents a property sheet.
- ROLE_SYSTEM_PUSHBUTTON
- The object represents a push button control.
- ROLE_SYSTEM_RADIOBUTTON
- The object represents an option button, also called a radio button. It is one of a group of mutually exclusive options. All objects sharing a single parent that have this attribute are assumed to be part of single mutually exclusive group. Use ROLE_SYSTEM_GROUPING objects to divide them into separate groups.
- ROLE_SYSTEM_ROW
- The object represents a row of cells within a table.
- ROLE_SYSTEM_ROWHEADER
- The object represents a row header, which provides a visual label for a table row.
- ROLE_SYSTEM_SCROLLBAR
- The object represents a vertical or horizontal scroll bar, which is part of the client area or used in a control.
- ROLE_SYSTEM_SEPARATOR
- The object is used to visually divide a space into two regions, such as a separator menu item or a bar that divides split panes within a window.
- ROLE_SYSTEM_SLIDER
- The object represents a slider, which allows the user to adjust a setting in given increments between minimum and maximum values.
- ROLE_SYSTEM_SOUND
- The object represents a system sound, which is associated with various system events.
- ROLE_SYSTEM_SPINBUTTON
- The object represents a spin box, which is a control that allows the user to increment or decrement the value displayed in a separate “buddy” control associated with the spin box.
- ROLE_SYSTEM_SPLITBUTTON
- The role represents a button on a toolbar that has a drop-down list icon directly adjacent to the button.
- ROLE_SYSTEM_STATICTEXT
- The object represents read-only text, such as labels for other controls or instructions in a dialog box. Static text cannot be modified or selected.
- ROLE_SYSTEM_STATUSBAR
- The object represents a status bar, which is an area at the bottom of a window that displays information about the current operation, state of the application, or selected object. The status bar has multiple fields, which display different kinds of information.
- ROLE_SYSTEM_TABLE
- The object represents a table that contains rows and columns of cells, and optionally, row headers and column headers.
- ROLE_SYSTEM_TEXT
- The object represents selectable text that allows edits or is designated read-only.
- ROLE_SYSTEM_TITLEBAR
- The object represents a title or caption bar for a window.
- ROLE_SYSTEM_TOOLBAR
- The object represents a toolbar, which is a grouping of controls that provides easy access to frequently used features.
- ROLE_SYSTEM_TOOLTIP
- The object represents a ToolTip that provides helpful hints.
- ROLE_SYSTEM_WHITESPACE
- The object represents blank space between other objects.
- ROLE_SYSTEM_WINDOW
- The object represents the window frame, which contains child objects such as a title bar, client, and other objects contained in a window.
Software Driving Software:
Active Accessibility-Compliant Apps Give Programmers New Tools to Manipulate Software
Dmitri Klementiev
| This article assumes you’re familiar with C++, COM, and Win32 |
| Level of Difficulty
These articles provide an overview of Microsoft technologies that can be used to make a custom control accessible. Techniques range from creating or overriding properties with Dynamic Annotation, to using the new IAccessibleEx interface to add UI Automation support to controls that already support Microsoft Active Accessibility. |