C# – AutomationElement left click and send keys

As I discussed in last post about UI Automation of .Net Framework http://hintdesk.com/c-list-all-opened-tabs-of-firefox/ , I am trying to make a shortcut of multimedia keys to any flash player in web browser, for example YouTube. At the end of last blog I mentioned about AutomationElement.Current.NativeWindowHandle but it doesn’t really help. If I try to get the Handle of ControlType.Custom, I always become an IntPtr.Zero as result. Therefore I can’t use SendMessage or something equivalent to send message to flash player.

It turns to be difficult now. When SendMessage doesn’t work, so far as I know, I can’t send keys to it without bringing it to front or rather bringing it to active window. That’s really not what I want. I would like to control the flash player with multimedia keys silently. The Firefox window should be “behind” as it was when I control it. So until now I don’t have a perfect solution for my case. However I will show you how to make the flash player to active and send keys to it. If you find a better solution (which doesn’t have to make YouTube as active windown), then share me. YouTube and Firefox again are our candidates to test.

As Firefox users, we already know that we can open many tabs in browser’s window. Our YouTube Tab can be hidden by other tabs like image below

Firefox Tab

So if we want to make flash player as active control, we must set this Tab as active tab by getting it as AutomationElement and call Invoke() function to simulate left click on control. To find the Tab in UI tree quickly, we can use the Ctrl-Mouse-Click function of UI Spy. Click on the Mouse Icon on menu and then Ctrl-Click on YouTube Tab, we’ll land on the correct node of UI tree.

UI Spy Ctrl-Mouse-Click

I just edit and expand the code of the blog post before to set YouTube as active tab. Please read that post to understand more about the variables and their meanings.

...
AutomationElement firstCustomControl = GetNextCustomControl(rootElement, condCustomControl);

Condition condToolbarControl = new AndCondition(
	new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.ToolBar),
	new PropertyCondition(AutomationElement.NameProperty, "Browser-Tabs"));

AutomationElement toolbarFF = firstCustomControl.FindFirst(TreeScope.Children, condToolbarControl);

Condition condTabItemControl =new AndCondition(
	new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.TabItem),
	new PropertyCondition(AutomationElement.NameProperty, "YouTube - Broadcast Yourself"));

AutomationElement youTubeTab = toolbarFF.FindFirst(TreeScope.Descendants, condTabItemControl);
(youTubeTab.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern).Invoke();

UI Spy

As you can see, I use an AndCondition to limit the search result and use two types of wide range searching: one is Children (not recursive) and the other is Descendants (recursive). The tab YouTube can be then set as active by treating it as InvokePattern and calling its Invoke() function. To check if a control supports Invoke() function, on the window of UI Spy, right click on it, choose Control Patterns and see in the new window if there’s a Invoke function

UI Spy Control Patterns

UI Spy Invoke Function

After setting YouTube tab as active one, we move along the UI Tree again, localize the Flash Player control and set it active as following code (which bases on UI tree in UI Spy). Because I use a hardly-attached-way therefore if YouTube changes its structur we must rewrite our code. The loosely-attached-way was discussed in the blog before, you can read it again or invent one way yourself.

AutomationElement flashElement = docElement.FindAll(TreeScope.Children, condCustomControl)[1].
	FindAll(TreeScope.Children, condCustomControl)[2]
	.FindFirst(TreeScope.Children, condCustomControl)
	.FindFirst(TreeScope.Children, condCustomControl)
	.FindAll(TreeScope.Children, condCustomControl)[1]
	.FindFirst(TreeScope.Children, condCustomControl)
	.FindAll(TreeScope.Children, condCustomControl)[0];
(flashElement.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern).Invoke();

UI tree

Thanks to screen reader function of YouTube http://www.google.com/support/youtube/bin/answer.py?answer=189278, we can send key to this flash player to order what he should do. This SendKeys.SendWait() function requires an active window/control. Therefore we must “prepare” a lot of things before just to be sure that this function sends key correctly to desired control. In this example, I send key “0” to rewind the current playing video.

(flashElement.GetCurrentPattern(InvokePattern.Pattern) as InvokePattern).Invoke();
Keys key = Keys.D0;
SendKeys.SendWait(key.ToString());

So I hope that after this blog post, you can automatically animate any control you want with UIAutomation. Remember to share me if you know how to send message to “hidden” tab in Firefox. I think there must be a solution that we can interact with the flash player without popping it up to be active. As usual, the complete source code can be download at the end “List all opened tabs of Firefox

6 thoughts on “C# – AutomationElement left click and send keys”

  1. Hi,

    I downloaded the code and tried to run but it doesnt do anything.

    For which version of FF is this. I want to read the content from the firefox.

    Thanks for the help.

    Regards,
    Ayush

  2. @Ayushman: Sorry I didn’t remember the version of firefox but I think if you understand how it works you can implement yourself for newer version of Firefox.

  3. The code is long and I have tried too. I’m still struggling to find a way to send a key to AutomationElement object in .NET.

  4. @Jenna: As I wrote in post, you just need to call SendKeys.SendWait() function to send keys but this function only works when your AutomationElement is active. For not active AutomationElement, I don’t have any solution.

  5. I have tried in my computer but is does nothing. Are you sure it really sends the keys to the window?

Leave a Reply

Your email address will not be published. Required fields are marked *