C# – List all opened tabs of Firefox with UIAutomation

I use at home a keyboard with multimedia keys looks like this one of Logitech Media Keyboard K200 With One-touch Media and Internet Keys (920-002719). It’s very comfortable with this type of keyboard because when I am listening with Windows Media Player, I can start/stop/forward/backward without switching to Windows Media Player. Unfortunately these multimedia keys don’t work with Firefox. Let’s consider this case: I have multi tabs opened in Firefox. One of them is YouTube website and is playing a video. I am on chatting with my family on Yahoo Messenger and would like to make a voice chat. So I would like to pause the video of YouTube and resume it after chatting. However the multimedia key don’t work with Firefox (when Firefox is in background and YouTube is not active tab), I must browse to Firefox and then YouTube site to pause the video. It’s really uncomfortable. Therefore I would like make the multimedia keys work with Firefox so that I don’t have to jump between windows to stop the sound.

Solution: Not yet. I haven’t found a complete solution until now. I have a concept and try to implement it. I think what I need is a program mapping the media keys from keyboard to YouTube shortcut keys. That means anytime when I press a Pause key on keyboard, my program will be notified and his task is to send a Spacebar code to the YouTube tab on Firefox. Why Spacebar? (Read here for shortcut of YouTube

http://www.google.com/support/youtube/bin/answer.py?answer=189278

That’s my concept. It sounds pretty easy but it’s really hard to implement completely. First, I have to globally hook up the keyboard and catch the key pressed event of multimedia key. Then I must check if YouTube tab stands on Firefox and send Spacebar key to this tab. Keyboard globally hooking has many examples outside. You can easily find one therefore I just skip this part. In this small blog I would like to discuss only small part of second part is how to check if YouTube is currently loaded in Firefox.

If you read my blog before maybe you are thinking of using Spy++ and FindWindowEx API to localize YouTube tab on Firefox as these posts

http://hintdesk.com/c-send-data-to-other-application/
http://hintdesk.com/c-send-data-to-other-application-menu-and-menuitem/

But it’s not so easy as before. If you open Spy++ and search for Firefox you only see a node with 2 sub nodes without any useful information about current opened tabs like image below

Spy++ Firefox

The reason is that Spy++ can “see” only windows or rather controls deriving from Windows Control class. The controls of Firefox against are custom control and don’t inherit from this class therefore Spy++ canโ€™t show them in his tree. In this case, we must use another technique to achieve our target which is UI Automation

http://msdn.microsoft.com/en-us/library/ms747327.aspx

This technique is normally used in Testing .NET Test Automation Recipes: A Problem-Solution Approach (Expert’s Voice in .NET)

UI Automation provides programmatic access to most user interface (UI) elements on the desktop, enabling assistive technology products such as screen readers to provide information about the UI to end users and to manipulate the UI by means other than standard input. UI Automation also allows automated test scripts to interact with the UI. To use UI Automation, we need to add a reference UIAutomationClient.dll and UIAutomationTypes.dll. These libraries are not available under .Net/COM tab in reference. We must browse to them which are usually under the path

C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.0\

So it’s short introduction about UI Automation. Then how to use it in our code?

Under

C:\Program Files (x86)\Windows Kits\8.1\bin\x64

you’ll a tool inspect.exe. With the help of this tool, we can view complete UI structure of all opening windows. Each node represents a control. It can be a Window, TitleBar, and Menu… The right panel shows more detailed information of each node, for example his ClassName, Name, ControlPatterns… even the URL of the tab like images below.

Inspect Mainwindow

So what all we need to do know is looping through the tree and list all found URLs.

Process firefox = Process.GetProcessesByName("firefox")[0];
AutomationElement rootElement = AutomationElement.FromHandle(firefox.MainWindowHandle);
Condition condDocAll = new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Document);
foreach (AutomationElement docElement in rootElement.FindAll(TreeScope.Descendants, condDocAll))
{
	 foreach (AutomationPattern pattern in docElement.GetSupportedPatterns())
	{
		if (docElement.GetCurrentPattern(pattern) is ValuePattern)
			Console.WriteLine((docElement.GetCurrentPattern(pattern) as ValuePattern).Current.Value.ToString() + Environment.NewLine);
	}
}

In the code above, I get the process of Firefox and pass his MainWindowHandle to the constructor of AutomationElement to initialize as rootElement of our search. With help of a condition, we can limit what we would like to search for. In this case, I would like to search only for ControlType.Document in his all descendants. If an AutomationElement with ControlType.Document is found, I loop through all of his patterns and write the URL back to screen. As you see, it’s pretty easy to use in comparing to method using FindWindowEx. However the code above is not optimized because he loops all of sub nodes of rootElement to find out the documents which causes large consuming and the results are sometimes false. However the advantage of this code is that he works with any version of Firefox because of loose attaching to the control trees although it’s time consuming.

If time consuming is a problem and it’s ok that we attach hardly our code to a specific version of Firefox I would like to recommend using this code below

Condition condCustomControl = new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Custom);
AutomationElement firstCustomControl = GetNextCustomControl(rootElement, condCustomControl);
AutomationElement secondCustomControl = GetNextCustomControl(firstCustomControl, condCustomControl);
foreach (AutomationElement thirdElement in secondCustomControl.FindAll(TreeScope.Children, condCustomControl))
{
	foreach (AutomationElement fourthElement in thirdElement.FindAll(TreeScope.Children, condCustomControl))
	{
		Condition condDocument = new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Document);
		AutomationElement docElement = fourthElement.FindFirst(TreeScope.Children, condDocument);
		if (docElement != null)
		{
			foreach (AutomationPattern pattern in docElement.GetSupportedPatterns())
			{
				if (docElement.GetCurrentPattern(pattern) is ValuePattern)
					Console.WriteLine((docElement.GetCurrentPattern(pattern) as ValuePattern).Current.Value.ToString() + Environment.NewLine);
			}
		}
	}
}

The block code above works thousand times faster than the first one because he does not loop through all nodes but basing on structure shown on UI Spy, he goes to a specific node and finds the next one. Then again goes to specific sub-node and finds the next one until he reaches the document node and print out the URL.

Url Node

As you can see with UI Automation we can access to each control of other application like traditional way of FindWindowEx. The handle of control can be got through property NativeWindowHandle if we want, then we can send any message to it. The solution for my problem is still not finished. I am trying to make it work. Hopefully I’ll get a working tool to control YouTube player from my keyboard.

Source code: https://bitbucket.org/hintdesk/dotnet-list-all-opened-tabs-of-firefox-with-uiautomation

14 thoughts on “C# – List all opened tabs of Firefox with UIAutomation”

  1. Hi,

    This is a very intersting solution and one of the only ones I could find with a coding solution. One problem I have with it though is that it only gives you the sites in the most recently used Firefox window. If I have three Firebox browser windows in my taskbar, it only picks up the most recently used one. I would have liked it to hook into all the firefox windows, albeit Firefox.exe only runs once.

    Can you think of a solution?

    Also, I ported this to VB.net, and find the following line to execute extremely slowely:

    For Each docElement As AutomationElement In rootElement.FindAll(TreeScope.Descendants, condDocAll)

    …it takes almost 10 seconds. Is there a “lighter” way to do it?

    Thanks buddy!

  2. This page doesn’t outline the GetNextCustomControl function. I found it in the source code that’s downloadeble here. Could you please help in translating it into VB.Net?

    What I’ve got so far is:

    Return rootElement.FindAll(TreeScope.Children, condCustomControl).Cast(Of AutomationElement)().ToList().Where(Function(x As Object) x.Current.BoundingRectangle Rectangle.Empty).FirstOrDefault()

    But it doesn’t like the “x as object” bit, it should be declared as something else.

    Please help, I can’t get your code to run otherwise…!

  3. @Johan: For question with many windows of one “firefox” process, you can use EnumWindows to enumerate all top-level windows and for each window use GetWindowThreadProcessId. Then compare the process id with process id of firefox to enumerate all windows of firefox.
    For question with GetNextCustomControl you can use a simple foreach loop instead of LINQ syntax. Moreover there are a lot of tools in Internet to convert C# to VB.NET, for example http://www.developerfusion.com/tools/convert/csharp-to-vb/ , just use the converter to convert code for you.

  4. Brilliant! I’ll use this solution in my application. Already can aquire url from IE, opera, ff, safari and chrome.

    Big thanks!

  5. @alexbard: I’m glad to help you. I already visit your website it seems to be very interesting but I can’t understant. ๐Ÿ˜‰

  6. Maybe I should clearify: the URL from the active tab is still retrieved but the non-visible tabs don’t supply a url. This can be checked with UI Spy, the problem is not with your code.

  7. @Lakritzator: As I already said in post, it can happen because firefox maybe changes his code and my code won’t work. But I hope that you’ll find your own way to make it work again.

  8. Hi admin,

    Some how it will not works for me. There is some error in line and returned null.
    AutomationElement firstCustomControl = GetNextCustomControl(rootElement, condCustomControl);

    Can you help me how to do this.

    Thanks in advance
    Chandan Dey

  9. @Chandan Dey: This code works only with old version of Firefox. You can apply its concept to work with new Firefox.

  10. Excellent example. But when I make my own modifications… nothing
    works.

    I only need to get 1 thing: The HTML that is currently loading into
    FireFox’s current tab.

    (I do NOT need to re-grab the new HTML from the server. The page will
    be different than what is CURRENTLY in the browser. Like a “news
    page” that changes every few minutes…. “new data” is useless to me.)

  11. @Tammy_Programmer: The concept for getting text of current tab is same as getting URL. Just use inspect.exe to find website windows and read text out.

Leave a Reply

Your email address will not be published. Required fields are marked *