Tech TLDR;

Running Any Linux Browser in (almost) Headless Mode

by

I just saw an articled on Hacker News announcing that Chrome 59 is going to have cross-platform headless support.

A lot of people are excited by this announcement, mainly because it will allow them to move away from PhantomJS – a buggy headless browser currently used as a de-facto solution to run unit tests and other automated tasks.

I too have had my struggles with PhantomJS. One of my first tasks at my current job was resurrecting our long abandoned test framework. I spent some time trying to get everything working with PhanotmJS, but ran into a lot of issues and was forced to look for an alternative solution.

Reading through comments at Hacker News I didn’t see anyone mentioned the approach that we ended up taking (update, some popped up after I finished writing this article). In addition, some people mentioned that the current headless Chrome is not quite where it needs to be. I decided to share our trick since it has served us well for the past 9 months AND it can work with other browsers.

TLDR; We are using xvfb as a virtual display server to run a real Chrome browser in effectively HEADLESS WAY.

Simple test on my Mac

At work we use Protractor as our test framework with Chrome and this approach will work with anything that runs in any browser supported by Unix/Linux system.

For the purpose of this exercise I wanted to write something simple, so I decided to use Python binding for Selenium. I will also use Firefox to demonstrate that this approach works with other browsers.

First I had to run the test on my Mac. I already had Firefox and Python installed, but I had to install the selenium binding pip install selenium and brew install geckodriver to allow selenium binding to interact with Firefox.

Then I ran my simple test script.

from selenium import webdriver
import time

driver = webdriver.Firefox()     # Connect to Firefox
driver.get("https://github.com") # Navigate to Github
print driver.title               # Print page title
time.sleep(3)                    # Wait 3 seconds
driver.quit()                    # Close the browser
Code language: PHP (php)

As expected, the script opened Firefox, printed out the GitHub’s page title and closed the browser. That is how I got the fancy screenshot at the top of this page 🙂

Simulating the build server

I’ve used vagrant to start an Ubuntu 14.04 in a Virtual machine from within same directory, so I can easily have access to my test script. Among other things, Vagrant synchronized current folder with the Virtual machine under /vagrant path, as we’ll see shortly.

vagrant init ubuntu/trusty64
vagrant up
vagrant ssh

After ssh’ing into the virtual machine, I did the following:

sudo apt-get update                # Update dependencies
sudo apt-get install python-pip    # Install pip (Ubuntu already had python 2)
sudo pip install selenium
Code language: JavaScript (javascript)

Installing geckodriver was a bit trickier:

wget https://github.com/mozilla/geckodriver/releases/download/v0.15.0/geckodriver-v0.15.0-linux64.tar.gz
tar -xvzf geckodriver-v0.15.0-linux64.tar.gz
chmod +x geckodriver
sudo cp geckodriver /usr/local/bin/
Code language: JavaScript (javascript)

At this point I’ve had all dependencies installed like I did on my Mac, except that my test was on a server, without X Window System running. As expected, the test hang for a while and failed with the following error:

$ cd /vagrant
$ python test.py 
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    driver = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 154, in __init__
    keep_alive=True)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 98, in __init__
    self.start_session(desired_capabilities, browser_profile)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 185, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 249, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 193, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
Code language: PHP (php)

Running it Headless

Fixing the error was as easy as:

sudo apt-get install xvfb
xvfb-run --server-args='-screen 0 1280x1024x24' python test.py

# -screen 0 1280x1024x24 means -- configured with a root window 
# of 1024 by 768 pixels and a color depth of 24 bits
# and taken right out of the man file :)
Code language: PHP (php)

Which printed out a nice message:

The world's leading software development platform · GitHub

So there you have it a (somewhat) headless Firefox running on a server.

Conclusion

Like everyone else, I am very excited about the Headless Chrome announcement. At the same time I wanted to share the xvfb-run technique because:

  1. It provides a good alternative to PhantomJS
  2. It works with other browsers, not just Chrome
  3. It allows to run identical browsers used by real users, instead of modified (headless) versions.
I hope this post will save somebody from the horrors of PhantomJS.

A day after headless Chrome announcement, PhantomJS maintainer announced that he will be stepping down from the project. I felt bad for bashing PhantomJS. It is a great project that I’ve use for majority of my development career.

Value this content? 🌟
Your $1 helps sustain our efforts.
Grateful for your support! 🙏