Overview
I write a lot of tutorials for Envato Tuts+. These tutorials have headings that need to follow certain rules of capitalization. Tuts+ provides us authors a web-based tool that takes the text of a heading and returns a properly capitalized heading. When I write my tutorials I’m trying to get into the flow, and switching to the capitalization tool breaks my flow. I used to just wing it and do the capitalization myself.
It turns out that I often made mistakes, which caused extra work to the Tuts+ editors who had to correct them. Being a programmer, I decided to program my way out of the problem. I still write my own headings, but when I’m done, I run a little Python program that parses my article, detects all the headers, and then runs them the through Tuts+ capitalization tool and properly capitalizes all the headings.
Since the capitalization tool is a web-based application and not an API, I had to automate the browser in order to invoke it and extract the capitalized headings. In this tutorial, you’ll learn how to control the browser in Python via Selenium and make it do your bidding.
How to Capitalize Headings
Capitalizing headings is not rocket science, but it’s not trivial either. There are several styles, with some overlap and some variations. This is more or less the consensus:
- Capitalize all words with four or more letters.
- Always capitalize the first and last words.
- Don’t capitalize articles: a, an, the.
- Don’t capitalize short conjunctions: and, or, nor, for, but, so, yet.
Then there are a bunch of exceptions and special situations (e.g. heading flows to the next line). But this is all moot. Even if I decided to write my own capitalization code then Tuts+ already picked their flavor and I need to conform to their chosen style.
The Online Case Converter
The Tuts+ capitalization tool is an internal tool intended to be used by Tuts+ instructors only, so I can’t use it as a demo. However, I found a similar tool called Title Case Converter. It’s a web application with a large text area where you can type your heading (or title), a convert button that you click to submit the form, an output area that appears with a properly capitalized heading, and finally a copy button that copies the converted heading to the clipboard.
A Plan for Automating the Capitalization Tool
OK. I’m going to use the online case converter, but there is no API. This is not a big problem. I can automate the browser and simulate a user typing the heading into the input field, clicking the convert button, waiting for the output to show up, clicking the copy button, and finally pasting the properly capitalized heading from the clipboard.
The first step is to pick a browser automation. I chose Selenium WebDriver, which I’ve used successfully before. The rest of the plan is:
- Launch a browser.
- Navigate to the online case converter URL.
- Find all the necessary elements.
- Populate the input field.
- Submit the form.
- Wait for the output.
- Click the copy button.
- Read the capitalized heading from the clipboard.
The complete source code can be found on GitLab.
Introduction to Selenium
Selenium has been automating browsers since 2004. In 2008 it merged with the WebDriver project, which addresses some of the limitations of the original Selenium (e.g. running in the JavaScript sandbox).
Selenium still offers the original flavor of Selenium called Selenium RC (Remote Control). It also has an IDE for writing automated test suites and a tool called Selenium Grid that scales Selenium RC for large test suites that must run in multiple environments. We will limit ourselves to programmatic access to the browser through the WebDriver API (a.k.a. Selenium 2).
Installing Selenium and a Web Driver
Installing Selenium is as simple as pipenv selenium. If you’re not familiar with Pipenv, check out Revisiting Python Packaging With Pipenv. You also need a specific web driver. There are web drivers for different browsers and back ends. You can find the full list on the Selenium website.
I chose the Chrome web driver for this tutorial. Here’s the latest version.
It’s a single zip file that contains a single executable (there are Windows, macOS, and Linux versions). Once you’ve downloaded it, unzip it and drop in your path.
Congratulations! You’re now ready to use Selenium WebDriver from Python.
Launching the Browser
Selenium makes it very easy to launch a browser. As long as you have the right web driver in your path, you just import the selenium.webdriver module and call the proper method to launch your browser of choice:
from selenium import webdriver driver = webdriver.Chrome()
Navigating to a URL
Once you have a driver object, you can call the get() method to navigate to any web page. Here is how to navigate to the Title Case Converter:
driver.get('https://titlecaseconverter.com')
Finding Elements
We need a way to locate the elements on the page you want to interact with. The simplest way is to inspect the web page in the browser and find the ids of the target elements. In the following screenshot, you can see that the input field has the id “title”:
The convert button has no id, but that’s not a problem as you’ll see soon. Here is the code to locate the form and the text fields by id:
input_field = driver.find_element_by_id('title')
If an element you want to interact with doesn’t have an id, you can find it using various other methods such as name, class name, or CSS selector. See all the options in this Selenium guide.
For example, to locate the convert button, I used its class name:
convertButton =
driver.find_element_by_class_name('convertButton')
Populating Text Fields
To populate the input field, we can use the send_keys() method of our input field element. But make sure to clear it first. Otherwise, you’ll just append to the existing text.
input_field.clear() input_field.send_keys(heading)
That was pretty easy.
Clicking Buttons and Submitting Forms
Now, it’s time to submit the form. You can do it in two ways:
- Find the form element on the page and call its
submit()method. - Find the convert button and click it.
I initially submitted the form directly:
form = driver.find_element_by_css_selector('body > form')
form.submit()
For some reason, it doesn’t work. There is no error, but nothing happens. I didn’t spend too much time investigating because the whole point of Selenium and this tutorial is to simulate a person. So I used the other method and just clicked the button I found earlier:
convertButton.click()
Waiting for Form Submission
The online case converter is somewhat fancy. The output field doesn’t exist initially. After you click the convert button and the form is submitted, the output is displayed along with the copy button. That means that we have to wait until the form submission is complete before the copy button appears and we can click it in order to copy the output to the clipboard.
Selenium has got you covered. It has support for waiting for arbitrary conditions and timing out if they don’t materialize. Here is the code that waits for the copy button. It creates a WebDriverWait object with a five-second timeout. It then creates a condition for the presence of an element with the class name copyButton, and then it calls the wait object’s until() method with the condition.
wait = WebDriverWait(driver, 5) buttonPresent = presence_of_element_located((By.CLASS_NAME, 'copyButton')) copyButton = wait.until(buttonPresent)
The result is that after clicking the convert button, it will wait until the copy button shows up or time out after five seconds. If all is well, it will return the copyButton element.
Reading the Capitalized Heading
You can read the content of text fields with field.get_attribute('value'). But, as I mentioned before, the online case converter is kinda fancy. Its output is a nested structure of spans and divs, and when you hover over each part of the output, it tells you why it is capitalized or not.
I could drill down this maze and parse the actual heading, but there is an easier way. The copy button copies the capitalized heading to the clipboard. We can just click the button and read the heading from the clipboard. I used the clipboard module, which you can install with pipenv install clipboard. The code below clears the clipboard by copying an empty string into it, clicks the copy button, and repeatedly reads the clipboard until it’s not empty.
clipboard.copy('')
copyButton.click()
result = clipboard.paste()
while not result:
time.sleep(0.1)
result = clipboard.paste()
Capitalizing a Whole Markdown Document
OK. We can capitalize a single heading. I put all this code in a single function:
def capitalize_heading(heading):
input_field = driver.find_element_by_id('title')
input_field.clear()
input_field.send_keys(heading)
# form = driver.find_element_by_css_selector('body > form')
# form.submit()
convertButton =
driver.find_element_by_class_name('convertButton')
convertButton.click()
# Wait for copy button to appear
wait = WebDriverWait(driver, 5)
buttonPresent = presence_of_element_located((By.CLASS_NAME,
'copyButton'))
copyButton = wait.until(buttonPresent)
clipboard.copy('')
copyButton.click()
result = clipboard.paste()
while not result:
time.sleep(0.1)
result = clipboard.paste()
return result
Now, armed with this capability, we can parse a whole article and capitalize all the headings. I write my tutorials in Markdown. All my headings start with one or more hashes (#).
I defined a helper function that takes a line and, if it contains a heading, capitalizes it properly using the capitalize_heading() function and returns the result. The main thing is stripping all the leading hashes and spaces and restoring them later. We can’t feed a heading with leading spaces because it confuses the online case converter:
def capitalize_line(line):
tokens = line.split('#')
heading = tokens[-1]
space_count = len(heading) - len(heading.lstrip())
spacing = heading[:space_count]
tokens[-1] = spacing + capitalize_heading(heading.lstrip())
result = '#'.join(tokens)
return result
At this point, we can capitalize a Markdown heading. It’s time to capitalize a whole Markdown document. This code is pretty simple—iterate over all the lines, capitalize every line that starts with a hash, and return the properly capitalized text:
def capitalize_all_headings(markdown): capitalized = []
lines = markdown.split('n')
for line in lines:
if line.startswith('#'):
line = capitalize_line(line)
print(line)
capitalized.append(line)
return 'n'.join(capitalized)
The main function takes an input Markdown document, capitalizes it, and saves the result as “capitalized.md”, which you can check and use. The only thing to be careful about is if your Markdown document contains non-heading lines that start with a hash. This can happen if you head code blocks that contain Python or bash comments.
Fun fact—the tutorial you are reading right now was capitalized using this program.
Conclusion
Automating the browser lets you programmatically take control of web applications that don’t provide an API. This is useful mostly for form filling and other interactive web activities (clicking “Next” on long EULAs?).
Selenium was primarily designed for testing web applications, but it’s great at automating any browser-based interaction. If you think a little bit, you can probably find many web applications that you can automate and make your life easier.
Powered by WPeMatico
