18. Regular Expressions, aka regex

This chapter covers how to regular expressions to perform processing of text.

Regex, aka regular expressions, provide a way to both search and change text. Their advantages are that they are concise, they run very quickly, they can be ported across languages (they are definitely not just a Python thing!), and they are very powerful. The disadvantage is that they are confusing and take some getting used to!

You can live code regex in a couple of places, the first is within Visual Studio Code itself. Do this by clicking the magnifying glass in the left-hand side panel of options. When the search strip appears, you can put a search term in. To the right of the text entry box, there are three buttons, one of which is a period (full stop) followed by an asterisk. This option allows the Visual Studio text search function to accept regular expressions. This will apply regex to all of the text in your current Visual Studio workspace.

Another approach is to head over to https://regex101.com/ or https://regexr.com/ and begin typing your regular expression there (regexr’s cheat sheets and reference patterns are well worth checking out too). You will need to add some text in the box for the regex to be applied to.

Try either of the above with the regex string \w+\s. This matches any occurrence of the word ‘string’ that is followed by another word and then a whitespace. As an example, ‘string cleaning ‘ would be picked up as a match when using this regex.

Within Python, the re library provides support for regular expressions. Let’s try it:

import re

text = "It is true that string cleaning is a topic in this chapter. string editing is another."
re.findall("string \w+\s", text)

['string cleaning ', 'string editing ']

re.findall() returns all matches. There are several useful search-like functions in re to be aware of that have a similar syntax of re.function(regex, text). The table shows what they all do

Function	What it does	Example of use	Output for given value of text
re.match()	Declares whether there is a match at the beginning of a string.	re.match("string \w+\s" , text) is True	None
re.search()	Declares whether there is a match anywhere in the string.	re.search("string \w+\s" , text) is True	True
re.findall()	Returns all matches.	re.findall("string \w+\s" , text)	['string cleaning ', 'string editing ']
re.split()	Splits text wherever a match occurs.	re.split("string \w+\s" , text)	['It is true that ', 'is a topic in this chapter. ', 'is another.']

Another really handy regex function is re.sub(), which substitutes one bit of text for another if it finds a match. Here’s an example:

new_text = "new text here! "
re.sub("string \w+\s", new_text, text)

'It is true that new text here! is a topic in this chapter. new text here! is another.'

18.1. Special Characters

So far, we’ve only seen a very simple application of regex involving a vanilla word, string, the code for another word \w+ and the code for a whitespace \s. Let’s take a more comprehensive look at the regex special characters:

Character	Description	Example Text	Example Regex	Example Match Text
\d	One Unicode digit in any script	“file_93 is open”	file_\d\d	“file_93”
\w	“word character”: Unicode letter, digit, or underscore	“blah hello-word blah”	\w-\w	“hello-world”
\s	“whitespace character”: any Unicode separator	“these are some words with spaces”	words\swith\sspaces	“words with spaces”
\D	Non-digit character (opposite of \d)	“ABC 10323982328”	\D\D\D	“ABC”
\W	Non-word character (opposite of \w)	“Once upon a time *”	\W	“*”
\S	Non-whitespace character (opposite of \s)	“y “	\S	“y”
\Z	End of string	“End of a string”	\w+\Z	“string””
.	Match any character except the newline	“ab=def”	ab.def	“ab=def”

Note that whitespace characters include newlines, \n, and tabs, \t.

18.2. Quantifiers

As well as these special characters, there are quantifiers which ask for more than one occurence of a character. For example, in the above, \w\w asked for two word characters, while \d\d asks for two digits. The next table shows all of the quantifiers.

Quantifier	Role	Example Text	Example Regex	Example Match
{m}	Exactly m repetitions	“936 and 42 are the codes”	\d{3}	“936”
{m,n}	From m (default 0) to n (default infinity)	“Words up to four letters”	\b\w{1,4}\b	“up”, “to”, “four”
*	0 or more. Same as {,}	“42 is the code”	\d*\s	“42”
+	1 or more. Same as {1,}	“4 323 hello”	\d+	“4”, “323”
?	Optional, so 0 or 1. Same as {,1}.	“4 323 hello”	\d?\s	“4”

Exercise

Find a single regex that will pick out only the percentage numbers from both “Inflation in year 3 was 2 percent” and “Interest rates were as high as 12 percent”.

18.3. Metacharacters

Now, as well as special characters and quantifiers, we can have meta-character matches. These are not characters per se, but starts, ends, and other bits of words. For example, \b matches strings at a word (\w+) boundary, so if we took the text “Three letter words only are captured” and ran \b\w\w\w\b we would return “are”. \B matches strings not at word (\w+) boundaries so the text “Bricks” with \B\w\w\B applied would yield “ri”. The next table contains some useful metacharacters.

Metacharacter Sequence	Meaning	Example Regex	Example Match
^	Start of string or line	^abc	“abc” (appearing at start of string or line)
$	End of string, or end of line	xyz$	“xyz” (appearing at end of string or line)
\b	Match string at word (\w+) boundary	ing\b	“matching” (matches ing if it is at the end of a word)
\B	Match string not at word (\w+) boundary	\Bing\B	“stinger” (matches ing if it is not at the beginning or end of the word)

Because so many characters have special meaning in regex, if you want to look for, say, a dollar sign or a dot, you need to escape the character first with a backward slash. So \${1}\d+ would look for a single dollar sign followed by some digits and would pick up the ‘$50’ in ‘she made $50 dollars’.

Exercise

Find the regex that will pick out only the first instance of the word ‘money’ and any word subsequent to ‘money’ from the following: “money supply has grown considerably. money demand has not kept up.”.

18.4. Ranges

You probably think you’re done with regex, but not so fast! There are more metacharacters to come. This time, they will represent ranges of characters.

Metacharacter Sequence	Description	Example Expression	Example Match
[characters]	The characters inside the brackets are part of a matching-character set	[abcd]	a, b, c, d, abcd
[^…]	Characters inside brackets are a non-matching set; a character not inside is a matching character.	[^abcd]	Any occurrence of any character EXCEPT a, b, c, d.
[character-character]	Any character in the range between two characters (inclusive) is part of the set	[a-z]	Any lowercase letter
[^character]	Any character that is not the listed character	[^A]	Any character EXCEPT capital A

Ranges have two more neat tricks. The first is that they can be concatenated. For example, [a-c-1-5] would match any of a, b, c, 1, 2, 3, 4, 5. They can also be modified with a quantifier, so [a-c0-2]{2} would match “a0” and “ab”.

18.5. Greedy versus lazy regexes

Buckle up, because this one is a bit tricky to grasp. Adding a ? after a regex will make it go from being ‘greedy’ to being ‘lazy’. Greedy means that you will match the longest possible string that hits the condition. Lazy will mean that you get the shortest possible string matching the condition. It’s easiest to demonstrate with an example:

test_string = "stackoverflow"
greedy_regex = "s.*o"
lazy_regex = "s.*?o"

print(f"The greedy match is {re.findall(greedy_regex, test_string)[0]}")
print(f"The lazy match is {re.findall(lazy_regex, test_string)[0]}")

The greedy match is stackoverflo
The lazy match is stacko

In the former (greedy) case, we get from an ‘s’ all the way to the last ‘o’ within the same word. In the latter (lazy) case we just get everything between the start and first occurrence of an ‘o’.

18.6. Matches versus capture groups

There is often a difference between what you might want to match and what you actually want to grab with your regex. Let’s say, for example, we’re parsing some text and we want any numbers that follow the format ‘$xx.xx’, where the ‘x’ are numbers but we don’t want the dollar sign. To do this, we can create a capture group using brackets. Here’s an example:

text = "Product 1 was $45.34, while product 2 came in at $50.00 however it was assessed that the $4.66 difference did not make up for the higher quality of product 2."
re.findall("\$(\d{2}.\d{2})", text)

['45.34', '50.00']

Let’s pick apart the regex here. First, we asked for a literal dollar sign using \$. Next, we opened up a capture group with (. Then we said only give us the numbers that are 2 digits, a period, and another 2 digits (thus excluding $4.66). Finally, we closed the capture group with ).

So while we specify a match using regex, while only want running the regex to return the capture group.

Let’s see a more complicated example.

sal_r_per = r"\b([0-9]{1,6}(?:\.)?(?:[0-9]{1,2})?(?:\s?-\s?|\s?to\s?)[0-9]{1,6}(?:\.)?(?:[0-9]{1,2})?)(?:\s?per)\b"
text = "This job pays gbp 30500.00 to 35000 per year. Apply at number 100 per the below address."
re.findall(sal_r_per, text)

['30500.00 to 35000']

In this case, the regex first looks for up to 6 digits, then optionally a period, then optionally another couple of digits, then either a dash or ‘to’ using the ‘|’ operator (which means or), followed by a similar number, followed by ‘per’.

But the capture group is only the subset of the match that is the number range-we discard most of the rest. Note also that other numbers, even if they are followed by ‘per’, are not picked up. (?:) begins a non-capture group, which matches only but does not capture, so that although (?:\s?per) looks for ” per” after a salary (with the space optional due to the second ?), it does not get returned.

Exercise

Find a regex that captures the wage range from “Salary Pay in range \(9.00 - \)12.02 but you must start at 8.00 - 8.30 every morning.”.

This has been a whirlwind tour of regexes. Although regex looks a lot like gobbledygook, it is a really useful tool to be able to deploy for more complex string cleaning and extraction tasks.