Loadclub195

• Unix Regex Cheat Sheet Download
• Regex Cheat Sheet In R

Regular Expressions Cheat Sheet by DaveChild - Cheatography.com Created Date: 4237Z. Linux comes with GNU grep, which supports extended regular expressions. GNU grep is the default on all Linux systems. The grep command is used to locate information stored anywhere on your server or workstation. Let us see fundamental of regex and how to use regular expressions in the Linux and Unix like systems. Regular Expressions in grep.

In this Linux/Unix command line cheat sheet, you will learn: Basic Linux commands File Permission commands Environment Variables command User management commands of linux Networking command Process co. Regex cheatsheet Many programs use regular expression to find & replace text. However, they tend to come with their own different flavor.

In this article I am going to share some bash scripting commands and regular expressions which I find useful in password cracking. Most of the time, we find hashes to crack via shared pastes websites (the most popular of them being Pastebin.) Isolating the hashes by hand can be a time consuming process; for that reason we are going to use regular expressions to make our life easier!

Extract md5 hashes

# egrep -oE '(^|[^a-fA-F0-9])[a-fA-F0-9]{32}([^a-fA-F0-9]|$)' *.txt | egrep -o '[a-fA-F0-9]{32}' > md5-hashes.txt

An alternative could be with sed

# sed -rn 's/.*[^a-fA-F0-9]([a-fA-F0-9]{32})[^a-fA-F0-9].*/1/p' *.txt > md5-hashes

Note: The above regexes can be used for SHA1, SHA256 and other unsalted hashes represented in hex. The only thing you have to do is change the '{32}' to the corresponding length for your desired hash-type.

Extract valid MySQL-Old hashes

# grep -e '[0-7][0-9a-f]{7}[0-7][0-9a-f]{7}' *.txt > mysql-old-hashes.txt

Extract blowfish hashes

# grep -e '$2a$08$(.){75}' *.txt > blowfish-hashes.txt

Extract Joomla hashes

# egrep -o '([0-9a-zA-Z]{32}):(w{16,32})' *.txt > joomla.txt

Extract VBulletin hashes

# egrep -o '([0-9a-zA-Z]{32}):(S{3,32})' *.txt > vbulletin.txt

Extraxt phpBB3-MD5

# egrep -o '$H$S{31}' *.txt > phpBB3-md5.txt

Extract Wordpress-MD5

# egrep -o '$P$S{31}' *.txt > wordpress-md5.txt

Extract Drupal 7

# egrep -o '$S$S{52}' *.txt > drupal-7.txt

Extract old Unix-md5

# egrep -o '$1$w{8}S{22}' *.txt > md5-unix-old.txt

Extract md5-apr1

# egrep -o '$apr1$w{8}S{22}' *.txt > md5-apr1.txt

Extract sha512crypt, SHA512(Unix)

# egrep -o '$6$w{8}S{86}' *.txt > sha512crypt.txt

Extract e-mails from text files

# grep -E -o 'b[a-zA-Z0-9.#?$*_-][email protected][a-zA-Z0-9.#?$*_-]+.[a-zA-Z0-9.-]+b' *.txt > e-mails.txt

Extract HTTP URLs from text files

# grep http | grep -shoP 'http.*?[' >]' *.txt > http-urls.txt

For extracting HTTPS, FTP and other URL format use# grep -E '(((https|ftp|gopher)|mailto)[.:][^ >' ]*|www.[-a-z0-9.]+)[^ .,; >'>):]' *.txt > urls.txt

Note: if grep returns 'Binary file (standard input) matches' use the following approaches # tr '[000-011013-037177-377]' '.' < *.log | grep -E 'Your_Regex' OR # cat -v *.log | egrep -o 'Your_Regex'

Extract Floating point numbers

# grep -E -o '^[-+]?[0-9]*.?[0-9]+([eE][-+]?[0-9]+)?$' *.txt > floats.txt

Extract credit card data

Visa# grep -E -o '4[0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' *.txt > visa.txt

MasterCard# grep -E -o '5[0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' *.txt > mastercard.txt

American Express# grep -E -o 'b3[47][0-9]{13}b' *.txt > american-express.txt

Diners Club# grep -E -o 'b3(?:0[0-5]|[68][0-9])[0-9]{11}b' *.txt > diners.txt

Discover# grep -E -o '6011[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' *.txt > discover.txt

JCB# grep -E -o 'b(?:2131|1800|35d{3})d{11}b' *.txt > jcb.txt

AMEX# grep -E -o '3[47][0-9]{2}[ -]?[0-9]{6}[ -]?[0-9]{5}' *.txt > amex.txt

Extract Social Security Number (SSN)

# grep -E -o '[0-9]{3}[ -]?[0-9]{2}[ -]?[0-9]{4}' *.txt > ssn.txt

Extract Indiana Driver License Number

# grep -E -o '[0-9]{4}[ -]?[0-9]{2}[ -]?[0-9]{4}' *.txt > indiana-dln.txt

Extract US Passport Cards

# grep -E -o 'C0[0-9]{7}' *.txt > us-pass-card.txt

Extract US Passport Number

# grep -E -o '[23][0-9]{8}' *.txt > us-pass-num.txt

Extract US Phone Numberss

# grep -Po 'd{3}[s-_]?d{3}[s-_]?d{4}' *.txt > us-phones.txt

Extract ISBN Numbers

# egrep -a -o 'bISBN(?:-1[03])?:? (?=[0-9X]{10}$|(?=(?:[0-9]+[- ]){3})[- 0-9X]{13}$|97[89][0-9]{10}$|(?=(?:[0-9]+[- ]){4})[- 0-9]{17}$)(?:97[89][- ]?)?[0-9]{1,5}[- ]?[0-9]+[- ]?[0-9]+[- ]?[0-9X]b' *.txt > isbn.txt

Remove the space character with sed

# sed -i 's/ //g' file.txtOR# egrep -v '^[[:space:]]*$' file.txt

Remove the last space character with sed

# sed -i s/.$// file.txt

Sorting Wordlists by Length

# awk '{print length, $0}' rockyou.txt | sort -n | cut -d ' ' -f2- > rockyou_length-list.txt

Convert uppercase to lowercase and the opposite

Remove blank lines with sed

# sed -i '/^$/d' List.txt

Remove defined character with sed

# sed -i 's/'//' file.txt

Delete a string with sed

# echo 'This is a foo test' | sed -e 's/<foo>//g'

Replace characters with tr

# tr '@' '#' < emails.txtOR# sed 's/@/#' file.txt

Print specific columns with awk

# awk -F ',' '{print $3}' infile.csv > outfile.csvOR# cut -d ',' -f 3 infile.csv > outfile.csv

Note: if you want to isolate all columns after column 3 use # cut -d ',' -f 3- infile.csv > outfile.csv

Generate Random Passwords with urandom

Remove Parenthesis with tr

# tr -d '()' < in_file > out_file

Generate wordlists from your file-names

# ls -A | sed 's/regexp/&/g'

Process text files when cat is unable to handle strange characters

# sed 's/([[:alnum:]]*)[[:space:]]*(.)(.*)/12/' *.txt

Generate length based wordlists with awk

# awk 'length 10' file.txt > 10-length.txt

Merge two different txt files

# paste -d' ' file1.txt file2.txt > new-file.txt

Faster sorting

# export alias sort='sort --parallel=<number_of_cpu_cores> -S <amount_of_memory>G ' && export LC_ALL='C' && cat file.txt | sort -u > new-file.txt

Mac to unix

# tr '015' '012' < in_file > out_file

Dos to Unix

# dos2unix file.txt

Unix to Dos

# unix2dos file.txt

Remove from one file what is in another file

# grep -F -v -f file1.txt -w file2.txt > file3.txt

Isolate specific line numbers with sed

# sed -n '1,100p' test.file > file.out

Create Wordlists from PDF files

# pdftotext file.pdf file.txt

Find the line number of a string inside a file

# awk '{ print NR, $0 }' file.txt | grep 'string-to-grep'

https://github.com/ggreer/the_silver_searcher

For faster searching, use all the above grep regular expressions with the command ag. The following is a proof of concept of its speed:

Useful Use of Cat

Contrary to what many veteran unix users may believe, this happens to be one of the rare opportunities where using cat can actually make your searches faster. The SilverSearcher utility is (at the time of this writing) not quite as efficient as cat when it comes to reading from file handles. Therefore, you can pipe output from cat into ag to see nearly a 2x real time performance gain:

The tables below are a reference to basic regex. While reading the rest of the site, when in doubt, you can always come back and look here. (It you want a bookmark, here's a direct link to the regex reference tables). I encourage you to print the tables so you have a cheat sheet on your desk for quick reference.
The tables are not exhaustive, for two reasons. First, every regex flavor is different, and I didn't want to crowd the page with overly exotic syntax. For a full reference to the particular regex flavors you'll be using, it's always best to go straight to the source. In fact, for some regex engines (such as Perl, PCRE, Java and .NET) you may want to check once a year, as their creators often introduce new features.
The other reason the tables are not exhaustive is that I wanted them to serve as a quick introduction to regex. If you are a complete beginner, you should get a firm grasp of basic regex syntax just by reading the examples in the tables. I tried to introduce features in a logical order and to keep out oddities that I've never seen in actual use, such as the 'bell character'. With these tables as a jumping board, you will be able to advance to mastery by exploring the other pages on the site.

How to use the tables

The tables are meant to serve as an accelerated regex course, and they are meant to be read slowly, one line at a time. On each line, in the leftmost column, you will find a new element of regex syntax. The next column, 'Legend', explains what the element means (or encodes) in the regex syntax. The next two columns work hand in hand: the 'Example' column gives a valid regular expression that uses the element, and the 'Sample Match' column presents a text string that could be matched by the regular expression.
You can read the tables online, of course, but if you suffer from even the mildest case of online-ADD (attention deficit disorder), like most of us… Well then, I highly recommend you print them out. You'll be able to study them slowly, and to use them as a cheat sheet later, when you are reading the rest of the site or experimenting with your own regular expressions.
Enjoy!
If you overdose, make sure not to miss the next page, which comes back down to Earth and talks about some really cool stuff: The 1001 ways to use Regex.

Regex Accelerated Course and Cheat Sheet

For easy navigation, here are some jumping points to various sections of the page:
✽ Characters
✽ Quantifiers
✽ More Characters
✽ Logic
✽ More White-Space
✽ More Quantifiers
✽ Character Classes
✽ Anchors and Boundaries
✽ POSIX Classes
✽ Inline Modifiers
✽ Lookarounds
✽ Character Class Operations
✽ Other Syntax
(direct link)

Characters

Character	Legend	Example	Sample Match
d	Most engines: one digit from 0 to 9	file_dd	file_25
d	.NET, Python 3: one Unicode digit in any script	file_dd	file_9੩
w	Most engines: 'word character': ASCII letter, digit or underscore	w-www	A-b_1
w	.Python 3: 'word character': Unicode letter, ideogram, digit, or underscore	w-www	字-ま_۳
w	.NET: 'word character': Unicode letter, ideogram, digit, or connector	w-www	字-ま‿۳
s	Most engines: 'whitespace character': space, tab, newline, carriage return, vertical tab	asbsc	a b c
s	.NET, Python 3, JavaScript: 'whitespace character': any Unicode separator	asbsc	a b c
D	One character that is not a digit as defined by your engine's d	DDD	ABC
W	One character that is not a word character as defined by your engine's w	WWWWW	*-+=)
S	One character that is not a whitespace character as defined by your engine's s	SSSS	Yoyo

(direct link)

Quantifiers

Quantifier	Legend	Example	Sample Match
+	One or more	Version w-w+	Version A-b1_1
{3}	Exactly three times	D{3}	ABC
{2,4}	Two to four times	d{2,4}	156
{3,}	Three or more times	w{3,}	regex_tutorial
*	Zero or more times	A*B*C*	AAACC
?	Once or none	plurals?	plural

(direct link)

More Characters

Character	Legend	Example	Sample Match
.	Any character except line break	a.c	abc
.	Any character except line break	.*	whatever, man.
.	A period (special character: needs to be escaped by a )	a.c	a.c
	Escapes a special character	.*+? $^/	.*+? $^/
	Escapes a special character	[{()}]	[{()}]

(direct link)

Logic

Logic	Legend	Example	Sample Match
|	Alternation / OR operand	22|33	33
( … )	Capturing group	A(nt|pple)	Apple (captures 'pple')
1	Contents of Group 1	r(w)g1x	regex
2	Contents of Group 2	(dd)+(dd)=2+1	12+65=65+12
(?: … )	Non-capturing group	A(?:nt|pple)	Apple

(direct link)

More White-Space

Character	Legend	Example	Sample Match
t	Tab	Ttw{2}	T ab
r	Carriage return character	see below
n	Line feed character	see below
rn	Line separator on Windows	ABrnCD	AB CD
N	Perl, PCRE (C, PHP, R…): one character that is not a line break	N+	ABC
h	Perl, PCRE (C, PHP, R…), Java: one horizontal whitespace character: tab or Unicode space separator
H	One character that is not a horizontal whitespace
v	.NET, JavaScript, Python, Ruby: vertical tab
v	Perl, PCRE (C, PHP, R…), Java: one vertical whitespace character: line feed, carriage return, vertical tab, form feed, paragraph or line separator
V	Perl, PCRE (C, PHP, R…), Java: any character that is not a vertical whitespace
R	Perl, PCRE (C, PHP, R…), Java: one line break (carriage return + line feed pair, and all the characters matched by v)

(direct link)

More Quantifiers

Quantifier	Legend	Example	Sample Match
+	The + (one or more) is 'greedy'	d+	12345
?	Makes quantifiers 'lazy'	d+?	1 in 12345
*	The * (zero or more) is 'greedy'	A*	AAA
?	Makes quantifiers 'lazy'	A*?	empty in AAA
{2,4}	Two to four times, 'greedy'	w{2,4}	abcd
?	Makes quantifiers 'lazy'	w{2,4}?	ab in abcd

(direct link)

Character Classes

Character	Legend	Example	Sample Match
[ … ]	One of the characters in the brackets	[AEIOU]	One uppercase vowel
[ … ]	One of the characters in the brackets	T[ao]p	Tap or Top
-	Range indicator	[a-z]	One lowercase letter
[x-y]	One of the characters in the range from x to y	[A-Z]+	GREAT
[ … ]	One of the characters in the brackets	[AB1-5w-z]	One of either: A,B,1,2,3,4,5,w,x,y,z
[x-y]	One of the characters in the range from x to y	[ -~]+	Characters in the printable section of the ASCII table.
[^x]	One character that is not x	[^a-z]{3}	A1!
[^x-y]	One of the characters not in the range from x to y	[^ -~]+	Characters that are not in the printable section of the ASCII table.
[dD]	One character that is a digit or a non-digit	[dD]+	Any characters, inc- luding new lines, which the regular dot doesn't match
[x41]	Matches the character at hexadecimal position 41 in the ASCII table, i.e. A	[x41-x45]{3}	ABE

(direct link)

Anchors and Boundaries

Anchor	Legend	Example	Sample Match
^	Start of string or start of line depending on multiline mode. (But when [^inside brackets], it means 'not')	^abc .*	abc (line start)
$	End of string or end of line depending on multiline mode. Many engine-dependent subtleties.	.*? the end$	this is the end
A	Beginning of string (all major engines except JS)	Aabc[dD]*	abc (string.. ..start)
z	Very end of the string Not available in Python and JS	the endz	this is..n..the end
Z	End of string or (except Python) before final line break Not available in JS	the endZ	this is..n..the endn
G	Beginning of String or End of Previous Match .NET, Java, PCRE (C, PHP, R…), Perl, Ruby
b	Word boundary Most engines: position where one side only is an ASCII letter, digit or underscore	Bob.*bcatb	Bob ate the cat
b	Word boundary .NET, Java, Python 3, Ruby: position where one side only is a Unicode letter, digit or underscore	Bob.*bкошкаb	Bob ate the кошка
B	Not a word boundary	c.*BcatB.*	copycats

(direct link)

POSIX Classes

Character	Legend	Example	Sample Match
[:alpha:]	PCRE (C, PHP, R…): ASCII letters A-Z and a-z	[8[:alpha:]]+	WellDone88
[:alpha:]	Ruby 2: Unicode letter or ideogram	[[:alpha:]d]+	кошка99
[:alnum:]	PCRE (C, PHP, R…): ASCII digits and letters A-Z and a-z	[[:alnum:]]{10}	ABCDE12345
[:alnum:]	Ruby 2: Unicode digit, letter or ideogram	[[:alnum:]]{10}	кошка90210
[:punct:]	PCRE (C, PHP, R…): ASCII punctuation mark	[[:punct:]]+	?!.,:;
[:punct:]	Ruby: Unicode punctuation mark	[[:punct:]]+	‽,:〽⁆

(direct link)

Inline Modifiers

None of these are supported in JavaScript. In Ruby, beware of (?s) and (?m).

Modifier	Legend	Example	Sample Match
(?i)	Case-insensitive mode (except JavaScript)	(?i)Monday	monDAY
(?s)	DOTALL mode (except JS and Ruby). The dot (.) matches new line characters (rn). Also known as 'single-line mode' because the dot treats the entire input as a single line	(?s)From A.*to Z	From A to Z
(?m)	Multiline mode (except Ruby and JS) ^ and $ match at the beginning and end of every line	(?m)1rn^2$rn^3$	1 2 3
(?m)	In Ruby: the same as (?s) in other engines, i.e. DOTALL mode, i.e. dot matches line breaks	(?m)From A.*to Z	From A to Z
(?x)	Free-Spacing Mode mode (except JavaScript). Also known as comment mode or whitespace mode	(?x) # this is a # comment abc # write on multiple # lines [ ]d # spaces must be # in brackets	abc d
(?n)	.NET, PCRE 10.30+: named capture only	Turns all (parentheses) into non-capture groups. To capture, use named groups.
(?d)	Java: Unix linebreaks only	The dot and the ^ and $ anchors are only affected by n
(?^)	PCRE 10.32+: unset modifiers	Unsets ismnx modifiers

(direct link)

Lookarounds

Lookaround	Legend	Example	Sample Match
(?=…)	Positive lookahead	(?=d{10})d{5}	01234 in 0123456789
(?<=…)	Positive lookbehind	(?<=d)cat	cat in 1cat
(?!…)	Negative lookahead	(?!theatre)thew+	theme
(?<!…)	Negative lookbehind	w{3}(?<!mon)ster	Munster

(direct link)

Character Class Operations

Class Operation	Legend	Example	Sample Match
[…-[…]]	.NET: character class subtraction. One character that is in those on the left, but not in the subtracted class.	[a-z-[aeiou]]	Any lowercase consonant
[…-[…]]	.NET: character class subtraction.	[p{IsArabic}-[D]]	An Arabic character that is not a non-digit, i.e., an Arabic digit
[…&&[…]]	Java, Ruby 2+: character class intersection. One character that is both in those on the left and in the && class.	[S&&[D]]	An non-whitespace character that is a non-digit.
[…&&[…]]	Java, Ruby 2+: character class intersection.	[S&&[D]&&[^a-zA-Z]]	An non-whitespace character that a non-digit and not a letter.
[…&&[^…]]	Java, Ruby 2+: character class subtraction is obtained by intersecting a class with a negated class	[a-z&&[^aeiou]]	An English lowercase letter that is not a vowel.
[…&&[^…]]	Java, Ruby 2+: character class subtraction	[p{InArabic}&&[^p{L}p{N}]]	An Arabic character that is not a letter or a number

(direct link)

Other Syntax

Syntax	Legend	Example	Sample Match
K	Keep Out Perl, PCRE (C, PHP, R…), Python's alternate regex engine, Ruby 2+: drop everything that was matched so far from the overall match to be returned	prefixKd+	12
Q…E	Perl, PCRE (C, PHP, R…), Java: treat anything between the delimiters as a literal string. Useful to escape metacharacters.	Q(C++ ?)E	(C++ ?)

The 1001 ways to use Regex

Unix Regex Cheat Sheet Download

Regex Cheat Sheet In R