No. 2. LINUX Basics for Hackers (Text Manipulation)

7 July 2023 12 minutes Author: Lady Liberty

Text Mastery in Linux: Text Manipulation and Processing Skills

In the Linux world, manipulation of textual data plays a large role in performing various tasks. It allows you to efficiently analyze, process and transform texts using various tools and commands. This is especially useful for working with large amounts of data, such as log files, configuration files, CSV files, and others. One of the powerful tools available in Linux for manipulating text is a command-line shell interpreter such as Bash. With its help, you can perform various operations with text, such as search, replace, sort, filter, and others. Additionally, you can use regular expressions to perform powerful text processing based on specific patterns. Linux also has a large number of utilities that simplify text manipulation.

For example, the AWK program allows you to perform complex text processing operations using fields and recording rules. The Linux Stream system allows you to combine several text processing commands, creating powerful pipelines. Linux Text Manipulation provides comprehensive information on various methods, tools, and techniques for working with text in Linux. You will learn about regular expressions, commands for finding and replacing text, trimming and sorting files, filtering, and other useful techniques. In addition, we provide practical examples and advice on how to use these skills in real-world scenarios. Get the skills you need to effectively manipulate text in Linux. “Linux Text Manipulation” is your key to successfully mastering this important skill. Start learning Linux text manipulation today and increase your productivity when working with text data.

Browse files

As demonstrated in Chapter 1, the simplest text display command is probably cat, but it has its limitations. Use cat to display the Snort configuration file (snort.conf) found in /etc/snort (see Listing 21).

kali >cat /etc/snort/snort.conf

Listing 21: Displaying snort.conf in a terminal window

The screen should now display the entire snort.conf file, which will stream until the file reaches the end, as shown here. This is not the most convenient or practical way to view and work with this file.

# include $SO_RULE_PATH/exploit.rules

In the next two sections, I’ll show you  the head and tail commands, which are two methods of displaying only part of a file’s contents to make it easier to see the key content.

If you just want to view the beginning of the file, you can use the head command. By default, this command displays the first 10 lines of the file. For example, the following command shows you the first 10 lines of snort.conf:

kali >head /etc/snort/snort.conf

If you want to see more or less than the default 10 rows, enter the number you want

A dash (-) switch after the head call and before the file name. For example, if you want to see the first 20 lines of a file, you would enter the command shown at the top of Listing 22.

kali >head -20 /etc/snort/snort.conf

VRT Snort.conf rule packages

Listing 22: Displaying the first 20 lines of snort.conf in a terminal window

You should only see the first 20 lines of snort.conf displayed in your terminal window.

Grabbing that tail

The tail command is similar to head, but is used to view the last lines of a file. Let’s use it in snort.conf:

kali >хвіст /etc/snort/snort.conf
#include $SO_RULE_PATH/smtp.rules
#include $SO_RULE_PATH/specificthreats.rules
#include $SO_RULE_PATH/webactivex.rules
#include $SO_RULE_PATH/webclient.rules
#include $SO_RULE_PATH/webiis.rules
#include $SO_RULE_PATH/webmiscp.rules

Note that this command displays some of the last lines of the include rule files, but not all, since, like head, the default for tail is to display 10 lines. You can display more lines by grabbing the last 20 lines of snort.conf. As with the  head command, you can tell the tail how many lines to display by entering a dash (-) followed by the number of lines between the command and the filename, as shown in Listing 23.

kali >хвіст -20 /etc/snort/snort.conf #include $SO_RULE_PATH/chat.rules
#include $SO_RULE_PATH/chat.rules
#include $SO_RULE_PATH/chat.rules
Снип
#Event порогові або придушувальні команди.

Listing 23: Displaying the last 20 lines of snort.conf in a terminal window Now we can see almost all the include lines of the rules files on one screen.

Line numbering

Sometimes, especially with very long files, we may want to display line numbers in the file. Since snort.conf has more than 600 lines, line numbers will be useful here. This makes it easy to reference changes and return to the same location in the file.

To display a file with line numbers, use the nl (number lines) command. Just enter the command shown in Listing 24.

kali >nl/etc/snort/snort.conf
612
#################################################################
613 #dynamic library rules
614 #include $SO_RULE_PATH/bad­traffic.rules
615 #include $SO_RULE_PATH/chat.rules
­­snip­­
630 #include $SO_RULE_PATH/web­iis.rules
631 #include $SO_RULE_PATH/web­misc.rules

Listing 24: Displaying line numbers in terminal output

Each line is now numbered, making it much easier to reference.

Text filtering using GREP

The grep command is probably the most widely used text manipulation command. This allows you to filter the contents of a file to display. If, for example, you want to see all the lines that contain the word source in your snort.conf file, you can use cat and tell it to display only those lines (see Listing 25).

kali >cat/etc/snort/snort.conf|grepoutput
# 6) Configure output plugins
# Step #6: Configure output plugins
# output unified2: filename merged.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
output unified2: filename merged.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
# output alert_unified2: filename merged.log, limit 128, nostamp
# output log_unified2: filename merged.log, limit 128, nostamp
# output alert_syslog: LOG_AUTH LOG_ALERT
# output log_tcpdump: tcpdump.log

Listing 25: Display strings with instances of the keyword or phrase specified by grep

This command will first look at snort.conf and then use a pipe (|) to send it to grep, which will take the file as input, look for lines with occurrences of the word, and display only those lines. The grep command is a very powerful and important command to use in Linux because it can save you hours of searching for every occurrence of a word or command in a file.

Hacker Challenge: Using grep, nl, tail and head

Let’s say you want to display five lines right before the line that says # Step #6:

Configure output plugins using at least the four commands you just learned. How would you do it? (Hint: There are many more variations of these commands than the ones we’ve discussed. You can learn more commands using Linux’s built-in command. For example, man  tail will show a help file for the tail command.)  There are many ways to solve this problem; here I will show you which lines to change to do it one way and your job is to find another method.

Step 1

kali >nl/etc/snort.conf|grepoutput
34 # 6) Configure output plugins
512 # Step #6: Configure output plugins
518 # output unified2: filename merged.log, limit 128, nostamp, mpls_event_types,
vlan_event_types
521 # output alert_unified2: filename snort.alert, limit 128, nostamp
522 # output log_unified2: filename snort.log, limit 128, nostamp
525 # output alert_syslog: LOG_AUTH LOG_ALERT
528 # output log_tcpdump: tcpdump.log
We can see that the line #Step#6:Configureoutputpluginsis line 512, and we know we want the
five lines preceding line 512 as well as line 512 itself (that is, lines 507 to 512).

Step 2

kali >tail-n+507/etc/snort/snort.conf|head-n6
nested_ip inner, \
whitelist $WHITE_LIST_PATH/white_list.rules, \
blacklist $BLACK_LIST_PATH/black_list.rules
###################################################
# Step #6: Configure output plugins
Here, we use tailto start at line 507 and then output into head, and we return just the top six
lines, giving us the five lines preceding the Step#6line, with that line included.

USING SED FOR SEARCH AND REPLACE

The sed command allows you to search for occurrences of a word or pattern of text and then perform an action on it. The command name is short for stream editor, as it follows the same concept as the stream editor. In its simplest form, sed works like a search and replace function in Windows.

Search for the word mysql in the snort.conf file with grep, for example:

kali >cat /etc/snort/snort.conf | grep  mysql include $RULE_PATH/mysql.rules
#include $RULE_PATH/servermysql.rules

You should see that the grep command found two instances of mysql.

Let’s say you want sed to replace every occurrence of mysql with MySQL (remember, Linux is case sensitive), then save the new file to snort2.conf. You can do this by entering the command shown in Listing 26.

kali >sed s/mysql/MySQL/g /etc/snort/snort.conf > snort2.conf

Listing 26: Using sed to search and replace keywords or phrases

The s command performs a search: first you give the term you’re looking for (mysql) and then the term you want to replace it with (MySQL), separated by a forward slash (/). The g command tells Linux that you want the replacement to be done globally. The result is then saved in a new file named snort2.conf.

Now when you use grep from snort2.conf to search for mysql you will see no instances found, but when you search for MySQL you will see two occurrences.

kali >cat snort2.conf | grep MySQL include $RULE_PATH/MySQL.rules #include $RULE_PATH/serverMySQL.rules

If you wanted to replace only the first occurrence of the term mysql, you would not use the final g command.

kali > sed s/mysql/MySQL/ snort.conf > snort2.conf

You can also use the sed command to find and replace any specific occurrence of a word, not all occurrences or just the first occurrence. For example, if you want to replace only the second occurrence of the word mysql, simply put the occurrence number (in this case 2) at the end of the command:

kali >sed s/mysql/MySQL/2 snort.conf > snort2.conf

This command affects only the second instance of mysql.

VIEW FILES WITH MORE AND LESS QUANTITY

Although cat is a good utility for displaying files and creating small files, it certainly has its limitations when displaying large files. When you use cat  with snort.conf, the file scrolls through each page until it reaches the end, which isn’t very practical if you want to extract any information from it.

For working with larger files, we have two more viewing utilities: more and less.

Display control with more

The Advanced command displays the file one page at a time and allows you to scroll down with the ENTER key. This is a utility used by human pages, so let’s take a look at it first. Open snort.conf with the more command, as shown in Listing 27.

kali >more /etc/snort/snort.conf

  • # Snort build options:

  • # Parameters: enablegre enablempls enabletargetbased enableppm enableperfprofiling enablezlib enableactive Response enablenormalizer enablereload enablereact enableflexresp3

More (2%) Listing 27: Using more to display terminal output one page at a time

Notice that more only displays the first page and then stops and tells us in the lower left corner what portion of the file is displayed (2 percent in this case). To view other lines or pages, press ENTER. To exit more, type q (for quit).

Mapping and filtering with less overhead

The less command is very similar to the more command, but with additional functionality – hence the common Linux aficionado saying, “Less is more.” With less, you can not only scroll through the file at your leisure, but also filter it by term. As in Listing 28, open snort.conf with a smaller number.

kali >less /etc/snort/snort.conf Snip

  • # Snort build options:

  • # Parameters: enablegre enablempls enabletargetbased enableppm enableperfprofiling enablezlib enableactive Response enablenormalizer enablereload enablereact /etc/snort/snort.conf

Listing 28: Using less for both displaying the terminal and outputting a page at a time and filtering the results

Notice in the lower left corner of the screen that the path to the file is less highlighted. If you press the forward slash (/) key, less will let you search for terms in the file. For example, when you first configure Snort, you need to define how and where you want to send the intrusion warning output. To find this section of the configuration file, you can simply search the output, for example:

  • # Snort build options:

  • # Parameters: enablegre enablempls enabletargetbased enableppm enableperfprofiling enablezlib enableactive Response enablenormalizer enablereload enablereact / output

This will immediately take you to the first exit appearance and highlight it. You can then search for the next occurrence of the output by typing n (for next).

# Step # 6: Configuring Output Plugins # For more information, see the Snort Guide, Configuring Snort Output Modules

#unified2

# Рекомендовано для більшості встановлень

# вихід unified2: ім'я файлу об'єднано.log, ліміт 128, nostamp, mpls_event_types,

vlan_event_types вивід unified2: ім'я файлу snort.log, limit 128, nostamp, mpls_event_types, vlan_event_types

# Додаткова конфігурація для певних типів інсталяцій

#  вихід alert_unified2: ім'я файлу snort.alert, ліміт 128, nostamp

# Вихід log_unified2: Назва файлу Snort.log, ліміт 128, nostamp

# syslog

#  alert_syslog на виході: LOG_AUTH LOG_ALERT

:

As you can see, less took you to the next occurrence of the output word and highlighted all the search terms. In this case, it went directly to the source section of Snort. How convenient!

Summary

Linux has many ways to manipulate text, and each method has its own strengths and weaknesses. We’ve touched on a few of the most useful methods in this section, but I encourage you to try each of them and develop your own feelings and preferences. For example, I think grep is indispensable and I use less extensively, but you may feel differently.

We used materials from the book “LINUX BASICS FOR HACKERS” written by William Pollock

Other related articles
Found an error?
If you find an error, take a screenshot and send it to the bot.