
In the Linux world, manipulation of textual data plays a large role in performing various tasks. It allows you to efficiently analyze, process and transform texts using various tools and commands. This is especially useful for working with large amounts of data, such as log files, configuration files, CSV files, and others. One of the powerful tools available in Linux for manipulating text is a command-line shell interpreter such as Bash. With its help, you can perform various operations with text, such as search, replace, sort, filter, and others. Additionally, you can use regular expressions to perform powerful text processing based on specific patterns. Linux also has a large number of utilities that simplify text manipulation.
For example, the AWK program allows you to perform complex text processing operations using fields and recording rules. The Linux Stream system allows you to combine several text processing commands, creating powerful pipelines. Linux Text Manipulation provides comprehensive information on various methods, tools, and techniques for working with text in Linux. You will learn about regular expressions, commands for finding and replacing text, trimming and sorting files, filtering, and other useful techniques. In addition, we provide practical examples and advice on how to use these skills in real-world scenarios. Get the skills you need to effectively manipulate text in Linux. “Linux Text Manipulation” is your key to successfully mastering this important skill. Start learning Linux text manipulation today and increase your productivity when working with text data.
As demonstrated in Chapter 1, the simplest text display command is probably cat, but it has its limitations. Use cat to display the Snort configuration file (snort.conf) found in /etc/snort (see Listing 21).
kali >cat /etc/snort/snort.conf
Listing 21: Displaying snort.conf in a terminal window
The screen should now display the entire snort.conf file, which will stream until the file reaches the end, as shown here. This is not the most convenient or practical way to view and work with this file.
# include $SO_RULE_PATH/exploit.rules
In the next two sections, I’ll show you the head and tail commands, which are two methods of displaying only part of a file’s contents to make it easier to see the key content.
If you just want to view the beginning of the file, you can use the head command. By default, this command displays the first 10 lines of the file. For example, the following command shows you the first 10 lines of snort.conf:
kali >head /etc/snort/snort.conf
If you want to see more or less than the default 10 rows, enter the number you want
A dash (-) switch after the head call and before the file name. For example, if you want to see the first 20 lines of a file, you would enter the command shown at the top of Listing 22.
kali >head -20 /etc/snort/snort.conf
VRT Snort.conf rule packages
Listing 22: Displaying the first 20 lines of snort.conf in a terminal window
You should only see the first 20 lines of snort.conf displayed in your terminal window.
Grabbing that tail
The tail command is similar to head, but is used to view the last lines of a file. Let’s use it in snort.conf:
kali >хвіст /etc/snort/snort.conf #include $SO_RULE_PATH/smtp.rules #include $SO_RULE_PATH/specificthreats.rules #include $SO_RULE_PATH/webactivex.rules #include $SO_RULE_PATH/webclient.rules #include $SO_RULE_PATH/webiis.rules #include $SO_RULE_PATH/webmiscp.rules
Note that this command displays some of the last lines of the include rule files, but not all, since, like head, the default for tail is to display 10 lines. You can display more lines by grabbing the last 20 lines of snort.conf. As with the head command, you can tell the tail how many lines to display by entering a dash (-) followed by the number of lines between the command and the filename, as shown in Listing 23.
kali >хвіст -20 /etc/snort/snort.conf #include $SO_RULE_PATH/chat.rules #include $SO_RULE_PATH/chat.rules #include $SO_RULE_PATH/chat.rules Снип #Event порогові або придушувальні команди.
Listing 23: Displaying the last 20 lines of snort.conf in a terminal window Now we can see almost all the include lines of the rules files on one screen.
Sometimes, especially with very long files, we may want to display line numbers in the file. Since snort.conf has more than 600 lines, line numbers will be useful here. This makes it easy to reference changes and return to the same location in the file.
To display a file with line numbers, use the nl (number lines) command. Just enter the command shown in Listing 24.
kali >nl/etc/snort/snort.conf 612 ################################################################# 613 #dynamic library rules 614 #include $SO_RULE_PATH/badtraffic.rules 615 #include $SO_RULE_PATH/chat.rules snip 630 #include $SO_RULE_PATH/webiis.rules 631 #include $SO_RULE_PATH/webmisc.rules
Listing 24: Displaying line numbers in terminal output
Each line is now numbered, making it much easier to reference.
The grep command is probably the most widely used text manipulation command. This allows you to filter the contents of a file to display. If, for example, you want to see all the lines that contain the word source in your snort.conf file, you can use cat and tell it to display only those lines (see Listing 25).
kali >cat/etc/snort/snort.conf|grepoutput # 6) Configure output plugins # Step #6: Configure output plugins # output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types # output alert_unified2: filename merged.log, limit 128, nostamp # output log_unified2: filename merged.log, limit 128, nostamp # output alert_syslog: LOG_AUTH LOG_ALERT # output log_tcpdump: tcpdump.log
Listing 25: Display strings with instances of the keyword or phrase specified by grep
This command will first look at snort.conf and then use a pipe (|) to send it to grep, which will take the file as input, look for lines with occurrences of the word, and display only those lines. The grep command is a very powerful and important command to use in Linux because it can save you hours of searching for every occurrence of a word or command in a file.
Let’s say you want to display five lines right before the line that says # Step #6:
Configure output plugins using at least the four commands you just learned. How would you do it? (Hint: There are many more variations of these commands than the ones we’ve discussed. You can learn more commands using Linux’s built-in command. For example, man tail will show a help file for the tail command.) There are many ways to solve this problem; here I will show you which lines to change to do it one way and your job is to find another method.
kali >nl/etc/snort.conf|grepoutput 34 # 6) Configure output plugins 512 # Step #6: Configure output plugins 518 # output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types 521 # output alert_unified2: filename snort.alert, limit 128, nostamp 522 # output log_unified2: filename snort.log, limit 128, nostamp 525 # output alert_syslog: LOG_AUTH LOG_ALERT 528 # output log_tcpdump: tcpdump.log We can see that the line #Step#6:Configureoutputpluginsis line 512, and we know we want the five lines preceding line 512 as well as line 512 itself (that is, lines 507 to 512).
kali >tail-n+507/etc/snort/snort.conf|head-n6 nested_ip inner, \ whitelist $WHITE_LIST_PATH/white_list.rules, \ blacklist $BLACK_LIST_PATH/black_list.rules ################################################### # Step #6: Configure output plugins Here, we use tailto start at line 507 and then output into head, and we return just the top six lines, giving us the five lines preceding the Step#6line, with that line included.
USING SED FOR SEARCH AND REPLACE
The sed command allows you to search for occurrences of a word or pattern of text and then perform an action on it. The command name is short for stream editor, as it follows the same concept as the stream editor. In its simplest form, sed works like a search and replace function in Windows.
Search for the word mysql in the snort.conf file with grep, for example:
kali >cat /etc/snort/snort.conf | grep mysql include $RULE_PATH/mysql.rules #include $RULE_PATH/servermysql.rules
You should see that the grep command found two instances of mysql.
Let’s say you want sed to replace every occurrence of mysql with MySQL (remember, Linux is case sensitive), then save the new file to snort2.conf. You can do this by entering the command shown in Listing 26.
kali >sed s/mysql/MySQL/g /etc/snort/snort.conf > snort2.conf
Listing 26: Using sed to search and replace keywords or phrases
The s command performs a search: first you give the term you’re looking for (mysql) and then the term you want to replace it with (MySQL), separated by a forward slash (/). The g command tells Linux that you want the replacement to be done globally. The result is then saved in a new file named snort2.conf.
Now when you use grep from snort2.conf to search for mysql you will see no instances found, but when you search for MySQL you will see two occurrences.
kali >cat snort2.conf | grep MySQL include $RULE_PATH/MySQL.rules #include $RULE_PATH/serverMySQL.rules
If you wanted to replace only the first occurrence of the term mysql, you would not use the final g command.
kali > sed s/mysql/MySQL/ snort.conf > snort2.conf
You can also use the sed command to find and replace any specific occurrence of a word, not all occurrences or just the first occurrence. For example, if you want to replace only the second occurrence of the word mysql, simply put the occurrence number (in this case 2) at the end of the command:
kali >sed s/mysql/MySQL/2 snort.conf > snort2.conf
This command affects only the second instance of mysql.
VIEW FILES WITH MORE AND LESS QUANTITY
Although cat is a good utility for displaying files and creating small files, it certainly has its limitations when displaying large files. When you use cat with snort.conf, the file scrolls through each page until it reaches the end, which isn’t very practical if you want to extract any information from it.
For working with larger files, we have two more viewing utilities: more and less.
The Advanced command displays the file one page at a time and allows you to scroll down with the ENTER key. This is a utility used by human pages, so let’s take a look at it first. Open snort.conf with the more command, as shown in Listing 27.
kali >more /etc/snort/snort.conf
# Snort build options:
# Parameters: enablegre enablempls enabletargetbased enableppm enableperfprofiling enablezlib enableactive Response enablenormalizer enablereload enablereact enableflexresp3
More (2%) Listing 27: Using more to display terminal output one page at a time
Notice that more only displays the first page and then stops and tells us in the lower left corner what portion of the file is displayed (2 percent in this case). To view other lines or pages, press ENTER. To exit more, type q (for quit).
Mapping and filtering with less overhead
The less command is very similar to the more command, but with additional functionality – hence the common Linux aficionado saying, “Less is more.” With less, you can not only scroll through the file at your leisure, but also filter it by term. As in Listing 28, open snort.conf with a smaller number.
kali >less /etc/snort/snort.conf Snip
# Snort build options:
# Parameters: enablegre enablempls enabletargetbased enableppm enableperfprofiling enablezlib enableactive Response enablenormalizer enablereload enablereact /etc/snort/snort.conf
Listing 28: Using less for both displaying the terminal and outputting a page at a time and filtering the results
Notice in the lower left corner of the screen that the path to the file is less highlighted. If you press the forward slash (/) key, less will let you search for terms in the file. For example, when you first configure Snort, you need to define how and where you want to send the intrusion warning output. To find this section of the configuration file, you can simply search the output, for example:
# Snort build options:
# Parameters: enablegre enablempls enabletargetbased enableppm enableperfprofiling enablezlib enableactive Response enablenormalizer enablereload enablereact / output
This will immediately take you to the first exit appearance and highlight it. You can then search for the next occurrence of the output by typing n (for next).
# Step # 6: Configuring Output Plugins # For more information, see the Snort Guide, Configuring Snort Output Modules
#unified2 # Рекомендовано для більшості встановлень # вихід unified2: ім'я файлу об'єднано.log, ліміт 128, nostamp, mpls_event_types, vlan_event_types вивід unified2: ім'я файлу snort.log, limit 128, nostamp, mpls_event_types, vlan_event_types # Додаткова конфігурація для певних типів інсталяцій # вихід alert_unified2: ім'я файлу snort.alert, ліміт 128, nostamp # Вихід log_unified2: Назва файлу Snort.log, ліміт 128, nostamp # syslog # alert_syslog на виході: LOG_AUTH LOG_ALERT :
As you can see, less took you to the next occurrence of the output word and highlighted all the search terms. In this case, it went directly to the source section of Snort. How convenient!
Linux has many ways to manipulate text, and each method has its own strengths and weaknesses. We’ve touched on a few of the most useful methods in this section, but I encourage you to try each of them and develop your own feelings and preferences. For example, I think grep is indispensable and I use less extensively, but you may feel differently.
We used materials from the book “LINUX BASICS FOR HACKERS” written by William Pollock