Introduction

We have all have heard of network analyzers or packet sniffers, Wireshark with its command-line counterpart Tshark, or tcpdump.

This article explores how Linux’s command-line power, combined with the tcpdump lexical parser’s filter expressions, can be used for some complex networking debugging. If you are bored at home in these COVID times, how about extracting the video of a video conference capture? (On a serious note, only do this for your video with the participants’ permission.)

What packets to capture

In using a protocol sniffer, the most important thing is identifying what headers and handshake packets mean. The start of a connection has SYN packets (Flag of [S]), while the end will have a reset (Flag of [R]), OR it could be a FIN from the client, as they close that TCP connection. Once identified, inspect the payloads and extract the data using a file transfer or a file upload using HTTP or even voice, video, or streaming traffic.

One cannot study the contents of a particular packet exchange between two hosts on the Internet without isolating the traffic between a particular set of ports. To do this, we must filter based on the IP address or protocol header or port numbers or a combination of this data.

A basic ping ICMP checks traffic and connectivity after the IPsec VPN setup. Usually, if the ping traffic flows then, most others also can. One thing to remember is that the handshake, mostly on the TCP layer, can work only when the packets flow in both directions to each other. Sometimes when you have a proxy redirection in between, the routing could be asymmetric, which means the reverse packets may flow on a different route based on the routing table and network topology.

libpcap

You can diagnose and identify most of the networking problems using the libpcap filters of tcpdump utility. The packet capture library uses a machine-readable format for faster writing and reading since packets travel at very high speeds, and putting a loop in between could make your host lose packets.

You can use Tcpdump to print out packet and payload contents and headers, and you can use the strings(1) utility to look at some of the text data code inside packets in network byte order (little-endian). Using tcpdump appropriately could discourage some people as it takes a lot of existing networking knowledge and in-depth knowledge of the protocol stack to figure out what is going on underneath. (Keep trying, it is worth it!)

Common use scenarios:

Some of the common usages of tcpdump:

root@ec2-user:~<strong># tcpdump -ni enp1s0</strong>
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp1s0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:58:09.368093 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [P.], seq 1109409432:1109409550, ack 2559495171, win 501, options [nop,nop,TS val 1310390170 ecr 3330220954], length 118
16:58:09.368235 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [P.], seq 118:858, ack 1, win 501, options [nop,nop,TS val 1310390170 ecr 3330220954], length 740
16:58:09.387823 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [.], ack 118, win 1050, options [nop,nop,TS val 3330226398 ecr 1310390170], length 0
16:58:09.387933 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [.], ack 858, win 1049, options [nop,nop,TS val 3330226398 ecr 1310390170], length 0
16:58:09.730289 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 1:95, ack 858, win 1050, options [nop,nop,TS val 3330226741 ecr 1310390170], length 94
16:58:09.732076 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 95:1455, ack 858, win 1050, options [nop,nop,TS val 3330226743 ecr 1310390170], length 1360
16:58:09.732215 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [.], ack 1455, win 501, options [nop,nop,TS val 1310390534 ecr 3330226741], length 0
16:58:09.732229 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 1455:2093, ack 858, win 1050, options [nop,nop,TS val 3330226743 ecr 1310390170], length 638
16:58:09.736054 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 2093:2166, ack 858, win 1050, options [nop,nop,TS val 3330226747 ecr 1310390170], length 73
16:58:09.736156 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 2166:2382, ack 858, win 1050, options [nop,nop,TS val 3330226747 ecr 1310390170], length 216
16:58:09.736170 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 2382:2421, ack 858, win 1050, options [nop,nop,TS val 3330226747 ecr 1310390170], length 39
16:58:09.736918 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [.], ack 2421, win 501, options [nop,nop,TS val 1310390538 ecr 3330226743], length 0
16:58:09.738487 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [P.], seq 858:897, ack 2421, win 501, options [nop,nop,TS val 1310390540 ecr 3330226743], length 39
16:58:09.758538 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [.], ack 897, win 1050, options [nop,nop,TS val 3330226769 ecr 1310390540], length 0
16:58:09.766096 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [P.], seq 897:1015, ack 2421, win 501, options [nop,nop,TS val 1310390568 ecr 3330226769], length 118
16:58:09.766165 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [P.], seq 1015:2166, ack 2421, win 501, options [nop,nop,TS val 1310390568 ecr 3330226769], length 1151
16:58:09.785298 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [.], ack 1015, win 1050, options [nop,nop,TS val 3330226796 ecr 1310390568], length 0
16:58:09.786467 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [.], ack 2166, win 1050, options [nop,nop,TS val 3330226797 ecr 1310390568], length 0
16:58:09.906618 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 2421:2514, ack 2166, win 1050, options [nop,nop,TS val 3330226917 ecr 1310390568], length 93
16:58:09.907797 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 2514:2974, ack 2166, win 1050, options [nop,nop,TS val 3330226918 ecr 1310390568], length 460
16:58:09.908384 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 2974:3058, ack 2166, win 1050, options [nop,nop,TS val 3330226919 ecr 1310390568], length 84
16:58:09.908432 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 3058:3231, ack 2166, win 1050, options [nop,nop,TS val 3330226919 ecr 1310390568], length 173
16:58:09.908530 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [P.], seq 3231:3270, ack 2166, win 1050, options [nop,nop,TS val 3330226919 ecr 1310390568], length 39
16:58:09.909605 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [.], ack 3270, win 501, options [nop,nop,TS val 1310390711 ecr 3330226917], length 0
16:58:09.910938 IP 192.168.1.12.34850 &gt; 172.217.163.206.443: Flags [P.], seq 2166:2205, ack 3270, win 501, options [nop,nop,TS val 1310390712 ecr 3330226917], length 39
16:58:09.930959 IP 172.217.163.206.443 &gt; 192.168.1.12.34850: Flags [.], ack 2205, win 1050, options [nop,nop,TS val 3330226942 ecr 1310390712], length 0
^C
26 packets captured
26 packets received by filter
0 packets dropped by kernel

The way tcpdump prints its packets are one packet per line with the source IP, source port, destination IP, and destination port. You can compress duplicate lines.

You can get different results using the -t switch multiple times for timestamp view or the -d switch for more and more debugging info. The -X switch is for printing the contents of packets, which means the output will be much more significant as each line follows with the packet payload contents.*

*if the connection is encrypted you might not be able to see all of the details of the packet.

Networking is all about traffic flowing between ports or IP addresses and not the contents since the lower layer fragmentation and reassembly usually do not affect us significantly.

Also, tcpdump cannot measure network speeds or identify speed bottlenecks. We need different approaches and different tools for such things.

The example of a filter using tcpdump is this:
# tcpdump -ni enp1s0 -host 192.168.1.5

The above is an RFC1918 address, so this must be a local LAN host we need to track. Using the -n switch is very useful since we can save a lot of latency by avoiding DNS host listings as we usually know the IP address pairs in question.

We can refine the above filter by using these filter expressions:
# tcpdump -ni enp1s0 -host 192.168.1.5 icmp

The above lists the ping packets, and the below gives out the web traffic on SSL port, which is almost all of the traffic these days:
# tcpdump -ni enp1s0 -host 192.168.1.5 port 443

You can also combine the tcpdump filters using the OR and AND methods commonly used in expressions to filter from two hosts or two ports in an OR combination or to narrow down traffic in an AND filter to avoid unwanted noise traffic.

The -w and -r switches are useful to write to a file at wire speed to inspect later, and the -r switch does the reading. This process works like a time machine, in which the packets are played back from a previous session. So, it is not live traffic on your network interface, but it works as if it were.

I have often enjoyed this feature since it usually takes a long time to develop software that can reconstruct network packets. Let alone one using a multimedia utility like GStreamer pipeline and tcpdump capture files, aka pcap files are usually played back much later and often many times till we can finish our software development.

Further

In addition to using tcpdump for troubleshooting, it is also a very educative tool for students who want to learn how the Internet works behind the scenes. It shows how low-level packets on the wire can enable various user experiences like a skype phone call or a youtube video streaming session.

The possibilities of using tcpdump and Wireshark GUI are endless, and this article only scratches the surface.

Did you enjoy this content? Follow our linkedin page!