Guest Blogger: Fred Peña Urbina

Malicious traffic rarely looks “broken” at a glance, but it often leaves subtle inconsistencies in how the TCP/IP stack behaves. These inconsistencies timing irregularities, retransmission bursts, or unexpected window advertisements, become valuable clues during early triage,
especially when detailed payload analysis is not immediately possible.

This post walks through how lightweight malicious activity can disrupt normal TCP behavior on a Linux server. The goal is not to model a full command-and-control channel, but to highlight practical, low-level signals that an analyst can spot quickly using standard tools like tcpdump or Wireshark.

Each example is intentionally minimal, focusing on what changes inside the stack rather than on the malware itself. This makes the anomalies easy to reproduce, easy to visualize, and easy to integrate into an incident responder toolkit.

Why this is important…

Understanding how Linux reacts to malformed TCP traffic is valuable for two reasons. First, it exposes how the kernel protects higher-level processes by validating packet structure before any userland code interacts with it. Second, these reactions create small but reliable forensic traces reset bursts, discarded segments, or kernel log entries that help distinguish legitimate noise from intentional probing. Even a single anomalous flag combination is often enough to reveal an early-stage  reconnaissance attempt or a poorly built malicious tool.

Methodology

This test uses two Linux machines connected to the same network: one acting as the target, and one generating anomalous packets. On the attacker system, hping3 is used to send a minimal sequence of TCP segments with invalid flag combinations (FIN+PUSH+URG) toward an open port on the target. On the target, tcpdump captures the incoming traffic while dmesg is monitored for kernel reactions.

Demonstration Case 1:

How NULL Packets Reveal Service State on Linux

To illustrate how malformed TCP segments distort the normal behavior of the Linux network stack, we begin with one of the most compact but telling probes: the NULL packet a TCP segment with all flags unset.
Although extremely simple, this probe reliably exposes how the kernel reacts when a packet does not match any valid transition in the TCP state machine.

Behavior on the Victim (Linux server)
sudo hping3 -p 80 192.168.133.129 -c 5
Flags: none
Capturing traffic with tcpdump:

If port 80 is open, the reaction is different: the packet is simply ignored.

Open ports never respond to a NULL probe.

This contrast: silence for open ports, RST+ACK for closed ports, is the core of the technique. Behavior on the Attacker Side with port 80 closed:

RA is the same RST+ACK observed by tcpdump, but displayed using hping3 internal notation.

This gives the attacker a clean inference channel:

• If RA → the port is closed.
• If no reply → the port is likely open.

Linux does not randomize this behavior, so the pattern is stable and reproducible even with minimal privileges.

Why this matters

Even though NULL packets are malformed from a protocol perspective, Linux still follows deterministic recovery rules to protect the stack’s internal state. This predictable behavior is what allows an attacker to:

More importantly, for defenders, these sequences produce clean, low-noise artifacts in tcpdump, conntrack, and in some cases kernel logs making them excellent training material for spotting malicious reconnaissance.

RA is the same RST+ACK observed by tcpdump, but displayed using hping3 internal notation.

This gives the attacker a clean inference channel:

• If RA → the port is closed.
• If no reply → the port is likely open.

Linux does not randomize this behavior, so the pattern is stable and reproducible even with minimal privileges.

Why this matters

Even though NULL packets are malformed from a protocol perspective, Linux still follows deterministic recovery rules to protect the stack’s internal state. This predictable behavior is what allows an attacker to:

• fingerprint service availability
• map basic reachability even through restrictive firewalls
• detect local filters (e.g., ufw, nftables) based on how they alter the default kernel response.

More importantly, for defenders, these sequences produce clean, low-noise artifacts in tcpdump, conntrack, and in some cases kernel logs making them excellent training material for spotting malicious reconnaissance.

Case 2: Timestamp (TSval/TSecr Manipulation)

In this second scenario, the attacker does not rely on unusual flag combinations.
Instead, the focus is on a subtler and harder-to-filter technique: manipulating the TCP Timestamp option.

Unlike malformed FIN/URG packets or NULL scans, here we tamper with metadata that the Linux TCP stack uses for RTT estimation, PAWS validation, and segment ordering.

Linux implements this logic in internal routines such as tcp_parse_options() and
tcp_validate_timestamp().

If the timestamp values do not match the expected progression e.g., a TSval far in the future, or a TSecr that the host never emitted the segment is silently rejected.

To trigger this behavior, the attacker sends a legitimate SYN but with fabricated timestamps:

Externally, this looks like a normal SYN.
Internally, the kernel processes the segment through the timestamp validation path.

If the timestamps fall outside the acceptable window, Linux marks the segment as invalid.

Depending on the code path, the kernel may drop the packet silently.

Why this matters

This technique shows that an attacker does not need malformed flags to disrupt the TCP handshake.
Simply manipulating the timing metadata is enough to:

• break or delay the handshake silently
• trigger kernel-level validation routines that normally remain invisible
• expose implementation differences across Linux kernels.

From a reconnaissance and fingerprinting perspective, timestamp anomalies reveal subtle behavioral traits of the remote stack without sending obviously suspicious traffic.

This case complements Case 1 by demonstrating that disruption can occur not only through invalid structures but also through invalid expectations.

Conclusion

The experiments and observations presented in this article highlight the  intrinsic behavior of the Linux TCP/IP stack when confronted with various scenarios involving malformed or intentionally manipulated TCP packets.

These responses, ranging from timestamp adjustments to subtle changes in sequence acknowledgment patterns, reveal a wealth of information about the underlying  network stack’s handling of edge cases and abnormal traffic. Such behaviors are not merely academic curiosities; they constitute a valuable source of signals for those skilled in traffic analysis and reconnaissance. By carefully observing these reactions, it becomes possible to perform sophisticated fingerprinting of systems and middleboxes, discerning the presence of firewalls, load balancers, or other intermediary devices. In essence, the Linux TCP stack’s nuanced responses to unusual packet patterns serve as a kind of “behavioral signature” that can be leveraged for network reconnaissance, security assessments, and advanced research in TCP/IP stack analysis. This underscores the importance of understanding low-level protocol behavior, as even minor, seemingly inconsequential reactions can provide meaningful insights to observers who know how to interpret them.