Active Stack Fingerprinting

The OS fingerprinting refer to any method used to determine what operating system is running on a remote computer. OS fingerprinting is an essential part of network reconnaissance, because the attacker has a greater possibility of succeeding in this attack if he can formulate his attack strategy based on operating system specific vulnerability.
Remote OS fingerprinting is carried out by noting the way the remote system responds to specifically crafted TCP packets. These can range from examining the default TCP window size in a packet, to measuring the amount of data in ICMP packets, and even gauging TCP initial sequence numbers. Similar to port scanning, there are several method to successfully fingerprint an OS. Querying the services running on a target machine is often the simplest way for OS fingerprinting.
Active stack fingerprint is based on the principle that an operating system's IP stack has its own unique way of responding to specially crafted TCP packets. This is due to the different interpretation that vendor abide with while implementing the TCP/IP stack on the particular OS. In active fingerprinting, a variety of malformed packets are sent to the remote host, and the responses compared to a database.
In Nmap, active stack fingerprint is done through eight tests:

1. A TCP packet with the SYN and ECN -Echo flags enabled is sent to an open TCP port.
2. A TCP packet with no flags enabled is sent to an open TCP port.
3. A TCP packet with the URG, PSH, SYN and FIN flags enabled is sent to an open TCP port.
4. A TCP packet with the ACK flag enabled is sent to an open TCP port.
5. A TCP packet with the SYN flag enabled is sent to a closed TCP port.
6. A TCP packet with the ACK flag enabled is sent to a closed TCP port.
7. A TCP packet with the URG, PSH and FIN flag enabled is sent to a closed TCP port.
8. A UDP packet is sent to a closed UDP port. The objective is to extract an ICMP port unreachable message back from the target machine.

One more test that Nmap performs is names TSeq for TCP sequenceability test. The test tries to determine the sequence generation patterns of the TCP initial sequence number also known as TCP ISN sampling , the IP identification numbers also known as IPID sampling and the TCP timestamp numbers. The test is performed by sending six TCP packets with the SYN flag enabled to an open TCP port. The objective is to find partterns in the initial sequence numbers chosen by TCP implementations when responding to a connection request. THese can be categorized into many groups such as traditional 64K (many old UNIX), Random increment (newer version of Solaris, IRIX, FreeBSD, Digital Unix, Cray etc), True "random" (Linux 2.0.*, OpenVMS, newer AIX etc). Windows boxes use a "time dependent" model where the ISN is incremented by a fixed amount each time period.
Most operating system increment a system-wide IPID value for each packet they send. Others such as OpenBSD, use a random IPID and some system (like Linux) use an IPID of 0 in many cases where the "Don't Fragment" bit is not set. Windows does not put the IPID in network byte order, so it increments by 256 for each packet. Another number that can be sequenced for OS detection purposes is the TCP timestamp option values. Some system do not support the feature; others increment the value at frequencies of 2HZ, 100HZ or 1000HZ and still other return 0.