A rootkit is a program or a program kit that hides the presence of malware in the system.
A rootkit for Windows systems is a program that penetrates into the system and intercepts the system functions (Windows API). It can effectively hide its presence by intercepting and modifying low-level API functions. Moreover it can hide the presence of particular processes, folders, files and registry keys. Some rootkits install its own drivers and services in the system (they also remain “invisible”).
Below are some anti-rootkit utilities:
TDSSKiller by Kaspersky Lab
TDL4 Removal Tool by BitDefender