In reading over TenForums this weekend, I came across a new potential source of disk space consumption in Windows. It turns out that NTFS can sometimes allocate lots of space for transactions. It uses a built-in facility called the Kernel Transaction Manager (KTM), along with the Common Log File System to implement transactional NTFS, aka TxF. Potential pathologies can result, and consume 100s of MBs or even GBs of disk space. I learned all this, and more, from Chapter 12 of Windows Internals, Part 2, 6th Edition (pp. 469-477). Fortunately, the “file system utility” command, fsutil cleans up excess old transactions. It can also easily report on what’s up with TxF. Here’s a screen cap of one particularly useful such command:
The “resource info \” parameters for fsutil describe transaction space and data for the boot/system drive.
[Click image to see full-sized view.]
Details: How Fsutil Cleans Up Excess Old Transactions
Just to be clear, the info shown in the preceding screen shot shows what a typical Windows 10 desktop looks like (based on my local sample of 7 physical machines, and 4 VMs). The TenForums message thread I saw voiced a desire to identify and repair the cause of 28GB of “missing” disk space on an SSD (see item#8 for details). It runs out that the poster’s system had been writing huge volumes of transaction logs and data that weren’t getting cleaned up. Running a specific fsutil command cleaned up and reclaimed the excess disk space consumed thereby:
fsutil resource setautoreset true <drive-spec>
There <drive-spec> is the volume ID for the volume you wish to clean up (such as C:\ or D:\). The poster also reported that this took upwards of 30 minutes to complete. Thus, for systems with lots of space consumed by TOPS, be prepared to spend some time to reclaim such space.
A Few Explanatory Notes on FSUTIL Output
TOPS stands for TxF Old Page Stream. It maintains a default data stream in the $Tops file, along with an alternate data stream called $T. Windows Internals says this about the function of the $T stream:
The $T stream contains file data that is partially overwritten by a transactional writer (as opposed to a full overwrite, which would move the file into the $TxT directory). NTFS keeps a structure in memory that keeps track of which parts of a file are being modified under a transaction so that nontransacted readers can still access the noncommitted data by have their reads forwarded to $Tops:$T. When the transaction is committed or aborted, the pages are … moved from the $T stream into the original file …” [pg. 474].
In a subsequent paragraph, the authors go onto observe that the TxF log files reside in a hidden directory named $Extend. To find his problem, the Tenforums poster used a file analysis program called WizTree. Turns out it’s adept at displaying both hidden files and on-disk space consumption. This combination is particularly important when detecting TxF related space consumption issues. In the poster’s case, WizTree showed space consumption of 34GB in the $Extend directory tree. Of course, you can always use fsutil to look for such issues explicitly. Because Fsutil cleans up excess old transactions, I’ve added it to my quarterly disk space inspection and cleanup routine.
More Info on the Fsutil Command
For more information on the fsutil command, see its Windows Server Command-Line Reference entry. There, you’ll find examples of proper syntax and usage. It’s worth consulting before putting this useful and informative tool to work.
[Note: The 7th edition of Volume 1 of Windows Internals is currently available, but the 7th edition of Volume 2 isn’t scheduled for release until late 2017 or early 2018. That’s why I had to consult the previous edition, instead of the current one, for the fsutil information cited here. The newest version of that coverage is apparently still in development!]