I'm going to assume that you have done everything perfect security wise on Windows NT 4.0 and that your domains are arranged the way you want them. None of your Windows NT 4.0 computers have been ever compromised, all user accounts belong to actual users and they are given the rights and access they need on the system. You only have a few administrative accounts, know who knows their passwords. You have no FAT volumes, (its all NTFS) and you've set good permissions. You've also applied all service packs and hot fixes, tweaked all security related registry entries and so forth.
There does not appear to be a comprehensive report on this issue, but one thing can be sure. If you have not changed security settings you may have a smoother upgrade, but if you have done the right thing and attempted to lock down NT, you may have some problems. According to Microsoft documents, when you upgrade NT 4.0 to Windows Server 2003 the following will happen:
• All service packs and hot fixes and I.E. upgrades are rolled back. (This shouldn't be a real problem, since new versions of OS files and IE will be installed.)
• The registry is refreshed and default values are set. (This could be a problem if you have adjusted security via registry entries and this differs from what Windows Server 2003 will do. A fresh install of Windows Server 2003 will not do this differently, but if you expect current security settings to be the same as in the Windows NT 4.0 computer, they may not be after an upgrade. Either way, you need to review security and set it according to your policy.
• Default permissions are applied. (This could be a problem if you have set permissions on operating systems folders. And, of course that's why they are reset. Windows Server 2003 may need different permissions than you have set for Windows NT 4.0.)
• Registers COM components. (This could be a problem if you have deleted any for security reasons. Guess what. They're back.)
• Renumerates plug and play. (This shouldn't be a problem.)
• Renumbers driver. (This should only be a problem if you have something hard-coded to use a specific drive.)
The upgrade does not change some system file permissions though. For example, if you removed permissions for the authenticated users groups from the Windows system folders, you may prevent many services from starting. (See KB # 827480)
In sum, I don't see any weakening of security, but I do see potential changes. In either case, upgrade or fresh install, you are going to want to review security requirements and your organizations policy to make sure security is what you think it is after the upgrade.
The main advantage to upgrading is you don't have to reinstall other applications (unless they are not compatible with the new OS), or add thousands of users, or set NTFS file permission for applications and files. The main disadvantage is that you keep your own security messes. Just installing Windows Server 2003 will not resolve security issues you may have.
The ultimate question to really ask here is which way will be the most risky? Not just from a security configuration standpoint, but from the big picture. Is there a possibility that an upgrade might result in a server that is non-functional? Yes. Are there potential changes to security settings either way? Yes. Might applications not run, or clients fail to connect? Yes. Is it possible that things will go without any problem? Yes. Which way, upgrade or fresh install is the least risk for you? That, I can't answer, but either way you've some more reading and testing to do before you make that decision.