With the random file access requested, the Windows file-system cache is growing so that it could fit all the accessed pages. If the whole database is being accessed (e.g. by the backup process or just very random I/O pattern) and its size is larger than either the configured FileSystemCacheSize setting or the available RAM size, the swapping occurs. In the worst case it leads to the shrinked working set of the Firebird server itself but sometimes even Windows itself can go out of the available physical memory and crash.
Confirmation by Microsoft: