The difference is very little to nothing. It's not like using a swap file would need to go through the file system layer and all that. Not since a long time. Here's a discussion thread about the perf difference from a couple of decades ago, when the 2.6 kernel was new:
formatting link