The amount of swap used is not a good indicative, you need to check if there is a big exchange of data per second/minute. This is the only indicative of an out of memory system.

Sometimes, some regions of data memory “age” in ram without any access for a long periods of time, the the kernel here has two options, it could destroy the region knowing it could recreate it when needed (with some cpu overhead) or moved this to a swap file when the ram structure already in the swap file and release than section.

Which regions are good candidates for this? Buffers, specially in the fs, code region used for processes or even data sections of a long sleeping process…

Checking your data, if those 5gb are created over a long period of time I would not care a lot about it. Remeber how big the swap is, isn’t that important vs real traffic (in or out) to it

IMO, the easiest way to make your system not swap into some disk you don’t want it to is to not enable any swap partition on that disk.

If it’s using that much RAM for cache, it will survive not having a swap with a minimum impact on performance.

Huh…

  System load:  1.26                Temperature:           57.0 C
  Usage of /:   20.4% of 454.86GB   Processes:             527
  Memory usage: 44%                 Users logged in:       1
  Swap usage:   0%                  IPv4 address for eno1: 192.168.1.190

Ubuntu 22.04.5 - 32 GB RAM - 53 total containers

cat /proc/sys/vm/swappiness: 60 (pretty much default)

cat /proc/pressure/cpu: some avg10=0.68 avg60=0.62 avg300=0.67 total=960097774
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
cat /proc/pressure/memory: some avg10=0.00 avg60=0.00 avg300=0.00 total=0
full avg10=0.00 avg60=0.00 avg300=0.00 total=0
cat /proc/pressure/io: some avg10=1.16 avg60=4.88 avg300=3.85 total=2831425045
full avg10=1.11 avg60=4.54 avg300=3.58 total=2653626377

also: I did not realize until now, and also it might just he a coincidence, but it looks like as if your max swap usage with plenty of cached memory would be 10% of your RAM. I wonder if it would top out at half of that with a swappiness setting of 5. if you try it, load back all your swapped memory first.

I disagree with most here.

You need to reduce swappiness even more, and tweak some other variables (like the memory “threshold” Linux starts to swap, and disk cache aggressiveness) to stop Linux from swapping so opportunistically under such a light load.

IMO, Linux is configured for “old” systems by default: slow HDDs, and constrained RAM pools, where disk IO really, really needs caching, and where idle background processes take a large fraction of RAM.

You have superfluous RAM for your workload. And very fast disk IO that isn’t such a hindrance to apps anyway. And a disk you don’t want to wear. This is the opposite scenario: you don’t want Linux to swap unless it absolutely has to.

For reference, this is part of my config. It’s rather niche and you probably shouldn’t use it, but you should consider looking up the variables:

# Keep min reserve reasonable for 8GB usable space
vm.min_free_kbytes = 262144        # Lower absolute minimum to 256MB

vm.watermark_scale_factor = 10     # Lower to 0.1% (which is ~128MB on 128GB)

# Disable watermark boosting completely
vm.watermark_boost_factor = 0

# Normal-ish metadata pressure so desktop doesn't stutter on disk reads
vm.vfs_cache_pressure = 120

# Allow reasonable swapping of inactive anonymous desktop pages, could be lower
vm.swappiness = 10

# Dirty bytes limits to limit caching
vm.dirty_background_bytes = 67108864
vm.dirty_bytes = 268435456

# Disable compaction & proactive scans to stop freezing with large portions of RAM mlocked
vm.compaction_proactiveness = 0
vm.compact_unevictable_allowed = 0

vm.page-cluster = 0 # 4kb pages for SSD

My system still uses RAM as disk cache with this config, it just won’t go out of its way to swap just to keep that cache, especially I lower swappiness to 1-3.

I also have a 1GB zram pool, prioritized over ssd swap. But you should make yours even larger (maybe 4GB?). This will intercept anything that does swap first.

I also start some applications with systemd-run and specify memory caps and swap limits (often forbidding them from swapping entirely).

It makes a night-and-day difference for some workloads on my system, that would otherwise swap pointlessly, just tank performance and even de-stabilize the system.

cache is good, but you shouldn’t have that much swap usage with that overall memory utilization. 8t seems the kernel is making a decision to move a significant amount of memory to swap to have just a little bit more cache. that could make sense in some situations, especially with lots of RAM and a slow disk that is frequently accessed… but probably not on your system. I see you have already reduced swappiness to 10, so maybe all that was snapped out is really rarely used memory, but I wouldn’t want even that on an SSD so you could try setting swappiness to 0. that way it should only use the swap when there is no more cache to free, I think, but the behaviour of this sysctl was not always clear like that.

https://docs.kernel.org/admin-guide/sysctl/vm.html?highlight=swappiness#swappiness

No point in messing with swappiness settings, trust me. You want to address the reason this happens in the first place, not disable or reduce swap.

I’ve had a nearly identical scenario at work. The reason? A data verification process that went through 100s of thousands of files to run their checksums and check integrity. This was working as intended, but the system alerts were firing because of that high buffer/cache utilization. So check which processes have highest IO and you might find the answer. Swap is just a side effect.

https://www.linuxatemyram.com/

This is normal. Linux caches disk reads so that you don’t have to go to her/SSD to reread blocks. If your app isn’t using the ram Linux will use it instead.

Zero reason to be concerned, because if your app allocates more memory Linux will evict the cached pages to make sure your apps have plenty of space.

In this case your apps have memory that isn’t being touched (sometimes a sign of a memory leak), and Linux has decided that caching your disk access is more advantageous than keeping the unused application memory in ram.

If you have apps that you don’t use often and are really slow to access when you finally do, keep lowering that swappiness value or just remove swap all together if it annoys you.

Looking at your graph you have a lot of ram that you don’t really need… I would just disable swap entirely. Otherwise you are just wasting write cycles on your SSD.