Response to "Optimizing Linux Memory Management..."

I read a fantastic article last week by some engineers from LinkedIn.  It was fantastic because it really made me think about how we want the kernel to work.  But, it also contains inaccuracies that trouble me.  I have a few thoughts on the article.

1. The "zone reclaim" feature at the root of the authors' troubles is not enabled everywhere, not even on all NUMA systems.  Hardware vendors are essentially responsible for whether this feature is on or not.
2. The "Linux is quite bad at cleaning up this garbage properly" comment really stings.  It's actually the opposite from what I believe and have been advising folks on for years.  Linux is actually fantastic at managing its garbage.  It is arguable that the kernel's default behavior is not obvious, that we should use zone_reclaim_mode=0 in all but the most extreme NUMA environments.  But, the fact is that the kernel was working both as designed and as documented.
3. Do not read in to this article too much, especially for trying to understand how the Linux VM or the kernel works.  The authors misread the "global spinlock on the zone" source code and the interpretation in the article is dead wrong.
4. Memory pressure is caused when someone needs a particular kind of memory.  Usually, that memory is simply any free memory.  But, not all memory is the same from the kernel's perspective: you can see pressure when there is lots of free memory of other kinds.  A few examples of these special needs would be DMA-capable memory, physical contiguity for large pages, "low" memory, and NUMA locality.  The authors made the fundamental mistake of assuming that having any free memory means that there is no pressure.
5. There is no such thing as "NUMA memory balancing" in the kernel that was running.  The authors are observant in noticing that direct page scans and thp_splits occur at the same times, but they are wholly incorrect in assuming that these constitute any intentional rebalancing.  "Transparent HugePages do not play nice with NUMA systems" is also a dangerously broad thing to say, and it is not supported by even the data that the authors present.

The thing that most troubles me is how difficult it was for these fellow software engineers (with access to the source code and documentation for their kernel) to figure out what the kernel was doing.  How do we get end users to make the leap from "I see latency spikes in my custom database" to "I should set zone_reclaim_mode=0"?

The LinkedIn folks also pointed out that very few Google searches end up pointing to the (wonderful) linux-mm wiki.  Does everybody know that it's there?  Does anybody actually use it?

USB (EHCI) Debugging Ports

 In my new job, I'm looking at kernel scalability on an 8-socket (160 logical cpu!) system.  Unfortunately, I'm having some troubles getting it to even boot wit
h 3.9-rc kernels.  It's been dying in very early boot, and doesn't have a hardware serial port!

I've recently discovered that there is early printk support for "USB Debug Ports", which I should be able to use in place of serial ports.  Darren Hart was kind enough to let me borrow his USB EHCI debug adapter, since the manufacturer is not selling them at the moment.


The debug adapter has two sides and it is not symmetric. This can be a bit confusing since if you plug it in to both systems while they are booted, you will get a /dev/ttyUSB0 on both that behaves the same.  In my case, only one side will power the device, and that is the side which goes in to the good system (the one you are not debugging).

I tried to get it to work on three different systems.  I only got it to work on one. :(

• Lenovo S10-3 netbook - Only one EHCI debug controller, but never got it to work. I assume the debug port is not connected to the outside world.
• Acer Veriton X4618G
• My system has two USB controllers,which means that I might need either "earlyprintk=dbgp" *OR* "earlyprintk=dbgp1". Theoretically, I could probably figure out which of the two physical controllers was which in lspci, then figure out the order in which earlyprintk=dbgp enumerates those, then track down how they're connected. But, I'm dumb, and I don't expect myself to perform this procedure correctly.
• My motherboard has 4 internal 10-pin USB headers (each with two actual ports), plus another 6 ports on the back. So, I've got 14 possible ports and two possible boot options. Worst case, I'm going to have to boot the system 28 times! I ended up finding the port on my 17th boot. Take good notes!
• I never did get "earlyprintk=dbgp" to work. Only "earlyprintk=dbgp1" worked. I'm not sure if this was my error somehow, or if that port is just not exposed.
• Some of the internal motherboard headers, despite having 10 pins, only actually have one port on them. I _believe_ this are intended for front-panel card readers, but just beware.
• Fujitsu Primequest 1800e2
• This systems problems in early boot were the reason I tried EHCI debugging to begin with.
• This is a large, expensive system.  The USB ports exposed to the outside world all appear to be on different USB buses than the ones that have the USB debug functionality.  The CD drive claimed to be on the right bus, but the USB bus was not accessible.
• I gave up and got a PCIe serial card instead

GFP_ATOMIC Allocation Failures

I see a good number of bug reports or complaints about GFP_ATOMIC allocations failing.  The general consensus seems to be that if there is an allocation failure in the kernel, something is wrong with the kernel (or specifically the VM).

GFP_ATOMIC allocations are special.  If you call kmalloc(GFP_KERNEL), or alloc_page(GFP_KERNEL), you actually implicitly pass a couple of other flags in:

 #define GFP_KERNEL (__GFP_WAIT | __GFP_IO | __GFP_FS)

The most important of those if __GFP_WAIT.  If you call in to the allocator with that flag, it tells the VM that it is OK for you to sleep for a bit.  The VM will actually put the allocating process to sleep in order to go off and free up other memory.

But, GFP_ATOMIC does not have the __GFP_WAIT bit set.  It effectively says to the kernel, "give me memory NOW and don't put me to sleep."  But, the implication here is that the kernel will not be given a chance to go off and free other memory.

Let's say you walk in to a fast food joint because you want french fries.  You walk up to the counter and order your fries, but they are out.  If you can wait (__GFP_WAIT), you'll get your fries in a moment once they cook another batch.  But, if you are in a hurry and can not wait for another batch to be cooked (GFP_ATOMIC) you are going to have to walk away empty handed.  You can complain that there should have been more fries cooked in advance (increase min_free_kbytes), but that only helps keep them from running out less often, not from ever running out.

Inherently, being in a hurry (GFP_ATOMIC) exposes you to the possibility of failure.  If you do not want failures, then do not use GFP_ATOMIC.

Honey, please stop spamming me!

Recently, I have received many emails that look like they're from my wife and other family members, and have innocuous subjects like "great" or "hi".  However, once opened, they usually contain a single http link to some suspicious-looking site.

My first reaction was, "oh no, my wife's computer is infected!"  But, upon closer examination, the messages all say:

From: MyWife Hansen <notmywife@yahoo.com>

The From: name matches, but the email address does not.  This is basically normal spam, but with a twist: someone has found out who I know in order to craft messages which I am very likely to open.   How are they making the connection?  There were two clues.

1. I only received these "from" people that have my same last name.  My dad, brother, and wife, but not my sister who goes by her married name.  
2. Of the messages from my wife, some included her maiden and married names, which is highly unusual.  She only goes by one or the other... except on Facebook.

Facebook has a "feature" where you can search for friends by email address.  I believe the spammers are creating fake Facebook and Yahoo accounts.  Once they are ready to spam me, they look up my email on Facebook and look at my friends list.  They pick a friend with the same last name, and set that as the "From" name.  From Facebook, they do not know what "MyWife Hansen"'s email address is, so they use one of the fraudulent addresses.  Some lessons:

1. Yahoo is the source of the spam, probably by letting fake accounts get created.   This has been going on for months at least.
2. Yahoo might check that the account and email address in "From:" match, but does not check the name, or use it to help indicate spam when messages are sent.
3. Once again, sharing information on Facebook is dangerous and has unintended consequences.  In this case, the only information the spammers needed was your email and your friends list.

Takeaway: Lock down your Facebook account so that your friends list is not available, especially to non-friends.  You can also remove the ability for people to find you by email address.