addr2line keeps me sane

Doing kernel work, I end up with a lot of text dumps of things. It's typical to get lots of junk that looks like gibberish:

[71818.339389] bash            S 0000000000000000     0  3829   3753 0x00000000
[71818.339389]  ffff88007a5fdd38 0000000000000086 0000000000000001 0000000000011e80
[71818.339389]  0000000000000000 ffff88007af31080 ffff88007bc15040 ffff88007fc11e80
[71818.339389]  ffff88007af31080 0000000000000000 ffff88007fc11e80 0000000000000000
[71818.339389] Call Trace:
[71818.339389]  [] ? check_preempt_curr+0x7a/0x90
[71818.339389]  [] ? try_to_wake_up+0x1e5/0x280
[71818.339389]  [] schedule+0x45/0x60
[71818.339389]  [] schedule_timeout+0x14f/0x250
[71818.339389]  [] n_tty_read+0x2f0/0x810
[71818.339389]  [] ? try_to_wake_up+0x280/0x280
[71818.339389]  [] tty_read+0xa6/0xe0
[71818.339389]  [] vfs_read+0xcb/0x170
[71818.339389]  [] sys_read+0x55/0x90
[71818.339389]  [] system_call_fastpath+0x16/0x1b

Let's say you were trying to interpret this stack trace. Sometimes, the compiler will inline function calls and they might not show up in a stack trace, so it is not immediately apparent how tty_read() might call try_to_wake_up(). You can disassemble or use a debugger, but those both require skill. I prefer to replace having skill with tools instead, which is why I love addr2line. You need to feed it a vmlinux (not a vmlinuz or bzImage mind you), but its output is wonderful:

dave@kernel:~/linux-2.6.git$ addr2line -e vmlinux ffffffff81389ff6
/home/dave/work/linux-2.6.git/drivers/tty/tty_io.c:959
dave@kernel:~/linux-2.6.git$ vi /home/dave/work/linux-2.6.git/drivers/tty/tty_io.c +959

Which points to:

                i = (ld->ops->read)(tty, file, buf, count);
        else
                i = -EIO;
        tty_ldisc_deref(ld); <------------------
        if (i > 0)
                inode->i_atime = current_fs_time(inode->i_sb);
        return i;
}

and it's fairly easy to follow the call path from there.

SSH Push?

Some VNC servers have a cool feature called "VNC Push". In the traditional setup, the VNC server (the thing with the display you are connecting to) sits around with an open socket and the client connects to it. "VNC Push"ing switches that around. The client sits around with an open socket and has the server connect to it.

This is handy when the "server" is behind a firewall that you do not control, for instance at your grandma's house. You can have an icon on the desktop that says something like "Get Help From Dave" to have it push a connection to your waiting VNC client.

Back to ssh... I have a similar situation to the VNC situation. I am installing a Chumby at a relative's house, and I'd like to still be able to get remote access to it. However, I do not want to depend on the router's port forwarding since it might get reset, or the router replaced. I'm also not sure if their router supports UPnP, nor if I trust it. I've done something similar in the past with ssh port forwarding tricks, but that requires keeping some kind of credentials on the remote side and maintaining a UNIX account (although it can be rssh protected).

I've managed to come up with a decent alternative using socat. socat is like netcat (UNIX command nc) on steroids since allows you to tie together all kinds of bidirectional communication like ptys, but especially sockets. It starts out with running a command on your internet-accessible system:

socat TCP4-LISTEN:12345 TCP4-LISTEN:56789

The '12345' and '56789' are arbitrary (just remember to open them up in your local firewall). You can use just about whatever TCP ports you want as long as they are >1024 and <65536. This tells socat to sit around on tcp port 12345 and wait for a connection. When it gets a connection, it should only then wait for another connection on port 56789. All the traffic that comes in
on port 56789 will be piped back out port 12345. The important distinction from netcat is that this works bidirectionally.

Next, run the following on your ssh server that's behind the firewall:

socat TCP:yourserverfoo.dyndns.org:12345 TCP4:localhost:22

That tells it to connect to yourserverfoo.dyndns.org's TCP port, and then pipe all that traffic to the local port 22 where the ssh daemon is listening. Finally, back on the internet-accessible system, you can do:

ssh -o HostName=localhost -p 56789 my-ssh-push

The "-o HostName" bit is helpful to keep your ~/.ssh/known_hosts file from getting confused about what the key for 'localhost' might be. Once you do this, verify and accept the SSH host key, you should get a chance to log in.

The last step is to set up your remote server to push the connection periodically. In my case, I just run this script on startup in the background. It's important when doing this to realize that just about anybody aware of your scheme could connect to your port and pretend to be your remote system. Be careful, make sure to pre-populate your ssh host keys as well as use key-based authentication instead of passwords.