When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

Kernel Debuggers

Although there are packages available such as KDB, KGDB and LTT, they all require the kernel itself to be patched before they can be used. Slugos includes no such patches (and maybe they would occupy more space in the Slug Flash than there is room for), but if someone is prepared to do the necessary groundwork, TPTB might well be prepared to incorporate them. So, in the meantime, let us see what other tools are available.

Strace

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also reveal to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

Maps and Caches

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files"

    /proc/net/rpc/*/content.

For example, on my own slug, 'more /proc/net/rpc/nfsd.export/content' gives me

    #path domain(flags)
    /media/sda1	clerew(rw,no_root_squash,sync,wdelay)
    /	clerew(ro,no_root_squash,sync,wdelay,fsid=0)

Syslog and [d]printk

Most Syslog messages that you see (/var/log/messages) arise from calls of printk in the kernel. and which you can locate in the kernel source code if you grep it diligently enough.

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you cannot deduce that that part of the kernel code had not been traversed because dprintk is actually a macro (<linux/sunrpc/debug.h>) that you have to activate by writing suitable magic numbers into /proc/sys/sunrpc/*_debug. Here are the magic numbers I have identified so far:

    For nfs_debug
    NFSDBG_VFS              0x0001 
    NFSDBG_PAGECACHE        0x0008
    NFSDBG_PROC             0x0010
    NFSDBG_XDR              0x0020
    NFSDBG_FILE             0x0040
    NFSDBG_ROOT             0x0080
    NFSDBG_CALLBACK         0x0100
    NFSDBG_CLIENT           0x0200

    For nfsd_debug
    NFSDDBG_FH              0x0002
    NFSDDBG_EXPORT          0x0004
    NFSDDBG_SVC             0x0008
    NFSDDBG_PROC            0x0010
    NFSDDBG_FILEOP          0x0020
    NFSDDBG_XDR             0x0100
    NFSDDBG_LOCKD           0x0200

    For nlm_debug (the lockd)
    NLMDBG_SVC              0x0001
    NLMDBG_CLIENT           0x0002
    NLMDBG_SVCLOCK          0x0008
    NLMDBG_MONITOR          0x0010
    NLMDBG_SVCSUBS          0x0040
    NLMDBG_HOSTCACHE        0x0080
    NLMDBG_XDR              0x0100

    For rpc_debug
    RPCDBG_XPRT             0x0001
    RPCDBG_CALL             0x0002
    RPCDBG_AUTH             0x0010
    RPCDBG_PMAP             0x0020
    RPCDBG_SCHED            0x0040
    RPCDBG_TRANS            0x0080
    RPCDBG_SVCSOCK          0x0100
    RPCDBG_SVCDSP           0x0200
    RPCDBG_MISC             0x0400
    RPCDBG_CACHE            0x0800

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff (or rather 32767, since /proc/sys/sunrpc/*_debug doesn't do hex) will give you "everything available". I found it especially useful to say

    echo 4 > /proc/sys/sunrpc/nfsd_debug

(i.e. NFSDDBG_EXPORT) when trying to discover why exportfs had not exported all the things I had asked it to.

All the output produced by this debugging appears in the Syslog. If you normally connect with your slug using ssh, then it is useful to open a second ssh window with

    tail -f /var/log/messages

permanently running in it, and then you will be able to follow events as they happen.

Other facilities

I am sure there must be lots more handles to cause the kernel to reveal its internal workings just waiting to be discovered, so I hope that others will add them to this Wiki as time goes by.