NSLU2-Linux
view · edit · print · history

HowTo.ProbingTheKernel History

Hide minor edits - Show changes to markup

August 15, 2008, at 11:47 AM by Charles Lindsey -- /proc/sys/sunrpc/*_debug doesn\'t do hex
Changed line 66 from:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to say

to:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff (or rather 32767, since /proc/sys/sunrpc/*_debug doesn't do hex) will give you "everything available". I found it especially useful to say

July 29, 2008, at 02:41 PM by Charles Lindsey -- Initial Version complete
Changed lines 13-15 from:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files"
/proc/net/rpc/*/content For example, on my own slug, 'more /proc/net/rpc/nfsd.export/content' gives me\\

to:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files"

    /proc/net/rpc/*/content.

For example, on my own slug, 'more /proc/net/rpc/nfsd.export/content' gives me

Added line 25:
Changed line 38 from:
    @@NFSDDBG_EXPORT          0x0004
to:
    NFSDDBG_EXPORT          0x0004
Changed lines 66-67 from:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to use NFSDDBG_EXPORT when trying to discover why exportfs had not exported all the things I had asked it to.

to:

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to say

    echo 4 > /proc/sys/sunrpc/nfsd_debug

(i.e. NFSDDBG_EXPORT) when trying to discover why exportfs had not exported all the things I had asked it to.

July 29, 2008, at 02:27 PM by Charles Lindsey --
Changed lines 9-10 from:

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also revel to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

to:

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also reveal to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

Changed lines 14-21 from:
 -> /proc/net/rpc/*/content
For example, on my own slug, more /proc/net/rpc/nfsd.export/content gives me
->
#path domain(flags)
/media/sda1	clerew(rw,no_root_squash,sync,wdelay)
/	clerew(ro,no_root_squash,sync,wdelay,fsid=0)

Syslog and printk

to:
    /proc/net/rpc/*/content

For example, on my own slug, 'more /proc/net/rpc/nfsd.export/content' gives me
#path domain(flags)

    /media/sda1	clerew(rw,no_root_squash,sync,wdelay)
    /	clerew(ro,no_root_squash,sync,wdelay,fsid=0)

Syslog and [d]printk

Changed lines 25-70 from:
For nfs_debug
NFSDBG_VFS              0x0001 
NFSDBG_PAGECACHE        0x0008
NFSDBG_PROC             0x0010
NFSDBG_XDR              0x0020
NFSDBG_FILE             0x0040
NFSDBG_ROOT             0x0080
NFSDBG_CALLBACK         0x0100
NFSDBG_CLIENT           0x0200

For nfsd_debug
NFSDDBG_FH              0x0002
NFSDDBG_EXPORT          0x0004
NFSDDBG_SVC             0x0008
NFSDDBG_PROC            0x0010
NFSDDBG_FILEOP          0x0020
NFSDDBG_XDR             0x0100
NFSDDBG_LOCKD           0x0200

For nlm_debug (the lockd)
NLMDBG_SVC              0x0001
NLMDBG_CLIENT           0x0002
NLMDBG_SVCLOCK          0x0008
NLMDBG_MONITOR          0x0010
NLMDBG_SVCSUBS          0x0040
NLMDBG_HOSTCACHE        0x0080
NLMDBG_XDR              0x0100

For rpc_debug
RPCDBG_XPRT             0x0001
RPCDBG_CALL             0x0002
RPCDBG_AUTH             0x0010
RPCDBG_PMAP             0x0020
RPCDBG_SCHED            0x0040
RPCDBG_TRANS            0x0080
RPCDBG_SVCSOCK          0x0100
RPCDBG_SVCDSP           0x0200
RPCDBG_MISC             0x0400
RPCDBG_CACHE            0x0800

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available".

THIS WIKI STILL UNDER CONSTRICTION :-(

to:
    For nfs_debug
    NFSDBG_VFS              0x0001 
    NFSDBG_PAGECACHE        0x0008
    NFSDBG_PROC             0x0010
    NFSDBG_XDR              0x0020
    NFSDBG_FILE             0x0040
    NFSDBG_ROOT             0x0080
    NFSDBG_CALLBACK         0x0100
    NFSDBG_CLIENT           0x0200

    For nfsd_debug
    NFSDDBG_FH              0x0002
    @@NFSDDBG_EXPORT          0x0004
    NFSDDBG_SVC             0x0008
    NFSDDBG_PROC            0x0010
    NFSDDBG_FILEOP          0x0020
    NFSDDBG_XDR             0x0100
    NFSDDBG_LOCKD           0x0200

    For nlm_debug (the lockd)
    NLMDBG_SVC              0x0001
    NLMDBG_CLIENT           0x0002
    NLMDBG_SVCLOCK          0x0008
    NLMDBG_MONITOR          0x0010
    NLMDBG_SVCSUBS          0x0040
    NLMDBG_HOSTCACHE        0x0080
    NLMDBG_XDR              0x0100

    For rpc_debug
    RPCDBG_XPRT             0x0001
    RPCDBG_CALL             0x0002
    RPCDBG_AUTH             0x0010
    RPCDBG_PMAP             0x0020
    RPCDBG_SCHED            0x0040
    RPCDBG_TRANS            0x0080
    RPCDBG_SVCSOCK          0x0100
    RPCDBG_SVCDSP           0x0200
    RPCDBG_MISC             0x0400
    RPCDBG_CACHE            0x0800

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available". I found it especially useful to use NFSDDBG_EXPORT when trying to discover why exportfs had not exported all the things I had asked it to.

All the output produced by this debugging appears in the Syslog. If you normally connect with your slug using ssh, then it is useful to open a second ssh window with

    tail -f /var/log/messages

permanently running in it, and then you will be able to follow events as they happen.

Other facilities

I am sure there must be lots more handles to cause the kernel to reveal its internal workings just waiting to be discovered, so I hope that others will add them to this Wiki as time goes by.

July 29, 2008, at 01:55 PM by Charles Lindsey --
Changed lines 1-4 from:

{:title Probing the Kernel:}

When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

to:

When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

Changed lines 24-26 from:

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you canot deduce that that part of the kernel code had not been traversed.

to:

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you cannot deduce that that part of the kernel code had not been traversed because dprintk is actually a macro (<linux/sunrpc/debug.h>) that you have to activate by writing suitable magic numbers into /proc/sys/sunrpc/*_debug. Here are the magic numbers I have identified so far:

For nfs_debug
NFSDBG_VFS              0x0001 
NFSDBG_PAGECACHE        0x0008
NFSDBG_PROC             0x0010
NFSDBG_XDR              0x0020
NFSDBG_FILE             0x0040
NFSDBG_ROOT             0x0080
NFSDBG_CALLBACK         0x0100
NFSDBG_CLIENT           0x0200

For nfsd_debug
NFSDDBG_FH              0x0002
NFSDDBG_EXPORT          0x0004
NFSDDBG_SVC             0x0008
NFSDDBG_PROC            0x0010
NFSDDBG_FILEOP          0x0020
NFSDDBG_XDR             0x0100
NFSDDBG_LOCKD           0x0200

For nlm_debug (the lockd)
NLMDBG_SVC              0x0001
NLMDBG_CLIENT           0x0002
NLMDBG_SVCLOCK          0x0008
NLMDBG_MONITOR          0x0010
NLMDBG_SVCSUBS          0x0040
NLMDBG_HOSTCACHE        0x0080
NLMDBG_XDR              0x0100

For rpc_debug
RPCDBG_XPRT             0x0001
RPCDBG_CALL             0x0002
RPCDBG_AUTH             0x0010
RPCDBG_PMAP             0x0020
RPCDBG_SCHED            0x0040
RPCDBG_TRANS            0x0080
RPCDBG_SVCSOCK          0x0100
RPCDBG_SVCDSP           0x0200
RPCDBG_MISC             0x0400
RPCDBG_CACHE            0x0800

Clearly, these numbers can be 'or'ed together, and if in doubt 0x7fff will give you "everything available".

July 23, 2008, at 11:32 AM by Charles Lindsey --
Changed lines 1-2 from:

Probing the Kernel

to:

{:title Probing the Kernel:}

Changed lines 15-18 from:

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files" /proc/net/rpc/*/contentFor example, on my own slug, @@

to:
The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files"
-> /proc/net/rpc/*/content
For example, on my own slug, more /proc/net/rpc/nfsd.export/content gives me
->
#path domain(flags)
/media/sda1	clerew(rw,no_root_squash,sync,wdelay)
/	clerew(ro,no_root_squash,sync,wdelay,fsid=0)

Syslog and printk

Most Syslog messages that you see (/var/log/messages) arise from calls of printk in the kernel. and which you can locate in the kernel source code if you grep it diligently enough.

However, if you look closely you will also spot calls of dprintk in the kernel source, and if you do not see the corresponding message in the Syslog, you canot deduce that that part of the kernel code had not been traversed.

July 22, 2008, at 06:30 PM by Charles Lindsey -- New Page
Added lines 1-20:

Probing the Kernel

When operations involving the Kernel don't go as expected, and attempts to find what is happening by studying the Kernel Source Code simply get you into a mire of confusion, what tools are available to guide you through the jungle?

Kernel Debuggers

Although there are packages available such as KDB, KGDB and LTT, they all require the kernel itself to be patched before they can be used. Slugos includes no such patches (and maybe they would occupy more space in the Slug Flash than there is room for), but if someone is prepared to do the necessary groundwork, TPTB might well be prepared to incorporate them. So, in the meantime, let us see what other tools are available.

Strace

strace is an already-available module from this site. It can be used to trace (selectively) all calls upon kernel services including, in particular syscall, which will also revel to you amazing behind-the-scenes activities involving the 'procfs' file system which, for sure, were never called directly by your application.

Maps and Caches

The kernel contains many maps and caches most of whose contents can be inspected if only you can discover the correct incantation. In particular the various caches maintained for services accessed through RPC can be inspected through the "files" /proc/net/rpc/*/contentFor example, on my own slug, @@

THIS WIKI STILL UNDER CONSTRICTION :-(

view · edit · print · history · Last edited by Charles Lindsey.
Originally by Charles Lindsey.
Page last modified on August 15, 2008, at 11:47 AM