Solaris crash dump analysisAnalyse de crash dump Solaris

You have encountered a solaris kernel crash. How you can resolve this issue ? I give you some tips to help you. The post can be updated if I found others tips.

The crash dump is localized in /var/crash/. To have access, you must be root. If you don’t find any crash with the day date, maybe the partition is full or the swap size is less than the memory size.

This is an example:

drwxr-xr-x 2 root root 512 Jun 29 10:04 .
drwxr-xr-x 3 root root 512 Feb 7 12:45 ..
-rw-r–r– 1 root root 2 Jun 29 10:04 bounds
-rw-r–r– 1 root root 868244 Jun 27 12:26 unix.4
-rw-r–r– 1 root root 861782 Jun 28 22:30 unix.5
-rw-r–r– 1 root root 857575 Jun 29 10:04 unix.6
-rw-r–r– 1 root root 126050304 Jun 27 12:26 vmcore.4
-rw-r–r– 1 root root 121364480 Jun 28 22:31 vmcore.5
-rw-r–r– 1 root root 98418688 Jun 29 10:04 vmcore.6

If you want to analyze the 29th June crash, enter this command:

mdb 6

Loading modules: [ unix krtld genunix ip usba ufs_log logindmux ptm cpc ipc random nfs ]

::status will give you the crash status

debugging crash dump vmcore.6 (64-bit) from daytona
operating system: 5.9 Generic_118558-06 (sun4u)
panic message: vn_rele: vnode ref count 0
dump content: kernel pages only

$c give you the stack when the crash occurs

vpanic(3, 1170b10, 20, 0, 3fb63a00, 0)
vn_rele+0x1c(300016d9ea8, 3000399b000, 20, 0, 30000267388, 0)
closef+0x84(30003aaa778, 3000399b000, 20, 2000, 100, 30008ff4490)
munlink+0x340(0, 30000bb7b98, 9, 30000267388, 2a1006e5aec, 5400)
strioctl+0x27c0(30000267388, 30001643788, 30001643878, 0, 5000, 0)
spec_ioctl+0x94(300039a7d98, 530d, bb, 202003, 30000267388, 2a1006e5aec)
ioctl+0x1f8(5, 530d, bb, 0, 10007adb8, 10007ac08)
syscall_trap+0x88(5, 530d, bb, 0, 10007adb8, 10007ac08)

::cpuinfo give you the running thread

ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 00001431638 1b 13 0 59 no no t-0 3000399b000 usam_daemon

In this example, the thread number is 3000399b000. To see the thread stack you can write

3000399b000::findstack

000002a1006e4b31 vn_rele+0x1c()
000002a1006e4be1 closef+0x84()
000002a1006e4c91 munlink+0x340()
000002a1006e4d71 strioctl+0x27c0()
000002a1006e50e1 spec_ioctl+0x94()
000002a1006e51a1 ioctl+0x1f8()
000002a1006e52f1 syscall_trap+0x88()

::thread give you the thread list

ADDR STATE FLG PFLG SFLG PRI EPRI PIL INTR
000000000140a000 stopped 0 0 3e03 96 0 0 n/a
000002a10001fd40 onproc 8 0 3 -1 0 0 n/a
000002a10003dd40 sleep 8 0 3 60 0 0 n/a
000002a100037d40 free 9 0 3 0 0 0 n/a
000002a100031d40 free 9 0 3 0 0 0 n/a
000002a10002bd40 free 9 0 3 0 0 0 n/a
000002a100025d40 free 9 0 3 0 0 0 n/a
000002a10005dd40 free 9 0 3 0 0 0 n/a
000002a100057d40 free 9 0 3 0 0 0 n/a
000002a100051d40 free 9 0 3 0 0 0 n/a
000002a10004bd40 free 9 0 3 169 0 10 2a100045d40
000002a100045d40 free 9 0 3 160 0 1 2a10001fd40
000002a10007dd40 free 9 0 3 168 0 9 2a100045d40
000002a100071d40 sleep 8 0 3 99 0 0 n/a
000002a10006bd40 sleep 8 0 3 99 0 0 n/a
000002a1007a5d40 sleep 8 0 3 60 0 0 n/a
000002a1006cbd40 run 8 0 13 60 0 0 n/a
000002a100605d40 sleep 8 0 3 60 0 0 n/a
000002a100691d40 sleep 8 0 3 60 0 0 n/a
000002a100697d40 sleep 8 0 3 60 0 0 n/a
000002a1006abd40 sleep 8 0 3 60 0 0 n/a

Another dcmd (a dcmd is a mdb command) is whatis.
For example If you get the l_qbot address 30003b01660, and you don’t know the object type you do:

30003b01660::whatis

30003b01660 is 30003b01570+f0, allocated from queue_cache

The l_qbot is a queue allocated from the queue_cache, you can do:

30003b01660::queue

ADDR MODULE FLAGS NBLK
0000030003b01660 mom 004022 0 0000000000000000

Don’t forget to use the help command to have more inforamation about a specific dcmd

:help queue

NAME
queue – filter and display STREAM queue

SYNOPSIS
addr ::queue [-q|v] [-m mod] [-f flag] [-F flag] [-s syncq_addr]

DESCRIPTION

Print queue information for a given queue pointer.

Without the address of a “queue_t” structure given, print information about al
l
queues in the “queue_cache”.

Options:
-v: be verbose – print symbolic flags falues
-q: be quiet – print queue pointer only
-f flag: print only queues with flag set
-F flag: print only queues with flag NOT set
-m modname: print only queues with specified module name
-s syncq_addr: print only queues which use specified syncq

Available conversions:
q2rdq: given a queue addr print read queue pointer
q2wrq: given a queue addr print write queue pointer
q2otherq: given a queue addr print other queue pointer
q2syncq: given a queue addr print syncq pointer (::help syncq)
q2stream: given a queue addr print its stream pointer
(see ::help stream and ::help stdata)

To walk q_next pointer of the queue use

queue_addr::walk qnext

ATTRIBUTES

Target: kvm
Module: genunix
Interface Stability: Unstable

If you do 30003b01660::queue -v you have

ADDR MODULE FLAGS NBLK
0000030003b01660 mom 004022 0 0000000000000000
|
+–> QWANTR Someone wants to read Q
QUSE This queue in use (allocation)
QMTSAFE stream module is MT-safe

When you have finished to quit you do:

$q

It’s all good debugging

UPDATE1 when you have a queue address you can dump all the associated stream by the q2stream dcmd.
Vous avez un Solaris kernel crash. Comment résoudre ce problème ? Je vous donne quelques tips. Ce post peut être mis à jour si je trouve d’autre tips.

Le dump crash est localisé sous /var/crash/. Pour y accéder, vous devez être root. Si vous ne trouvez aucun fichier avec la date du jour, c’est que peut être la partition est pleine ou la taille de swap est inférieur à la taille de la mémoire.

Voici un exemple:

drwxr-xr-x 2 root root 512 Jun 29 10:04 .
drwxr-xr-x 3 root root 512 Feb 7 12:45 ..
-rw-r–r– 1 root root 2 Jun 29 10:04 bounds
-rw-r–r– 1 root root 868244 Jun 27 12:26 unix.4
-rw-r–r– 1 root root 861782 Jun 28 22:30 unix.5
-rw-r–r– 1 root root 857575 Jun 29 10:04 unix.6
-rw-r–r– 1 root root 126050304 Jun 27 12:26 vmcore.4
-rw-r–r– 1 root root 121364480 Jun 28 22:31 vmcore.5
-rw-r–r– 1 root root 98418688 Jun 29 10:04 vmcore.6

Si vous voulez analyser le crash du 29 juin, entrez la commande:

mdb 6

Loading modules: [ unix krtld genunix ip usba ufs_log logindmux ptm cpc ipc random nfs ]

::status will give you the crash status

debugging crash dump vmcore.6 (64-bit) from daytona
operating system: 5.9 Generic_118558-06 (sun4u)
panic message: vn_rele: vnode ref count 0
dump content: kernel pages only

$c vous donne la pile au moment du crash

vpanic(3, 1170b10, 20, 0, 3fb63a00, 0)
vn_rele+0x1c(300016d9ea8, 3000399b000, 20, 0, 30000267388, 0)
closef+0x84(30003aaa778, 3000399b000, 20, 2000, 100, 30008ff4490)
munlink+0x340(0, 30000bb7b98, 9, 30000267388, 2a1006e5aec, 5400)
strioctl+0x27c0(30000267388, 30001643788, 30001643878, 0, 5000, 0)
spec_ioctl+0x94(300039a7d98, 530d, bb, 202003, 30000267388, 2a1006e5aec)
ioctl+0x1f8(5, 530d, bb, 0, 10007adb8, 10007ac08)
syscall_trap+0x88(5, 530d, bb, 0, 10007adb8, 10007ac08)

::cpuinfo vous donne le thread qui tournait

ID ADDR FLG NRUN BSPL PRI RNRN KRNRN SWITCH THREAD PROC
0 00001431638 1b 13 0 59 no no t-0 3000399b000 usam_daemon

Dans l’exemple, l’adresse de la thread est 3000399b000. Pour voir la pile de la thread exécutez:

3000399b000::findstack

000002a1006e4b31 vn_rele+0x1c()
000002a1006e4be1 closef+0x84()
000002a1006e4c91 munlink+0x340()
000002a1006e4d71 strioctl+0x27c0()
000002a1006e50e1 spec_ioctl+0x94()
000002a1006e51a1 ioctl+0x1f8()
000002a1006e52f1 syscall_trap+0x88()

::thread vous donne la liste des thread

ADDR STATE FLG PFLG SFLG PRI EPRI PIL INTR
000000000140a000 stopped 0 0 3e03 96 0 0 n/a
000002a10001fd40 onproc 8 0 3 -1 0 0 n/a
000002a10003dd40 sleep 8 0 3 60 0 0 n/a
000002a100037d40 free 9 0 3 0 0 0 n/a
000002a100031d40 free 9 0 3 0 0 0 n/a
000002a10002bd40 free 9 0 3 0 0 0 n/a
000002a100025d40 free 9 0 3 0 0 0 n/a
000002a10005dd40 free 9 0 3 0 0 0 n/a
000002a100057d40 free 9 0 3 0 0 0 n/a
000002a100051d40 free 9 0 3 0 0 0 n/a
000002a10004bd40 free 9 0 3 169 0 10 2a100045d40
000002a100045d40 free 9 0 3 160 0 1 2a10001fd40
000002a10007dd40 free 9 0 3 168 0 9 2a100045d40
000002a100071d40 sleep 8 0 3 99 0 0 n/a
000002a10006bd40 sleep 8 0 3 99 0 0 n/a
000002a1007a5d40 sleep 8 0 3 60 0 0 n/a
000002a1006cbd40 run 8 0 13 60 0 0 n/a
000002a100605d40 sleep 8 0 3 60 0 0 n/a
000002a100691d40 sleep 8 0 3 60 0 0 n/a
000002a100697d40 sleep 8 0 3 60 0 0 n/a
000002a1006abd40 sleep 8 0 3 60 0 0 n/a

Une autre dcmd est whatis (une dcmd est une mdb commande)

Dans notre exemple, si vous voulez connaitre le type de l’objet l_qbot à l’adresse 30003b01660

30003b01660::whatis

30003b01660 is 30003b01570+f0, allocated from queue_cache

l’objet l_qbot est une queue allouée de la queue_cache, vous pouvez exécuter:

30003b01660::queue

ADDR MODULE FLAGS NBLK
0000030003b01660 mom 004022 0 0000000000000000

Ne pas oublier d’utiliser la commande help pour avoir plus d’information sur une dcmd spécifique

:help queue

NAME
queue – filter and display STREAM queue

SYNOPSIS
addr ::queue [-q|v] [-m mod] [-f flag] [-F flag] [-s syncq_addr]

DESCRIPTION

Print queue information for a given queue pointer.

Without the address of a “queue_t” structure given, print information about al
l
queues in the “queue_cache”.

Options:
-v: be verbose – print symbolic flags falues
-q: be quiet – print queue pointer only
-f flag: print only queues with flag set
-F flag: print only queues with flag NOT set
-m modname: print only queues with specified module name
-s syncq_addr: print only queues which use specified syncq

Available conversions:
q2rdq: given a queue addr print read queue pointer
q2wrq: given a queue addr print write queue pointer
q2otherq: given a queue addr print other queue pointer
q2syncq: given a queue addr print syncq pointer (::help syncq)
q2stream: given a queue addr print its stream pointer
(see ::help stream and ::help stdata)

pour aller au pointeur q_next de la queue utiliser:

queue_addr::walk qnext

ATTRIBUTES

Target: kvm
Module: genunix
Interface Stability: Unstable

Si vous faites 30003b01660::queue -v, vous allez avoir:

ADDR MODULE FLAGS NBLK
0000030003b01660 mom 004022 0 0000000000000000
|
+–> QWANTR Someone wants to read Q
QUSE This queue in use (allocation)
QMTSAFE stream module is MT-safe

Pour terminer exécuter:

$q

C’est tout, bon debug

UPDATE1 quand vous avez une adresse de queue, vous pouvez dumper la stream associée avec la dcmd q2stream.

New gamesnouveau jeu

I have to continue to develop my second game on iOS. I will see what new feature has been Introduced by Apple on the last iOS version.
I didn’t like theobject’s animation in my game, and I have to rework this part.
As I work on the game only after my work, the project goes slowly.

Je dois continuer le développement de mon second jeu sous iOS. Je vais regarder les fonctionnalités nouvelles introduite par Apple dans la dernière version d’iOS:
Je n’aime pas l’animation des objets de mon jeu et je dois retravailler cette partie
Comme je travaille dessus seulement après le boulot, le travail avance vraiment lentement.