Назад | Перейти на главную страницу

отслеживание процесса bash для команды df

Я использую советы по это ТАК нить.

ps -awux|grep df
root       15826  0.0  0.0      0     0 ?        I<   May22   0:00 [cifs-dfscache]
myuser 3086246  0.0  0.0 216860  3212 ?        Ss   16:06   0:02 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df;echo '==> /proc/who <==';who;echo '==> /proc/end <==';echo '##Moba##'; done
myuser 3137650  0.0  0.0 215348   616 ?        D    16:27   0:00 df

Так что я cat стек процесса 3137650:

cat /proc/3137650/stack
[<0>] autofs_wait+0x25b/0x723
[<0>] autofs_mount_wait+0x49/0xf0
[<0>] autofs_d_automount+0xdb/0x200
[<0>] follow_managed+0x110/0x2c0
[<0>] walk_component+0x1e9/0x2f0
[<0>] path_lookupat+0x70/0x120
[<0>] filename_lookup+0x97/0x180
[<0>] user_statfs+0x33/0xa0
[<0>] __do_sys_statfs+0x10/0x30
[<0>] do_syscall_64+0x5b/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

И идентификатор другого процесса 3086246:

cat /proc/3086246/stack
[<0>] do_wait+0x1b3/0x220
[<0>] kernel_wait4+0x96/0x120
[<0>] do_syscall_64+0x5b/0xf0
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9

Затем:

xargs -0 -n 1 echo < /proc/3137650/environ
SHELL=/bin/bash
MATHEMATICA_HOME=/usr/local/Wolfram/Mathematica/11.3
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
XDG_CONFIG_HOME=/home/myuser/.config
SPARK_LOCAL_HOSTNAME=localhost
LMOD_DIR=/usr/share/lmod/lmod/libexec
PWD=/home/myuser
LOGNAME=myuser
XDG_SESSION_TYPE=tty
MODULESHOME=/usr/share/lmod/lmod
MANPATH=/usr/share/lmod/lmod/share/man:
CUDA_INCLUDE_DIRS=/usr/include/cuda
SPARK_MASTER_IP=127.0.0.1
HOME=/home/myuser
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
LANG=en_US.UTF-8
XDG_CONFIG_DIR=/home/myuser/.config
LMOD_SETTARG_FULL_SUPPORT=no
CUDA_INC_PATH=/usr/include/cuda
LMOD_VERSION=8.2.10
SSH_CONNECTION=x.x.x.x 51696 x.x.x.x 22
MODULEPATH_ROOT=/usr/share/modulefiles
XDG_SESSION_CLASS=user
LMOD_PKG=/usr/share/lmod/lmod
HADOOP_HOME=/usr/local/bin/hadoop-2.9.0
GUROBI_HOME=/home/student/gurobi811/linux64/
LESSOPEN=||/usr/bin/lesspipe.sh %s
USER=kudyba
LMOD_ROOT=/usr/share/lmod
SHLVL=1
BASH_ENV=/usr/share/lmod/lmod/init/bash
LMOD_sys=Linux
SPARK_HOME=/usr/local/bin/spark
SPARK_LOCAL_IP=127.0.0.1
XDG_SESSION_ID=7778
LD_LIBRARY_PATH=:/home/student/gurobi811/linux64//lib
XDG_RUNTIME_DIR=/run/user/6105
SSH_CLIENT=x.x.x.x 51696 22
PIG_INSTALL=/usr/local/bin/pig-0.17.0
SPARK_EXAMPLES_JAR=/usr/local/bin/spark-2.1.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.1.1.jar
KDEDIRS=/usr
XDG_DATA_DIRS=/home/myuser/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
PATH=/usr/local/bin/anaconda3/bin:/home/users/mzilversmit/ncbi-blast-2.7.1+/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/users/students/mchen177/gurobi811/linux64//bin:/usr/local/bin/spark/bin:/usr/local/bin/pig-0.17.0/bin:/opt/dell/srvadmin/bin:/usr/local/bin/spark/bin
MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core
SPARK_CLASSPATH=/usr/share/java/mysql-connector-java.jar
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/6105/bus
LMOD_CMD=/usr/share/lmod/lmod/libexec/lmod
BASH_FUNC_ml%%=() {  eval $($LMOD_DIR/ml_cmd "$@")
}
BASH_FUNC_module%%=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
_=/usr/bin/df

и из другого процесса:

xargs -0 -n 1 echo < /proc/3086246/environ
USER=myuser
LOGNAME=myuser
HOME=/home/myuser
PATH=/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
SHELL=/bin/bash
XDG_SESSION_ID=7778
XDG_RUNTIME_DIR=/run/user/xxxx
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/xxxx/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
SSH_CLIENT=edited 51696 22
SSH_CONNECTION=edited 51696 edited 22

И просто для подтверждения идентификатора родительского процесса:

ps --ppid  3086246
    PID TTY          TIME CMD
3137650 ?        00:00:00 df

Не уверен, что echo ##Moba## подсказка, так как я использую Mobaxterm.

Когда я пробую strace родительского процесса, которого он просто ждет:

strace -p  3086246
strace: Process 3086246 attached
wait4(-1,

Однако дочерний процесс возвращает:

strace -p 3137650
strace: attach: ptrace(PTRACE_SEIZE, 3137650): Operation not permitted

Для хорошей меры:

pstree -pl 3137650
df(3137650)

и:

pstree -pl 3086246
bash(3086246)───df(3137650)

и:

ps -fp 3086246
UID          PID    PPID  C STIME TTY          TIME CMD
myuser   3086246       1  0 16:06 ?        00:00:02 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head

Так TTY это '?', ничего из cronjobs, что я мог видеть.

Сделайте 2 разных статуса Ss для 3086246 и D для 3137650, хороший стол здесь:

D    uninterruptible sleep (usually IO)
S    interruptible sleep (waiting for an event to complete)
s    is a session leader

Я также попробовал gdb на PID 3086246:

Reading symbols from /usr/bin/bash...
Reading symbols from /usr/lib/debug/usr/bin/bash-5.0.17-1.fc32.x86_64.debug...
Reading symbols from /lib64/libtinfo.so.6...
Reading symbols from /usr/lib/debug/usr/lib64/libtinfo.so.6.1-6.1-15.20191109.fc32.x86_64.debug...
Reading symbols from /lib64/libdl.so.2...
Reading symbols from /usr/lib/debug/usr/lib64/libdl-2.31.so.debug...
Reading symbols from /lib64/libc.so.6...
Reading symbols from /usr/lib/debug/usr/lib64/libc-2.31.so.debug...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
Reading symbols from /usr/lib/debug/usr/lib64/ld-2.31.so.debug...
0x00007f8e059ccf3a in __GI___wait4 (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7fffa494ff10, options=options@entry=0,
    usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:27
27        return SYSCALL_CANCEL (wait4, pid, stat_loc, options, usage);

Есть мысли или другие команды отладки, которые стоит попробовать?

Изменить: добавлен статус systemctl для PID в соответствии с предложением @ michael-hampton:

systemctl status 3086246 -l --no-pager
● session-7778.scope - Session 7778 of user myuser
     Loaded: loaded (/run/systemd/transient/session-7778.scope; transient)
  Transient: yes
     Active: active (abandoned) since Mon 2020-06-29 13:03:32 EDT; 11h ago
      Tasks: 2
     Memory: 316.9M
        CPU: 11min 46.751s
     CGroup: /user.slice/user-6105.slice/session-7778.scope
             ├─3086246 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df;echo '==> /proc/who <==';who;echo '==> /proc/end <==';echo '##Moba##'; done
             └─3137650 df

Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           20 May 29 14:35 usr/share/unimaps -> /usr/lib/kbd/unimaps
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   3 root     root            0 May 29 14:35 var
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           11 May 29 14:35 var/lock -> ../run/lock
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root            6 May 29 14:35 var/run -> ../run
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   2 root     root            0 May 29 14:35 var/tmp
Jun 29 16:26:21 ourserver  dracut[3126800]: ========================================================================
Jun 29 16:26:21 ourserver  dracut[3126800]: *** Creating initramfs image file '/boot/initramfs-5.6.19-300.fc32.x86_64.tmp' done ***
Jun 29 16:27:21 ourserver  systemd-tmpfiles[3137785]: /usr/lib/tmpfiles.d/lxdm.conf:1: Line references path below legacy directory /var/run/, updating /var/run/lxdm → /run/lxdm; please update the tmpfiles.d/ drop-in file accordingly.
Jun 29 16:48:05 ourserver  su[2983955]: pam_unix(su:session): session closed for user root
Jun 29 16:48:07 ourserver  sshd[2983731]: pam_unix(sshd:session): session closed for user myuser

и:

systemctl status 3137650 -l --no-pager
● session-7778.scope - Session 7778 of user myuser
     Loaded: loaded (/run/systemd/transient/session-7778.scope; transient)
  Transient: yes
     Active: active (abandoned) since Mon 2020-06-29 13:03:32 EDT; 11h ago
      Tasks: 2
     Memory: 316.9M
        CPU: 11min 46.751s
     CGroup: /user.slice/user-6105.slice/session-7778.scope
             ├─3086246 bash -c while [ -d /proc/$PPID ]; do sleep 1;head -v -n 8 /proc/meminfo; head -v -n 2 /proc/stat /proc/version /proc/uptime /proc/loadavg /proc/sys/fs/file-nr /proc/sys/kernel/hostname; tail -v -n 16 /proc/net/dev;echo '==> /proc/df <==';df;echo '==> /proc/who <==';who;echo '==> /proc/end <==';echo '##Moba##'; done
             └─3137650 df

Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           20 May 29 14:35 usr/share/unimaps -> /usr/lib/kbd/unimaps
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   3 root     root            0 May 29 14:35 var
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root           11 May 29 14:35 var/lock -> ../run/lock
Jun 29 16:26:21 ourserver  dracut[3126800]: lrwxrwxrwx   1 root     root            6 May 29 14:35 var/run -> ../run
Jun 29 16:26:21 ourserver  dracut[3126800]: drwxr-xr-x   2 root     root            0 May 29 14:35 var/tmp
Jun 29 16:26:21 ourserver  dracut[3126800]: ========================================================================
Jun 29 16:26:21 ourserver  dracut[3126800]: *** Creating initramfs image file '/boot/initramfs-5.6.19-300.fc32.x86_64.tmp' done ***
Jun 29 16:27:21 ourserver  systemd-tmpfiles[3137785]: /usr/lib/tmpfiles.d/lxdm.conf:1: Line references path below legacy directory /var/run/, updating /var/run/lxdm → /run/lxdm; please update the tmpfiles.d/ drop-in file accordingly.
Jun 29 16:48:05 ourserver  su[2983955]: pam_unix(su:session): session closed for user root
Jun 29 16:48:07 ourserver  sshd[2983731]: pam_unix(sshd:session): session closed for user myuser

Я начинаю думать, что это могло быть эта ошибка за исключением того, что ssh не медленный. Бег:

systemctl | grep "abandoned" | grep -e "-[[:digit:]]"

вернул несколько заброшенных сессий ssh. Бег:

systemctl | grep "abandoned" | grep -e "-[[:digit:]]" | sed "s/.scope.*/.scope/" | xargs systemctl stop

удаляет все прерванные сеансы и 'df'команда исчезает из'ps'.

Тайна раскрыта. Это действительно MobaXterm, внизу слева есть опция удаленного мониторинга: