Mulinux | Linux & DevOps Notes

MySQL 常用排障与救火速查 (Runbook)

Mon, 17 Oct 2022 13:57:34 +0800

🚨 使用指南： 这是一份专为生产环境数据库宕机、卡顿、爆满设计的救火急诊室指南。越致命的状况对应的代码越靠前，日常业务代码被折叠至附录。在此页面所抄写的每一行指令，都应该怀揣对生产环境的敬畏。

01. 致命故障阻断：连接爆满与死锁

定位卡死的源头查询

当业务发生大面积连接超时（Too many connections），必须立即揪出源头。

# 查看全盘正在执行的所有线程与查询。重点盯住 `Time` 耗时超长的行，和 `State` 为 Locked 的进程。
show full processlist;

# 选定特定祸首（譬如 Id 为 1024的查询），手动狙杀以释放表锁：
kill 1024;

批量猎杀僵尸查询（雪崩级自救）

当数据库瞬间被几十上百个卡死的查询打满，一个个 COPY ID 根本来不及，你可以利用下面语句跑出批量 KILL 脚本：

# 找出所有运行超过 30 秒并在指定库下执行的坏死语句，让引擎自动吐出批量 KILL 指令。
select concat('KILL ',id,';') from information_schema.processlist 
where db='你的库名' and time > 30 and command = 'Query';
# 将上一步产出的多行 KILL 代码复制，并一键黏贴至控制台批量斩杀！

定位深层死锁与哪张库表堵塞

# 查询有哪张表正在被死锁占用处于写入阻塞中 (In_use 大于 0 代表被锁)
show open tables where In_use > 0;

# 查询底层 InnoDB 引擎的运转全貌与事务挂起细节（硬核排查指令）
show engine innodb status;

02. 特权拉起与账号盲切抢救

免密强拉服务（忘 Root 密码 / 权鉴紊乱自救）

如果因为乱改权限导致数据库完全锁死谁也连不进去，连 Root 密码都被遗忘：

# 1. 在 Linux 服务台停掉 MySQL 服务
systemctl stop mysqld
# 2. 利用跳过授权表模式强行拉起服务，直接跨过登陆鉴权网
mysqld_safe --skip-grant-tables &
# 3. 此时可以直接由于空密码登入 MySQL 开始重置你的正常帐户和权限了！
mysql -u root

阻断异常暴露：给内网降级配置临时入口

# 紧急创建一个只能在专门客户端网段 (%) 连入的排障专用号，锁定至单一业务库避免祸及池鱼
create user 'sos_user'@'%' identified by 'strong_password';
grant all privileges on your_db_name.* to 'sos_user'@'%';
flush privileges;

03. 性能深潜与大口径清洗

执行计划：质检索引缺失导致的隐性卡顿

业务不会宕机，就是单纯拖慢了 CPU 和 I/O，通常是索引缺失。

# 在极慢的日志查询前加上 EXPLAIN，重点看 `type` 字段，如果是 'ALL' 即引发了最坏的全表扫描。
explain select * from user_logs where created_at > '2022-10-01';

脏数据防炸阻断（防爬虫注入崩溃）

# 上游送来的数据主键如果冲突了不报错终止写入，转为安全的 Update 覆盖策略
insert into my_table (id, name, count) values (1, 'apple', 1) 
on duplicate key update count = count + 1;

大面数据纠偏清洗 (利用高频的 JOIN UPDATE)

# 利用 B 表（准数据字典）的最新数据去强制刷新修复 A 表损坏的数据
update table_a a 
inner join table_b b on a.id = b.a_id 
set a.status = b.status 
where a.status != b.status;

04. 灾后数据导出抢救

当磁盘快要报警红线：舍车保帅库抛离算法

如果一个几百 G 的历史归档库马上要吃光操作系统的 IO 容量，你需要强行抽出核心表保命。

# `--ignore-table` 极度实用：导出时强行剔除无用的海量流水线日志大表，秒速把最核心库备份抽离换台机器
mysqldump -uroot -p"Password" --databases itoa126 --ignore-table=itoa126.scr_big_log_table > /app/dump_core.sql

# 也可以借助 Linux 管道过滤，一通导出来只包含你需要的关键业务分库
mysql -e "show databases;" -uroot -p"Pass" | grep -Ev "Database|test|information_schema" | xargs mysqldump -uroot -p"Pass" --databases > /app/dump_clean.sql

百万数据导出为 CSV 将压力脱手

# 让 MySQL 引擎的 I/O 底层直写文件，防止导出几百万行数据把你的运维客户端直接卡死爆炸
select from_unixtime(left(event_time,10)),origin_id from tb_event 
where merge_key=1 order by event_time 
into outfile '/tmp/tb_event_20221017.csv' fields terminated by ',';

附录：日常业务打磨工具箱

核级删除：极速清空表

# 相比 Delete 性能极速，且不记入日志，不可回滚，自增 ID 会重置！
truncate table table_name;

UNIX 时间戳转换

select unix_timestamp('2022-03-21 14:28:30') from tb_event;
# 时间戳转回可用日期版
select from_unixtime(1666060280, '%Y-%m-%d %H:%i:%S') from tb_event;

nginx logrotate日志回滚

Tue, 09 Aug 2022 10:04:22 +0800

logrotate配置

vi /etc/logrotate.d/nginx
/var/log/nginx/*.log {
    rotate 7
    size 5k
    dateext
    dateformat -%Y-%m-%d-%s
    missingok
#    compress
    sharedscripts
    postrotate
        test -r /run/nginx.pid && kill -USR1 `cat /run/nginx.pid`
    endscript
}

日志回滚命令

logrotate -v -f /etc/logrotate.d/nginx

定时任务

crontab -e
*/5 * * * * /usr/sbin/logrotate -v -f /etc/logrotate.d/nginx

CDH中Kafka实现多端口监听

Mon, 07 Mar 2022 14:45:11 +0800

节点分布

名称	集群IP	业务IP
AIOPS01CALC	192.168.1.1	10.10.1.1
AIOPS02CALC	192.168.1.2	10.10.1.2
AIOPS03CALC	192.168.1.3	10.10.1.3

需求描述

集群内部通过192.168.1.X网段通信，对外通过10.10.1.X网段通信，可使用如下参数配置实现多网段不同端口监听访问。

配置参数

kafka.properties的Kafka Broker高级配置代码	Kafka Broker(aiops01calc)	listeners=PLAINTEXT://192.168.1.1:29092,EXTERNAL://10.10.1.1:19092advertised.listeners=PLAINTEXT://192.168.1.1:29092,EXTERNAL://10.10.1.1:19092listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
	Kafka Broker(aiops02calc)	listeners=PLAINTEXT://192.168.1.2:29092,EXTERNAL://10.10.1.2:19092advertised.listeners=PLAINTEXT://192.168.1.2:29092,EXTERNAL://10.10.1.2:19092listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
	Kafka Broker(aiops03calc)	listeners=PLAINTEXT://192.168.1.3:29092,EXTERNAL://10.10.1.3:19092advertised.listeners=PLAINTEXT://192.168.1.3:29092,EXTERNAL://10.10.1.3:19092listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT

logstash使用示例

Sun, 12 Dec 2021 10:04:22 +0800

es2csv

input {
 elasticsearch {
    hosts => "localhost:9200"
    index => "kibana_sample_data_ecommerce"
	user => "elastic"
	password => "password"
    query => '{  # query可不填写
    "query": {
      "match_all": {}
      }
    }'
  }
}
 
output {
  csv {
    fields => ["@message"]
    path => "/tmp/csv-export.csv"
  }
}

es2file

input {
 elasticsearch {
    hosts => "localhost:9200"
    index => "kibana_sample_data_ecommerce"
	user => "elastic"
	password => "password"
    query => '{  
    "query": {
      "match_all": {}
      }
    }'
  }
}
 
output {
  file {
    path => "/tmp/csv-export.txt"
	codec => line { format => "%{@message}" }
  }
}

启动命令

/app/logstash/bin/logstash -f /app/logstash/config/esTocsv.conf

Redis 生产环境主从架构配置参考

Mon, 08 Feb 2021 15:59:42 +0800

prod xxxx redis

这是一份基于生产环境的主从 Redis 配置脱敏留档。供部署搭建时参考。

master redis.conf

bind 127.0.0.1 192.168.1.112
protected-mode yes
port 8888
tcp-backlog 2048
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /usr/local/redis/run/redis.pid
loglevel notice
logfile /usr/local/redis/log/redis.log
databases 16
always-show-logo yes
save ""
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /mds/redis
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 0 0 0
client-output-buffer-limit pubsub 0 0 0
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
maxmemory 0
# BEGIN 20200909 ALTERATION
requirepass password
# END 20200909 ALTERATION

slave redis.conf

bind 127.0.0.1 192.168.1.111
protected-mode yes
port 8888
tcp-backlog 2048
timeout 0
tcp-keepalive 300
daemonize yes
supervised no
pidfile /usr/local/redis/run/redis.pid
loglevel notice
logfile /usr/local/redis/log/redis.log
databases 16
always-show-logo yes
save ""
stop-writes-on-bgsave-error no
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /mds/redis
replica-serve-stale-data yes
replica-read-only yes
replicaof 192.168.1.112 9264
replica-priority 100
replica-lazy-flush yes
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-disable-tcp-nodelay no
lazyfree-lazy-eviction no
lazyfree-lazy-expire no
lazyfree-lazy-server-del no
appendonly no
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble no
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
latency-monitor-threshold 0
notify-keyspace-events ""
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-size -2
list-compress-depth 0
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
hll-sparse-max-bytes 3000
stream-node-max-bytes 4096
stream-node-max-entries 100
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit replica 0 0 0
client-output-buffer-limit pubsub 0 0 0
hz 10
dynamic-hz yes
aof-rewrite-incremental-fsync yes
rdb-save-incremental-fsync yes
maxmemory 0
# BEGIN 20200909 ALTERATION
requirepass password
masterauth password
# END 20200909 ALTERATION

Windows Find Software

Fri, 05 Feb 2021 14:11:42 +0800

查找windows是否安装某个软件

通过服务查找

powershell

Get-WmiObject -class Win32_Product |Select-Object -Property name,version

批处理

reg query HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\|find /i "sogou input"

通过路径名称匹配

Get-ChildItem C:\ProgramData\Microsoft\Windows\\'Start Menu\'\Programs\ -Filter *Premium*"
Get-ChildItem C:\\'Program Files\' -Filter *Premium*"
Get-ChildItem C:\\'Program Files (x86)\' -Filter *Premium*"

Zabbix监控断线，proxy日志提示Connection Restored

Thu, 14 Jan 2021 11:14:03 +0800

问题说明

zabbix监控数据出现断线

zabbixproxy日志

 12250:20210114:104415.661 resuming Zabbix agent checks on host "hostname": connection restored

系统日志

Jan 14 10:46:19 hostname kernel: nf_conntrack: table full, dropping packet.

解决

从日志上可看到，nf_conntrack设置的值不够大，导致包被丢弃，proxy与agent间歇性的断开重连，可进行系统配置调整进行解决，在/etc/sysctl.conf上增加以下配置。

# nf_conntrack
net.nf_conntrack_max = 655360
net.netfilter.nf_conntrack_tcp_timeout_established = 1200

容器化部署Wordpress

Wed, 23 Dec 2020 13:30:28 +0800

Install docker and docker-compose

wget http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
yum install docker-ce -y
systemctl start docker
systemctl enable docker
curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Deploy wordpress using official examples

Create wordpress directory

mkdir wordpress
cd wordpress/

Create docker-compose file

vi docker-compose.yml

version: '3.3'

services:
   db:
     image: mysql:5.7
     volumes:
       - db_data:/var/lib/mysql
     restart: always
     environment:
       MYSQL_ROOT_PASSWORD: password
       MYSQL_DATABASE: wordpress
       MYSQL_USER: wordpress
       MYSQL_PASSWORD: wordpress

   wordpress:
     depends_on:
       - db
     image: wordpress:latest
     ports:
       - "8000:80"
     restart: always
     environment:
       WORDPRESS_DB_HOST: db:3306
       WORDPRESS_DB_USER: wordpress
       WORDPRESS_DB_PASSWORD: wordpress
       WORDPRESS_DB_NAME: wordpress
volumes:
    db_data: {}

Start docker-compose

docker-compose up -d

Visit wordpress and install it

http://192.168.122.87:8000/

refer to

https://docs.docker.com/compose/wordpress/
https://github.com/docker-library/mysql
https://github.com/docker-library/wordpress

Kvm Windows时间异常问题

Wed, 16 Dec 2020 11:23:23 +0800

KVM xml配置

<clock offset='localtime'>
  <timer name='rtc' tickpolicy='catchup' track='guest'/>
  <timer name='pit' tickpolicy='delay'/>
  <timer name='hpet' present='no'/>
</clock>

or

<clock offset='timezone' timezone='Asia/Shanghai'/>

Windows服务器配置

bcdedit /set USEPLATFORMCLOCK on #设置
bcdedit /enum #查看是否生效
bcdedit / deletevalue useplatformclock #取消配置

Nginx 代理示例

Tue, 24 Nov 2020 10:22:53 +0800

stream代理

本地80代理到192.168.0.111:8081

stream {
    server {
        listen 80;
        proxy_pass shterm;
    }

    upstream shterm {
        server 192.168.0.111:8081;
    }

}

https ssl 代理示例

将本地443代理到test.mulinux.com:8080

#http {
#    upstream test.mulinux.com {
#        server test.mulinux.com:8080;
#   }

    server {
        listen      443 ssl;
        server_name test.mulinux.com;

        ssl_certificate        /mds/nginx/cert/server.crt;
        ssl_certificate_key    /mds/nginx/cert/server.key;
        #ssl_client_certificate /etc/ssl/certs/ca.crt;
        ssl_verify_client      off;

        location / {
            proxy_pass https://test.mulinux.com:8080/;
            proxy_redirect off;
        }
    }
}

proxy_pass + upstream

http {
   upstream mulinux_com {
       server blog.mulinux.com;
       server www.mulinux.com;
   }

   server {
        listen       8080;
        listen [::]:8081 ipv6only=on;
        listen       8080 ssl;
        listen [::]:8081 ssl ;
        server_name  localhost;

            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;


        ssl_certificate /usr/local/nginx/cert/mulinux.crt;
        ssl_certificate_key /usr/local/nginx/cert/mulinux.key;
        ssl_session_cache    shared:SSL:1m;
        ssl_session_timeout  5m;

        ssl_ciphers  HIGH:!aNULL:!MD5;
        ssl_prefer_server_ciphers  on;

        location / {
           proxy_pass http://mulinux_com;
        }
    }
}

参考: https://docs.nginx.com/nginx/admin-guide/security-controls/securing-http-traffic-upstream/