二环外的开阳里

一、环境说明

所有环境已提前以docker安装方式配置好服务

服务器	IP	服务
Prometheus-server	192.168.13.80	prometheus、node-exporter、grafana、alertmanagert
agent-centos7	192.168.13.81	docker、nginx、redis、mysql
agent-ubuntu22.04	192.168.13.82
agent-windows2019	192.168.13.83	mssql

二、Prometheus监控linux服务器

linux下安装node-export

1. 二进制形式安装

下载地址：官网下载

# 下载
wget https://objects.githubusercontent.com/github-production-release-asset-2e65be/9524057/a7e04f41-5543-40e2-9060-26fefe32bb4b?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240918%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240918T033247Z&X-Amz-Expires=300&X-Amz-Signature=00acdf0927397f4d03a82b2746743e77b8aa5f7d7764022d35223aa04759ab8d&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=9524057&response-content-disposition=attachment%3B%20filename%3Dnode_exporter-1.8.2.linux-amd64.tar.gz&response-content-type=application%2Foctet-stream
# 解压
tar -xf node_exporter-1.8.2.linux-amd64.tar.gz
# 移动到/opt/prometheus目录下
mv node_exporter-1.8.2.linux-amd64 /opt/prometheus/node_exporter

# 更改/opt/prometheus/alertmanager的权限
chown prometheus:prometheus -R /opt/prometheus

# 创建systemd服务，开机自启动
vi /usr/lib/systemd/system/node_exporter.service
[Unit]
Description=node_exporter
After=network.target
 
[Service]
User=prometheus
Group=prometheus

 
ExecStart=/opt/prometheus/node_exporter/node_exporter
 
Restart=on-failure

 
[Install]
WantedBy=multi-user.target

# 给权限
chown prometheus:prometheus -R /opt/prometheus

# 启动测试
systemctl daemon-reload
systemctl enable node_exporter --now
systemctl status node_exporter -l

#如有错误自行检查日志
journalctl -u node_exporter -f

# 端口监听9100
netstat -lnpt

应用	地址
node_exporter	http://192.168.13.81:9100/metrics

2. docker方式安装

mkdir -p /data/node-exporter
cd /data/node-exporter

cat >docker-compose.yaml <<"EOF"
version: '3.3'
services:
  node_exporter:
    image: prom/node-exporter:v1.5.0
    container_name: node_exporter
    restart: always 
    volumes:
        - /proc:/host/proc:ro
        - /sys:/host/sys:ro
        - /:/rootfs:ro
    command:
        - '--path.procfs=host/proc'
        - '--path.sysfs=host/sys'
        - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc|rootfs/var/lib/docker)($$|/)'
    ports:
        - '9100:9100'
EOF

# 注：Ubuntu系统会出现 yaml: line 8: found character that cannot start any token 报错
# yaml文件不能出现制表符，全部改为空格

3. 修改prometheus配置（注意文件格式）

# 在prometheus服务器上配置
vi /opt/prometheus/prometheus/prometheus.yml

  # node-exporter配置
  - job_name : 'node-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9100']
      labels:
        instance: Prometheus服务器
    # 在此添加一个target
    - targets: ['192.168.13.81:9100']
      labels:
        instance: agent-contos7

# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

ubuntu系统类似，自行安装并更改配置文件即可

三、Prometheus监控windows服务器

windows监控使用的采集器为：wmi-exporter

安装过程

1. 下载

下载地址：Releases · prometheus-community/windows_exporter (github.com)

下载64位msi程序

2. 安装

运行了之后，我们发现好像什么都没有发生，但实际上可能已经安装好了，我们查看服务

-- 这里推荐把启动类型，修改成自动（延迟启动）

-- 或者可以 cmd命令执行 sc config windows_exporter start= delayed-auto

我们还可以看看它的执行文件路径，里面就是它的启动命令了，默认端口是9182，需要开放防火墙端口

3. 核验 wmi_exporter 是否运行

访问http://192.168.13.83:9182/metrics

4. 修改prometheus配置

# 在prometheus服务器上配置
vi /opt/prometheus/prometheus/prometheus.yml

# node-exporter配置
  - job_name : 'node-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['localhost:9100']
      labels:
        instance: Prometheus服务器
    - targets: ['192.168.13.81:9100']
      labels:
        instance: agent-contos7
    - targets: ['192.168.13.82:9100']
      labels:
        instance: agent-ubuntu22.04
# 在此添加配置
  # windows-exporter配置
  - job_name : 'windows-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.13.83:9182']
      labels:
        instance: agent-windows2022

# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

四、Prometheus监控nginx

nginx监控使用的采集器为：nginx-exporter

安装过程

1. 环境说明

# docker安装nginx 卷映射
/data/nginx/conf:/etc/nginx/conf.d
/data/nginx/logs:/var/log/nginx
/data/nginx/html:/usr/share/nginx/html

# nginx需要开启stub_status配置 即需要with-http_stub_status_module模块
# 检查是否安装with-http_stub_status_module模块
docker exec -it nginx nginx -V 2>&1 | grep -o --with-http_stub_status_module

# nginx开启stub_status
server{
    ....
    location stub_status {
        stub_status on;
        access_log off;
        allow 0.0.0.0/0;
        deny all;
    }
    ....
}
# 重载nginx配置
docker exec -it nginx nginx -s reload
# 检查
curl http://192.168.13.81/stub_status

2. c

下载地址：Release v1.3.0 · nginxinc/nginx-prometheus-exporter · GitHub

在此使用docker安装演示，二进制安装直接从上述地址下载安装就可以了

# 创建docker-compose.yaml文件
mkdir /data/nginx-exporter
cd /data/nginx-exporter
vim docker-compose.yaml

version: '3.3'

services:
  nginx_exporter:
    image: nginx/nginx-prometheus-exporter:0.11
    container_name: nginx_exporter
    hostname: nginx_exporter
    command:
        - '-nginx.scrape-uri=http://192.168.13.81/stub_status'
    restart: always
    ports:
        - "9113:9113" 
 
# 启动容器
docker compose up -d
# 访问 http://192.168.13.81:9113/metrics 测试

3. 修改prometheus配置

# 在prometheus服务器上配置
vi /opt/prometheus/prometheus/prometheus.yml

# 末尾添加如下配置
# nginx-export配置
  - job_name : 'nginx-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.13.81:9113']
      labels:
        instance: agent-nginx


# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

五、Prometheus监控redis

1. 环境说明

redis:
    image: redis:latest
    container_name: redis
    ports:
      - "6379:6379"
    volumes:
      - /data/redis/data:/data
    networks:
      - app-network

2.下载并安装redis_exporter

下载地址：Releases · oliver006/redis_exporter · GitHub

在此使用docker安装演示，二进制安装直接从上述地址下载安装就可以了

# 创建docker-compose.yaml文件
mkdir /data/redis-exporter
cd /data/redis-exporter
vim docker-compose.yaml

version: '3.3'
services:
    redis-exporter:
        image: oliver006/redis_exporter
        container_name: redis-exporter
        ports:
          - "9121:9121"
        environment:
          REDIS_ADDR: "192.168.13.81:6379"
          REDIS_PASSWORD: 123456
        restart: always
 
# 启动容器
docker compose up -d
# 访问 http://192.168.13.81:9121/metrics 测试

3. 修改prometheus配置

# 在prometheus服务器上配置
vi /opt/prometheus/prometheus/prometheus.yml

# 末尾添加如下配置
# redis-export配置
  - job_name : 'redis-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.13.81:9121']
      labels:
        instance: agent-redis


# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

六、Prometheus监控mysql

1. 环境说明

mysql:
    image: mysql:latest
    container_name: mysql
    environment:
      MYSQL_ROOT_PASSWORD: 123456
    ports:
      - "3306:3306"
    volumes:
      - /data/mysql/data:/var/lib/mysql
    networks:
      - app-network

可选操作：创建单独的exporter用户用于监控，不用root

# 登录数据库
docker exec -it mysql mysql -u root -p

# 创建用户并赋权
mysql> CREATE USER 'exporter'@'%' IDENTIFIED BY '123456' WITH MAX_USER_CONNECTIONS 3;
mysql> GRANT PROCESS,REPLICATION CLIENT,SELECT on *.* TO 'exporter'@'%';

2.下载并安装mysqld_exporter

下载地址：GitHub - prometheus/mysqld_exporter: Exporter for MySQL server metrics

# 下载解压
wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.15.1/mysqld_exporter-0.15.1.linux-amd64.tar.gz
tar xf mysqld_exporter-0.15.1.linux-amd64.tar.gz
mv mysqld_exporter-0.15.1.linux-amd64 /data/mysqld-exporter

# 配置用于监控的配置文件
cd /data/mysqld_exporter
vim .my.cnf

[client]
host = 192.168.13.81
user = root
password = 123456

# 配置开机启动
vim /etc/systemd/system/mysqld-exporter.service 

[Unit]
Description=mysqld_exporter


[Service]
ExecStart=/data/mysqld_exporter/mysqld_exporter --config.my-cnf /data/mysqld_exporter/.my.cnf --web.listen-address=0.0.0.0:9104 \
--collect.slave_status \
--collect.binlog_size \
--collect.info_schema.processlist \
--collect.info_schema.innodb_metrics \
--collect.engine_innodb_status \
--collect.perf_schema.file_events \
--collect.perf_schema.replication_group_member_stats
Restart=on-failure

[Install]
WantedBy=multi-user.target


# 设置开机启动
systemctl daemon-reload
systemctl enable mysqld-exporter --now

# 访问 http://192.168.13.81:9104/metrics 测试

3. 修改prometheus配置

# 在prometheus服务器上配置
vi /opt/prometheus/prometheus/prometheus.yml

# 末尾添加如下配置
# redis-export配置
  - job_name : 'redis-exporter'
    scrape_interval: 15s
    static_configs:
    - targets: ['192.168.13.81:9121']
      labels:
        instance: agent-redis


# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

七、Prometheus监控docker

使用CAdvisor

CAdvisor是Google开源的一款用于展示和分析容器运行状态的可视化工具。通过在主机上运行CAdvisor用户可以轻松的获取到当前主机上容器的运行统计信息，并以图表的形式向用户展示

1. 安装CAdvisor

docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --detach=true --name=cadvisor google/cadvisor:latest
# 访问 http://192.168.13.81:8080/metrics 测试

2. 修改prometheus配置

# 在prometheus服务器上配置
vi /opt/prometheus/prometheus/prometheus.yml

# 末尾添加如下配置
# cadvisor配置
  - job_name: cadvisor
  static_configs:
  - targets:
    - 192.168.13.81:8080


# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

八、Prometheus黑盒监控

黑盒监控相较于白盒监控最大的不同在于黑盒监控是以故障为导向当故障发生时，黑盒监控能快速发现故障，而白盒监控则侧重于主动发

现或者预测潜在的问题。一个完善的监控目标是要能够从白盒的角度发现潜在问题，能够在黑盒的角度快速发现已经发生的问题。

1. 安装blackbox_exporter

# 下载安装
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.16.0/blackbox_exporter-0.16.0.linux-amd64.tar.gz
tar -xvf  blackbox_exporter-0.16.0.linux-amd64.tar.gz
mv blackbox_exporter-0.16.0.linux-amd64/ /data/redis_exporter

# 创建配置文件
cat > /data/black_exporter/black_exporter.yml << EOF
modules:
  http_2xx:
    prober: http
    timeout: 20s
    http:
      preferred_ip_protocol: "ip4"
  http_post_2xx_query:
    prober: http
    timeout: 20s
    http:
      preferred_ip_protocol: "ip4" ##使用ipv4
      method: POST
      headers:
        Content-Type: application/json ##header头
      body: '{"hmac":"","params":{"publicFundsKeyWords":"xxx"}}' ##传参
  tls_connect_tls:
    prober: tcp
    timeout: 5s
    tcp:
      tls: true
  tcp_connect:
    prober: tcp
    timeout: 5s
 #
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  irc_banner:
    prober: tcp
    tcp:
      query_response:
      - send: "NICK prober"
      - send: "USER prober prober prober :prober"
      - expect: "PING :([^ ]+)"
        send: "PONG ${1}"
      - expect: "^:[^ ]+ 001"
  icmp:
    prober: icmp
    timeout: 20s
EOF

# 配置systemd
cat > /etc/systemd/system/blackbox_exporter.service <<EOF
[Unit]
Description=blackbox_exporter
Documentation=https://prometheus.io/
After=network.target

[Service]
Type=simple
User=prometheus
ExecStart=/data/blackbox_exporter/blackbox_exporter  --config.file=/data/blackbox_exporter/blackbox.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

# 启动服务
systemctl enable blackbox_exporter.service 
systemctl start blackbox_exporter.service 
systemctl status blackbox_exporter.service 

# 配置Consul自动发现
cat > /data/consul/server1/config/blackbox-exporter.json <<EOF
{
    "service":{
        "name":"blackbox-exporter",
        "tags":[
            "prod",
            "blackbox-exporter"
        ],
        "address": "172.26.42.229",
        "port":9115,
        "check":{
            "http":"http://172.26.42.229:9115",
            "interval":"10s"
        },
        "token":"your_token"
    }
}
EOF
# 热加载Consul配置
consul reload
# Grafana导入Dashboard-ID：7587

2. 修改prometheus配置

 - job_name: 'blackbox'
    metrics_path: /probe
    params:
      # blackbox_exporter中定义的模块
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      # 监控的目标
      - targets:
        - http://example.com    # Target to probe with http.
          labels:
            app: cmoon
            dev: prod
        - https://example.com   # Target to probe with https.
          labels:
            app: cmoon-test
        - http://example.com:8080 # Target to probe with http on port 8080.
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        # blackbox_exporter服务的连接地址
        replacement: 192.168.13.80:9115  # The blackbox exporter's real hostname:port.


# 检查配置
cd /opt/prometheus/prometheus/
./promtool check config prometheus.yml
# 重载配置
curl -X POST http://localhost:9090/-/reload

Prometheus添加监控项

一、环境说明

二、Prometheus监控linux服务器

linux下安装node-export

1. 二进制形式安装

2. docker方式安装

3. 修改prometheus配置（注意文件格式）

三、Prometheus监控windows服务器

安装过程

1. 下载

2. 安装

3. 核验 wmi_exporter 是否运行

4. 修改prometheus配置

四、Prometheus监控nginx

安装过程

1. 环境说明

2. c

3. 修改prometheus配置

五、Prometheus监控redis

1. 环境说明

2.下载并安装redis_exporter

3. 修改prometheus配置

六、Prometheus监控mysql

1. 环境说明

2.下载并安装mysqld_exporter

3. 修改prometheus配置

七、Prometheus监控docker

使用CAdvisor

1. 安装CAdvisor

2. 修改prometheus配置

八、Prometheus黑盒监控

1. 安装blackbox_exporter

2. 修改prometheus配置