index.json

[{"content":"1. 概述 无意间发现了Hugo，尝试后发现效果不错，同时把过程记录下来，主要有以下几部分组成：\nHugo v0.106 hugo-PaperMod v6.0 Github Pages Github Actions 2. 实现过程 Hugo Hugo是由Go语言实现的静态网站生成器。简单、易用、高效、易扩展、快速部署。Hugo获取源文件和模板，并将它们作为输入来创建一个完整的html网站。\n二进制安装Hugo 到 Hugo Releases 下载对应的操作系统版本的Hugo二进制文件。\n生成站点 使用Hugo快速生成站点，初始文件都生成在目录里/path/to/site，配置文件格式为yaml。\nhugo new site /path/to/site --format yaml 自动生成文件夹内容如下：\n. ├── archetypes │ └── default.md ├── config.yaml ├── content ├── data ├── layouts ├── static └── themes 切换到站点目录\ncd /path/to/site 创建第一篇文章 posts文件夹会自动创建\nhugo new posts/about.md 自动生成文件到路径下content/posts/about.md，文件初始内容为：\n--- date: 2022-11-19T23:51:58+08:00 draft: false title: \u0026#34;about\u0026#34; --- # 在文末输入正文内容，以markdown格式 安装主题 到官网Hugo Themes选择心仪的主题。比如PaperMod，找到PaperMod Releases下载源码包。解压到themes目录下。\ncd themes wget https://github.com/adityatelange/hugo-PaperMod/archive/v6.0.zip unzip v6.0.zip # 将主题文件夹重命名为PaperMod ls PaperMod 启用主题 修改配置文件config.yaml\nbaseURL: \u0026#34;https://gnail89.github.io\u0026#34; # 站点URL languageCode: \u0026#34;en-us\u0026#34; # 语言 title: \u0026#34;📕W.\u0026#39;s Blog\u0026#34; # 站点名称 theme: \u0026#34;PaperMod\u0026#34; # 主题名，与themes目录下对应 paginate: 5 # 每页显示文章数量 运行Hugo 在站点根目录下执行Hugo命令进行调试。\nhugo server -D 在浏览器里打开预览：http://localhost:1313\n部署 生成静态页面，然后就可以部署到GitHub Pages，在站点根目录下执行。\n# (可选)清空public目录下的旧文件，全部重新生成。 hugo -F --cleanDestinationDir hugo 执行完成后，会在public目录下生成静态站点文件，全部复制到服务器即可。\nPaperMod主题设置 配置文件config.yaml示例参考PaperMod配置示例。\nHugo 主题配置参数大同小异，按喜好配置即可。\nbaseURL: \u0026#34;https://gnail89.github.io\u0026#34; languageCode: \u0026#34;en-us\u0026#34; title: \u0026#34;📕W.\u0026#39;s Blog\u0026#34; theme: \u0026#34;PaperMod\u0026#34; paginate: 5 enableInlineShortcodes: true enableRobotsTXT: true buildDrafts: false buildFuture: false buildExpired: false enableEmoji: true minify: disableXML: true # minifyOutput: true languages: en: languageName: \u0026#34;English\u0026#34; weight: 1 menu: main: - name: \u0026#34;Search\u0026#34; url: search/ weight: 10 - name: \u0026#34;Archive\u0026#34; url: archives/ weight: 20 - name: \u0026#34;Tags\u0026#34; url: tags/ weight: 30 - name: \u0026#34;Categories\u0026#34; url: categories/ weight: 40 outputs: home: - HTML - RSS - JSON params: env: production DateFormat: \u0026#34;2006-01-02\u0026#34; defaultTheme: auto disableThemeToggle: false ShowReadingTime: true ShowShareButtons: true ShowPostNavLinks: true ShowBreadCrumbs: true ShowCodeCopyButtons: true ShowWordCount: true ShowRssButtonInSectionTermList: true UseHugoToc: true disableSpecial1stPost: false disableScrollToTop: false comments: false hidemeta: false hideSummary: false showtoc: true tocopen: true # profile-mode profileMode: enabled: false title: \u0026#34;W.\u0026#39;s Blog\u0026#34; subtitle: \u0026#34;messages\u0026#34; imageUrl: \u0026#34;/images/logo.jpg\u0026#34; # imageWidth: 120 # imageHeight: 120 imageTitle: \u0026#34;image\u0026#34; buttons: - name: \u0026#34;Search\u0026#34; url: search - name: \u0026#34;Posts\u0026#34; url: posts - name: \u0026#34;Archive\u0026#34; url: archives # home-info mode homeInfoParams: Title: \u0026#34;W.\u0026#39;s Blog\u0026#34; Content: \u0026#34;messages\u0026#34; socialIcons: - name: github url: \u0026#34;https://github.com/\u0026#34; - name: email url: \u0026#34;mailto:user@localhost\u0026#34; - name: rss url: \u0026#34;index.xml\u0026#34; # cover: # hidden: true # hide everywhere but not in structured data # hiddenInList: true # hide on list pages and home # hiddenInSingle: true # hide on single page # for search # https://fusejs.io/api/options.html fuseOpts: isCaseSensitive: false shouldSort: true location: 0 distance: 1000 threshold: 0.4 minMatchCharLength: 0 keys: [\u0026#34;title\u0026#34;, \u0026#34;permalink\u0026#34;, \u0026#34;summary\u0026#34;, \u0026#34;content\u0026#34;] menu: main: - identifier: Search name: \u0026#34;Search\u0026#34; url: search/ weight: 10 - identifier: Archive name: \u0026#34;Archive\u0026#34; url: archives/ weight: 20 - identifier: Tags name: \u0026#34;Tags\u0026#34; url: tags/ weight: 30 - identifier: Categories name: \u0026#34;Categories\u0026#34; url: categories/ weight: 40 # Read: https://github.com/adityatelange/hugo-PaperMod/wiki/FAQs#using-hugos-syntax-highlighter-chroma pygmentsUseClasses: true markup: # goldmark: # renderer: # unsafe: true highlight: noClasses: false # anchorLineNos: true # codeFences: true # guessSyntax: true # lineNos: true # style: monokai Github Pages 必要的条件：\ngithub账户 创建Github Pages仓库 新建一个public仓库，并命名为username.github.io，其中username指github账户名。\nGitHub Pages 项目需要符合 username.github.io 的特殊命名格式，如果仓库名和账户名没有完全匹配的话，网站就不能运行，所以一定要保证完全匹配。 Github Actions 使用Github Actions是为了实现CI/CD，自动发布文章到网站，如果喜欢手动发布静态页面的也是可以的。\n必要的条件 准备一个存放Hogo源码的私有仓库 创建好的Github Pages仓库 主要流程是配置私有仓库的Github Actions和token\n配置私有仓库的Github Actions 在Hugo博客根目录下，创建Github Actions配置文件及目录，配置文件命名为gh-pages.yml，完整路径为.github/workflows/gh-pages.yml，这就是Github Action需要的一个工作流的配置，如果需要多个工作流可以创建多个yml文件。Hugo官方文档Build Hugo With GitHub Action有提供配置文件示例：\nname: github pages on: push: branches: - main # Set a branch that will trigger a deployment pull_request: jobs: deploy: runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v3 with: submodules: true # Fetch Hugo themes (true OR recursive) fetch-depth: 0 # Fetch all history for .GitInfo and .Lastmod - name: Setup Hugo uses: peaceiris/actions-hugo@v2 with: hugo-version: \u0026#39;latest\u0026#39; # extended: true - name: Build run: hugo --minify - name: Deploy uses: peaceiris/actions-gh-pages@v3 if: github.ref == \u0026#39;refs/heads/main\u0026#39; with: github_token: ${{ secrets.GITHUB_TOKEN }} publish_dir: ./public 整个工作流包括主要4个步骤：\n拉取代码 准备Hugo环境 编译生成静态文件 将静态文件发布出去 对于个人站点而言，示例配置文件已经足够满足需求，其中需要将token类型改为personal_token。\n示例：\nname: github pages on: push: branches: - main # Set a branch that will trigger a deployment pull_request: jobs: deploy: runs-on: ubuntu-22.04 steps: - uses: actions/checkout@v3 with: submodules: true # Fetch Hugo themes (true OR recursive) fetch-depth: 0 # Fetch all history for .GitInfo and .Lastmod - name: Setup Hugo uses: peaceiris/actions-hugo@v2 with: hugo-version: \u0026#39;latest\u0026#39; # extended: true - name: Build run: hugo --minify - name: Deploy uses: peaceiris/actions-gh-pages@v3 if: github.ref == \u0026#39;refs/heads/main\u0026#39; with: personal_token: ${{ secrets.PERSONAL_TOKEN }} # token name external_repository: username/username.github.io # 发布到Github Pages仓库 publish_branch: main # 指定分支 publish_dir: ./public # 将public目录发布出去 配置token 生成一个Personal access tokens\n登录GitHub，进入Settings-\u0026gt;Developer settings-\u0026gt;Personal access tokens页面。 点击Generate new token，在Note中输入名称，在Select scopes选择workflow 将生成的token保存下来，离开页面将无法查看。 进入需要配置工作流的仓库，进入Settings-\u0026gt;Secrets-\u0026gt;Actions页面，点击New repository secret按钮。 在页面中name部分输入在yml中设置的值PERSONAL_TOKEN，在Secret中输入token值，然后点击Add secret按钮。 3. 最终效果 在本地Hugo站点源码目录里新增/更新页面，文件格式为markdown，本地调试和预览：hugo server -D。 通过git push将源码推送到GitHub仓库。 触发GitHub Actions，按照workflow流程自动生成静态文件，然后推送的GitHub Pages仓库。 GitHub Pages仓库更新后，触发官方页面部署CI，最终实现自动发布。 ","permalink":"https://gnail89.github.io/posts/start-hugo/","summary":"1. 概述 无意间发现了Hugo，尝试后发现效果不错，同时把过程记录下来，主要有以下几部分组成：\nHugo v0.106 hugo-PaperMod v6.0 Github Pages Github Actions 2. 实现过程 Hugo Hugo是由Go语言实现的静态网站生成器。简单、易用、高效、易扩展、快速部署。Hugo获取源文件和模板，并将它们作为输入来创建一个完整的html网站。\n二进制安装Hugo 到 Hugo Releases 下载对应的操作系统版本的Hugo二进制文件。\n生成站点 使用Hugo快速生成站点，初始文件都生成在目录里/path/to/site，配置文件格式为yaml。\nhugo new site /path/to/site --format yaml 自动生成文件夹内容如下：\n. ├── archetypes │ └── default.md ├── config.yaml ├── content ├── data ├── layouts ├── static └── themes 切换到站点目录\ncd /path/to/site 创建第一篇文章 posts文件夹会自动创建\nhugo new posts/about.md 自动生成文件到路径下content/posts/about.md，文件初始内容为：\n--- date: 2022-11-19T23:51:58+08:00 draft: false title: \u0026#34;about\u0026#34; --- # 在文末输入正文内容，以markdown格式 安装主题 到官网Hugo Themes选择心仪的主题。比如PaperMod，找到PaperMod Releases下载源码包。解压到themes目录下。","title":"Start"},{"content":"基础安装 # node安装 # https://nodejs.org/en/download curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.5/install.sh | bash \\. \u0026#34;$HOME/.nvm/nvm.sh\u0026#34; nvm install 24 node -v npm -v # npm源 npm config set registry https://mirrors.huaweicloud.com/repository/npm/ # npm安装opencode # https://opencode.ai npm install -g opencode-ai # npm安装openspec # https://www.npmjs.com/package/@fission-ai/openspec npm install -g @fission-ai/openspec@latest ","permalink":"https://gnail89.github.io/posts/npm-install-opencode-openspec/","summary":"基础安装 # node安装 # https://nodejs.org/en/download curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.5/install.sh | bash \\. \u0026#34;$HOME/.nvm/nvm.sh\u0026#34; nvm install 24 node -v npm -v # npm源 npm config set registry https://mirrors.huaweicloud.com/repository/npm/ # npm安装opencode # https://opencode.ai npm install -g opencode-ai # npm安装openspec # https://www.npmjs.com/package/@fission-ai/openspec npm install -g @fission-ai/openspec@latest ","title":"npm install opencode \u0026 openspec"},{"content":"使用Dockerfile构建Zabbix Agent程序 创建Dockerfile文件 # 阶段 1: 构建环境 (Builder) FROM alpine:latest AS builder # 1. 设置镜像源并安装编译依赖 RUN sed -i \u0026#39;s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g\u0026#39; /etc/apk/repositories \u0026amp;\u0026amp; \\ apk add --no-cache \\ make automake autoconf gcc g++ libc-dev pkgconfig \\ tar linux-headers \\ pcre2-dev pcre2-static # 2. 复制源码包 COPY ./openssl-3.5.6.tar.gz /tmp/ COPY ./zabbix-7.0.25.tar.gz /tmp/ WORKDIR /tmp # 3. 编译 OpenSSL (静态库) RUN tar -zxvf openssl-3.5.6.tar.gz \u0026amp;\u0026amp; \\ cd openssl-3.5.6 \u0026amp;\u0026amp; \\ ./config --prefix=/opt/openssl no-shared no-zlib -static \u0026amp;\u0026amp; \\ make -j$(nproc) \u0026amp;\u0026amp; \\ make install_sw \u0026amp;\u0026amp; \\ cd /tmp \u0026amp;\u0026amp; rm -rf openssl-3.5.6 openssl-3.5.6.tar.gz # 4. 编译 Zabbix Agent (静态链接) RUN tar -zxvf zabbix-7.0.25.tar.gz \u0026amp;\u0026amp; \\ cd zabbix-7.0.25 \u0026amp;\u0026amp; \\ ./configure \\ --prefix=/opt/zabbix \\ --enable-agent \\ --enable-static \\ --with-libpcre2 \\ --enable-ipv6 \\ --with-openssl=/opt/openssl \\ CFLAGS=\u0026#34;-static -std=gnu99\u0026#34; \\ LDFLAGS=\u0026#34;-static\u0026#34; \u0026amp;\u0026amp; \\ make -j$(nproc) \u0026amp;\u0026amp; \\ make install \u0026amp;\u0026amp; \\ cd /tmp \u0026amp;\u0026amp; rm -rf zabbix-7.0.25 zabbix-7.0.25.tar.gz # 阶段 2: 运行环境 (Output) FROM alpine:latest AS output # 1. 设置镜像源 (用于安装验证工具) RUN sed -i \u0026#39;s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g\u0026#39; /etc/apk/repositories # 2. 从 builder 阶段复制编译好的二进制文件 COPY --from=builder /opt/zabbix /opt/zabbix # 3. 安装 file 工具用于验证静态链接状态 RUN apk add --no-cache file \u0026amp;\u0026amp; \\ echo \u0026#34;=== 文件类型验证 ===\u0026#34; \u0026amp;\u0026amp; \\ file /opt/zabbix/sbin/zabbix_agentd \u0026amp;\u0026amp; \\ echo -e \u0026#34;\\n=== 动态链接检查 (显示: not a dynamic executable) ===\u0026#34; \u0026amp;\u0026amp; \\ ldd /opt/zabbix/sbin/zabbix_agentd || true # 4. 设置默认启动命令 CMD [\u0026#34;/opt/zabbix/sbin/zabbix_agentd\u0026#34;, \u0026#34;--version\u0026#34;] 构建镜像 docker build -f Dockerfile -t zabbix-agent:latest . 运行镜像 docker run -it zabbix-agent:latest ","permalink":"https://gnail89.github.io/posts/build-agent-with-docker/","summary":"使用Dockerfile构建Zabbix Agent程序 创建Dockerfile文件 # 阶段 1: 构建环境 (Builder) FROM alpine:latest AS builder # 1. 设置镜像源并安装编译依赖 RUN sed -i \u0026#39;s/dl-cdn.alpinelinux.org/mirrors.ustc.edu.cn/g\u0026#39; /etc/apk/repositories \u0026amp;\u0026amp; \\ apk add --no-cache \\ make automake autoconf gcc g++ libc-dev pkgconfig \\ tar linux-headers \\ pcre2-dev pcre2-static # 2. 复制源码包 COPY ./openssl-3.5.6.tar.gz /tmp/ COPY ./zabbix-7.0.25.tar.gz /tmp/ WORKDIR /tmp # 3. 编译 OpenSSL (静态库) RUN tar -zxvf openssl-3.5.6.tar.gz \u0026amp;\u0026amp; \\ cd openssl-3.5.6 \u0026amp;\u0026amp; \\ ./config --prefix=/opt/openssl no-shared no-zlib -static \u0026amp;\u0026amp; \\ make -j$(nproc) \u0026amp;\u0026amp; \\ make install_sw \u0026amp;\u0026amp; \\ cd /tmp \u0026amp;\u0026amp; rm -rf openssl-3.","title":"Build Agent With Docker"},{"content":"ironic IPA 镜像构建案例 - 1 基础信息 类型 值 说明 构建环境 openEuler 24.03 (LTS-SP3) download Python Python 3.11.6 OS diskimage-builder 3.41.0 pip3 install ironic-python-agent-builder 7.2.0 pip3 install BaseOS IMAGE openEuler 24.03 (LTS-SP3) download ironic-python-agent stable/2025.2 source code requirements stable/2025.2 source code 环境配置 构建环境安装依赖包 openEuler 24.03 (LTS-SP3) # from https://opendev.org/openstack/diskimage-builder/raw/branch/master/bindep.txt dnf install -y binutils bzip2 util-linux dosfstools gdisk kpartx sudo curl git squashfs-tools qemu-img libguestfs policycoreutils-python-utils procps-ng yum yum-utils # other packages dnf install -y gcc make cmake net-tools rsyslog libaio* libnuma* glibc glibc-common glibc-devel* gcc-c++* libgcc* make* sysstat* telnet sg3_utils targetcli device-mapper-multipath sysfsutils fio iperf unzip nfs-utils gdisk lvm2 acpid qemu* 基础镜像下载 # 基础镜像中需要内置tar命令 wget https://dl-cdn.openeuler.openatom.cn/openEuler-24.03-LTS-SP3/virtual_machine_img/x86_64/openEuler-24.03-LTS-SP3-x86_64.qcow2.xz xz -d openEuler-24.03-LTS-SP3-x86_64.qcow2.xz pip 国内源配置 # 切换pip源 mkdir -p ~/.pip cat \u0026gt; ~/.pip/pip.conf \u0026lt;\u0026lt; EOF [global] index-url = https://mirrors.aliyun.com/pypi/simple trusted-host = mirrors.aliyun.com EOF 离线源码准备 openstack 2025.2 版本 # 拉取ipa源码，指定版本 git clone https://opendev.org/openstack/ironic-python-agent.git ./ironic-python-agent-src cd ironic-python-agent-src git checkout stable/2025.2 # 拉取requirements源码，指定版本 git clone https://opendev.org/openstack/requirements.git ./requirements-src cd requirements-src git checkout stable/2025.2 安装DIB和IPA builder pip3 install diskimage-builder pip3 install ronic-python-agent-builder diskimage-builder elements 改造 修改 diskimage_builder/elements/openeuler-minimal/element-deps # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/openeuler-minimal/element-deps cloud-init growroot openssh-server yum redhat-common 新增 diskimage_builder/elements/openeuler-minimal/root.d/10-openeuler-image # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/openeuler-minimal/root.d/10-openeuler-image # 参考centos元素中的配置 #!/bin/bash if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then set -x fi set -eu set -o pipefail DIB_LOCAL_IMAGE=${DIB_LOCAL_IMAGE:-} if [ -n \u0026#34;$DIB_LOCAL_IMAGE\u0026#34; ]; then IMAGE_LOCATION=$DIB_LOCAL_IMAGE # No need to copy a local image into the cache directory, so just specify # the cached path as the original path. CACHED_IMAGE=$IMAGE_LOCATION BASE_IMAGE_FILE=$(basename $DIB_LOCAL_IMAGE) BASE_IMAGE_TAR=$BASE_IMAGE_FILE.tgz fi echo \u0026#34;Extracting image...\u0026#34; $TMP_HOOKS_PATH/bin/extract-image $BASE_IMAGE_FILE $BASE_IMAGE_TAR $IMAGE_LOCATION $CACHED_IMAGE 修改 diskimage_builder/elements/dhcp-all-interfaces/pkg-map # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/dhcp-all-interfaces/pkg-map # 将 \u0026#34;ifupdown\u0026#34; 值改为 \u0026#34;NetworkManager\u0026#34; \u0026#34;family\u0026#34;: { \u0026#34;redhat\u0026#34;: { \u0026#34;dhcp-client\u0026#34;: \u0026#34;dhclient\u0026#34;, \u0026#34;ifupdown\u0026#34;: \u0026#34;NetworkManager\u0026#34;, \u0026#34;ndisc6\u0026#34;: \u0026#34;\u0026#34; }, 构建镜像 配置环境变量 # 指定元素路径 export ELEMENTS_PATH=/usr/local/share/ironic-python-agent-builder/dib # 指定目标镜像中的内核和ramdisk export DIB_BAREMETAL_INITRD_PATTERN=\u0026#34;initramfs-6.6.0-144.0.0.130.oe2403sp3.x86_64.img\u0026#34; export DIB_BAREMETAL_KERNEL_PATTERN=\u0026#34;vmlinuz-6.6.0-144.0.0.130.oe2403sp3.x86_64\u0026#34; # 开启debug信息 export DIB_DEBUG_TRACE=\u0026#34;1\u0026#34; # 指定python版本 # export DIB_PYTHON_VERSION=2 # 配置用户信息 export DIB_DEV_USER_PASSWORD=\u0026#34;openeuler\u0026#34; export DIB_DEV_USER_PWDLESS_SUDO=\u0026#34;YES\u0026#34; export DIB_DEV_USER_USERNAME=\u0026#34;openeuler\u0026#34; # 指定镜像源 export DIB_DISTRIBUTION_MIRROR=\u0026#34;https://mirror.lzu.edu.cn/openeuler\u0026#34; # 指定本地基础镜像 export DIB_LOCAL_IMAGE=\u0026#34;/root/openEuler-24.03-LTS-SP3-x86_64.qcow2\u0026#34; # 配置release信息 export DIB_RELEASE=\u0026#34;24.03-LTS-SP3\u0026#34; # 配置本地源码路径 # export DIB_REPOLOCATION_ironic_lib=\u0026#34;/root/ironic-lib-src\u0026#34; export DIB_REPOLOCATION_ironic_python_agent=\u0026#34;/root/ironic-python-agent-src\u0026#34; export DIB_REPOLOCATION_requirements=\u0026#34;/root/requirements-src\u0026#34; # 配置源码分支信息 # export DIB_REPOREF_ironic_lib=\u0026#34;stable/train\u0026#34; export DIB_REPOREF_ironic_python_agent=\u0026#34;stable/2025.2\u0026#34; export DIB_REPOREF_requirements=\u0026#34;stable/2025.2\u0026#34; 开始构建镜像 disk-image-create -a amd64 ironic-python-agent-ramdisk openeuler-minimal dynamic-login block-device-efi dhcp-all-interfaces devuser 镜像构建成功后，在当前目录下，生成文件：initramfs和kernel\nimage.initramfs image.kernel 以及image.d\n# ls -R image.d image.d: dib-manifests image.d/dib-manifests: dib_arguments dib_environment dib-manifest-git-image 镜像验证 启动虚拟机(x86_64) # qemu-kvm qemu-kvm \\ -enable-kvm \\ -m 4096 \\ -cpu host \\ -kernel image.kernel \\ -initrd image.initramfs \\ -append \u0026#34;root=/dev/ram0 console=ttyS0\u0026#34; \\ -nographic # libvirt virt-install \\ --name ipa-test \\ --memory 4096 \\ --vcpus 1 \\ --cpu host \\ --disk none \\ --boot kernel=/path/to/image.kernel,initrd=/path/to/image.initramfs,kernel_args=\u0026#34;root=/dev/ram0 console=ttyS0\u0026#34; \\ --graphics none \\ --console pty,target_type=serial ironic IPA 镜像构建案例 - 2 基础信息 类型 值 说明 构建环境 openEuler 24.03 (LTS-SP3) download Python Python 3.11.6 OS diskimage-builder 3.41.0 pip3 install ironic-python-agent-builder 2.7.0 pip3 install BaseOS IMAGE openEuler 22.03 LTS download ironic-python-agent stable/pike-eol source code ironic-lib stable/pike-eol source code requirements stable/pike-eol source code 环境配置 构建环境安装依赖包 openEuler 24.03 (LTS-SP3) # from https://opendev.org/openstack/diskimage-builder/raw/branch/master/bindep.txt dnf install -y binutils bzip2 util-linux dosfstools gdisk kpartx sudo curl git squashfs-tools qemu-img libguestfs policycoreutils-python-utils procps-ng yum yum-utils # other packages dnf install -y gcc make cmake net-tools rsyslog libaio* libnuma* glibc glibc-common glibc-devel* gcc-c++* libgcc* make* sysstat* telnet sg3_utils targetcli device-mapper-multipath sysfsutils fio iperf unzip nfs-utils gdisk lvm2 acpid qemu* 基础镜像下载 # 基础镜像中需要内置tar命令 wget https://dl-cdn.openeuler.openatom.cn/openEuler-22.03-LTS/virtual_machine_img/x86_64/openEuler-22.03-LTS-x86_64.qcow2.xz xz -d openEuler-22.03-LTS-x86_64.qcow2.xz pip 国内源配置 # 切换pip源 mkdir -p ~/.pip cat \u0026gt; ~/.pip/pip.conf \u0026lt;\u0026lt; EOF [global] index-url = https://mirrors.aliyun.com/pypi/simple trusted-host = mirrors.aliyun.com EOF 离线源码准备 openstack pike 版本 # 拉取ipa源码，指定版本 git clone https://opendev.org/openstack/ironic-python-agent.git ./ironic-python-agent-src cd ironic-python-agent-src # 获取所有tags信息 git fetch --all --tags # 检查目标tag名称 git tag -l | grep pike-eol # 从指定tag创建新分支 git checkout -b stable/pike tags/pike-eol # 验证新分支 git branch --list stable/pike git log -1 --oneline # 拉取requirements源码，指定版本 git clone https://opendev.org/openstack/requirements.git ./requirements-src cd requirements-src # 获取所有tags信息 git fetch --all --tags # 检查目标tag名称 git tag -l | grep pike-eol # 从指定tag创建新分支 git checkout -b stable/pike tags/pike-eol # 验证新分支 git branch --list stable/pike git log -1 --oneline # 拉取ironic-lib源码，指定版本 git clone https://opendev.org/openstack/ironic-lib.git ./ironic-lib-src cd ironic-lib-src # 获取所有tags信息 git fetch --all --tags # 检查目标tag名称 git tag -l | grep pike-eol # 从指定tag创建新分支 git checkout -b stable/pike tags/pike-eol # 验证新分支 git branch --list stable/pike git log -1 --oneline 安装DIB和IPA builder pip3 install diskimage-builder pip3 install ironic-python-agent-builder==2.7.0 diskimage-builder elements 改造 修改 diskimage_builder/elements/openeuler-minimal/element-deps # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/openeuler-minimal/element-deps cloud-init growroot openssh-server yum redhat-common 新增 diskimage_builder/elements/openeuler-minimal/root.d/10-openeuler-image # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/openeuler-minimal/root.d/10-openeuler-image # 参考centos元素中的配置 #!/bin/bash if [ ${DIB_DEBUG_TRACE:-0} -gt 0 ]; then set -x fi set -eu set -o pipefail DIB_LOCAL_IMAGE=${DIB_LOCAL_IMAGE:-} if [ -n \u0026#34;$DIB_LOCAL_IMAGE\u0026#34; ]; then IMAGE_LOCATION=$DIB_LOCAL_IMAGE # No need to copy a local image into the cache directory, so just specify # the cached path as the original path. CACHED_IMAGE=$IMAGE_LOCATION BASE_IMAGE_FILE=$(basename $DIB_LOCAL_IMAGE) BASE_IMAGE_TAR=$BASE_IMAGE_FILE.tgz fi echo \u0026#34;Extracting image...\u0026#34; $TMP_HOOKS_PATH/bin/extract-image $BASE_IMAGE_FILE $BASE_IMAGE_TAR $IMAGE_LOCATION $CACHED_IMAGE 修改 diskimage_builder/elements/dhcp-all-interfaces/pkg-map # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/dhcp-all-interfaces/pkg-map # 将 \u0026#34;ifupdown\u0026#34; 值改为 \u0026#34;NetworkManager\u0026#34; \u0026#34;family\u0026#34;: { \u0026#34;redhat\u0026#34;: { \u0026#34;dhcp-client\u0026#34;: \u0026#34;dhclient\u0026#34;, \u0026#34;ifupdown\u0026#34;: \u0026#34;NetworkManager\u0026#34;, \u0026#34;ndisc6\u0026#34;: \u0026#34;\u0026#34; }, 修改 diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip # cat /usr/local/lib/python3.11/site-packages/diskimage_builder/elements/pip-and-virtualenv/install.d/pip-and-virtualenv-source-install/04-install-pip # 245行后新增openeuler的支持 elif [[ $DISTRO_NAME = openeuler ]]; then python3 -m pip install pip pip3 install setuptools ironic-python-agent-builder elements 改造 修改 ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/pkg-map # cat /usr/local/share/ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/pkg-map # 新增对openEuler的支持 \u0026#34;family\u0026#34;: { # ...(skip)... \u0026#34;openeuler\u0026#34;: { \u0026#34;libffi-dev\u0026#34;: \u0026#34;libffi-devel\u0026#34;, \u0026#34;libssl-dev\u0026#34;: \u0026#34;openssl-devel\u0026#34;, \u0026#34;python\u0026#34;: \u0026#34;python3\u0026#34;, \u0026#34;python-dev\u0026#34;: \u0026#34;python3-devel\u0026#34; }, # ...(skip)... } 修改 ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/package-installs.yaml # cat /usr/local/share/ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/package-installs.yaml # 注释以下内容 #python-dev: # build-only: true 修改 ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/install.d/ironic-python-agent-ramdisk-source-install/60-ironic-python-agent-ramdisk-install # cat /usr/local/share/ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/install.d/ironic-python-agent-ramdisk-source-install/60-ironic-python-agent-ramdisk-install # 将IPA_PYTHON指向特定python版本 IPA_PYTHON=\u0026#34;/opt/python3.6/bin/python3\u0026#34; 增加 python 3.6 编译安装脚本: ironic-python-agent-builder/dib/ironic-python-agent-ramdisk/install.d/ironic-python-agent-ramdisk-source-install/50-python36-install # 示例 #!/bin/bash if [ \u0026#34;${DIB_DEBUG_TRACE:-0}\u0026#34; -gt 0 ]; then set -x fi set -eu set -o pipefail export LC_ALL=C.UTF-8 dnf install -y gcc make zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel wget tar gdbm-devel libdb-devel libnsl2-devel openssl-devel gdbm-devel libdb-devel wget https://www.python.org/ftp/python/3.6.15/Python-3.6.15.tgz tar -zxf Python-3.6.15.tgz \u0026amp;\u0026amp; cd Python-3.6.15 ./configure --prefix=/opt/python3.6 --enable-optimizations --with-ensurepip=install make \u0026amp;\u0026amp; make install chmod +x 50-python36-install 构建镜像 配置环境变量 # 指定元素路径 export ELEMENTS_PATH=/usr/local/share/ironic-python-agent-builder/dib # 指定目标镜像中的内核和ramdisk export DIB_BAREMETAL_INITRD_PATTERN=\u0026#34;initramfs-5.10.0-60.139.0.166.oe2203.x86_64.img\u0026#34; export DIB_BAREMETAL_KERNEL_PATTERN=\u0026#34;vmlinuz-5.10.0-60.139.0.166.oe2203.x86_64\u0026#34; # 开启debug信息 export DIB_DEBUG_TRACE=\u0026#34;1\u0026#34; # 指定python版本 export DIB_PYTHON_VERSION=3 # 配置用户信息 export DIB_DEV_USER_PASSWORD=\u0026#34;openeuler\u0026#34; export DIB_DEV_USER_PWDLESS_SUDO=\u0026#34;YES\u0026#34; export DIB_DEV_USER_USERNAME=\u0026#34;openeuler\u0026#34; # 指定镜像源 export DIB_DISTRIBUTION_MIRROR=\u0026#34;https://mirror.lzu.edu.cn/openeuler\u0026#34; # 指定本地基础镜像 export DIB_LOCAL_IMAGE=\u0026#34;/root/openEuler-22.03-LTS-x86_64.qcow2\u0026#34; # 配置release信息 export DIB_RELEASE=\u0026#34;22.03-LTS\u0026#34; # 配置本地源码路径 export DIB_REPOLOCATION_ironic_lib=\u0026#34;/root/ironic-lib-src\u0026#34; export DIB_REPOLOCATION_ironic_python_agent=\u0026#34;/root/ironic-python-agent-src\u0026#34; export DIB_REPOLOCATION_requirements=\u0026#34;/root/requirements-src\u0026#34; # 配置源码分支信息 export DIB_REPOREF_ironic_lib=\u0026#34;stable/pike\u0026#34; export DIB_REPOREF_ironic_python_agent=\u0026#34;stable/pike\u0026#34; export DIB_REPOREF_requirements=\u0026#34;stable/pike\u0026#34; 开始构建镜像 disk-image-create -a amd64 ironic-python-agent-ramdisk openeuler-minimal dynamic-login block-device-efi dhcp-all-interfaces devuser 镜像构建成功后，在当前目录下，生成文件：initramfs和kernel\nimage.initramfs image.kernel 以及image.d\n# ls -R image.d image.d: dib-manifests image.d/dib-manifests: dib_arguments dib_environment dib-manifest-git-image 镜像验证 启动虚拟机(x86_64) # qemu-kvm qemu-kvm \\ -enable-kvm \\ -m 4096 \\ -cpu host \\ -kernel image.kernel \\ -initrd image.initramfs \\ -append \u0026#34;root=/dev/ram0 console=ttyS0\u0026#34; \\ -nographic # libvirt virt-install \\ --name ipa-test \\ --memory 4096 \\ --vcpus 1 \\ --cpu host \\ --disk none \\ --boot kernel=/path/to/image.kernel,initrd=/path/to/image.initramfs,kernel_args=\u0026#34;root=/dev/ram0 console=ttyS0\u0026#34; \\ --graphics none \\ --console pty,target_type=serial ","permalink":"https://gnail89.github.io/posts/ironic-ipa-build/","summary":"ironic IPA 镜像构建案例 - 1 基础信息 类型 值 说明 构建环境 openEuler 24.03 (LTS-SP3) download Python Python 3.11.6 OS diskimage-builder 3.41.0 pip3 install ironic-python-agent-builder 7.2.0 pip3 install BaseOS IMAGE openEuler 24.03 (LTS-SP3) download ironic-python-agent stable/2025.2 source code requirements stable/2025.2 source code 环境配置 构建环境安装依赖包 openEuler 24.03 (LTS-SP3) # from https://opendev.org/openstack/diskimage-builder/raw/branch/master/bindep.txt dnf install -y binutils bzip2 util-linux dosfstools gdisk kpartx sudo curl git squashfs-tools qemu-img libguestfs policycoreutils-python-utils procps-ng yum yum-utils # other packages dnf install -y gcc make cmake net-tools rsyslog libaio* libnuma* glibc glibc-common glibc-devel* gcc-c++* libgcc* make* sysstat* telnet sg3_utils targetcli device-mapper-multipath sysfsutils fio iperf unzip nfs-utils gdisk lvm2 acpid qemu* 基础镜像下载 # 基础镜像中需要内置tar命令 wget https://dl-cdn.","title":"ironic IPA 镜像构建案例"},{"content":"VM installation for aarch64 on x86_64 host 案例一，centos 7运行aarch64虚拟机，完成ntp软件编译 1. 基础环境 基础操作系统: CentOS 7 x86_64 虚拟机操作系统: CentOS 7 aarch64 # 安装CentOS 7基础操作系统。并下载qemu源码包。 wget https://download.qemu.org/qemu-5.2.0.tar.xz #qemu源码 wget http://releases.linaro.org/components/kernel/uefi-linaro/16.02/release/qemu64/QEMU_EFI.fd #aarch64 UEFI引导程序 # 安装libvirt管理软件 yum install libvirt virt-manager systemctl start libvirtd 2. 编译qemu yum install Ninja-build gcc gcc-c++ bzip2 tar xvJf qemu-5.2.0.tar.xz cd qemu-5.2.0 ./configure --target-list=aarch64-softmmu make \u0026amp;\u0026amp; make install 3. 安装aarch64虚拟机 # 准备镜像：CentOS-7-aarch64-Minimal-2009.iso # 创建卷 qemu-img create -f qcow2 centos7_aarch64.qcow2 10G # 启动虚拟机 qemu-system-aarch64 -m 2048 -cpu cortex-a72 -smp 2 -M virt \\ -bios QEMU_EFI.fd -nographic \\ -drive if=none,file=centos7_aarch64.qcow2,id=hd0 \\ -device virtio-blk-device,drive=hd0 \\ -drive if=none,file=CentOS-7-aarch64-Minimal-2009.iso,id=cdrom,media=cdrom \\ -device virtio-scsi-device \\ -device scsi-cd,drive=cdrom \\ -net nic -net tap,ifname=tap0,script=no,downscript=no 4. 启动虚拟机 qemu-system-aarch64 -m 2048 -cpu cortex-a72 -smp 2 -M virt \\ -bios QEMU_EFI.fd -nographic \\ -drive if=none,file=centos7_aarch64.qcow2,id=hd0 \\ -device virtio-blk-device,drive=hd0 \\ -device virtio-scsi-device \\ -net nic -net tap,ifname=tap0,script=no,downscript=no 5. 创建透明网桥 # 将虚拟机网卡tap0加入br0，以实现网络互通 brctl addbr br0 brctl addif br0 eth0 brctl addif br0 tap0 ifup tap0 添加网桥：brctl addbr br0 删除网桥：brctl delbr br0 显示网桥信息：brctl show 将eth0端口加入网桥br0：brctl addif br0 eth0 从网桥br0中删除eth0端口：brctl delif br0 eth0 6. 编译ntp软件 wget http://archive.ntp.org/ntp4/ntp-4.2/ntp-4.2.8p15.tar.gz tar zxf ntp-4.2.8p15.tar.gz cd ntp-4.2.8p15 ./configure --host=aarch64-linux-gnu --prefix=/opt/ntp-4.2.8p15 --enable-static make \u0026amp;\u0026amp; make install 案例二，OpenEuler 22.03运行aarch64虚拟机，zabbix agent编译 1. 基础环境 基础操作系统: OpenEuler 22.03 x86_64 虚拟机操作系统: alpine 3.23.3 aarch64 # 安装依赖 yum install libvirt qemu-system-aarch64 # 安装UEFI引导程序 wget https://dl-cdn.openeuler.openatom.cn/openEuler-22.03-LTS-SP3/everything/aarch64/Packages/edk2-aarch64-202011-13.oe2203sp3.noarch.rpm # 解压包 rpm2cpio edk2-aarch64-202011-13.oe2203sp3.noarch.rpm | cpio -idmv # 使用其中的QEMU_EFI-pflash.raw和vars-template-pflash.raw固件 2. 安装虚拟机 # 创建卷 qemu-img create -f qcow2 /var/lib/libvirt/images/alpine-aarch64.qcow2 10G # 通过virt-install启动虚拟机 virt-install \\ --name alpine-aarch64 \\ --arch aarch64 \\ --machine virt \\ --vcpus 2 \\ --cpu cortex-a72 \\ --memory 1024 \\ --os-variant alpinelinux3.12 \\ --disk path=/var/lib/libvirt/images/alpine-aarch64.qcow2,format=qcow2,size=10 \\ --cdrom /var/lib/libvirt/images/alpine-standard-3.23.3-aarch64.iso \\ --network network=default \\ --boot loader=/var/lib/libvirt/images/QEMU_EFI-pflash.raw,loader_ro=yes,loader_type=pflash,nvram_template=/var/lib/libvirt/images/vars-template-pflash.raw \\ --graphics vnc,listen=0.0.0.0 --noautoconsole # 连接控制台执行安装 virsh console alpine-aarch64 # 停止虚拟机 virsh destroy alpine-aarch64 # 删除虚拟机 virsh undefine --nvram alpine-aarch64 3. 编译软件 基础配置 # 替换为国内软件源 echo \u0026#34;https://mirrors.ustc.edu.cn/alpine/v3.23/main https://mirrors.ustc.edu.cn/alpine/v3.23/community\u0026#34; \u0026gt; /etc/apk/repositories # 安装依赖软件 apk add --no-cache \\ make automake autoconf gcc libc-dev pkgconfig \\ tar g++ linux-headers \\ pcre2-dev pcre2-static 编译openssl tar -zxvf openssl-1.1.1w.tar.gz \u0026amp;\u0026amp; \\ cd openssl-1.1.1w \u0026amp;\u0026amp; \\ ./config --prefix=/opt/openssl no-shared no-zlib -static \u0026amp;\u0026amp; \\ make -j$(nproc) \u0026amp;\u0026amp; \\ make install 编译zabbix agent tar -zxvf zabbix-7.0.22.tar.gz cd zabbix-7.0.22 ./configure \\ --prefix=/opt/zabbix \\ --enable-agent \\ --enable-static \\ --with-libpcre2 \\ --enable-ipv6 \\ --with-openssl=/opt/openssl \\ CFLAGS=\u0026#34;-static -std=gnu99\u0026#34; \\ LDFLAGS=\u0026#34;-static\u0026#34; make -j$(nproc) \u0026amp;\u0026amp; make install ","permalink":"https://gnail89.github.io/posts/vm_installation_for_aarch64_on_x86_64_host/","summary":"VM installation for aarch64 on x86_64 host 案例一，centos 7运行aarch64虚拟机，完成ntp软件编译 1. 基础环境 基础操作系统: CentOS 7 x86_64 虚拟机操作系统: CentOS 7 aarch64 # 安装CentOS 7基础操作系统。并下载qemu源码包。 wget https://download.qemu.org/qemu-5.2.0.tar.xz #qemu源码 wget http://releases.linaro.org/components/kernel/uefi-linaro/16.02/release/qemu64/QEMU_EFI.fd #aarch64 UEFI引导程序 # 安装libvirt管理软件 yum install libvirt virt-manager systemctl start libvirtd 2. 编译qemu yum install Ninja-build gcc gcc-c++ bzip2 tar xvJf qemu-5.2.0.tar.xz cd qemu-5.2.0 ./configure --target-list=aarch64-softmmu make \u0026amp;\u0026amp; make install 3. 安装aarch64虚拟机 # 准备镜像：CentOS-7-aarch64-Minimal-2009.iso # 创建卷 qemu-img create -f qcow2 centos7_aarch64.qcow2 10G # 启动虚拟机 qemu-system-aarch64 -m 2048 -cpu cortex-a72 -smp 2 -M virt \\ -bios QEMU_EFI.","title":"VM installation for aarch64 on x86_64 host"},{"content":"自签名私有证书制作 V1 版本 ssl.key（私钥） ssl.crt（证书） dhparam.pem（DH 参数）文件 # 生成私钥文件 (ssl.key) openssl genrsa -out ssl.key 2048 # 生成证书签名请求 (CSR) openssl req -new -key ssl.key -out ssl.csr # 生成自签名证书 (ssl.crt) openssl x509 -req -days 365 -in ssl.csr -signkey ssl.key -out ssl.crt #生成 DH 参数文件 (dhparam.pem) openssl dhparam -out dhparam.pem 2048 # 验证私钥 openssl rsa -in ssl.key -check # 查看证书详细信息 openssl x509 -in ssl.crt -text -noout # 验证私钥和证书是否匹配, 如果两个哈希值相同，则私钥和证书匹配 openssl pkey -pubout -in ssl.key | openssl sha256 openssl x509 -pubkey -in ssl.crt -noout | openssl sha256 # 检查证书有效期 openssl x509 -in ssl.crt -noout -dates # 验证DH参数 openssl dhparam -in dhparam.pem -check V3 版本 ssl.key（私钥） ssl.crt（证书） dhparam.pem（DH 参数）文件 # 创建 OpenSSL 配置文件 (ssl_v3.cnf) cat \u0026gt; ssl_v3.cnf \u0026lt;\u0026lt; \u0026#34;EOF\u0026#34; [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = v3_req x509_extensions = v3_ca prompt = no default_md = sha256 [ req_distinguished_name ] C = CN ST = State L = City O = Organization OU = Organizational Unit CN = example.com [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = @alt_names [ v3_ca ] basicConstraints = critical,CA:TRUE subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always keyUsage = critical, digitalSignature, keyCertSign [ alt_names ] DNS.1 = localhost DNS.2 = example.com DNS.3 = www.example.com IP.1 = 127.0.0.1 IP.2 = ::1 EOF # 生成私钥文件 (ssl.key) openssl genrsa -out ssl.key 2048 # 生成证书签名请求 (CSR) openssl req -new -key ssl.key -out ssl.csr -config ssl_v3.cnf # 生成自签名证书 (ssl.crt) (弃用扩展alt_names) openssl x509 -req -in ssl.csr -signkey ssl.key -out ssl.crt -days 365 -sha256 -extfile ssl_v3.cnf # 生成自签名证书 (ssl.crt) (引入扩展alt_names) openssl x509 -req -in ssl.csr -signkey ssl.key -out ssl.crt -days 365 -sha256 -extfile ssl_v3.cnf -extensions v3_req #生成 DH 参数文件 (dhparam.pem) openssl dhparam -out dhparam.pem 2048 # 查看证书详细信息 openssl x509 -in ssl.crt -text -noout 测试实例 # 1. 创建 OpenSSL 配置文件 (ssl_v3.cnf) cat \u0026gt; ssl_v3.cnf \u0026lt;\u0026lt; \u0026#34;EOF\u0026#34; [ req ] default_bits = 2048 distinguished_name = req_distinguished_name req_extensions = v3_req x509_extensions = v3_ca prompt = no default_md = sha256 [ req_distinguished_name ] C = CN ST = State L = City O = Organization OU = Organizational Unit CN = example.com [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth, clientAuth subjectAltName = @alt_names [ v3_ca ] basicConstraints = critical,CA:TRUE subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer:always keyUsage = critical, digitalSignature, keyCertSign [ alt_names ] DNS.1 = localhost DNS.2 = example.com DNS.3 = www.example.com IP.1 = 127.0.0.1 IP.2 = ::1 EOF # 2. 生成CA私钥文件 openssl genrsa -out ca.key 2048 # 3. 生成CA证书签名请求 openssl req -new -key ca.key -out ca.csr # 4. 生成CA自签名证书 openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt -days 3650 -sha256 -extfile ssl_v3.cnf -extensions v3_ca # 5. 生成服务器私钥文件 openssl genrsa -out ssl.key 2048 # 6. 生成服务器证书签名请求 openssl req -new -key ssl.key -out ssl.csr -config ssl_v3.cnf -extensions v3_req # 7. 签名服务器证书 openssl x509 -req -in ssl.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ssl.crt -days 365 -sha256 -extfile ssl_v3.cnf -extensions v3_req # 8. 生成 DH 参数文件 (dhparam.pem) openssl dhparam -out dhparam.pem 2048 # 9. 查看证书详细信息 openssl x509 -in ssl.crt -text -noout ","permalink":"https://gnail89.github.io/posts/create_ssl_private_certificate/","summary":"自签名私有证书制作 V1 版本 ssl.key（私钥） ssl.crt（证书） dhparam.pem（DH 参数）文件 # 生成私钥文件 (ssl.key) openssl genrsa -out ssl.key 2048 # 生成证书签名请求 (CSR) openssl req -new -key ssl.key -out ssl.csr # 生成自签名证书 (ssl.crt) openssl x509 -req -days 365 -in ssl.csr -signkey ssl.key -out ssl.crt #生成 DH 参数文件 (dhparam.pem) openssl dhparam -out dhparam.pem 2048 # 验证私钥 openssl rsa -in ssl.key -check # 查看证书详细信息 openssl x509 -in ssl.crt -text -noout # 验证私钥和证书是否匹配, 如果两个哈希值相同，则私钥和证书匹配 openssl pkey -pubout -in ssl.","title":"Create SSL private certificate"},{"content":"引导虚拟机实例简单流程(AI) #!/bin/bash set -Eeuo pipefail basepath=$(cd `dirname $0`;pwd) readonly VM_FORMAT_FILE=\u0026#34;vm_info_format.txt\u0026#34; readonly HYPERVISOR_FILE=\u0026#34;env_available_host_list.txt\u0026#34; readonly VOL_TYPE_FILE=\u0026#34;env_cinder_volume_type.txt\u0026#34; readonly IMAGES_FILE=\u0026#34;env_image_list.txt\u0026#34; readonly DATA_SEPARATOR=\u0026#34;+\u0026#34; readonly MAX_PARALLEL=2 log() { local lvl=\u0026#34;$1\u0026#34; msg=\u0026#34;$2\u0026#34; printf \u0026#34;%s [%4s] %s\\n\u0026#34; \u0026#34;$(date +\u0026#39;%F %T\u0026#39;)\u0026#34; \u0026#34;$lvl\u0026#34; \u0026#34;$msg\u0026#34; \u0026gt;\u0026amp;2 [[ \u0026#34;$lvl\u0026#34; == \u0026#34;ERROR\u0026#34; ]] \u0026amp;\u0026amp; exit 1 return 0 } trap \u0026#39;log ERROR \u0026#34;Script aborted in $FUNCNAME at line $LINENO\u0026#34;\u0026#39; ERR require_cmd() { command -v \u0026#34;$1\u0026#34; \u0026amp;\u0026gt;/dev/null || log ERROR \u0026#34;缺少命令: $1\u0026#34;;} init_deps() { for cmd in openstack nova awk grep sed tee; do require_cmd \u0026#34;$cmd\u0026#34;; done; } check_volumes(){ local retries=$1 count=0 vol_ids=(\u0026#34;${@:2}\u0026#34;) for ((i=0; i\u0026lt;retries; i++)); do count=0 for vid in \u0026#34;${vol_ids[@]}\u0026#34;; do [[ -z \u0026#34;$vid\u0026#34; || \u0026#34;$vid\u0026#34; == \u0026#34;None\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;无效的卷ID\u0026#34; local st; st=$(openstack volume show -f value --column status \u0026#34;$vid\u0026#34;) case \u0026#34;$st\u0026#34; in available) count=$((count+1)) ;; error*) log ERROR \u0026#34;卷状态显示error, ID: $vid\u0026#34; ;; esac done [[ $count -eq ${#vol_ids[@]} ]] \u0026amp;\u0026amp; { echo ok; return 0; } sleep 30 done echo \u0026#34;None\u0026#34; return 0 } export_volume_types(){ \u0026gt; \u0026#34;$1\u0026#34; log INFO \u0026#34;正在导出卷类型列表\u0026#34; openstack volume type list -f value --column Name | grep -Ev \u0026#39;^(#|$)\u0026#39; \u0026gt; \u0026#34;$1\u0026#34; || echo \u0026#34;None\u0026#34; \u0026gt; \u0026#34;$1\u0026#34; } export_hypervisors(){ \u0026gt; \u0026#34;$1\u0026#34; log INFO \u0026#34;正在导出可用的宿主机列表\u0026#34; openstack compute service list -f value --service nova-compute \\ | awk \u0026#39;$5 == \u0026#34;enabled\u0026#34; \u0026amp;\u0026amp; $6 == \u0026#34;up\u0026#34; {print $4\u0026#34;:\u0026#34;$3}\u0026#39; | grep -Ev \u0026#39;^(#|$)\u0026#39; \u0026gt; \u0026#34;$1\u0026#34; || echo \u0026#34;None\u0026#34; \u0026gt; \u0026#34;$1\u0026#34; } export_images(){ \u0026gt; \u0026#34;$1\u0026#34; log INFO \u0026#34;正在导出镜像列表\u0026#34; openstack image list -f value --column Name | grep -Ev \u0026#39;^(#|$)\u0026#39; \u0026gt; \u0026#34;$1\u0026#34; || echo \u0026#34;None\u0026#34; \u0026gt; \u0026#34;$1\u0026#34; } create_boot_volume(){ local name=$1 size=$2 image=$3 type=$4 [[ -z \u0026#34;$name\u0026#34; || -z \u0026#34;$size\u0026#34; || -z \u0026#34;$image\u0026#34; || -z \u0026#34;$type\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;缺少卷创建参数\u0026#34; local vol_name=\u0026#34;${name}-sysvol\u0026#34; log INFO \u0026#34;创建启动卷: $vol_name\u0026#34; local opts=(--size \u0026#34;$size\u0026#34; --image \u0026#34;$(openstack image show -f value --column id \u0026#34;$image\u0026#34;)\u0026#34;) [[ \u0026#34;$type\u0026#34; != \u0026#34;None\u0026#34; ]] \u0026amp;\u0026amp; opts+=(--type \u0026#34;$type\u0026#34;) local id; id=$(openstack volume create -f value --column id \u0026#34;${opts[@]}\u0026#34; \u0026#34;$vol_name\u0026#34;) log INFO \u0026#34;启动卷已创建, ID: $id\u0026#34; echo \u0026#34;$id\u0026#34; return 0 } create_data_vols(){ local name=$1 size_list=$2 type=$3 [[ -z \u0026#34;$name\u0026#34; || -z \u0026#34;$size_list\u0026#34; || -z \u0026#34;$type\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;缺少卷创建参数\u0026#34; IFS=\u0026#34;$DATA_SEPARATOR\u0026#34; read -r -a sizes \u0026lt;\u0026lt;\u0026lt;\u0026#34;$size_list\u0026#34; for s in \u0026#34;${sizes[@]}\u0026#34;; do [[ $s -le 0 ]] \u0026amp;\u0026amp; continue local vol_name=\u0026#34;${name}-datavol-${s}\u0026#34; log INFO \u0026#34;创建数据卷: $vol_name\u0026#34; local opts=(--size \u0026#34;$s\u0026#34;) [[ \u0026#34;$type\u0026#34; != \u0026#34;None\u0026#34; ]] \u0026amp;\u0026amp; opts+=(--type \u0026#34;$type\u0026#34;) local id; id=$(openstack volume create -f value --column id \u0026#34;${opts[@]}\u0026#34; \u0026#34;$vol_name\u0026#34;) log INFO \u0026#34;数据卷已创建, ID: $id\u0026#34; echo \u0026#34;$id\u0026#34; done return 0 } boot_instance(){ local name=$1 ip=$2 flavor=$3 net=$4 zone=$5; shift 5 local vol_ids=(\u0026#34;$@\u0026#34;) bdevs=() n=0 [[ -z \u0026#34;$name\u0026#34; || -z \u0026#34;$ip\u0026#34; || -z \u0026#34;$flavor\u0026#34; || -z \u0026#34;$net\u0026#34; || -z \u0026#34;$zone\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;缺少启动参数\u0026#34; for vid in \u0026#34;${vol_ids[@]}\u0026#34;; do bdevs+=(--block-device source=volume,id=$vid,dest=volume,shutdown=preserve,bootindex=$n) n=$((n+1)) done log INFO \u0026#34;启动虚拟机: $name\u0026#34; log INFO \u0026#34;虚拟机启动指令: $(echo \u0026#34;nova boot --flavor $flavor ${bdevs[@]} --nic net-id=$net,v4-fixed-ip=$ip --availability-zone $zone $name\u0026#34;)\u0026#34; if [ $(openstack server list --ip \u0026#34;${ip}$\u0026#34; -f value 2\u0026gt;/dev/null | wc -l) -ne 0 ]; then log INFO \u0026#34;虚拟机IP已使用, 忽略启动\u0026#34; return 0 fi nova boot --flavor \u0026#34;$flavor\u0026#34; \u0026#34;${bdevs[@]}\u0026#34; \\ --nic net-id=\u0026#34;$net\u0026#34;,v4-fixed-ip=\u0026#34;$ip\u0026#34; \\ --availability-zone \u0026#34;$zone\u0026#34; \u0026#34;$name\u0026#34; return 0 } process_vm_line(){ local line=$1 local vol_ids=() IFS=\u0026#39;,\u0026#39; read -r image cpu_mem sys data name vlan ip zone voltype \u0026lt;\u0026lt;\u0026lt;\u0026#34;$line\u0026#34; vol_ids+=( $(create_boot_volume \u0026#34;$name\u0026#34; \u0026#34;$sys\u0026#34; \u0026#34;$image\u0026#34; \u0026#34;${voltype:-None}\u0026#34;) ) vol_ids+=( $(create_data_vols \u0026#34;$name\u0026#34; \u0026#34;$data\u0026#34; \u0026#34;${voltype:-None}\u0026#34;) ) [[ \u0026#34;$(check_volumes 30 \u0026#34;${vol_ids[@]}\u0026#34;)\u0026#34; == \u0026#34;ok\u0026#34; ]] \u0026amp;\u0026amp; \\ boot_instance \u0026#34;$name\u0026#34; \u0026#34;$ip\u0026#34; \u0026#34;$cpu_mem\u0026#34; \u0026#34;$(openstack network list -f csv --column ID --column Name |grep -w \u0026#34;${vlan}\u0026#34; |awk -F\u0026#39;\\\u0026#34;\u0026#39; \u0026#39;{print $2}\u0026#39;)\u0026#34; \u0026#34;$zone\u0026#34; \u0026#34;${vol_ids[@]}\u0026#34; return 0 } main_vm_task(){ local infile=$1 log INFO \u0026#34;开始并发创建虚拟机流程\u0026#34; local fifo_file=\u0026#34;/tmp/$$.vm_fifo\u0026#34; mkfifo \u0026#34;$fifo_file\u0026#34; exec 9\u0026lt;\u0026gt;\u0026#34;$fifo_file\u0026#34; rm -f \u0026#34;$fifo_file\u0026#34; for ((i=0;i\u0026lt;MAX_PARALLEL;i++)); do echo \u0026gt;\u0026amp;9; done while IFS= read -r line; do [[ -z $line || ${line:0:1} == \u0026#39;#\u0026#39; ]] \u0026amp;\u0026amp; continue read -t 1000 -u9 { process_vm_line \u0026#34;$line\u0026#34; echo \u0026gt;\u0026amp;9 } \u0026amp; done \u0026lt; \u0026#34;$infile\u0026#34; wait exec 9\u0026gt;\u0026amp;- log INFO \u0026#34;所有虚拟机创建流程已执行完成\u0026#34; return 0 } get_item_by_ip() { local ip=\u0026#34;$1\u0026#34; file=\u0026#34;$2\u0026#34; if [ ! -r \u0026#34;$file\u0026#34; ]; then log WARN \u0026#34;无法访问文件: $file\u0026#34; echo -n \u0026#34;None\u0026#34; return 0 fi local valid_items=$(awk \u0026#39;!/^#|^$/ {print $1}\u0026#39; \u0026#34;$file\u0026#34;) local item_count=$(echo \u0026#34;$valid_items\u0026#34; | wc -l) if [ \u0026#34;$item_count\u0026#34; -gt 0 ]; then local index=$(( (${ip##*.} % item_count) + 1 )) echo -n \u0026#34;$(echo \u0026#34;$valid_items\u0026#34; | sed -n \u0026#34;${index}p\u0026#34;)\u0026#34; else echo -n \u0026#34;None\u0026#34; fi return 0 } trim(){ local str=\u0026#34;$1\u0026#34; echo \u0026#34;$str\u0026#34; | sed \u0026#39;s/^[[:space:]]*//;s/[[:space:]]*$//\u0026#39; } main_build_task(){ local file=\u0026#34;$1\u0026#34; zone=\u0026#34;None\u0026#34; voltype=\u0026#34;None\u0026#34; [[ ! -f \u0026#34;$file\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;读取虚拟机资产信息失败：$file\u0026#34; echo \u0026#34;#镜像名称, 规格名称, 系统盘大小, 数据盘大小, 虚拟机名称, VLAN名称, IP地址, 宿主机, 卷类型\u0026#34; \u0026gt; \u0026#34;$VM_FORMAT_FILE\u0026#34; while IFS=\u0026#39;,\u0026#39; read -r -a fields ; do [[ \u0026#34;${#fields[@]}\u0026#34; -eq 0 || \u0026#34;${fields[0]}\u0026#34; =~ ^(#|机房和平台) ]] \u0026amp;\u0026amp; continue [[ ${#fields[@]} -lt 14 ]] \u0026amp;\u0026amp; continue img=$(trim \u0026#34;${fields[6]}\u0026#34;) cpu=$(trim \u0026#34;${fields[7]}\u0026#34;) mem=$(trim \u0026#34;${fields[8]}\u0026#34;) sys=$(trim \u0026#34;${fields[9]}\u0026#34;) data=$(trim \u0026#34;${fields[10]}\u0026#34;) name=$(trim \u0026#34;${fields[11]}\u0026#34;) net=$(trim \u0026#34;${fields[12]}\u0026#34;) vmip=$(trim \u0026#34;${fields[13]}\u0026#34;) [[ -z \u0026#34;$img\u0026#34; || -z \u0026#34;$cpu\u0026#34; || -z \u0026#34;$mem\u0026#34; || -z \u0026#34;$sys\u0026#34; || -z \u0026#34;$data\u0026#34; || -z \u0026#34;$name\u0026#34; || -z \u0026#34;$net\u0026#34; || -z \u0026#34;$vmip\u0026#34; ]] \u0026amp;\u0026amp; continue flavor=\u0026#34;${cpu}C${mem}G${sys}G\u0026#34; if [[ -n \u0026#34;$vmip\u0026#34; ]]; then zone=$(get_item_by_ip \u0026#34;$vmip\u0026#34; \u0026#34;$HYPERVISOR_FILE\u0026#34;) voltype=$(get_item_by_ip \u0026#34;$vmip\u0026#34; \u0026#34;$VOL_TYPE_FILE\u0026#34;) fi if [ \u0026#34;$(grep -c \u0026#34;$img\u0026#34; \u0026#34;$IMAGES_FILE\u0026#34;)\u0026#34; -eq 1 ]; then img=$(grep \u0026#34;$img\u0026#34; \u0026#34;$IMAGES_FILE\u0026#34;) else log WARN \u0026#34;镜像: $img 不存在, 已忽略\u0026#34; fi echo \u0026#34;$img,$flavor,$sys,$data,$name,$net,$vmip,$zone,$voltype\u0026#34; | tee -a \u0026#34;$VM_FORMAT_FILE\u0026#34; done \u0026lt; \u0026#34;$file\u0026#34; log INFO \u0026#34;---- 【重要】请核查配置文件: ${VM_FORMAT_FILE}, 并修正错误 ----\u0026#34; return 0 } main_review_task(){ local file=\u0026#34;$1\u0026#34; [[ ! -f \u0026#34;$file\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;读取配置文件失败：$file\u0026#34; local cluster_images=($(openstack image list -f value -c Name)) local cluster_flavors=($(openstack flavor list -f value -c Name)) local cluster_vlans=($(openstack network list -f csv | awk -F\u0026#39;\\\u0026#34;\u0026#39; \u0026#39;{print $4}\u0026#39;)) local cluster_voltypes=($(openstack volume type list -f value -c Name)) [[ \u0026#34;${#cluster_voltypes[@]}\u0026#34; -eq 0 ]] \u0026amp;\u0026amp; cluster_voltypes[0]=\u0026#34;None\u0026#34; while IFS=\u0026#39;,\u0026#39; read -r img flavor sys data name net vmip zone voltype; do [[ -z \u0026#34;$img\u0026#34; || -z \u0026#34;$flavor\u0026#34; || -z \u0026#34;$sys\u0026#34; || -z \u0026#34;$data\u0026#34; || -z \u0026#34;$name\u0026#34; || -z \u0026#34;$net\u0026#34; || -z \u0026#34;$vmip\u0026#34; ]] \u0026amp;\u0026amp; continue [[ \u0026#34;$img\u0026#34; =~ ^(#|$) ]] \u0026amp;\u0026amp; continue if ! [[ \u0026#34; ${cluster_images[@]} \u0026#34; =~ \u0026#34; ${img} \u0026#34; ]]; then log WARN \u0026#34;镜像不存在: ${img} (IP: $vmip)\u0026#34; fi if ! [[ \u0026#34; ${cluster_flavors[@]} \u0026#34; =~ \u0026#34; ${flavor} \u0026#34; ]]; then log WARN \u0026#34;规格不存在: ${flavor} (IP: $vmip)\u0026#34; fi if ! [[ \u0026#34; ${cluster_vlans[@]} \u0026#34; =~ \u0026#34; ${net} \u0026#34; ]]; then log WARN \u0026#34;网络不存在: ${net} (IP: $vmip)\u0026#34; fi if ! [[ \u0026#34; ${cluster_voltypes[@]} \u0026#34; =~ \u0026#34; ${voltype} \u0026#34; ]]; then log WARN \u0026#34;卷类型不存在: ${voltype} (IP: $vmip)\u0026#34; fi IFS=\u0026#34;$DATA_SEPARATOR\u0026#34; read -r -a sizes \u0026lt;\u0026lt;\u0026lt;\u0026#34;$data\u0026#34; #local arr=(${data//${DATA_SEPARATOR}/ }) for size in \u0026#34;${sizes[@]}\u0026#34;; do if ! [[ \u0026#34;$size\u0026#34; =~ ^[0-9]+$ ]]; then log WARN \u0026#34;数据盘格式检测异常: $size (IP: $vmip), 间隔符号为: ${DATA_SEPARATOR}\u0026#34; fi done log INFO \u0026#34;IP: ${vmip}, 5项检查流程执行完毕\u0026#34; done \u0026lt; \u0026#34;$file\u0026#34; return 0 } main(){ init_deps case \u0026#34;$1\u0026#34; in init) export_hypervisors \u0026#34;$HYPERVISOR_FILE\u0026#34; export_volume_types \u0026#34;$VOL_TYPE_FILE\u0026#34; export_images \u0026#34;$IMAGES_FILE\u0026#34; log INFO \u0026#34;【重要】请人工核实生成的系统环境信息文件【重要】\u0026#34; ;; build) main_build_task \u0026#34;$2\u0026#34; ;; review) main_review_task \u0026#34;$VM_FORMAT_FILE\u0026#34; ;; run) main_vm_task \u0026#34;$VM_FORMAT_FILE\u0026#34; ;; *) log ERROR \u0026#34;Usage: $0 {init|build \u0026lt;src\u0026gt;|review|run}\u0026#34; ;; esac } main \u0026#34;$@\u0026#34; ","permalink":"https://gnail89.github.io/posts/openstack_mk_vm/","summary":"引导虚拟机实例简单流程(AI) #!/bin/bash set -Eeuo pipefail basepath=$(cd `dirname $0`;pwd) readonly VM_FORMAT_FILE=\u0026#34;vm_info_format.txt\u0026#34; readonly HYPERVISOR_FILE=\u0026#34;env_available_host_list.txt\u0026#34; readonly VOL_TYPE_FILE=\u0026#34;env_cinder_volume_type.txt\u0026#34; readonly IMAGES_FILE=\u0026#34;env_image_list.txt\u0026#34; readonly DATA_SEPARATOR=\u0026#34;+\u0026#34; readonly MAX_PARALLEL=2 log() { local lvl=\u0026#34;$1\u0026#34; msg=\u0026#34;$2\u0026#34; printf \u0026#34;%s [%4s] %s\\n\u0026#34; \u0026#34;$(date +\u0026#39;%F %T\u0026#39;)\u0026#34; \u0026#34;$lvl\u0026#34; \u0026#34;$msg\u0026#34; \u0026gt;\u0026amp;2 [[ \u0026#34;$lvl\u0026#34; == \u0026#34;ERROR\u0026#34; ]] \u0026amp;\u0026amp; exit 1 return 0 } trap \u0026#39;log ERROR \u0026#34;Script aborted in $FUNCNAME at line $LINENO\u0026#34;\u0026#39; ERR require_cmd() { command -v \u0026#34;$1\u0026#34; \u0026amp;\u0026gt;/dev/null || log ERROR \u0026#34;缺少命令: $1\u0026#34;;} init_deps() { for cmd in openstack nova awk grep sed tee; do require_cmd \u0026#34;$cmd\u0026#34;; done; } check_volumes(){ local retries=$1 count=0 vol_ids=(\u0026#34;${@:2}\u0026#34;) for ((i=0; i\u0026lt;retries; i++)); do count=0 for vid in \u0026#34;${vol_ids[@]}\u0026#34;; do [[ -z \u0026#34;$vid\u0026#34; || \u0026#34;$vid\u0026#34; == \u0026#34;None\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;无效的卷ID\u0026#34; local st; st=$(openstack volume show -f value --column status \u0026#34;$vid\u0026#34;) case \u0026#34;$st\u0026#34; in available) count=$((count+1)) ;; error*) log ERROR \u0026#34;卷状态显示error, ID: $vid\u0026#34; ;; esac done [[ $count -eq ${#vol_ids[@]} ]] \u0026amp;\u0026amp; { echo ok; return 0; } sleep 30 done echo \u0026#34;None\u0026#34; return 0 } export_volume_types(){ \u0026gt; \u0026#34;$1\u0026#34; log INFO \u0026#34;正在导出卷类型列表\u0026#34; openstack volume type list -f value --column Name | grep -Ev \u0026#39;^(#|$)\u0026#39; \u0026gt; \u0026#34;$1\u0026#34; || echo \u0026#34;None\u0026#34; \u0026gt; \u0026#34;$1\u0026#34; } export_hypervisors(){ \u0026gt; \u0026#34;$1\u0026#34; log INFO \u0026#34;正在导出可用的宿主机列表\u0026#34; openstack compute service list -f value --service nova-compute \\ | awk \u0026#39;$5 == \u0026#34;enabled\u0026#34; \u0026amp;\u0026amp; $6 == \u0026#34;up\u0026#34; {print $4\u0026#34;:\u0026#34;$3}\u0026#39; | grep -Ev \u0026#39;^(#|$)\u0026#39; \u0026gt; \u0026#34;$1\u0026#34; || echo \u0026#34;None\u0026#34; \u0026gt; \u0026#34;$1\u0026#34; } export_images(){ \u0026gt; \u0026#34;$1\u0026#34; log INFO \u0026#34;正在导出镜像列表\u0026#34; openstack image list -f value --column Name | grep -Ev \u0026#39;^(#|$)\u0026#39; \u0026gt; \u0026#34;$1\u0026#34; || echo \u0026#34;None\u0026#34; \u0026gt; \u0026#34;$1\u0026#34; } create_boot_volume(){ local name=$1 size=$2 image=$3 type=$4 [[ -z \u0026#34;$name\u0026#34; || -z \u0026#34;$size\u0026#34; || -z \u0026#34;$image\u0026#34; || -z \u0026#34;$type\u0026#34; ]] \u0026amp;\u0026amp; log ERROR \u0026#34;缺少卷创建参数\u0026#34; local vol_name=\u0026#34;${name}-sysvol\u0026#34; log INFO \u0026#34;创建启动卷: $vol_name\u0026#34; local opts=(--size \u0026#34;$size\u0026#34; --image \u0026#34;$(openstack image show -f value --column id \u0026#34;$image\u0026#34;)\u0026#34;) [[ \u0026#34;$type\u0026#34; !","title":"Openstack_mk_vm"},{"content":"软件环境 openEuler 22.03 Python 3.9.22 OpenStack 2025.1 软件版本上限约束 根据安装要求，选定软件版本限制范围。https://opendev.org/openstack/requirements 或者 https://releases.openstack.org/constraints/upper/2025.1 # e.g # 版本约束 curl https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/upper-constraints.txt 基础环境配置 # 关闭SELinux sudo setenforce 0 sudo sed -i \u0026#39;s/^SELINUX=.*/SELINUX=disabled/\u0026#39; /etc/selinux/config # 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 停用NetworkManager sudo systemctl stop NetworkManager sudo systemctl disable NetworkManager sudo dnf install -y network-scripts sudo systemctl enable network # 配置ulimit cat \u0026gt;\u0026gt; /etc/security/limits.conf \u0026lt;\u0026lt;EOF * soft nproc 65535 * hard nproc 65535 * soft nofile 65535 * hard nofile 65535 EOF # 配置epel源 cat \u0026gt; /etc/yum.repos.d/epel-OpenEuler.repo \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [epel] name=Extra Packages for Enterprise Linux $releasever - $basearch # It is much more secure to use the #metalink, but if you wish to use a local mirror # place its address here. baseurl=https://repo.huaweicloud.com/epel/8/Everything/$basearch #metalink=https://mirrors.fedoraproject.org/#metalink?repo=epel-$releasever\u0026amp;arch=$basearch\u0026amp;infra=$infra\u0026amp;content=$contentdir enabled=1 gpgcheck=0 countme=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8 [epel-debuginfo] name=Extra Packages for Enterprise Linux $releasever - $basearch - Debug # It is much more secure to use the #metalink, but if you wish to use a local mirror # place its address here. baseurl=https://repo.huaweicloud.com/epel/$releasever/Everything/$basearch/debug #metalink=https://mirrors.fedoraproject.org/#metalink?repo=epel-debug-$releasever\u0026amp;arch=$basearch\u0026amp;infra=$infra\u0026amp;content=$contentdir enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8 gpgcheck=0 [epel-source] name=Extra Packages for Enterprise Linux $releasever - $basearch - Source # It is much more secure to use the #metalink, but if you wish to use a local mirror # place it\u0026#39;s address here. baseurl=https://repo.huaweicloud.com/$releasever/Everything/source/tree/ #metalink=https://mirrors.fedoraproject.org/#metalink?repo=epel-source-$releasever\u0026amp;arch=$basearch\u0026amp;infra=$infra\u0026amp;content=$contentdir enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-8 gpgcheck=0 EOF 系统软件依赖 # 系统依赖列表 curl https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/bindep.txt # 源码编译安装liberasurecode yum install gcc make autoconf automake libtool wget https://gitee.com/src-openeuler/liberasurecode/raw/Multi-Version_OpenStack-Wallaby_openEuler-22.03-LTS-SP1/liberasurecode-1.6.2.tar.gz tar zxf liberasurecode-1.6.2.tar.gz \u0026amp;\u0026amp; cd liberasurecode-1.6.2 ./autogen.sh ./configure make \u0026amp;\u0026amp; make install echo \u0026#34;/usr/local/lib\u0026#34; \u0026gt; /etc/ld.so.conf.d/liberasurecode.conf \u0026amp;\u0026amp; ldconfig 使用python虚拟环境 # 安装依赖 yum install gcc gcc-c++ make openssl-devel bzip2-devel libffi-devel \\ zlib-devel xz-devel ncurses-devel sqlite-devel readline-devel \\ tk-devel gdbm-devel libnsl2-devel uuid-devel # 安装python ./configure --prefix=/opt/python-3.9.22 --with-openssl=/opt/openssl --enable-optimizations make \u0026amp;\u0026amp; make install # 创建python虚拟测试环境 python3 -m venv /opt/openstack # 进入虚拟环境 source /opt/openstack/bin/activate # 退出虚拟环境 deactivate # 升级pip pip install --upgrade pip -i https://pypi.tuna.tsinghua.edu.cn/simple 安装软件 # 检查组件版本release，确定版本信息 # https://releases.openstack.org/epoxy/index.html pip install \\ -i https://pypi.tuna.tsinghua.edu.cn/simple \\ -c https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/upper-constraints.txt \\ -r https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/requirements.txt \\ -r https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/test-requirements.txt \\ keystone==27.0.0 \\ nova==31.0.0 \\ cinder==26.1.0 \\ neutron==26.0.1 \\ glance==30.0.0 \\ horizon==25.3.0 \\ ironic==29.0.3 \\ swift==2.35.0 # 下载离线包 pip download -d ~/packages/ \\ -i https://pypi.tuna.tsinghua.edu.cn/simple \\ -c https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/upper-constraints.txt \\ -r https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/requirements.txt \\ -r https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/test-requirements.txt \\ keystone==27.0.0 \\ nova==31.0.0 \\ cinder==26.1.0 \\ neutron==26.0.1 \\ glance==30.0.0 \\ horizon==25.3.0 \\ ironic==29.0.3 \\ swift==2.35.0 获取源码 # e.g # git获取 # https://opendev.org/openstack/cinder/src/branch/stable/2025.1/ # # release获取 # https://tarballs.openstack.org/cinder/cinder-26.1.0.tar.gz 使用tox生成默认配置文件 # 创建python虚拟测试环境 python3 -m venv /opt/tox # 进入虚拟环境 source /opt/tox/bin/activate # 安装tox pip install tox -i https://pypi.tuna.tsinghua.edu.cn/simple # 生成默认配置文件 # grep \u0026#34;:genconfig\u0026#34; tox.ini tox -e genconfig 测试镜像 # 下载测试镜像 wget http://download.cirros-cloud.net/0.5.2/cirros-0.5.2-x86_64-disk.img openstack image create --file cirros-0.5.2-x86_64-disk.img --public cirros # 创建实例 openstack network list # 获取网络ID openstack server create --flavor m1.tiny --image cirros --nic net-id=\u0026lt;网络ID\u0026gt; test-vm ","permalink":"https://gnail89.github.io/posts/openstack_build_case/","summary":"软件环境 openEuler 22.03 Python 3.9.22 OpenStack 2025.1 软件版本上限约束 根据安装要求，选定软件版本限制范围。https://opendev.org/openstack/requirements 或者 https://releases.openstack.org/constraints/upper/2025.1 # e.g # 版本约束 curl https://opendev.org/openstack/requirements/raw/branch/stable/2025.1/upper-constraints.txt 基础环境配置 # 关闭SELinux sudo setenforce 0 sudo sed -i \u0026#39;s/^SELINUX=.*/SELINUX=disabled/\u0026#39; /etc/selinux/config # 关闭防火墙 systemctl stop firewalld systemctl disable firewalld # 停用NetworkManager sudo systemctl stop NetworkManager sudo systemctl disable NetworkManager sudo dnf install -y network-scripts sudo systemctl enable network # 配置ulimit cat \u0026gt;\u0026gt; /etc/security/limits.conf \u0026lt;\u0026lt;EOF * soft nproc 65535 * hard nproc 65535 * soft nofile 65535 * hard nofile 65535 EOF # 配置epel源 cat \u0026gt; /etc/yum.","title":"OpenStack build case"},{"content":"案例 - 1 软件版本 openEuler 22.03 openssl-1.1.1w.tar.gz Python-3.9.22.tgz openssl dnf install -y gcc gcc-c++ make perl perl-devel tar zxf openssl-1.1.1w.tar.gz ./config --prefix=/opt/openssl make \u0026amp;\u0026amp; make install Python dnf install -y bzip2-devel libffi-devel zlib-devel xz-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel libnsl2-devel uuid-devel ./configure --prefix=/opt/python-3.9.22 --with-openssl=/opt/openssl --enable-shared --enable-optimizations make \u0026amp;\u0026amp; make install 案例 - 2 软件版本 openEuler 24.03 (LTS-SP3) openssl-1.0.2u.tar.gz Python-2.7.18.tgz openssl # 依赖安装 dnf install -y gcc gcc-c++ make perl perl-devel # 编译安装 tar zxf openssl-1.0.2u.tar.gz ./config --prefix=/opt/openssl-1.0.2 shared make \u0026amp;\u0026amp; make install_sw # 配置文件 cat \u0026gt; /etc/ld.so.conf.d/openssl-1.0.2.conf \u0026lt;\u0026lt;EOF /opt/openssl-1.0.2/lib EOF # 生效 ldconfig python # 依赖安装 dnf install -y gcc make zlib-devel bzip2 bzip2-devel readline-devel sqlite sqlite-devel openssl-devel tk-devel libffi-devel wget tar gdbm-devel libdb-devel libnsl2-devel openssl-devel gdbm-devel libdb-devel # 设置环境变量 export LDFLAGS=\u0026#34;-L/opt/openssl-1.0.2/lib\u0026#34; export CPPFLAGS=\u0026#34;-I/opt/openssl-1.0.2/include\u0026#34; export PKG_CONFIG_PATH=/opt/openssl-1.0.2/lib/pkgconfig # 编译安装 tar zxf Python-2.7.18.tgz ./configure --prefix=/opt/python2.7 --enable-optimizations make \u0026amp;\u0026amp; make install # 安装pip /opt/python2.7/bin/python2 -m ensurepip --upgrade # 验证ssl python -c \u0026#34;import ssl; print(ssl.OPENSSL_VERSION)\u0026#34; ","permalink":"https://gnail89.github.io/posts/python_source_build/","summary":"案例 - 1 软件版本 openEuler 22.03 openssl-1.1.1w.tar.gz Python-3.9.22.tgz openssl dnf install -y gcc gcc-c++ make perl perl-devel tar zxf openssl-1.1.1w.tar.gz ./config --prefix=/opt/openssl make \u0026amp;\u0026amp; make install Python dnf install -y bzip2-devel libffi-devel zlib-devel xz-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel libnsl2-devel uuid-devel ./configure --prefix=/opt/python-3.9.22 --with-openssl=/opt/openssl --enable-shared --enable-optimizations make \u0026amp;\u0026amp; make install 案例 - 2 软件版本 openEuler 24.03 (LTS-SP3) openssl-1.0.2u.tar.gz Python-2.7.18.tgz openssl # 依赖安装 dnf install -y gcc gcc-c++ make perl perl-devel # 编译安装 tar zxf openssl-1.","title":"Python源码编译简要步骤"},{"content":"AI写的SQL表结构 CREATE TABLE vms ( id INT NOT NULL AUTO_INCREMENT, location VARCHAR(255) NOT NULL, vm_uuid VARCHAR(36) NOT NULL, vm_name VARCHAR(255) NOT NULL, vm_state VARCHAR(255) DEFAULT NULL, power_state VARCHAR(36) DEFAULT NULL, vm_nets VARCHAR(255) DEFAULT NULL, flavor_name VARCHAR(255) DEFAULT NULL, flavor_id VARCHAR(36) DEFAULT NULL, availability_zone VARCHAR(255) DEFAULT NULL, vm_host VARCHAR(255) DEFAULT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, deleted_at TIMESTAMP DEFAULT NULL, is_deleted BOOLEAN DEFAULT FALSE, PRIMARY KEY (id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; CREATE TABLE vms_archive ( id INT NOT NULL, location VARCHAR(255) NOT NULL, vm_uuid VARCHAR(36) NOT NULL, vm_name VARCHAR(255) NOT NULL, vm_state VARCHAR(255) DEFAULT NULL, power_state VARCHAR(36) DEFAULT NULL, vm_nets VARCHAR(255) DEFAULT NULL, flavor_name VARCHAR(255) DEFAULT NULL, flavor_id VARCHAR(36) DEFAULT NULL, availability_zone VARCHAR(255) DEFAULT NULL, vm_host VARCHAR(255) DEFAULT NULL, created_at TIMESTAMP NOT NULL, updated_at TIMESTAMP NOT NULL, deleted_at TIMESTAMP DEFAULT NULL, is_deleted BOOLEAN DEFAULT FALSE, archive_batch TIMESTAMP NOT NULL, PRIMARY KEY (id, archive_batch), INDEX idx_vm_uuid (vm_uuid), INDEX idx_archive_batch (archive_batch), INDEX idx_vm_uuid_batch (vm_uuid, archive_batch), INDEX idx_active_time_batch (vm_state, archive_batch, vm_uuid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; AI写的SQL查询 #查询指定批次的差异。 SELECT cur.vm_uuid, cur.vm_name, cur.vm_state AS current_state, cur.archive_batch AS current_batch, prev.vm_state AS previous_state, prev.archive_batch AS previous_batch FROM vms_archive cur JOIN vms_archive prev ON cur.vm_uuid = prev.vm_uuid AND prev.archive_batch = \u0026#39;2025-06-30 00:00:00\u0026#39; AND prev.is_deleted = 0 WHERE cur.archive_batch = \u0026#39;2025-07-01 00:00:00\u0026#39; AND cur.is_deleted = 0 AND cur.vm_state != \u0026#39;ACTIVE\u0026#39; AND prev.vm_state = \u0026#39;ACTIVE\u0026#39;; # 统计 SELECT archive_batch, COUNT(vm_uuid) AS \u0026#39;主机ACTIVE监控\u0026#39; FROM vms_archive WHERE vm_state = \u0026#39;ACTIVE\u0026#39; AND archive_batch \u0026gt;= NOW() - INTERVAL 12 HOUR GROUP BY archive_batch ORDER BY archive_batch; AI写的shell脚本 #!/bin/bash # 配置MySQL连接参数 MYSQL_HOST=\u0026#34;\u0026#34; MYSQL_PORT=\u0026#34;\u0026#34; MYSQL_USER=\u0026#34;\u0026#34; MYSQL_PASSWORD=\u0026#34;\u0026#34; MYSQL_DATABASE=\u0026#34;\u0026#34; MYSQL_CMD=\u0026#34;/usr/bin/mysql\u0026#34; # 生成当前批次号（Unix时间戳格式） ARCHIVE_BATCH=$(date +%s) ### 阶段1：数据归档 ### ARCHIVE_SQL=$(cat \u0026lt;\u0026lt;SQL -- 设置事务隔离级别确保一致性 SET TRANSACTION ISOLATION LEVEL REPEATABLE READ; START TRANSACTION; -- 执行数据归档 INSERT INTO vms_archive ( id, location, vm_uuid, vm_name, vm_state, power_state, vm_nets, flavor_name, flavor_id, availability_zone, vm_host, created_at, updated_at, deleted_at, is_deleted, archive_batch ) SELECT id, location, vm_uuid, vm_name, vm_state, power_state, vm_nets, flavor_name, flavor_id, availability_zone, vm_host, created_at, updated_at, deleted_at, is_deleted, FROM_UNIXTIME(${ARCHIVE_BATCH}) FROM vms WHERE is_deleted = 0; COMMIT; SQL ) # 执行归档 echo \u0026#34;${ARCHIVE_SQL}\u0026#34; | $MYSQL_CMD -h \u0026#34;$MYSQL_HOST\u0026#34; -P \u0026#34;$MYSQL_PORT\u0026#34; \\ -u \u0026#34;$MYSQL_USER\u0026#34; -p\u0026#34;$MYSQL_PASSWORD\u0026#34; -D \u0026#34;$MYSQL_DATABASE\u0026#34; --ssl-mode=DISABLED \u0026gt; /dev/null 2\u0026gt;\u0026amp;1 # 检查归档结果 if [ $? -ne 0 ]; then echo \u0026#34;[$(date +\u0026#39;%F %T\u0026#39;)] 错误: 数据归档失败!\u0026#34; \u0026gt;\u0026amp;2 exit 1 fi ### 阶段2：清理历史数据 ### CLEANUP_SQL=$(cat \u0026lt;\u0026lt;SQL -- 开启事务确保原子性 START TRANSACTION; -- 删除12小时前的归档批次 DELETE FROM vms_archive WHERE archive_batch \u0026lt; NOW() - INTERVAL 12 HOUR; -- 获取删除行数 SELECT ROW_COUNT() AS deleted_rows; COMMIT; SQL ) # 执行清理操作并捕获结果 CLEANUP_RESULT=$(echo \u0026#34;${CLEANUP_SQL}\u0026#34; | $MYSQL_CMD -h \u0026#34;$MYSQL_HOST\u0026#34; -P \u0026#34;$MYSQL_PORT\u0026#34; \\ -u \u0026#34;$MYSQL_USER\u0026#34; -p\u0026#34;$MYSQL_PASSWORD\u0026#34; -D \u0026#34;$MYSQL_DATABASE\u0026#34; --ssl-mode=DISABLED -s -N 2\u0026gt;/dev/null) # 检查清理结果 if [ $? -eq 0 ]; then DELETED_ROWS=$(echo \u0026#34;$CLEANUP_RESULT\u0026#34; | awk \u0026#39;NR==1{print $1}\u0026#39;) echo \u0026#34;[$(date +\u0026#39;%F %T\u0026#39;)] 归档成功! 批次: $ARCHIVE_BATCH, 清理记录: $DELETED_ROWS 行\u0026#34; else echo \u0026#34;[$(date +\u0026#39;%F %T\u0026#39;)] 警告: 归档成功但清理失败! 批次: $ARCHIVE_BATCH\u0026#34; \u0026gt;\u0026amp;2 fi ","permalink":"https://gnail89.github.io/posts/vm_state_shell_ai/","summary":"AI写的SQL表结构 CREATE TABLE vms ( id INT NOT NULL AUTO_INCREMENT, location VARCHAR(255) NOT NULL, vm_uuid VARCHAR(36) NOT NULL, vm_name VARCHAR(255) NOT NULL, vm_state VARCHAR(255) DEFAULT NULL, power_state VARCHAR(36) DEFAULT NULL, vm_nets VARCHAR(255) DEFAULT NULL, flavor_name VARCHAR(255) DEFAULT NULL, flavor_id VARCHAR(36) DEFAULT NULL, availability_zone VARCHAR(255) DEFAULT NULL, vm_host VARCHAR(255) DEFAULT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, deleted_at TIMESTAMP DEFAULT NULL, is_deleted BOOLEAN DEFAULT FALSE, PRIMARY KEY (id) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; CREATE TABLE vms_archive ( id INT NOT NULL, location VARCHAR(255) NOT NULL, vm_uuid VARCHAR(36) NOT NULL, vm_name VARCHAR(255) NOT NULL, vm_state VARCHAR(255) DEFAULT NULL, power_state VARCHAR(36) DEFAULT NULL, vm_nets VARCHAR(255) DEFAULT NULL, flavor_name VARCHAR(255) DEFAULT NULL, flavor_id VARCHAR(36) DEFAULT NULL, availability_zone VARCHAR(255) DEFAULT NULL, vm_host VARCHAR(255) DEFAULT NULL, created_at TIMESTAMP NOT NULL, updated_at TIMESTAMP NOT NULL, deleted_at TIMESTAMP DEFAULT NULL, is_deleted BOOLEAN DEFAULT FALSE, archive_batch TIMESTAMP NOT NULL, PRIMARY KEY (id, archive_batch), INDEX idx_vm_uuid (vm_uuid), INDEX idx_archive_batch (archive_batch), INDEX idx_vm_uuid_batch (vm_uuid, archive_batch), INDEX idx_active_time_batch (vm_state, archive_batch, vm_uuid) ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci; AI写的SQL查询 #查询指定批次的差异。 SELECT cur.","title":"AI写shell脚本"},{"content":"python 加速 # 创建python虚拟测试环境 python3 -m venv /opt/test # 进入虚拟环境 source /opt/test/bin/activate # 退出虚拟环境 deactivate # 镜像加速地址： pip install sqlite -i https://pypi.tuna.tsinghua.edu.cn/simple -c ~/requirements_upper.txt docker-compose # install the Docker Compose standalone curl -SL https://github.com/docker/compose/releases/download/v2.33.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose docker-ce repo示例 # docker-ce/阿里云 dnf config-manager --add-repo=https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo docker 加速 # ref: https://github.com/DaoCloud/public-image-mirror # 添加到 /etc/docker/daemon.json { \u0026#34;registry-mirrors\u0026#34;: [ \u0026#34;https://docker.m.daocloud.io\u0026#34; ], \u0026#34;ipv6\u0026#34;: false, \u0026#34;fixed-cidr\u0026#34;: \u0026#34;10.10.0.0/24\u0026#34;, \u0026#34;bip\u0026#34;: \u0026#34;10.10.0.1/24\u0026#34;, \u0026#34;log-driver\u0026#34;: \u0026#34;json-file\u0026#34;, \u0026#34;log-opts\u0026#34;: { \u0026#34;max-size\u0026#34;: \u0026#34;100m\u0026#34;, \u0026#34;max-file\u0026#34;: \u0026#34;3\u0026#34; } } epel repo示例 # epel(RHEL 8) yum install -y https://mirrors.aliyun.com/epel/epel-release-latest-8.noarch.rpm sed -i \u0026#39;s|^#baseurl=https://download.example/pub|baseurl=https://mirrors.aliyun.com|\u0026#39; /etc/yum.repos.d/epel* sed -i \u0026#39;s|^metalink|#metalink|\u0026#39; /etc/yum.repos.d/epel* # epel(RHEL 7) wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo # epel(RHEL 6) wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-archive-6.repo Centos 7 repo示例 [base] name=CentOS-7 - Base - Aliyun baseurl=http://mirrors.aliyun.com/centos/7/os/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 enabled=1 [updates] name=CentOS-7 - Updates - Aliyun baseurl=http://mirrors.aliyun.com/centos/7/updates/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 enabled=1 [extras] name=CentOS-7 - Extras - Aliyun baseurl=http://mirrors.aliyun.com/centos/7/extras/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7 enabled=1 Centos 6 repo示例 [base] name=CentOS-6 - Base - Aliyun baseurl=https://mirrors.aliyun.com/centos-vault/6.10/os/$basearch/ gpgcheck=1 gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-6 enabled=1 [updates] name=CentOS-6 - Updates - Aliyun baseurl=https://mirrors.aliyun.com/centos-vault/6.10/updates/$basearch/ gpgcheck=1 gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-6 enabled=1 [extras] name=CentOS-6 - Extras - Aliyun baseurl=https://mirrors.aliyun.com/centos-vault/6.10/extras/$basearch/ gpgcheck=1 gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-6 enabled=1 ","permalink":"https://gnail89.github.io/posts/setup_repo/","summary":"python 加速 # 创建python虚拟测试环境 python3 -m venv /opt/test # 进入虚拟环境 source /opt/test/bin/activate # 退出虚拟环境 deactivate # 镜像加速地址： pip install sqlite -i https://pypi.tuna.tsinghua.edu.cn/simple -c ~/requirements_upper.txt docker-compose # install the Docker Compose standalone curl -SL https://github.com/docker/compose/releases/download/v2.33.1/docker-compose-linux-x86_64 -o /usr/local/bin/docker-compose chmod +x /usr/local/bin/docker-compose docker-ce repo示例 # docker-ce/阿里云 dnf config-manager --add-repo=https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo docker 加速 # ref: https://github.com/DaoCloud/public-image-mirror # 添加到 /etc/docker/daemon.json { \u0026#34;registry-mirrors\u0026#34;: [ \u0026#34;https://docker.m.daocloud.io\u0026#34; ], \u0026#34;ipv6\u0026#34;: false, \u0026#34;fixed-cidr\u0026#34;: \u0026#34;10.10.0.0/24\u0026#34;, \u0026#34;bip\u0026#34;: \u0026#34;10.10.0.1/24\u0026#34;, \u0026#34;log-driver\u0026#34;: \u0026#34;json-file\u0026#34;, \u0026#34;log-opts\u0026#34;: { \u0026#34;max-size\u0026#34;: \u0026#34;100m\u0026#34;, \u0026#34;max-file\u0026#34;: \u0026#34;3\u0026#34; } } epel repo示例 # epel(RHEL 8) yum install -y https://mirrors.","title":"常用软件源repo"},{"content":"rsync升级简要步骤 编译环境: redhat 6.5 x86_64 # 安装依赖包 yum install gcc gcc-c++ make automake acl libacl-devel popt-devel # 配置，去除部分特性 ./configure --prefix=/opt/rsync \\ --disable-openssl \\ --disable-xxhash \\ --disable-zstd \\ --disable-lz4 # 编译 make \u0026amp;\u0026amp; make install # 将生成的二进制文件部署到指定路径下进行测试: /usr/bin/rsync ","permalink":"https://gnail89.github.io/posts/rsync_notes/","summary":"rsync升级简要步骤 编译环境: redhat 6.5 x86_64 # 安装依赖包 yum install gcc gcc-c++ make automake acl libacl-devel popt-devel # 配置，去除部分特性 ./configure --prefix=/opt/rsync \\ --disable-openssl \\ --disable-xxhash \\ --disable-zstd \\ --disable-lz4 # 编译 make \u0026amp;\u0026amp; make install # 将生成的二进制文件部署到指定路径下进行测试: /usr/bin/rsync ","title":"rsync升级简要步骤"},{"content":"OpenStack example # 将image镜像设置为vga显示驱动，以支持更大分辨率 openstack image set --property hw_video_model=vga image_id # 将volume卷设置为vga显示驱动，以支持更大分辨率（已有虚拟机需要重建） openstack volume set --image-property hw_video_model=vga volume_id ","permalink":"https://gnail89.github.io/posts/openstack_notes/","summary":"OpenStack example # 将image镜像设置为vga显示驱动，以支持更大分辨率 openstack image set --property hw_video_model=vga image_id # 将volume卷设置为vga显示驱动，以支持更大分辨率（已有虚拟机需要重建） openstack volume set --image-property hw_video_model=vga volume_id ","title":"OpenStack_notes"},{"content":"ICMP Timestamp Request Remote Date Disclosure Synopsis It is possible to determine the exact time set on the remote host.\nDescription The remote host answers to an ICMP timestamp request. This allows an attacker to know the date that is set on the targeted machine, which may assist an unauthenticated, remote attacker in defeating time-based authentication protocols.\nTimestamps returned from machines running Windows Vista / 7 / 2008 / 2008 R2 are deliberately incorrect, but usually within 1000 seconds of the actual system time.\nSolution Filter out the ICMP timestamp requests (13), and the outgoing ICMP timestamp replies (14).\n# Block ICMP Timestamp requests (type 13) with iptables: iptables -I INPUT -p icmp --icmp-type timestamp-request -j DROP # Block ICMP Timestamp reply (type 14) with iptables: iptables -I OUTPUT -p icmp --icmp-type timestamp-reply -j DROP # firewalld firewall-cmd --zone=public --add-icmp-block={timestamp-request,timestamp-reply} --permanent firewall-cmd --reload References CVE-1999-0524\n","permalink":"https://gnail89.github.io/posts/cve-1999-0524/","summary":"ICMP Timestamp Request Remote Date Disclosure Synopsis It is possible to determine the exact time set on the remote host.\nDescription The remote host answers to an ICMP timestamp request. This allows an attacker to know the date that is set on the targeted machine, which may assist an unauthenticated, remote attacker in defeating time-based authentication protocols.\nTimestamps returned from machines running Windows Vista / 7 / 2008 / 2008 R2 are deliberately incorrect, but usually within 1000 seconds of the actual system time.","title":"CVE-1999-0524"},{"content":"开源项目地址: https://github.com/thu-pacman/AIPerf\n基础环境 SSH免密 ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub server-ip 部署NFS 安装软件包 yum install -y nfs-utils 配置服务端 mkdir /userhome chmod -R 777 /userhome echo \u0026#34;/userhome *(rw,sync,insecure,no_root_squash)\u0026#34; \u0026gt;\u0026gt; /etc/exports # 启动服务 systemctl enable --now nfs-server systemctl status nfs-server 配置客户端 mkdir /userhome mount server-ip:/userhome /userhome 准备数据集 数据集下载 Imagenet官方地址：http://www.image-net.org/index 使用ImageNet-2012数据集 cd /userhome/AIPerf/scripts/build_data ./download_imagenet.sh ls ./imagenet 原始的ImageNet-2012包含以下两个文件: ILSVRC2012_img_val.tar ILSVRC2012_img_train.tar TFReord制作 训练集和验证集需要按照1000个子目录下包含图片的格式，处理步骤： 将train 和 val 的数据按照文件夹分类 指定参数运行build_imagenet_data.py # 做验证集 cd /userhome/AIPerf/scripts/build_data mkdir -p ILSVRC2012/raw-data/imagenet-data/validation/ tar -xvf imagenet/ILSVRC2012_img_val.tar -C ILSVRC2012/raw-data/imagenet-data/validation/ python preprocess_imagenet_validation_data.py ILSVRC2012/raw-data/imagenet-data/validation/ imagenet_2012_validation_synset_labels.txt # 做训练集 mkdir -p ILSVRC2012/raw-data/imagenet-data/train/ tar -xvf imagenet/ILSVRC2012_img_train.tar -C ILSVRC2012/raw-data/imagenet-data/train/ \u0026amp;\u0026amp; cd ILSVRC2012/raw-data/imagenet-data/train find . -name \u0026#34;*.tar\u0026#34; | while read NAE ; do mkdir -p \u0026#34;${NAE%.tar}\u0026#34;; tar -xvf \u0026#34;${NAE}\u0026#34; -C \u0026#34;${NAE%.tar}\u0026#34;; rm -f \u0026#34;${NAE}\u0026#34;; done cd /userhome/AIPerf/scripts/build_data # TensorFlow需要制作TFRecord mkdir -p ILSVRC2012/output python build_imagenet_data.py \\ --train_directory=ILSVRC2012/raw-data/imagenet-data/train \\ --validation_directory=ILSVRC2012/raw-data/imagenet-data/validation \\ --output_directory=ILSVRC2012/output \\ --imagenet_metadata_file=imagenet_metadata.txt \\ --labels_file=imagenet_lsvrc_2015_synsets.txt # 在ILSVRC2012/output目录下生成128个validation开头的验证集文件和1024个train开头的训练集文件。 分发数据集 对TensorFlow，将验证集和数据集移动到slave节点。 mkdir -p /root/datasets/imagenet/train mkdir -p /root/datasets/imagenet/val mv ILSVRC2012/output/train-* /root/datasets/imagenet/train mv ILSVRC2012/output/validation-* /root/datasets/imagenet/val 对其他模型，将解压后验证集和数据集移动到slave节点，根据框架自行调整。 # 对PyTorch mkdir -p /root/datasets/imagenet/ mv ILSVRC2012/train /root/datasets/imagenet mv ILSVRC2012/val /root/datasets/imagenet 项目安装 环境变量声明 export AIPERF_WORKDIR=/userhome export AIPERF_SLAVE_WORKDIR=/root/ export AIPERF_MASTER_IP=10.0.1.100 export AIPERF_MASTER_PORT=9987 * AIPERF_WORKDIR AIPerf工作目录，必须在共享存储上，能被所有节点访问到 AIPerf代码，部分log文件等将会被放到这个目录 * AIPERF_SLAVE_WORKDIR 计算节点工作目录，不需要在共享存储上（也可以放在共享存储） 部分计算节点log文件将会被生成到这个目录下 * AIPERF_MASTER_IP AIPerf调度服务IP，即控制节点IP * AIPERF_MASTER_PORT AIPerf调度服务端口 下载源代码 # 在共享存储上创建AIPerf工作目录 mkdir -p $AIPERF_WORKDIR # 下载源代码 git clone https://github.com/thu-pacman/AIPerf.git $AIPERF_WORKDIR/AIPerf AIPerf目录下aiperf_setenv.sh脚本设置环境变量。 source $AIPERF_WORKDIR/AIPerf/aiperf_setenv.sh 安装依赖包 安装Python wget https://www.python.org/ftp/python/3.8.20/python-3.9.20.tgz tar -zxvf python-3.9.20.tgz # 安装依赖包 yum install -y gcc gcc-c++ make openssl-devel bzip2-devel libffi-devel # 编译安装 cd python-3.9.20 ./configure --prefix=/usr/local/python3 \\ --with-ensurepip=install \\ --enable-optimizations make \u0026amp;\u0026amp; make install # 设置环境变量 # ln -s /usr/local/python3/bin/python3.9 /usr/bin/python3 echo \u0026#39;export PATH=/usr/local/python3/bin:$PATH\u0026#39; \u0026gt;\u0026gt; ~/.bash_profile source ~/.bash_profile # 升级pip和setuptools # 临时镜像源: https://mirrors.tuna.tsinghua.edu.cn/help/pypi/ python3 -m pip install --upgrade pip -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple python3 -m pip install --upgrade setuptools -i https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple 控制节点 # python3 -m pip install -r $AIPERF_WORKDIR/AIPerf/requirements_master.txt python3 -m pip install scikit-learn pyyaml django schema colorama ruamel.yaml matplotlib jinja2 zmq torch==1.8.0 hyperopt==0.1.2 json_tricks \u0026#34;numpy\u0026lt;2.0\u0026#34; scipy coverage psutil 计算节点 # python3 -m pip install -r $AIPERF_WORKDIR/AIPerf/requirements_slave.txt # 根据CUDA和cuDNN版本选择计算框架版本，例如对于CUDA 11.2，cuDNN 8.1，可选择tensorflow 2.5.0 # python3 -m pip install tensorflow==2.5.0 python3 -m pip install zmq scikit-learn psutil dill hyperopt==0.1.2 json_tricks \u0026#34;numpy\u0026lt;2.0\u0026#34; scipy coverage tensorflow==2.5.0 torch==1.8.0 # sample install nvidia driver wget https://us.download.nvidia.com/tesla/460.106.00/NVIDIA-Linux-x86_64-460.106.00.run sh NVIDIA-Linux-x86_64-460.106.00.run # CUDA Toolkit 11.2 # https://developer.nvidia.com/cuda-11.2.0-download-archive wget https://developer.download.nvidia.com/compute/cuda/11.2.0/local_installers/cuda_11.2.0_460.27.04_linux.run sh cuda_11.2.0_460.27.04_linux.run # environment variables export PATH=/usr/local/cuda/bin${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} # cuDNN 8.1 # https://developer.nvidia.com/rdp/cudnn-archive # https://developer.nvidia.com/compute/machine-learning/cudnn/secure/8.1.0.77/11.2_20210127/cudnn-11.2-linux-x64-v8.1.0.77.tgz wget https://developer.download.nvidia.com/compute/redist/cudnn/v8.1.0/cudnn-11.2-linux-x64-v8.1.0.77.tgz tar -zvxf cudnn-11.2-linux-x64-v8.1.0.77.tgz cp -P cuda/include/cudnn.h /usr/local/cuda/include cp -P cuda/lib64/libcudnn* /usr/local/cuda/lib64/ chmod a+r /usr/local/cuda/include/cudnn.h #check cuda nvidia-smi #check cudnn nvcc -V 安装AIPerf 控制节点 # 安装AutoML组件 cd $AIPERF_WORKDIR/AIPerf/src/sdk/pynni/ python3 -m pip install -e . # 安装aiperf控制组件 cd $AIPERF_WORKDIR/AIPerf/src/aiperf_manager/ python3 -m pip install -e . aiperf --help 计算节点 # 安装AutoML组件 cd $AIPERF_WORKDIR/AIPerf/src/sdk/pynni/ python3 -m pip install -e . 下载模型权重 将权重文件resnet50_weights_tf_dim_ordering_tf_kernels.h5下载并放到$AIPERF_WORKDIR下 # 百度云盘 https://pan.baidu.com/s/1JBOh2XwetZAalLdmULBN2Q 提取码: 5nbc mv resnet50_weights_tf_dim_ordering_tf_kernels.h5 $AIPERF_WORKDIR (md5sum: a7b3fe01876f51b976af0dea6bc144eb) 启动测试(控制节点) 启动调度服务 配置计算节点信息 ${AIPERF_WORKDIR}/AIPerf/aiperf_ctrl/servers.json # 例如，集群有2个节点，每个节点有4张GPU，则配置如下： [ { \u0026#34;ip\u0026#34;: \u0026#34;172.23.33.33\u0026#34;, \u0026#34;tag\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;waiting\u0026#34;, \u0026#34;CUDA_VISIBLE_DEVICES\u0026#34;: \u0026#34;0,1,2,3\u0026#34; }, { \u0026#34;ip\u0026#34;: \u0026#34;172.23.33.34\u0026#34;, \u0026#34;tag\u0026#34;: \u0026#34;\u0026#34;, \u0026#34;status\u0026#34;: \u0026#34;waiting\u0026#34;, \u0026#34;CUDA_VISIBLE_DEVICES\u0026#34;: \u0026#34;0,1,2,3\u0026#34; } ] 启动命令 cd $AIPERF_WORKDIR/AIPerf/aiperf_ctrl python3 manage.py runserver ${AIPERF_MASTER_IP}:${AIPERF_MASTER_PORT} 启动AIPerf测试 测试基本条件：\n1.测试运行时间应不少于1小时；\n2.测试的计算精度不低于FP-16；\n3.测试完成时所取得的最高正确率应大于70%；\n初始化配置 ${AIPERF_WORKDIR}/AIPerf/examples/trials/network_morphism/imagenet/config.yml\n关键参数解释:\n可选参数 说明 默认值 1 trialConcurrency 同时运行的trial数 1 2 maxExecDuration 设置测试时间(单位 ：h) 12 3 CUDA_VISIBLE_DEVICES 指定测试程序可用的gpu索引 0,1,2,3,4,5,6,7 4 srun：\u0026ndash;cpus-per-task=30 参数为slurm可用cpu核数减 1 30 5 \u0026ndash;slave 跟 trialConcurrency参数保持一致 1 6 \u0026ndash;ip master节点ip，直接使用默认值 ${AIPERF_MASTER_IP} 7 \u0026ndash;batch_size batch size 448 8 \u0026ndash;epochs 正常训练epoch数 60 9 \u0026ndash;initial_lr 初始学习率 1e-1 10 \u0026ndash;final_lr 最终学习率 0 11 \u0026ndash;train_data_dir 训练数据集路径 None 12 \u0026ndash;val_data_dir 验证数据集路径 None 13 \u0026ndash;warmup_1 warm up机制第一轮epoch数 15 14 \u0026ndash;warmup_2 warm up机制第二轮epoch数 30 15 \u0026ndash;warmup_3 warm up机制第三轮epoch数 45 16 \u0026ndash;num_parallel_calls tfrecord数据加载加速 48 # 示例 authorName: default experimentName: example_imagenet-network-morphism-test trialConcurrency: 2 maxExecDuration: 24h maxTrialNum: 9999 trainingServicePlatform: local useAnnotation: false logLevel: trace tuner: #choice: TPE, Random, Anneal, Evolution, BatchTuner, NetworkMorphism #SMAC (SMAC should be installed through nnictl) builtinTunerName: NetworkMorphism classArgs: #choice: maximize, minimize optimize_mode: maximize #for now, this tuner only supports cv domain task: cv #input image width input_width: 224 #input image channel input_channel: 3 #number of classes n_output_node: 1000 trial: command: CUDA_VISIBLE_DEVICES=0 \\ python3 imagenet_train.py \\ --slave 2 \\ --ip ${AIPERF_MASTER_IP} \\ --batch_size 448 \\ --epoch 60 \\ --initial_lr 1e-1 \\ --final_lr 0 \\ --train_data_dir /root/datasets/imagenet/train \\ --val_data_dir /root/datasets/imagenet/val codeDir: . gpuNum: 0 # 注意配置文件中slave参数需要和trialConcurrency参数一致，修改为计算节点个数 # 修改CUDA_VISIBLE_DEVICES为计算节点所用的GPU # 需要修改--train_data_dir和--var_data_dir的位置以指向计算节点数据集存储位置 运行benchmark\n在${AIPERF_WORKDIR}/AIPerf/examples/trials/network_morphism/imagenet/目录下执行以下命令运行用例\n注：若使用pyTorch，请在 ${AIPERF_WORKDIR}/AIPerf/examples/trials/network_morphism/imagenetTorch/下执行\n# 启动 aiperf create -c config.yml # 停止,ctrl-c终止 # 清理所有计算节点上的训练进程 aiperf clean 查看状态\n当测试运行过程中，运行以下程序会在终端打印experiment的Error、Score、Regulated Score等信息\npython3 $AIPERF_WORKDIR/AIPerf/scripts/reports/report.py --id experiment_ID 同时会产生实验报告存放在experiment_ID的对应路径${AIPERF_WORKDIR}/mountdir/nni/experiments/experiment_ID/results目录下。\n实验成功时报告为 Report_Succeed.html。\n实验失败时报告为 Report_Failed.html。\n实验失败会报告失败原因，请查阅AI Benchmark测试规范分析失败原因。 保存日志\u0026amp;结果数据 运行以下程序可将测试产生的日志以及数据统一保存到${AIPERF_WORKDIR}/mountdir/nni/experiments/experiment_ID/results/logs中，便于实验分析。 由于实验数据在复制过程中会导致额外的网络、内存、cpu等资源开销，建议在实验停止/结束后再执行日志保存操作。 python3 $AIPERF_WORKDIR/AIPerf/scripts/reports/report.py --id experiment_ID --logs True 测试参数设置及推荐环境配置(官方) 可变设置 slave计算节点的GPU卡数：默认将单个物理服务器作为一个slave节点，并使用其所有GPU； 深度学习框架：默认使用keras+tensorflow； 数据集加载方式：默认将数据预处理成TFRecord格式，以加快数据加载的效率； 数据集存储方式：默认采用网络共享存储； 超参设置：默认初始batch size=448，默认初始学习率=0.1，默认最终学习率=0，默认正常训练epochs=60，默认从第四轮trial开始，每个trial搜索1次，默认超参为kernel size和batch size。 推荐环境配置\n软件：TensorFlow2.5.0，CUDA11.2，cuDNN8.1，python3.8\n","permalink":"https://gnail89.github.io/posts/aiperf-setup/","summary":"开源项目地址: https://github.com/thu-pacman/AIPerf\n基础环境 SSH免密 ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub server-ip 部署NFS 安装软件包 yum install -y nfs-utils 配置服务端 mkdir /userhome chmod -R 777 /userhome echo \u0026#34;/userhome *(rw,sync,insecure,no_root_squash)\u0026#34; \u0026gt;\u0026gt; /etc/exports # 启动服务 systemctl enable --now nfs-server systemctl status nfs-server 配置客户端 mkdir /userhome mount server-ip:/userhome /userhome 准备数据集 数据集下载 Imagenet官方地址：http://www.image-net.org/index 使用ImageNet-2012数据集 cd /userhome/AIPerf/scripts/build_data ./download_imagenet.sh ls ./imagenet 原始的ImageNet-2012包含以下两个文件: ILSVRC2012_img_val.tar ILSVRC2012_img_train.tar TFReord制作 训练集和验证集需要按照1000个子目录下包含图片的格式，处理步骤： 将train 和 val 的数据按照文件夹分类 指定参数运行build_imagenet_data.py # 做验证集 cd /userhome/AIPerf/scripts/build_data mkdir -p ILSVRC2012/raw-data/imagenet-data/validation/ tar -xvf imagenet/ILSVRC2012_img_val.","title":"AIPerf 基准测试工具安装"},{"content":"","permalink":"https://gnail89.github.io/posts/kubectl-notes/","summary":"","title":"Kubectl notes"},{"content":"安装步骤 下载二进制包 # docker-ce 26.1.3 curl -O https://mirrors.ustc.edu.cn/docker-ce/linux/static/stable/x86_64/docker-26.1.3.tgz # cri-dockerd 0.3.10 curl -O https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.10/cri-dockerd-0.3.10.amd64.tgz 部署docker-ce # 解压缩 tar zxf docker-26.1.3.tgz # 复制二进制文件 cp docker/* /usr/bin/ 配置containerd.service文件 cat \u0026gt;/etc/systemd/system/containerd.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=1048576 TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target EOF # 设置开机自启 systemctl enable --now containerd.service 配置docker.service文件 cat \u0026gt; /etc/systemd/system/docker.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target NetworkManager.service firewalld.service cri-docker.service docker.socket containerd.service Wants=network-online.target NetworkManager.service Requires=docker.socket containerd.service [Service] Type=notify ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process OOMScoreAdjust=-500 [Install] WantedBy=multi-user.target EOF 配置docker.socket文件 cat \u0026gt; /etc/systemd/system/docker.socket \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=Docker Socket for the API [Socket] ListenStream=/var/run/docker.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target EOF 配置docker的daemon.json文件 mkdir /etc/docker/ cat \u0026gt;/etc/docker/daemon.json \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; { \u0026#34;exec-opts\u0026#34;: [\u0026#34;native.cgroupdriver=systemd\u0026#34;], \u0026#34;registry-mirrors\u0026#34;: [ \u0026#34;https://docker.m.daocloud.io\u0026#34;, \u0026#34;https://dockerproxy.com\u0026#34; ], \u0026#34;max-concurrent-downloads\u0026#34;: 10, \u0026#34;log-driver\u0026#34;: \u0026#34;json-file\u0026#34;, \u0026#34;log-level\u0026#34;: \u0026#34;warn\u0026#34;, \u0026#34;log-opts\u0026#34;: { \u0026#34;max-size\u0026#34;: \u0026#34;10m\u0026#34;, \u0026#34;max-file\u0026#34;: \u0026#34;3\u0026#34; }, \u0026#34;data-root\u0026#34;: \u0026#34;/data\u0026#34; } EOF # exec-opts: 用于设置Docker守护进程的选项，native.cgroupdriver=systemd表示使用systemd作为Cgroup驱动程序。 # registry-mirrors: 用于指定Docker镜像的镜像注册服务器。示例：https://docker.m.daocloud.io、https://docker.mirrors.ustc.edu.cn、http://hub-mirror.c.163.com # max-concurrent-downloads: 用于设置同时下载镜像的最大数量，默认值为3，这里设置为10。 # log-driver: 用于设置Docker守护进程的日志驱动程序，这里设置为json-file。 # log-level: 用于设置日志的级别，这里设置为warn。 # log-opts: 用于设置日志驱动程序的选项，这里有两个选项：max-size和max-file。max-size表示每个日志文件的最大大小，这里设置为10m，max-file表示保存的最大日志文件数量，这里设置为3。 # data-root: 用于设置Docker守护进程的数据存储根目录，默认为/var/lib/docker，这里设置为/data。 启动docker # 创建docker组 groupadd docker systemctl daemon-reload systemctl enable --now containerd.service systemctl enable --now docker.socket systemctl enable --now docker.service # 验证 docker info 部署cri-dockerd 用于kubelet交互 tar zxf cri-dockerd-0.3.10.amd64.tgz cp -r cri-dockerd/ /usr/bin/ chmod +x /usr/bin/cri-dockerd/cri-dockerd 配置cri-docker.service文件 cat \u0026gt; /usr/lib/systemd/system/cri-docker.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOF 配置cri-docker.socket文件 cat \u0026gt; /usr/lib/systemd/system/cri-docker.socket \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=CRI Docker Socket for the API PartOf=cri-docker.service [Socket] ListenStream=%t/cri-dockerd.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target EOF 启动cri-docker systemctl daemon-reload systemctl enable --now cri-docker.service systemctl status cri-docker.service ","permalink":"https://gnail89.github.io/posts/docker-ce-static/","summary":"安装步骤 下载二进制包 # docker-ce 26.1.3 curl -O https://mirrors.ustc.edu.cn/docker-ce/linux/static/stable/x86_64/docker-26.1.3.tgz # cri-dockerd 0.3.10 curl -O https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.10/cri-dockerd-0.3.10.amd64.tgz 部署docker-ce # 解压缩 tar zxf docker-26.1.3.tgz # 复制二进制文件 cp docker/* /usr/bin/ 配置containerd.service文件 cat \u0026gt;/etc/systemd/system/containerd.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=1048576 TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target EOF # 设置开机自启 systemctl enable --now containerd.service 配置docker.service文件 cat \u0026gt; /etc/systemd/system/docker.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.","title":"docker-ce二进制安装"},{"content":"Enable TLSv1.2 Registry Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server] \u0026#34;Enabled\u0026#34;=dword:00000001 \u0026#34;DisabledByDefault\u0026#34;=dword:00000000 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client] \u0026#34;Enabled\u0026#34;=dword:00000001 \u0026#34;DisabledByDefault\u0026#34;=dword:00000000 Powershell New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;1\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 0 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;1\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 0 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null Disable TLS 1.0 Registry Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Server] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Client] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 Powershell New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Server\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Server\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Server\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Client\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Client\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.0\\Client\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null Disable TLS 1.1 Registry Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Server] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Client] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 Powershell New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Server\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Server\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Server\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Client\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Client\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.1\\Client\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null Disabe SSL v2.0 and SSL v3.0 Registry Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Server] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Client] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Server] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Client] \u0026#34;Enabled\u0026#34;=dword:00000000 \u0026#34;DisabledByDefault\u0026#34;=dword:00000001 Powershell New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Server\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Server\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Server\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Client\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Client\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 2.0\\Client\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Server\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Server\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Server\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Client\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Client\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;0\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\SSL 3.0\\Client\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 1 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null Disable Ciphers RC4 AND 3DES Registry Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Ciphers] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Ciphers\\RC4 128/128] \u0026#34;Enabled\u0026#34;=dword:00000000 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Ciphers\\RC4 40/128] \u0026#34;Enabled\u0026#34;=dword:00000000 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Ciphers\\RC4 56/128] \u0026#34;Enabled\u0026#34;=dword:00000000 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Ciphers\\RC4 64/128] \u0026#34;Enabled\u0026#34;=dword:00000000 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Ciphers\\Triple DES 168] \u0026#34;Enabled\u0026#34;=dword:00000000 Cipher Suites in TLS/SSL https://learn.microsoft.com/en-us/windows/win32/secauthn/cipher-suites-in-schannel 运行gpedit.msc，打开“本地组策略编辑器”-“计算机配置”-“管理模板”-“网络”-“SSL配置设置”， 在“SSL密码套件顺序”选项上，右键“编辑”。 选择“已启用”，在右侧说明页中找到“TLS 1.2 SHA256 and SHA384 cipher suites”和“TLS 1.2 ECC GCM cipher suites”复制下面的选项，然后以逗号分隔的形式形成一整串字符串，填至左侧输入框中。 示例： TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256_P521,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P384,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384_P521,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P384,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P521,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P384,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P521,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P384,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256_P521,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P384,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384_P521,TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_256_CBC_SHA ","permalink":"https://gnail89.github.io/posts/windows-security-for-ssl/","summary":"Enable TLSv1.2 Registry Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2] [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server] \u0026#34;Enabled\u0026#34;=dword:00000001 \u0026#34;DisabledByDefault\u0026#34;=dword:00000000 [HKEY_LOCAL_MACHINE\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client] \u0026#34;Enabled\u0026#34;=dword:00000001 \u0026#34;DisabledByDefault\u0026#34;=dword:00000000 Powershell New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;1\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Server\u0026#39; -name \u0026#39;DisabledByDefault\u0026#39; -value 0 -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-Item \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.2\\Client\u0026#39; -name \u0026#39;Enabled\u0026#39; -value \u0026#39;1\u0026#39; -PropertyType \u0026#39;DWord\u0026#39; -Force | Out-Null New-ItemProperty -path \u0026#39;HKLM:\\SYSTEM\\CurrentControlSet\\Control\\SecurityProviders\\SCHANNEL\\Protocols\\TLS 1.","title":"Windows Enable TLSv1.2 And Other Disable"},{"content":"NFS Server笔记 exports配置 # ro # 只读 # rw # 读写 # sync # 同步写入资料到内存与硬盘中 # async # 资料会先暂存于内存中，而非直接写入硬盘 # secure # NFS通过1024以下的安全TCP/IP端口发送 # insecure # NFS通过1024以上的端口发送 # wdelay #如果多个用户要写入NFS目录，则归组写入（默认） # no_wdelay # 如果多个用户要写入NFS目录，则立即写入，当使用async时，无需此设置。 # hide # 在NFS共享目录中不共享其子目录 # no_hide # 共享NFS目录的子目录 # subtree_check # 如果共享/usr/bin之类的子目录时，强制NFS检查父目录的权限 # no_subtree_check # 同上，但不检查父目录权限 # all_squash # 将所有共享文件的UID和GID映射匿名用户anonymous，适合公用目录。 # no_all_squash # 保留共享文件的UID和GID（默认） # root_squash # root用户的所有请求映射成如anonymous用户一样的权限（默认） # no_root_squash # root用户具有根目录的完全管理访问权限 # anonuid=xxx # 指定NFS服务器/etc/passwd文件中匿名用户的UID # anongid=xxx # 指定NFS服务器/etc/passwd文件中匿名用户的GID # cat /etc/exports /nfs 192.168.1.2(rw,async,no_root_squash) nfs.conf配置文件 # 在/etc/nfs.conf中配置threads线程数量 [nfsd] threads=64 [mountd] threads=4 # 改端口 RQUOTAD_PORT=30000 LOCKD_TCPPORT=30001 LOCKD_UDPPORT=30002 MOUNTD_PORT=30003 STATD_PORT=30004 内核参数配置 sunrpc.tcp_slot_table_entries=2048 #rpc并发请求数 net.core.rmem_max=16777216 #socket读缓冲区大小 net.core.wmem_max=16777216 #socket写缓冲区大小 net.core.somaxconn=32768 #tcp队列 net.ipv4.tcp_max_sync_backlog=8192 #syn队列 NFS Client笔记 挂载参数： # _netdev # 挂载时等待网络设备就绪 # noatime # 禁止atime # nodiratime # 禁止diratime # vers=3 # 版本v3 # noac # 禁用cache缓存机制 # rsize=8192 # 读缓存大小 # wsize=8192 # 写缓存大小 # tcp # 使用tcp协议 #cat /etc/fstab 192.168.1.1:/nfs /nfs nfs _netdev,noatime,nodiratime,vers=3,proto=tcp,rsize=262144,wsize=262144,timeo=50 0 0 # mount挂载 mount -t nfs -o vers=3,proto=tcp,async,noatime,nodiratime,wsize=262144,rsize=262144,timeo=50 192.168.1.1:/nfs /nfs mount -t nfs -o proto=tcp,noatime,nodiratime,rsize=262144,wsize=262144,timeo=50 192.168.1.1:/nfs /nfs 查看状态 rpcinfo -p # 查看rpc端口使用情况 exportfs -arv # 刷新exports nfsstat -r # 查看nfsd状态 nfsstat -s # 查看nfsd状态 nfsstat -c # 查看client状态 cat /proc/net/rpc/nfsd # 查看nfs线程运行状态 cat /proc/fs/nfsd/exports # 查看client挂载状态 ","permalink":"https://gnail89.github.io/posts/nfs-optimize/","summary":"NFS Server笔记 exports配置 # ro # 只读 # rw # 读写 # sync # 同步写入资料到内存与硬盘中 # async # 资料会先暂存于内存中，而非直接写入硬盘 # secure # NFS通过1024以下的安全TCP/IP端口发送 # insecure # NFS通过1024以上的端口发送 # wdelay #如果多个用户要写入NFS目录，则归组写入（默认） # no_wdelay # 如果多个用户要写入NFS目录，则立即写入，当使用async时，无需此设置。 # hide # 在NFS共享目录中不共享其子目录 # no_hide # 共享NFS目录的子目录 # subtree_check # 如果共享/usr/bin之类的子目录时，强制NFS检查父目录的权限 # no_subtree_check # 同上，但不检查父目录权限 # all_squash # 将所有共享文件的UID和GID映射匿名用户anonymous，适合公用目录。 # no_all_squash # 保留共享文件的UID和GID（默认） # root_squash # root用户的所有请求映射成如anonymous用户一样的权限（默认） # no_root_squash # root用户具有根目录的完全管理访问权限 # anonuid=xxx # 指定NFS服务器/etc/passwd文件中匿名用户的UID # anongid=xxx # 指定NFS服务器/etc/passwd文件中匿名用户的GID # cat /etc/exports /nfs 192.","title":"NFS笔记"},{"content":"环境变量： export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_DEFAULT_REGION= S3基本操作命令： # 将bucket中特定前缀的对象全部复制到另一个bucket aws s3 cp s3://bucket-name/example s3://my-bucket/ # 将本地文件复制到s3存储中 aws s3 cp filename.txt s3://bucket-name # 将s3中的对象复制到本地目录 aws s3 cp s3://bucket-name/filename.txt ./ #以流式传输到s3的对象 echo \u0026#34;hello world\u0026#34; | aws s3 cp - s3://bucket-name/filename.txt # 以流式传输stdout打印在控制台中 $ aws s3 cp s3://bucket-name/filename.txt - hello world # 以流式传输压缩对象 aws s3 cp s3://bucket-name/pre - | bzip2 --best | aws s3 cp - s3://bucket-name/key.bz2 # bucket列表 aws s3 ls # bucket中的对象列表 aws s3 ls s3://bucket-name # bucket中特定前缀的对象列表 aws s3 ls s3://bucket-name/example/ # 创建bucket aws s3 mb s3://bucket-name # 将example中所有对象移动到另一个bucket aws s3://bucket-name/example s3://my-bucket/ # 将本地文件移动到s3 bucket中 aws s3 mv filename.txt s3://bucket-name # 将s3中的对象移动到本地目录 aws s3 mv s3://bucket-name/filename.txt ./ # 删除bucket及其对象 aws s3 rb s3://bucket-name --force # 删除bucket中的单个对象 aws s3 rm s3://bucket-name/example/filename.txt # 删除bucket中特定前缀的所有对象 aws s3 rm s3://bucket-name/example --recursive # 同步当前目录中的文件到s3特定前缀的内容（更新和追加内容） aws s3 sync . s3://my-bucket/path # 当前目录已删除的不存在文件，在s3中同步删除（强一致性） aws s3 sync . s3://my-bucket/path --delete ","permalink":"https://gnail89.github.io/posts/s3-awscli-basic/","summary":"环境变量： export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= export AWS_DEFAULT_REGION= S3基本操作命令： # 将bucket中特定前缀的对象全部复制到另一个bucket aws s3 cp s3://bucket-name/example s3://my-bucket/ # 将本地文件复制到s3存储中 aws s3 cp filename.txt s3://bucket-name # 将s3中的对象复制到本地目录 aws s3 cp s3://bucket-name/filename.txt ./ #以流式传输到s3的对象 echo \u0026#34;hello world\u0026#34; | aws s3 cp - s3://bucket-name/filename.txt # 以流式传输stdout打印在控制台中 $ aws s3 cp s3://bucket-name/filename.txt - hello world # 以流式传输压缩对象 aws s3 cp s3://bucket-name/pre - | bzip2 --best | aws s3 cp - s3://bucket-name/key.bz2 # bucket列表 aws s3 ls # bucket中的对象列表 aws s3 ls s3://bucket-name # bucket中特定前缀的对象列表 aws s3 ls s3://bucket-name/example/ # 创建bucket aws s3 mb s3://bucket-name # 将example中所有对象移动到另一个bucket aws s3://bucket-name/example s3://my-bucket/ # 将本地文件移动到s3 bucket中 aws s3 mv filename.","title":"S3 awscli Basic"},{"content":"OpenStack client 3.12.2离线安装 下载python源码包：https://www.python.org/ftp/python/3.8.18/Python-3.8.18.tgz\n查找pip包：https://pypi.org/\nOpenStack各版本release版号：https://releases.openstack.org/\nOpenStack各组件release版号：https://docs.openstack.org/releasenotes/\n编译安装python tar zxf Python-3.8.18.tgz ./configure --prefix=/opt/python make \u0026amp;\u0026amp; make install 通过pip下载离线安装包 创建虚拟环境 python3.8 -m venv /opt/env_cli3.12 source /opt/env_cli3.12/bin/activate #（退出虚拟环境使用deactivate） 下载安装包 /opt/python/bin/pip3 download \\ -d /opt/python-openstackclient-3.12.2 \\ --index https://mirrors.bfsu.edu.cn/pypi/web/simple/ \\ python-openstackclient==3.12.2 \\ python-novaclient==9.1.3 \\ python-keystoneclient==3.12.0 \\ python-glanceclient==2.8.1 \\ python-cinderclient==3.1.1 \\ openstacksdk==0.9.17 \\ keystoneauth1==3.1.1 \\ os-client-config==1.28.1 \\ osc-lib==1.7.1 \\ oslo.config==4.13.2 \\ oslo.i18n==3.23.1 \\ oslo.serialization==2.20.3 \\ oslo.utils==3.28.4 通过pip执行离线安装 cd /opt/python-openstackclient-3.12.2 /opt/python/bin/pip3 install --no-index -f . * ","permalink":"https://gnail89.github.io/posts/setup-openstackclient-3.12.2-by-pip/","summary":"OpenStack client 3.12.2离线安装 下载python源码包：https://www.python.org/ftp/python/3.8.18/Python-3.8.18.tgz\n查找pip包：https://pypi.org/\nOpenStack各版本release版号：https://releases.openstack.org/\nOpenStack各组件release版号：https://docs.openstack.org/releasenotes/\n编译安装python tar zxf Python-3.8.18.tgz ./configure --prefix=/opt/python make \u0026amp;\u0026amp; make install 通过pip下载离线安装包 创建虚拟环境 python3.8 -m venv /opt/env_cli3.12 source /opt/env_cli3.12/bin/activate #（退出虚拟环境使用deactivate） 下载安装包 /opt/python/bin/pip3 download \\ -d /opt/python-openstackclient-3.12.2 \\ --index https://mirrors.bfsu.edu.cn/pypi/web/simple/ \\ python-openstackclient==3.12.2 \\ python-novaclient==9.1.3 \\ python-keystoneclient==3.12.0 \\ python-glanceclient==2.8.1 \\ python-cinderclient==3.1.1 \\ openstacksdk==0.9.17 \\ keystoneauth1==3.1.1 \\ os-client-config==1.28.1 \\ osc-lib==1.7.1 \\ oslo.config==4.13.2 \\ oslo.i18n==3.23.1 \\ oslo.serialization==2.20.3 \\ oslo.utils==3.28.4 通过pip执行离线安装 cd /opt/python-openstackclient-3.12.2 /opt/python/bin/pip3 install --no-index -f . * ","title":"Setup Openstackclient 3.12.2 by pip"},{"content":"裸金属镜像配置 镜像制作 type: whole disk image\n准备 系统镜像ISO\nUEFI 引导\nqemu-kvm环境\n命名规范 区域-版本-架构-启动类型-文件系统-镜像大小: C01-AnolisOS-8.8-x86_64-UEFI-XFS-LVM-8G-IRONIC-yyymmdd.qcow2\n分区规划参考 镜像大小8GB, 可按需求进行分区, 后面配合userdata使用\nFS VG LV SIZE /boot - - 1 GB /boot/efi - - 600 MB / rootvg root 1 GB /usr rootvg usr 4 GB /var rootvg var 1 GB /home rootvg home 1 GB 软件安装 cloud-init\nqemu-guest-agent (可选)\ndnf install -y cloud-init qemu-guest-agent cloud-init配置文件更新 # 更新的内容 no_ssh_fingerprints: true disable_root: false ssh_pwauth: true network: config: disabled datasource_list: [OpenStack, ConfigDrive] datasource: OpenStack: metadata_urls: [\u0026#39;http://169.254.169.254\u0026#39;] max_wait: 120 timeout: 5 # 示例 sed -i \u0026#39;/# from the distro configuration/a\\no_ssh_fingerprints: true\u0026#39; /etc/cloud/cloud.cfg sed -i \u0026#39;/disable_root: false/a\\ssh_pwauth: 1\u0026#39; /etc/cloud/cloud.cfg echo \u0026#39; network: config: disabled\u0026#39; \u0026gt;\u0026gt; /etc/cloud/cloud.cfg echo \u0026#34; datasource_list: [OpenStack, ConfigDrive] datasource: OpenStack: metadata_urls: [\u0026#39;http://169.254.169.254\u0026#39;] max_wait: 120 timeout: 5\u0026#34; \u0026gt;\u0026gt; /etc/cloud/cloud.cfg # 开机自启动 systemctl enable cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service 添加raid驱动 cat \u0026gt; /etc/dracut.conf.d/raid_drives.conf \u0026lt;\u0026lt;EOF logfile=/var/log/dracut.log add_drivers+=\u0026#34; ahci megaraid_sas mpt3sas mpt2sas \u0026#34; EOF dracut -f -N UserData模板 #!/bin/bash node_bond0_macs=\u0026#34;08:30:ce:06:2b:cc 08:30:ce:06:33:aa\u0026#34; node_bond0_mode=\u0026#34;mode=active-backup,miimon=100\u0026#34; node_bond0_ip=\u0026#34;192.168.1.1\u0026#34; node_bond0_prefix=\u0026#34;24\u0026#34; node_bond0_netmask=\u0026#34;255.255.255.0\u0026#34; node_bond0_gw=\u0026#34;192.168.1.1\u0026#34; create_rootvg_partition(){ # 使用xfs文件系统 os_rootvg=\u0026#34;$(lsblk -r |grep -w \u0026#34;/\u0026#34; |awk -F\u0026#39;-\u0026#39; \u0026#39;{print $1}\u0026#39;)\u0026#34; if [ x\u0026#34;${os_rootvg}\u0026#34; != x\u0026#34;\u0026#34; ]; then local parts=\u0026#39; resize|root|/|50G resize|user|/usr|50G resize|var|/var|50G resize|home|/home|50G create|swap|swap|8G create|tmp|/tmp|20G \u0026#39; for line in ${parts};do local type=\u0026#34;$(echo ${line} |awk -F\u0026#39;|\u0026#39; \u0026#39;{print $1}\u0026#39;)\u0026#34; local part_name=\u0026#34;$(echo ${line} |awk -F\u0026#39;|\u0026#39; \u0026#39;{print $2}\u0026#39;)\u0026#34; local part_dir=\u0026#34;$(echo ${line} |awk -F\u0026#39;|\u0026#39; \u0026#39;{print $3}\u0026#39;)\u0026#34; local part_size=\u0026#34;$(echo ${line} |awk -F\u0026#39;|\u0026#39; \u0026#39;{print $4}\u0026#39;)\u0026#34; case ${type} in create) lvcreate -y -L ${part_size} -n ${part_name} ${os_rootvg} mkfs.xfs -f /dev/${os_rootvg}/${part_name} if [ $(grep \u0026#34;/dev/${os_rootvg}/${part_name}\u0026#34; /etc/fstab |wc -l) -eq 0 ]; then [ ! -d ${part_dir} ] \u0026amp;\u0026amp; mkdir -p ${part_dir} echo \u0026#34;/dev/${os_rootvg}/${part_name} ${part_dir} xfs defaults 0 0\u0026#34; \u0026gt;\u0026gt; /etc/fstab fi ;; resize) lvextend -L ${part_size} -n /dev/${os_rootvg}/${part_name} xfs_growfs /dev/${os_rootvg}/${part_name} # xfs_growfs ${part_dir} ;; *) echo \u0026#34;type error\u0026#34; ;; esac done else echo \u0026#34;get rootvg info failed\u0026#34; fi } create_users(){ local users=\u0026#39; user:pass \u0026#39; for line in ${users};do local name=\u0026#34;$(echo ${line} |awk -F\u0026#39;:\u0026#39; \u0026#39;{print $1}\u0026#39;)\u0026#34; useradd -m ${name} echo \u0026#34;${$line}\u0026#34; |chpasswd done } resize_os_partition(){ # 获取安装系统的磁盘 os_disk=\u0026#34;$(lsblk -r |grep -w \u0026#34;/boot\u0026#34; |cut -c1-3)\u0026#34; # 获取/根文件系统所在的vg os_rootvg=\u0026#34;$(lsblk -r |grep -w \u0026#34;/\u0026#34; |awk -F\u0026#39;-\u0026#39; \u0026#39;{print $1}\u0026#39;)\u0026#34; # 获取/根文件系统所在的盘符路径 if [ x\u0026#34;${os_disk}\u0026#34; != x\u0026#34;\u0026#34; ] \u0026amp;\u0026amp; [ x\u0026#34;${os_rootvg}\u0026#34; != x\u0026#34;\u0026#34; ]; then os_rootvg_path=\u0026#34;$(pvs |grep -w \u0026#34;${os_rootvg}\u0026#34; |egrep -o \u0026#34;/dev/${os_disk}[[:digit:]]+\u0026#34;)\u0026#34; else os_rootvg_path=\u0026#34;\u0026#34; fi # 获取/根文件系统所在分区号 if [ x\u0026#34;${os_rootvg_path}\u0026#34; != x\u0026#34;\u0026#34; ]; then os_rootvg_path_num=\u0026#34;$(lsblk -r |grep -w \u0026#34;$(echo ${os_rootvg_path} |awk -F\u0026#39;/\u0026#39; \u0026#39;{print $NF}\u0026#39;)\u0026#34; |awk \u0026#39;{print $2}\u0026#39; |awk -F\u0026#39;:\u0026#39; \u0026#39;{print $NF}\u0026#39;)\u0026#34; else os_rootvg_path_num=\u0026#34;\u0026#34; fi # resize系统盘，扩容至100% if [ x\u0026#34;${os_disk}\u0026#34; != x\u0026#34;\u0026#34; ] \u0026amp;\u0026amp; [ x\u0026#34;${os_rootvg_path}\u0026#34; != x\u0026#34;\u0026#34; ] \u0026amp;\u0026amp; [ x\u0026#34;${os_rootvg_path_num}\u0026#34; != x\u0026#34;\u0026#34; ]; then parted /dev/${os_disk} resizepart ${os_rootvg_path_num} 100% pvresize ${os_rootvg_path} else echo \u0026#34;resize root disk failed\u0026#34; fi } net_setup_nmcli(){ if [ x\u0026#34;$(systemctl is-enabled NetworkManager)\u0026#34; != x\u0026#34;enabled\u0026#34; ]; then systemctl enable NetworkManager fi if [ x\u0026#34;$(systemctl is-active NetworkManager)\u0026#34; != x\u0026#34;active\u0026#34; ]; then systemctl restart NetworkManager fi # 删除现有网络配置 nmcli connection delete $(nmcli connection show |awk \u0026#39;{print $(NF-2)}\u0026#39;) # 根据MAC遍历接口 local ports=() for m in ${node_bond0_macs};do for i in $(nmcli device status |grep ethernet |awk \u0026#39;{print $1}\u0026#39;);do if [ $(nmcli device show $i |grep -i \u0026#34;${m}\u0026#34; |wc -l) -eq 1 ]; then local n=${#ports[@]} ports[${n}]=\u0026#34;${i}\u0026#34; break fi done done # 创建bond0接口 nmcli connection add type bond con-name bond0 ifname bond0 autoconnect yes bond.options \u0026#34;${node_bond0_mode}\u0026#34; nmcli connection modify bond0 ipv4.addresses ${node_bond0_ip}/${node_bond0_prefix} ipv4.gateway ${node_bond0_gw} ipv4.method manual # 绑定子接口 for ((i=0;i\u0026lt;${#ports[@]};i++)); do nmcli connection add type ethernet con-name ${ports[${i}]} ifname ${ports[${i}]} master bond0 done # 重载配置 nmcli connection reload nmcli connection down bond0 \u0026amp;\u0026amp; nmcli connection up bond0 } add_repos(){ # 增加软件源配置 if [ -d /etc/yum.repos.d ]; then mkdir -p /etc/yum.repos.d/bak [ -d /etc/yum.repos.d/bak ] \u0026amp;\u0026amp; mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak/ cat \u0026gt; /etc/yum.repos.d/base.repo \u0026lt;\u0026lt;EOF [os_base] name=base packages baseurl=http://172.16.1.1:8080/anolis/BaseOS enabled=1 gpgcheck=0 [os_stream] name=stream packages baseurl=http://172.16.1.1:8080/anolis/AppStream enabled=1 gpgcheck=0 EOF fi } main(){ # 系统磁盘分区扩容 resize_os_partition # 系统分区扩容 create_rootvg_partition # 配置bond和ip net_setup_nmcli # 创建账号密码 create_users # 配置软件源 add_repos # 停用cloud-init systemctl disable cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service reboot } main ironic node推送示例 # 示例 nova boot --config-drive true \\ --user-data ${userdata_file} \\ --flavor ${flavor_id} \\ --image ${image_name} \\ --nic net-name=\u0026#34;${network_name}\u0026#34;,v4-fixed-ip=${instance_ip} \\ --availability-zone nova::${ironic_node_uuid} \\ ${instance_name} ","permalink":"https://gnail89.github.io/posts/ironic_image_userdata/","summary":"裸金属镜像配置 镜像制作 type: whole disk image\n准备 系统镜像ISO\nUEFI 引导\nqemu-kvm环境\n命名规范 区域-版本-架构-启动类型-文件系统-镜像大小: C01-AnolisOS-8.8-x86_64-UEFI-XFS-LVM-8G-IRONIC-yyymmdd.qcow2\n分区规划参考 镜像大小8GB, 可按需求进行分区, 后面配合userdata使用\nFS VG LV SIZE /boot - - 1 GB /boot/efi - - 600 MB / rootvg root 1 GB /usr rootvg usr 4 GB /var rootvg var 1 GB /home rootvg home 1 GB 软件安装 cloud-init\nqemu-guest-agent (可选)\ndnf install -y cloud-init qemu-guest-agent cloud-init配置文件更新 # 更新的内容 no_ssh_fingerprints: true disable_root: false ssh_pwauth: true network: config: disabled datasource_list: [OpenStack, ConfigDrive] datasource: OpenStack: metadata_urls: [\u0026#39;http://169.","title":"ironic image and userdata"},{"content":"环境 主机名 IP地址 备注 VIP 192.168.100.100 浮动IP k8s-master01 192.168.100.101 master节点 k8s-master02 192.168.100.102 master节点 k8s-master03 192.168.100.103 master节点 k8s-node01 192.168.100.104 node节点 k8s-node02 192.168.100.105 node节点 网段说明\nservice: 10.96.0.0/12 pod: 172.16.0.0/12 k8s基础系统环境配置 配置IP # 网卡UUID不能重复，UUID重复无法获取到IPV6地址 # # 查看当前的网卡列表和 UUID： # nmcli con show # 删除要更改 UUID 的网络连接： # nmcli con delete uuid \u0026lt;原 UUID\u0026gt; # 重新生成 UUID： # nmcli con add type ethernet ifname \u0026lt;接口名称\u0026gt; con-name \u0026lt;新名称\u0026gt; # 重新启用网络连接： # nmcli con up \u0026lt;新名称\u0026gt; # 更改网卡的UUID nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 # 修改静态的IPv4地址 nmcli con mod ens1 ipv4.addresses 192.168.100.101/24; nmcli con mod ens1 ipv4.gateway 192.168.100.1; nmcli con mod ens1 ipv4.method manual; nmcli con mod ens1 ipv4.dns \u0026#34;192.168.100.1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv4.addresses 192.168.100.102/24; nmcli con mod ens1 ipv4.gateway 192.168.100.1; nmcli con mod ens1 ipv4.method manual; nmcli con mod ens1 ipv4.dns \u0026#34;192.168.100.1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv4.addresses 192.168.100.103/24; nmcli con mod ens1 ipv4.gateway 192.168.100.1; nmcli con mod ens1 ipv4.method manual; nmcli con mod ens1 ipv4.dns \u0026#34;192.168.100.1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv4.addresses 192.168.100.104/24; nmcli con mod ens1 ipv4.gateway 192.168.100.1; nmcli con mod ens1 ipv4.method manual; nmcli con mod ens1 ipv4.dns \u0026#34;192.168.100.1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv4.addresses 192.168.100.105/24; nmcli con mod ens1 ipv4.gateway 192.168.100.1; nmcli con mod ens1 ipv4.method manual; nmcli con mod ens1 ipv4.dns \u0026#34;192.168.100.1\u0026#34;; nmcli con up ens1 # 没有IPv6不配置即可(没有测试IPv6) nmcli con mod ens1 ipv6.addresses fc00:43f4:1eea:1::10; nmcli con mod ens1 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod ens1 ipv6.method manual; nmcli con mod ens1 ipv6.dns \u0026#34;2400:3200::1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv6.addresses fc00:43f4:1eea:1::20; nmcli con mod ens1 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod ens1 ipv6.method manual; nmcli con mod ens1 ipv6.dns \u0026#34;2400:3200::1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv6.addresses fc00:43f4:1eea:1::30; nmcli con mod ens1 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod ens1 ipv6.method manual; nmcli con mod ens1 ipv6.dns \u0026#34;2400:3200::1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv6.addresses fc00:43f4:1eea:1::40; nmcli con mod ens1 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod ens1 ipv6.method manual; nmcli con mod ens1 ipv6.dns \u0026#34;2400:3200::1\u0026#34;; nmcli con up ens1 nmcli con mod ens1 ipv6.addresses fc00:43f4:1eea:1::50; nmcli con mod ens1 ipv6.gateway fc00:43f4:1eea:1::1; nmcli con mod ens1 ipv6.method manual; nmcli con mod ens1 ipv6.dns \u0026#34;2400:3200::1\u0026#34;; nmcli con up ens1 # 查看网卡配置 # nmcli device show ens1 # nmcli con show ens1 [root@localhost ~]# cat /etc/sysconfig/network-scripts/ifcfg-ens1 TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none IPADDR=192.168.100.101 PREFIX=24 GATEWAY=192.168.100.1 DNS1=192.168.100.1 DEFROUTE=yes IPV4_FAILURE_FATAL=no IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=ens1 UUID=a5078897-4aae-3fa8-8b7a-0773d924901e DEVICE=ens1 ONBOOT=yes AUTOCONNECT_PRIORITY=-999 IPV6ADDR=fc00:43f4:1eea:1::10/128 IPV6_DEFAULTGW=fc00:43f4:1eea:1::1 DNS2=2400:3200::1 设置主机名 hostnamectl set-hostname k8s-master01 hostnamectl set-hostname k8s-master02 hostnamectl set-hostname k8s-master03 hostnamectl set-hostname k8s-node01 hostnamectl set-hostname k8s-node02 配置yum源 sudo sed -e \u0026#39;s|^mirrorlist=|#mirrorlist=|g\u0026#39; \\ -e \u0026#39;s|^#baseurl=http://mirror.centos.org/$contentdir|baseurl=https://mirrors.tuna.tsinghua.edu.cn/centos|g\u0026#39; \\ -i.bak \\ /etc/yum.repos.d/CentOS-*.repo 安装一些软件(可选) yum update -y \u0026amp;\u0026amp; yum -y install wget psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git network-scripts tar curl 下载离线文件(可选) # 下载必要工具 yum -y install createrepo yum-utils wget epel-release elrepo-release # 下载全量依赖包 repotrack wget psmisc vim net-tools nfs-utils telnet yum-utils device-mapper-persistent-data lvm2 git network-scripts tar curl gcc keepalived haproxy bash-completion chrony sshpass ipvsadm ipset sysstat conntrack libseccomp # 创建yum源信息 createrepo -u -d /data/centos8/ # 拷贝包到内网机器上 scp -r centos8/ root@192.168.100.101: scp -r centos8/ root@192.168.100.102: scp -r centos8/ root@192.168.100.103: scp -r centos8/ root@192.168.100.104: scp -r centos8/ root@192.168.100.105: # 在内网机器上创建repo配置文件 rm -rf /etc/yum.repos.d/* cat \u0026gt; /etc/yum.repos.d/base.repo \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [base] name=CentOS-$releasever - Media baseurl=file:///root/centos8/ gpgcheck=0 enabled=1 EOF # 安装下载好的包 yum clean all yum makecache yum install -y /root/centos8/* 下载软件包(必要) #!/bin/bash # 查看版本地址： # # https://github.com/containernetworking/plugins/releases/ # https://github.com/containerd/containerd/releases/ # https://github.com/kubernetes-sigs/cri-tools/releases/ # https://github.com/Mirantis/cri-dockerd/releases/ # https://github.com/etcd-io/etcd/releases/ # https://github.com/cloudflare/cfssl/releases/ # https://github.com/kubernetes/kubernetes/tree/master/CHANGELOG # https://download.docker.com/linux/static/stable/x86_64/ # https://github.com/opencontainers/runc/releases/ # https://mirrors.tuna.tsinghua.edu.cn/elrepo/kernel/el7/x86_64/RPMS/ # https://github.com/helm/helm/tags # http://nginx.org/download/ # Version numbers cni_plugins_version=\u0026#39;v1.3.0\u0026#39; cri_containerd_cni_version=\u0026#39;1.7.6\u0026#39; crictl_version=\u0026#39;v1.28.0\u0026#39; cri_dockerd_version=\u0026#39;0.3.4\u0026#39; etcd_version=\u0026#39;v3.5.9\u0026#39; cfssl_version=\u0026#39;1.6.4\u0026#39; kubernetes_server_version=\u0026#39;1.27.6\u0026#39; docker_version=\u0026#39;24.0.6\u0026#39; runc_version=\u0026#39;1.1.9\u0026#39; kernel_version=\u0026#39;5.4.256\u0026#39; helm_version=\u0026#39;3.12.3\u0026#39; nginx_version=\u0026#39;1.25.2\u0026#39; # URLs base_url=\u0026#39;https://ghproxy.com/https://github.com\u0026#39; kernel_url=\u0026#34;http://mirrors.tuna.tsinghua.edu.cn/elrepo/kernel/el7/x86_64/RPMS/kernel-lt-${kernel_version}-1.el7.elrepo.x86_64.rpm\u0026#34; runc_url=\u0026#34;${base_url}/opencontainers/runc/releases/download/v${runc_version}/runc.amd64\u0026#34; docker_url=\u0026#34;https://download.docker.com/linux/static/stable/x86_64/docker-${docker_version}.tgz\u0026#34; cni_plugins_url=\u0026#34;${base_url}/containernetworking/plugins/releases/download/${cni_plugins_version}/cni-plugins-linux-amd64-${cni_plugins_version}.tgz\u0026#34; cri_containerd_cni_url=\u0026#34;${base_url}/containerd/containerd/releases/download/v${cri_containerd_cni_version}/cri-containerd-cni-${cri_containerd_cni_version}-linux-amd64.tar.gz\u0026#34; crictl_url=\u0026#34;${base_url}/kubernetes-sigs/cri-tools/releases/download/${crictl_version}/crictl-${crictl_version}-linux-amd64.tar.gz\u0026#34; cri_dockerd_url=\u0026#34;${base_url}/Mirantis/cri-dockerd/releases/download/v${cri_dockerd_version}/cri-dockerd-${cri_dockerd_version}.amd64.tgz\u0026#34; etcd_url=\u0026#34;${base_url}/etcd-io/etcd/releases/download/${etcd_version}/etcd-${etcd_version}-linux-amd64.tar.gz\u0026#34; cfssl_url=\u0026#34;${base_url}/cloudflare/cfssl/releases/download/v${cfssl_version}/cfssl_${cfssl_version}_linux_amd64\u0026#34; cfssljson_url=\u0026#34;${base_url}/cloudflare/cfssl/releases/download/v${cfssl_version}/cfssljson_${cfssl_version}_linux_amd64\u0026#34; helm_url=\u0026#34;https://files.m.daocloud.io/get.helm.sh/helm-v${helm_version}-linux-amd64.tar.gz\u0026#34; kubernetes_server_url=\u0026#34;https://dl.k8s.io/v${kubernetes_server_version}/kubernetes-server-linux-amd64.tar.gz\u0026#34; nginx_url=\u0026#34;http://nginx.org/download/nginx-${nginx_version}.tar.gz\u0026#34; # Download packages packages=( #$kernel_url $runc_url $docker_url $cni_plugins_url $cri_containerd_cni_url $crictl_url $cri_dockerd_url $etcd_url $cfssl_url $cfssljson_url $helm_url $kubernetes_server_url $nginx_url ) for package_url in \u0026#34;${packages[@]}\u0026#34;; do filename=$(basename \u0026#34;$package_url\u0026#34;) if wget -cq --progress=bar:force:noscroll -nc \u0026#34;$package_url\u0026#34;; then echo \u0026#34;Downloaded $filename\u0026#34; else echo \u0026#34;Failed to download $filename\u0026#34; exit 1 fi done 关闭防火墙 systemctl disable --now firewalld 关闭SELinux setenforce 0 sed -i \u0026#39;s#SELINUX=enforcing#SELINUX=disabled#g\u0026#39; /etc/selinux/config 关闭交换分区 sed -ri \u0026#39;s/.*swap.*/#\u0026amp;/\u0026#39; /etc/fstab swapoff -a \u0026amp;\u0026amp; sysctl -w vm.swappiness=0 cat /etc/fstab # /dev/mapper/centos-swap swap swap defaults 0 0 网络配置 # NetworkManager cat \u0026gt; /etc/NetworkManager/conf.d/calico.conf \u0026lt;\u0026lt; EOF [keyfile] unmanaged-devices=interface-name:cali*;interface-name:tunl* EOF systemctl restart NetworkManager 时间同步 yum install chrony -y systemctl enable --now chronyd chronyc sources -v 配置ulimit ulimit -SHn 65535 cat \u0026gt;\u0026gt; /etc/security/limits.conf \u0026lt;\u0026lt;EOF * soft nofile 655360 * hard nofile 655360 * soft nproc 655360 * hard nproc 655360 * soft memlock unlimited * hard memlock unlimitedd EOF 配置免密登录 yum install -y sshpass ssh-keygen -f /root/.ssh/id_rsa -P \u0026#39;\u0026#39; export IP=\u0026#34;192.168.100.101 192.168.100.102 192.168.100.103 192.168.100.104 192.168.100.105\u0026#34; export SSHPASS=12345678 for HOST in $IP;do sshpass -e ssh-copy-id -o StrictHostKeyChecking=no $HOST done cat \u0026gt; /root/.ssh/config \u0026lt;\u0026lt;EOF Host * StrictHostKeyChecking no EOF 添加elrepo源 yum install -y elrepo-release sed -i \u0026#34;s@mirrorlist@#mirrorlist@g\u0026#34; /etc/yum.repos.d/elrepo.repo sed -i \u0026#34;s@elrepo.org/linux@mirrors.tuna.tsinghua.edu.cn/elrepo@g\u0026#34; /etc/yum.repos.d/elrepo.repo # 查看可用安装包 yum --disablerepo=\u0026#34;*\u0026#34; --enablerepo=\u0026#34;elrepo-kernel\u0026#34; list available 升级内核至4.18版本以上 # 稳定版kernel-ml 长期维护版kernel-lt yum -y --enablerepo=elrepo-kernel install kernel-lt # 查看已安装那些内核 rpm -qa | grep kernel # 查看默认内核 grubby --default-kernel # 查看所有内核版本 grubby --info=ALL # 若不是最新的使用命令设置 grubby --set-default /boot/vmlinuz-5.4.256-1.el8.elrepo.x86_64 # 重启生效 init 6 # 或 systemctl --force reboot 安装ipvsadm yum install -y ipvsadm ipset sysstat conntrack libseccomp cat \u0026gt;\u0026gt; /etc/modules-load.d/ipvs.conf \u0026lt;\u0026lt;EOF ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack ip_tables ip_set xt_set ipt_set ipt_rpfilter ipt_REJECT ipip EOF systemctl restart systemd-modules-load.service lsmod | grep -e ip_vs -e nf_conntrack ip_vs_sh 16384 0 ip_vs_wrr 16384 0 ip_vs_rr 16384 16 ip_vs 176128 22 ip_vs_rr,ip_vs_sh,ip_vs_wrr nf_conntrack 159744 4 xt_conntrack,nf_nat,xt_MASQUERADE,ip_vs nf_defrag_ipv6 24576 2 nf_conntrack,ip_vs nf_defrag_ipv4 16384 1 nf_conntrack libcrc32c 16384 4 nf_conntrack,nf_nat,xfs,ip_vs 修改内核参数 cat \u0026lt;\u0026lt;EOF \u0026gt; /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 65536 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 net.ipv6.conf.all.disable_ipv6 = 0 net.ipv6.conf.default.disable_ipv6 = 0 net.ipv6.conf.lo.disable_ipv6 = 0 net.ipv6.conf.all.forwarding = 1 EOF sysctl --system 所有节点配置hosts本地解析 cat /etc/hosts 192.168.100.100 k8s-vip 192.168.100.101 k8s-master01 192.168.100.102 k8s-master02 192.168.100.103 k8s-master03 192.168.100.104 k8s-node01 192.168.100.105 k8s-node02 k8s基础组件安装 注意: docker 和 containerd 二选其一即可\n安装Containerd作为Runtime(方式一，推荐) # 参考链接 # https://github.com/containernetworking/plugins/releases/ # https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz #创建cni插件所需目录 mkdir -p /etc/cni/net.d /opt/cni/bin #解压cni二进制包 tar xf cni-plugins-linux-amd64-v*.tgz -C /opt/cni/bin/ # 参考链接 # https://github.com/containerd/containerd/releases/ # https://github.com/containerd/containerd/releases/download/v1.7.6/cri-containerd-cni-1.7.6-linux-amd64.tar.gz #解压containerd到根目录 tar -xzf cri-containerd-cni-*-linux-amd64.tar.gz -C / #创建服务启动文件 cat \u0026gt; /etc/systemd/system/containerd.service \u0026lt;\u0026lt;EOF [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/local/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=infinity TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target EOF 配置Containerd所需的模块 cat \u0026lt;\u0026lt;EOF | sudo tee /etc/modules-load.d/containerd.conf overlay br_netfilter EOF # 加载模块 systemctl restart systemd-modules-load.service 配置Containerd所需的内核 cat \u0026lt;\u0026lt;EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF # 加载内核 sysctl --system 创建Containerd的配置文件 # 创建默认配置文件 mkdir -p /etc/containerd containerd config default \u0026gt; /etc/containerd/config.toml # 启用systemd cgroup sed -i \u0026#34;s#SystemdCgroup\\ \\=\\ false#SystemdCgroup\\ \\=\\ true#g\u0026#34; /etc/containerd/config.toml cat /etc/containerd/config.toml | grep SystemdCgroup sed -i \u0026#34;s#registry.k8s.io#m.daocloud.io/registry.k8s.io#g\u0026#34; /etc/containerd/config.toml cat /etc/containerd/config.toml | grep sandbox_image sed -i \u0026#34;s#config_path\\ \\=\\ \\\u0026#34;\\\u0026#34;#config_path\\ \\=\\ \\\u0026#34;/etc/containerd/certs.d\\\u0026#34;#g\u0026#34; /etc/containerd/config.toml cat /etc/containerd/config.toml | grep certs.d mkdir -pv /etc/containerd/certs.d/docker.io # 配置国内加速 cat \u0026gt; /etc/containerd/certs.d/docker.io/hosts.toml \u0026lt;\u0026lt; EOF server = \u0026#34;https://docker.io\u0026#34; [host.\u0026#34;https://hub-mirror.c.163.com\u0026#34;] capabilities = [\u0026#34;pull\u0026#34;, \u0026#34;resolve\u0026#34;] EOF 启动并设置为开机启动 systemctl daemon-reload systemctl enable --now containerd systemctl restart containerd 配置crictl客户端连接的运行时位置 # https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz #解压 tar xf crictl-v*-linux-amd64.tar.gz -C /usr/bin/ #生成配置文件 cat \u0026gt; /etc/crictl.yaml \u0026lt;\u0026lt;EOF runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock timeout: 10 debug: false EOF #测试 systemctl restart containerd crictl info 安装docker作为Runtime(方式二) 安装docker # 二进制包下载地址：https://download.docker.com/linux/static/stable/x86_64/ # wget https://download.docker.com/linux/static/stable/x86_64/docker-24.0.6.tgz #解压 tar xf docker-*.tgz #拷贝二进制文件 cp docker/* /usr/bin/ #创建containerd的service文件 cat \u0026gt;/etc/systemd/system/containerd.service \u0026lt;\u0026lt;EOF [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target local-fs.target [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 LimitNPROC=infinity LimitCORE=infinity LimitNOFILE=1048576 TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target EOF # 设置开机自启 systemctl enable --now containerd.service #准备docker的service文件 cat \u0026gt; /etc/systemd/system/docker.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service containerd.service Wants=network-online.target Requires=docker.socket containerd.service [Service] Type=notify ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process OOMScoreAdjust=-500 [Install] WantedBy=multi-user.target EOF #准备docker的socket文件 cat \u0026gt; /etc/systemd/system/docker.socket \u0026lt;\u0026lt;EOF [Unit] Description=Docker Socket for the API [Socket] ListenStream=/var/run/docker.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target EOF #创建docker组 groupadd docker #启动docker systemctl enable --now docker.socket systemctl enable --now docker.service #验证 docker info # 配置国内加速器 mkdir -pv /etc/docker/ cat \u0026gt;/etc/docker/daemon.json \u0026lt;\u0026lt;EOF { \u0026#34;exec-opts\u0026#34;: [\u0026#34;native.cgroupdriver=systemd\u0026#34;], \u0026#34;registry-mirrors\u0026#34;: [ \u0026#34;https://docker.m.daocloud.io\u0026#34;, \u0026#34;https://docker.mirrors.ustc.edu.cn\u0026#34;, \u0026#34;http://hub-mirror.c.163.com\u0026#34; ], \u0026#34;max-concurrent-downloads\u0026#34;: 10, \u0026#34;log-driver\u0026#34;: \u0026#34;json-file\u0026#34;, \u0026#34;log-level\u0026#34;: \u0026#34;warn\u0026#34;, \u0026#34;log-opts\u0026#34;: { \u0026#34;max-size\u0026#34;: \u0026#34;10m\u0026#34;, \u0026#34;max-file\u0026#34;: \u0026#34;3\u0026#34; }, \u0026#34;data-root\u0026#34;: \u0026#34;/var/lib/docker\u0026#34; } EOF systemctl daemon-reload systemctl stop docker systemctl restart docker 安装cri-docker # 由于1.24以及更高版本不支持docker所以安装cri-docker # 下载cri-docker # wget https://ghproxy.com/https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd-0.3.4.amd64.tgz # 解压cri-docker tar xvf cri-dockerd-*.amd64.tgz cp -r cri-dockerd/ /usr/bin/ chmod +x /usr/bin/cri-dockerd/cri-dockerd # 写入启动配置文件 cat \u0026gt; /usr/lib/systemd/system/cri-docker.service \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd/cri-dockerd --network-plugin=cni --pod-infra-container-image=m.daocloud.io/registry.k8s.io/pause:3.8 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOF # 写入socket配置文件 cat \u0026gt; /usr/lib/systemd/system/cri-docker.socket \u0026lt;\u0026lt;EOF [Unit] Description=CRI Docker Socket for the API PartOf=cri-docker.service [Socket] ListenStream=%t/cri-dockerd.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target EOF # 进行启动cri-docker systemctl daemon-reload systemctl enable --now cri-docker systemctl restart cri-docker systemctl status cri-docker k8s与etcd安装(仅在master01操作) 解压k8s安装包 # 下载安装包 # wget https://dl.k8s.io/v1.27.6/kubernetes-server-linux-amd64.tar.gz # wget https://github.com/etcd-io/etcd/releases/download/v3.5.9/etcd-v3.5.9-linux-amd64.tar.gz # 解压k8s安装文件 tar -xf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} # 解压etcd安装文件 tar -xf etcd*.tar.gz \u0026amp;\u0026amp; mv etcd-*/etcd /usr/local/bin/ \u0026amp;\u0026amp; mv etcd-*/etcdctl /usr/local/bin/ # 查看/usr/local/bin下内容 ls /usr/local/bin/ containerd crictl etcdctl kube-proxy containerd-shim critest kube-apiserver kube-scheduler containerd-shim-runc-v1 ctd-decoder kube-controller-manager containerd-shim-runc-v2 ctr kubectl containerd-stress etcd kubelet 查看版本 # kubelet --version Kubernetes v1.27.6 # etcdctl version etcdctl version: 3.5.9 API version: 3.5 将组件复制到其他k8s节点 Master=\u0026#39;k8s-master02 k8s-master03\u0026#39; Work=\u0026#39;k8s-node01 k8s-node02\u0026#39; # 拷贝master组件 for NODE in $Master; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done # 拷贝work组件 for NODE in $Work; do scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done # 所有节点执行 mkdir -pv /opt/cni/bin 相关证书生成 # master01节点下载证书生成工具 # wget \u0026#34;https://ghproxy.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssl_1.6.4_linux_amd64\u0026#34; -O /usr/local/bin/cfssl # wget \u0026#34;https://ghproxy.com/https://github.com/cloudflare/cfssl/releases/download/v1.6.4/cfssljson_1.6.4_linux_amd64\u0026#34; -O /usr/local/bin/cfssljson # 软件包内有 cp cfssl_*_linux_amd64 /usr/local/bin/cfssl cp cfssljson_*_linux_amd64 /usr/local/bin/cfssljson # 添加执行权限 chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson 生成etcd证书 所有master节点创建证书存放目录 mkdir -pv /etc/etcd/ssl 在master01节点生成etcd证书 mkdir pki \u0026amp;\u0026amp; cd pki cat \u0026gt; etcd-ca-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;etcd\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;etcd\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Etcd Security\u0026#34; } ], \u0026#34;ca\u0026#34;: { \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; } } EOF cat \u0026gt; etcd-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;etcd\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;etcd\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Etcd Security\u0026#34; } ] } EOF cat \u0026gt; ca-config.json \u0026lt;\u0026lt;EOF { \u0026#34;signing\u0026#34;: { \u0026#34;default\u0026#34;: { \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; }, \u0026#34;profiles\u0026#34;: { \u0026#34;kubernetes\u0026#34;: { \u0026#34;usages\u0026#34;: [ \u0026#34;signing\u0026#34;, \u0026#34;key encipherment\u0026#34;, \u0026#34;server auth\u0026#34;, \u0026#34;client auth\u0026#34; ], \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; } } } } EOF # 生成etcd证书和etcd证书的key（如果你觉得以后可能会扩容，可以在ip那多写几个预留出来） # 如果没有IPv6 可删除可保留 cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /etc/etcd/ssl/etcd-ca cfssl gencert \\ -ca=/etc/etcd/ssl/etcd-ca.pem \\ -ca-key=/etc/etcd/ssl/etcd-ca-key.pem \\ -config=ca-config.json \\ -hostname=127.0.0.1,k8s-master01,k8s-master02,k8s-master03,192.168.100.101,192.168.100.102,192.168.100.103,fc00:43f4:1eea:1::10,fc00:43f4:1eea:1::20,fc00:43f4:1eea:1::30,::1 \\ -profile=kubernetes \\ etcd-csr.json | cfssljson -bare /etc/etcd/ssl/etcd 将证书复制到其他节点 Master=\u0026#39;k8s-master02 k8s-master03\u0026#39; for NODE in $Master; do ssh $NODE \u0026#34;mkdir -p /etc/etcd/ssl\u0026#34;; for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do scp /etc/etcd/ssl/${FILE} $NODE:/etc/etcd/ssl/${FILE}; done; done 生成k8s相关证书 所有k8s节点创建证书存放目录 mkdir -pv /etc/kubernetes/pki master01节点生成k8s证书 cat \u0026gt; ca-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;kubernetes\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;Kubernetes\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Kubernetes-manual\u0026#34; } ], \u0026#34;ca\u0026#34;: { \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; } } EOF cat \u0026gt; ca-config.json \u0026lt;\u0026lt;EOF { \u0026#34;signing\u0026#34;: { \u0026#34;default\u0026#34;: { \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; }, \u0026#34;profiles\u0026#34;: { \u0026#34;kubernetes\u0026#34;: { \u0026#34;usages\u0026#34;: [ \u0026#34;signing\u0026#34;, \u0026#34;key encipherment\u0026#34;, \u0026#34;server auth\u0026#34;, \u0026#34;client auth\u0026#34; ], \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; } } } } EOF cat \u0026gt; apiserver-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;kube-apiserver\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;Kubernetes\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Kubernetes-manual\u0026#34; } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare /etc/kubernetes/pki/ca # 生成一个k8s根证书 ，可以多写一些IP作为预留IP，为将来添加node做准备 # 10.96.0.1是service网段的第一个地址，需要计算，192.168.100.100为高可用vip地址 # 如果没有IPv6 可删除可保留 cfssl gencert \\ -ca=/etc/kubernetes/pki/ca.pem \\ -ca-key=/etc/kubernetes/pki/ca-key.pem \\ -config=ca-config.json \\ -hostname=10.96.0.1,192.168.100.100,127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,192.168.100.101,192.168.100.102,192.168.100.103,192.168.100.104,192.168.100.105,192.168.100.106,192.168.100.107,192.168.100.108,192.168.100.109,192.168.100.110,fc00:43f4:1eea:1::10,fc00:43f4:1eea:1::20,fc00:43f4:1eea:1::30,fc00:43f4:1eea:1::40,fc00:43f4:1eea:1::50,fc00:43f4:1eea:1::60,fc00:43f4:1eea:1::70,fc00:43f4:1eea:1::80,fc00:43f4:1eea:1::90,fc00:43f4:1eea:1::100,::1 \\ -profile=kubernetes apiserver-csr.json | cfssljson -bare /etc/kubernetes/pki/apiserver 生成apiserver聚合证书 cat \u0026gt; front-proxy-ca-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;kubernetes\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;ca\u0026#34;: { \u0026#34;expiry\u0026#34;: \u0026#34;876000h\u0026#34; } } EOF cat \u0026gt; front-proxy-client-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;front-proxy-client\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 } } EOF cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-ca # 有一个警告，可以忽略 cfssl gencert \\ -ca=/etc/kubernetes/pki/front-proxy-ca.pem \\ -ca-key=/etc/kubernetes/pki/front-proxy-ca-key.pem \\ -config=ca-config.json \\ -profile=kubernetes front-proxy-client-csr.json | cfssljson -bare /etc/kubernetes/pki/front-proxy-client 生成controller-manage的证书 在高可用配置部分选择使用哪种高可用方案 若使用 haproxy、keepalived 那么为 --server=https://192.168.100.100:9443 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443 cat \u0026gt; manager-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;system:kube-controller-manager\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;system:kube-controller-manager\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Kubernetes-manual\u0026#34; } ] } EOF cfssl gencert \\ -ca=/etc/kubernetes/pki/ca.pem \\ -ca-key=/etc/kubernetes/pki/ca-key.pem \\ -config=ca-config.json \\ -profile=kubernetes \\ manager-csr.json | cfssljson -bare /etc/kubernetes/pki/controller-manager # 设置一个集群项 kubectl config set-cluster kubernetes \\ --certificate-authority=/etc/kubernetes/pki/ca.pem \\ --embed-certs=true \\ --server=https://192.168.100.100:9443 \\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # 设置一个环境项，一个上下文 kubectl config set-context system:kube-controller-manager@kubernetes \\ --cluster=kubernetes \\ --user=system:kube-controller-manager \\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # 设置一个用户项 kubectl config set-credentials system:kube-controller-manager \\ --client-certificate=/etc/kubernetes/pki/controller-manager.pem \\ --client-key=/etc/kubernetes/pki/controller-manager-key.pem \\ --embed-certs=true \\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig # 设置默认环境 kubectl config use-context system:kube-controller-manager@kubernetes \\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig 在高可用配置部分选择使用哪种高可用方案 若使用 haproxy、keepalived 那么为 --server=https://192.168.100.100:9443 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443 cat \u0026gt; scheduler-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;system:kube-scheduler\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;system:kube-scheduler\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Kubernetes-manual\u0026#34; } ] } EOF cfssl gencert \\ -ca=/etc/kubernetes/pki/ca.pem \\ -ca-key=/etc/kubernetes/pki/ca-key.pem \\ -config=ca-config.json \\ -profile=kubernetes \\ scheduler-csr.json | cfssljson -bare /etc/kubernetes/pki/scheduler kubectl config set-cluster kubernetes \\ --certificate-authority=/etc/kubernetes/pki/ca.pem \\ --embed-certs=true \\ --server=https://192.168.100.100:9443 \\ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config set-credentials system:kube-scheduler \\ --client-certificate=/etc/kubernetes/pki/scheduler.pem \\ --client-key=/etc/kubernetes/pki/scheduler-key.pem \\ --embed-certs=true \\ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config set-context system:kube-scheduler@kubernetes \\ --cluster=kubernetes \\ --user=system:kube-scheduler \\ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig kubectl config use-context system:kube-scheduler@kubernetes \\ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig 在高可用配置部分选择使用哪种高可用方案 若使用 haproxy、keepalived 那么为 --server=https://192.168.100.100:9443 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443 cat \u0026gt; admin-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;admin\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;system:masters\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Kubernetes-manual\u0026#34; } ] } EOF cfssl gencert \\ -ca=/etc/kubernetes/pki/ca.pem \\ -ca-key=/etc/kubernetes/pki/ca-key.pem \\ -config=ca-config.json \\ -profile=kubernetes \\ admin-csr.json | cfssljson -bare /etc/kubernetes/pki/admin kubectl config set-cluster kubernetes \\ --certificate-authority=/etc/kubernetes/pki/ca.pem \\ --embed-certs=true \\ --server=https://192.168.100.100:9443 \\ --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config set-credentials kubernetes-admin \\ --client-certificate=/etc/kubernetes/pki/admin.pem \\ --client-key=/etc/kubernetes/pki/admin-key.pem \\ --embed-certs=true \\ --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config set-context kubernetes-admin@kubernetes \\ --cluster=kubernetes \\ --user=kubernetes-admin \\ --kubeconfig=/etc/kubernetes/admin.kubeconfig kubectl config use-context kubernetes-admin@kubernetes --kubeconfig=/etc/kubernetes/admin.kubeconfig 生成kube-proxy证书 在高可用配置部分选择使用哪种高可用方案 若使用 haproxy、keepalived 那么为 --server=https://192.168.100.100:9443 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443 cat \u0026gt; kube-proxy-csr.json \u0026lt;\u0026lt;EOF { \u0026#34;CN\u0026#34;: \u0026#34;system:kube-proxy\u0026#34;, \u0026#34;key\u0026#34;: { \u0026#34;algo\u0026#34;: \u0026#34;rsa\u0026#34;, \u0026#34;size\u0026#34;: 2048 }, \u0026#34;names\u0026#34;: [ { \u0026#34;C\u0026#34;: \u0026#34;CN\u0026#34;, \u0026#34;ST\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;L\u0026#34;: \u0026#34;Beijing\u0026#34;, \u0026#34;O\u0026#34;: \u0026#34;system:kube-proxy\u0026#34;, \u0026#34;OU\u0026#34;: \u0026#34;Kubernetes-manual\u0026#34; } ] } EOF cfssl gencert \\ -ca=/etc/kubernetes/pki/ca.pem \\ -ca-key=/etc/kubernetes/pki/ca-key.pem \\ -config=ca-config.json \\ -profile=kubernetes \\ kube-proxy-csr.json | cfssljson -bare /etc/kubernetes/pki/kube-proxy kubectl config set-cluster kubernetes \\ --certificate-authority=/etc/kubernetes/pki/ca.pem \\ --embed-certs=true \\ --server=https://192.168.100.100:9443 \\ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config set-credentials kube-proxy \\ --client-certificate=/etc/kubernetes/pki/kube-proxy.pem \\ --client-key=/etc/kubernetes/pki/kube-proxy-key.pem \\ --embed-certs=true \\ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config set-context kube-proxy@kubernetes \\ --cluster=kubernetes \\ --user=kube-proxy \\ --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig kubectl config use-context kube-proxy@kubernetes --kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig 生成ServiceAccount Key ——secret openssl genrsa -out /etc/kubernetes/pki/sa.key 2048 openssl rsa -in /etc/kubernetes/pki/sa.key -pubout -out /etc/kubernetes/pki/sa.pub 将证书发送到其他master节点 # 其他节点创建目录 # mkdir -pv /etc/kubernetes/pki/ for NODE in k8s-master02 k8s-master03; do for FILE in $(ls /etc/kubernetes/pki | grep -v etcd); do scp /etc/kubernetes/pki/${FILE} $NODE:/etc/kubernetes/pki/${FILE}; done; for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do scp /etc/kubernetes/${FILE} $NODE:/etc/kubernetes/${FILE}; done; done 查看证书 # ls /etc/kubernetes/pki/ admin.csr apiserver.pem controller-manager-key.pem front-proxy-ca.pem kube-proxy-key.pem scheduler-key.pem admin-key.pem ca.csr controller-manager.pem front-proxy-client.csr kube-proxy.pem scheduler.pem admin.pem ca-key.pem front-proxy-client-key.pem sa.key apiserver.csr ca.pem front-proxy-ca.csr front-proxy-client.pem sa.pub apiserver-key.pem controller-manager.csr front-proxy-ca-key.pem kube-proxy.csr scheduler.csr # 共26个 # ls /etc/kubernetes/pki/ |wc -l 26 k8s系统组件etcd配置 etcd配置 master01配置 # 如果要用IPv6那么把IPv4地址修改为IPv6即可 cat \u0026gt; /etc/etcd/etcd.config.yml \u0026lt;\u0026lt; EOF name: \u0026#39;k8s-master01\u0026#39; data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: \u0026#39;https://192.168.100.101:2380\u0026#39; listen-client-urls: \u0026#39;https://192.168.100.101:2379,http://127.0.0.1:2379\u0026#39; max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: \u0026#39;https://192.168.100.101:2380\u0026#39; advertise-client-urls: \u0026#39;https://192.168.100.101:2379\u0026#39; discovery: discovery-fallback: \u0026#39;proxy\u0026#39; discovery-proxy: discovery-srv: initial-cluster: \u0026#39;k8s-master01=https://192.168.100.101:2380,k8s-master02=https://192.168.100.102:2380,k8s-master03=https://192.168.100.103:2380\u0026#39; initial-cluster-token: \u0026#39;etcd-k8s-cluster\u0026#39; initial-cluster-state: \u0026#39;new\u0026#39; strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: \u0026#39;off\u0026#39; proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd.pem\u0026#39; key-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-key.pem\u0026#39; client-cert-auth: true trusted-ca-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-ca.pem\u0026#39; auto-tls: true peer-transport-security: cert-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd.pem\u0026#39; key-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-key.pem\u0026#39; peer-client-cert-auth: true trusted-ca-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-ca.pem\u0026#39; auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF master02配置 # 如果要用IPv6那么把IPv4地址修改为IPv6即可 cat \u0026gt; /etc/etcd/etcd.config.yml \u0026lt;\u0026lt; EOF name: \u0026#39;k8s-master02\u0026#39; data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: \u0026#39;https://192.168.100.102:2380\u0026#39; listen-client-urls: \u0026#39;https://192.168.100.102:2379,http://127.0.0.1:2379\u0026#39; max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: \u0026#39;https://192.168.100.102:2380\u0026#39; advertise-client-urls: \u0026#39;https://192.168.100.102:2379\u0026#39; discovery: discovery-fallback: \u0026#39;proxy\u0026#39; discovery-proxy: discovery-srv: initial-cluster: \u0026#39;k8s-master01=https://192.168.100.101:2380,k8s-master02=https://192.168.100.102:2380,k8s-master03=https://192.168.100.103:2380\u0026#39; initial-cluster-token: \u0026#39;etcd-k8s-cluster\u0026#39; initial-cluster-state: \u0026#39;new\u0026#39; strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: \u0026#39;off\u0026#39; proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd.pem\u0026#39; key-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-key.pem\u0026#39; client-cert-auth: true trusted-ca-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-ca.pem\u0026#39; auto-tls: true peer-transport-security: cert-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd.pem\u0026#39; key-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-key.pem\u0026#39; peer-client-cert-auth: true trusted-ca-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-ca.pem\u0026#39; auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF master03配置 # 如果要用IPv6那么把IPv4地址修改为IPv6即可 cat \u0026gt; /etc/etcd/etcd.config.yml \u0026lt;\u0026lt; EOF name: \u0026#39;k8s-master03\u0026#39; data-dir: /var/lib/etcd wal-dir: /var/lib/etcd/wal snapshot-count: 5000 heartbeat-interval: 100 election-timeout: 1000 quota-backend-bytes: 0 listen-peer-urls: \u0026#39;https://192.168.100.103:2380\u0026#39; listen-client-urls: \u0026#39;https://192.168.100.103:2379,http://127.0.0.1:2379\u0026#39; max-snapshots: 3 max-wals: 5 cors: initial-advertise-peer-urls: \u0026#39;https://192.168.100.103:2380\u0026#39; advertise-client-urls: \u0026#39;https://192.168.100.103:2379\u0026#39; discovery: discovery-fallback: \u0026#39;proxy\u0026#39; discovery-proxy: discovery-srv: initial-cluster: \u0026#39;k8s-master01=https://192.168.100.101:2380,k8s-master02=https://192.168.100.102:2380,k8s-master03=https://192.168.100.103:2380\u0026#39; initial-cluster-token: \u0026#39;etcd-k8s-cluster\u0026#39; initial-cluster-state: \u0026#39;new\u0026#39; strict-reconfig-check: false enable-v2: true enable-pprof: true proxy: \u0026#39;off\u0026#39; proxy-failure-wait: 5000 proxy-refresh-interval: 30000 proxy-dial-timeout: 1000 proxy-write-timeout: 5000 proxy-read-timeout: 0 client-transport-security: cert-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd.pem\u0026#39; key-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-key.pem\u0026#39; client-cert-auth: true trusted-ca-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-ca.pem\u0026#39; auto-tls: true peer-transport-security: cert-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd.pem\u0026#39; key-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-key.pem\u0026#39; peer-client-cert-auth: true trusted-ca-file: \u0026#39;/etc/kubernetes/pki/etcd/etcd-ca.pem\u0026#39; auto-tls: true debug: false log-package-levels: log-outputs: [default] force-new-cluster: false EOF 创建service(所有master节点操作) 创建etcd.service并启动 cat \u0026gt; /usr/lib/systemd/system/etcd.service \u0026lt;\u0026lt; EOF [Unit] Description=Etcd Service Documentation=https://coreos.com/etcd/docs/latest/ After=network.target [Service] Type=notify ExecStart=/usr/local/bin/etcd --config-file=/etc/etcd/etcd.config.yml Restart=on-failure RestartSec=10 LimitNOFILE=65535 [Install] WantedBy=multi-user.target Alias=etcd3.service EOF 创建etcd证书目录 mkdir /etc/kubernetes/pki/etcd ln -s /etc/etcd/ssl/* /etc/kubernetes/pki/etcd/ systemctl daemon-reload systemctl enable --now etcd 查看etcd状态 # 如果要用IPv6那么把IPv4地址修改为IPv6即可 export ETCDCTL_API=3 etcdctl --endpoints=\u0026#34;192.168.100.101:2379,192.168.100.102:2379,192.168.100.103:2379\u0026#34; --cacert=/etc/kubernetes/pki/etcd/etcd-ca.pem --cert=/etc/kubernetes/pki/etcd/etcd.pem --key=/etc/kubernetes/pki/etcd/etcd-key.pem endpoint stat --write-out=table +----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 192.168.100.101:2379 | c88a7ad9caa0caa0 | 3.5.9 | 10 MB | false | false | 6 | 1607415 | 1607415 | | | 192.168.100.102:2379 | 66a0add9c0721410 | 3.5.9 | 10 MB | true | false | 6 | 1607415 | 1607415 | | | 192.168.100.103:2379 | 5d94bf57d46a5d6f | 3.5.9 | 10 MB | false | false | 6 | 1607415 | 1607415 | | +----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ 高可用配置(在Master服务器上操作) 注意 nginx 和 haproxy+keepalived 方案二选一即可\n选择使用那种高可用方案，同时可以俩种都选用，比如： nginx方案实现集群内的高可用\nhaproxy、keepalived 方案实现集群外访问\n在此之前已经生成k8s相关证书 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443\n若使用 haproxy、keepalived 那么为 --server=https://192.168.100.100:9443\nnginx高可用方案 编译nginx # 安装编译环境 yum install gcc -y # 下载解压nginx二进制文件 # wget http://nginx.org/download/nginx-1.25.1.tar.gz tar xvf nginx-*.tar.gz cd nginx-* # 进行编译 ./configure --with-stream --without-http --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module make \u0026amp;\u0026amp; make install # 拷贝编译好的nginx node=\u0026#39;k8s-master02 k8s-master03 k8s-node01 k8s-node02\u0026#39; for NODE in $node; do scp -r /usr/local/nginx/ $NODE:/usr/local/nginx/; done 写入启动配置 在所有master主机上执行 # 写入nginx配置文件 cat \u0026gt; /usr/local/nginx/conf/kube-nginx.conf \u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; worker_processes 1; events { worker_connections 1024; } stream { upstream backend { least_conn; hash $remote_addr consistent; server 192.168.100.101:6443 max_fails=3 fail_timeout=30s; server 192.168.100.102:6443 max_fails=3 fail_timeout=30s; server 192.168.100.103:6443 max_fails=3 fail_timeout=30s; } server { listen 127.0.0.1:8443; proxy_connect_timeout 1s; proxy_pass backend; } } EOF # 写入启动配置文件 cat \u0026gt; /etc/systemd/system/kube-nginx.service \u0026lt;\u0026lt;EOF [Unit] Description=kube-apiserver nginx proxy After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking ExecStartPre=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -t ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx ExecReload=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/kube-nginx.conf -p /usr/local/nginx -s reload PrivateTmp=true Restart=always RestartSec=5 StartLimitInterval=0 LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF # 设置开机自启 systemctl enable --now kube-nginx systemctl restart kube-nginx systemctl status kube-nginx keepalived和haproxy高可用方案 安装keepalived和haproxy服务 systemctl disable --now firewalld setenforce 0 sed -i \u0026#39;s#SELINUX=enforcing#SELINUX=disabled#g\u0026#39; /etc/selinux/config yum -y install keepalived haproxy 修改haproxy配置文件(所有master节点配置文件一样) # cp /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.bak cat \u0026gt;/etc/haproxy/haproxy.cfg\u0026lt;\u0026lt;\u0026#34;EOF\u0026#34; global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-in bind *:33305 mode http option httplog monitor-uri /monitor frontend k8s-master bind 0.0.0.0:9443 bind 127.0.0.1:9443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend k8s-master backend k8s-master mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server k8s-master01 192.168.100.101:6443 check server k8s-master02 192.168.100.102:6443 check server k8s-master03 192.168.100.103:6443 check EOF master01配置keepalived master节点 #cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak cat \u0026gt; /etc/keepalived/keepalived.conf \u0026lt;\u0026lt; EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script \u0026#34;/etc/keepalived/check_apiserver.sh\u0026#34; interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state MASTER # 注意网卡名 interface ens1 mcast_src_ip 192.168.100.101 virtual_router_id 51 priority 100 nopreempt advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.100.100 } track_script { chk_apiserver } } EOF master02配置keepalived backup节点 # cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak cat \u0026gt; /etc/keepalived/keepalived.conf \u0026lt;\u0026lt; EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script \u0026#34;/etc/keepalived/check_apiserver.sh\u0026#34; interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP # 注意网卡名 interface ens1 mcast_src_ip 192.168.100.102 virtual_router_id 51 priority 80 nopreempt advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.100.100 } track_script { chk_apiserver } } EOF master03配置keepalived backup节点 # cp /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak cat \u0026gt; /etc/keepalived/keepalived.conf \u0026lt;\u0026lt; EOF ! Configuration File for keepalived global_defs { router_id LVS_DEVEL } vrrp_script chk_apiserver { script \u0026#34;/etc/keepalived/check_apiserver.sh\u0026#34; interval 5 weight -5 fall 2 rise 1 } vrrp_instance VI_1 { state BACKUP # 注意网卡名 interface ens1 mcast_src_ip 192.168.100.103 virtual_router_id 51 priority 50 nopreempt advert_int 2 authentication { auth_type PASS auth_pass K8SHA_KA_AUTH } virtual_ipaddress { 192.168.100.100 } track_script { chk_apiserver } } EOF 配置健康检查脚本(keepalived主机上配置) cat \u0026gt; /etc/keepalived/check_apiserver.sh \u0026lt;\u0026lt; EOF #!/bin/bash err=0 for k in \\$(seq 1 3) do check_code=\\$(pgrep haproxy) if [[ \\$check_code == \u0026#34;\u0026#34; ]]; then err=\\$(expr \\$err + 1) sleep 1 continue else err=0 break fi done if [[ \\$err != \u0026#34;0\u0026#34; ]]; then echo \u0026#34;systemctl stop keepalived\u0026#34; /usr/bin/systemctl stop keepalived exit 1 else exit 0 fi EOF # 给脚本执行权限 chmod +x /etc/keepalived/check_apiserver.sh 启动服务 systemctl daemon-reload systemctl enable --now haproxy systemctl enable --now keepalived 测试高可用 # 能ping通 # ping 192.168.100.100 # 能telnet访问 # telnet 192.168.100.100 9443 # 关闭主节点，看vip是否漂移到备节点 k8s组件配置 所有k8s节点创建以下目录 mkdir -p /etc/kubernetes/manifests/ /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes 创建apiserver(所有master节点) master01节点配置 cat \u0026gt; /usr/lib/systemd/system/kube-apiserver.service \u0026lt;\u0026lt;EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \\\\ --v=2 \\\\ --allow-privileged=true \\\\ --bind-address=0.0.0.0 \\\\ --secure-port=6443 \\\\ --advertise-address=192.168.100.101 \\\\ --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\\\ --service-node-port-range=30000-32767 \\\\ --etcd-servers=https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379 \\\\ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \\\\ --etcd-certfile=/etc/etcd/ssl/etcd.pem \\\\ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\\\ --client-ca-file=/etc/kubernetes/pki/ca.pem \\\\ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \\\\ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \\\\ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \\\\ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \\\\ --service-account-key-file=/etc/kubernetes/pki/sa.pub \\\\ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \\\\ --service-account-issuer=https://kubernetes.default.svc.cluster.local \\\\ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \\\\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \\ --authorization-mode=Node,RBAC \\\\ --enable-bootstrap-token-auth=true \\\\ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \\\\ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \\\\ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \\\\ --requestheader-allowed-names=aggregator \\\\ --requestheader-group-headers=X-Remote-Group \\\\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\\\ --requestheader-username-headers=X-Remote-User \\\\ --enable-aggregator-routing=true Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF master02节点配置 cat \u0026gt; /usr/lib/systemd/system/kube-apiserver.service \u0026lt;\u0026lt;EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \\\\ --v=2 \\\\ --allow-privileged=true \\\\ --bind-address=0.0.0.0 \\\\ --secure-port=6443 \\\\ --advertise-address=192.168.100.102 \\\\ --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\\\ --service-node-port-range=30000-32767 \\\\ --etcd-servers=https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379 \\\\ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \\\\ --etcd-certfile=/etc/etcd/ssl/etcd.pem \\\\ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\\\ --client-ca-file=/etc/kubernetes/pki/ca.pem \\\\ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \\\\ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \\\\ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \\\\ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \\\\ --service-account-key-file=/etc/kubernetes/pki/sa.pub \\\\ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \\\\ --service-account-issuer=https://kubernetes.default.svc.cluster.local \\\\ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \\\\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \\ --authorization-mode=Node,RBAC \\\\ --enable-bootstrap-token-auth=true \\\\ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \\\\ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \\\\ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \\\\ --requestheader-allowed-names=aggregator \\\\ --requestheader-group-headers=X-Remote-Group \\\\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\\\ --requestheader-username-headers=X-Remote-User \\\\ --enable-aggregator-routing=true Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF master03节点配置 cat \u0026gt; /usr/lib/systemd/system/kube-apiserver.service \u0026lt;\u0026lt;EOF [Unit] Description=Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \\\\ --v=2 \\\\ --allow-privileged=true \\\\ --bind-address=0.0.0.0 \\\\ --secure-port=6443 \\\\ --advertise-address=192.168.100.103 \\\\ --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\\\ --service-node-port-range=30000-32767 \\\\ --etcd-servers=https://192.168.100.101:2379,https://192.168.100.102:2379,https://192.168.100.103:2379 \\\\ --etcd-cafile=/etc/etcd/ssl/etcd-ca.pem \\\\ --etcd-certfile=/etc/etcd/ssl/etcd.pem \\\\ --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\\\ --client-ca-file=/etc/kubernetes/pki/ca.pem \\\\ --tls-cert-file=/etc/kubernetes/pki/apiserver.pem \\\\ --tls-private-key-file=/etc/kubernetes/pki/apiserver-key.pem \\\\ --kubelet-client-certificate=/etc/kubernetes/pki/apiserver.pem \\\\ --kubelet-client-key=/etc/kubernetes/pki/apiserver-key.pem \\\\ --service-account-key-file=/etc/kubernetes/pki/sa.pub \\\\ --service-account-signing-key-file=/etc/kubernetes/pki/sa.key \\\\ --service-account-issuer=https://kubernetes.default.svc.cluster.local \\\\ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \\\\ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \\ --authorization-mode=Node,RBAC \\\\ --enable-bootstrap-token-auth=true \\\\ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem \\\\ --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.pem \\\\ --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client-key.pem \\\\ --requestheader-allowed-names=aggregator \\\\ --requestheader-group-headers=X-Remote-Group \\\\ --requestheader-extra-headers-prefix=X-Remote-Extra- \\\\ --requestheader-username-headers=X-Remote-User \\\\ --enable-aggregator-routing=true Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF 启动apiserver(所有master节点) systemctl daemon-reload systemctl enable --now kube-apiserver systemctl restart kube-apiserver systemctl status kube-apiserver 配置kube-controller-manager service # 所有master节点配置，且配置相同 # 172.16.0.0/12为pod网段，按需求设置你自己的网段 cat \u0026gt; /usr/lib/systemd/system/kube-controller-manager.service \u0026lt;\u0026lt; EOF [Unit] Description=Kubernetes Controller Manager Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-controller-manager \\\\ --v=2 \\\\ --bind-address=0.0.0.0 \\\\ --root-ca-file=/etc/kubernetes/pki/ca.pem \\\\ --cluster-signing-cert-file=/etc/kubernetes/pki/ca.pem \\\\ --cluster-signing-key-file=/etc/kubernetes/pki/ca-key.pem \\\\ --service-account-private-key-file=/etc/kubernetes/pki/sa.key \\\\ --kubeconfig=/etc/kubernetes/controller-manager.kubeconfig \\\\ --leader-elect=true \\\\ --use-service-account-credentials=true \\\\ --node-monitor-grace-period=40s \\\\ --node-monitor-period=5s \\\\ --controllers=*,bootstrapsigner,tokencleaner \\\\ --allocate-node-cidrs=true \\\\ --service-cluster-ip-range=10.96.0.0/12,fd00:1111::/112 \\\\ --cluster-cidr=172.16.0.0/12,fc00:2222::/112 \\\\ --node-cidr-mask-size-ipv4=24 \\\\ --node-cidr-mask-size-ipv6=120 \\\\ --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem Restart=always RestartSec=10s [Install] WantedBy=multi-user.target EOF 启动kube-controller-manager systemctl daemon-reload systemctl enable --now kube-controller-manager systemctl restart kube-controller-manager systemctl status kube-controller-manager 配置kube-scheduler service 所有master节点配置，且配置相同 cat \u0026gt; /usr/lib/systemd/system/kube-scheduler.service \u0026lt;\u0026lt; EOF [Unit] Description=Kubernetes Scheduler Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-scheduler \\\\ --v=2 \\\\ --bind-address=0.0.0.0 \\\\ --leader-elect=true \\\\ --kubeconfig=/etc/kubernetes/scheduler.kubeconfig Restart=always RestartSec=10s [Install] WantedBy=multi-user.target EOF 启动并查看服务状态 systemctl daemon-reload systemctl enable --now kube-scheduler systemctl restart kube-scheduler systemctl status kube-scheduler TLS Bootstrapping配置 在master01上配置 在高可用配置部分选择使用哪种高可用方案 若使用 haproxy、keepalived 那么为 --server=https://192.168.100.100:9443 若使用 nginx方案，那么为 --server=https://127.0.0.1:8443 cd bootstrap cat \u0026gt; bootstrap.secret.yaml \u0026lt;\u0026lt;EOF apiVersion: v1 kind: Secret metadata: name: bootstrap-token-c8ad9c namespace: kube-system type: bootstrap.kubernetes.io/token stringData: description: \u0026#34;The default bootstrap token generated by \u0026#39;kubelet \u0026#39;.\u0026#34; token-id: c8ad9c token-secret: 2e4d610cf3e7426e usage-bootstrap-authentication: \u0026#34;true\u0026#34; usage-bootstrap-signing: \u0026#34;true\u0026#34; auth-extra-groups: system:bootstrappers:default-node-token,system:bootstrappers:worker,system:bootstrappers:ingress --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubelet-bootstrap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:node-bootstrapper subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:default-node-token --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-autoapprove-bootstrap roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:nodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:bootstrappers:default-node-token --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: node-autoapprove-certificate-rotation roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient subjects: - apiGroup: rbac.authorization.k8s.io kind: Group name: system:nodes --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: rbac.authorization.kubernetes.io/autoupdate: \u0026#34;true\u0026#34; labels: kubernetes.io/bootstrapping: rbac-defaults name: system:kube-apiserver-to-kubelet rules: - apiGroups: - \u0026#34;\u0026#34; resources: - nodes/proxy - nodes/stats - nodes/log - nodes/spec - nodes/metrics verbs: - \u0026#34;*\u0026#34; --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: system:kube-apiserver namespace: \u0026#34;\u0026#34; roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:kube-apiserver-to-kubelet subjects: - apiGroup: rbac.authorization.k8s.io kind: User name: kube-apiserver EOF kubectl config set-cluster kubernetes \\ --certificate-authority=/etc/kubernetes/pki/ca.pem \\ --embed-certs=true --server=https://127.0.0.1:8443 \\ --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-credentials tls-bootstrap-token-user \\ --token=c8ad9c.2e4d610cf3e7426e \\ --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config set-context tls-bootstrap-token-user@kubernetes \\ --cluster=kubernetes \\ --user=tls-bootstrap-token-user \\ --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig kubectl config use-context tls-bootstrap-token-user@kubernetes \\ --kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig # token的位置在bootstrap.secret.yaml，如果修改的话到这个文件修改 mkdir -pv /root/.kube ; cp /etc/kubernetes/admin.kubeconfig /root/.kube/config # 查看集群状态，没问题的话继续后续操作 # kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-1 Healthy etcd-0 Healthy etcd-2 Healthy # 集群正常后执行以下操作 kubectl create -f bootstrap.secret.yaml node节点配置 在master01上将证书复制到node节点 cd /etc/kubernetes/ for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do ssh $NODE mkdir -p /etc/kubernetes/pki; for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig kube-proxy.kubeconfig; do scp /etc/kubernetes/$FILE $NODE:/etc/kubernetes/${FILE}; done; done kubelet配置 当使用docker作为Runtime cat \u0026gt; /usr/lib/systemd/system/kubelet.service \u0026lt;\u0026lt;EOF [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes [Service] ExecStart=/usr/local/bin/kubelet \\\\ --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \\\\ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\\ --config=/etc/kubernetes/kubelet-conf.yml \\\\ --container-runtime-endpoint=unix:///run/cri-dockerd.sock \\\\ --node-labels=node.kubernetes.io/node= [Install] WantedBy=multi-user.target EOF 当使用Containerd作为Runtime (推荐) mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/ # 所有k8s节点配置kubelet service cat \u0026gt; /usr/lib/systemd/system/kubelet.service \u0026lt;\u0026lt; EOF [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=containerd.service Requires=containerd.service [Service] ExecStart=/usr/local/bin/kubelet \\\\ --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.kubeconfig \\\\ --kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\\\ --config=/etc/kubernetes/kubelet-conf.yml \\\\ --container-runtime-endpoint=unix:///run/containerd/containerd.sock \\\\ --node-labels=node.kubernetes.io/node= [Install] WantedBy=multi-user.target EOF 所有k8s节点创建kubelet的配置文件 cat \u0026gt; /etc/kubernetes/kubelet-conf.yml \u0026lt;\u0026lt;EOF apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: 0.0.0.0 port: 10250 readOnlyPort: 10255 authentication: anonymous: enabled: false webhook: cacheTTL: 2m0s enabled: true x509: clientCAFile: /etc/kubernetes/pki/ca.pem authorization: mode: Webhook webhook: cacheAuthorizedTTL: 5m0s cacheUnauthorizedTTL: 30s cgroupDriver: systemd cgroupsPerQOS: true clusterDNS: - 10.96.0.10 clusterDomain: cluster.local containerLogMaxFiles: 5 containerLogMaxSize: 10Mi contentType: application/vnd.kubernetes.protobuf cpuCFSQuota: true cpuManagerPolicy: none cpuManagerReconcilePeriod: 10s enableControllerAttachDetach: true enableDebuggingHandlers: true enforceNodeAllocatable: - pods eventBurst: 10 eventRecordQPS: 5 evictionHard: imagefs.available: 15% memory.available: 100Mi nodefs.available: 10% nodefs.inodesFree: 5% evictionPressureTransitionPeriod: 5m0s failSwapOn: true fileCheckFrequency: 20s hairpinMode: promiscuous-bridge healthzBindAddress: 127.0.0.1 healthzPort: 10248 httpCheckFrequency: 20s imageGCHighThresholdPercent: 85 imageGCLowThresholdPercent: 80 imageMinimumGCAge: 2m0s iptablesDropBit: 15 iptablesMasqueradeBit: 14 kubeAPIBurst: 10 kubeAPIQPS: 5 makeIPTablesUtilChains: true maxOpenFiles: 1000000 maxPods: 110 nodeStatusUpdateFrequency: 10s oomScoreAdj: -999 podPidsLimit: -1 registryBurst: 10 registryPullQPS: 5 resolvConf: /etc/resolv.conf rotateCertificates: true runtimeRequestTimeout: 2m0s serializeImagePulls: true staticPodPath: /etc/kubernetes/manifests streamingConnectionIdleTimeout: 4h0m0s syncFrequency: 1m0s volumeStatsAggPeriod: 1m0s EOF 启动kubelet systemctl daemon-reload systemctl enable --now kubelet systemctl restart kubelet systemctl status kubelet 查看集群 # kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 Ready \u0026lt;none\u0026gt; 13d v1.27.6 k8s-master02 Ready \u0026lt;none\u0026gt; 13d v1.27.6 k8s-master03 Ready \u0026lt;none\u0026gt; 13d v1.27.6 k8s-node01 Ready \u0026lt;none\u0026gt; 13d v1.27.6 k8s-node02 Ready \u0026lt;none\u0026gt; 13d v1.27.6 查看容器运行时 # containerd # kubectl describe node | grep Runtime Container Runtime Version: containerd://1.7.6 Container Runtime Version: containerd://1.7.6 Container Runtime Version: containerd://1.7.6 Container Runtime Version: containerd://1.7.6 Container Runtime Version: containerd://1.7.6 # docker # kubectl describe node | grep Runtime Container Runtime Version: docker://24.0.2 Container Runtime Version: docker://24.0.2 Container Runtime Version: docker://24.0.2 Container Runtime Version: docker://24.0.2 Container Runtime Version: docker://24.0.2 kube-proxy配置 将kubeconfig发送至其他节点 for NODE in k8s-master02 k8s-master03 k8s-node01 k8s-node02; do scp /etc/kubernetes/kube-proxy.kubeconfig $NODE:/etc/kubernetes/kube-proxy.kubeconfig; done 所有k8s节点添加kube-proxy的service文件 cat \u0026gt; /usr/lib/systemd/system/kube-proxy.service \u0026lt;\u0026lt; EOF [Unit] Description=Kubernetes Kube Proxy Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-proxy \\\\ --config=/etc/kubernetes/kube-proxy.yaml \\\\ --v=2 Restart=always RestartSec=10s [Install] WantedBy=multi-user.target EOF 所有k8s节点添加kube-proxy的配置 cat \u0026gt; /etc/kubernetes/kube-proxy.yaml \u0026lt;\u0026lt; EOF apiVersion: kubeproxy.config.k8s.io/v1alpha1 bindAddress: 0.0.0.0 clientConnection: acceptContentTypes: \u0026#34;\u0026#34; burst: 10 contentType: application/vnd.kubernetes.protobuf kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig qps: 5 clusterCIDR: 172.16.0.0/12,fc00:2222::/112 configSyncPeriod: 15m0s conntrack: max: null maxPerCore: 32768 min: 131072 tcpCloseWaitTimeout: 1h0m0s tcpEstablishedTimeout: 24h0m0s enableProfiling: false healthzBindAddress: 0.0.0.0:10256 hostnameOverride: \u0026#34;\u0026#34; iptables: masqueradeAll: false masqueradeBit: 14 minSyncPeriod: 0s syncPeriod: 30s ipvs: masqueradeAll: true minSyncPeriod: 5s scheduler: \u0026#34;rr\u0026#34; syncPeriod: 30s kind: KubeProxyConfiguration metricsBindAddress: 127.0.0.1:10249 mode: \u0026#34;ipvs\u0026#34; nodePortAddresses: null oomScoreAdj: -999 portRange: \u0026#34;\u0026#34; udpIdleTimeout: 250ms EOF 启动kube-proxy systemctl daemon-reload systemctl restart kube-proxy systemctl enable --now kube-proxy systemctl status kube-proxy 安装网络插件 注意: 网络插件calico和cilium二选一即可 升级runc(可选) # https://github.com/opencontainers/runc/releases # 升级runc # wget https://ghproxy.com/https://github.com/opencontainers/runc/releases/download/v1.1.4/runc.amd64 install -m 755 runc.amd64 /usr/local/sbin/runc cp -p /usr/local/sbin/runc /usr/local/bin/runc cp -p /usr/local/sbin/runc /usr/bin/runc 安装Calico 更改calico网段 wget https://ghproxy.com/https://github.com/projectcalico/calico/blob/master/manifests/calico-typha.yaml cp calico-typha.yaml calico.yaml cp calico-typha.yaml calico-ipv6.yaml # vi calico.yaml # 需要更新以下配置： \u0026#34;ipam\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;calico-ipam\u0026#34;, }, - name: IP value: \u0026#34;autodetect\u0026#34; - name: CALICO_IPV4POOL_CIDR value: \u0026#34;172.16.0.0/12\u0026#34; # vi calico-ipv6.yaml # 需要更新以下配置： \u0026#34;ipam\u0026#34;: { \u0026#34;type\u0026#34;: \u0026#34;calico-ipam\u0026#34;, \u0026#34;assign_ipv4\u0026#34;: \u0026#34;true\u0026#34;, \u0026#34;assign_ipv6\u0026#34;: \u0026#34;true\u0026#34; }, - name: IP value: \u0026#34;autodetect\u0026#34; - name: IP6 value: \u0026#34;autodetect\u0026#34; - name: CALICO_IPV4POOL_CIDR value: \u0026#34;172.16.0.0/12\u0026#34; - name: CALICO_IPV6POOL_CIDR value: \u0026#34;fc00::/48\u0026#34; - name: FELIX_IPV6SUPPORT value: \u0026#34;true\u0026#34; # 若docker镜像拉不下来，可以使用国内的仓库 sed -i \u0026#34;s#docker.io/calico/#m.daocloud.io/docker.io/calico/#g\u0026#34; calico.yaml sed -i \u0026#34;s#docker.io/calico/#m.daocloud.io/docker.io/calico/#g\u0026#34; calico-ipv6.yaml # 本地没有公网 IPv6 使用 calico.yaml kubectl apply -f calico.yaml # 本地有公网 IPv6 使用 calico-ipv6.yaml kubectl apply -f calico-ipv6.yaml 查看容器状态 # calico 初始化会很慢 需要耐心等待一下 # kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-76754ff848-gmx8n 1/1 Running 3 (29h ago) 13d kube-system calico-node-845jj 1/1 Running 2 (47h ago) 13d kube-system calico-node-m7p9n 1/1 Running 2 (47h ago) 13d kube-system calico-node-qqj7g 1/1 Running 2 (47h ago) 13d kube-system calico-node-rfq9m 1/1 Running 4 (11m ago) 13d kube-system calico-node-tdnmt 1/1 Running 3 (29h ago) 13d kube-system calico-typha-59d75c5dd4-8fg4d 1/1 Running 2 (47h ago) 13d 安装cilium(示例) 安装helm(示例) # curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 # chmod 700 get_helm.sh # ./get_helm.sh wget https://files.m.daocloud.io/get.helm.sh/helm-v3.12.1-linux-amd64.tar.gz tar xvf helm-*-linux-amd64.tar.gz cp linux-amd64/helm /usr/local/bin/ 安装(示例) # 添加源 helm repo add cilium https://helm.cilium.io # 修改为国内源 helm pull cilium/cilium tar xvf cilium-*.tgz cd cilium/ sed -i \u0026#34;s#quay.io/#m.daocloud.io/quay.io/#g\u0026#34; values.yaml # 默认参数安装 helm install cilium ./cilium/ -n kube-system # 启用ipv6 # helm install cilium cilium/cilium --namespace kube-system --set ipv6.enabled=true # 启用路由信息和监控插件 # helm install cilium cilium/cilium --namespace kube-system --set hubble.relay.enabled=true --set hubble.ui.enabled=true --set prometheus.enabled=true --set operator.prometheus.enabled=true --set hubble.enabled=true --set hubble.metrics.enabled=\u0026#34;{dns,drop,tcp,flow,port-distribution,icmp,http}\u0026#34; 查看(示例) # kubectl get pod -A | grep cilium kube-system cilium-gmr6c 1/1 Running 0 5m3s kube-system cilium-kzgdj 1/1 Running 0 5m3s kube-system cilium-operator-69b677f97c-6pw4k 1/1 Running 0 5m3s kube-system cilium-operator-69b677f97c-xzzdk 1/1 Running 0 5m3s kube-system cilium-q2rnr 1/1 Running 0 5m3s kube-system cilium-smx5v 1/1 Running 0 5m3s kube-system cilium-tdjq4 1/1 Running 0 5m3s 下载专属监控面板(示例) # wget https://ghproxy.com/https://raw.githubusercontent.com/cilium/cilium/1.12.1/examples/kubernetes/addons/prometheus/monitoring-example.yaml # sed -i \u0026#34;s#docker.io/#m.daocloud.io/docker.io/#g\u0026#34; monitoring-example.yaml # kubectl apply -f monitoring-example.yaml namespace/cilium-monitoring created serviceaccount/prometheus-k8s created configmap/grafana-config created configmap/grafana-cilium-dashboard created configmap/grafana-cilium-operator-dashboard created configmap/grafana-hubble-dashboard created configmap/prometheus created clusterrole.rbac.authorization.k8s.io/prometheus created clusterrolebinding.rbac.authorization.k8s.io/prometheus created service/grafana created service/prometheus created deployment.apps/grafana created deployment.apps/prometheus created 下载部署测试用例(示例) 说明 测试用例 需要在 安装CoreDNS 之后完成 # wget https://ghproxy.com/https://raw.githubusercontent.com/cilium/cilium/master/examples/kubernetes/connectivity-check/connectivity-check.yaml # sed -i \u0026#34;s#google.com#baidu.cn#g\u0026#34; connectivity-check.yaml sed -i \u0026#34;s#quay.io/#m.daocloud.io/quay.io/#g\u0026#34; connectivity-check.yaml # kubectl apply -f connectivity-check.yaml deployment.apps/echo-a created deployment.apps/echo-b created deployment.apps/echo-b-host created deployment.apps/pod-to-a created deployment.apps/pod-to-external-1111 created deployment.apps/pod-to-a-denied-cnp created deployment.apps/pod-to-a-allowed-cnp created deployment.apps/pod-to-external-fqdn-allow-google-cnp created deployment.apps/pod-to-b-multi-node-clusterip created deployment.apps/pod-to-b-multi-node-headless created deployment.apps/host-to-b-multi-node-clusterip created deployment.apps/host-to-b-multi-node-headless created deployment.apps/pod-to-b-multi-node-nodeport created deployment.apps/pod-to-b-intra-node-nodeport created service/echo-a created service/echo-b created service/echo-b-headless created service/echo-b-host-headless created ciliumnetworkpolicy.cilium.io/pod-to-a-denied-cnp created ciliumnetworkpolicy.cilium.io/pod-to-a-allowed-cnp created ciliumnetworkpolicy.cilium.io/pod-to-external-fqdn-allow-google-cnp created 查看pod(示例) # kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE cilium-monitoring grafana-59957b9549-6zzqh 1/1 Running 0 10m cilium-monitoring prometheus-7c8c9684bb-4v9cl 1/1 Running 0 10m default test-75b5d7fbfb-7zjsr 1/1 Running 0 27h default test-75b5d7fbfb-hbvr8 1/1 Running 0 27h default test-75b5d7fbfb-ppbzg 1/1 Running 0 27h default echo-a-6799dff547-pnx6w 1/1 Running 0 10m default echo-b-fc47b659c-4bdg9 1/1 Running 0 10m default echo-b-host-67fcfd59b7-28r9s 1/1 Running 0 10m default host-to-b-multi-node-clusterip-69c57975d6-z4j2z 1/1 Running 0 10m default host-to-b-multi-node-headless-865899f7bb-frrmc 1/1 Running 0 10m default pod-to-a-allowed-cnp-5f9d7d4b9d-hcd8x 1/1 Running 0 10m default pod-to-a-denied-cnp-65cc5ff97b-2rzb8 1/1 Running 0 10m default pod-to-a-dfc64f564-p7xcn 1/1 Running 0 10m default pod-to-b-intra-node-nodeport-677868746b-trk2l 1/1 Running 0 10m default pod-to-b-multi-node-clusterip-76bbbc677b-knfq2 1/1 Running 0 10m default pod-to-b-multi-node-headless-698c6579fd-mmvd7 1/1 Running 0 10m default pod-to-b-multi-node-nodeport-5dc4b8cfd6-8dxmz 1/1 Running 0 10m default pod-to-external-1111-8459965778-pjt9b 1/1 Running 0 10m default pod-to-external-fqdn-allow-google-cnp-64df9fb89b-l9l4q 1/1 Running 0 10m kube-system cilium-7rfj6 1/1 Running 0 56s kube-system cilium-d4cch 1/1 Running 0 56s kube-system cilium-h5x8r 1/1 Running 0 56s kube-system cilium-operator-5dbddb6dbf-flpl5 1/1 Running 0 56s kube-system cilium-operator-5dbddb6dbf-gcznc 1/1 Running 0 56s kube-system cilium-t2xlz 1/1 Running 0 56s kube-system cilium-z65z7 1/1 Running 0 56s kube-system coredns-665475b9f8-jkqn8 1/1 Running 1 (36h ago) 36h kube-system hubble-relay-59d8575-9pl9z 1/1 Running 0 56s kube-system hubble-ui-64d4995d57-nsv9j 2/2 Running 0 56s kube-system metrics-server-776f58c94b-c6zgs 1/1 Running 1 (36h ago) 37h 修改为NodePort(示例) # kubectl edit svc -n kube-system hubble-ui service/hubble-ui edited # kubectl edit svc -n cilium-monitoring grafana service/grafana edited # kubectl edit svc -n cilium-monitoring prometheus service/prometheus edited type: NodePort 查看端口(示例) # kubectl get svc -A | grep monit cilium-monitoring grafana NodePort 10.100.250.17 \u0026lt;none\u0026gt; 3000:30707/TCP 15m cilium-monitoring prometheus NodePort 10.100.131.243 \u0026lt;none\u0026gt; 9090:31155/TCP 15m # kubectl get svc -A | grep hubble kube-system hubble-metrics ClusterIP None \u0026lt;none\u0026gt; 9965/TCP 5m12s kube-system hubble-peer ClusterIP 10.100.150.29 \u0026lt;none\u0026gt; 443/TCP 5m12s kube-system hubble-relay ClusterIP 10.109.251.34 \u0026lt;none\u0026gt; 80/TCP 5m12s kube-system hubble-ui NodePort 10.102.253.59 \u0026lt;none\u0026gt; 80:31219/TCP 5m12s 访问(示例) http://192.168.100.101:30707 http://192.168.100.101:31155 http://192.168.100.101:31219 安装CoreDNS 以下步骤只在master01操作 修改文件 # 下载tgz包 helm repo add coredns https://coredns.github.io/helm helm pull coredns/coredns tar xvf coredns-*.tgz cd coredns/ # 修改IP地址 vi values.yaml cat values.yaml | grep clusterIP: clusterIP: \u0026#34;10.96.0.10\u0026#34; # 示例 --- service: # clusterIP: \u0026#34;\u0026#34; # clusterIPs: [] # loadBalancerIP: \u0026#34;\u0026#34; # externalIPs: [] # externalTrafficPolicy: \u0026#34;\u0026#34; # ipFamilyPolicy: \u0026#34;\u0026#34; # The name of the Service # If not set, a name is generated using the fullname template clusterIP: \u0026#34;10.96.0.10\u0026#34; name: \u0026#34;\u0026#34; annotations: {} --- # 修改为国内源 docker源可选 sed -i \u0026#34;s#coredns/#m.daocloud.io/docker.io/coredns/#g\u0026#34; values.yaml sed -i \u0026#34;s#registry.k8s.io/#m.daocloud.io/registry.k8s.io/#g\u0026#34; values.yaml # 默认参数安装 helm install coredns ./coredns/ -n kube-system 安装Metrics Server 以下步骤只在master01操作 安装 在新版的Kubernetes中系统资源的采集均使用Metrics-server，可以通过Metrics采集节点和Pod的内存、磁盘、CPU和网络的使用率\n# 单机版 wget https://ghproxy.com/https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml # 高可用版本 wget https://ghproxy.com/https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability.yaml # 修改配置 vim components.yaml vim high-availability.yaml --- # 1 defaultArgs: - --cert-dir=/tmp - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-use-node-status-port - --metric-resolution=15s - --kubelet-insecure-tls - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.pem - --requestheader-username-headers=X-Remote-User - --requestheader-group-headers=X-Remote-Group - --requestheader-extra-headers-prefix=X-Remote-Extra- # 2 volumeMounts: - mountPath: /tmp name: tmp-dir - name: ca-ssl mountPath: /etc/kubernetes/pki # 3 volumes: - emptyDir: {} name: tmp-dir - name: ca-ssl hostPath: path: /etc/kubernetes/pki --- # 修改为国内源 docker源可选 sed -i \u0026#34;s#registry.k8s.io/#m.daocloud.io/registry.k8s.io/#g\u0026#34; *.yaml # 二选一 kubectl apply -f components.yaml # kubectl apply -f high-availability.yaml 查看状态 # kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-master01 94m 2% 1344Mi 35% k8s-master02 98m 2% 1412Mi 36% k8s-master03 87m 2% 1351Mi 35% k8s-node01 51m 1% 756Mi 19% k8s-node02 57m 1% 475Mi 12% 集群验证 部署pod资源 cat\u0026lt;\u0026lt;EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: busybox namespace: default spec: containers: - name: busybox image: docker.io/library/busybox:1.28 command: - sleep - \u0026#34;3600\u0026#34; imagePullPolicy: IfNotPresent restartPolicy: Always EOF # 查看 # kubectl get pod NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 17s 用pod解析默认命名空间中的kubernetes # 查看name # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 \u0026lt;none\u0026gt; 443/TCP 14d # 进行解析 # kubectl exec busybox -n default -- nslookup kubernetes Server: 10.96.0.10 Address 1: 10.96.0.10 coredns-coredns.kube-system.svc.cluster.local Name: kubernetes Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local 测试跨命名空间是否可以解析 # 查看有那些name # kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 \u0026lt;none\u0026gt; 443/TCP 14d ingress-nginx ingress-nginx-controller LoadBalancer 10.107.76.220 \u0026lt;pending\u0026gt; 80:30451/TCP,443:32101/TCP 13d ingress-nginx ingress-nginx-controller-admission ClusterIP 10.110.40.24 \u0026lt;none\u0026gt; 443/TCP 13d kube-system calico-typha ClusterIP 10.104.191.196 \u0026lt;none\u0026gt; 5473/TCP 13d kube-system coredns-coredns ClusterIP 10.96.0.10 \u0026lt;none\u0026gt; 53/UDP,53/TCP 13d kube-system default-http-backend ClusterIP 10.100.145.35 \u0026lt;none\u0026gt; 80/TCP 13d kube-system metrics-server ClusterIP 10.107.79.100 \u0026lt;none\u0026gt; 443/TCP 13d # 进行解析 # kubectl exec busybox -n default -- nslookup coredns-coredns.kube-system Server: 10.96.0.10 Address 1: 10.96.0.10 coredns-coredns.kube-system.svc.cluster.local Name: coredns-coredns.kube-system Address 1: 10.96.0.10 coredns-coredns.kube-system.svc.cluster.local 每个节点都必须要能访问Kubernetes的kubernetes svc 443和kube-dns的service 53 telnet 10.96.0.1 443 Trying 10.96.0.1... Connected to 10.96.0.1. Escape character is \u0026#39;^]\u0026#39;. telnet 10.96.0.10 53 Trying 10.96.0.10... Connected to 10.96.0.10. Escape character is \u0026#39;^]\u0026#39;. curl 10.96.0.10:53 curl: (52) Empty reply from server Pod和Pod之间要能通 # kubectl get po -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES busybox 1/1 Running 0 18m 172.27.14.202 k8s-node02 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; # kubectl get po -n kube-system -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES calico-kube-controllers-76754ff848-gmx8n 1/1 Running 3 (30h ago) 13d 172.25.244.199 k8s-master01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; calico-node-845jj 1/1 Running 2 (2d ago) 13d 192.168.100.102 k8s-master02 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; calico-node-m7p9n 1/1 Running 2 (2d ago) 13d 192.168.100.103 k8s-master03 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; calico-node-qqj7g 1/1 Running 2 (2d ago) 13d 192.168.100.104 k8s-node01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; calico-node-rfq9m 1/1 Running 4 (53m ago) 13d 192.168.100.105 k8s-node02 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; calico-node-tdnmt 1/1 Running 3 (30h ago) 13d 192.168.100.101 k8s-master01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; calico-typha-59d75c5dd4-8fg4d 1/1 Running 2 (2d ago) 13d 192.168.100.103 k8s-master03 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; coredns-coredns-5f4fc468-t5l96 1/1 Running 2 (2d ago) 13d 172.25.92.73 k8s-master02 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; default-http-backend-cf5544789-7qw27 1/1 Running 2 (2d ago) 13d 172.17.125.11 k8s-node01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; kubernetes-dashboard-5948b5f5d7-t2nkl 1/1 Running 3 (2d ago) 13d 172.25.92.74 k8s-master02 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; metrics-server-65875bc4f9-wcxlv 1/1 Running 4 (2d ago) 13d 172.17.125.10 k8s-node01 \u0026lt;none\u0026gt; \u0026lt;none\u0026gt; # 进入busybox ping其他节点上的pod #kubectl exec -ti busybox -- sh # ping 192.168.100.102 PING 192.168.100.102 (192.168.100.102): 56 data bytes 64 bytes from 192.168.100.102: seq=0 ttl=63 time=0.487 ms 64 bytes from 192.168.100.102: seq=1 ttl=63 time=0.432 ms 64 bytes from 192.168.100.102: seq=2 ttl=63 time=0.334 ms # 可以连通证明这个pod是可以跨命名空间和跨主机通信的 创建三个副本，可以看到3个副本分布在不同的节点上 cat \u0026gt; deployments.yaml \u0026lt;\u0026lt; EOF apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 EOF # kubectl apply -f deployments.yaml deployment.apps/nginx-deployment created # kubectl get pod NAME READY STATUS RESTARTS AGE busybox 1/1 Running 0 21m nginx-deployment-55f598f8d-4rnwj 1/1 Running 0 8s nginx-deployment-55f598f8d-tvfgk 1/1 Running 0 8s nginx-deployment-55f598f8d-xsc9p 1/1 Running 0 8s # 删除nginx # kubectl delete -f deployments.yaml 安装dashboard helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/ helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --namespace kube-system 更改dashboard的svc为NodePort # kubectl edit svc kubernetes-dashboard -n kube-system type: NodePort 查看端口号 # kubectl get svc kubernetes-dashboard -n kube-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes-dashboard NodePort 10.100.112.99 \u0026lt;none\u0026gt; 443:31048/TCP 13d 创建token cat \u0026gt; dashboard-user.yaml \u0026lt;\u0026lt; EOF apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kube-system EOF # kubectl apply -f dashboard-user.yaml # 创建token # kubectl -n kube-system create token admin-user eyJhbGciOiJSUzI1NiIsImtpZCI6IlA1MVFqS1lwOTU2S1pMSWxC...... 登录dashboard https://192.168.100.100:31048 ingress安装 部署 wget https://ghproxy.com/https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml # 修改为国内源 docker源可选 sed -i \u0026#34;s#registry.k8s.io/#m.daocloud.io/registry.k8s.io/#g\u0026#34; *.yaml cat \u0026gt; backend.yaml \u0026lt;\u0026lt; EOF apiVersion: apps/v1 kind: Deployment metadata: name: default-http-backend labels: app.kubernetes.io/name: default-http-backend namespace: kube-system spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: default-http-backend template: metadata: labels: app.kubernetes.io/name: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend image: registry.cn-hangzhou.aliyuncs.com/test/defaultbackend-amd64:1.5 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: kube-system labels: app.kubernetes.io/name: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: app.kubernetes.io/name: default-http-backend EOF kubectl apply -f deploy.yaml kubectl apply -f backend.yaml cat \u0026gt; ingress-demo-app.yaml \u0026lt;\u0026lt; EOF apiVersion: apps/v1 kind: Deployment metadata: name: hello-server spec: replicas: 2 selector: matchLabels: app: hello-server template: metadata: labels: app: hello-server spec: containers: - name: hello-server image: registry.cn-hangzhou.aliyuncs.com/lfy_k8s_images/hello-server ports: - containerPort: 9000 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: nginx-demo name: nginx-demo spec: replicas: 2 selector: matchLabels: app: nginx-demo template: metadata: labels: app: nginx-demo spec: containers: - image: nginx name: nginx --- apiVersion: v1 kind: Service metadata: labels: app: nginx-demo name: nginx-demo spec: selector: app: nginx-demo ports: - port: 8000 protocol: TCP targetPort: 80 --- apiVersion: v1 kind: Service metadata: labels: app: hello-server name: hello-server spec: selector: app: hello-server ports: - port: 8000 protocol: TCP targetPort: 9000 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress-host-bar spec: ingressClassName: nginx rules: - host: \u0026#34;hello.test.cn\u0026#34; http: paths: - pathType: Prefix path: \u0026#34;/\u0026#34; backend: service: name: hello-server port: number: 8000 - host: \u0026#34;demo.test.cn\u0026#34; http: paths: - pathType: Prefix path: \u0026#34;/nginx\u0026#34; backend: service: name: nginx-demo port: number: 8000 EOF # 等创建完成后在执行： kubectl apply -f ingress-demo-app.yaml kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE ingress-host-bar nginx hello.test.cn,demo.test.cn 192.168.100.102 80 7s 过滤查看ingress端口 # 修改为nodeport kubectl edit svc -n ingress-nginx ingress-nginx-controller type: NodePort # kubectl get svc -A | grep ingress ingress-nginx ingress-nginx-controller LoadBalancer 10.107.76.220 \u0026lt;pending\u0026gt; 80:30451/TCP,443:32101/TCP 13d ingress-nginx ingress-nginx-controller-admission ClusterIP 10.110.40.24 \u0026lt;none\u0026gt; 443/TCP 13d IPv6测试(示例) #部署应用 cat\u0026lt;\u0026lt;EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: test spec: replicas: 3 selector: matchLabels: app: test template: metadata: labels: app: test spec: containers: - name: test image: docker.io/library/nginx resources: limits: memory: \u0026#34;128Mi\u0026#34; cpu: \u0026#34;500m\u0026#34; ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: test spec: ipFamilyPolicy: PreferDualStack ipFamilies: - IPv6 - IPv4 type: NodePort selector: app: test ports: - port: 80 targetPort: 80 EOF #查看端口 # kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE test NodePort fd00::a29c \u0026lt;none\u0026gt; 80:30779/TCP 5s #使用内网访问 # curl -I http://[fd00::a29c] HTTP/1.1 200 OK Server: nginx/1.21.6 Date: Thu, 05 May 2022 10:20:35 GMT Content-Type: text/html Content-Length: 615 Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT Connection: keep-alive ETag: \u0026#34;61f01158-267\u0026#34; Accept-Ranges: bytes # curl -I http://192.168.100.101:30779 HTTP/1.1 200 OK Server: nginx/1.21.6 Date: Thu, 05 May 2022 10:20:59 GMT Content-Type: text/html Content-Length: 615 Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT Connection: keep-alive ETag: \u0026#34;61f01158-267\u0026#34; Accept-Ranges: bytes #使用公网访问 # curl -I http://[2409:8a10:9e18:9020::10]:30779 HTTP/1.1 200 OK Server: nginx/1.21.6 Date: Thu, 05 May 2022 10:20:54 GMT Content-Type: text/html Content-Length: 615 Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT Connection: keep-alive ETag: \u0026#34;61f01158-267\u0026#34; Accept-Ranges: bytes 安装命令行自动补全功能(可选) yum install bash-completion -y source /usr/share/bash-completion/bash_completion source \u0026lt;(kubectl completion bash) echo \u0026#34;source \u0026lt;(kubectl completion bash)\u0026#34; \u0026gt;\u0026gt; ~/.bashrc 附录(转载) # 镜像加速器可以使用DaoCloud仓库，替换规则如下 cr.l5d.io/ ===\u0026gt; m.daocloud.io/cr.l5d.io/ docker.elastic.co/ ===\u0026gt; m.daocloud.io/docker.elastic.co/ docker.io/ ===\u0026gt; m.daocloud.io/docker.io/ gcr.io/ ===\u0026gt; m.daocloud.io/gcr.io/ ghcr.io/ ===\u0026gt; m.daocloud.io/ghcr.io/ k8s.gcr.io/ ===\u0026gt; m.daocloud.io/k8s.gcr.io/ mcr.microsoft.com/ ===\u0026gt; m.daocloud.io/mcr.microsoft.com/ nvcr.io/ ===\u0026gt; m.daocloud.io/nvcr.io/ quay.io/ ===\u0026gt; m.daocloud.io/quay.io/ registry.jujucharms.com/ ===\u0026gt; m.daocloud.io/registry.jujucharms.com/ registry.k8s.io/ ===\u0026gt; m.daocloud.io/registry.k8s.io/ registry.opensource.zalan.do/ ===\u0026gt; m.daocloud.io/registry.opensource.zalan.do/ rocks.canonical.com/ ===\u0026gt; m.daocloud.io/rocks.canonical.com/ # 镜像版本要自行查看，因为镜像版本是随时更新的，文档无法做到实时更新 # docker pull 镜像 docker pull registry.cn-hangzhou.aliyuncs.com/test/cni:master docker pull registry.cn-hangzhou.aliyuncs.com/test/node:master docker pull registry.cn-hangzhou.aliyuncs.com/test/kube-controllers:master docker pull registry.cn-hangzhou.aliyuncs.com/test/typha:master docker pull registry.cn-hangzhou.aliyuncs.com/test/coredns:v1.10.0 docker pull registry.cn-hangzhou.aliyuncs.com/test/pause:3.6 docker pull registry.cn-hangzhou.aliyuncs.com/test/metrics-server:v0.5.2 docker pull kubernetesui/dashboard:v2.7.0 docker pull kubernetesui/metrics-scraper:v1.0.8 docker pull quay.io/cilium/cilium:v1.12.6 docker pull quay.io/cilium/certgen:v0.1.8 docker pull quay.io/cilium/hubble-relay:v1.12.6 docker pull quay.io/cilium/hubble-ui-backend:v0.9.2 docker pull quay.io/cilium/hubble-ui:v0.9.2 docker pull quay.io/cilium/cilium-etcd-operator:v2.0.7 docker pull quay.io/cilium/operator:v1.12.6 docker pull quay.io/cilium/clustermesh-apiserver:v1.12.6 docker pull quay.io/coreos/etcd:v3.5.4 docker pull quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26 # docker 保存镜像 docker save registry.cn-hangzhou.aliyuncs.com/test/cni:master -o cni.tar docker save registry.cn-hangzhou.aliyuncs.com/test/node:master -o node.tar docker save registry.cn-hangzhou.aliyuncs.com/test/typha:master -o typha.tar docker save registry.cn-hangzhou.aliyuncs.com/test/kube-controllers:master -o kube-controllers.tar docker save registry.cn-hangzhou.aliyuncs.com/test/coredns:v1.10.0 -o coredns.tar docker save registry.cn-hangzhou.aliyuncs.com/test/pause:3.6 -o pause.tar docker save registry.cn-hangzhou.aliyuncs.com/test/metrics-server:v0.5.2 -o metrics-server.tar docker save kubernetesui/dashboard:v2.7.0 -o dashboard.tar docker save kubernetesui/metrics-scraper:v1.0.8 -o metrics-scraper.tar docker save quay.io/cilium/cilium:v1.12.6 -o cilium.tar docker save quay.io/cilium/certgen:v0.1.8 -o certgen.tar docker save quay.io/cilium/hubble-relay:v1.12.6 -o hubble-relay.tar docker save quay.io/cilium/hubble-ui-backend:v0.9.2 -o hubble-ui-backend.tar docker save quay.io/cilium/hubble-ui:v0.9.2 -o hubble-ui.tar docker save quay.io/cilium/cilium-etcd-operator:v2.0.7 -o cilium-etcd-operator.tar docker save quay.io/cilium/operator:v1.12.6 -o operator.tar docker save quay.io/cilium/clustermesh-apiserver:v1.12.6 -o clustermesh-apiserver.tar docker save quay.io/coreos/etcd:v3.5.4 -o etcd.tar docker save quay.io/cilium/startup-script:d69851597ea019af980891a4628fb36b7880ec26 -o startup-script.tar # 传输到各个节点 for NODE in k8s-master01 k8s-master02 k8s-master03 k8s-node01 k8s-node02; do scp -r images/ $NODE:/root/ ; done # 创建命名空间 ctr ns create k8s.io # 导入镜像 ctr --namespace k8s.io image import images/cni.tar ctr --namespace k8s.io image import images/node.tar ctr --namespace k8s.io image import images/typha.tar ctr --namespace k8s.io image import images/kube-controllers.tar ctr --namespace k8s.io image import images/coredns.tar ctr --namespace k8s.io image import images/pause.tar ctr --namespace k8s.io image import images/metrics-server.tar ctr --namespace k8s.io image import images/dashboard.tar ctr --namespace k8s.io image import images/metrics-scraper.tar ctr --namespace k8s.io image import images/dashboard.tar ctr --namespace k8s.io image import images/metrics-scraper.tar ctr --namespace k8s.io image import images/cilium.tar ctr --namespace k8s.io image import images/certgen.tar ctr --namespace k8s.io image import images/hubble-relay.tar ctr --namespace k8s.io image import images/hubble-ui-backend.tar ctr --namespace k8s.io image import images/hubble-ui.tar ctr --namespace k8s.io image import images/cilium-etcd-operator.tar ctr --namespace k8s.io image import images/operator.tar ctr --namespace k8s.io image import images/clustermesh-apiserver.tar ctr --namespace k8s.io image import images/etcd.tar ctr --namespace k8s.io image import images/startup-script.tar # pull tar包 解压后 helm pull cilium/cilium # 查看镜像版本 # cat values.yaml| grep tag: -C1 repository: \u0026#34;quay.io/cilium/cilium\u0026#34; tag: \u0026#34;v1.12.6\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; -- repository: \u0026#34;quay.io/cilium/certgen\u0026#34; tag: \u0026#34;v0.1.8@sha256:4a456552a5f192992a6edcec2febb1c54870d665173a33dc7d876129b199ddbd\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; -- repository: \u0026#34;quay.io/cilium/hubble-relay\u0026#34; tag: \u0026#34;v1.12.6\u0026#34; # hubble-relay-digest -- repository: \u0026#34;quay.io/cilium/hubble-ui-backend\u0026#34; tag: \u0026#34;v0.9.2@sha256:a3ac4d5b87889c9f7cc6323e86d3126b0d382933bd64f44382a92778b0cde5d7\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; -- repository: \u0026#34;quay.io/cilium/hubble-ui\u0026#34; tag: \u0026#34;v0.9.2@sha256:d3596efc94a41c6b772b9afe6fe47c17417658956e04c3e2a28d293f2670663e\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; -- repository: \u0026#34;quay.io/cilium/cilium-etcd-operator\u0026#34; tag: \u0026#34;v2.0.7@sha256:04b8327f7f992693c2cb483b999041ed8f92efc8e14f2a5f3ab95574a65ea2dc\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; -- repository: \u0026#34;quay.io/cilium/operator\u0026#34; tag: \u0026#34;v1.12.6\u0026#34; # operator-generic-digest -- repository: \u0026#34;quay.io/cilium/startup-script\u0026#34; tag: \u0026#34;d69851597ea019af980891a4628fb36b7880ec26\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; -- repository: \u0026#34;quay.io/cilium/cilium\u0026#34; tag: \u0026#34;v1.12.6\u0026#34; # cilium-digest -- repository: \u0026#34;quay.io/cilium/clustermesh-apiserver\u0026#34; tag: \u0026#34;v1.12.6\u0026#34; # clustermesh-apiserver-digest -- repository: \u0026#34;quay.io/coreos/etcd\u0026#34; tag: \u0026#34;v3.5.4@sha256:795d8660c48c439a7c3764c2330ed9222ab5db5bb524d8d0607cac76f7ba82a3\u0026#34; pullPolicy: \u0026#34;IfNotPresent\u0026#34; 参考资料: https://github.com/cby-chen/Kubernetes\n","permalink":"https://gnail89.github.io/posts/kubernetes-v1.27.6-centos8stream-binary-install/","summary":"环境 主机名 IP地址 备注 VIP 192.168.100.100 浮动IP k8s-master01 192.168.100.101 master节点 k8s-master02 192.168.100.102 master节点 k8s-master03 192.168.100.103 master节点 k8s-node01 192.168.100.104 node节点 k8s-node02 192.168.100.105 node节点 网段说明\nservice: 10.96.0.0/12 pod: 172.16.0.0/12 k8s基础系统环境配置 配置IP # 网卡UUID不能重复，UUID重复无法获取到IPV6地址 # # 查看当前的网卡列表和 UUID： # nmcli con show # 删除要更改 UUID 的网络连接： # nmcli con delete uuid \u0026lt;原 UUID\u0026gt; # 重新生成 UUID： # nmcli con add type ethernet ifname \u0026lt;接口名称\u0026gt; con-name \u0026lt;新名称\u0026gt; # 重新启用网络连接： # nmcli con up \u0026lt;新名称\u0026gt; # 更改网卡的UUID nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 nmcli con delete uuid a5078897-4aae-3fa8-8b7a-0773d924901d;nmcli con add type ethernet ifname ens1 con-name ens1;nmcli con up ens1 # 修改静态的IPv4地址 nmcli con mod ens1 ipv4.","title":"Kubernetes V1.27.6 Centos8stream Binary Install"},{"content":"1. 软件版本： OS: Anolis OS release 8.8 SSH: openssh-9.3p1.tar.gz SSL: openssl-1.1.1t.tar.gz 2. 安装基础依赖包 dnf install -y gcc gcc-c++ make zlib zlib-devel pam pam-devel tar 3. 手动编译OpenSSL # 解压缩 tar zxf openssl-1.1.1t.tar.gz cd openssl-1.1.1t # 编译安装 ./config --prefix=/opt/openssl/ make \u0026amp;\u0026amp; make install 安装目录结构 /opt/openssl/ ├── bin │ ├── c_rehash │ └── openssl ├── include │ └── openssl ├── lib │ ├── engines-1.1 │ ├── libcrypto.a │ ├── libcrypto.so -\u0026gt; libcrypto.so.1.1 │ ├── libcrypto.so.1.1 │ ├── libssl.a │ ├── libssl.so -\u0026gt; libssl.so.1.1 │ ├── libssl.so.1.1 │ └── pkgconfig ├── share │ ├── doc │ └── man └── ssl ├── certs ├── ct_log_list.cnf ├── ct_log_list.cnf.dist ├── misc ├── openssl.cnf ├── openssl.cnf.dist └── private 4. 手动编译OpenSSH # 解压缩 tar zxf openssh-9.3p1.tar.gz cd openssh-9.3p1 # 指定LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/openssl/include/openssl:/opt/openssl/lib mv /opt/openssl/lib/libcrypto.so{,.bak} # 配置 ./configure --prefix=/usr/local/openssh \\ --bindir=/usr/bin \\ --sbindir=/usr/sbin \\ --sysconfdir=/etc/ssh \\ --with-zlib \\ --with-pam \\ --with-ssl-dir=/opt/openssl \\ --with-ldflags=\u0026#34;-L/opt/openssl/lib -l:libcrypto.a -pthread\u0026#34; # 编译安装 make \u0026amp;\u0026amp; make install # 升级后的版本sshd -V OpenSSH_9.3, OpenSSL 1.1.1t 7 Feb 2023 sshd 链接库信息 $ ldd /usr/sbin/sshd linux-vdso.so.1 (0x00007ffdcac9f000) libcrypt.so.1 =\u0026gt; /lib64/libcrypt.so.1 (0x00007fe856a00000) libpam.so.0 =\u0026gt; /lib64/libpam.so.0 (0x00007fe856600000) libdl.so.2 =\u0026gt; /lib64/libdl.so.2 (0x00007fe856200000) libutil.so.1 =\u0026gt; /lib64/libutil.so.1 (0x00007fe855e00000) libresolv.so.2 =\u0026gt; /lib64/libresolv.so.2 (0x00007fe855a00000) libz.so.1 =\u0026gt; /lib64/libz.so.1 (0x00007fe855600000) libpthread.so.0 =\u0026gt; /lib64/libpthread.so.0 (0x00007fe855200000) libc.so.6 =\u0026gt; /lib64/libc.so.6 (0x00007fe854e00000) libaudit.so.1 =\u0026gt; /lib64/libaudit.so.1 (0x00007fe854a00000) /lib64/ld-linux-x86-64.so.2 (0x00007fe8574c4000) libcap-ng.so.0 =\u0026gt; /lib64/libcap-ng.so.0 (0x00007fe854600000) systemd启动配置示例 $ cat /usr/lib/systemd/system/sshd.service [Unit] Description=OpenSSH server daemon Documentation=man:sshd(8) man:sshd_config(5) After=network.target sshd-keygen.target Wants=sshd-keygen.target [Service] Type=simple ExecStart=/usr/sbin/sshd -D ExecReload=/bin/kill -HUP $MAINPID KillMode=process Restart=on-failure RestartSec=42s [Install] WantedBy=multi-user.target sshd服务重启 systemctl daemon-reload systemctl restart sshd systemctl status sshd ","permalink":"https://gnail89.github.io/posts/openssh_upgrade/","summary":"1. 软件版本： OS: Anolis OS release 8.8 SSH: openssh-9.3p1.tar.gz SSL: openssl-1.1.1t.tar.gz 2. 安装基础依赖包 dnf install -y gcc gcc-c++ make zlib zlib-devel pam pam-devel tar 3. 手动编译OpenSSL # 解压缩 tar zxf openssl-1.1.1t.tar.gz cd openssl-1.1.1t # 编译安装 ./config --prefix=/opt/openssl/ make \u0026amp;\u0026amp; make install 安装目录结构 /opt/openssl/ ├── bin │ ├── c_rehash │ └── openssl ├── include │ └── openssl ├── lib │ ├── engines-1.1 │ ├── libcrypto.a │ ├── libcrypto.so -\u0026gt; libcrypto.so.1.1 │ ├── libcrypto.","title":"OpenSSH 9.3p1编译安装,链接OpenSSL 1.1.1t静态库"},{"content":"工具 需要以下最新版本工具\nsmartctl\nMegaCli64\nlsscsi\n获取基本信息 使用lsscsi获取SCSI设备信息和属性 其中id的含义可以从/proc/scsi/scsi文件中简单获取。 [0:2:2:0]分别表示Host, Channel, Id, Lun\n# lsscsi -g [0:0:2:0] enclosu SAS/SATA Expander RevA - /dev/sg0 [0:2:0:0] disk LSI LSI 3.40 /dev/sda /dev/sg1 [0:2:1:0] disk LSI LSI 3.40 /dev/sdb /dev/sg2 [0:2:2:0] disk LSI LSI 3.40 /dev/sdc /dev/sg3 [0:2:3:0] disk LSI LSI 3.40 /dev/sdd /dev/sg4 [0:2:4:0] disk LSI LSI 3.40 /dev/sde /dev/sg5 [0:2:5:0] disk LSI LSI 3.40 /dev/sdf /dev/sg6 [0:2:6:0] disk LSI LSI 3.40 /dev/sdg /dev/sg7 [0:2:7:0] disk LSI LSI 3.40 /dev/sdh /dev/sg8 [0:2:8:0] disk LSI LSI 3.40 /dev/sdi /dev/sg9 [0:2:9:0] disk LSI LSI 3.40 /dev/sdj /dev/sg10 [0:2:10:0] disk LSI LSI 3.40 /dev/sdk /dev/sg11 [0:2:11:0] disk LSI LSI 3.40 /dev/sdl /dev/sg12 [0:2:12:0] disk LSI LSI 3.40 /dev/sdm /dev/sg13 # cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 02 Lun: 00 Vendor: SAS/SATA Model: Expander Rev: RevA Type: Enclosure ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 00 Lun: 00 Vendor: LSI Model: LSI Rev: 3.40 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi0 Channel: 02 Id: 01 Lun: 00 Vendor: LSI Model: LSI Rev: 3.40 Type: Direct-Access ANSI SCSI revision: 05 ... 使用MegaCli64获取硬盘信息 # 获取MegaRAID的bus id为：02:00.0 # lspci |grep MegaRAID 02:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS 2208 # MegaCli64 -LdPdInfo -aALL |egrep \u0026#34;Target Id:|Enclosure Device ID:|^Slot Number:|^Device Id:|^Inquiry Data:\u0026#34; Virtual Drive: 0 (Target Id: 0) Enclosure Device ID: 2 Slot Number: 12 Device Id: 18 Inquiry Data: HGST xxxxxxxx xxxxxxxx Enclosure Device ID: 2 Slot Number: 13 Device Id: 19 Inquiry Data: HGST xxxxxxxx xxxxxxxx Virtual Drive: 1 (Target Id: 1) Enclosure Device ID: 2 Slot Number: 0 Device Id: 6 Inquiry Data: HGST xxxxxxxx xxxxxxxx Virtual Drive: 2 (Target Id: 2) Enclosure Device ID: 2 Slot Number: 1 Device Id: 14 Inquiry Data: YHH7HWLAA xxxxxxxx xxxxxxxx ... # MegaCli64 -pdInfo -PhysDrv[2:1] -aALL Enclosure Device ID: 2 Slot Number: 1 Drive\u0026#39;s position: DiskGroup: 10, Span: 0, Arm: 0 Enclosure position: 1 Device Id: 14 WWN: 5000cca225d181c1 Sequence Number: 2 Media Error Count: 0 Other Error Count: 0 Predictive Failure Count: 0 Last Predictive Failure Event Seq Number: 0 PD Type: SATA ... 使用smartctl扫描设备 # smartctl --scan-open /dev/sda -d scsi # /dev/sda, SCSI device /dev/sdb -d scsi # /dev/sdb, SCSI device /dev/sdc -d scsi # /dev/sdc, SCSI device /dev/sdd -d scsi # /dev/sdd, SCSI device ... /dev/bus/0 -d sat+megaraid,3 # /dev/bus/0 [megaraid_disk_03] [SAT]，ATA device /dev/bus/0 -d sat+megaraid,11 # /dev/bus/0 [megaraid_disk_11] [SAT]，ATA device /dev/bus/0 -d sat+megaraid,14 # /dev/bus/0 [megaraid_disk_14] [SAT]，ATA device /dev/bus/0 -d sat+megaraid,17 # /dev/bus/0 [megaraid_disk_17] [SAT]，ATA device ... 方法 从lsscsi中可以得到设备ID 例子：[0:2:2:0] disk LSI \u0026hellip; /dev/sdc ==\u0026gt; Host id: 0 Channel id: 2 Target id: 2 # 对应MegaRAID中的Target id Lun id: 0\n从MegaCli64中可以得到设备的Target id，Enclosure Device ID，Slot Number，Inquiry Data等信息 例子：PhysDrv[2:1] Virtual Drive: 2 (Target Id: 2) # 对应scsi中的Target id Device Id: 14 # 对应smartctl中的megaraid id Enclosure Device ID: 2 Slot Number: 1 # 对应机箱槽位号 Inquiry Data: YHH7HWLAA xxxxxxxx xxxxxxxx # 硬盘序列号信息\n从smartctl中可以得到megaraid信息 /dev/bus/0 -d sat+megaraid,14 # 对应MegaRAID中Device id\n以/dev/sdc为例。\n/dev/sdc的scsi id为[0:2:2:0] =\u0026gt; MegaRAID硬盘PhysDrv[2:1] =\u0026gt; smartctl中megaraid,14\n通过序列号验证MegaRAID与smartctl的关系：\n# smartctl -a /dev/bus/0 -d sat+megaraid,14 |grep \u0026#34;Serial Number\u0026#34; Serial Number: YHH7HWLAA # MegaC1164 -pdInfo -PhysDrv[2:1] -aALL ... Enclosure Device ID: 2 Slot Number: 1 Device Id: 14 Inquiry Data: YHH7HWLAA xxxxxxxx xxxxxxxx ... ","permalink":"https://gnail89.github.io/posts/find-disk-slot/","summary":"工具 需要以下最新版本工具\nsmartctl\nMegaCli64\nlsscsi\n获取基本信息 使用lsscsi获取SCSI设备信息和属性 其中id的含义可以从/proc/scsi/scsi文件中简单获取。 [0:2:2:0]分别表示Host, Channel, Id, Lun\n# lsscsi -g [0:0:2:0] enclosu SAS/SATA Expander RevA - /dev/sg0 [0:2:0:0] disk LSI LSI 3.40 /dev/sda /dev/sg1 [0:2:1:0] disk LSI LSI 3.40 /dev/sdb /dev/sg2 [0:2:2:0] disk LSI LSI 3.40 /dev/sdc /dev/sg3 [0:2:3:0] disk LSI LSI 3.40 /dev/sdd /dev/sg4 [0:2:4:0] disk LSI LSI 3.40 /dev/sde /dev/sg5 [0:2:5:0] disk LSI LSI 3.40 /dev/sdf /dev/sg6 [0:2:6:0] disk LSI LSI 3.40 /dev/sdg /dev/sg7 [0:2:7:0] disk LSI LSI 3.","title":"硬盘驱动器与物理硬盘对应关系"},{"content":" 分享ssh安全配置实践\n1. openssh基本配置 关注安全漏洞信息，定期升级ssh1版本，关键参数配置参考：\ncat /etc/ssh/sshd_config Port 22 # 建议更改默认端口 Protocol 2 PermitRootLogin no MaxAuthTries 3 PubkeyAuthentication yes PermitEmptyPasswords no PasswordAuthentication no UsePAM yes X11Forwarding no PrintMotd no ClientAliveInterval 180 ClientAliveCountMax 0 UseDNS no Banner /etc/ssh_banner AllowUsers username 2. 使用PAM模块锁定账户 PAM2模块，pam_tally2和pam_faillock模块功能相似，以pam_faillock模块为例：\ncat /etc/pam.d/password-auth #%PAM-1.0 # This file is auto-generated. # User changes will be destroyed the next time authselect is run. auth required pam_env.so auth required pam_faillock.so deny=3 unlock_time=900 even_deny_root root_unlock_time=900 auth sufficient pam_unix.so try_first_pass nullok auth required pam_deny.so account required pam_unix.so ... cat /etc/pam.d/system-auth #%PAM-1.0 # This file is auto-generated. # User changes will be destroyed the next time authselect is run. auth required pam_env.so auth required pam_faillock.so deny=3 unlock_time=900 even_deny_root root_unlock_time=900 auth sufficient pam_unix.so try_first_pass nullok auth required pam_deny.so account required pam_unix.so ... 3. 利用iptables + ipset 启用iptables3，并配置ssh端口阻止策略，示例：\niptables策略 systemctl enable iptables \u0026amp;\u0026amp; systemctl start iptables cat /etc/sysconfig/iptables # sample configuration for iptables service # you can edit this manually or use system-config-firewall # please do not ask us to add additional ports/services to this default configuration *filter :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT -A INPUT -p icmp -j ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m set --match-set banlist src -p tcp -m tcp --dport 22 -j DROP -m comment --comment \u0026#34;block ipset banlist\u0026#34; -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT -A INPUT -j REJECT --reject-with icmp-host-prohibited -A FORWARD -j REJECT --reject-with icmp-host-prohibited COMMIT ipset策略 启用ipset4服务，并新建策略banlist。\n# 安装ipset服务 yum install -y ipset-service systemctl enable ipset # 创建ipset策略 ipset create banlist hash:ip hashsize 4096 maxelem 1000000 timeout 86400 # 保存ipset配置 ipset save \u0026gt; /etc/sysconfig/ipset # 启动ipset服务 systemctl restart ipset #查看ipset策略 ipset list 制定ipset控制脚本 以CentOS 7为例，shell脚本示例如下：\n#!/bin/bash ipset_name=\u0026#34;banlist\u0026#34; num=3 date_time=\u0026#34;$(date -d \u0026#34;1 hours ago\u0026#34; \u0026#34;+%Y-%m-%d %H:%M:%S\u0026#34;)\u0026#34; [[ $(ipset list ${ipset_name}) ]] || ipset create ${ipset_name} hash:ip hashsize 4096 maxelem 1000000 timeout 86400 ipset_info=\u0026#34;$(ipset save ${ipset_name})\u0026#34; lastb -a -s \u0026#34;${date_time}\u0026#34; |awk \u0026#39;/ssh/{print $NF}\u0026#39; |sort -n |uniq -c |while read line; do count=\u0026#34;$(echo $line |awk \u0026#39;{print $1}\u0026#39;)\u0026#34; ipaddr=\u0026#34;$(echo $line |awk \u0026#39;{print $2}\u0026#39;)\u0026#34; if [ $count -ge $num ] \u0026amp;\u0026amp; [ $(echo ${ipset_info} |grep -wc \u0026#34;${ipaddr}\u0026#34;) -eq 0 ]; then /usr/sbin/ipset add ${ipset_name} ${ipaddr} fi done 配置root用户crontab定时任务： crontab -l * * * * * /bin/bash /opt/ban_ssh_ip.sh \u0026amp;\u0026gt;/dev/null openssh\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nLinux PAM\u0026#160;\u0026#x21a9;\u0026#xfe0e;\niptables\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nipset\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://gnail89.github.io/posts/ssh-security/","summary":"分享ssh安全配置实践\n1. openssh基本配置 关注安全漏洞信息，定期升级ssh1版本，关键参数配置参考：\ncat /etc/ssh/sshd_config Port 22 # 建议更改默认端口 Protocol 2 PermitRootLogin no MaxAuthTries 3 PubkeyAuthentication yes PermitEmptyPasswords no PasswordAuthentication no UsePAM yes X11Forwarding no PrintMotd no ClientAliveInterval 180 ClientAliveCountMax 0 UseDNS no Banner /etc/ssh_banner AllowUsers username 2. 使用PAM模块锁定账户 PAM2模块，pam_tally2和pam_faillock模块功能相似，以pam_faillock模块为例：\ncat /etc/pam.d/password-auth #%PAM-1.0 # This file is auto-generated. # User changes will be destroyed the next time authselect is run. auth required pam_env.so auth required pam_faillock.so deny=3 unlock_time=900 even_deny_root root_unlock_time=900 auth sufficient pam_unix.","title":"SSH Security"},{"content":"1. 环境信息 CentOS 7 禁用firewalld 禁用selinux 时钟同步 设置主机名 ip addr hostname 192.168.61.129 ceph-node01 192.168.61.130 ceph-node02 192.168.61.131 ceph-node03 2. 安装依赖包 所有节点都需要安装\n安装epel和elrepo软件源 yum install -y epel-release elrepo-release 安装python3 yum install -y python3 升级kernel至最新 yum --enablerepo=elrepo-kernel install -y kernel-ml # 设置默认使用的内核 grubby --default-index grubby --info=ALL grubby --set-default-index=0 安装docker-ce软件源 cd /etc/yum.repos.d/ curl --remote-name https://download.docker.com/linux/centos/docker-ce.repo 安装最新版本docker yum install -y docker-ce systemctl start docker systemctl enable docker 重启生效 3. 安装cephadm curl --remote-name --location https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-octopus/elnoarch/cephadm chmod +x ./cephadm ./cephadm add-repo --release octopus #添加repo软件源 ./cephadm install #升级最新cephadm包 4. 初始化ceph集群 初始化第一个mon节点 cephadm install ceph-common #安装ceph cli工具 #查看编排后端是cephadm，如果是显示rook这里后端就会显示rook。 ceph orch status mkdir -p /etc/ceph cephadm bootstrap --mon-ip 192.168.61.129 #初始化一个mon节点，获取最新的容器镜像到本地 # 获取mon容器shell cephadm shell --fsid ec45b570-5432-11ed-9ea9-000c29cb9f51 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 引导完成一个单节点群集，程序会做如下事情：\n在本地主机上为新集群创建monitor 和 manager daemon守护程序。 为Ceph集群生成一个新的SSH密钥，并将其添加到root用户的/root/.ssh/authorized_keys文件中。 将与新群集进行通信所需的最小配置文件保存到/etc/ceph/ceph.conf。 向/etc/ceph/ceph.client.admin.keyring写入client.admin管理secret key的副本。 将public key的副本写入/etc/ceph/ceph.pub。 初始的mgr信息：\nURL: https://ceph-node01:8443 User: admin Password: 安装提示获取密码 (可选)如果忘记mgr密码可以通过以下方法重置密码(将密码写入password文件中，通过命令导入密码)\nceph dashboard ac-user-set-password admin -i password 当前节点安装后容器列表中的角色： mon.ceph-node01 # monitor节点 mgr.ceph-node01.hnlomt # mgr管理节点 prometheus.ceph-node01 # prometheus监控 alertmanager.ceph-node01 # prometheus告警相关 grafana.ceph-node01 # 监控告警展示 node-exporter.ceph-node01 # 数据采集node节点 crash.ceph-node01 # crash相关 分发cephadm节点公钥到其余节点： ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node02 ssh-copy-id -f -i /etc/ceph/ceph.pub root@ceph-node03 新增ceph主机到集群新增ceph主机到集群 ceph orch host add ceph-node02 192.168.61.130 ceph orch host add ceph-node03 192.168.61.131 调整mon节点分布 默认mon节点是5个，随机分布在集群主机上，可以根据实际情况，指定部署ceph mon节点（orch模块自动管理）\nceph orch apply mon \u0026#34;ceph-node01,ceph-node02,ceph-node03\u0026#34; 指定部署mon数量，orch模块自动部署，默认值为5 ceph orch apply mon 3 指定部署mon到指定标签节点（orch模块自动管理） ceph orch apply mon label:mon 禁用mon自动部署（可选） 这种情况下需要手动部署mon节点，mon节点不受orch模块自动管理\nceph orch apply mon --unmanaged 手动指定节点部署mon（可选） ceph orch daemon add mon ceph-node02:192.168.61.130 ceph orch daemon add mon ceph-node03:192.168.61.131 对节点打标签(可选) ceph orch host label add ceph-node01 mon ceph orch host label add ceph-node02 mon ceph orch host label add ceph-node03 mon ceph orch host label add ceph-node01 _admin ceph orch host label add ceph-node01 mgr ceph orch host label add ceph-node02 mgr 查询集群状态 查看Service状态 ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGNAME IMAGE ID alertmanager 1/1 7m ago 23h count:1 quay.iprometheus/alertmanager:v0.20.0 0881eb8f169f crash 3/3 7m ago 23h * quay.io/cepceph:v15 93146564743f grafana 1/1 7m ago 23h count:1 quay.io/cepceph-grafana:6.7.4 557c83e11646 mgr 2/2 7m ago 23h count:2 quay.io/cepceph:v15 93146564743f mon 3/3 7m ago 118m ceph-node01;ceph-node02;ceph-node03 quay.io/cepceph:v15 93146564743f node-exporter 3/3 7m ago 23h * quay.iprometheus/node-exporter:v0.18.1 e5a616e4b9cf prometheus 1/1 7m ago 23h count:1 quay.iprometheus/prometheus:v2.18.1 de242295e225 查询Deamon状态 ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.ceph-node01 ceph-node01 running (2h) 3m ago 4w 0.20.0 quay.io/prometheus/alertmanager:v0.20.0 0881eb8f169f 689490756acc crash.ceph-node01 ceph-node01 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 5c05118fec32 crash.ceph-node02 ceph-node02 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f bbc408935136 crash.ceph-node03 ceph-node03 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 87c1e705053e grafana.ceph-node01 ceph-node01 running (2h) 3m ago 4w 6.7.4 quay.io/ceph/ceph-grafana:6.7.4 557c83e11646 e211ea6be042 mgr.ceph-node01.wmeddo ceph-node01 running (2h) 3m ago 5d 15.2.17 quay.io/ceph/ceph:v15 93146564743f e465de9c3e9d mgr.ceph-node02.hyipxw ceph-node02 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 5026b159b240 mon.ceph-node01 ceph-node01 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 3f48f4045628 mon.ceph-node02 ceph-node02 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 3cf45764cafb mon.ceph-node03 ceph-node03 running (2h) 3m ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f f689c97923f6 node-exporter.ceph-node01 ceph-node01 running (2h) 3m ago 4w 0.18.1 quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf 12aec21e4c45 node-exporter.ceph-node02 ceph-node02 running (2h) 3m ago 4w 0.18.1 quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf 4fcfb820aa3b node-exporter.ceph-node03 ceph-node03 running (2h) 3m ago 4w 0.18.1 quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf 8aee37a4868b prometheus.ceph-node01 ceph-node01 running (2h) 3m ago 4w 2.18.1 quay.io/prometheus/prometheus:v2.18.1 de242295e225 aa30615a1ce0 查看orch管理的主机列表 ceph orch host ls HOST ADDR LABELS STATUS ceph-node01 ceph-node01 _admin mgr mon ceph-node02 ceph-node02 mgr mon ceph-node03 ceph-node03 mon 5. 部署osd 添加OSD需求满足以下所有条件：\n设备必须没有分区。\n设备不得具有任何LVM状态。\n不得安装设备。\n该设备不得包含文件系统。\n该设备不得包含Ceph BlueStore OSD。\n设备必须大于5 GB。\norch模块自动部署osd ceph orch apply osd --all-available-devices 暂停orch模块自动管理osd（可选） ceph orch apply osd --all-available-devices --unmanaged=true 手动新增osd（可选） ceph orch device ls ceph orch device zap ceph-node01 /dev/sdb --force ceph orch daemon add osd ceph-node01:/dev/sdb 移除osd ceph orch daemon stop osd.2 ceph orch osd rm 2 #等待数据平衡 ceph orch osd rm status ceph osd tree ceph osd out osd.2 #等待数据平衡 ceph osd purge osd.2 --force ceph osd crush ls ceph-node01 ceph orch daemon rm osd.2 --force 问题：wipefs: error: /dev/sdb: probing initialization failed: Device or resource busy 解决办法：\n移除逻辑卷： dmsetup remove --force ceph--cd03e720--2f3e--497e--8f7a--97432c2b675d-osd--block--c4b4700e--1001--473a--ab05--470b7e9c711e 擦除文件系统的签名 wipefs -a /dev/sdb 重新初始化 ceph orch device zap ceph-node01 /dev/sdb --force 问题：集群所有mgr全部down，集群报warning，无法使用orch管理。 找到mgr一个节点，mgr一般保存位置为：/var/lib/ceph/{fsid}/mgr.node.name\n通过cephadm临时拉起mgr服务：\ncephadm run --name {mgr.node.name} --fsid {fsid} 然后用orch模块拉起剩余mgr服务： ceph orch ps ceph orch daemon start mgr.ceph-node01.name 停掉cephadm临时拉起的mgr，然后用orch模块重启所有mgr： ceph orch restart mgr 6. 部署CEPHFS服务 使用ceph fs创建 创建cephfs文件系统，新版本ceph会自动创建mds服务，使用cephfs文件系统需要一个或多个MDS服务。\nceph fs volume create cephfs --placement=3 ceph fs status # 或者手动部署mds守护进程： ceph orch apply mds cephfs --placement 3 ceph orch ls 查看ceph fs状态，可以显示主备mds节点信息，文件系统情况\nceph fs status ceph fs ls ceph mds stat 手动创建 # 创建fs data池，启用erasure纠删功能，也可以使用replicated功能。 ceph osd pool create cephfs.cephfs.data 32 erasure # 创建fs metadata池，只能默认replicated复制功能 ceph osd pool create cephfs.cephfs.meta 32 # 创建ceph fs，附上metadata和data池 ceph fs new cephfs cephfs.cephfs.meta cephfs.cephfs.data # 手动部署cephfs mds服务 ceph orch apply mds cephfs --placement=3 （可选）可将cephfs的数据池设置启用纠删码(pool创建时需要指定为erasure参数)，然后启用ec overwrites。但cephfs metadata元数据池不能使用纠删码，因为元数据使用OMAP数据结构所以不支持。\nceph osd pool set cephfs.cephfs.data allow_ec_overwrites true Linux挂载cephfs文件系统方法 生成ceph配置文件至客户端主机 mkdir -p -m 755 /etc/ceph ssh {user}@{mon-host} \u0026#34;sudo ceph config generate-minimal-conf\u0026#34; | sudo tee /etc/ceph/ceph.conf chmod 644 /etc/ceph/ceph.conf 生成cephx认证信息 ssh {user}@{mon-host} \u0026#34;sudo ceph fs authorize cephfs client.mountfs / rw\u0026#34; | sudo tee /etceph/ceph.client.mountfs.keyring chmod 600 /etc/ceph/ceph.client.mountfs.keyring 当使用内核驱动挂载cephfs时使用（泄露密钥风险） # mount -t ceph {device-string}:{path-to-mounted} {mount-point} -o {key-value-args{other-args} mount -t ceph 192.168.61.129:6789,192.168.61.130:6789,192.168.61.131:6789:/ /mnt -name=mountfs,secret=AQBtO3dj8bjLNxAAGFHw02lPnlbapyAVwwIBuA== 使用mount.ceph挂载（需要安装ceph-common包） mount -t ceph :/ /mnt -o name=mountfs,fs=cephfs 写入/etc/fstab文件 :/ /mnt ceph name=mountfs,[fs=cephfs,secret=str,]noatime,[nodiratime,]_netdev 0 2 7. 部署CEPH NFS服务 仅支持 NFSv4 协议。\n创建新池 ceph osd pool create ganesha_data 32 ceph osd pool application enable ganesha_data nfs 创建NFS网关 ceph orch apply nfs nfs ganesha_data --placement=2 查看Service状态 ceph orch ls NAME RUNNING REFRESHED AGE PLACEMENT IMAGE NAME IMAGE ID alertmanager 1/1 46s ago 4w count:1 quay.io/prometheus/alertmanager:v0.20.0 0881eb8f169f crash 3/3 48s ago 4w * quay.io/ceph/ceph:v15 93146564743f grafana 1/1 46s ago 4w count:1 quay.io/ceph/ceph-grafana:6.7.4 557c83e11646 mds.cephfs 2/2 48s ago 5d count:2 quay.io/ceph/ceph:v15 93146564743f mgr 2/2 48s ago 9d ceph-node01;ceph-node02 quay.io/ceph/ceph:v15 93146564743f mon 3/3 48s ago 4w ceph-node01;ceph-node02;ceph-node03 quay.io/ceph/ceph:v15 93146564743f nfs.nfs 2/2 46s ago 11m count:2 quay.io/ceph/ceph:v15 93146564743f node-exporter 3/3 48s ago 4w * quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf osd.all-available-devices 3/3 48s ago 9d * quay.io/ceph/ceph:v15 93146564743f prometheus 1/1 46s ago 4w count:1 quay.io/prometheus/prometheus:v2.18.1 de242295e225 查看daemon状态 ceph orch ps NAME HOST STATUS REFRESHED AGE VERSION IMAGE NAME IMAGE ID CONTAINER ID alertmanager.ceph-node01 ceph-node01 running (5h) 81s ago 4w 0.20.0 quay.io/prometheus/alertmanager:v0.20.0 0881eb8f169f 689490756acc crash.ceph-node01 ceph-node01 running (5h) 81s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 5c05118fec32 crash.ceph-node02 ceph-node02 running (5h) 84s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f bbc408935136 crash.ceph-node03 ceph-node03 running (5h) 58s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 87c1e705053e grafana.ceph-node01 ceph-node01 running (5h) 81s ago 4w 6.7.4 quay.io/ceph/ceph-grafana:6.7.4 557c83e11646 e211ea6be042 mds.cephfs.ceph-node02.mxqelr ceph-node02 running (5h) 84s ago 5d 15.2.17 quay.io/ceph/ceph:v15 93146564743f 549606028df4 mds.cephfs.ceph-node03.xhcgto ceph-node03 running (5h) 58s ago 5d 15.2.17 quay.io/ceph/ceph:v15 93146564743f ee51fb9c1a8b mgr.ceph-node01.wmeddo ceph-node01 running (5h) 81s ago 6d 15.2.17 quay.io/ceph/ceph:v15 93146564743f e465de9c3e9d mgr.ceph-node02.hyipxw ceph-node02 running (5h) 84s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 5026b159b240 mon.ceph-node01 ceph-node01 running (5h) 81s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 3f48f4045628 mon.ceph-node02 ceph-node02 running (5h) 84s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 3cf45764cafb mon.ceph-node03 ceph-node03 running (5h) 58s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f f689c97923f6 nfs.nfs.ceph-node01 ceph-node01 running (11m) 81s ago 11m 3.3 quay.io/ceph/ceph:v15 93146564743f 117e58e5990c nfs.nfs.ceph-node03 ceph-node03 running (60s) 58s ago 11m 3.3 quay.io/ceph/ceph:v15 93146564743f 3662ecedd542 node-exporter.ceph-node01 ceph-node01 running (5h) 81s ago 4w 0.18.1 quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf 12aec21e4c45 node-exporter.ceph-node02 ceph-node02 running (5h) 84s ago 4w 0.18.1 quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf 4fcfb820aa3b node-exporter.ceph-node03 ceph-node03 running (5h) 58s ago 4w 0.18.1 quay.io/prometheus/node-exporter:v0.18.1 e5a616e4b9cf 8aee37a4868b osd.0 ceph-node03 running (5h) 58s ago 4w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 4d8635a833cc osd.1 ceph-node02 running (5h) 84s ago 3w 15.2.17 quay.io/ceph/ceph:v15 93146564743f 6fa84fe728c7 osd.2 ceph-node01 running (5h) 81s ago 9d 15.2.17 quay.io/ceph/ceph:v15 93146564743f 16d42b3583ea prometheus.ceph-node01 ceph-node01 running (5h) 81s ago 4w 2.18.1 quay.io/prometheus/prometheus:v2.18.1 de242295e225 aa30615a1ce0 NFS 高可用(高版本支持) ingress nfs 部署文件：ingress-nfs.yml service_type: ingress service_id: nfs.nfs placement: count: 2 spec: backend_service: nfs.nfs frontend_port: 2050 # 对外服务端口 monitor_port: 9000 # haproxy查看状态页面 virtual_ip: 192.168.61.140/24 部署 ceph orch apply -i ingress-nfs.yaml 8. 部署RGW服务 ceph orch apply rgw object-service us-east-1 --placement=3 默认情况下，cephadm会自动创建realm和zone，或者也可以手动创建realm，zonegroup，zone。\nradosgw-admin realm create --rgw-realm=\u0026lt;realm-name\u0026gt; --default radosgw-admin zonegroup create --rgw-zonegroup=\u0026lt;zonegroup-name\u0026gt; --master --default radosgw-admin zone create --rgw-zonegroup=\u0026lt;zonegroup-name\u0026gt; --rgw-zone=\u0026lt;zone-name\u0026gt; --master --default radosgw-admin period update --rgw-realm=\u0026lt;realm-name\u0026gt; --commit ","permalink":"https://gnail89.github.io/posts/install-ceph-octopus-with-cephadm/","summary":"1. 环境信息 CentOS 7 禁用firewalld 禁用selinux 时钟同步 设置主机名 ip addr hostname 192.168.61.129 ceph-node01 192.168.61.130 ceph-node02 192.168.61.131 ceph-node03 2. 安装依赖包 所有节点都需要安装\n安装epel和elrepo软件源 yum install -y epel-release elrepo-release 安装python3 yum install -y python3 升级kernel至最新 yum --enablerepo=elrepo-kernel install -y kernel-ml # 设置默认使用的内核 grubby --default-index grubby --info=ALL grubby --set-default-index=0 安装docker-ce软件源 cd /etc/yum.repos.d/ curl --remote-name https://download.docker.com/linux/centos/docker-ce.repo 安装最新版本docker yum install -y docker-ce systemctl start docker systemctl enable docker 重启生效 3. 安装cephadm curl --remote-name --location https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-octopus/elnoarch/cephadm chmod +x .","title":"Install Ceph Octopus With Cephadm"},{"content":"1. 基本语法 Markdown语法主要分类: 标题, 段落, 区块引用, 代码区块, 强调, 列表, 分割线, 链接, 图片, 反斜杠\\, 符号\u0026rsquo;`' 标题 使用=和-符号标记一级和二级标题 这是一级标题 ========== 这是二级标题 ---------- 使用#符号，可以表示1-6级标题 # 一级标题 ## 二级标题 ### 三级标题 #### 四级标题 ##### 五级标题 ###### 六级标题 段落 段落内强制换行使用两个以上空格加上回车, 引用中换行可省略回车 区块引用 在段落的每行或只在第一行使用符号\u0026gt;,还可使用多个嵌套引用 \u0026gt; 区块引用 \u0026gt;\u0026gt; 嵌套引用 代码区块 代码区块的建立是在每行加上4个空格或者一个制表符 注意:需要和普通段落之间存在空行。\nvoid main() { printf(\u0026quot;Hello, Markdown.\u0026quot;); } 强调 在强调内容两侧分别加上*或者_ *斜体* _斜体_ **粗体** __粗体__ ***斜体加粗*** ~~删除线~~ 列表 使用星号*,加号+,减号-标记无序列表 * 无序列表文字 * 无序列表文字 * 无序列表文字 + 无序列表文字 + 无序列表文字 + 无序列表文字 - 无序列表文字 - 无序列表文字 - 无序列表文字 有序列表的标记方式是将上述的符号换成数字,并加上英文句号.,后面还有一个空格 1. 有序列表 2. 有序列表 3. 有序列表 分割线 分割线最常使用就是三个或以上*或-或_ 链接 链接可以由两种形式生成：行内式和参考式\n行内式:\n[显示名称](https://www.google.com \u0026#34;链接别名\u0026#34;) 效果:\n显示名称\n参考式: [显示名称1][1] [显示名称2][2] [1]:https://www.google.com \u0026#34;链接别名1\u0026#34; [2]:https://www.google.com \u0026#34;链接别名2\u0026#34; 效果: 显示名称1 显示名称2\n图片 添加图片的形式和链接相似，只需在链接的基础上前方加一个英文感叹号! ![显示名称](https://www.google.com/logo.png \u0026#34;链接别名\u0026#34;) 反斜杠\\ 相当于反转义作用。使符号成为普通符号 以下列出的字符都可以通过使用反斜杠字符从而达到转义目的。 Character Name \\ backslash ` backtick (see also escaping backticks in code) * asterisk _ underscore { } curly braces [ ] brackets ( ) parentheses # pound sign + plus sign - minus sign (hyphen) . dot ! exclamation mark | pipe (see also escaping pipe in tables) 符号\u0026rsquo;`' 起到标记的作用, 例如: `point` 效果:\npoint\n2. 扩展语法 表格 要添加表，请使用三个或多个连字符（---）创建每列的标题，并使用管道（|）分隔每列。您可以选择在表的任一端添加管道。\n| Syntax | Description | | ----------- | ----------- | | Header | Title | | Paragraph | Text | 效果： Syntax Description Header Title Paragraph Text 对齐 通过在标题行中的连字符的左侧，右侧或两侧添加冒号（:），将列中的文本对齐到左侧，右侧或中心。\n| Syntax | Description | Test Text | | :--- | :----: | ---: | | Header | Title | Here\u0026#39;s this | | Paragraph | Text | And more | 效果： Syntax Description Test Text Header Title Here\u0026rsquo;s this Paragraph Text And more 围栏代码块 Markdown基本语法允许您通过将行缩进四个空格或一个制表符来创建代码块。如果发现不方便，请尝试使用受保护的代码块。根据Markdown处理器或编辑器的不同，您将在代码块之前和之后的行上使用三个反引号(```）或三个波浪号（~~~）。\n``` { \u0026quot;firstName\u0026quot;: \u0026quot;John\u0026quot;, \u0026quot;lastName\u0026quot;: \u0026quot;Smith\u0026quot;, \u0026quot;age\u0026quot;: 25 } ``` 效果： { \u0026#34;firstName\u0026#34;: \u0026#34;John\u0026#34;, \u0026#34;lastName\u0026#34;: \u0026#34;Smith\u0026#34;, \u0026#34;age\u0026#34;: 25 } 语法高亮 许多Markdown处理器都支持受围栏代码块的语法突出显示。使用此功能，您可以为编写代码的任何语言添加颜色突出显示。要添加语法突出显示，请在受防护的代码块之前的反引号旁边指定一种语言。\n```json { \u0026quot;firstName\u0026quot;: \u0026quot;John\u0026quot;, \u0026quot;lastName\u0026quot;: \u0026quot;Smith\u0026quot;, \u0026quot;age\u0026quot;: 25 } ``` 效果： { \u0026#34;firstName\u0026#34;: \u0026#34;John\u0026#34;, \u0026#34;lastName\u0026#34;: \u0026#34;Smith\u0026#34;, \u0026#34;age\u0026#34;: 25 } 定义列表 一些Markdown处理器允许您创建术语及其对应定义的定义列表。要创建定义列表，请在第一行上键入术语。在下一行，键入一个冒号，后跟一个空格和定义。\nFirst Term : This is the definition of the first term. Second Term : This is one definition of the second term. : This is another definition of the second term. 效果： First Term This is the definition of the first term. Second Term This is one definition of the second term. This is another definition of the second term. 任务列表语法 任务列表使您可以创建带有复选框的项目列表。在支持任务列表的Markdown应用程序中，复选框将显示在内容旁边。要创建任务列表，请在任务列表项之前添加破折号-和方括号[ ]，并在[ ]前面加上空格。要选择一个复选框，请在方括号[x]之间添加 x 。\n- [x] Write the press release - [ ] Update the website - [ ] Contact the media 效果： Write the press release Update the website Contact the media 使用 Emoji 表情 有两种方法可以将表情符号添加到Markdown文件中：将表情符号复制并粘贴到Markdown格式的文本中，或者键入emoji shortcodes。\n复制和粘贴表情符号 在大多数情况下，您可以简单地从Emojipedia等来源复制表情符号并将其粘贴到文档中。许多Markdown应用程序会自动以Markdown格式的文本显示表情符号。从Markdown应用程序导出的HTML和PDF文件应显示表情符号。\nTip: 如果您使用的是静态网站生成器，请确保将HTML页面编码为UTF-8。\n使用表情符号简码 一些Markdown应用程序允许您通过键入表情符号短代码来插入表情符号。这些以冒号开头和结尾，并包含表情符号的名称。\n图钉 :pushpin: 效果： 图钉 \u0026#x1f4cc;\n自动网址链接 许多Markdown处理器会自动将URL转换为链接。这意味着如果您输入http://www.example.com，即使您未使用方括号，您的Markdown处理器也会自动将其转换为链接。\nhttp://www.example.com 效果： http://www.example.com\n禁用自动URL链接 如果您不希望自动链接URL，则可以通过将URL表示为带反引号(`)的代码来删除该链接。\n`http://www.example.com` 效果： http://www.example.com\n脚注 Here\u0026#39;s a simple footnote,[^1] and here\u0026#39;s a longer one.[^bignote] [^1]: This is the first footnote. [^bignote]: Here\u0026#39;s one with multiple paragraphs and code. Indent paragraphs to include them in the footnote. `{ my code }` Add as many paragraphs as you like. 效果： Here\u0026rsquo;s a simple footnote,1 and here\u0026rsquo;s a longer one.2\n参考资料3\nThis is the first footnote.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nHere\u0026rsquo;s one with multiple paragraphs and code.\nIndent paragraphs to include them in the footnote.\n{ my code }\nAdd as many paragraphs as you like.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nhttps://markdown.com.cn\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://gnail89.github.io/posts/markdown/","summary":"1. 基本语法 Markdown语法主要分类: 标题, 段落, 区块引用, 代码区块, 强调, 列表, 分割线, 链接, 图片, 反斜杠\\, 符号\u0026rsquo;`' 标题 使用=和-符号标记一级和二级标题 这是一级标题 ========== 这是二级标题 ---------- 使用#符号，可以表示1-6级标题 # 一级标题 ## 二级标题 ### 三级标题 #### 四级标题 ##### 五级标题 ###### 六级标题 段落 段落内强制换行使用两个以上空格加上回车, 引用中换行可省略回车 区块引用 在段落的每行或只在第一行使用符号\u0026gt;,还可使用多个嵌套引用 \u0026gt; 区块引用 \u0026gt;\u0026gt; 嵌套引用 代码区块 代码区块的建立是在每行加上4个空格或者一个制表符 注意:需要和普通段落之间存在空行。\nvoid main() { printf(\u0026quot;Hello, Markdown.\u0026quot;); } 强调 在强调内容两侧分别加上*或者_ *斜体* _斜体_ **粗体** __粗体__ ***斜体加粗*** ~~删除线~~ 列表 使用星号*,加号+,减号-标记无序列表 * 无序列表文字 * 无序列表文字 * 无序列表文字 + 无序列表文字 + 无序列表文字 + 无序列表文字 - 无序列表文字 - 无序列表文字 - 无序列表文字 有序列表的标记方式是将上述的符号换成数字,并加上英文句号.","title":"Markdown基本语法"},{"content":"教程 Markdown 教程 Pro Git 2 简体中文\ngit在线学习工具\n菜鸟教程\nLinux命令大全\n鸟哥教程\n工具 开源中国在线工具\n菜鸟教程在线工具\n程序员在线工具\n脚本之家在线工具\nw3cschool在线工具\n在线压缩转换工具\n码云开源项目推荐列表\n博客 Java全栈知识体系 ","permalink":"https://gnail89.github.io/posts/website-favorites/","summary":"教程 Markdown 教程 Pro Git 2 简体中文\ngit在线学习工具\n菜鸟教程\nLinux命令大全\n鸟哥教程\n工具 开源中国在线工具\n菜鸟教程在线工具\n程序员在线工具\n脚本之家在线工具\nw3cschool在线工具\n在线压缩转换工具\n码云开源项目推荐列表\n博客 Java全栈知识体系 ","title":"网站收藏夹"}]