最近升级服务器的一些笔记

Leave a reply

最近新采购了一批硬件用于升级现有DevOps服务，相关的软件也都更新到较新的版本。本次升级比较顺利，没有遇到担心的兼容性问题，放一下配置供参考。

Storage Server

Intel Xeon E5-2603 v3
Supermicro X10SRL-F
Samsung ECC DDR4-2400 16GB x 4
Chelsio T520-CR
LSI SAS9217-8i
InnoDisk SSD 32GB
WD HUS726T6TALE6L4 6TB 7.2K SATA x12 (2 stripped raidz2)
Intel Optane 900p 280G PCI-E (as ZIL/SLOG)
Samsung 970 EVO PLUS 1TB M.2 (as L2ARC)
FreeNAS 11.2-U2

Virtualization Server

Intel Xeon E5-2699 v3 x 2
Supermicro X10DAi
Samsung ECC DDR4-2400 32GB x 8
Chelsio T520-CR
InnoDisk SSD 32GB
VMware ESXi 6.5 U2

Switch & Cables

UBNT ES-16-XG
10GTek SFP+DAC

Tips:

ESXi 6.5 U2 安装 Chelsio T520-CR 驱动
https://service.chelsio.com/beta/drivers/esx_2_0_0_32_uwire/Chelsio_UnifiedWire_ESXi_UserGuide.pdf

ZIL/SLOG
对于没有强制 SYNC WRITE 的场景，一点儿用处都没有。

MTU
9000 比 1500 性能好？未必！相反 9000 会带来额外的一些问题。

MPIO
iSCSI 实测可以跑满两根 10Gbps 。

WSL cause C1083

Leave a reply

遇坑：
在 WSL 下用 git clone 的代码，可以在编辑器（Qt Creator）中正常打开；qmake 执行也正常，但是编译时报 c1xx : fatal error C1083。

解决：
发现是 WSL 的问题，在 Windows 下重新复制（或打包再解包）就 OK 了。

参考：
https://developercommunity.visualstudio.com/content/problem/240102/problem-with-compile-c-project-c1xx-fatal-error-c1.html

Wake on LAN in Windows 10

Leave a reply

最近新组装了一台小钢炮（Ryzen 2700x），打算利用 WOL 方便远程使用，没想到遇到一个小坑。
一开始，折腾了 BIOS 中所有能改的设置，不行；
后来安装了 Intel 的官方驱动，设置了还是不行：可以在睡眠状态（S4）下唤醒，但关机状态（S5）下无效。
后来在 ASUS Forum 看到类似的问题，其中提到可能是 Windows 10 的原因：

“Wake on LAN” (WOL) behavior in Windows 8, Windows 8.1 and Windows 10

The “Wake on LAN” (WOL) feature wakes a computer from a low-power state when a network adapter detects a WOL event. Typically, such an event is a specially constructed Ethernet packet. The default behavior in response to WOL events has changed from Windows 7 to Windows 8 and Windows 10.

Windows 7

In Windows 7, the default shutdown operation puts the system into the classic shutdown state (S5), and all devices are put into the lowest power state (D3). WOL from S5 is not officially supported in Windows 7. However, some network adapters can be left armed for waking if enough residual power is available. Therefore, waking from S5 is possible on some systems if enough residual power is supplied to the network adapter even though the system is in the S5 state and devices are in D3.

Windows 8, Windows 8.1 and Windows 10

In Windows 8, 8.1 and Windows 10, the default shutdown behavior puts the system into the hybrid shutdown state (S4), and all devices are put into D3. WOL from S4 or S5 is unsupported. Network adapters are explicitly not armed for WOL in either S5 or S4 cases because users expect zero power consumption and battery drain in the shutdown state. This behavior removes the possibility of invalid wake-ups when an explicit shutdown is requested. Therefore, WOL is supported only from sleep (S3) or hibernation (S4) states in Windows 8, 8.1 and Windows 10.

In Windows 8, 8.1 and Windows 10, hybrid shutdown (S4) stops user sessions but lets the contents of kernel sessions be written to the hard disk. This enables faster startups.

To disable the S4 state in Windows 8, 8.1 and Windows 10, follow these steps.

Note We do not recommend that you disable the hybrid shutdown (S4) state.

  1. In Control Panel, open the Power Options item.
  2. Click the Choose what the power buttons do link.
  3. Clear the Turn on fast startup (recommended) check box.
  4. Click Save Settings.

按照文中的说明，改了电源计划，果然 OK 了。

Linux clock source problem with Skylake-X processors

Leave a reply

0x00

最近将一个在CentOS/RHEL 7.x下开发、运行的程序尝试移植到Ubuntu 16.04下，发现性能骤降，用perf top检查发现__vdso_clock_gettime和read_hpet的时间开销非常巨大。

0x01

Google搜了一圈：Two frequently used system calls are ~77% slower on AWS EC2
感觉是__vdso_clock_gettime没有能按照预期工作，而是每次都进内核读取硬件时钟。
但是我们的环境都是baremetal，并不是VM，理论上不应该是上述文中提到的原因。

0x02

因为代码中不少地方使用了clock_gettime和asio的timer，跟踪下去发现asio的底层使用了HPET，但是每个io_service仅使用了一个timerfd，大量的read_hpet应该是由clock_gettime调用的，一般来说clock_gettime不应该使用HPET这么高精度的时钟源。
顺着这个思路，发现两台机器的时钟源果然不同
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
出问题的这台显示是hpet，而正常应该为tsc。

再查看当前可用时钟源，发现竟然没有tsc！
$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource

离真相已经不远了。

0x03

出问题的是Intel Core i9-7900x（代号Skylake-X），怀疑是内核对新CPU的支持还不完善。
在 Kernel.org 的 Bugzilla 和 git log 里面发现类似的反馈：

Skylake-X的TSC晶振频率应该是25MHz，不是24MHz。内核在检测TSC时发现偏差，从而禁用了TSC，转用了HPET作为默认时钟源。

0x04

解决方案就是换最新的内核，我选用的是4.15.3，再验证可用时钟源和当前时钟源均为tsc无误，程序运行也恢复正常。

PS：至于开头__vdso_clock_gettime为何没有按照预期运行，而是每次都执行read_hpet，暂时未深入研究具体代码，以后有空再补……

Build Skia & libyuv

Leave a reply

Skia

export HTTPS_PROXY=<YOUR_HTTP_PROXY>
export PATH="${DEPOT_TOOLS_DIR}:${PATH}"
git clone https://skia.googlesource.com/skia.git
cd skia
python tools/git-sync-deps
bin/gn gen out/Static --args='is_official_build=true skia_enable_gpu=false skia_enable_pdf=false skia_enable_discrete_gpu=false skia_use_system_libjpeg_turbo=false skia_use_system_libpng=false skia_use_system_libwebp=false'
ninja -C out/Static

libyuv

export HTTPS_PROXY=<YOUR_HTTP_PROXY>
export PATH="${DEPOT_TOOLS_DIR}:${PATH}"
gclient config --name src https://chromium.googlesource.com/libyuv/libyuv
mkdir -p cache
sed -i 's/cache_dir = None/cache_dir = \"cache\"/g' .gclient
gclient sync
cd src
gn gen out/Release "--args=is_debug=false"
ninja -C out/Release

注：在梯子不稳定的情况下，启用cache_dir能解决git clone third_party中断的问题。

 

eglMakeCurrent返回EGL_BAD_ALLOC

Leave a reply

环境：Android、OpenGL ES 3.0
硬件：T-Firefly 3288（RK3288）、昂达 V10 Pro（MTK8173）、小米平板2（Intel Atom Z8500）

现象：创建两个EGLSurface，一个为offscreen的FBO，另一个为SurfaceView，先对第一个EGLSurface调用makeCurrent，不进行任何绘制操作，再对第二个EGLSurface调用makeCurrent，程序崩溃，堆栈如下：
04-18 20:46:19.597 3232-3328/sanbu.mc_400 E/mali_so: encounter the first mali_error : 0x0003 : execution failed (gles_fb_context_flush at hardware/arm/maliT760/driver/product/gles/src/fb/mali_gles_fb_module_api.c:1395)
04-18 20:46:19.597 3232-3328/sanbu.mc_400 E/mali_so: to dump the call_stack of the first error :
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #00 pc 0042fbf4 /system/vendor/lib/egl/rk3288/libGLES_mali.so
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #01 pc 003fbfa4 /system/vendor/lib/egl/rk3288/libGLES_mali.so
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #02 pc 003d5874 /system/vendor/lib/egl/rk3288/libGLES_mali.so (eglMakeCurrent+980)
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #03 pc 0001016f /system/lib/libEGL.so (android::egl_display_t::makeCurrent(android::egl_context_t*, android::egl_context_t*, void*, void*, void*, void*, void*, void*)+96)
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #04 pc 0001286f /system/lib/libEGL.so (eglMakeCurrent+266)
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #05 pc 0006323d /system/lib/libandroid_runtime.so
04-18 20:46:19.676 3232-3328/sanbu.mc_400 D/mali_so: #06 pc 00bf6ef7 /data/dalvik-cache/arm/system@framework@boot.oat
04-18 20:46:19.676 3232-3328/sanbu.mc_400 E/libEGL: eglMakeCurrent:786 error 3003 (EGL_BAD_ALLOC)
这个问题仅在瑞芯微的RK3288平台上存在，另外两家（MTK和Intel）设备运行正常。

根据堆栈打印的信息和在其它设备上测试的结果，初步推断该问题很大程度上与GPU硬件实现或驱动相关。

规避的方式：在切换两个surface的context之间进行一次绘制操作。

不知道这个锅是ARM Mali的还是RK的，不过RK的东西真不咋的。

rsyslog处理大量冗余message

Leave a reply

一年前给 Buffalo LS-WXL 重新灌了 Debian Wheezy 后，貌似 smartd/smartctl 和原先的 kernel “水土不服“，dmesg 中频繁打印：

program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO

估计是 smartctl 对 marvell 的支持滞后了，刚开始也懒得理它，后来发现打印是在太频繁了，导致 /var/log 下的日志文件占用了太多的空间：

Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 10:00:21 HOME-NAS kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO

在不想花费太多时间的情况下，想到从 rsyslog 下手，有两种方式：

方法一：合并重复的message，合并输出

在 /etc/rsyslog.conf 中加入
###########################
#### GLOBAL DIRECTIVES ####
###########################
$RepeatedMsgReduction on

最终日志文件会生成如下内容：
Jan 19 18:01:34 home-nas kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Jan 19 18:02:05 home-nas kernel: last message repeated 9 times
Jan 19 18:03:06 home-nas kernel: last message repeated 28 times
Jan 19 18:04:07 home-nas kernel: last message repeated 29 times
Jan 19 18:05:25 home-nas kernel: last message repeated 30 times
Jan 19 18:06:32 home-nas kernel: last message repeated 53 times
Jan 19 18:07:32 home-nas kernel: last message repeated 28 times
Jan 19 18:08:32 home-nas kernel: last message repeated 28 times

方法二：过滤不需要记录的message，完全禁止输出
在 /etc/rsyslog.conf 中加入
###############
#### RULES ####
###############

:msg, contains, "program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO" ~

注意：末尾的~符号是同一行，格式为：
:msg contains "要匹配的文本" ~

完美解决~~~

 

 

宽带升级到100M之后……

Leave a reply

前两天电信客服来电说可以把家里的宽带免费升级到100M，在外面想用家里的设备测一下实际的上下行速度。家里 7×24 小时开机的设备有一台 Mikrotik RB751G-2HnD 和一台 Buffalo LS-WXL （重刷了 Debian Wheezy 7.9），那就看看SpeedTest有没有 Linux Console 版本的吧。

果然在GitHub 上找到一个：https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
# wget https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
# chmod +x speedtest_cli.py
# ./speedtest_cli.py

执行下，结果还不错
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from China Telecom (121.231.9.15)...
Selecting best server based on latency...
Hosted by CHINA UNICOM JIANGSU COMPANY (Nanjing) [117.84 km]: 7.761 ms
Testing download speed........................................
Download: 93.06 Mbit/s
Testing upload speed..................................................
Upload: 4.23 Mbit/s

Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from China Telecom (49.89.188.61)...
Selecting best server based on latency...
Hosted by China Telecom JiangSu Branch (Nanjing) [198.70 km]: 11.072 ms
Testing download speed........................................
Download: 102.80 Mbit/s
Testing upload speed..................................................
Upload: 3.79 Mbit/s

再测一下 BT 下载速度，NAS 上用 transmission 下Fedora-24-Server 的镜像，速度完全只能达到3~4MB/s，dstat 一看，CPU 竟然满载……

再看一下 ROS 的 CPU Usage 竟然也达到 60~70%

看来是时候考虑新一波的设备升级计划了~~~