【最佳實踐】prometheus 監控 sql server （使用sql_exporter）

【0】核心參考

sql_exporter使用、采集器源碼： https://github.com/free/sql_exporter 

MSSQL簡略儀表盤：https://grafana.com/grafana/dashboards/9336

更多、更高級的sql server采集指標：https://github.com/influxdata/telegraf/tree/master/plugins/inputs/sqlserver?tdsourcetag=s_pctim_aiomsg

sql server 對象監控 參考：https://docs.microsoft.com/zh-cn/sql/relational-databases/performance-monitor/use-sql-server-objects?view=sql-server-ver15

【簡述】

MSSQL的監控，官網沒有提供采集器，所以只能用其他程序來監控，本文就用了 sql-exporter。

它可以理解成一個遠程連接數據庫的工具，可以用它來連接sql server/mysql 等等數據庫，並以SQL查詢方式采集SQL查詢結果。

SQL_Exporter 是中心化的，可以把對不同實例的agent鏈接，都放在一台linux服務器上，以便管理、修改。你想想看，如果你更新了采集指標，只需要在該台中心化服務器重啟sql_exporter agent采集客戶端即可生效。

不像其他 mysql/linux/windows采集器，他們是部署在實際被采集的服務器上的，如果你想要更新采集器（雖然它做的很好了，不需要更新。除此之外就是無法加入自定義的采集指標或者采集項），那么需要到那么多台被采集的客戶端機器上去覆蓋更新，很痛苦的。

【1】安裝配置 sql_exporter

【1.1】下載解壓 sql_exporter

下載：https://github.com/free/sql_exporter/releases

#wget https://github.com/free/sql_exporter/releases/download/0.5/sql_exporter-0.5.linux-amd64.tar.gz

mkdir  /soft
cd /soft
wget https://github.com/free/sql_exporter/releases/download/0.5/sql_exporter-0.5.linux-amd64.tar.gz
tar -zxf sql_exporter-0.5.linux-amd64.tar.gz 
ln -s sql_exporter-0.5.linux-amd64 sql_exporter
cd sql_exporter

 【1.2】修改配置文件

# Global defaults.
global:
  # Subtracted from Prometheus' scrape_timeout to give us some headroom and prevent Prometheus from timing out first.
  scrape_timeout_offset: 500ms
  # Minimum interval between collector runs: by default (0s) collectors are executed on every scrape.
  min_interval: 0s
  # Maximum number of open connections to any one target. Metric queries will run concurrently on multiple connections,
  # as will concurrent scrapes.
  max_connections: 10
  # Maximum number of idle connections to any one target. Unless you use very long collection intervals, this should
  # always be the same as max_connections.
  max_idle_connections: 5

# The target to monitor and the collectors to execute on it.
target:
  # Data source name always has a URI schema that matches the driver name. In some cases (e.g. MySQL)
  # the schema gets dropped or replaced to match the driver expected DSN format.
  # data_source_name: 'sqlserver://sql_exporter:a123456!@192.168.191.81:1433/?encrypt=disable'
  data_source_name: 'sqlserver://sa:a123456!@192.168.191.81:1433'
   


  # Collectors (referenced by name) to execute on the target.
  collectors: [mssql_standard]

# Collector files specifies a list of globs. One collector definition is read from each matching file.
collector_files:
  - "*.collector.yml"

解析：

（1）global

收集器中允許最慢的SQL執行超時時間，注意該超時時間應小於prometheus中的 scrape_time

scrape_timeout_offset: 500ms #用於從 prometheus的 scrape_timeout 中減去一個偏移時間，防止 prometheus 先超時，如果設置了 scrape_timeout,scrape
min_interval: 0s  #收集器每隔0運行一次（默認情況下）收集器每隔0運行一次。
max_connections: 10 #到任何一個目標的最大打開連接數。采集器信息查詢將在多個連接上並發運行，
max_idle_connections: 5 #到任何一個目標的最大空閑連接數。除非使用很長的收集間隔，否則應該

（2）target

#數據源

data_source_name: 'sqlserver://sa:a123456!@192.168.191.81:1433'

 

（3）collector

#引用收集器文件

collector_files:

  - "*.collector.yml"

【1.3】自帶的sql server監控采集器 

　　

 

 這里我們配置文件中 已經引用了 配置文件相同目錄下的 "*.collector.yml"，所以該文件也被包含進來了。

【2】整合 prometheus + sql_exporter

那么這個東西其實是一個采集器啊，但是為什么不能放到windows上呢.......好吧，不管了好像沒什么很好的windows采集器，自己也不會做，先用着吧

【2.1】修改prometheus.yml配置文件

　　

 

 

 

 

 

【2.2】啟動 sql_exporter  

（1）封裝成系統服務

[Unit]
Description=sql_exporter

[Service]
Type=simple
ExecStart=/soft/sql_exporter/sql_exporter -config.file /soft/sql_exporter/sql_exporter.yml
Restart=on-failure

[Install]
WantedBy=multi-user.target

（2）啟動、查看

systemctl daemon-reload
systemctl start sql_exporter
systemctl status sql_exporter -l

啟動成功，並且 默認端口是 9399

　　

 

 

【2.3】核驗

http://192.168.175.131:9399/metrics

如下圖，這就成功了啊

　　

 

 

【3】結合 grafana 顯示

【3.1】導入MSSQL模板

https://grafana.com/grafana/dashboards?dataSource=prometheus&search=mssql

 　　

　　

 

 導入這個 9336 模板

　　

 

【3.2】查看儀表盤 

最后結果：好像還行，但很多圖表no data ，這個模板和采集器不是很配，有空可以自己配置修改一下

　　

 

【4】自定義MSSQL

【4.0】MSSQL權限與監控賬戶

DECLARE @sql VARCHAR(max)
SET @sql=CAST('use master;CREATE LOGIN [sql_exporter] WITH PASSWORD=N''qICJEasdqwDiOSrdT96'', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF; GRANT VIEW SERVER STATE TO [sql_exporter];
GRANT VIEW ANY DEFINITION TO [sql_exporter];' AS VARCHAR(max))

select @sql=@sql+CAST('use '+name+';CREATE USER [sql_exporter] FOR LOGIN [sql_exporter];
EXEC sp_addrolemember N''db_datareader'', N''sql_exporter'';'+CHAR(10) AS VARCHAR(max)) 
from master.sys.databases  and state=0
EXEC(@sql)

防火牆什么的就不用我說了吧？

【4.1】自定義采集器

【4.2】采集器啟動

布置在linux 中間節點，也可以直接布置在prometheus節點上

如果密碼有特殊字符，在URL上使用報錯，則參見附錄，用特殊字符替換

比如密碼： !@#$%^qwe123   轉移成 %21%40%23%24%25%5Eqwe123

nohup /data/mssql/sql_exporter -config.data-source-name=sqlserver://sql_exporter:qICJEasdqwDiOSrdT96@10.112.5.106:1433/?encrypt=disable -config.file=/data/mssql/sql_exporter.yml -web.listen-address=127.0.0.1:9400 -log_dir=/data/mssql_log &

nohup /data/mssql/sql_exporter -config.data-source-name=sqlserver://sql_exporter:qICJEasdqwDiOSrdT96@10.112.5.105:1433/?encrypt=disable -config.file=/data/mssql/sql_exporter.yml -web.listen-address=127.0.0.1:9401 -log_dir=/data/mssql_log &

【4.3】prometheus配置

因為是自定義，所以在job name中特地加了 mssql 關鍵字，以便【4.4】中的變量好獲取到所有mssql 的job，以供篩選

- job_name: '大連娛網_mssql'
    static_configs:
    - targets: ['127.0.0.1:9400']
      labels:
        name: '我是第一台機器DB[10.112.5.106]' 
　　- targets: ['127.0.0.1:9401'] 
　　　　labels: 
　　　　name: '我是第二台機器DB[10.112.5.10]'

這里的配置，要和【4.2】的相互對應，否則會采集出問題。

這里的name 為什么會寫上IP呢，這是因為為了辨識是哪台機器，也是為了【4.4】中的大盤 儀表盤上可以顯示出機器IP來，因為我們這個是自定義的，無法像官網提供的一樣。其實我們這個有點類似於pushgateway

【4.4】自定義儀表盤

 

 核心變量

　　

 

 騷氣界面預覽

　　

 

【4.5】報警規則

groups:
- name: MSSQL告警規則
  rules:

  - alert: mssql引擎服務宕機
    expr:  windows_service_state{state="running",exported_name="mssqlserver"}!=1
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"

   
  - alert: mssql代理服務宕機
    expr:  windows_service_state{exported_name="sqlserveragent",state="running"}!=1                                                     
    for: 1m         
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"

  - alert: mssql引擎服務重啟
    expr: mssql_db_uptime < 3600
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "mssql引擎服務1小時內有過重啟，現已重啟{{ $value }} 秒"

  - alert: mssql數據庫不可用/不可訪問
    expr: mssql_current_state_dbState !=0
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "db:{{ $labels.db }}\n value:{{ $labels.value }}={{ $value }} "

  - alert: mssql阻塞
    expr: sum(mssql_current_state_blocking)>5
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "mssql請求阻塞數>5，當前：{{ $value }} "

  - alert: mssql請求過多
    expr: sum(mssql_current_state_requests)>100
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "mssql請求數>100，當前：{{ $value }} "


  - alert: mssql死鎖產生
    expr: increase(mssql_counter{type_object="SQLServer:Locks",type_counter="Number of Deadlocks/sec",type_instance="_Total"}[5m])>0
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "mssql 5分鍾內死鎖產生次數：{{ $value }} "


  - alert: mssql作業執行錯誤
    expr: increase(mssql_job_state_today[5m])>0
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "mssql 今天作業運行錯誤次數：{{ $value }} "


  - alert: mssql鏡像狀態變化
    expr: increase(mssql_mirror_sync{value="status"} [5m])!=0
    for: 1m
    labels:
      severity: warning
    annotations:
      summary: "詳細: {{ $labels }}"
      description: "db:{{ $labels.db }}\n value:{{ $labels.value }}={{ $value }} "
  

【4.6】報警模板

email

{{ define "email.html" }}
{{- if gt (len .Alerts.Firing) 0 -}}{{ range.Alerts }}
告警項: {{ .Labels.alertname }} <br>
項目組:{{ .Labels.job }} <br>
實例名:{{ .Labels.name }}-{{ .Labels.instance }}  <br>
詳情:  {{ .Annotations.description }} <br>
級別:  {{ .Labels.severity }}  <br>
開始時間:  {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
++++++++++++++++++++++++++++++++++++<br>
+++++++++++++++++++++++++++++++++++++<br>
{{ end }}{{ end -}}
{{- if gt (len .Alerts.Resolved) 0 -}}{{ range.Alerts }}
Resolved<br>
告警項: {{ .Labels.alertname }} <br>
項目組:{{ .Labels.job }} <br>
實例名:{{ .Labels.name }}-{{ .Labels.instance }}  <br>
詳情:  {{ .Annotations.description }} <br>
級別:  {{ .Labels.severity }}  <br>
開始時間:  {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
恢復時間:  {{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}<br>
++++++++++++++++++++++++++++++++++++<br>
+++++++++++++++++++++++++++++++++++++<br>
{{ end }}{{ end -}}
{{- end }}

企業微信

{{ define "wechat.default.message" }}
{{- if gt (len .Alerts.Firing) 0 -}}{{ range.Alerts }}
告警項: {{ .Labels.alertname }}
項目組:{{ .Labels.job }}
實例名:{{ .Labels.name }}-{{ .Labels.instance }}
詳情:  {{ .Annotations.description }}
級別:  {{ .Labels.severity }}
開始時間:  {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
------------------------------------
------------------------------------
{{ end }}{{ end -}}
{{- if gt (len .Alerts.Resolved) 0 -}}{{ range.Alerts }}
Resolved
告警項: {{ .Labels.alertname }}
項目組:{{ .Labels.job }}
實例名:{{ .Labels.name }}-{{ .Labels.instance }}
詳情:  {{ .Annotations.description }}
級別:  {{ .Labels.severity }}
開始時間:  {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
恢復時間:  {{ (.EndsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}
------------------------------------
------------------------------------
{{ end }}{{ end -}}
{{- end }}

 

【最佳實踐】安裝采集器、配置sql server權限

【1】 windows采集器文件

【1.1】上傳文件 windows_exporter 采集器文件

【1.2】雙擊運行

可能有東西出來，也可能一閃而過，也可能一點反應都沒有。

【1.3】核驗

Win+R 運行  =》Services.msc

查看服務， windows_exporter，這就表示部署好了。

【2】mssql 訪問配置

【2.1】防火牆配置/MSSQL權限配置（打開cmd，在dos命令窗口下執行）

netsh advfirewall firewall add rule name="prometheus_monitor" dir=in action=allow remoteip="192.168.1.1,192.168.1.2" protocol=TCP localport="1433,9182"
 
net stop wuauserv
sc config wuauserv start= disabled
sc config TrustedInstaller start= disabled
sc config windows_exporter start= delayed-auto
sc config MSSQLSERVER start= delayed-auto
sc config SQLSERVERAGENT start= delayed-auto


sqlcmd -E
USE [master]
GO

CREATE LOGIN [sql_exporter] WITH PASSWORD=N'qwer1234qwer', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF
GO

ALTER SERVER ROLE [sysadmin] ADD MEMBER [sql_exporter]
GO

 

-------------下面的不要運行，只是參考！--------

 

參考：

DECLARE @sql VARCHAR(max)
SET @sql=CAST('use master;CREATE LOGIN [sql_exporter] WITH PASSWORD=N''qwer1234qwer'', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF; GRANT VIEW SERVER STATE TO [sql_exporter];
GRANT VIEW ANY DEFINITION TO [sql_exporter];' AS VARCHAR(max))
select @sql=@sql+CAST('use '+name+';CREATE USER [sql_exporter] FOR LOGIN [sql_exporter];

EXEC sp_addrolemember N''db_datareader'', N''sql_exporter'';'+CHAR(10) AS VARCHAR(max))
from master.sys.databases where state=0 and is_read_only=0
EXEC(@sql)

Go

ALTER SERVER ROLE [sysadmin] ADD MEMBER [sql_exporter]

GO

參考：

Declare @login varchar(200),@role varchar(200), @login_pwd varchar(200)

Set @login='business_query'
Set @login_pwd='qwer1234qwer'
SET @role='db_datareader'

DECLARE @sql VARCHAR(max)

SET @sql=CAST('use master;CREATE LOGIN '+@login+' WITH PASSWORD=N'''+@login_pwd +''', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF; GRANT VIEW SERVER STATE TO '+@login+';
GRANT VIEW ANY DEFINITION TO '+@login+';' AS VARCHAR(max))
select @sql=@sql+CAST('use '+name+';CREATE USER '+@login+' FOR LOGIN '+@login+';
EXEC sp_addrolemember N'''+@role+''', N''business_query'';'+CHAR(10) AS VARCHAR(max))
from master.sys.databases where state=0 and is_read_only=0
EXEC(@sql)

Go

 

【3】采集器服務器配置參考

進入 115.230.30.138——10.20.53.12

cd /data/mssql/

修改 mssql_agent.sh

也要修改prometheus的配置文件；參考：

nohup /data/mssql/sql_exporter -config.data-source-name=sqlserver://sql_exporter:qwer1234qwer@10.20.54.59:1433/?encrypt=disable \
-config.file=/data/mssql/sql_exporter.yml -web.listen-address=127.0.0.1:9402  -log_dir=/data/mssql_log &

 

【參考文檔】

參考：https://www.bilibili.com/read/cv7134580/

官網：https://github.com/free/sql_exporter

 【附錄】

由於在url中特殊符號都有特殊意義或者被認為是不安全的字符，所以在拼接url時應當替換出url中的特殊字符

例如 

var x = "2# 前緣肋"

var rp= x.replace('#','%23'); // %23是#的URL編碼 要用他來替換原有的#

 

w3schools網站上列出了此類編碼參考

 

Character	From Windows-1252	From UTF-8
space	%20	%20
!	%21	%21
"	%22	%22
#	%23	%23
$	%24	%24
%	%25	%25
&	%26	%26
'	%27	%27
(	%28	%28
)	%29	%29
*	%2A	%2A
+	%2B	%2B
,	%2C	%2C
-	%2D	%2D
.	%2E	%2E
/	%2F	%2F
0	%30	%30
1	%31	%31
2	%32	%32
3	%33	%33
4	%34	%34
5	%35	%35
6	%36	%36
7	%37	%37
8	%38	%38
9	%39	%39
:	%3A	%3A
;	%3B	%3B
<	%3C	%3C
=	%3D	%3D
>	%3E	%3E
?	%3F	%3F
@	%40	%40
A	%41	%41
B	%42	%42
C	%43	%43
D	%44	%44
E	%45	%45
F	%46	%46
G	%47	%47
H	%48	%48
I	%49	%49
J	%4A	%4A
K	%4B	%4B
L	%4C	%4C
M	%4D	%4D
N	%4E	%4E
O	%4F	%4F
P	%50	%50
Q	%51	%51
R	%52	%52
S	%53	%53
T	%54	%54
U	%55	%55
V	%56	%56
W	%57	%57
X	%58	%58
Y	%59	%59
Z	%5A	%5A
[	%5B	%5B
\	%5C	%5C
]	%5D	%5D
^	%5E	%5E
_	%5F	%5F
`	%60	%60
a	%61	%61
b	%62	%62
c	%63	%63
d	%64	%64
e	%65	%65
f	%66	%66
g	%67	%67
h	%68	%68
i	%69	%69
j	%6A	%6A
k	%6B	%6B
l	%6C	%6C
m	%6D	%6D
n	%6E	%6E
o	%6F	%6F
p	%70	%70
q	%71	%71
r	%72	%72
s	%73	%73
t	%74	%74
u	%75	%75
v	%76	%76
w	%77	%77
x	%78	%78
y	%79	%79
z	%7A	%7A
{	%7B	%7B
|	%7C	%7C
}	%7D	%7D
~	%7E	%7E
	%7F	%7F
`	%80	%E2%82%AC
	%81	%81
‚	%82	%E2%80%9A
ƒ	%83	%C6%92
„	%84	%E2%80%9E
…	%85	%E2%80%A6
†	%86	%E2%80%A0
‡	%87	%E2%80%A1
ˆ	%88	%CB%86
‰	%89	%E2%80%B0
Š	%8A	%C5%A0
‹	%8B	%E2%80%B9
Œ	%8C	%C5%92
	%8D	%C5%8D
Ž	%8E	%C5%BD
	%8F	%8F
	%90	%C2%90
‘	%91	%E2%80%98
’	%92	%E2%80%99
“	%93	%E2%80%9C
”	%94	%E2%80%9D
•	%95	%E2%80%A2
–	%96	%E2%80%93
—	%97	%E2%80%94
˜	%98	%CB%9C
™	%99	%E2%84
š	%9A	%C5%A1
›	%9B	%E2%80
œ	%9C	%C5%93
	%9D	%9D
ž	%9E	%C5%BE
Ÿ	%9F	%C5%B8
	%A0	%C2%A0
¡	%A1	%C2%A1
¢	%A2	%C2%A2
£	%A3	%C2%A3
¤	%A4	%C2%A4
¥	%A5	%C2%A5
¦	%A6	%C2%A6
§	%A7	%C2%A7
¨	%A8	%C2%A8
©	%A9	%C2%A9
ª	%AA	%C2%AA
«	%AB	%C2%AB
¬	%AC	%C2%AC
	%AD	%C2%AD
®	%AE	%C2%AE
¯	%AF	%C2%AF
°	%B0	%C2%B0
±	%B1	%C2%B1
²	%B2	%C2%B2
³	%B3	%C2%B3
´	%B4	%C2%B4
µ	%B5	%C2%B5
¶	%B6	%C2%B6
·	%B7	%C2%B7
¸	%B8	%C2%B8
¹	%B9	%C2%B9
º	%BA	%C2%BA
»	%BB	%C2%BB
¼	%BC	%C2%BC
½	%BD	%C2%BD
¾	%BE	%C2%BE
¿	%BF	%C2%BF
À	%C0	%C3%80
Á	%C1	%C3%81
Â	%C2	%C3%82
Ã	%C3	%C3%83
Ä	%C4	%C3%84
Å	%C5	%C3%85
Æ	%C6	%C3%86
Ç	%C7	%C3%87
È	%C8	%C3%88
É	%C9	%C3%89
Ê	%CA	%C3%8A
Ë	%CB	%C3%8B
Ì	%CC	%C3%8C
Í	%CD	%C3%8D
Î	%CE	%C3%8E
Ï	%CF	%C3%8F
Ð	%D0	%C3%90
Ñ	%D1	%C3%91
Ò	%D2	%C3%92
Ó	%D3	%C3%93
Ô	%D4	%C3%94
Õ	%D5	%C3%95
Ö	%D6	%C3%96
×	%D7	%C3%97
Ø	%D8	%C3%98
Ù	%D9	%C3%99
Ú	%DA	%C3%9A
Û	%DB	%C3%9B
Ü	%DC	%C3%9C
Ý	%DD	%C3%9D
Þ	%DE	%C3%9E
ß	%DF	%C3%9F
à	%E0	%C3%A0
á	%E1	%C3%A1
â	%E2	%C3%A2
ã	%E3	%C3%A3
ä	%E4	%C3%A4
å	%E5	%C3%A5
æ	%E6	%C3%A6
ç	%E7	%C3%A7
è	%E8	%C3%A8
é	%E9	%C3%A9
ê	%EA	%C3%AA
ë	%EB	%C3%AB
ì	%EC	%C3%AC
í	%ED	%C3%AD
î	%EE	%C3%AE
ï	%EF	%C3%AF
ð	%F0	%C3%B0
ñ	%F1	%C3%B1
ò	%F2	%C3%B2
ó	%F3	%C3%B3
ô	%F4	%C3%B4
õ	%F5	%C3%B5
ö	%F6	%C3%B6
÷	%F7	%C3%B7
ø	%F8	%C3%B8
ù	%F9	%C3%B9
ú	%FA	%C3%BA
û	%FB	%C3%BB
ü	%FC	%C3%BC
ý	%FD	%C3%BD
þ	%FE	%C3%BE
ÿ	%FF	%C3%BF

 

---

URL Encoding Reference

 

The ASCII control characters %00-%1F were originally designed to control hardware devices.

Control characters have nothing to do inside a URL.

ASCII Character	Description	URL-encoding
NUL	null character	%00
SOH	start of header	%01
STX	start of text	%02
ETX	end of text	%03
EOT	end of transmission	%04
ENQ	enquiry	%05
ACK	acknowledge	%06
BEL	bell (ring)	%07
BS	backspace	%08
HT	horizontal tab	%09
LF	line feed	%0A
VT	vertical tab	%0B
FF	form feed	%0C
CR	carriage return	%0D
SO	shift out	%0E
SI	shift in	%0F
DLE	data link escape	%10
DC1	device control 1	%11
DC2	device control 2	%12
DC3	device control 3	%13
DC4	device control 4	%14
NAK	negative acknowledge	%15
SYN	synchronize	%16
ETB	end transmission block	%17
CAN	cancel	%18
EM	end of medium	%19
SUB	substitute	%1A
ESC	escape	%1B
FS	file separator	%1C
GS	group separator	%1D
RS	record separator	%1E
US	unit separator