轉載自博客:
https://my.oschina.net/yyqz/blog/5085650
1.背景
按照Skywalking官網的客戶端搭建方式,基本采取配置agent.properties文件,或者通過java -D 帶參數方式(也可以直接使用環境變量進行配置),這些操作辦法都屬於靜態配置。如果在業務高峰期,可能需要調整采樣率 agent.sample_n_per_3_secs 的數值,只能通過重新啟動agent方式更新配置信息。
那么如何才能做到通過后台,動態控制agent端的采樣率、鏈路跨度等配置信息呢?正巧,Skywalking提供了動態更新功能,
1.1官網文檔索引
agent支持動態代理:https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/readme/#dynamic-configurations 詳細說明: https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/configuration-discovery/
客戶端允許動態更新的參數:
| Key | 參數描述 | 參數示例 | 依賴插件 |
|---|---|---|---|
| agent.sample_n_per_3_secs | 每3秒采集鏈路數據,-1表示全部收集 |
-1 | - |
| agent.ignore_suffix | 不記錄鏈路信息url后綴(第一個節點開始算起),如果多個,采用逗號,隔開 |
.txt,.log | - |
| agent.trace.ignore_path | 不記錄調用鏈的路徑,多個可以采用逗號,隔開 詳細 |
/your/path/1/**,/your/path/2/** |
apm-trace-ignore-plugin |
| agent.span_limit_per_segment | 鏈路最大跨度 | 300 |
- |
也就是說,skywalking提供能動態修改上面配置的方案。那么具體要怎么才能實現呢,需要agent和Skywalking Server端共同配置才能實現。 服務端配置官網文檔:https://skywalking.apache.org/docs/main/v8.6.0/en/setup/backend/dynamic-config/
SkyWalking Configurations mostly are set through application.yml and OS system environment variables. At the same time, some of them are supporting dynamic settings from upstream management system.
Right now, SkyWalking supports following dynamic configurations.
... 省略具體參數,詳細見上面官網鏈接
| Config Key | Value Description | Value Format Example |
|---|---|---|
| configuration-discovery.default.agentConfigurations | The ConfigurationDiscovery settings,agent端動態配置key | look at configuration-discovery.md |
其中第一列是表示key值,后面是描述及示例;
官網還有提到,服務端動態參數配置,可以支持多種模式,大致列舉如下:
grpc表示可以直接通過調用grpc接口修改配置zookeeper表示可以直接集成zookeeper實現配置刷新etcd表示可以通過etcd配置中心的方式實現動態更新配置Consul表示可以通過監聽Consul配置進行動態更新Apollo表示可以集成Apollo配置中心實現動態更新Nacos表示可以集成Nacos配置中心實現動態更新configmap表示在雲原生k8s容器內部,可以通過自帶配置管理功能監聽來實現動態更新
默認此功能是關閉的,需要在config/application.yml文件中,修改如下配置,決定使用哪一個作為監聽來源。注意:這里只能同時選擇一種,防止出現數據不一致導致服務混亂問題。
configuration: selector: ${SW_CONFIGURATION:none}
當然上述配置均可以直接采用服務端修改環境變量方式實現,減少對文件的個性化修改,本文也是采用環境變量模式。
綜上所屬,根據查看源碼流程所得,agent端在啟動的時候,會開啟配置變更監聽,直接連接server端接口;而server端在啟動的時候,會去監聽所選擇的配置來源,例如本文接下來介紹的使用zookeeper為例,配置修改變化下發流程如下:
zookeeper節點數據變化 --> server監聽到變化 --> agent監聽到變化 --> 修改本地相關參數值
1.2 zk節點路徑
下面是application.yml中zk配置的采樣率作為入口,尋找zk節點路徑規則:
zookeeper: period: ${SW_CONFIG_ZK_PERIOD:60} # Unit seconds, sync period. Default fetch every 60 seconds. nameSpace: ${SW_CONFIG_ZK_NAMESPACE:/default} hostPort: ${SW_CONFIG_ZK_HOST_PORT:localhost:2181} # Retry Policy baseSleepTimeMs: ${SW_CONFIG_ZK_BASE_SLEEP_TIME_MS:1000} # initial amount of time to wait between retries maxRetries: ${SW_CONFIG_ZK_MAX_RETRIES:3} # max number of times to retry
對應了服務端源碼:org.apache.skywalking.oap.server.configuration.zookeeper.ZookeeperServerSettings
private String nameSpace = "/default"; private String hostPort; private int baseSleepTimeMs = 1000; private int maxRetries = 3; private int period = 60;
其中nameSpace可以自行修改。 服務端啟動監聽類org.apache.skywalking.oap.server.configuration.zookeeper.ZookeeperConfigWatcherRegister
public class ZookeeperConfigWatcherRegister extends ConfigWatcherRegister { private final PathChildrenCache childrenCache; private final String prefix; public ZookeeperConfigWatcherRegister(ZookeeperServerSettings settings) throws Exception { super(settings.getPeriod()); prefix = settings.getNameSpace() + "/"; RetryPolicy retryPolicy = new ExponentialBackoffRetry(settings.getBaseSleepTimeMs(), settings.getMaxRetries()); CuratorFramework client = CuratorFrameworkFactory.newClient(settings.getHostPort(), retryPolicy); client.start(); this.childrenCache = new PathChildrenCache(client, settings.getNameSpace(), true); this.childrenCache.start(); } @Override public Optional<ConfigTable> readConfig(Set<String> keys) { ConfigTable table = new ConfigTable(); keys.forEach(s -> { ChildData data = this.childrenCache.getCurrentData(this.prefix + s); table.add(new ConfigTable.ConfigItem(s, data == null ? null : new String(data.getData()))); }); return Optional.of(table); } }
prefix = settings.getNameSpace() + "/";:表示URL前綴是在上面application.yml中配置的nameSpace,本文環境變量配置示例export SW_CONFIG_ZK_NAMESPACE=/skywalking/gongstring/agent
ChildData data = this.childrenCache.getCurrentData(this.prefix + s);:表示監聽的ZK節點路徑,需要將上面的URL前綴與當前節點key合並,例如agent相關的配置,服務端配置的keyconfiguration-discovery.default.agentConfigurations,所以使用zookeeper做配置下發的時候,需要在zk中找到如下節點(服務啟動會自動創建,如果沒有手工創建):/skywalking/gongstring/agent/configuration-discovery.default.agentConfigurations
1.3節點數據
下面是服務啟動監聽node節點源碼: org.apache.skywalking.apm.agent.core.conf.dynamic.ConfigurationDiscoveryService
/** * Register dynamic configuration watcher. * * @param watcher dynamic configuration watcher */ public void registerAgentConfigChangeWatcher(AgentConfigChangeWatcher watcher) { WatcherHolder holder = new WatcherHolder(watcher); if (register.containsKey(holder.getKey())) { throw new IllegalStateException("Duplicate register, watcher=" + watcher); } register.put(holder.getKey(), holder); } /** * Process ConfigurationDiscoveryCommand and notify each configuration watcher. * * @param configurationDiscoveryCommand Describe dynamic configuration information */ public void handleConfigurationDiscoveryCommand(ConfigurationDiscoveryCommand configurationDiscoveryCommand) { final String responseUuid = configurationDiscoveryCommand.getUuid(); if (responseUuid != null && Objects.equals(this.uuid, responseUuid)) { return; } List<KeyStringValuePair> config = readConfig(configurationDiscoveryCommand); config.forEach(property -> { String propertyKey = property.getKey(); WatcherHolder holder = register.get(propertyKey); if (holder != null) { AgentConfigChangeWatcher watcher = holder.getWatcher(); String newPropertyValue = property.getValue(); if (StringUtil.isBlank(newPropertyValue)) { if (watcher.value() != null) { // Notify watcher, the new value is null with delete event type. watcher.notify( new AgentConfigChangeWatcher.ConfigChangeEvent( null, AgentConfigChangeWatcher.EventType.DELETE )); } else { // Don't need to notify, stay in null. } } else { if (!newPropertyValue.equals(watcher.value())) { watcher.notify(new AgentConfigChangeWatcher.ConfigChangeEvent( newPropertyValue, AgentConfigChangeWatcher.EventType.MODIFY )); } else { // Don't need to notify, stay in the same config value. } } } else { LOGGER.warn("Config {} from OAP, doesn't match any watcher, ignore.", propertyKey); } }); this.uuid = responseUuid; LOGGER.trace("Current configurations after the sync, configurations:{}", register.toString()); }
該方法是將讀取參數添加到待監聽列表中,在agent端的,org.apache.skywalking.apm.agent.core.sampling.SamplingService源碼中,有關於監聽采樣率初始化的key值,關鍵代碼:
@Override public void boot() { samplingRateWatcher = new SamplingRateWatcher("agent.sample_n_per_3_secs", this); ServiceManager.INSTANCE.findService(ConfigurationDiscoveryService.class) .registerAgentConfigChangeWatcher(samplingRateWatcher); handleSamplingRateChanged(); }
到此處,說明,需要從服務端讀取並監聽agent.sample_n_per_3_secs數據變化,同時參考官網說明:https://skywalking.apache.org/docs/main/v8.6.0/en/setup/service-agent/java-agent/configuration-discovery/
configurations: //service name serviceA: // Configurations of service A // Key and Value are determined by the agent side. // Check the agent setup doc for all available configurations. key1: value1 key2: value2 ... serviceB: ...
表示如果要實現動態修改功能,只需要修改zk節點的數據值為上面這個格式即可。
下面是我的配置示例。
2.示例配置
2.1環境變量
export SW_STORAGE=elasticsearch export SW_STORAGE_ES_CLUSTER_NODES=192.168.0.4:9200 export SW_KAFKA_FETCHER=default export SW_KAFKA_FETCHER_SERVERS=192.168.0.4:9093,192.168.0.4:9094,192.168.0.4:9095 export SW_NAMESPACE=gongstring export SW_CONFIGURATION=zookeeper export SW_CONFIG_ZK_NAMESPACE=/skywalking/gongstring/agent export SW_CONFIG_ZK_HOST_PORT=192.168.0.4:2181 export SW_CONFIG_ZK_PERIOD=3
2.2 zookeeper節點數據配置
zk節點路徑:/skywalking/gongstring/agent/configuration-discovery.default.agentConfigurations
configurations: dubbo.provider: agent.sample_n_per_3_secs: 5 dubbo.consumer: agent.sample_n_per_3_secs: 2
2.3 agent配置
provider:
-javaagent:D:/dev/tools/skywalking/agent/skywalking-agent.jar -DSW_AGENT_NAMESPACE=gongstring -DSW_AGENT_NAME=dubbo.provider -DSW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.0.4:11800 -Dplugin.dubbo.collect_provider_arguments=true -Dplugin.dubbo.collect_consumer_arguments=true
consumer:
-javaagent:D:/dev/tools/skywalking/agent/skywalking-agent.jar -DSW_AGENT_NAMESPACE=gongstring -DSW_AGENT_NAME=dubbo.consumer -DSW_AGENT_COLLECTOR_BACKEND_SERVICES=192.168.0.4:11800 -Dplugin.dubbo.collect_provider_arguments=true -Dplugin.dubbo.collect_consumer_arguments=true
