【生活工具】你能帶我回家么，可能明天要下雨了。

前言

你是否也這樣？每天加班完后只想回家躺着，經常忘記帶傘回家。如果第二天早上有雨，往往就會成為落湯雞，特別是筆者所在的深圳，更是喜歡下雨，稍不注意，就成落湯雞。其實想想，這種情況也是可以有效避免的，只需要晚上帶傘回家，然后第二天早上帶出來，最后美滋滋的吃早餐。但前提是晚上帶傘回家，你知道的，做IT的都在忙着改變世界，帶傘這種小事當然不值一提，華麗忘記。這時候默想，要是有人每天晚上提醒我帶傘回家就好了，這種想法似乎有些奢侈。既然別人做不到，那就讓程序來做吧。

思路

本項目其實就是個天氣提醒器，用來提醒我們廣大IT同仁們明天天氣，思路大致分為如下幾步。

• 從網上爬取深圳明天天氣情況並解析。
• 解析天氣信息后發送郵件提醒。
• 將項目打包后上傳至服務器。
• 編寫Linux的定時任務，定時運行啟動腳本。

整體框架

整體框架包括Linux的定時任務部分和weather-service中處理部分，系統會在每天啟動定時任務(自動運行指定腳本)，啟動腳本會啟動weather-service服務完成天氣信息的爬取和郵件提醒。

技術點

整個項目涉及的技術點如下。

• Crontab，定時任務命令。
• Shell腳本，啟動腳本編寫。
• weather-service涉及技術如下
  • Maven，項目使用Maven構建。
  • HttpClient，爬取網頁信息。
  • JSoup，解析網頁信息。
  • JavaMail，發送郵件。
  • log4j、slf4j，日志輸出。

源碼

weather-service的核心模塊為爬取模塊和郵件模塊；而完成自動化執行動作則需要編寫Crontab定時任務和Shell腳本，定時任務定時啟動Shell腳本。

爬取模塊

主要完成從互聯網上爬取天氣信息並進行解析。

• WeatherCrawler

package com.hust.grid.weather;

import org.apache.http.HttpEntity;
import org.apache.http.HttpStatus;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.hust.grid.bean.WeatherInfo;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class WeatherCrawler {
    private Logger logger = LoggerFactory.getLogger(WeatherCrawler.class);

    public WeatherInfo crawlWeather(String url) {
        CloseableHttpClient client = null;
        HttpGet get;
        WeatherInfo weatherInfo = null;
        try {
            client = HttpClients.custom().setRetryHandler(DefaultHttpRequestRetryHandler.INSTANCE).build();

            RequestConfig config = RequestConfig
                    .custom()
                    .setConnectionRequestTimeout(3000)
                    .setConnectTimeout(3000)
                    .setSocketTimeout(30 * 60 * 1000)
                    .build();

            get = new HttpGet(url);
            get.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
            get.setHeader("Accept-Encoding", "gzip, deflate");
            get.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
            get.setHeader("Host", "www.weather.com.cn");
            get.setHeader("Proxy-Connection", "keep-alive");
            get.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");

            get.setConfig(config);
            CloseableHttpResponse response = client.execute(get);
            int statusCode = response.getStatusLine().getStatusCode();
            if (statusCode == HttpStatus.SC_OK) {
                HttpEntity entity = response.getEntity();
                String content = EntityUtils.toString(entity, "utf8");
                logger.debug("content =====>" + content);
                if (content != null)
                    weatherInfo = parseResult(content);
            }
        } catch (Exception e) {
            logger.error(e.getMessage());
        } finally {
            if (client != null) {
                try {
                    client.close();
                } catch (Exception e) {
                   logger.error("close client error " + e.getMessage());
                }
            }
        }


        return weatherInfo;
    }

    public WeatherInfo parseResult(String content) {
        Document document = Jsoup.parse(content);
        Element element = document.getElementById("7d");
        Elements elements = element.getElementsByTag("ul");
        Element clearFix = elements.get(0);
        Elements lis = clearFix.getElementsByTag("li");
        // 7 days weather info, we just take tomorrow weather info
        Element tomorrow = lis.get(1);
        logger.debug("tomorrow =====> " + tomorrow);
        return parseWeatherInfo(tomorrow);

    }

    private WeatherInfo parseWeatherInfo(Element element) {
        Elements weathers = element.getElementsByTag("p");
        String weather = weathers.get(0).text();
        String temp = weathers.get(1).text();
        String wind = weathers.get(2).text();

        WeatherInfo weatherInfo = new WeatherInfo(weather, temp, wind);

        logger.info("---------------------------------------------------------------------------------");
        logger.info("---------------------------------------------------------------------------------");
        logger.info("weather is " + weather);
        logger.info("temp is " + temp);
        logger.info("wind is " + wind);
        logger.info("---------------------------------------------------------------------------------");
        logger.info("---------------------------------------------------------------------------------");

        return weatherInfo;
    }

    public static void main(String[] args) {
        WeatherCrawler crawlWeatherInfo = new WeatherCrawler();
        crawlWeatherInfo.crawlWeather("http://www.weather.com.cn/weather/101280601.shtml");
    }
}

可以看到爬取模塊首先使用HttpClient從指定網頁獲取信息，然后對響應結果使用JSoup進行解析，並且只解析了明天的天氣信息，最后將解析后的天氣信息封裝成WeatherInfo對象返回。

郵件模塊

主要完成將解析的信息以郵件發送給指定收件人。

• MailSender

package com.hust.grid.email;

import com.hust.grid.cache.ConstantCacheCenter;
import com.hust.grid.bean.WeatherInfo;
import com.sun.mail.util.MailSSLSocketFactory;

import javax.mail.*;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeMessage;
import java.io.UnsupportedEncodingException;
import java.security.GeneralSecurityException;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.Properties;

public class MailSender {
    private WeatherInfo weatherInfo;
    private static Properties prop = new Properties();
    private ConstantCacheCenter constantCacheCenter = ConstantCacheCenter.getInstance();

    private static class MyAuthenticator extends Authenticator {
        private String username;
        private String password;

        public MyAuthenticator(String username, String password) {
            this.username = username;
            this.password = password;
        }

        @Override
        protected PasswordAuthentication getPasswordAuthentication() {
            return new PasswordAuthentication(username, password);
        }
    }

    public MailSender(WeatherInfo weatherInfo) {
        this.weatherInfo = weatherInfo;
    }

    public void sendToAll() {
        List<String> receivers = constantCacheCenter.getReceivers();
        for (String receiver : receivers) {
            send(receiver);
        }
    }

    private void send(String receiver) {
        prop.setProperty("mail.transport.protocol", constantCacheCenter.getProtocol());
        prop.setProperty("mail.smtp.host", constantCacheCenter.getHost());
        prop.setProperty("mail.smtp.port", constantCacheCenter.getPort());
        prop.setProperty("mail.smtp.auth", "true");
        MailSSLSocketFactory mailSSLSocketFactory = null;
        try {
            mailSSLSocketFactory = new MailSSLSocketFactory();
            mailSSLSocketFactory.setTrustAllHosts(true);
        } catch (GeneralSecurityException e1) {
            e1.printStackTrace();
        }
        prop.put("mail.smtp.ssl.enable", "true");
        prop.put("mail.smtp.ssl.socketFactory", mailSSLSocketFactory);
        //
        Session session = Session.getDefaultInstance(prop, new MyAuthenticator(constantCacheCenter.getUsername(), constantCacheCenter.getAuthorizationCode()));
        session.setDebug(true);
        MimeMessage mimeMessage = new MimeMessage(session);
        try {
            mimeMessage.setFrom(new InternetAddress(constantCacheCenter.getSenderEmail(), constantCacheCenter.getSenderName()));
            mimeMessage.addRecipient(Message.RecipientType.TO, new InternetAddress(receiver));
            mimeMessage.setSubject("明日天氣");
            mimeMessage.setSentDate(new Date());

            mimeMessage.setText("Hi, Dear: \n\n     明天天氣狀態如下：" + weatherInfo.toString());
            mimeMessage.saveChanges();
            Transport.send(mimeMessage);
        } catch (MessagingException e) {
            e.printStackTrace();
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
    }


    public static void main(String[] args) {
        WeatherInfo weatherInfo = new WeatherInfo("晴朗", "27/33", "3級");
        List<String> receivers = new ArrayList<String>();
        receivers.add("[e-mail]");
        MailSender s = new MailSender(weatherInfo);
        s.sendToAll();
    }
}

可以看到郵件發送模塊需要進行一系列的設置，如端口號、認證、服務、發送人和收信人等信息。本發送郵件模塊使用QQ郵箱進行發送，需要在QQ郵箱的設置中獲取對應的授權碼。

微信提醒模塊

經過讀者提醒，可以使用微信進行提醒，現在使用微信的頻率太高了，在網上找到Server醬做微信提醒接口，接口非常簡單，源碼如下

package com.hust.grid.weixin;

import com.hust.grid.bean.WeatherInfo;
import com.hust.grid.cache.ConstantCacheCenter;
import org.apache.http.Consts;
import org.apache.http.HttpHost;
import org.apache.http.HttpStatus;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.util.ArrayList;
import java.util.List;

public class WeiXinSender {
    private Logger logger = LoggerFactory.getLogger(WeiXinSender.class);

    private static final String PREFIX = "https://sc.ftqq.com/";
    private ConstantCacheCenter constantCacheCenter = ConstantCacheCenter.getInstance();

    private WeatherInfo weatherInfo;

    public WeiXinSender(WeatherInfo weatherInfo) {
        this.weatherInfo = weatherInfo;
    }

    public void sendToAll() {
        List<String> receiverKeys = constantCacheCenter.getWeixinReceiverKeysList();
        logger.info("receiverKeys " + receiverKeys);
        for (String key : receiverKeys) {
            send(key);
        }
    }

    private void send(String key) {
        CloseableHttpClient client = null;
        HttpPost post;
        StringBuffer stringBuffer = new StringBuffer();
        stringBuffer.append(PREFIX);
        stringBuffer.append(key);
        stringBuffer.append(".send");

        try {
            client = HttpClients.custom().setRetryHandler(DefaultHttpRequestRetryHandler.INSTANCE).build();
            RequestConfig config = RequestConfig
                    .custom()
                    .setConnectionRequestTimeout(3000)
                    .setConnectTimeout(3000)
                    .setSocketTimeout(30 * 60 * 1000)
                    .build();


            String text = "明日天氣情況";
            String desp = weatherInfo.getWeixinFormatString();

            List<BasicNameValuePair> postDatas = new ArrayList<>();
            postDatas.add(new BasicNameValuePair("text", text));
            postDatas.add(new BasicNameValuePair("desp", desp));

            logger.info("url is " + stringBuffer.toString());
            post = new HttpPost(stringBuffer.toString());
            post.setConfig(config);
            post.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8");
            post.setHeader("Accept-Encoding", "gzip, deflate, br");
            post.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
            post.setHeader("Cache-Control", "max-age=0");
            post.setHeader("Connection", "keep-alive");
            post.setHeader("Host", "sc.ftqq.com");
            post.setHeader("Upgrade-Insecure-Requests", "1");
            post.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36");

            UrlEncodedFormEntity formEntity = new UrlEncodedFormEntity(postDatas , Consts.UTF_8) ;
            post.setEntity(formEntity);
            CloseableHttpResponse response = client.execute(post);
            int statusCode = response.getStatusLine().getStatusCode();
            if (statusCode == HttpStatus.SC_OK) {
                logger.info("call the cgi success");
            }
        } catch (Exception e) {
            logger.error(e.getMessage());
        } finally {
            if (client != null) {
                try {
                    client.close();
                } catch (Exception e) {
                    logger.error("close client error " + e.getMessage());
                }
            }
        }
    }
}

可以看到只需要使用SCKey調用Server醬提供的接口，並關注該服務號便可完成微信提醒功能。

定時任務

本項目使用Linux中定時任務命令crontab完成定時任務的編寫，其命令如下。

53 19 * * * (. ~/.bashrc; cd /home/robbinli/weather-service/bin; ~/weather-service/bin/start.sh > /tmp/weather-service-monitor.log 2>&1)

該crontab命令表示在每天的19:53分執行對應的腳本，並將信息定向至指定文件中，關於crontab命令的編寫感興趣的朋友可自行上網查閱。值得注意的是執行的start.sh啟動腳本最好使用絕對路徑，可避免找不到腳本的問題。

啟動腳本

啟動腳本指定了如何啟動weather-service的jar包。

#!/bin/sh

# export jdk env
export JAVA_HOME=/home/robbinli/program/java/jdk1.8.0_45
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

# get root path
base_path=`cd "$(dirname "$0")"; cd ..; pwd`

if [ ! -d "../log" ]; then
  mkdir ../log
fi


# auto format the jars in classpath
lib_jars=`ls $base_path/lib/ | grep jar | awk -v apppath=$base_path 'BEGIN{jars="";}{jars=sprintf("%s:%s/lib/%s", jars, apppath, $1);} END{print jars}'`
conf_path="$base_path/conf"

main_class="com.hust.grid.entry.WeatherServiceMain ${conf_path}"
jar_file="weather-service.jar"

run_cmd="java -Dlog.home=${base_path}/log -Dlog4j.configuration=file:${base_path}/conf/log4j.properties -verbosegc -XX:+PrintGCDetails -cp ${conf_path}:${base_path}/bin/${jar_file}${lib_jars} ${main_class} "

echo "start command: $run_cmd"
echo "start..."
       $run_cmd > $base_path/log/jvm.log 2>&1 &

值得注意的是啟動腳本中設置了JDK環境，並創建了log目錄記錄日志文件，然后使用java運行weather-service jar包。的目錄結構，若不設置，同時，

問題記錄

在完成整個項目過程中也遇到了些問題，記錄如下。

定時任務無法啟動腳本

需要在start.sh中設置JDK環境，若不設置，會出現直接使用sh start.sh可啟動jar包，而使用crontab定時任務時無法啟動，因為crontab啟動時不會加載JDK環境變量，因此無法啟動，需要在啟動腳本中指定JDK環境。

運行啟動腳本異常

由於筆者是在Windows環境下編程，上傳start.sh腳本至Linux后，使用sh start.sh運行啟動腳本時出現異常，這是由於Windows和Linux的文件格式不相同，需要使用dos2unix start.sh命令將Windows格式轉化為Linux下格式。

運行效果

啟動腳本后，郵件截圖如下。

開源地址

由於所花時間有限，只完成了很基礎的功能，現將其開源，有興趣的讀者可自行修改源碼，添加想要的功能，如使用其他郵箱(google、163)發送、添加短信提醒、可直接在配置文件中配置不同地區(非URL)、未來七天的天氣；歡迎讀者Fork And Star。
鏈接如下：weather-service in github

總結

利用差不多1天時間，折騰了這個mini 項目，希望能夠發揮它的價值，也感謝各位讀者的閱讀。