接口中轉stream傳輸 request/response

 

php實現

CURLOPT_WRITEFUNCTION: for response ，把接口返回的結果拿回來，會進行多次回調，直到接口中的內容全部讀完
CURLOPT_READFUNCTION ：for request  ，把要請求接口的參數數據寫出去

CURLOPT_READFUNCTION    
    回調函數名。該函數應接受三個參數。第一個是 cURL resource；
    第二個是通過選項 CURLOPT_INFILE 傳給 cURL 的 stream resource；
    第三個參數是最大可以讀取的數據的數量。回 調函數必須返回一個字符串，長度小於或等於請求的數據量（第三個參數）。一般從傳入的 stream resource 讀取。返回空字符串作為 EOF（文件結束） 信號。
CURLOPT_WRITEFUNCTION    
    回調函數名。該函數應接受兩個參數。
    第一個是 cURL resource；
    第二個是要寫入的數據字符串。數 據必須在函數中被保存。 
    函數必須准確返回寫入數據的字節數，否則傳輸會被一個錯誤所中斷。

 

A bit more documentation (without minimum version numbers):

- CURLOPT_WRITEFUNCTION
- CURLOPT_HEADERFUNCTION

  Pass a function which will be called to write data or headers respectively. The callback function prototype: long write_callback (resource ch, string data)

The ch argument is CURL session handle. The data argument is data received. Note that its size is variable. When writing data, as much data as possible will be returned in all invokes. When writing headers, exactly one complete header line is returned for better parsing.
The function must return number of bytes actually taken care of. If that amount differs from the amount passed to this function, an error will occur.

- CURLOPT_READFUNCTION
  Pass a function which will be called to read data. The callback function prototype:

string read_callback (resource ch, resource fd, long length) 
The ch argument is CURL session handle. The fd argument is file descriptor passed to CURL by CURLOPT_INFILE option. The length argument is maximum length which can be returned.
The function must return string containing the data which were read. If length of the data is more than maximum length, it will be truncated to maximum length. Returning anything else than a string means an EOF.

[Note: there is more callbacks implemented in current cURL library but they aren't unfortunately implemented in php curl interface yet.]

 

　　文件下載

client.php

<?php

function receiveResponse($curlHandle, $xmldata)
{
    var_dump($xmldata);
    flush();
    ob_flush();
    //需要准確返回讀取的字節數
    return strlen($xmldata);
}

$ch = curl_init('http://jksong.cm/api.php?id=1&name=jksong');
curl_setopt($ch, CURLOPT_WRITEFUNCTION, "receiveResponse");
curl_exec($ch);
curl_close($ch);

 

api.php

<?php

for ($i=0;$i<10;$i++){

    var_dump(str_repeat("a", 20));
//    var_dump(time());

    sleep(1);

    ob_flush();
    flush();
}

 

JAVA httpclient 實現

CloseableHttpClient client = HttpClients.createDefault();
        HttpGet httpGet = new HttpGet("http://jksong.cm/api.php");
        try (CloseableHttpResponse response1 = client.execute(httpGet)) {
            final HttpEntity entity = response1.getEntity();
            if (entity != null) {
                try (InputStream inputStream = entity.getContent()) {
                    // do something useful with the stream
                    //System.out.println(IOUtils.toString(inputStream));

                    BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream);
                    
                    int b = 0;
                    while ((b = inputStream.read()) != -1) {
                        System.out.println("===>"+(char)b);
                    }

                }
            }
        }

 

 

 node

http

var http = require('http');
var body = '';

http.get("http://jksong.cm/api.php", function(res) {
    res.on('data', function(chunk) {
        body += chunk;
        console.log(chunk);
    });
    res.on('end', function() {
        // all data has been downloaded
    });
});

 

axios

var fs = require('fs');
var http = require('axios');
// 獲取遠端圖片
http({
    method: 'get',
    url: 'http://jksong.cm/api.php',
    responseType: 'stream'
})
    .then(function (response) {
        // console.log(response.data.read);
        response.data.pipe(fs.createWriteStream('ada_lovelace.txt'))
    });

 

　　文件上傳

PHP

php中form-data不支持 php://input 方式讀取，可以采用 binary或urlencode 方式 直接上傳文件

驗證：

api.php

//可以實時讀取最新上傳的數據
$handle = fopen("php://input", "rb");
$contents = '';
while (!feof($handle)) {
    $body = fread($handle, 8192);
    var_dump($body);
    file_put_contents("/tmp/a.txt", $body, FILE_APPEND);
}
fclose($handle);

 

curl請求

#注意發送速率，由於服務端緩沖區的存在，太小時不容易測出實時接收的效果
curl --limit-rate 100k -L -X GET 'http://jksong.cm/api.php' -H 'Content-Type: application/x-rar-compressed' --data-binary '@/tmp/file.data'
curl --limit-rate 100k -L -X GET 'http://jksong.cm/api.php' -H 'Content-Type: application/x-rar-compressed' --data-urlencode 'song@/tmp/file.data'

 

JAVA

request.getInputStream() 中的流可以實時獲取，這樣就可以邊上傳，邊進行轉發

如果獲取 inputstream 原因參看   接口中轉stream傳輸

 

由於multipart方式上傳不夠優雅

　　1、要不就是不可以實時獲取到流，必須要等到全上傳完畢 【MultipartFile 參數方式】

　　2、要不就是關閉框架中 multipart【MultipartResolver】 解析的功能 【spring.servlet.multipart.enabled=false】，這樣會導致 MultipartFile 參數方式全部失敗

　　    並且獲取到流中的body是按照 multipart 方式處理的，有boundary邊界等新，非原始的文件信息

 

更好的處理文件上傳，直接使用binary方式，可以直接流中獲取文件的原始內容

curl --limit-rate 100k  -X POST 'http://127.0.0.1:8080/upload' -H 'Content-Type: application/x-sql' --data-binary '@/Users/siqi/Desktop/file.sql'>out.txt

 

控制器示例

    @RequestMapping(value = "/upload")
    public IdNamePair upload(HttpServletRequest request) throws IOException { 
　　　　　//實時讀取上傳的文件信息 ServletInputStream inputStream = request.getInputStream(); StringBuffer sb = new StringBuffer(); ServletInputStream fis = inputStream; int b = 0; while ((b = fis.read()) != -1) { sb.appendCodePoint(b); } inputStream.close(); System.out.println(sb.toString()); IdNamePair idNamePair = new IdNamePair(); idNamePair.setId(123); idNamePair.setName("藍銀草"); return idNamePair; }

php請求示例

<?php


$curl = curl_init();

curl_setopt_array($curl, array(
    CURLOPT_URL => "http://127.0.0.1:8080/upload",
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_ENCODING => "",
    CURLOPT_MAXREDIRS => 10,
    CURLOPT_TIMEOUT => 0,
    CURLOPT_FOLLOWLOCATION => true,
    CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
    CURLOPT_CUSTOMREQUEST => "POST",
    CURLOPT_POSTFIELDS => file_get_contents("/Users/siqi/Desktop/file.sql"),
    CURLOPT_HTTPHEADER => array(
        "Content-Type: application/octet-stream",
    ),
));

$response = curl_exec($curl);

curl_close($curl);
echo $response;

 

  

參考：

https://stackoverflow.com/questions/16462088/php-curl-and-stream-forwarding

https://www.php.net/manual/zh/function.curl-setopt.php

https://www.ashleysheridan.co.uk/blog/Creating+a+Streaming+Proxy+with+PHP#stream-the-response