使用Qt連接scrcpy-server控制手機

這篇具有很好參考價值的文章主要介紹了使用Qt連接scrcpy-server控制手機。希望對大家有所幫助。如果存在錯誤或未考慮完全的地方，請大家不吝賜教，您也可以點擊"舉報違法"按鈕提交疑問。

測試環(huán)境

首先放一些測試環(huán)境，不保證其他環(huán)境也能夠這樣使用：

Qt庫：5.12.2，mscv2019_64
scrcpy：2.3.1
FFmpeg：ffmpeg-n5.1.4-1-gae14d9c06b-win64-gpl-shared-5.1
Adb：34.0.5
Android環(huán)境：MuMu模擬器12

如何啟動scrcpy-server

首先需要說明的是，我們是與scrcpy-server建立連接，而單純想顯示手機上的畫面與控制，作者github發(fā)布有scrcpy.exe可以直接運行使用，而這里我們相當于做另一個scrcpy，從而達到一些自定義控制的目的。與scrcpy-server建立連接，github上開發(fā)文檔也說明了https://github.com/Genymobile/scrcpy/blob/master/doc/develop.md，這里更詳細的說明下與scrcpy-server建立連接的具體細節(jié)。為了更好的描述細節(jié)，下面所有操作使用Qt代碼演示。

1. 連接設(shè)備

啟動scrcpy-server的所有操作都是經(jīng)過Adb進行的，不了解Adb命令建議先學(xué)習(xí)一下相關(guān)命令，因此，連接設(shè)備前先確保手機上打開了“USB調(diào)試”開關(guān)。連接設(shè)備使用命令adb connect，Qt中執(zhí)行Adb命令使用QProcess類，這里我們封裝一個Adb工具類以方便的執(zhí)行命令：

//頭文件
#pragma once

#include <qobject.h>
#include <qprocess.h>

/**
 * @brief Adb命令執(zhí)行封裝類
 */
class AdbCommandRunner {
public:
    explicit AdbCommandRunner(const QString& deviceName = QString());

    ~AdbCommandRunner();

    /**
     * @brief 執(zhí)行Adb命令
     * @param cmds 參數(shù)列表
     * @param waitForFinished 是否等待執(zhí)行完成
     */
    void runAdb(const QStringList& cmds, bool waitForFinished = true);

    /**
     * @brief 獲取執(zhí)行結(jié)果的錯誤
     * @return
     */
    QString getLastErr();

    QString lastFeedback; //執(zhí)行結(jié)果返回的字符串

private:
    QProcess process;
    QString deviceName;
};

//cpp
#include "adbcommandrunner.h"

#include <qdebug.h>

AdbCommandRunner::AdbCommandRunner(const QString &deviceName)
    : deviceName(deviceName)
{}

AdbCommandRunner::~AdbCommandRunner()  {
    if (process.isOpen()) {
        process.kill();
        process.waitForFinished();
    }
}

void AdbCommandRunner::runAdb(const QStringList &cmds, bool waitForFinished) {
    if (deviceName.isEmpty()) {
        process.start("adb/adb", cmds);
    } else {
        process.start("adb/adb", QStringList({"-s", deviceName}) + cmds);
    }
    qDebug() << "do adb execute command:" << "adb " + cmds.join(' ');
    if (waitForFinished) {
        process.waitForFinished();
    }
    lastFeedback = process.readAllStandardOutput();
}

QString AdbCommandRunner::getLastErr() {
    QString failReason = process.readAllStandardError();
    if (failReason.isEmpty()) {
        failReason = lastFeedback;
    }
    return failReason;
}

需要注意的是，Adb服務(wù)是后臺運行的，我們可以直接執(zhí)行adb connect命令連接設(shè)備，adb會自動啟動服務(wù)，然而啟動服務(wù)是需要個幾秒鐘，直接QProcess執(zhí)行會有個等待時間，正確的做法是，先使用adb start-server啟動服務(wù)，這個過程可以在線程中執(zhí)行：

QThread::create([] {
	QProcess process;
    process.start("adb/adb", {"start-server"});
    process.waitForFinished();
    if (process.exitCode() == 0 && process.exitStatus() == QProcess::NormalExit) {
        qDebug() << "adb server start finished!";
    } else {
    	qDebug() << "adb server start failed:" << process.readAll();
	}
))->start();

如果服務(wù)啟動成功，并且設(shè)備存在，連接時幾乎沒有等待時間

bool connectToDevice() {
    AdbCommandRunner runner;
    runner.runAdb({"connect", deviceAddress});
    if (runner.lastFeedback.contains("cannot connect to")) {
        qDebug() << "connect device:" << deviceAddress << "failed, error:" << runner.getLastErr();
        return false;
    }
    qInfo() << "connect device:" << deviceAddress << "success!";
    return true;
}

2. 推送scrcpy-server到手機上

推送文件自然是使用adb push命令，建議是推送到臨時目錄/data/local/tmp下：

bool pushServiceToDevice() {
    auto scrcpyFilePath = QDir::currentPath() + "/scrcpy/scrcpy-server";
    qDebug() << "scrcpy path:" << scrcpyFilePath;

    AdbCommandRunner runner;
    runner.runAdb({"-s", deviceAddress, "push", scrcpyFilePath, "/data/local/tmp/scrcpy-server.jar"});
    if (!runner.lastFeedback.contains("1 file pushed")) {
        qDebug() << runner.getLastErr();
        return false;
    }
    return true;
}

3. 建立Adb隧道連接

默認情況下，scrcpy-server是作為客戶端，通過adb隧道連接到電腦端的本地Tcp服務(wù)器，如開發(fā)者文檔上描述，這個角色也是可以反轉(zhuǎn)的，只需要在啟動服務(wù)命令里面添加tunnel_forward=true（注意不是啟動scrcpy.exe的命令行參數(shù)）。默認角色下，使用adb reverse命令開啟隧道連接，需要注意的是，隧道名中需要攜帶一個8位字符串scid作為標識，這里我們可以使用時間戳代替：

scid = QString::asprintf("%08x", QDateTime::currentSecsSinceEpoch());
AdbCommandRunner runner;
runner.runAdb({"-s", deviceAddress, "reverse", "localabstract:scrcpy_" + scid, "tcp:27183")});

記住這個27183端口，下面使用QTcpServer時正是使用這個端口監(jiān)聽服務(wù)的連接。

4. 啟動服務(wù)

scrcpy-server本身是一個可執(zhí)行的jar包，啟動這個jar包，使用adb shell命令：

serverRunner = new AdbCommandRunner;
QStringList scrcpyServiceOpt;
scrcpyServiceOpt << "-s" << deviceAddress << "shell";
scrcpyServiceOpt << "CLASSPATH=/data/local/tmp/scrcpy-server.jar";
scrcpyServiceOpt << "app_process";
scrcpyServiceOpt << "/";
scrcpyServiceOpt << "com.genymobile.scrcpy.Server";
scrcpyServiceOpt << SCRCPY_VERSION;
scrcpyServiceOpt << "scid=" + scid;
scrcpyServiceOpt << "audio=false"; //不傳輸音頻
scrcpyServiceOpt << "max_fps=" + QString::number(maxFrameRate); //最大幀率
scrcpyServiceOpt << "max_size=1920"; //視頻幀最大尺寸
serverRunner->runAdb(scrcpyServiceOpt, false);

需要注意的是，這里QProcess對象需要保存，關(guān)閉服務(wù)時需要殺死對應(yīng)的adb shell子進程。在上面參數(shù)中scid以及之前的參數(shù)是必要的，如果版本號和scid對應(yīng)不上無法啟動服務(wù)。更多的控制參數(shù)可以參考源代碼scrcpy\app\src\server.c第212行開始，其中參數(shù)的默認值在scrcpy\app\src\options.c中，啟動成功后就會立即通過adb與電腦端本地服務(wù)建立連接。

5. 關(guān)閉服務(wù)

關(guān)閉服務(wù)時，首先需要結(jié)束shell進程，然后關(guān)閉隧道即可：

if (serverRunner) {
    delete serverRunner;
    serverRunner = nullptr;
}

AdbCommandRunner runner;
runner.runAdb({"-s", deviceAddress, "reverse", "--remove", "localabstract:scrcpy_" + scid});

關(guān)閉服務(wù)之后，scrcpy-server會自己在設(shè)備中刪除，重新啟動服務(wù)需要從第2步驟推送文件開始。

使用QTcpServer與scrcpy-server建立連接

上面說了，默認情況下電腦端作為tcp服務(wù)器，scrcpy-server作為客戶端建立連接，因此，使用QTcpServer監(jiān)聽本地adb隧道連接端口即可：

ScrcpyServer::ScrcpyServer(QObject *parent)
    : QObject(parent)
{
    //tcp服務(wù)
    tcpServer = new QTcpServer(this);
    connect(tcpServer, &QTcpServer::acceptError, this, [] (QAbstractSocket::SocketError socketError) {
        qCritical() << "scrcpy server accept error:" << socketError;
    });
    connect(tcpServer, &QTcpServer::newConnection, this, &ScrcpyServer::handleNewConnection);
}

void ScrcpyServer::handleNewConnection() {
    auto socket = tcpServer->nextPendingConnection();
    //第一個socket為視頻流
    if (!videoSocket) {
        videoSocket = socket;
        connect(socket, &QTcpSocket::readyRead, this, &ScrcpyServer::receiveVideoBuffer);
        qInfo() << "video socket pending connect...";
    } else if (!controlSocket) {
        controlSocket = socket;
        connect(socket, &QTcpSocket::readyRead, this, &ScrcpyServer::receiveControlBuffer);
        qInfo() << "control socket pending connect...";
    } else {
        qWarning() << "unexpect socket appending...";
    }
    connect(socket, &QTcpSocket::stateChanged, this, [=] (QAbstractSocket::SocketState state) {
        qDebug() << "socket state changed:" << state;
        if (state == QAbstractSocket::UnconnectedState) {
            socket->deleteLater();
        }
    });
}

bool ScrcpyServer::start() {
    if (!tcpServer->isListening()) {
        bool success = tcpServer->listen(QHostAddress::AnyIPv4, 27183);
        if (!success) {
            qDebug() << "tcp server listen failed:" << tcpServer->errorString();
        }
    }
}

根據(jù)開發(fā)者文檔描述，scrcpy-server連接到QTcpServer后，會有3個tcp連接分別用來傳輸：視頻、音頻、控制命令，這里我們在啟動時設(shè)置了audio=false關(guān)閉了音頻傳輸，因此第2個為控制socket。

建立連接并視頻推流完整流程

上面講了啟動scrcpy-server和使用QTcpServer建立連接，事實上，建立連接和啟動tcp服務(wù)是需要按照順序進行的：

1. 開啟視頻推流過程

開啟QTcpServer服務(wù)，監(jiān)聽指定端口如27183
推送scrcpy-server到手機上
使用tcp服務(wù)監(jiān)聽的端口，和8位隨機字符串作為scid，建立Adb隧道連接
使用adb shell命令啟動scrcpy-server服務(wù)
QTcpServer等待視頻流和控制socket連接

2. 關(guān)閉視頻推流過程

結(jié)束adb shell子進程
關(guān)閉Adb隧道連接
關(guān)閉Tcp服務(wù)

視頻流的解碼

1. 數(shù)據(jù)包協(xié)議解析

文檔中詳細描述了視頻流的數(shù)據(jù)組成，最開始視頻流會傳輸64字節(jié)表示設(shè)備的名稱，然后依次傳輸4字節(jié)編碼方式、4字節(jié)幀圖像寬度、4字節(jié)幀圖像高度，接著開始傳輸視頻幀，其中視頻幀由幀頭和數(shù)據(jù)組成，幀頭中包含有PTS標志（8字節(jié)）和幀數(shù)據(jù)長度（4字節(jié)）兩個信息，后面接收幀數(shù)據(jù)長度的數(shù)據(jù)即可，然后等待接收下一幀數(shù)據(jù)。視頻默認編碼為H.264，可以通過啟動服務(wù)參數(shù)更改編碼類型，這里我們使用FFmpeg來解析視頻幀。
由于解碼是個耗時任務(wù)，需要放到線程中運行，這里就需要與QTcpSocket接收到的數(shù)據(jù)進行線程同步處理，為了讓解碼線程看起來像是以同步方式讀取數(shù)據(jù)，編寫一個工具類來接收QTcpSocket發(fā)送來的數(shù)據(jù)：

//頭文件
#pragma once

#include <qobject.h>
#include <qmutex.h>
#include <qwaitcondition.h>

#include "byteutil.h"

class BufferReceiver : public QObject {
public:
    explicit BufferReceiver(QObject *parent = nullptr);

    void sendBuffer(const QByteArray& data);

    void endCache();

    template<typename T>
    T receive() {
        enum {
            T_Size = sizeof(T)
        };

        T value = T();
        receive((void*)&value, T_Size);
        ByteUtil::swapBits(value);
        return value;
    }

    void receive(void* data, int len);

    bool isEndReceive() const {
        return endBufferCache;
    }

private:
    QByteArray receiveBuffer;
    QMutex mutex;
    QWaitCondition receiveWait;

    bool endBufferCache;
};

//cpp
#include "bufferreceiver.h"

BufferReceiver::BufferReceiver(QObject *parent)
    : QObject(parent)
    , endBufferCache(false)
{}

void BufferReceiver::sendBuffer(const QByteArray &data) {
    QMutexLocker locker(&mutex);
    receiveBuffer.append(data);
    receiveWait.notify_all();
}

void BufferReceiver::endCache() {
    QMutexLocker locker(&mutex);
    endBufferCache = true;
    receiveWait.notify_all();
}

void BufferReceiver::receive(void *data, int len) {
    mutex.lock();
    if (endBufferCache) {
        mutex.unlock();
        return;
    }
    while (receiveBuffer.size() < len && !endBufferCache) {
        receiveWait.wait(&mutex);
    }
    if (!endBufferCache) {
        memcpy(data, receiveBuffer.data(), len);
        receiveBuffer = receiveBuffer.mid(len);
    }
    mutex.unlock();
}

在主線程中收到視頻流數(shù)據(jù)就緩存到BufferReceiver中：

void ScrcpyServer::receiveVideoBuffer() {
    if (videoDecoder) {
        videoDecoder->appendBuffer(videoSocket->readAll());
    }
}

解碼器線程按照協(xié)議依次接收數(shù)據(jù)包：

void VideoDecoder::run() {
    QByteArray remoteDeviceName(64, '\0');
    bufferReceiver.receive(remoteDeviceName.data(), remoteDeviceName.size());
    auto name = QString::fromUtf8(remoteDeviceName);
    if (!name.isEmpty()) {
        qInfo() << "device name received:" << name;
    }
    if (bufferReceiver.isEndReceive()) {
        return;
    }

    if (codecCtx == nullptr) {
        auto codecId = bufferReceiver.receive<uint32_t>();
        auto width = bufferReceiver.receive<int>();
        auto height = bufferReceiver.receive<int>();
        if (!codecInit(codecId, width, height)) {
            codecRelease();
            qCritical() << "video decode init failed!";
            return;
        }
    }
    qInfo() << "video decode is running...";
    for (;;) {
        if (!frameReceive()) {
            break;
        }
        if (!frameMerge()) {
            av_packet_unref(packet);
            break;
        }
        frameUnpack();
        av_packet_unref(packet);
    }

    //釋放資源
    codecRelease();

    qInfo() << "video decoder exit...";
}

2. 解碼流程

注意上面解碼線程的讀取數(shù)據(jù)步驟，在讀取到解碼器和幀大小時就可以進行解碼器初始化了：

//初始化解碼器
auto codec = avcodec_find_decoder(AV_CODEC_ID_H264);
if (!codec) {
    qDebug() << "find codec h264 fail!";
    return false;
}

//初始化解碼器上下文
codecCtx = avcodec_alloc_context3(codec);
if (!codecCtx) {
    qDebug() << "allocate codec context fail!";
    return false;
}

codecCtx->width = width;
codecCtx->height = height;
codecCtx->pix_fmt = AV_PIX_FMT_YUV420P;

int ret = avcodec_open2(codecCtx, codec, nullptr);
if (ret < 0) {
    qDebug() << "open codec fail!";
    return false;
}

packet = av_packet_alloc();
if (!packet) {
    qDebug() << "alloc packet fail!";
    return false;
}

decodeFrame = av_frame_alloc();
if (!decodeFrame) {
    qDebug() << "alloc frame fail!";
    return false;
}

獲取到幀數(shù)據(jù)時，依次讀取PTS和幀數(shù)據(jù)大小，設(shè)置到AVPacket中：

bool VideoDecoder::frameReceive() {
    auto ptsFlags = bufferReceiver.receive<uint64_t>();
    auto frameLen = bufferReceiver.receive<int32_t>();
    if (bufferReceiver.isEndReceive()) {
        return false;
    }
    Q_ASSERT(frameLen != 0);

    if (av_new_packet(packet, frameLen)) {
        qDebug() << "av new packet failed!";
        return false;
    }
    bufferReceiver.receive(packet->data, frameLen);
    if (bufferReceiver.isEndReceive()) {
        return false;
    }

    if (ptsFlags & SC_PACKET_FLAG_CONFIG) {
        packet->pts = AV_NOPTS_VALUE;
    } else {
        packet->pts = ptsFlags & SC_PACKET_PTS_MASK;
    }

    if (ptsFlags & SC_PACKET_FLAG_KEY_FRAME) {
        packet->flags |= AV_PKT_FLAG_KEY;
    }
    packet->dts = packet->pts;
    return true;
}

根據(jù)PTS判斷是否需要進行幀合并：

bool VideoDecoder::frameMerge() {
    bool isConfig = packet->pts == AV_NOPTS_VALUE;
    if (isConfig) {
        free(mergeBuffer);
        mergeBuffer = (uint8_t*)malloc(packet->size);
        if (!mergeBuffer) {
            qDebug() << "merge buffer malloc failed! required size:" << packet->size;
            return false;
        }
        memcpy(mergeBuffer, packet->data, packet->size);
        mergedSize = packet->size;
    }
    else if (mergeBuffer) {
        if (av_grow_packet(packet, mergedSize)) {
            qDebug() << "av grow packet failed!";
            return false;
        }
        memmove(packet->data + mergedSize, packet->data, packet->size);
        memcpy(packet->data, mergeBuffer, mergedSize);

        free(mergeBuffer);
        mergeBuffer = nullptr;
    }
    return true;
}

視頻幀解包分別使用avcodec_send_packet和avcodec_receive_frame，下面代碼中演示了如何循環(huán)解包，然后轉(zhuǎn)換為QVideoFrame對象（供后面視頻渲染使用），注意這里圖像格式為YUV420P：

void VideoDecoder::frameUnpack() {
    if (packet->pts == AV_NOPTS_VALUE) {
        return;
    }
    int ret = avcodec_send_packet(codecCtx, packet);
    if (ret < 0 && ret != AVERROR(EAGAIN)) {
        qCritical() << "send packet error:" << ret;
    } else {
        //循環(huán)解析數(shù)據(jù)幀
        for (;;) {
            ret = avcodec_receive_frame(codecCtx, decodeFrame);
            if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF) {
                break;
            }

            if (ret) {
                qCritical() << "could not receive video frame:" << ret;
                break;
            }

            QVideoFrame cachedFrame(codecCtx->width * codecCtx->height * 3 / 2,
                                    QSize(codecCtx->width, codecCtx->height),
                                    codecCtx->width, QVideoFrame::Format_YUV420P);
            int imageSize = av_image_get_buffer_size(codecCtx->pix_fmt, codecCtx->width, codecCtx->height, 1);
            if (cachedFrame.map(QAbstractVideoBuffer::WriteOnly)) {
                uchar *dstData = cachedFrame.bits();
                av_image_copy_to_buffer(dstData, imageSize, decodeFrame->data, decodeFrame->linesize,
                                        codecCtx->pix_fmt,
                                        codecCtx->width, codecCtx->height, 1);
                cachedFrame.unmap();

                emit frameDecoded(cachedFrame);
            }
            av_frame_unref(decodeFrame);
        }
    }
}

3. 視頻幀轉(zhuǎn)QImage

有時候我們需要提取視頻的一幀圖像，例如截圖操作，需要直接轉(zhuǎn)RGB圖像，這時候有兩種方法，一是直接對AVFrame進行轉(zhuǎn)換，也就是上面提到的decodeFrame，使用sws_scale函數(shù)，但是需要先初始化一個SwsContext，初始化可以在CodecContext初始化之后進行：

swsContext = sws_getContext(codecCtx->width, codecCtx->height, codecCtx->pix_fmt,
                            codecCtx->width, codecCtx->height, AV_PIX_FMT_RGB24, SWS_BILINEAR,
                            nullptr, nullptr, nullptr);

轉(zhuǎn)換時根據(jù)codecCtx信息先構(gòu)造一個QImage，再調(diào)用sws_scale即可：

QImage image = QImage(codecCtx->width, codecCtx->height, QImage::Format_RGB888);
auto imagePtr = image.bits();
//將YUV420p轉(zhuǎn)換為RGB24
const int lineSize[4] = {3*codecCtx->width, 0, 0, 0};
sws_scale(swsContext, (const uint8_t* const*)decodeFrame->data, decodeFrame->linesize,
          0, codecCtx->height, (uint8_t**)&imagePtr, lineSize);

第二種方法就比較簡單了，通過Qt內(nèi)置的方法轉(zhuǎn)換，正如上面提到，通過av_image_copy_to_buffer函數(shù)將AVFrame轉(zhuǎn)成了QVideoFrame最后發(fā)送了出來，獲取QImage直接調(diào)用image函數(shù)即可，此時轉(zhuǎn)換出來的格式是ARGB32：

QImage image = cachedFrame.image();

使用OpenGL渲染顯示視頻流

顯示視頻最好的辦法就是使用OpenGL渲染，這樣不會消耗大量的CPU資源，并且原視頻幀解碼出來的YUV420P也可以在OpenGL中計算。Qt中使用OpenGL自然是繼承QOpenGLWidget，Qt官方正好有一個顯示視頻的控件QVideoWidget，只是沒有提供直接設(shè)置視頻流的方法，仔細閱讀Multimedia模塊中的QVideoWidget源代碼發(fā)現(xiàn)，如果使用GLSL，經(jīng)過QPainterVideoSurface實例，最終進行渲染使用的是QVideoSurfaceGlslPainter，其中支持各種圖像幀類型的渲染，其中YUV420P也包含在內(nèi)，對于YUV420P轉(zhuǎn)RGB使用的是BT709標準。復(fù)制源代碼中multimediawidgets/qmediaopenglhelper_p.h、multimediawidgets/qpaintervideosurface_p.h、multimediawidgets/qpaintervideosurface.cpp3個文件，自定義一個VideoWidget其中實例化一個QPainterVideoSurface，刷新圖片是使用QPainterVideoSurface::present函數(shù)即可：

//.h
#pragma once

#include <qwidget.h>
#include <qopenglwidget.h>

#include "qpaintervideosurface_p.h"

class VideoWidget : public QOpenGLWidget {
public:
    explicit VideoWidget(QWidget *parent = nullptr);
    ~VideoWidget();

    QPainterVideoSurface *videoSurface() const;

    QSize sizeHint() const override;

public:
    void setAspectRatioMode(Qt::AspectRatioMode mode);

protected:
    void hideEvent(QHideEvent *event) override;
    void resizeEvent(QResizeEvent *event) override;
    void paintEvent(QPaintEvent *event) override;

private slots:
    void formatChanged(const QVideoSurfaceFormat &format);
    void frameChanged();

private:
    void updateRects();

private:
    QPainterVideoSurface *m_surface;
    Qt::AspectRatioMode m_aspectRatioMode;
    QRect m_boundingRect;
    QRectF m_sourceRect;
    QSize m_nativeSize;
    bool m_updatePaintDevice;
};

//.cpp
#include "videowidget.h"

#include <qevent.h>
#include <qvideosurfaceformat.h>

VideoWidget::VideoWidget(QWidget *parent)
    : QOpenGLWidget(parent)
    , m_aspectRatioMode(Qt::KeepAspectRatio)
    , m_updatePaintDevice(true)
{
    m_surface = new QPainterVideoSurface(this);
    
    connect(m_surface, &QPainterVideoSurface::frameChanged, this, &VideoWidget::frameChanged);
    connect(m_surface, &QPainterVideoSurface::surfaceFormatChanged, this, &VideoWidget::formatChanged);
}

QPainterVideoSurface *VideoWidget::videoSurface() const {
    return m_surface;
}

VideoWidget::~VideoWidget() {
    delete m_surface;
}

void VideoWidget::setAspectRatioMode(Qt::AspectRatioMode mode)
{
    m_aspectRatioMode = mode;
    updateGeometry();
}

QSize VideoWidget::sizeHint() const
{
    return m_surface->surfaceFormat().sizeHint();
}

void VideoWidget::hideEvent(QHideEvent *event)
{
    m_updatePaintDevice = true;
}

void VideoWidget::resizeEvent(QResizeEvent *event)
{
    updateRects();
}

void VideoWidget::paintEvent(QPaintEvent *event)
{
    QPainter painter(this);

    if (testAttribute(Qt::WA_OpaquePaintEvent)) {
        QRegion borderRegion = event->region();
        borderRegion = borderRegion.subtracted(m_boundingRect);

        QBrush brush = palette().window();

        for (const QRect &r : borderRegion)
            painter.fillRect(r, brush);
    }

    if (m_surface->isActive() && m_boundingRect.intersects(event->rect())) {
        m_surface->paint(&painter, m_boundingRect, m_sourceRect);

        m_surface->setReady(true);
    } else {
        if (m_updatePaintDevice && (painter.paintEngine()->type() == QPaintEngine::OpenGL
                                    || painter.paintEngine()->type() == QPaintEngine::OpenGL2)) {
            m_updatePaintDevice = false;

            m_surface->updateGLContext();
            if (m_surface->supportedShaderTypes() & QPainterVideoSurface::GlslShader) {
                m_surface->setShaderType(QPainterVideoSurface::GlslShader);
            } else {
                m_surface->setShaderType(QPainterVideoSurface::FragmentProgramShader);
            }
        }
    }
}

void VideoWidget::formatChanged(const QVideoSurfaceFormat &format)
{
    m_nativeSize = format.sizeHint();

    updateRects();
    updateGeometry();
    update();
}

void VideoWidget::frameChanged()
{
    update(m_boundingRect);
}

void VideoWidget::updateRects()
{
    QRect rect = this->rect();

    if (m_nativeSize.isEmpty()) {
        m_boundingRect = QRect();
    } else if (m_aspectRatioMode == Qt::IgnoreAspectRatio) {
        m_boundingRect = rect;
        m_sourceRect = QRectF(0, 0, 1, 1);
    } else if (m_aspectRatioMode == Qt::KeepAspectRatio) {
        QSize size = m_nativeSize;
        size.scale(rect.size(), Qt::KeepAspectRatio);

        m_boundingRect = QRect(0, 0, size.width(), size.height());
        m_boundingRect.moveCenter(rect.center());

        m_sourceRect = QRectF(0, 0, 1, 1);
    } else if (m_aspectRatioMode == Qt::KeepAspectRatioByExpanding) {
        m_boundingRect = rect;

        QSizeF size = rect.size();
        size.scale(m_nativeSize, Qt::KeepAspectRatio);

        m_sourceRect = QRectF(
                0, 0, size.width() / m_nativeSize.width(), size.height() / m_nativeSize.height());
        m_sourceRect.moveCenter(QPointF(0.5, 0.5));
    }
}

開始視頻推流之前，初始化Surface，設(shè)置使用OpenGL渲染，并指定視頻格式為YUV420P：

videoWidget->videoSurface()->setShaderType(QPainterVideoSurface::GlslShader);
videoWidget->videoSurface()->start(QVideoSurfaceFormat(QSize(1920, 1080), QVideoFrame::Format_YUV420P));

從VideoDecoder獲取到視頻幀時發(fā)送到Surface：

connect(decorder, &VideoDecoder::frameDecoded, this, [&](const QVideoFrame& frame) {
	videoWidget->videoSurface()->present(frame);
});

關(guān)閉推流時，同時關(guān)閉Surface渲染：

videoWidget->videoSurface()->stop();

控制命令的下發(fā)

命令的控制是通過第二個socket發(fā)送數(shù)據(jù)，其命令的編碼協(xié)議定義和編碼在源代碼scrcpy\app\src\control_msg.h、scrcpy\app\src\control_msg.c這兩個文件中。例如，發(fā)送一個點擊事件：

namespace ByteUtil {
    /**
     * @brief 字節(jié)序交換
     * @tparam T 數(shù)值類型
     * @param data 轉(zhuǎn)換目標數(shù)值
     * @param size 字節(jié)序交換大小
    */
    template<typename T>
    static void swapBits(T& data, size_t size = sizeof(T)) {
        for (size_t i = 0; i < size / 2; i++) {
            char* pl = (char*)&data + i;
            char* pr = (char*)&data + (size - i - 1);
            if (*pl != *pr) {
                *pl ^= *pr;
                *pr ^= *pl;
                *pl ^= *pr;
            }
        }
    }
    
	/**
     * @brief char*轉(zhuǎn)指定數(shù)值類型（大端序）
     * @tparam T 數(shù)值類型
     * @param data 轉(zhuǎn)換目標數(shù)值
     * @param src 原字節(jié)數(shù)組
     * @param srcSize 原字節(jié)數(shù)組大小
    */
    template<typename T>
    static void bitConvert(T& data, const void* src, int srcSize = sizeof(T)) {
        memcpy(&data, src, srcSize);
        swapBits(data, srcSize);
    }
}

class ControlMsg {
public:
    static QByteArray injectTouchEvent(android_motionevent_action action, android_motionevent_buttons actionButton,
                                       android_motionevent_buttons buttons, uint64_t pointerId,
                                       const QSize& screenSize, const QPoint& point, float pressure) 
    {
		char bytes[32];
	    bytes[0] = SC_CONTROL_MSG_TYPE_INJECT_TOUCH_EVENT;
	    bytes[1] = action;
	    ByteUtil::bitConvert(*(uint64_t*)(bytes + 2), &pointerId);
	    uint32_t x = point.x();
	    ByteUtil::bitConvert(*(uint32_t*)(bytes + 10), &x);
	    uint32_t y = point.y();
	    ByteUtil::bitConvert(*(uint32_t*)(bytes + 14), &y);
	    uint16_t w = screenSize.width();
	    ByteUtil::bitConvert(*(uint16_t*)(bytes + 18), &w);
	    uint16_t h = screenSize.height();
	    ByteUtil::bitConvert(*(uint16_t*)(bytes + 20), &h);
	    uint16_t pressureValue = sc_float_to_u16fp(pressure);
	    ByteUtil::bitConvert(*(uint16_t*)(bytes + 22), &pressureValue);
	    ByteUtil::bitConvert(*(uint32_t*)(bytes + 24), &actionButton);
	    ByteUtil::bitConvert(*(uint32_t*)(bytes + 28), &buttons);
	    return { bytes, 32 };
	}
};

注冊videoWidget事件過濾器，模擬發(fā)送鼠標事件：

bool App::eventFilter(QObject *watched, QEvent *event) {
    if (watched == videoWidget) {
        if (auto mouseEvent = dynamic_cast<QMouseEvent*>(event)) {
            auto dstPos = QPoint(qRound(mouseEvent->x() * framePixmapRatio.width()), qRound(mouseEvent->y() * framePixmapRatio.height()));
            if (mouseEvent->type() == QEvent::MouseButtonPress) {
                scrcpyServer->sendControl(ControlMsg::injectTouchEvent(AMOTION_EVENT_ACTION_DOWN, AMOTION_EVENT_BUTTON_PRIMARY,
                                                                       AMOTION_EVENT_BUTTON_PRIMARY, 0,
                                                                       frameSrcSize, dstPos, 1.0));
            } else if (mouseEvent->type() == QEvent::MouseButtonRelease) {
                scrcpyServer->sendControl(ControlMsg::injectTouchEvent(AMOTION_EVENT_ACTION_UP, AMOTION_EVENT_BUTTON_PRIMARY,
                                                                       AMOTION_EVENT_BUTTON_PRIMARY, 0,
                                                                       frameSrcSize, dstPos, 0.0));
            } else if (mouseEvent->type() == QEvent::MouseMove) {
                scrcpyServer->sendControl(ControlMsg::injectTouchEvent(AMOTION_EVENT_ACTION_MOVE, AMOTION_EVENT_BUTTON_PRIMARY,
                                                                       AMOTION_EVENT_BUTTON_PRIMARY, 0,
                                                                       frameSrcSize, dstPos, 1.0));
            }
        }
    }
    return QObject::eventFilter(watched, event);
}

//ScrcpyServer
void ScrcpyServer::sendControl(const QByteArray &controlMsg) {
    if (controlSocket) {
        controlSocket->write(controlMsg);
    }
}