Improving RTP latency with OAK-1 Lite Camera

Switchesapleo_O

Hello.
I am currently working on an RTP stream with the OAK-1 Lite camera, and I'm looking for ways to improve the video latency. Here is the code I'm using:

#include <iostream>
#include <depthai/depthai.hpp>
#include <iostream>
#include <vector>
#include <boost/asio.hpp>

/* Demonstration RTP stream example with OAK-1 Lite */
int main(int argc, char** argv, char ** envp){
	boost::asio::io_context io_context;
	boost::asio::ip::udp::socket socket(io_context, boost::asio::ip::udp::v4());
	boost::asio::ip::udp::endpoint remote_endpoint(boost::asio::ip::address::from_string("127.0.0.1"), 5600);
    boost::asio::socket_base::send_buffer_size option(0);
    socket.set_option(option);

    dai::Pipeline pipeline;

    auto colorCam = pipeline.create<dai::node::ColorCamera>();
    colorCam->setBoardSocket(dai::CameraBoardSocket::AUTO);
    // colorCam->setIspScale(1, 3);
    colorCam->setColorOrder(dai::ColorCameraProperties::ColorOrder::BGR);
    colorCam->setResolution(dai::ColorCameraProperties::SensorResolution::THE_1080_P);
    colorCam->setPreviewSize(1280, 720);
    colorCam->setVideoSize(1280, 720);
    colorCam->setInterleaved(false);
    colorCam->setFps(40);

    auto videoEnc = pipeline.create<dai::node::VideoEncoder>();
    colorCam->video.link(videoEnc->input);
    videoEnc->setDefaultProfilePreset(40, dai::VideoEncoderProperties::Profile::H265_MAIN);
    videoEnc->setBitrateKbps(2000);
    videoEnc->setLossless(false);
    videoEnc->setRateControlMode(dai::VideoEncoderProperties::RateControlMode::CBR);
    videoEnc->setKeyframeFrequency(5);

    auto xout = pipeline.create<dai::node::XLinkOut>();
    videoEnc->bitstream.link(xout->input);
    xout->setStreamName("video");

    dai::Device device(pipeline);
    std::shared_ptr<dai::DataOutputQueue> q = device.getOutputQueue("video", 40, true);

    uint16_t seqNum = 0;
    while(true) {
    	std::shared_ptr<dai::ImgFrame> h265Packet = q->get<dai::ImgFrame>();

		uint8_t rtpHeader[12] = {0x80, 0xE0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
		rtpHeader[2] = (seqNum >> 8) & 0xFF;
		rtpHeader[3] = seqNum & 0xFF;

		// Increment the sequence number
		seqNum++;

		// Combine the RTP header and the H.265 data into a single buffer
		std::vector<uint8_t> rtpPacket(rtpHeader, rtpHeader + sizeof(rtpHeader));
		rtpPacket.insert(rtpPacket.end(), h265Packet->getData().begin(), h265Packet->getData().end());

		// Send the RTP packet
		socket.send_to(boost::asio::buffer(rtpPacket), remote_endpoint); /* may fail if exceeds UDP packet size(TODO add fragmentation), just for demonstration */
    }
    return 0;
}

With this code, I'm able to successfully stream H.265 video from the camera over RTP. However, I am experiencing some latency, and I would like to minimize it as much as possible.
Is there are anything I can do to improve the latency in this setup? Are there any specific settings or configurations that could help reduce latency in the video streaming process?
Thank you in advance!

jakaskerl

Hi Switchesapleo_O
Things you can do (courtesy of ChatGPT):

Decrease the video encoder's bitrate: The higher the bitrate, the more data needs to be sent over the network, which can increase latency. By decreasing the bitrate, you can reduce the amount of data that needs to be sent, which can help to reduce latency.
Increase the frame rate: By increasing the frame rate, you can reduce the amount of time between frames, which can help to reduce latency.
Reduce the resolution: By reducing the resolution of the video stream, you can reduce the amount of data that needs to be sent over the network, which can help to reduce latency.
Use a faster network connection: If possible, use a faster network connection, such as a wired Ethernet connection, to reduce latency.
Optimize the network settings: Make sure that the network settings are optimized for low latency. For example, you may want to disable any network services that are not necessary for streaming video, such as file sharing or printer sharing.
Use a lower latency protocol: RTP is a relatively high latency protocol. If latency is a major concern, you may want to consider using a lower latency protocol, such as WebRTC. <--- also supported by depthai

Hope it helps,
Jaka

IvanW

Did you have any luck in getting reduced latency? Running into the same issue, even running a basic UDP stream on the camera directly.