MartnTous

  • Apr 25, 2024
  • Joined Nov 9, 2023
  • 0 best answers
  • Hi @jakaskerl

    I assumed it had to do with hardware limitations, but after running in debug mode, I noticed that the memory and cpu usage is low, which I found strange (this can be seen in the image I attached above).

    As for the ISP processing capability being 600 MP/s I also find it odd that this is a limiter, since I am working with ISP downscaling. Even if I wasn't using downscaling and working with the 12 MP, I'm working at 5fps, with two cameras at a time, so 12x5x2=120 MP/s, which is considerably less than the 600 MP/s capacity.

    Maybe it's another limiting hardware component? Or does it have to do with my acquisition script (maybe there is a way to make it more efficient)?

  • Hi @jakaskerl! Thanks for your response.

    What is the purpose of synchronizing both cameras? I did it but I don't notice an improvement in acquisition latency.

    • Hello everybody!

      I am using two IMX378 cameras connected to an OAK-FFC 4P module. I need to improve the delay between image acquisition and display on screen, which is more than one second. At the same time, I have a very low fluency in the video, this is because it does not work at the fps that I indicate. Below is an image that shows the delay mentioned and the fps.

      I don't know if it is a hardware limitation and therefore a delay that I have to deal with, or if there is any possible way or technique to improve this code.

      If I do not use "cam1.setIspScale(1,2)" the delay is ~4 seconds.

      import cv2
      import depthai as dai
      import time
      
      def getFrame(queue):
          frame = queue.get()
          return frame.getCvFrame()
      
      if __name__ == '__main__':
      
          fps = 10
          frame_count = 0
          
          # Define a pipeline
          pipeline = dai.Pipeline()
          device = dai.Device()
      
          # DEFINE SOURCES AND OUTPUTS
          cam1 = pipeline.create(dai.node.ColorCamera)
          cam2 = pipeline.create(dai.node.ColorCamera)
          cam1.setBoardSocket(dai.CameraBoardSocket.CAM_A)  # 4-lane MIPI  IMX378
          cam2.setBoardSocket(dai.CameraBoardSocket.CAM_D)  # 4-lane MIPI  IMX378
          cam1.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
          cam2.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
          cam1.setIspScale(1,2)
          cam2.setIspScale(1,2)
          cam1.setFps(fps)
          cam2.setFps(fps)
          # Set output Xlink 
          outcam1 = pipeline.create(dai.node.XLinkOut)
          outcam2 = pipeline.create(dai.node.XLinkOut)
          outcam1.setStreamName("cam1")
          outcam2.setStreamName("cam2")
          # LINKING
          cam1.isp.link(outcam1.input)
          cam2.isp.link(outcam2.input)
      
          outcam1.input.setBlocking(False)
          outcam2.input.setBlocking(False)
          outcam1.input.setQueueSize(1)
          outcam2.input.setQueueSize(1)
      
          with device:
              device.startPipeline(pipeline)
              cam1 = device.getOutputQueue(name="cam1", maxSize=1, blocking=False)
              cam2 = device.getOutputQueue(name="cam2", maxSize=1, blocking=False)
              n=0
      
              while True:
                  frame_count += 1
                  n+=1
                  
                  elapsed_time = time.time() - start_time
                  if elapsed_time > 1.0:
                      fps = frame_count / elapsed_time
                      print(f"FPS: {fps:.2f}\n")
                      frame_count = 0
                      start_time = time.time()
      
                  start_adq = time.time()  # Inicia el temporizador
      
                  Frame1 = getFrame(cam1)
                  #Frame2 = getFrame(cam2)
      
                  end_adq = time.time()  # Detiene el temporizador
                  tiempo_adquisicion = end_adq - start_adq
                  print(n,f"- acquisition time: {tiempo_adquisicion} seconds")
      
                  # Check for keyboard input
                  key = cv2.waitKey(1) & 0xFF
      
                  # Display the current state in the window
                  cv2.imshow("Frame1", Frame1)
                  #cv2.imshow("Frame2", Frame2)
      
                  if key == ord('q'):
                      break

      Thank you very much

      • Hi @jakaskerl, thank yo very much!

        It works with blocking=False, I no longer see those old frames when I switch cameras.

        Although, I was trying to avoid using this because it increases a lot (about double) the delay time between capture and display. At the same time, the frame acquisition rate is quite irregular.

        I think this is because in the first case (blocking=True) there are already frames available in the buffer to be taken when requested, and in the second case there may or may not be frames available in the buffer, depending on whether or not they were read in time, right?

      • Hi @jakaskerl , thanks for your help.

        It seems that it is not due to the high resolution, because even if I decrease it to 1080_P, although the effect is more difficult to appreciate, because the program runs faster, when I switch camera I still see some past frames until it jumps to the current ones.

        I also tried with the command you mention "DEPTHAI_LEVEL=trace python3 script.py", but it's still the same.

        • Hello everyone, I am having a problem and I would appreciate if you could help me.

          I have two modules IXM378 connected to an OAK-FFC-4P module, I want to see on screen only one of the cameras at a time, therefore, when I press the space bar I make a switch to the opposite camera. The problem is that, when I make a switch it shows me 9 frames of the previous moment in which I was using the current camera, instead of showing me directly the current moment, and that there is a fluid transition between both videos. Here is a graphical example to make it clear:

          What I find strange is that the buffer is configured with a queue size of 1 frame, that is, all the time the buffer is being updated with the last frame (current moment), so I don't understand where that additional 9 frames from a past moment come from.

          Here is the code I am using:

          import cv2	
          import depthai as dai
          import numpy as np
          
          def getFrame(queue):
              # Get frame from queue
              frame = queue.get()
              # Convert frame to OpenCV format and return
              return frame.getCvFrame()
          
          
          if __name__ == '__main__':
          
              fps = 3
              current_state = "Framecam1a"
          
              # Define a pipeline
              pipeline = dai.Pipeline()
              device = dai.Device()
          
          
              # DEFINE SOURCES AND OUTPUTS
              cam1a = pipeline.create(dai.node.ColorCamera)
              cam1b = pipeline.create(dai.node.ColorCamera)
              cam1a.setBoardSocket(dai.CameraBoardSocket.CAM_A)  # 4-lane MIPI  IMX378
              cam1b.setBoardSocket(dai.CameraBoardSocket.CAM_D)  # 4-lane MIPI  IMX378
              cam1a.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
              cam1b.setResolution(dai.ColorCameraProperties.SensorResolution.THE_12_MP)
              cam1a.setFps(fps)
              cam1b.setFps(fps)
          
              # Set output Xlink 
              outcam1a = pipeline.create(dai.node.XLinkOut)
              outcam1b = pipeline.create(dai.node.XLinkOut)
              outcam1a.setStreamName("cam1a")
              outcam1b.setStreamName("cam1b")
                
              # LINKING
              cam1a.isp.link(outcam1a.input)
              cam1b.isp.link(outcam1b.input)
          
              # Pipeline is defined, now we can connect to the device
              with device:
                  device.startPipeline(pipeline)
                  # Get output queues.
                  cam1a = device.getOutputQueue(name="cam1a", maxSize=1)
                  cam1b = device.getOutputQueue(name="cam1b", maxSize=1)
                      
                  while True:
          
                       # Acquire frames only for the current state
                      if current_state == "Framecam1a":
                          Frame = getFrame(cam1a)
                      else:
                          Frame = getFrame(cam1b)
          
                      # Display the current state in the window
                      cv2.imshow("Frame", Frame)
          
                      # Check for keyboard input
                      key = cv2.waitKey(1) & 0xFF
          
                      if key == ord(' '):
                          n=0
                          # Change the state
                          if current_state == "Framecam1a":
                              current_state = "Framecam1b"
                          else:
                              current_state = "Framecam1a"
          
                      if key == ord('q'):
                          break
          • Hello!

            I wonder if there is any way to flash the EEPROM manually instead of with calibrate.py

            I am using an OAK-FFC 4P with two IMX378 modules, I have run calibrate.py and the EEPROM has been flashed. I have to prepare several of these cameras (OAK-FFC 4P + 2x IMX378), the extrinsic values are the same for all cameras, so I wonder if it is possible to flash manually those others, with the intrinsic values of the first flashed camera, instead of running calibrate.py too many times with each camera.

            If possible, what intrinsic values should I get from the first flashed camera and how do I get them? Additionally, how do I flash those values obtained from the first camera to the rest of the cameras?

            Thanks!

            • jakaskerl

              Create an RGB cam with high resolution, then output one end directly to xlink node and the other end (preview) to stereo depth node (along with the other RGB camera - also preview output).

              When I do this, I get the following error:

              .

              There is cam_d queue missing... Since you have created the Xlink node, but it's queue can't be read, it fill's up, backfeeding the rest of the pipeline and blocking it.

              There were two problems, one, the one you mention, and the other, the enableRectified option. Even adding the cam_d queue or even deleting the whole cam_d output node it still froze. It was solved by disabling this enableRectified option and adding the cam_d queue or deleting the cam_d output node (since I only need the output of one of the two RGB images and the stereo output).

            • Hi @jakaskerl thanks for your response.

              1-

              You can downsize the frames using ImgManip node, or you can set the preview size, and pipe the preview output to the stereo node (I'm not entirely sure, what type of input the node expects - RGB/NV12).

              So, using the ImgManip node can I get two different images from the same camera (a high-resolution image and a downscaled image that I can link to the stereo output)?

              I wonder if there is any code example of this implementation, I would appreciate it very much.

              2-

              Can you make sure you are getting an output? Cam-d is sometimes causing issues.

              Yeah, I'm sure that all ports are working correctly, I had an issue with port-D but I solved it with "pip3 install depthai -U"

              As mentioned above, I think the problem with the execution freezing has to do with connecting cam_a and cam_d to two different outputs (one for rgb image and one for stereoscopy)

                  # LINKING
                  cam_a.isp.link(outcam_a.input)
                  cam_d.isp.link(outcam_d.input)
                  cam_a.isp.link(stereo.right) 
                  cam_d.isp.link(stereo.left) 

              because if I test each system in different codes (on one side only rgb image and on the other side only stereoscopy) it works. What I find strange is that for a moment it allows me to work with both at the same time and then it freezes.

              • Hi everybody!

                I am using an OAK-FFC-4P, and it has two IMX378 modules connected.

                My goal is to be able to perform stereoscopy with both modules and at the same time use the RGB image of one of them. The reason why I opted for these IMX378 modules is because of their high resolution of 12 MP, since I need to have a high quality RGB image.

                The problem is that for stereoscopy the maximum quality supported is 1080 P, that is, I have to set both modules to 1080 P, but this way I do not have the high quality RGB image I need. If I set the resolution to a higher value, I get the following error:

                1. I wonder if there is a way to configure the modules in 12 MP (to have a high quality RGB image), and take from it a lower resolution image to perform the stereoscopy.

                2. After a moment, the code freezes and stops working. I do not understand why. I think it has to do with connecting cam_a and cam_d to two different outputs (one for rgb image and one for stereoscopy)

                  # LINKING

                      cam_a.isp.link(outcam_a.input)

                      cam_d.isp.link(outcam_d.input)
                      cam_a.isp.link(stereo.right)   

                  cam_d.isp.link(stereo.left)

                  because if I test each system in different codes (on one side only rgb image and on the other side only stereoscopy) it works fine. What I find strange is that for a moment it allows me to work with both at the same time and then it freezes.

                Below I show the code used:

                import cv2
                import depthai as dai
                import numpy as np
                
                def getFrame(queue):
                    # Get frame from queue
                    frame = queue.get()
                    # Convert frame to OpenCV format and return
                    return frame.getCvFrame()
                
                def StereoConfig(stereo):
                
                    # Prioritize fill-rate, sets Confidence threshold to 255
                    # Prioritize accuracy, sets Confidence threshold to 200
                    stereo.initialConfig.setConfidenceThreshold(245)
                    # LR-check is required for depth alignment. Better handling for occlusions
                    stereo.setLeftRightCheck(True)  
                    # Closer-in minimum depth, disparity range is doubled (from 95 to 190):
                    stereo.setExtendedDisparity(False)
                    # Better accuracy for longer distance, fractional disparity 32-levels:
                    stereo.setSubpixel(False)
                
                    # FILTERS CONFIGURATION
                    # Options: MEDIAN_OFF, KERNEL_3x3, KERNEL_5x5, KERNEL_7x7 (default)
                    stereo.initialConfig.setMedianFilter(dai.MedianFilter.KERNEL_7x7)
                    config = stereo.initialConfig.get()
                    config.postProcessing.speckleFilter.enable = True
                    config.postProcessing.speckleFilter.speckleRange = 5
                    config.postProcessing.temporalFilter.enable = True
                    config.postProcessing.temporalFilter.alpha = 0.01
                    config.postProcessing.spatialFilter.enable = True
                    config.postProcessing.spatialFilter.holeFillingRadius = 10
                    config.postProcessing.spatialFilter.numIterations = 3
                    # config.postProcessing.thresholdFilter.minRange = 400
                    # config.postProcessing.thresholdFilter.maxRange = 1000
                    config.postProcessing.decimationFilter.decimationFactor = 3
                    stereo.initialConfig.set(config)
                
                
                if name == 'main':
                
                     fps = 30
                     disparityMultiplier=2.125
                     enableRectified = True
                
                     # Define a pipeline
                     pipeline = dai.Pipeline()
                     device = dai.Device()
                
                     # DEFINE SOURCES AND OUTPUTS
                     cam_a = pipeline.create(dai.node.ColorCamera)
                     cam_d = pipeline.create(dai.node.ColorCamera)
                     stereo = pipeline.create(dai.node.StereoDepth)
                
                     cam_a.setBoardSocket(dai.CameraBoardSocket.CAM_A)  # IMX378    
                     cam_d.setBoardSocket(dai.CameraBoardSocket.CAM_D)  # IMX378
                
                     cam_a.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)    
                     cam_d.setResolution(dai.ColorCameraProperties.SensorResolution.THE_1080_P)
                
                     cam_a.setFps(fps)
                     cam_d.setFps(fps)
                
                     cam_a.setIspScale(2, 3)
                     cam_d.setIspScale(2, 3)
                
                     StereoConfig(stereo)
                
                     # Set output Xlink
                     outcam_a = pipeline.create(dai.node.XLinkOut)
                     outcam_d = pipeline.create(dai.node.XLinkOut)
                     outstereo = pipeline.create(dai.node.XLinkOut)
                
                     outcam_a.setStreamName("cam_a")
                     outcam_d.setStreamName("cam_d")
                     outstereo.setStreamName("stereo")
                
                     if enableRectified:
                          xoutl = pipeline.create(dai.node.XLinkOut)        
                          xoutr = pipeline.create(dai.node.XLinkOut)        
                          xoutl.setStreamName("rectifiedLeft")        
                          xoutr.setStreamName("rectifiedRight")
                
                      # LINKING
                      cam_a.isp.link(outcam_a.input)
                      cam_d.isp.link(outcam_d.input)
                      cam_a.isp.link(stereo.right)
                      cam_d.isp.link(stereo.left)
                      stereo.disparity.link(outstereo.input)
                
                      if enableRectified:        
                            stereo.rectifiedLeft.link(xoutl.input)        
                            stereo.rectifiedRight.link(xoutr.input)        
                            stereo.depth.link(depthQueue.input)    
                
                      # Pipeline is defined, now we can connect to the device    
                      with device:        
                            device.startPipeline(pipeline)
                            CAM_a = device.getOutputQueue(name="cam_a", maxSize=1)        
                            stereo1 = device.getOutputQueue(name="stereo", maxSize=1)                
                
                            while True:
                                  Framecam1a = getFrame(CAM_a)
                                  Framestereo = getFrame(stereo1)
                
                                  # Colormap disparity for display
                                  Framestereo = (Framestereo \* disparityMultiplier).astype(np.uint8)            
                                  Framestereo = cv2.applyColorMap(Framestereo, cv2.COLORMAP_JET)                        
                
                                  # SHOW IMAGES            
                                  cv2.imshow("Framecam1a", Framecam1a)            
                                  cv2.imshow("Framestereo", Framestereo)
                
                                  # Check for keyboard input
                                  key = cv2.waitKey(1)
                                  if key == ord('q'):
                                        break

                I would really appreciate it if you could help me solve these problems. Thank you