I understand the hardware does not support that. But can we support it by our DepthAI pipeline by doing following things?
Make sense?
Hi erik
Most of disparity error come from the mismatching of the correspondent pixels from left and right due to noise, repetitive texture or lack of texture, occlusion, etc.
If we already know certain disparity range smaller than the default search range for a given pixel, then we can narrow the search range thus less susceptible to noise, repetitive texture, etc. Make sense?
Before requesting support, please see: https://docs.luxonis.com/en/latest/pages/support/
I wonder if the current DepthAI can support "Range Disparity".
A Range Disparity is defined as: Given a range of disparity (d1,d2) of a pixel, the disparity computation will output the best match disparity within (d1, d2). That is the "Range Disparity" would only search the lowest cost disparity in the range of (d1, d2).
So let's define D1 and D2 map for left image. That is, D1 is same dimension as left image with different d1 value for each pixel. Likewise, D2 map has d2 value for each pixel.
Disparity = RangeDisparity(left, right, inputConfig, D1, D2)
The trivial case which the current DepthAI already support is the constant range for each pixel say from 0, to 96..
But for RangeDisparity, each pixel can have different (d1, d2) range.
Please let me know if you have any questions regarding the definition of Range Disparity.
Thank you for your help.
Thank you very much for your information. Disparity shift is a very interesting and useful concept for an up-close object depth detection. However, it's not quite the same as image shift.
The use case I am trying to do is to use stereoDepth disparity computation to emulate "optical flow" computation:
monoRight(n)=> {imgFrame[:,0:width-s]=imgFrame[:,s:width]} => stereo.right
monoRight(n-1) =======================================> stereo.leftBy feeding monoRight frame n and monoRight previous frame n-1, I plan to get the optical flow using disparity computation. The way to deal with negative direction is to provide an image shift of 95/2 ~ 48. In this case, if we get disparity of 49 which will correspond to optical flow +1, and disparity of 47 correspond to optical flow -1. The Disparity shift won't be able to do the job. Because it is missing the "disparity" values between 0..47 which are "optical flow" -48..-1.
I already implemented the above pipeline in a stereo_by_host.py because I can shift the frame in host. Now if we want to put this pipeline in the device, the missing node is the image shift node.
Make sense? Please advise. Thanks a lot for your help again.
Hi,
I understand the Manip can do crop, flip, etc operation.
I wonder if there is equivalent operation of, for example, shift 48 pixels to left without cropping and maintain the original image size (for example, 640x400) after shifting. I know we might use manip crop then resize, but that would destroy the pixel position relationship, thus won't be acceptable.
So what I need is a node that can shift +/-s pixels of an imgFrame without changing the frame dimensions. For example, this node would be used in the following pipeline application:
monoRight => {imgFrame[:,0:width-s]=imgFrame[:,s:width]} => stereo.right
monoLeft =======================================> stereo.leftIs there a way to construct the above pipeline using existing depthai node? If not, could you please point me to a reference/code example on how to construct a custom image shift node (similar to such as edge detector node or feature tracker node) by a user?
Please advise. Thanks a lot for your help.
erik for such cropping you should use ImageManip node. Thoughts?
Nice! Thanks a lot for your help!
After tested it, I can crop it with manip but the resulting 592x400 imgFrame cannot be accepted by stereoDepth and generate the following error:
[184430106183EB0F00] [1.11] [2.314] [system] [critical] Fatal error. Please report to developers. Log: 'Fatal error on MSS CPU: trap: 2A, address: 8009F440' '0'
Any thoughts, Thanks.
Hi @erik
Thank you for your help.
Indeed the code is working!
However, I would like to shift the buffer by 48 pixels before sending it out as below:
script = pipeline.create(dai.node.Script)
script.setScript("""
width,height=640,400
imgs = [ImgFrame(height*width),ImgFrame(height*width)]
for i in range(2):
imgs[i].setWidth(width)
imgs[i].setHeight(height)
i=0
while True:
imgs[i]=node.io['in'].get()
imgs[i][:,0:width-48] = imgs[i][:,48:width]
o = (i+1)%2
node.io['out'].send(imgs[o])
i = o
""")
monoRight.out.link(script.inputs['in'])Then I encounter the following error message:
[184430106183EB0F00] [1.11] [1.487] [Script(3)] [critical] TypeError: 'lpb.ImgFrame' object is not subscriptable
I did try use imgs.getFrame() but fail too.
Please advise how to get ImgFrame in 2D shape and do frame[:,0:width-48]=frame[:,48:width] shift and then save it back into imgs again.
Thank you very much for your help.
Hi @erik
I understand the normal path for stereoDepth is following:
monoleft/right => rectification => disparity computation
I wonder if there is a way to configure stereoDepth to perform disparity computation without doing retification:
monoleft/right => disparity computation
If yes, how? Please advise. Thanks a lot for your help.
Hi @erik
Thank you for the information.
The following is the code I plan to test, please take a look if there is an obvious blunder.
script = pipeline.create(dai.node.Script)
script.setScript("""
imgs = [ImgFrame(400*640),ImgFrame(400*640)]
i=0
while True:
imgs[i]=node.io['in'].get()
o = (i+1)%2
node.io['out'].send(imgs[o])
i = o
""")As you can see, I tried to create 2 img buffers (imgs) as a cyclic buffer. So the while loop just to get the current img and send the previous img.
The input of this script node may be from, says, a monoRight node as below:
`monoRight = pipeline.create(dai.node.MonoCamera)`Do I need any clocking information to sync up the frame? or the node.io[].get() will automatically sync up the frame from the source?
In another words, would the following connection give me frame(n) and frame(n-1) to the sink node at the time n?
monoRight.out.link(sink.input1)
script.outputs['out'].link(sink.input2)Please advise. Thanks.
Hi
I would like to use a node to process frame(n) and frame(n-1) in the pipeline in real time.
How do I buffer a frame in the pipeline to sync up with next frame to achieve the above goal?
Thank you for your help.
Hi @erik
I tried to use -debug option to run stereo_depth_from_host.py and try to see the lrcheck=data.getData().
It seems that lrcheck has 2 elements for each pixel: for example if an image.shape = (h,w), then the lrcheck.shape = (h,w,2).
Could you provide definition for those 2 elements for each pixel in lrcheck?
For example: image.shape => (540,960)
after line #707 data = q.get()
lrcheck = data.getData()
lrcheck.shape = (1036800,)
lrcheck=lrcheck.reshape((540,960,2))
print(lrcheck[145,119])then I will get:
array([154, 1], dtype=uint8)Thank you for your help.
Very nice! It works now.
One more question: in order to correspond to the image left/right pixel position of resolution 1280x800, (image_left/right.shape => (800,1280) row index first format), what reshape order should I use (800,1280,96)? or (1280,800,96)?
That is which is correct?
framedata = image.getData().reshape((800,1280,96))
or
framedata = image.getData().reshape((1280,800,96))
please confirm. Thanks a lot!
"You can uncomment this line to get widthheight96 bytes disparity cost dump:
https://github.com/luxonis/depthai-python/blob/main/examples/StereoDepth/stereo_depth_from_host.py#L605"
Thank you for the recommendation. Actually I already did that and I examed the disparity_cost_dump.raw only has 1 16-bit integer for each pixel not 96 bytes!
Any other suggestion to dump 96 bytes of cost values for each pixel?
Thank you very much for your help.
Hi @erik
Thanks for the link.
I tried it and save the disparity_cost_dump.raw.
But it seems only save one 16-bit cost value for each pixel not 96 byte for each pixel.
I looked at the code at line 16:
parser.add_argument('-dumpdispcost', "--dumpdisparitycostvalues", action="store_true", help="Dumps the disparity cost values for each disparity range. 96 byte for each pixel.")
We should expect to have a dump of 96 cost values for each pixel, is that right?
What else of option I need to turn on to dump the 96 cost values for each pixel?
Thanks a lot for your help again.
@erik
If I want to study how the disparity cost is computed and also would like to output disparity cost vector (i.e. each pixel will have a vector of 95 cost for it's neighbors), where can I find the source code of the implmentation of the disparity cost formula? Is it possible to change the formula and output disparity cost vector map for debug/development? Thank you very much for your help.
stephansturges
could you share your captured left/right images of this "repeative texture" case?
I would like to try my algorithms to see if it can alleviate the "blue" area (no depth).
Thanks.
erik
Thanks a lot for your help.
I go back to look at my error message again:
[184430106183EB0F00] [400.856] [NeuralNetwork(7)] [critical] Fatal error in openvino '2021.4'. Likely because the model was compiled for different openvino version. If you want to select an explicit openvino version use: setOpenVINOVersion while creating pipeline. If error persists please report to developers. Log: 'softMaxNClasses' '157'
And did not see what you see at the end 'CMX memory is not enough!'
Just curious what's the different between your platform compared with mine. If I can see that message early, probably wouldn't need to go this far to find out ; ))
By the way, would RVP3 (KeemBay) version have bigger memory to run this model?
Thank you for your help along the way....