Updated Oct 2021
There are a number of ways to achieve portrait video and photo capture with RightBooth.
1 Physically rotate and mount the camera at 90 degrees and also rotate the camera input by 90 degrees in RightBooth
This is explained in the following tutorial:
The above tutorial discusses using a webcam, but the same principle also applies when using a DSLR camera. You can physically mount the DSLR camera at 90 degrees, then in RightBooth Camera DSLR settings you need to rotate the input by 90 degrees. Make sure the video recording W and H values are set to the video landscape values of the camera and also the 'Reduce photo size' W and H values are set to the photo landscape aspect ratio size of the camera. In the following example I am using a Canon 1100D set to capture photos at S1 quality, which is 2256x1504 pixels, and this is an aspect ratio of 1.5:1. So I have the photo reduction also set to 1.5:1, namely: W = 1500, H = 1000. I have also set the video W and H to be the same as the camera's video recording size 1920x1080. Finally, I tick the Rotate 90 checkbox and RightBooth will now ensure that the camera input appears in Portrait mode at the correct aspect ratio:
2 Mount the camera horizontally (un-rotated) and crop the input in RightBooth
Using this solution RightBooth will crop the landscape camera feed (webcam or DSLR), discarding the left and right sections of the input to leave a central portion that is any aspect ratio you require, such as square or portrait. Obviously this solution is not as good as the first solution because you are discarding useful sections of the camera input and you are therefore effectively reducing the resolution in your captured files.
In this example I have chosen to crop both DSLR video and photo inputs. I have cropped the video to 604x1080, which is an aspect ratio that is close to a rotated HD input. With photos I have cropped to 700x1000 which is providing a slightly wider Portrait input, as seen in the live view:
It is worth mentioning here that there are no hard and fast rules on cropping. Simply choose values that produce video and photos aspect ratios that work for you.