I recently encountered a problem with screen tearing – or so called "flickering" – in OBS (Open Broadcaster Software), which is likely connected to a setting for the graphics card. This happened both in the preview and in the recorded video, but only seems to happen when using the XSHM Screen Capture, and not when using the window recorder.
My first step in solving a problem like this is to Google the problem. I searched both for flickering on Ubuntu, and flickering generally. Presumably problems like this is not just limited to whatever OS or graphics card we are using, but the cause may still be the same, and could be software related.
However, to my surprise, there's a lot of people reporting the problem, but very few solutions. None of which worked in my case. I am aware that some of these problems had solutions, and were likely not related to this one at all. So, if your flickering also occurs while on the desktop, you should read this carefully, regardless of what OS or graphics card you are on. If you are technical enough, it might help you solve the problem in your specific circumstances.
I do a lot of desktop recordings when making tutorials, and while this does not ruin the information presented in my videos, it does somewhat mess up the quality of the videos. So, based on past experiences, and gut feeling, I suspected the graphics card drivers.
The problem with tearing is similar to what you might experience in some games, such as Dungeon Keeper 2 on modern systems. Many problems with flickering is most likely caused by the same issues, yet it seems very few people have actually found solutions.
The solution to flickering in OBS
Note. I do not know how to change settings for AMD cards, but you can likely solve the problem by doing similar fiddling with the settings for AMD/ATI, and I suspect the problem is software related.
At first, I also suspected V-sync (sync to vblank), however disabling that seemed to have no influence on the problem, and I am also using the proprietary drivers.
Symptoms: In Ubuntu 16.04, and perhaps on other operating systems, one might encounter screen tearing / flickering when windows are moved, if they change focus, and when using screen capture in OBS. Moving the mouse fast, or hovering icons in unity seem to especially trigger the problem.
Solution: Open "Nvidia X Server Settings" from Dash. In "X Server Display Configuration" check "Force Composition Pibeline" and "Force Full Composition Pipeline". Doing this gets rid of the flickering in OBS.
It may be related to the desktop environment in Ubuntu, as well as the way that some games and programs is displaying their HUD (Heads Up Display) and GUIs. If you search around, this problem appears to be caused by Xorg.
People have reported that enabling Full Composition Pipeline also fixes other problems issues, including tearing in Firefox, and while watching movies.
The Nvidia readme states the following about composition pipeline:
The NVIDIA X driver can use a composition pipeline to apply X screen transformations and rotations. "ForceCompositionPipeline" can be used to force the use of this pipeline, even when no transformations or rotations are applied to the screen.
Flickering doing full-screen capturing
According developers to the OBS forums, using screen capture for full-screen applications can actually cause screen flickering.
We should instead setup a window capture, and then specifically target the window of the game we want to record, which should then get rid of flickering doing recording and streaming.