0

I am trying to record a video from the camera, through AVCapture and AVAssetWriter, and to overlay some text and an image on the final video but I cannot achieve good performance and the device thermal throttles quite fast.

In order to do so I get the buffer from

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {

if output == videoDataOutput && self.videoInput.isReadyForMoreMediaData && self.isSessionStarted {

guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
    return
}
...

// i write the new buffer

var newimageBuffer = overlayFunction(image: self.mp, toBuffer: imageBuffer)
let newsb = try CMSampleBuffer(imageBuffer: newimageBuffer, formatDescription: sampleBuffer.formatDescription!, sampleTiming: newsampletiming)

// and i append it
self.videoInput.append(newsb)
...

}

}

And I draw on it through this function

func overlayFunction(image mp:UIImage?, toBuffer pixelBuffer: CVImageBuffer) -> CVImageBuffer{

guard
            CVPixelBufferLockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0)) == kCVReturnSuccess,
        let context = CGContext(data: CVPixelBufferGetBaseAddress(pixelBuffer),
                                width: CVPixelBufferGetWidth(pixelBuffer),
                                height: CVPixelBufferGetHeight(pixelBuffer),
                                bitsPerComponent: 8,
                                bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer),
                                space: CGColorSpaceCreateDeviceRGB(),
                                bitmapInfo: CGBitmapInfo(rawValue: CGBitmapInfo.byteOrder32Little.rawValue|CGImageAlphaInfo.premultipliedFirst.rawValue).rawValue)
else{
  return pixelBuffer
}

UIGraphicsPushContext(context)
context.scaleBy(x: 1.0, y:-1.0)
context.setFillColor(CGColor(red: 255, green: 255, blue: 255, alpha: 255))

mp.draw(in: CGRect(...), blendMode: CGBlendMode.normal, alpha: 1)

context.scaleBy(x: 1.0, y: 1.0)
        
let attributes : [ NSAttributedString.Key : Any ] = [.font : UIFont(name: "Helvetica", size: 36.0), .foregroundColor : UIColor(red: 255, green: 255, blue: 255, alpha: 0.8), .strokeWidth : -5]
var stringWithAttributes = NSAttributedString(string: str, attributes: attributes)
stringWithAttributes.draw(at: CGPoint(...))

UIGraphicsPopContext()

CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer

}

However the performance is terrible. The resulting video lags a lot. What could I do in order to improve its speed?

Thanks for any help!

1
  • Have you tried using instruments to find out where the time is spent? E.g. re-creating an NSAttributedString in every drawing cycle looks like a possible issue to me. Try to move as much code as you can out of the overlayFunction and use cached variables instead. If possible, you could also try prerendering the attributed string and UIImage into a single image and only draw that?
    – fabianm
    Commented Jul 12 at 10:16

0