iOS SDK >
Augmented Video

Welcome to the Augmented video tutorial. Using the complete Xcode project that you can download from this link, we are now going to add an augmented video to a marker.
Note: Our SDK is designed for use with Xcode (versions 8.2+) and supports iOS SDK 10.2+. The code below is compliant with the Swift 4.0 programming language. 

An important thing to know for people who wish to use the demo license, is that since you are limited to one bundle identifier for your apps (com.pikkart.trial) it is advisable to delete any app created with the Pikkart SDK before installing a new one.

First we need to copy some premade base classes from our sample package. 

Copy the two bundles from folder <pikkart_sample_dir>/sample into the folder <your_Xcode_Project_root>/<app_name>/VideoRendering/ as shown here:

The PikkartVideoPlayer class is a class encapsulating an AVPlayer and an OpenGL rendering surface to which video frames are redirected. This is the main class managing our video/audio data. The VideoMesh class is our 3D Object (a simple plane with our video as texture) that will be drawn on top of the tracked image. 

Let's see some interesting bits from ViewController:

public class ViewController:PKTViewController {
[..]
var _videoMesh:VideoMesh?; 
var context:EAGLContext:?
override func viewDidLoad() {
    EAGLContext.setCurrent(self.context)
    gl.glClearColor(1.0f, 1.0f, 1.0f, 1.0f);
    monkeyMesh=new Mesh();
    monkeyMesh.InitMesh(context.getAssets(),"media/monkey.json", "media/texture.png");
    _videoMesh = VideoMesh(parentCtrl: self)
    let mediaBundlePath=Bundle.main.bundlePath+"/media.bundle"
    let mediaBundle=Bundle(path: mediaBundlePath)
    _videoMesh!.InitMesh(mediaBundle!.path(forResource: "pikkart_video", ofType: "mp4")!, keyFrameUrl: mediaBundle!.path(forResource: "pikkart_keyframe", ofType: "png")!, seekPosition: 0, autostart: false, videoPlayer: nil) 
    glClearColor(1.0, 1.0, 1.0, 1.0)
}

The VideoMesh is created and initalized by passing to it the URL of the video file (can be a locally stored file or a web URL), the path to a keyframe image that will be shown to the user when the video is not playing (path to a locally stored image, in this case inside the media bundle) a starting seek position (in milliseconds), and whether the video has to autostart upon first detection or not. You can also pass a pointer to an external PikkartVideoPlayer object if you don't want the VideoMesh object to use its internal one.

In the ViewController listing we also have a couple more support functions that modify and rotate the projection matrix depending on device screen orientation. We originally saw them in the Cloud Planar Marker tutorial.

 internal func computeModelViewProjectionMatrix( _ mvMatrix: inout [Float], pMatrix:[Float]) -> Bool
 internal func computeProjectionMatrix(_ mvpMatrix:[Float]) -> Bool

Again, in the ViewController class the glkView method is tasked with drawing the video mesh whenever a specific marker is recognized:

  override func glkView(_ view: GLKView, drawIn rect: CGRect) {
  ...
  if self.isTracking() {
            if let currentMarker = self.getCurrentMarker() {
                if (currentMarker.markerId! == "3_543") {
                    var mvMatrix:[Float] = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
                    var pMatrix:[Float] = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
                    if (self.computeModelViewProjectionMatrix(&mvMatrix, pMatrix: pMatrix))
                    {
                    
                        _videoMesh!.DrawMesh(mvMatrix, projection: pMatrix)
                        RenderUtils.checkGLError()
                    }
                } 
            }
        }
   }
 
We also have two support functions that will either play or pause the video. These will be used  to respond to user interaction and app state change (viewWillDisappear mainly)

    internal func playOrPauseVideo() { _videoMesh!.playOrPauseVideo() } 
    internal func pauseVideo() { _videoMesh!.pauseVideo() } 

Please note how the  UITapGestureRecognizer object tracks touches on screen and is used to play/pause video (from the viewDidLoad method):

    let playVideoGesture = UITapGestureRecognizer(target:self, action:#selector(self.PlayOrPauseVideoAction(_:)))
    self.view.addGestureRecognizer(playVideoGesture)

And that's all folks for this tutorial. Once compiled and run on a device you should see something like this when targeting one of our test markers:

If you want more details on how our VideoMesh is made and how our AR video playback works take a look at the code and read the comments of our VideoMesh and PikkartVideoPlayerclasses.