Computer vision tutorial for pygame.camera newbies - Nirav Patel

Now that I know how to capture images using pygame.camera, I am moving on the computer vision for newbies.  I read the following thing and found too many technical jargons.  I guess I should first read colour basics, then tutorials of the pygame.transformation, pygame.threshold etc.

Perhaps I should also read the following or other articles about v4l, uvc etc

https://help.ubuntu.com/community/Webcam

I call it a day.


Camera Module Introduction by Nirav Patel nrp@eclecti.cc Revision 1.0, May 25th, 2009

http://www.pygame.org/docs/tut/camera/CameraIntro.html

Import and Init

...

Capturing a Single Image

...

Capturing a Live Stream

...

Basic Computer Vision

By using the camera, transform, and mask modules, pygame can do some basic computer vision.

Colorspaces

When initializing a camera, colorspace is an optional parameter, with 'RGB', 'YUV', and 'HSV' as the possible choices. YUV and HSV are both generally more useful for computer vision than RGB, and allow you to more easily threshold by color, something we will look at later in the tutorial.

    self.cam = pygame.camera.Camera(self.clist[0], self.size, "RGB")
 
    self.cam = pygame.camera.Camera(self.clist[0], self.size, "YUV")
 
    self.cam = pygame.camera.Camera(self.clist[0], self.size, "HSV")
 
Thresholding

Using the threshold() function from the transform module, one can do simple green screen like effects, or isolate specifically colored objects in a scene. In the below example, we threshold out just the green tree and make the rest of the image black. Check the reference documentation for details on the threshold function.

    self.thresholded = pygame.surface.Surface(self.size, 0, self.display)

    self.snapshot = self.cam.get_image(self.snapshot)

    pygame.transform.threshold(self.thresholded,self.snapshot,(0,255,0),(90,170,170),(0,0,0),2)
 
Of course, this is only useful if you already know the exact color of the object you are looking for. To get around this and make thresholding usable in the real world, we need to add a calibration stage where we identify the color of an object and use it to threshold against.

We will be using the average_color() function of the transform module to do this. Below is an example calibration function that you could loop until an event like a key press, and an image of what it would look like. The color inside the box will be the one that is used for the threshold. Note that we are using the HSV colorspace in the below images.

    def calibrate(self):
        # capture the image
        self.snapshot = self.cam.get_image(self.snapshot)
        # blit it to the display surface
        self.display.blit(self.snapshot, (0,0))
        # make a rect in the middle of the screen
        crect = pygame.draw.rect(self.display, (255,0,0), (145,105,30,30), 4)
        # get the average color of the area inside the rect
        self.ccolor = pygame.transform.average_color(self.snapshot, crect)
        # fill the upper left corner with that color
        self.display.fill(self.ccolor, (0,0,50,50))
        pygame.display.flip()
 
    pygame.transform.threshold(self.thresholded,self.snapshot,self.ccolor,(30,30,30),(0,0,0),2)
 
You can use the same idea to do a simple green screen/blue screen, by first getting a background image and then thresholding against it. The below example just has the camera pointed at a blank white wall in HSV colorspace.

    def calibrate(self):
        # capture a bunch of background images
        bg = []
        for i in range(0,5):
          bg.append(self.cam.get_image(self.background))
        # average them down to one to get rid of some noise
        pygame.transform.average_surfaces(bg,self.background)
        # blit it to the display surface
        self.display.blit(self.background, (0,0))
        pygame.display.flip()
 
    pygame.transform.threshold(self.thresholded,self.snapshot,(0,255,0),(30,30,30),(0,0,0),1,self.background)
 
Using the Mask Module

The stuff above is great if you just want to display images, but with the mask module, you can also use a camera as an input device for a game. For example, going back to the example of thresholding out a specific object, we can find the position of that object and use it to control an on screen object.

    def get_and_flip(self):
        self.snapshot = self.cam.get_image(self.snapshot)
        # threshold against the color we got before
        mask = pygame.mask.from_threshold(self.snapshot, self.ccolor, (30, 30, 30))
        self.display.blit(self.snapshot,(0,0))
        # keep only the largest blob of that color
        connected = mask.connected_component()
        # make sure the blob is big enough that it isn't just noise
        if mask.count() > 100:
            # find the center of the blob
            coord = mask.centroid()
            # draw a circle with size variable on the size of the blob
            pygame.draw.circle(self.display, (0,255,0), coord, max(min(50,mask.count()/400),5))
        pygame.display.flip()
 
This is just the most basic example. You can track multiple different colored blobs, find the outlines of objects, have collision detection between real life and in game objects, get the angle of an object to allow for even finer control, and more. Have fun!

.END