Breakdown

This page simply has a breakdwon of some of the code file names and what they all do. We understand that it can be a little intimidating to delve into large pools of unfamiliar code, so we hope this helps you get startd. We recommend sticking to the current code which uses a neural network, however, if you decide that the obsolete code could be refined to be more stable, that is an option as well. The obsolete code involves using the Haar cascade.

Current Code

record.py

POSSIBLY THE MOST IMPORTANT FILE

The existing header comments on this file are extremely helpful. However, here’s a walkthrough and some additional notes from this file. Keep in mind that this code was not written by us, so we can only provide a theoretical description of the code flow. Assuming the libraries used are intact, this description should be accurate

• This code starts by first loading up some sounds. These sounds are saved in the “sounds” folder as .wav files
• It starts by setting up connections to the camera (from the xiapi library) and the proximity sensor (which, based on past reports, didn’t fully function and might need more work)
• Note that this does use multithreading and thus, you will see “threading.Lock()” used, in order to avoid a race condition. If you’re unsure what this is, try looking up race conditions and mutex locks in multithreading
• Before we get into the meat of the code, it is important to provide a good description of the measure() function. This function
• Reads from the proximity sensor for a short duration
• Throws out a couple of dummy readings
• Takes the third value and decodes it
• Sends back decoded values
• Now let’s look at the main part of this code
• We start by playing a sound indicating that we are waiting to start recording.
• It will then wait for the sensor to detect a person. Note line 257 of the code for this. As soon as it detects a person, the recording starts, and a new sound is played. The recording lasts for 15 seconds
• After recording, the end sound is played
• Now, each thread heads on to process the recorded frames. The work is split evenly amongst them. This is where multithreading is important to ensure quick processing
• These threads (in the ProcessFrame function) use the tensorflow model to ensure that the measured data is indeed an eye. Note that the option for using the old cascade classifier still exists and changing the initialization macro at the beginning of the code can let you switch between the classifiers
• These frames are then downsampled and added to a buffer
• Once the data is stored in the buffer and we have ensured that the data is indeed an eye (line 286), another sound is played. The new video file is complied and outputted
• If the video was not an eye, it is instantly rejected

src/PublicEye Folder

To run, open the file within processing and press the “Run” button. Ensure that a video archive and visitor archive folders exist in a folder called “data/archive” and “data/visitor_movies” folder, respectively, and that the archive has at least 6 video clips. Uses four subfiles: DrawMovie.pde, Log.java, MovieManager.java, VisitorClip.java:

• The program begins by doing some setup and creating two watchdog threads
• The screen size is 480x96 pixels. This is the native resolution of the six panels that are being used for display
• One thread watches for new visitor clips (MovieManager.watchForNewVisitorClips())
• One thread watches for expired visitor clips (MovieManager.watchForExpiredClips())
• The program also checks that the video archive and visitor archive exists, in a “data/archive” and “data/visitor_movies” folder, respectively If any setup fails, the program exits
• initializeMovieDisplay() sets up the video clips in the correct screen-locations and renders the first frame of the animation, drawMovie() takes it from there
• drawMovie() is called every frame by the main thread. It creates a thread to load new movies when needed (to help prevent stuttering)
• Log.java logs different information based on the log level (FATAL, ERROR, WARN, INFO, DEBUG)
• MovieManager.java stores information about the archive and visitor clips. It also chooses a random archived (or visitor) clip when a new clip is needed This file contains two watchdog functions, watchForNewVisitorClips() and watchForExpiredClips() that are given their own threads to watch the video folders.
• VisitorClip.java has some utility functions for loading a new clip from a visitor

classify_image.py

As the name implies, this file is used to determine if an image contains an eye or not. It appears to mostly be taken directly from some TensorFlow library or another.

• load_graph() takes in a model file as input, which seems to simply be a file containing a string. It opens the file and reads it using graph_def.ParseFromString. This seems to be setting up a graph object in tensorflow (imported as tf), which will be used to operate on the image in classify_img()
• read_tensor_from_image_file() is creating a tensor object of the supplied image. It does some resizing of the image, and then executes the functions listed by the graph contained by tf on the image
• load_labels() simply reads all of the possible image labels listed in the target file and stores them in a label array
• classify_img() starts by creating a lot of objects, such as the name of the image file, the trained model to be used, the labels needed, and parameters of the image. It then calls load_graph() on the trained model’s file, read_tensor_from_image_file() on the image, and executes the graph input and output operations on what looks like a file path
• It then executes the image classification using tf.Session.run, and returns the label it gave the image (either eye or not eye)

tf_files folder

This contains the files for a tensorflow model. The important thing to note here is that this contains:

• A pre-trained model off the internet
• A re-trained model that simply does binary classification (eye vs not eye)

Use:

Note: this folder is referenced in the file classify_image.py

Classify_image.py has been imported by record.py

Essentially, here’s the flow of code

• Record.py:
• Get images and ask classify_image.py to see if it is an eye
• Classify_image.py
• Take the passed down image and open the tensorflow retrained model from the tf_files folder
• Use this model to check if its an eye and return results to record.py