CNN’s working with big datasets (Dogs VS Cats)
One of the greatest and coolest thing which I like about machine learning is Image vision or manipulating images. You can give an image input to the algorithm and it can create a filter (like Instagram/Snapchat) or you can detect the emotion of a person in an image. This all is done using CNN layer. CNN gives an extra edge. CNN reduces the size of the image while at the same time learning important features. Most of the application of CNNs is for image or video data but I have seen people using this on graphs (like spectrogram).
Today I will provide you a basic implementation of a neural network mostly using CNNs. I have used image generators which are a powerful tool provided by Keras which help us to process the image without loading it into memory. It pulls the image straight from the directory and we can manipulate it once loaded by image generator.
So lets start by loading all the packages
import os import zipfile import random import tensorflow as tf from tensorflow.keras.optimizers import RMSprop from tensorflow.keras.preprocessing.image import ImageDataGenerator from shutil import copyfile
Prepare the dataset
print(len(os.listdir('PetImages/Cat/'))) print(len(os.listdir('PetImages/Dog/'))) # Expected Output: # 12501 # 12501
We can see we have 12501 photos of Cats and 12501 of Dogs. This dataset is very large and loading whole 25002 photos in memory are computationally very expensive that is why we are using image generators. Image generators load the images only batch-wise and not the whole dataset at a time which takes less memory. Separate folders in data directory will be treated as labels. Further, we can distribute the training and testing set in the data directory by making training and testing folders within the label directories and keeping the images there. Let us start by creating the training and testing
os.mkdir('cats-v-dogs') os.mkdir('cats-v-dogs/training') os.mkdir('cats-v-dogs/testing') os.mkdir('cats-v-dogs/training/cats') os.mkdir('cats-v-dogs/training/dogs') os.mkdir('cats-v-dogs/testing/cats') os.mkdir('cats-v-dogs/testing/dogs')
Lets now split our images in 2 sets (testing and training). Lets create a function to do this.
def split_data(SOURCE, TRAINING, TESTING, SPLIT_SIZE): files =  for filename in os.listdir(SOURCE): file = SOURCE + filename if os.path.getsize(file) > 0: files.append(filename) else: print(filename + " is zero length, so ignoring.") training_length = int(len(files) * SPLIT_SIZE) testing_length = int(len(files) - training_length) shuffled_set = random.sample(files, len(files)) training_set = shuffled_set[0:training_length] testing_set = shuffled_set[-testing_length:] for filename in training_set: this_file = SOURCE + filename destination = TRAINING + filename copyfile(this_file, destination) for filename in testing_set: this_file = SOURCE + filename destination = TESTING + filename copyfile(this_file, destination) CAT_SOURCE_DIR = "PetImages/Cat/" TRAINING_CATS_DIR = "cats-v-dogs/training/cats/" TESTING_CATS_DIR = "cats-v-dogs/testing/cats/" DOG_SOURCE_DIR = "PetImages/Dog/" TRAINING_DOGS_DIR = "cats-v-dogs/training/dogs/" TESTING_DOGS_DIR = "cats-v-dogs/testing/dogs/" split_size = .9 split_data(CAT_SOURCE_DIR, TRAINING_CATS_DIR, TESTING_CATS_DIR, split_size) split_data(DOG_SOURCE_DIR, TRAINING_DOGS_DIR, TESTING_DOGS_DIR, split_size) # Expected output # 666.jpg is zero length, so ignoring # 11702.jpg is zero length, so ignoring
Now that we have created our training and testing datasets. Let us check how many files we have in those directories.
print(len(os.listdir('cats-v-dogs/training/cats/'))) print(len(os.listdir('cats-v-dogs/training/dogs/'))) print(len(os.listdir('cats-v-dogs/testing/cats/'))) print(len(os.listdir('cats-v-dogs/testing/dogs/'))) # Expected output: # 11250 # 11250 # 1250 # 1250
Build the model
The next step is to build a model. I have used keras to make that model.
model = tf.keras.models.Sequential([ tf.keras.layers.Conv2D(16, (3, 3), activation='relu', input_shape=(150, 150, 3)), tf.keras.layers.MaxPooling2D(2, 2), tf.keras.layers.Conv2D(32, (3, 3), activation='relu'), tf.keras.layers.MaxPooling2D(2, 2), tf.keras.layers.Conv2D(64, (3, 3), activation='relu'), tf.keras.layers.MaxPooling2D(2, 2), tf.keras.layers.Flatten(), tf.keras.layers.Dense(512, activation='relu'), tf.keras.layers.Dense(1, activation='sigmoid') ]) model.compile(optimizer=RMSprop(lr=0.001), loss='binary_crossentropy', metrics=['acc'])
I have made the above model. You can experiment by adding or removing various layer.
Train the model
We are using ImageGenerators which will pick the images from directories as needed by our model. Let’s start by making our ImageDataGenetators for training and testing dataset.
TRAINING_DIR = "/tmp/cats-v-dogs/training/" train_datagen = ImageDataGenerator(rescale=1.0/255.) train_generator = train_datagen.flow_from_directory(TRAINING_DIR, batch_size=100, class_mode='binary', target_size=(150, 150)) VALIDATION_DIR = "/tmp/cats-v-dogs/testing/" validation_datagen = ImageDataGenerator(rescale=1.0/255.) validation_generator = validation_datagen.flow_from_directory(VALIDATION_DIR, batch_size=100, class_mode='binary', target_size=(150, 150)) # Expected Output: # Found 22498 images belonging to 2 classes. # Found 2500 images belonging to 2 classes.
Now that we have created the genertors we can use them to train our model.
# Note that this may take some time. model.fit_generator(train_generator, epochs=50, verbose=1, validation_data=validation_generator)
Now we have trained our model lets check it on an image.
from keras.preprocessing import image # predicting images path = 'content/test.jpg' img = image.load_img(path, target_size=(150, 150)) x = image.img_to_array(img) x = np.expand_dims(x, axis=0) images = np.vstack([x]) classes = model.predict(images, batch_size=10) print(classes) if classes>0.5: print(fn + " is a dog") else: print(fn + " is a cat")
This codeblock will help you to check if the image has a dog or a cat.
In this post, we saw how can we use Convolutional Neural Networks to predict cats and dogs. Also, we learned how can we use image generators provide by Keras. We learned how we can handle large datasets without destroying our memory. We can use these concepts in transfer learning as well (maybe a future post). If you liked this you might also like PUBG – predicting win percentage and Emotion prediction.