Poor Accuracy When Evaluating the Model on My Own Dataset

[amit154154]

I'm attempting to evaluate the model using a custom dataset containing both real and fake videos (MP4 files with audio). I've closely followed the preprocessing instructions provided in this repository, but the model consistently classifies nearly all my real videos as fake.

⚙️ My Preprocessing Steps

1. Converted videos to exactly 25 FPS:

ffmpeg -y -i input.mp4 -r 25 -c:v libx264 -preset fast -crf 23 -c:a aac output.mp4

2. Extracted mono WAV audio at 16 kHz from converted videos:

ffmpeg -y -i video.mp4 -vn -ac 1 -ar 16000 audio.wav

3. Cropped videos to 224×224 face-centered images using MediaPipe (scale=1.2):

import cv2, mediapipe as mp

face_detection = mp.solutions.face_detection.FaceDetection(
    model_selection=0, min_detection_confidence=0.5
)

def crop_video(input_video, output_video, scale=1.2, size=224):
    # OpenCV and MediaPipe face-centered cropping logic
    pass  # simplified for brevity

4. Ran your provided preprocess.py script to generate concatenated frames and mel-spectrograms.

📉 Evaluation Results

• Accuracy: ~31%
• False Positive Rate (real videos predicted as fake): ~99%

Essentially, nearly every real video from my dataset is misclassified as fake.

✅ What I've Verified / Tried

• ✅ All videos have FPS exactly 25.
• ✅ Extracted audio is exactly 16 kHz mono.
• ✅ Visually checked the face-cropped videos (224×224, centered) to ensure similarity to your provided dataset.
• ✅ Tested multiple cropping scales; scale=1.2 visually matched your dataset best, but all tested scales produced similarly poor results.
• ✅ Experimented with different audio sampling rates without improvements.

I suspect there's a subtle preprocessing mismatch or additional step I may have overlooked. Could you please:

• Confirm if there are any other preprocessing steps or parameters not clearly documented?
• Share a complete example code snippet demonstrating preprocessing an external MP4 video?

Below is an example image from a real video in my dataset after running the full preprocessing pipeline:

Thank you for your support and your excellent research!