Pigs’ aggressive behavior represents one of the common issues that occur inside pigpens and which harm pigs’ health and welfare, resulting in a financial burden to farmers. Continuously monitoring several pigs for 24 hours to identify those behaviors manually is a very difficult task for pig caretakers. In this study, we propose a lightweight video-based approach for monitoring pigs’ aggressive behavior that can be implemented even in small-scale farms. The proposed system receives sequences of frames extracted from an RGB video stream containing pigs and uses MnasNet with a DM value of 0.5 to extract image features from pigs’ ROI identified by predefined annotations. These extracted features are then forwarded to a lightweight LSTM to learn temporal features and perform behavior recognition. The experimental results show that our proposed model achieved 0.92 in recall and F1-score with an execution time of 118.16 ms/sequence.