LSTM Autoencoder vs. Isolation Forest vs. One-Class SVM for Financial Transaction Fraud Detection

Financial fraud detection under extreme class imbalance (fraud rate < 1%) motivates unsupervised or semi-supervised anomaly detection approaches. This study compares three paradigms—LSTM Autoencoder (LSTM-AE), Isolation Forest (IF), and One-Class SVM (OCSVM)—on 100,000 synthetic transactions with a 0.35% fraud rate and 30 features (28 PCA-transformed components plus scaled Amount and Time). Using a stratified 60/20/20 train-validation-test split, evaluation is primarily via AUC-PR, which is more informative than AUC-ROC under extreme imbalance. IF achieves the highest AUC-PR (0.4169), vastly outperforming both OCSVM (0.0616) and LSTM-AE (0.0171). AUC-ROC rankings are IF (0.9784) > OCSVM (0.9100) > LSTM-AE (0.8148). McNemar tests confirm all pairwise differences are statistically significant (p < 0.001). Both LSTM-AE and OCSVM exhibit near-random precision on the minority class, suggesting that only IF effectively exploits the isolation structure of PCA-transformed fraud patterns. The LSTM-AE's failure is attributed to PCA destroying sequential dependencies that recurrent architectures require.