Aquatic ecosystems depend greatly on microscopic phytoplankton species as the primary producers and bio-indicators of water quality. Accurate detection and identification of phytoplankton species is a necessary step for monitoring their ecological risk and the following in-depth ecological and environmental studies and analyses, due to their significance and potential indication of environmental stress, eutrophication and habitat change, and HABs incidents. Yet, microscopic image-based phytoplankton species identification is labor-intensive and time-consuming. It requires expert taxonomic skills which make large-scale biodiversity screening an unthinkable task using traditional approaches. This paper proposes a novel automated system that integrates an Oriented YOLOv8-OBB object detection engine with a hybrid ConvNeXt–CBAM–Vision Transformer (ViT) classifier to tackle some fundamental challenges for microscopic phytoplankton species identification, including rotational-variation and small-object detection, background noise from other particles floating near the target organisms, and subtle morphological difference among species. For detection, an anchor-free oriented bounding box regression module was adopted to preserve the 360° rotational-invariance of the target organisms, minimizing errors of background inclusion of organisms rotating diagonally. After extracting the contour-integrating organism patches, a triple-hybrid deep learning classifier was deployed, by cascading a feature extractor ConvNeXt, an attention model CBAM, and a global relational aggregator ViT into the pipeline. The trained detection and classification models are validated over 23,352 microscopic images with various magnification factors, obtaining an accuracy of 94.2%, detection mean Average Precision (mAP) of 0.89, and test latency of 34.7 ms/image. Finally, a rule engine-based Ecological Risk Assessment (ERA) is devised to generate timely alarm by converting identified taxa to a Saprobic Index.