Automated garment defect detection has faced serious challenges in today's textile manufacturing industry, especially when dealing with subtle anomalies, varieties of defect types, and fast inspection times. Addressing these complexities, the present work proposes a novel hybrid framework that effectively integrates Reinforcement Learning (RL) with high-end visual differentiation methodologies. Using the benchmark DAGM Class10 dataset [25], our framework synergizes robust image feature extraction using a Convolutional Neural Network (CNN) with an Adaptive Visual Attention Mechanism, driven dynamically by an RL agent to intelligently explore image regions of interest. The RL agent thus seeks, through a process of trial and error, to optimize the path taken in inspection, rewarding those strategies that can identify subtle defects and minimize false positives quickly. Unlike other supervised models, this agent-driven exploration greatly enhances adaptive capability, allowing for fast detection of complex textures and minimally distinguishable defects. A proposed method has also been shown to be significantly superior in real-time performance and is thus suitable for deployment on edge-computing platforms in industrial settings. This study not only adds to the various techniques in garment inspection but also sets a milestone, the first combination of reinforcement learning with visual quality control, paving the road for innovative self-optimizing inspection systems within textile manufacturing.