Infrared small-target detection has always been an important technology in infrared tracking systems. The current infrared approaches for small-target detection in complex backgrounds are prone to generating false alarms and exhibit sluggish detection speeds from the perspective of the human visual system. Using the multiscale local contrast measure using a local energy factor (MLCM-LEF) method, an infrared small-target detection method based on a double-layer local energy factor is proposed. The target detection was performed from the perspectives of the local energy difference and local brightness difference. The double-layer local energy factor was used to describe the difference between the small target and the background from the energy perspective, and the weighted luminance difference factor was used to detect the target from the brightness angle. The infrared small target was extracted by a two-dimensional Gaussian fusion of the processing results of the two approaches. Finally, the image mean and standard deviation were used for adaptive threshold segmentation to extract the small infrared target. In experimental tests on public datasets, this method improved the performance in suppressing background compared with the MLCM-LEF algorithm, DLEF (double-layer local energy factor) reduced the detection of a single frame time by one-third.