This study investigates the randomness detection of cryptographic algorithms in network security. To effectively test and verify the security of ZigBee protocol in the Internet of Things, the study combines with the characteristics of ZigBee networks, and it rationally organizes and divides test modes based on the binary matrix rank theory test. Then this paper proposes a randomness detection method of ZigBee protocol in a wireless sensor network. The proposed method solves the one-sidedness that the binary matrix rank test simply assesses random sequences by linear correlation. The proposed assessment method can effectively appraise whether the ZigBee protocol has an encryption mechanism and encryption strength. Simulation results show that this method has the characteristics of fewer errors and high reliability.