This study proposes a novel model-based automatic search algorithm to realize the self-calibration of nonlinear signal model for angular position sensors. In some high-precision angular position sensors, nonlinearity of the signal model is the main source of errors and cannot be handled effectively. By constructing a signal flow network framework and by embedding a modeling search network, the parameters of the nonlinear signal model can be searched, and the calibration signal can be obtained. The convergence of the network search process was analyzed. The relationship between the optimization threshold and the convergence accuracy was also studied in simulations. Compared with the maximum angular error reduction to 47.42% after the calibration with simplified model that ignores signal nonlinearities, the proposed scheme was able to reduce this error to 0.0025% in simulations. By implementing the technique in a capacitive angular position sensor, the experimental results showed that the maximum angular error was reduced to 1.63% compared to a reduction of 86.02% achieved with the simplified model calibration. The effects of the search network order and layer number on the calibration accuracy were also analyzed, and the optimal parameters under experimental conditions were obtained. Correspondingly, the proposed scheme is able to handle calibration of nonlinear signal model and further improve sensor accuracy.