A nonlinear filter is the filter whose output is a nonlinear function of the input. By definition, any filter that is not a linear filter is a nonlinear filter. One practical reason to use nonlinear filters instead of linear filters is that linear filters may be too sensitive to a small fraction of anomalously large observations at the input.
One of the most commonly used nonlinear filters is the median filter.