I just realised, that this setting is working only for the main products page.
Is it possible to make this setting for displaying products in categories as well?