We tried using only brand and then only category driven by the menu string. Both gave us teh same behaviour. What we’re trying to achieve is a shop that only works for a particular category or brand. Then only show the brand or category filter options in the sidebar applicable to those displayed products. Are you saying this is not possible with your theme and plugins?
One thing we feel is a bad customer experience is loads of colour swatches are displayed that are not applicable to the current displayed products. Ideally we need to find a solution to this or not use the filter at all.