Hi – I think you can do the following by adjusting the ‘Default Sort Order’
If you go in the WordPress dashboard under products, attributes, and click on the attribute you want to make changes to, you’ll see a setting for ‘Default Sort Order’, make the ‘Custom Sort Ordering’ then go to you terms within here and drag and drop them in the order as you need. This should do the trick.
If you want to hide it from the widget, this might not be possible unless there’s no products associated with the term. This is how Woocommerce attributes work as default.
Your other option is to use a products filter plugin. These plugins give you a bit more flexibility and let you control what you want to show and hide and take filtering a step further. The XStore theme thankfully works well with the ‘Woocommerce Products Filter’.
Hope this helps!