nservicebus - How to interpret number of messages in Distributor's Storage and Control queues? -

this article tells control queues nsb master node uses control message load, though me it's still not clear how interpret disproportions in number of messages in queues: https://docs.particular.net/nservicebus/msmq/distributor/

i'm observing slowness in nsb service have never experienced slowness before. reason less parallel threads created per every master node comparing past time, , there have been no change in workers or master nodes configuration, max amount of threads allocate. i'm trying figure out if it's master node not want feed workers, or workers not want take more job.

i see amount of messages in control queue jumps 15 40, while storage has 5-8. should interpret workers ready work, while distributor can't send them more messages? thanks

the numbers in control , storage queue jump , down long distributor handing out messages. message coming control queue popped off queue , onto storage queue. message coming primary queue of distributor result in first message of storage queue popped off. it's hard interpret numbers of messages in queues of running distributor, because, time @ numbers computer management or queue explorer, have changed.

the extreme cases this:

1. no messages in primary input queue of distributor , no work happening on of workers.

• input queue: 0
• control queue: 0
• storage queue: number of workers*configured threads per worker

2. workers working @ full capacity. none able take on more work.

• input queue: 0+ (grows new messages comes in)
• control queue: 0
• storage queue: 0

in running system, can between these 2 extremes, so, unfortunately, it's hard snapshot of control , storage queue.

some troubleshooting tips:

• if storage queue empty, distributor can not hand out more work. not know send it. happens if workers occupied not sending ready-messages control queue until finish handling message.

• if storage queue consistently small compared total number of worker threads across workers, approaching total maximum capacity of workers.

• i suggest start looking @ logs of workers , see if work doing taking longer usual. slower database/third party integration?

• another thing check if there has been io-heavy added machine hosting distributor. if distributor running @ close max capacity, adding io might slow down msmq on box, giving worse throughput.