hello guys i have a problem at hand and a solution in mind however i would like to share my thoughts and get some feedback, i would appreciate it very much if you can share your thoughts. problem: I have three instances of a message bus (a messaging queue) in different parts of the world consuming messages. Messages are forwarded to these instances by a load balancer. Now currently all three instances are independent and do not communicate with one another. The problem arises when instance A of the messaging bus is expecting new messages, however they are forwarded to instance B. Instance A needs to simply know that the messages were forwarded to instance B instead of A so that it can change its behaviour accordingly. What would be the best approach to accomplish this? I am not looking for technologies specifically just ideas and thoughts on how this can be accomplished many thanks
Depends what kind of solution you're after. The point of a load balancer is to distribute work across systems that are equally capable of handling work. And the point of a messaging system is to encapsulate work into neatly divisible chunks. So the problem with your setup is that A is dependent on the message sent to B, which it shouldn't be. One solution is to modify the overall system so that while waiting for a message, a system is in state X, then after processing a message the system returns to exactly the same state, X. All systems waiting for a message should be in state X. Messages should contain a precise definition of the work that any system can handle regardless of which message goes to which system. The other solution is to kludge it so that the load balancer is capable of handling system specific messages, in such a way as to be able to send a message to the load balancer from the messaging source (or auto-generated by the load balancer) that says "To System A: Message M is being handled by System B", where M is the message you're asking about. To be honest this is really a nasty way of "fixing" the problem and at best should only be considered a horrible kludge until the real problem is solved, which is that your system design is broken. Alternatively, if the reason for asking is that A is connected to some specific hardware that B doesn't have, so only A is capable of handling message M, then the messages, message originator and "load balancer" (quoted because it's ceasing to be a true load balancer at this point) need modifying so that they are capable of handling system-specific messages.
Is there no way to assign sessions to each instance of the messaging queue? If you assign a session id with all messages going to A, or assign that at the start, you will ensure that all messages destined for A will continue to go there. You will take a performance hit, as you will now be load balancing at a coarser level (the session level rather than the message or request level) but it will make your life much simpler. As a side note, avoid a random or equalized load balancing solution. Measuring load on each of your messaging queues and assign work (or as I suggest, sessions) will be vastly more efficient. There was a high profile instance where a major ruby provider (heroku) moved over to randomly assigning load rather than measuring it and assigning based on which nodes has spare capacity.
