Motivation Some time ago, we started looking for an event dispatching framework, with support for python users, that gives our applications the possibility of interconnecting each other. The reason behind this was to design a flexible system of applications that communicate and exchange information, instead of designing a monolithic application that is hard to debug and not very flexible in deployment. Scaling We also wanted to be able to scale the entire system easily, to add components without affecting other applications that do not require them and also to replace components with others that perform the same type of services, without refactoring the consumer of those services. Network Another hard requirement is to be able to interconnect these applications, even though they run on different workstations (or servers). Intra-process or Inter-Process communication was not satisfactory enough. The network support we had to design for this framework needed to be flexible enough to permit working in TCP/IP based networks, as well as other types of communication mediums (e.g. a SBC connected via a UART serial connection). Simplicity For our possible users and for ourselves as well, we've wanted to design an API that is very simple to use and understand and pretty much plug&play for our existing applications. Without any complex configuration or fancy mechanisms, the entire terminology of the library is based on common-sense. Solution The requirements above and not only, lead us to a single decision: we need a flexible and centralized event dispatching framework that decouples our applications and divides them in two categories: data providers and data consumers. Why designing something new We've looked up several implementations on the world wide web. We don't want to perform negative advertisement, so we won't make any side-by-side comparison with any of them, but we will present some advantages our design has over what we have found: Statefulness Most of the solutions we've found didn't provide any support to know if a certain data provider (or sender) is connected to the system, without requiring explicit support from that sender. We didn't want to add explicit functionality to our applications in order to inform the consumers that they are connected, we've wanted our framework to be able to handle that. Therefore, this framework is able to notify any interested parties, if any sender is connected to the system, via built-on events that will be distributed by the framework. Flexible distribution options Some of the frameworks we've found, only provided the concept of channels for publishing messages. From the perspective of event dispatching, the simple concept of a channel can be resumed like this: it receives messages from a plural of publishers and routes them to a list of registered listeners. The concept of channel was not satisfactory enough for our applications. From a design perspective, we've considered that they may create the tendency to be used for dispatching messages that don't share the same characteristics or format with all the others previously sent through the same channel and therefore it may create problems for listeners that are not prepared to receive them in other formats. In this case, listeners are forced to perform some sort of message validation before interpreting them to be sure they won't collapse if any other publisher sends some other type of message through the same channel, later on in the project. In order to avoid that you need to be defensive when you write the listener of a channel event and you have one more thing to worry about when designing your event distribution system. Targeted event distribution Excepting these plausible architectural problems, supporting only the channel concept could also create unwanted "noise" in a network of applications, because some listeners may receive some messages even though they are not interested of them, from publishers that reuse the channel to reach a sub-set of listeners that are already connected to it. Our pysig framework is designed to support channels as well, but the mainstream use is the targeted distribution. That is, when a sender registers to pysig it defines the list of events that will propagate over the system and listeners can register to all of them or just one (or more) of them, whatever is desired. Therefore, if a sender publishes a series of events, a listener has the option to receive only a specific event generated only by that specific sender. If another sender publishes the same event, the listener will not receive it, if it doesn't explicitly wants to. If a listener wants to connect to all the events generated by a specific sender, it has this option via broadcast events. Transport optimization This also implies a good optimization of network use, because listeners will only receive what they expect and nothing they have to ignore. The selection of what messages are distributed to listeners is made by the server, therefore no unwanted messages will be transported by the network. Channels As a single form of communication, channels may pose the problems listed above, but this doesn't mean they are not required nor useful. Targeted events cannot cover the need of listening for events published by a plural of senders. Let's assume you have a series of sensors connected to your network, that all need to publish their temperature and you also have listeners interested in finding out temperature measurements from all of them. Without channels you will be forced to register to each sensor in particular, which is not quite pleasant nor good for the scalability of the system. One thing this framework does differently in this regard, in order to avoid the traps mentioned before, is that when an event is sent over a channel and not only directly to its registered listeners, the sender must explicitly approve this by adding a prefix to the event name (the # character) in order to mark it as channel. The same applies for listeners, that need to explicitly avoid to specify no sender upon registration. If a listener desires to receive messages only from a specific sender, event though they are marked as channels, they can still do so by just specifying the sender upon registration. Code: # register a sender with a normal event and a channel event senderobj = router.addRemoteSender("mysender", ["event", "#channel_event"]) Built-in events are channels Please note that the built-in events of pysig framework, the connect and disconnect events, ar marked as channel events. That is because it may be useful to receive connect/disconnect events from senders you don't know about, for tracking purposes or else. If you are intersted in knowing when a sender connects in particular, you can still do so by registering just only to it's corresponding connect event. Code: # register a listener for a channel event router.addRemoteListener(callback_channel_event, None, "#channel_event") # register a listener for a specific event router.addRemoteListener(callback_event, "mysender", "event") # register a listener to know when mysender is connected router.addRemoteListener(callback_connect_mysender, "mysender", "#connect") # register a listener to know when any sender is connecter router.addRemoteListener(callback_connect_mysender, None, "#connect") Requests The distribution frameworks we've studied so far are more about dispatching messages constructed by specific senders to some listeners, than requesting data from senders at any point in time. In order to allow listeners to acquire data immediately, without waiting for a specific trigger the sender considers acceptable for publishing the event, the pysig framework supports the concept of requests. Using this feature, listeners can ask for specific data from senders, if they support to respond in this manner, the response being replied just to the listener in question. Code: # requests supported requests = { None: default_request_handler, "get_ticks" : get_ticks } # sender with support for requests senderobj = router.addRemoteSender("mysender", ["event"], requests= requests) # listener requesting something response = client.remoteRequest("mysender", "get_ticks", None) Transport flexibility Although our main target was to use TCP/IP networks for distributing messages, we've constructed the design flexible enough to permit other type of carriers in the near future. Currently pysig has built-in support for TCP transportation but users may add support for other types of networks or other types of mediums, without modifying the core or the API. The transportation and data packing, unpacking is decoupled from the framework implementation. Network outage support The current implementation for carrying messages over TCP is trivial to use and has the ability to handle network outages. The TCP client has support for automatically reconnecting to the server if a network outage occurs for the client, or the server is down for a period of time, allowing senders or listeners to easily redo their registrations once the network is ready to go. IOT The world of IOT was not forgotten. Framework is designed for interconnected applications but for interconnected devices as well. These may publish, listen or request data from each other which allows great flexibility for home automation systems where you need transparently scale the communication and the flow of data without modifying old behaviors. We've also planned to create projections of pysig framework in other programming languages as well (like C) to increase the number of possible use-cases based on the same concepts and architecture. Ready to use examples Most of the frameworks have plenty of examples and step-by-step tutorials, but few have ready to use examples that can make the entire system work with just a couple of command lines in an isolated environment. The examples provided within this framework, can run a pysig server, register listeners and senders with just a couple of command lines and requires only some intuitive configuration settings (like port, server address, sender name etc.). Diving in The following section will give you all the necessary details to start connecting your existing python applications to pysig. Installation The pysig framework is published on the python package repository and you can install it right away using pip: Documentation The documentation of pysig is prepared with step-by-step tutorials and examples, so for further study please visit it at http://pysig.rtfd.org. Example For this article, we will present an example of a sender that publishes a list of events and a listener that receives some of them. Starting the server Checkout the repository indicated by the documentation and run the generic_server.py application, stored withing the examples folder in the root of the repository. For the sake of exemplifying, the server comes with a pre-registered sender named timer which triggers an event called tick every 5 seconds (these identification names are customizable at startup). Writing the client In the same folder we also have two other examples name generic_client_sender.py and generic_client_sender.py that do what their names suggests: they register a remote sender and a remote listener to the pysig server. You can simply run these examples, to test the framework, but in this article we will rather do it step by step to explain each section in particular. Importing We need to import the library and the carrier we want to use. In this case the carrier will be a TCP client. Code: import sig from sig.carrier.tcpclient import * Creating the router The entity responsible for dispatching messages throughout the system is called router in pysig. There are three types of router: the server router, the client router and the local router. Local router is used only when no network connectivity is required. In the case of the local router, the application uses the framework just for dispatching messages within the application and no support is required for communicating those outside the process. Please note that pysig has built-in logging support, for tracing problems. You can still use the standard logging library in python for this matter, which you can pass via setup_logger. Code: if -_name__ == "__main__": # setup logging logger = sig.SimpleLogger(level= sig.LOG_LEVELS.LEVEL_INFO) sig.setup_logger(logger) # create tcp carrier tcpclient = CarrierTCPClient() # create client router client = sig.ClientRouter(tcpclient) The above code, creates the router, the carrier and binds them together. It is still necessary to connect to a server, in order to be able to operate the router. Connecting to server We will now connect to the pysig server we started above. Replace localhost and 3000 port parameters, passed to the connect function with the correct address of the server, if it's not running on the same machine. Code: tcpclient.on_connected = register_on_connect tcpclient.on_disconnected = cleanup_on_disconnect tcpclient.connect("localhost", 3000) Please note that the client will attempt to automatically reconnect to server, if the server is not up or cannot be reached. You can disable this behavior by doing: Code: tcpclient.connect("localhost", 3000, auto_reconnect= False) Registrations Notice the register_on_connect and cleanup_on_disconnect callback functions, we haven't presented yet. Like their name suggests, these are called when a connection is established successfully or lost, respectively. Disregarding the role this client plays (listener, sender or both) these functions are used to perform registrations, by adding a sender or a listener to the router. Please note that once a sender a sender is disconnected, the server will automatically trigger the disconnect or connect events corresponding for this sender and it will remove it from the router, without waiting for the register to unregister itself explicitly. Please note that we need to put these functions before the if statement above. Let's see how the functions look like: Code: def register_on_connect(): global senderobj, signal_tick, signal_tac # add listener logger.info("Connecting listener to timer->tick") client.addListener(listen_to_tick,"timer","tick") # add sender sender_name = "mysender" sender_events = ["tick", "tac"] logger.info("Connecting %s sender" % (sender_name)) senderobj = client.addRemoteSender(sender_name, [sender_events]) # get signals signal_tick = senderobj.getSignal("tick") signal_tac = senderobj.getSignal("tac") def cleanup_on_disconnect(): logger.info("Disconnected!") As you can see, in just a couple of lines we have registered a sender and a listener in the same time. The listener is registered for the tick event triggered by the timer sender, which is actually the sender declared by the generic_server.py example, as mentioned above. The disconnect callback is not used, just logs the tragedy of disconnecting from the server. We will rely on the server to remove us automatically upon disconnect. The listener The callback function for this listener listen_to_tick was not defined in our previous code section. Here is how it looks like: Code: def listen_to_tick(info, data): sender = info.get("sender") event = info.get("event") logger.info("[%s] Event %s received with data %s" % ( sender, event, data )) As you can see, any callback registered by any listener, receives two parameters: info and data. The info parameter carries our information about the event received, the sender who triggered the event and other attributes presented by the documentation. The data parameter holds the data published by the sender when it triggered the event, it may be None and it may differ each time the event is triggered. The sender Our listener implementation is now over, let's move on to the sender. We have registered two events, but we did nothing with them. We will finish this example by providing the main loop, which should be appended to the if statement above, with the proper identation, of course. Code: import time count_of_ticks = 0 try: while True: # ticking if tcpclient.isConnected(): count_of_ticks += 1 logger.info("[%s] Triggering event tick with data '%s" % (count_of_ticks)) signal_tick.trigger(count_of_ticks) # waiting for tac time.sleep(5) # tac if tcpclient.isConnected(): logger.info("[%s] Triggering event tac with no data") signal_tac.trigger(None) # sleep time.sleep(5) except KeyboardInterrupt as e: logger.info("Diconnecting client..") tcpclient.disconnect() As you can see, we used the signals we've previously obtained for the connect callback, to trigger our events. These signals are responsible with propagating sender events throughout the system. The server will receive these signals and route them to the corresponding listeners. If no listeners are registered, they will only be communicated to the server and to no one else. Done This is it! We can now run the application and see how we receive tick events from our server. To see the events we triggered, we can use the generic_client_listener.py example to register to our own sender and see what happens. You can play with various events and data in your application to see what happens, but the first thing you may want to try is to register to the None event using the generic_client_listener.py and observe how you receive both tick and tac events from your sender, with a single registration. This is the special sender broadcast event that is sent for each specific event that sender triggers. Another idea would be to type None when the example asks for both the sender and event name to register to and see how you receive all messages that passes through this router. This is a registration to the special router broadcast event that is sent for each event passed to the router. Enjoy. Authors For the moment I am the solely developer of this framework, but I chose we as a form of addressing to myself in this article because its nicer to read and mentioning myself repeatedly just sounds egotistical. I am more than happy to welcome other contributors in this project with ideas, patches or feature requests. References Please note that at the time this article was written pysig framework version was 0.7.3. Documentation: http://pysig.rtfd.org Repository: https://bitbucket.org/madlex/pysig Package: https://pypi.python.org/pypi/pysig
