Messaging

The messaging pattern allows for record operations to be sent an received as events to another external system. Messages are grouped into a named process that represents a sequential queue to where any change in the tables belonging to that process will be published. In a similar way you can receive messages from another system by subscribing to that named process and you will receive those events into target tables that should match the same schema.

Definition

The Messaging definition can be found on the Processes section of Genio and is divided into Publications and Subscriptions.

Publication

Publication definition

Publications are identified by their name and group. Both identifiers toghether must be globally unique, so use the name to create a unique name for your process and use the group to create a unique name for you system. Groups also create a future proof way to create aliases for systems without renaming every single message in case the need arises.

Version allows to inform other systems of changes in your message structures and provide retrocompatibility (when its possible).

You can require that the result of the message is send back to you. This will cause a aditional Ack queue to be setup where the subscribers will send back their results of their processing.

To avoid cycles you can mark a process to not re-publish messages in case tables change during a subscription processing. This allows for 2 systems to both publish changes to a shared business object without running into infinite cycles. Use with care since this prevents publishing of this process during any subscription process, not of any specific one.

A publication defines a list of tables that will publish their CRUD operations (Create, Retrieve, Update, Delete) to the process queue. Each table can setup:

  • The table to publish
  • The list of fields of that table to be sent on the message
  • An optional filter condition that rows need to fulfill to be sent
  • If this is an anexed table to this process

Anex table is a concept that means this table is not sent when CRUD operations are made to it. Instead if any other related table of this process add a CRUD message then the related row of the anex table is added. In practice Anex tables are usually categorizations tables of the system that although their are not part of the business process their have information the external system might need to interpret the business. A common example is a table of countries or item categories.

Note: Resend condition is an experimental option that is meant to trigger a publication in a row even when it has not changed, but the framework does not support this kind of trigger yet

Subscription

Subscription definition

Subscriptions need to declare the id and group matching the process they intend to receive and process.

The version allows declaring the expected message version you want to receive. Its up to the processing routine to handle mismatches of version and handle them properly. There is no automatic rejection mechanism implemented at the moment.

The subscription should also match the intention of the publication to receive back a result of the processing. Failing to match this setting will result in either; sending results to a queue that will never be read by the publisher; or never sending results that the publisher is expecting to receive.

Subscribers are expected to coordinate with Pubilshers during schema changes to properly match and handle these modifications.

Subscriptions also need to match the declarations of the list of tables that is published. Its up to the subscriber to declare receiving tables with matching fields (of name, type and size) of the ones that are published. This is recommended to do in virtual tables, since these tables don't actually need to persist anything to the database, and will only be used to help parsing the published message.

Each subscribed table will define:

  • The local name of the receiving table
  • The alias name to match the name the publisher gave to the table
  • A filter condition to automatically ignore rows that don't match the condition

The filter condition should be used sparringly and its preferred the publisher doesn't send the row at all to reduce bandwidth usage. It also has limitations on the fields it can reference since it only supports own table fields.

Cascade insert handling

A important aspect of this pattern to note is the handling of cascade inserts in the source system. During a cascade insert on some object, the details of that object finish their insertion before the user saves the main object. The main object is marked with a zzstate in the database during this time.

The messaging system handles this situation by inhibiting the publication of the changes while a parent zzstate row is still present, and then when it transitions to a unmarked zzstate during user save operation, the messaging then sends all detail tables that are declared in the process.

This results in receiving the cascade insert as a single message, as well as avoiding external systems having to process orphan details rows.

This behaviour might cause some issues during state transitions of the rows, especially if they interact with filter conditions or external system errors, so you must be aware of this. This issues should be however far less problematic and sporadic than constantly handling orphaned cascade inserts and cancelations.

Manual code

Although the pattern has been designed to open the door to modeling the subscription processors that is not yet implemented. Instead you need to program your message processor in manual code, altough a lot of the parsing is already taken care of, and you can focus on matching tables and fields to update in the database and error checking.

Use /[MANUAL GQT MESSAGE_PROCESS [Group][Id]]/ to setup the body of the message processor. You will have access to the message data, the metadata of that subscription and a context that allows you to interact with the messaging system, for getting send system details like the origin database, or sending progress messages. Take into consideration that is up to the processing routing to start and stop any needed database transactions and to decide the correct database to use if you have multiple. This avoid having to clone or broacast messages that are actually meant for a whole system, not just to a single database.

You can always force the response to ignore the message in the context, and its also up to the manual to add business errors detected to the reply context.

Use /[MANUAL GQT MESSAGE_ACK [Group][Id]]/ to setup the body of the ack processor. You will have access to the message data, the metadata of that publication. The ack message receives the Ids of all the rows that were processed by the subscriber so you can match them and update whatever status fields you deem necessary. You will also have a list of errors the other system detected meaning the processing was not complete. You can also choose how best to persist these errors (log file, database field, database error table) depending on business needs.

Configuration

The configuration of the publications and subscriptions in Webadmin is reduced to:

  • Enabling/Disabling the messaging service
  • Declaring the message broker endpoint (only RabbitMq provider supported at the moment)
  • Enabling/Disabling publications
  • Enabling/Disabling subscriptions

All enabled publications and subscriptions will automatically create and connect to their respective queues in the broker during messaging service start.

Note that if you have the messaging service enabled and the network connection to the broker fails this is considered a transactional failure in the same way a network connection to the database would. This is fundamental to ensuring guaranteed delivery of messages. If you need to mitigate this risk you must cluster the message broker in the same way you would to the database service.

Installation requirements

The current provider implementation is based on RabbitMq. This is a well known standard message broker optimized for very high message loads.

RabbitMq is available for Windows, Linux and Docker containers.

Once installed its highly recommended you create a new virtual host for your solution (groups of interacting systems) to reuse the same broker for other external applications or to setup isolated quality or development enviroments. It also allows a more granular permission and user accound managment.

RabbitMq depends on Erlang runtime. You will need to install Erlang in your server prior to the RabbitMq instalation.

After installing RabbitMq using the downloaded installer there is usually a mismatch of the machine key Erlang uses and the one RabbitMq uses. To fix that problem:

  • copy the file in C:\Users\%USERNAME%\.erlang.cookie
  • to C:\Windows\System32\config\systemprofile\.erlang.cookie

To create a new vhost go to the rabbit installation directory and execute the batches with the applicational username and password you want:

./rabbitmqctl.bat add_vhost <sys>
./rabbitmqctl.bat add_user -p <sys> <username> <password>
./rabbitmqctl.bat set_user_tags -p <sys> <username> administrator
./rabbitmqctl.bat set_permissions -p <sys> <username> .* .* .*

If the user already exist you might only need to add permissions to the new vhost.

It is very useful to activate the Administration plugin that provides a helpful UI to manage RabbitMq, and the tracing plugin that allows you to log to file all the messages as they are queued. To enable them use the commands:

./rabbitmq-plugins enable rabbitmq_management
./rabbitmq-plugins enable rabbitmq_tracing

Its recommended you use tracing sparringly rather than leave it on at all times.

The admin ui will be usually be available at http://localhost:15672/

The endpoint you can use in the Webadmin configuration should be available at ampq://localhost/vhost (replaced with chosen computer name and vhost name)