wiki:portingWikiChat
Last modified 5 months ago Last modified on 05/20/14 01:31:49

WIKI CHAT

I have successfully implemented a old google style CHAT for the TRAC wiki system. In this document I describe the solution I have found.

MYchatbox.png

There are a few constraints/requirements I was facing when starting this project:

  • CHAT to integrate with the TRAC wiki system.
  • Use BASH shell scripting for the server side and JQuery for the client side.
  • Use the new technology called WEBSOCKET.
  • Allows multiple conversations at the same time.
  • Facilitate group conversations.
  • Allow the same user to have several sessions opened at the same time (a user logged in from different computers or using more than one browser).

Demo

Open this website in two different browsers otherwise it wont work.

logon as:

browser A:

  • username: newUser
  • password: new

browser B:

  • username: otherUser
  • password: other

optionally you can register yourself and have more users to test it with

WEBSOCKET

WebSockets represent a standard for bi-directional realtime communication between servers and clients. Firstly in web browsers, but ultimately between any server and any client.

It is an persistent connection between the client and the server and both parties can start sending data at any time.

It allows to send messages to a server and receive event-driven responses without having to poll the server for a reply.

meet websocketd

It turns anything that takes standard-in and standard-out into a websocket server!.

So long as you can write an executable program that reads STDIN and writes STDOUT, you can build a WebSocket server.

https://github.com/joewalnes/websocketd/wiki

This is perfect for developing the server using BASH shell script!

WEBSOCKETs the good news

no need for polling for new messages.

WEBSOCKETs the bad news

Every time the page is re-loaded the connection dies and a new connection needs to be re-established... which is a process that takes a minute or so.

the Problem

TRAC reloads a page every time a link is clicked on.

the Solution

load TRAC into an iframe and have the wiki chat running on the parent page... clicking a link in the wiki page causes the content of iframe to be re-loaded but not the parent page where the chat code and the websocket connection reside.

IFRAME

child --> parent redirection

It is possible that the TRAC project (the child page) would be addressed directly. In this case I have added code into site.html in the template dir that will redirect to the parent page (index.html).

If that was the only thing we did, the site would be opening to the default page specified in the src attribute of the iframe which is not the intent of the user.

What I have done to remedy to this is to set a cookie within the child page with the url the user entered, access the cookie from the parent page as it loads and use its content to set the src attribute of the iframe.

child page code (site.html head section):

  <script type="text/javascript" src="/chat/cookiePlugin.js"></script>
  <script language="javascript" type="text/javascript">
      //if the TARC page is referenced directly use the iframeSRC cookie to tell the parent page the actual href that the parent will load into an iframe
      if(self==top) { $.cookie('iframeSRC', self.location.href, { path: '/' }); top.location.href="/" }
  </script>

NOTE: I am using the JQuery cookiePlugin.

parent page (index.html) code:

<body>
    <script>
        //if for any reason the child page is referenced directly the child page populate the iframeSRC cookie with its href and call the parent (this page) which will load the child page into the iframe
        if($.cookie('iframeSRC') == null) { $.cookie('iframeSRC', '/yourTRACprojectPATH/wiki', { path: '/' }) }; // if the parent page is open directly the cookie would be null
        //by creating the iframe dinamically I can set the src. This will make the brower back button to work properly. (back button did not work when setting dinamically the iframe src attribute only)
        $('<iframe id="iframeID" name="iframeName" src="'+$.cookie('iframeSRC')+'" frameborder="0" width="100%" scrolling="no"></iframe>').appendTo('body')
        .resizeiframe();
    </script>

NOTE: if the cookie was NULL it means that the user addressed the parent page, in that case I set a default ifrane src attribute. Also note that all there is in the parent page body is the iframe plus a bunch of scripts for the chat.

Removing the IFRAME vertical scroll bar

If the height of the parent page is not enough to contain the iframe automatically a scroll bar will be added to the iframe potentially we will have 2 vertical scroll bars: one for the parent page and one for the child frame into the iframe.

The following code is a JQuery plugin that will resize the parent page height adjusting to the iframe height.

    <script>
      //plugin for setting the hight of the iframe to the hight of the page sourced into it
      //this will remove the vertical scroll bar for the iframe but have the scroll bar for the parent page
      $.fn.resizeiframe=function(){
         $(this).load(function() {
             $(this).height( $(this).contents().find("body").height() );
         });
         $(this).click(function() {
             $(this).height( $(this).contents().find("body").height() );
         });
      }
    </script>
</head>

NOTE: this code is found in the parent page (index.html) in the head section

Some more document sizing problems

When typing a new ticket in TRAC an automatic preview is shown which is updated periodically. In this case the iframe size is changed dynamically as you type.

I also have included code for showing the directory content as an expandable tree view, in this case the iframe size changes dynamically as you click the tree view.

In order to address these situation here is the code located into the child page site.html:

    <script>
    // this part of the code address the iframe size changes that don't have a page load event associated
    // when the content of the page gets changed by a click (see wiki documnents page)
        $('body').click(function(event){
          parent.clearFlashingTitle();
          setTimeout(function(){ $('#iframeID', window.parent.document).height($('body').height()) }, 10);
        });
    // when the page content changes as a result of typing (see adding a comment on a ticket page with the auto-preview)
        if($('#comment').length == 1){
            $('#comment').keydown(function(event) {
                if (event.keyCode == 13) { //resize the parent document height when pressing ENTER key only
                    setTimeout(function(){ $('#iframeID', window.parent.document).height($('body').height()) }, 3000);
                }
            });
        };
    </script>
  </body>

NOTE: is positioned at the bottom of the body section

The CHAT box

It is the box that is used to display the received messages and to input a new message.

  • It has a title bar with the name of the recipient of our communication and close button.
  • A content area where the messages received are displayed.
  • A input text area for composing the new messages to be sent.

CHAT box JQuery Plugin

I have implemented the CHAT box as a JQuery Plugin and using the DialogBox? offered by JQuery UI.

The Plugin accept the following options:

  • from: is the user using the chatbox. It is passed back to the sendCallBack function
  • to: is the user we send the messages to. It is also the ID of the box and the name in the box title bar.
  • width: is the width of the chatbox.
  • sendCallBack(from, id, message): is the function called when the ENTER key is pressed in the input text area.from is the user name, id is the chatbox ID which is also the recipient name, and message
  • closeCallBack(id): is the function called when the box close button is clicked. id is the chatbox ID being closed.

The Plugin offers 2 functions:

  • MYchatbox.addTXT(id, text) where id is the chatbox ID which corresponds to the recipient name, text is the formatted text that is going to be displayed in the content area.
  • MYchatbox.addMsg(id, from, message) where id is the chatbox ID which corresponds to the recipient name, from is the one that sent the message, and message is the message. This function will display the message in the content area prepended by from:.

When the Plugin is called it generates a chatbox in the right bottom corner of the browser, if more than one box is present it will position the chatboxes next to each other. If a chat box is closed, it will reposition the chatboxes side-by-side. If the browser is resized it will reposition the chatboxes to fit the new browser geometry.

NOTE: when a chatbox is closed the whole element is removed from the DOM loosing the content of the box. If the box is re-opened it is up to the calling program to add the content back if it wishes to.

If the messages in the content area don't fit a vertical scroll bar is shown and the content scrolls to bottom message.

CHATbox Usage Examples

Opening a new chatbox:

user='Matteo';
id='Marco';
closeBox = function(id) {
    DN[id] = 0; //cleanup when the box is closed
};
sendme=function(from, to, msg){
        console.log('send: '+msg); //your send logic goes here
};
$().MYchatbox({from: user, to: id, sendCallBack: sendme, closeCallBack: closeBox});

adding some text in the content area:

box=$("#"+id);
box.MYchatbox.addTXT(id, "<b>html formatted text</b>");

starting a websocket connection

There is some logic to determine whether the user is logged in or not. I the case the user is logged in if there is no connection already established a new websocket connection is started.

   var serverIPaddress = $(location).attr('href').replace(new RegExp('http://', ''), '').replace(new RegExp('/.*', ''), '');
   ws = new WebSocket('ws://'+serverIPaddress+':1212');

port 1212 is the one the websocketd is listening to.

I have added some logic to the code because I usually keep several browser TABs open with TRAC wiki pages.

I want only one page to establish the connection and the other pages poll in case the TAB with the connection get's closed to be ready to get a new connection started.

I use cookies to implement this cross TAB logic.

              ws.onopen = function() {

in this function goes the code when the websocket connection is opened, including sending to the websocket server the user name and session ID.

              ws.onclose = function() {

in this function goes the code when the websocket connection is colsed.

              ws.onmessage = function(event) {

in this function goes the code executed when a message is received by the client. In my case I parse the message and see if it carries a user list, a command to be executed by the client, or a message to be displayed in one of the chatboxes.

the logged in user list

Every time a user logs in or just becomes available (a new websocket connection is made) or becomes unavailable (websocket connection dies) a new user list is compiled and sent to each of the users logged in and connected.

The user list is displayed on the left side of the page under the logo and above the navigation bar. Each user name in the user list is a link and when pressed it opens a chatbox to start communicating with that user.

TRACnavBARadditions.PNG

sending a new message

Since originally this chat program was implemented using polling for new messages using at each poll an ajax call, also the sendme function responsible for sending the message was implemented using ajax call.

Because the sending happens only when the ENTER key is pressed it doesn't take up resources and thus I kept it as ajax implementation instead of converting it to use the websocket connection.

Once the message is sent the ajax call will execute chat_send.sh BASH shell script which will enter the message into the messages table in the DB.

It will also add an entry to the common file telling all the websocket servers a new message has been sent.

It will then add the message to the chatbox the user used for entering the text.

receiving a new message

Each websocket server (one for each session) gets the wakeup call that a new message has been sent and entered into the DB, then the WS server checks the DB if among the new messages is there any for the user they are serving.

In the case there are new messages for the user it serves, the websocket server relays the new messages to the client using the websocket connection.

Once the client receives it, the message gets added to a cookie. This allows me to keep history of the conversation even if the browser TAB where the websocket connection reside is close, since if there is another TAB with a TRAC wiki page open it will start a new connection and the conversation can be continued in the current TAB.

If the message comes and there is not chatbox for the conversation already opened, a new chatbox will be created.

the DB table for the messages

root@Lubuntu:~# mysql -pletmein -t <<< "use groupChat;select * from messages LIMIT 10;"
+------+-----------+---------+---------+----------------------+------------+------------------------------------------------------------------------------------+
| id   | from      | to      | box     | message              | sent       | recd                                                                               |
+------+-----------+---------+---------+----------------------+------------+------------------------------------------------------------------------------------+
| 1431 | newUser   | gg11    | gg11    | go gg11              | 1397659634 | matteo0 matteo1397659542 newUser0 newUser1397659551 otherUser0 otherUser1397659547 |
| 1432 | otherUser | gg11    | gg11    | yeah                 | 1397659653 | matteo0 matteo1397659542 newUser0 newUser1397659551 otherUser0 otherUser1397659547 |
| 1433 | matteo    | gg11    | gg11    | we will win          | 1397659662 | matteo0 matteo1397659542 newUser0 newUser1397659551 otherUser0 otherUser1397659547 |
| 1434 | otherUser | gg11    | gg11    | hello                | 1397861499 | matteo0 matteo1397861062 newUser0 newUser1397861040 otherUser0 otherUser1397861098 |
| 1435 | newUser   | newUser | matteo  | HI                   | 1398209795 | newUser0 newUser1398207807                                                         |
| 1436 | newUser   | matteo  | newUser | HI                   | 1398209795 | matteo0 matteo1398207790                                                           |
| 1437 | matteo    | matteo  | newUser | how are you doing%3F | 1398209833 | matteo0 matteo1398207790                                                           |
| 1438 | matteo    | newUser | matteo  | how are you doing%3F | 1398209833 | newUser0 newUser1398207807                                                         |
| 1439 | newUser   | newUser | matteo  | not too bad!         | 1398209844 | newUser0 newUser1398207807                                                         |
| 1440 | newUser   | matteo  | newUser | not too bad!         | 1398209844 | matteo0 matteo1398207790                                                           |
+------+-----------+---------+---------+----------------------+------------+------------------------------------------------------------------------------------+

same user multiple sessions

Since there is the possibility that the same user has multiple sessions opened (different computers or different browsers), in order to each session to get the new messages, the server assigned to each session will mark the messaged as received (recd field) with its user name and session number.

Group CHATs

There is a new button on the main TRAC navigation bar that will allow the user to start a group chat.

TRACnavBARadditions.PNG

If a group chat already exists, there is the option of adding new users to the group chat.

The user can leave the group chat by pressing the "x" next to the group chat name in the user list. chat.goups_parentpage.js has the javascript code to handle this functionality and chat_leaveGroup.sh, chat_addGroup.sh were also added in the /usr/lib/cgi-bin/ directory.

NOTE in the table reported above, that some of the messages are sent to a group (gg11) which is meant to be sent to multiple users. When a WS server looks into the DB for new messages, it looks for the messages with the recd field with its username and a 0 appended to it, once the message is relayed to its client, it will tag the message with the username and its session number.

A new MySQL DB table has been added to support group chat: when a group is added the group name and the users that are part of the group get stored into a new record.

root@ip-172-31-44-196:~# mysql -uroot -pletmein -t <<< 'USE groupChat;select * from users;'
+----+-----------+----------------+
| id | groupName | users          |
+----+-----------+----------------+
|  5 | gg11      | matteo newUser |
+----+-----------+----------------+

The websocket server

Once the websocket connection is started (from the client side) websocketd runs an new instance of the script which works as the websocket server, in my case called chat_newMSGs.sh. Thus we will have a server running for each chat session. chat_newMSGs.sh is implemented using BASH shell scripting.

Identifying the CHAT session

The first thing the server does is to establish the identity of the user opening the session.

The first parameter that it receives is the username, although the username is unique the same user could have different TRAC sessions open on different computers (or the same computer but different browsers).

For this reason TRAC generates as a user logs in a unique ID which is stored into the web-browser and the TRAC DB, which is called TRAC_AUTH. Thus same user can be logged in more than once and each session will be uniquely identified.

I am using this to help identify the chat session as well; and it is the 2nd parameter the websocket server receives from the client.

Another parameter we find in the TRAC DB which is unique: it is the time in seconds of the user logged in which is a more manageable number (less digits). I have thus decided to use TRAC_AUTH to find out this time and use the time as CHAT session ID.

This ID is then passed back to the web-browser and used for between browser <--> server connection.

I realized this would be un-necessary as I could use the TRAC_AUTH instead, but for historical reasons I found myself doing this way and left it so.

Avoiding polling the DB for new messages

Using the websocket allows us to avoid polling for new messages from the client side (it was indeed my first implementation for this chat program, with a periodic ajax call to the server for checking on new messages for the user).

But all we have done is to transfer the problem to the server side as it is the server role now to look for the presence of new messages for the user.

Is there a way to avoid that?

One way is somehow to communicate to the server every time a user enters a new message into the DB, only at that time the server goes and checks if a new message for it has been entered. If it does find a new message for the user it is serving then will relay the message to the client for displaying.

Sending commands to the server from the server side

All the websocked provides is a way of connecting to the client where server's STDIN is a message from the client and the server's STDOUT is a message for the client.

What we need is a way for whoever is entering the new message into the DB to broadcast a message to all the servers (one for each chat session) waking them up from their "sleep" so that they can go and check the DB for new messages for their user.

I have used a named pipe for communicating to the server from other means than the client. So each server has its own pipe named according to its unique session ID.

When a new message is entered into the DB a new message command entry is recorded into a common file. Each server tails -f the common file to its own name pipe, receiving this way the "wake up" call.

Avoiding polling the DB for new users

The other polling that would be needed is for checking on the logged in user list, the above logic is used for this as well. When a new user logs in an new user list command entry is recorded into the common file and the new logged in user list is relayed to the client for displaying.

Avoiding polling for commands to be relayed to the client

The last use of the above logic is for a command to be sent to the client initiated by another server side code.

receiving commands from the client side

As mentioned earlier the STDIN is the server input from the client. Once the server has fished setting up the connection it starts a loop listening for commands coming from the STDIN.

Keeping a record of CHATs

When a new message is entered by any of the active chat users, the chat_send.sh script is called which will also append the message to a text file for each of the users involved in the chatting session.

Accessing to the CHATs record

I have added a new "Chats" button to the TRAC main navigation bar, when pressing the button it will open a wiki page [wiki:user/chat].

TRACnavBARadditions.PNG

Stored into the wiki page I have the following script: user chat list Script which populates the page with the list of the user's chats.

CODE

  • /cgi-bin/defs.conf defines where to find the sqlite db file
  • /cgi-bin/chat_groupUserList.sh?groupName="'+id+'"' returns the list of users for the group
  • /cgi-bin/chat_send.sh?from="'+from+'";to="'+to+'";message="'+msg+'";userID="'+userID+'"' adds the new message to the DB, writes the new message to the chat log files, flags the presence of a new message to the WS servers
  • /cgi-bin/chat_getGroups.sh?user=%22"+user+"%22" returns the groups list the user is part of for "adding users to the existing group" function
  • /cgi-bin/chat_addGroup.sh?users=%22"+users+"%22;newGroup=%22"+newGroup+"%22;groupName=%22"+groupName+"%22" adds a new group
  • /cgi-bin/chat_leaveGroup.sh?user=%22"+user+"%22;groupName=%22"+groupName+"%22;type=%22leaveGroup%22" removes user from the group, if the group has less than 2 users the group itself is removed
  • /cgi-bin/chat_list.sh?user=%22"+user+"%22" lists the chat logs for the user
  • /cgi-bin/UserList.sh compiles the list of loggedin users and active groups and flags the WS servers for a new user list available
  • /var/www/index.html the parent page
  • /yourTRACprojPATH/template/site.html the child page

included within the index.html

  • <script type="text/javascript" src="/chat/jquery.js"></script> jquery library
  • <script type="text/javascript" src="/chat/cookiePlugin.js"></script> jquery cookie plugin
  • <link rel="stylesheet" href="/chat/jquery-ui-1.8.2.custom.css" /> jquery UI
  • <script type="text/javascript" src="/chat/jquery-ui-1.8.2.custom.min.js"></script> jquery UI
  • <script type="text/javascript" src="/chat/chat_MYchatbox_websocket_parentpage.js"></script> main JS code for implementing the CHAT
  • <link type="text/css" href="/chat/chat_groups.css" rel="stylesheet" /> provide the groupchat support
  • <script type="text/javascript" src="/chat/chat_groups_parentpage.js"></script> provide the groupchat support
  • <script type="text/javascript" src="/chat/MYchatbox.js"></script> JQuery chatbox Plugin
  • /yourWEBserverROOTdir/chat/groupChat.sql SQL file for creating chat DB
  • /root/bin/chat_newMSGs.sh the WS server
  • /root/bin/websocketd the WS deamon

Installation

  • copy the /cgi-bin/ files into your cgi-bin directory and make sure they are executable (chmod +x filename)
  • edit /cgi-bin/defs.conf file to fit your TRAC installation
  • copy the /root/bin files and make sure they are executable
  • copy the JS and the CSS files into /yourWEBserverROOTdir/chat/ directory
  • copy the index.html file into /yourWEBserverROOTdir/ directory
  • edit the index.html file and replace the projectPATH with your TRAC project path
  • modify the site.html by adding the following code:

(site.html <head> section):

  <!-- force the links in the iframe to open within the iframe itself -->
  <base target="iframeName" />

  <script type="text/javascript" src="/chat/cookiePlugin.js"></script>
  <script language="javascript" type="text/javascript">
      //if the TARC page is referenced directly use the iframeSRC cookie to tell the parent page the actual href that the parent will load into an iframe
      if(self==top) { $.cookie('iframeSRC', self.location.href, { path: '/' }); top.location.href="/" }
  </script>
<script>
$(document).ready(function(){
  $.cookie('iframeSRC', self.location.href, { path: '/' }); //every time the iframe is reloaded it updates the URL cookie, in case the browser reload button is pressed and the parent page is reloaded as well, it will know where the iframe was before the reload.
  console.log($.cookie('iframeSRC'));
  $('#logo').css("float","left");
  $('#metanav').prepend('<ul id="users" style="float:left"></ul>');
});
</script>

(site.html at the end of the <body> section):

    <script>
    // this part of the code address the iframe size changes that don't have a page load event associated
    // when the content of the page gets changed by a click (see wiki documnents page)
        $('body').click(function(event){
          parent.clearFlashingTitle();
          setTimeout(function(){ $('#iframeID', window.parent.document).height($('body').height()) }, 10);
        });
    // when the page content changes as a result of typing (see adding a comment on a ticket page with the auto-preview)
        if($('#comment').length == 1){
            $('#comment').keydown(function(event) {
                if (event.keyCode == 13) { //resize the parent document height when pressing ENTER key only
                    setTimeout(function(){ $('#iframeID', window.parent.document).height($('body').height()) }, 3000);
                }
            });
        };
    </script>
  </body>
  • set the root user mysql password
  • add the groupChat DB into mysql
    mysql -u root -p yourROOTpassword < /yourWEBserverROOTdir/chat/groupChat.sql
    
  • update all the *.sh files with a mysql command into them to reflect the password chosen
  • edit /etc/rc.local file to start websocketd whenever the linux computer is started by adding to it the following code
    source /usr/lib/cgi-bin/defs.conf
    sqlite3 $db "delete from auth_cookie;" #everybody is forced to logout
    nohup /root/bin/websocketd --port 1212 /root/bin/chat_newMSGs.sh &
    rm /.*fifo /.*.connected /.*.pid /.*.loggedIN
    echo -n > /tmp/.msgLoop.cmd; chown www-data:www-data /tmp/.msgLoop.cmd
    
  • reboot the linux computer
  • log in the WIKI and press the "Chats" button on the main navigation bar. It will try opening a page that does not exist yet, select to create it.

and paste the content found here: user chats list Script

Downloads

parent page:

Error: Macro ShellScript("'/usr/lib/cgi-bin/LinkToFile.sh /matteoProjectDocs/Docs/Projects/wikiCHAT/index.html index.html'") failed
12Cannot allocate memory
the /root/bin scripts:
Error: Macro ShellScript("'/usr/lib/cgi-bin/LinkToFile.sh /matteoProjectDocs/Docs/Projects/wikiCHAT/wikiCHATrootBIN.zip wikiCHATrootBIN.zip'") failed
12Cannot allocate memory
the cgi-bin scripts:
Error: Macro ShellScript("'/usr/lib/cgi-bin/LinkToFile.sh /matteoProjectDocs/Docs/Projects/wikiCHAT/wikiCHATcgi-bin.zip wikiCHATcgi-bin.zip'") failed
12Cannot allocate memory
the JQuery scripts:
Error: Macro ShellScript("'/usr/lib/cgi-bin/LinkToFile.sh /matteoProjectDocs/Docs/Projects/wikiCHAT/wikiCHATjs.zip wikiCHATjs.zip'") failed
12Cannot allocate memory
VirtualBox Appliance with Ubuntu Linux TRAC from TURNKEY and the wikiCHAT installed 250MB 7z file Download just extract it into your Virtualbox VMs directory. The VM is configured with 192.168.1.222 IP address, you might want to reconfigure to your like.

Linux root password: letmein
TRAC admin password: letmein
MYSQL root password: letmein
webmin port: https://IP:12321
webshell port: https://IP:12320

testing the CHAT

MAKE SURE YOUR COMPUTER ACCEPTS INBOUND CONNECTION TO THE 1212 PORT!!'''

I have used the following tool which is a command line websocket client for debugging

Once connected you can use STDIN to send messages. Each line is a message. You can just as well open a peristent client connection that prints incoming messages to STDOUT and sends messages from STDIN interactively:

wssh localhost:1212

Please feel free to leave a comment

Comment by anonymous on Tue Apr 29 16:42:45 2014

Nice write up! I will try to learn more about Websockets!

[AddComment?(appendonly)]