A sychronous communication protocol is designed and implemented for a distributed real-time system. The protocol operates on hierarchical ringsand forms the communication backbone of a cluster based distributed system model that is desugned to provide various distributed system functions such as clock synchronization, total event ordering, process group management, multicasting and fault tolerance. The multilevel hierarchy of the cluster based system model allows intracluster communication to be carried out independent of other clusters and hence provides parallelism in communication of cluster nodes. The intercluster communication among cluster representatives allows the leader to uniquely dictate various decisions to the whole system. The semantics of fault management for the cluster based model is also given and the interrelation of this fault mode with various layers of the system architecture is given by state transition diagrams. Finally, a distributed clock synchronization algorithm is implemented using the protocol and its performance is discussed.