Classification of real time multicast data using payload-based analysis is becoming increasingly difficult with many applications that a network supports. In this paper, we set our goal to identify the recurrent patterns and classification of transport layer data, as an effective measure of anomaly-based intrusion detection. These patterns are identified by using association rules techniques such as Apriori and clustering algorithms. A simulation experiment was configured to verify the efficacy of the algorithms. We are able to find an association between flow parameters for network traffic from the simulated data. This paper contributes a possible approach of analyzing behavior patterns for building a network traffic intrusion detection system and firewall at Transport layer, by using unsupervised association rule mining and clustering techniques. © 2011 Springer-Verlag.