In this paper, we introduce a novel method for human activity recognition that benefits from the structure and sequential properties of the test data as well as the training data. In the training phase, we obtain a fraction of data labels at constant time intervals and use them in a semi-supervised graph-based method for recognizing the user’s activities. We use label propagation on a k-nearest neighbor graph to calculate the probability of association of the unlabeled data to each class in this phase. Then we use these probabilities to train an HMM in a way that each of its hidden states corresponds to one class of activity. These probabilities are used to learn the transition probabilities between hidden states of the HMM which is used to predict the classes of the test data. Experimental results shows that the proposed method consistently outperforms the existing state of the art semi-supervised methods.