Yes you should look worried, Jon.
This was certainly one of the more interesting episodes this season. Not do to cool things that happened not due to complex plot points that will have to be remembered for the rest of the season to see what they lead to but what was added.
The logistics of splitting the one book into two seasons is certainly complex and because of this you frequently find the writers trimming unnecessary subplots (sometimes embellishing them. Because they made him older in this season Margaery Tyrell’s approach to her new betrothed Tommen Baratheon is quite different (scaring the poor little virgin out of his mind)) dropping tertiary characters who are little more than ongoing cameos… until they are not. A good example there’s a minstrel who accompanied Catelyn to the Aerie in Game of Thrones (season 1) He would have reappeared in the next episode to be a lesser but necessary piece in the game. At least he would have if Joffrey hadn’t destroyed his career in the first season. Not really a problem and it’s easy enough for the writers to replace the lost Marillion with another completely expendable minstrel and none will be overly concerned.
The problem goes both ways though while some of the storylines are complex that could be show in their own right others move more slowly or had most of their plot points handled in the previous season and if things went as they do in the books they wouldn’t have any screen time until the last few episodes.
This is the case of Bran and Jon Snow. In Bran’s case his role in the book Sword of Storms ends with his band crossing the Wall. They’ll reappear in the next few books heading farther north. In the case of Jon, he is preparing for the inevitable Wilding attack which I expect the creators to have spent the bulk of the season’s budget on. In either case this makes for great prose in the book but rather dull television.
Because of this the writers have padded with several lesser subplots involving deserter Black Brothers, a Bolton operative hunting Bran and Rickon and since Jon Snow has been told that Bran is alive in the show (In the book Sam, who is the only one who knows that Bran is alive and north of the wall has been sworn to secrecy by Coldhands) it looks like the two brothers paths will cross. I’m sure this will be fun to watch but it surprised me as this was the farthest from the path I’ve seen the writers go so far.
Not to say that this wasn’t fun to watch and the stuff they added with the White Walkers was lots of fun to watch!