Seller is trying to sue

My dad, brother and I signed our names to buy a house. The seller accepted the offer, but we gave them the "earnest money first". Which was 5,000. It went through on January 8th. Since then, there has been no communication from the seller or their closing lawyers, etc. we finally heard back last week of when the closing was going to be. February 4th (tmr). However, we receive an email from our closing lawyer stating everything we would need to pay including 6,000 more (aside from the 5,000). This was a turn off to my dad. Logistically, we won't have enough money left over. We went to our real estate agents and spoke with our closing lawyer and agreed to step back from closing/finalizing the sale. A few hours ago, our lawyer called saying the seller wants to take all of us to court for "breaching the contract" and for "losing money when the house was pending". Unless we "close the deal tomorrow". Is this an empty threat or should my family and I take this to court?