Many working Americans are tapping into their retirement funds for emergencies and unexpected expenses, and millions more have no savings for retirement at all, according to a 2015 survey by financial services website Bankrate.com.

The survey also revealed that fewer than one-fourth of workers ages 50 to 64 have seen their retirement savings suffer over the past year, making them the hardest hit age group. And perhaps the most troublesome statistic that came from the survey was that approximately 21 million Americans are not saving for retirement at all.

Failing to save for your golden years or raiding your current retirement savings can be detrimental and cause long-lasting repercussions to your financial wellness. There are ways to help improve your finances, though, and to prepare for when you eventually stop working.

The big drain on retirement savings
The survey found that nearly one-fifth of retirement-age consumers recently used their retirement savings for an emergency – a not-so-surprising finding to some financial professionals. “The majority of the times you have to debit a retirement account, it’s going to be because of a job loss or medical issue,” says Jim Saulnier, Certified Financial Planner (CFP®) at Jim Saulnier & Associates, in Fort Collins, Colo. These types of emergencies may be more of an issue for those closer to retirement, as they may have a harder time finding new work or recovering from a medical problem. Without other emergency funds, retirement savings may seem like an attractive option.

Still, this is worrisome because of the effects that early withdrawal can have on your finances in retirement. You may encounter tax penalties for early withdrawal, or, if you’re unable to replace the depleted funds, you may face the chance of running out of money during your retirement.

Millennials were the least likely to report a negative change to their retirement savings, at 8 percent, and they were the most likely to mark an improvement in their finances overall. This may be because younger workers generally have less saved for retirement—meaning they have less to draw upon—but they may also be able to recover faster from a job loss or medical challenge.

21 million Americans workers aren’t saving for retirement
While saving for retirement may not seem like a priority for you if you’re a younger worker, the earlier you start putting money away, the better. By delaying your retirement savings, you will lose out on years of compound interest, which could leave you significantly less to live on in retirement.

“It’s very hard for 20- and 30-year-olds to understand the concept of retirement,” Saulnier says. While many younger people have some sort of retirement savings, they may not concentrate on the end-goal, instead focusing on more immediate uses for their money, he says.

But without enough dependable funds in retirement, you may have trouble financing even a basic lifestyle for yourself. If you don’t have close relatives to rely on for this, you may face some particularly challenging decisions in retirement.

Steps to repair your retirement savings
Fortunately, you can help improve your retirement savings in a variety of ways, even if you’ve delayed getting started for years. It’s never too late to begin saving some part of your income from each paycheck, though the later you start, the more of your salary you’ll likely want to put away. A CFP® or other financial professional may be able to review your unique situation and make recommendations for how you can best save.

If you’re still working, Saulnier says to always put the minimum amount into your retirement accounts to get the maximum company match. After that, you can shift your focus to other accounts and saving funds. For those just starting out, though, a more liquid account may be a greater priority.

“Younger people need an emergency fund to guard against job loss and other things of that nature,” Saulnier says, while those closer to retirement want to gain as much extra savings as possible to live on in their later years.

You might worry that your chance to save for retirement has passed you by, but starting now is certainly better than giving up completely. Your ability to save will depend on your particular situation, but there are many online and in-person resources to help you get started.

Dustin Pellegrini is a senior web producer and writer at Think Glink Media, where he specializes in reporting on identity protection and credit. He studied writing and visual media at Columbia College Chicago.