Should I file even if I cannot pay?
Yes. The IRS charges a penalty for failing to file and a separate one for failing to pay, so filing on time limits the damage even when you cannot pay (IRS payment plans). The IRS also says to file every return that is due, whether or not you can pay in full (IRS: filing past due tax returns).
What happens if I keep not filing?
The IRS can file a substitute return for you. It says that return might not give you credit for deductions and exemptions you are entitled to. You then get a Notice of Deficiency (CP3219N), and you have 90 days to file your own return or petition the Tax Court; if you do neither, the IRS assesses its figure (IRS: filing past due tax returns).
That bill then moves into the normal collection process, which can mean a wage or bank levy or a federal tax lien.
What if the IRS owes me money for those years?
You can lose it. To claim a refund of withholding or estimated tax, you must file within three years of the return's due date, and the same rule applies to credits such as the Earned Income Credit (IRS: filing past due tax returns). The IRS also holds refunds when its records show a past-due return.
Self-employed and never filed? Your self-employment income is not reported to the Social Security Administration, so it does not count toward your Social Security credits.