To understand the consequences of borrowing from a deferred annuity (one in which annuity payments are not scheduled to commence within one year of issue), Is the annuity being used to fund an IRA or "qualified plan"? If the annuity is funding an IRA, no borrowing is permissible, because IRA rules do not permit borrowing from one's IRA. If the annuity is funding an employer-sponsored retirement plan (such as a 401(k) plan), borrowing may or may not be permitted by the plan (and the annuity contract).
If the deferred annuity is being purchased with after-tax dollars, not in an IRA or employer-sponsored plan, then borrowing is not forbidden by law, but most deferred annuity contracts do not allow it.
It should be noted that borrowing against such an annuity, or even pledging the annuity value as collateral for a loan (such as, from a bank) will cause the untaxed "gain" in the annuity to be taxable in the year of the pledging (up to the value of the amount borrowed) (IRC 72(e)(4))