There is a "trailer" style plug in the wiring harness from the factory and it is notorious for causing problems like you are describing. Many threads to look at.
To check you need to just do a hot wire start and see if the problem abates. If the problem does abate then the plug is degrading. Many here on this site including myself have had this issue and eliminated the connection or wired around it.
If you are unsure of hot-wiring get back for instructions and again there are some threads that cover it too.
Also give the "ground" for the battery and the A-4 a good look too.
Dave Neptune