PIC ports behaves randomly

I am trying for hours now to light up a simple LED on a button press with no luck. The pins seem to randomly get low and high for some reason. The idea is when I hold the button the LED should light up and when I release the button the LED should turn off.
I am using MPLAB X IDE v5.54 and PIC12F1822. Can anyone spot my mistake?
Here is the code:
 

// CONFIG1
#pragma config FOSC = ECH // Oscillator Selection (ECH, External Clock, High Power Mode (4-32 MHz): device clock supplied to CLKIN pin)
#pragma config WDTE = ON // Watchdog Timer Enable (WDT enabled)
#pragma config PWRTE = OFF // Power-up Timer Enable (PWRT disabled)
#pragma config MCLRE = ON // MCLR Pin Function Select (MCLR/VPP pin function is MCLR)
#pragma config CP = OFF // Flash Program Memory Code Protection (Program memory code protection is disabled)
#pragma config CPD = OFF // Data Memory Code Protection (Data memory code protection is disabled)
#pragma config BOREN = ON // Brown-out Reset Enable (Brown-out Reset enabled)
#pragma config CLKOUTEN = OFF // Clock Out Enable (CLKOUT function is disabled. I/O or oscillator function on the CLKOUT pin)
#pragma config IESO = ON // Internal/External Switchover (Internal/External Switchover mode is enabled)
#pragma config FCMEN = ON // Fail-Safe Clock Monitor Enable (Fail-Safe Clock Monitor is enabled)

// CONFIG2
#pragma config WRT = OFF // Flash Memory Self-Write Protection (Write protection off)
#pragma config PLLEN = ON // PLL Enable (4x PLL enabled)
#pragma config STVREN = ON // Stack Overflow/Underflow Reset Enable (Stack Overflow or Underflow will cause a Reset)
#pragma config BORV = LO // Brown-out Reset Voltage Selection (Brown-out Reset Voltage (Vbor), low trip point selected.)
#pragma config LVP = ON // Low-Voltage Programming Enable (Low-voltage programming enabled)

#include <xc.h>

#define _XTAL_FREQ 8000000 // SET frequency as 8000000 bit because of the 8 MHz clock

int main(int argc, char** argv) {
    OSCCON = 0xF0; // Internal oscillator 8MHz and PLL enabled
    ADCON0 = 0; // Disable ADC module

    ANSELAbits.ANSA2 = 0; // SET LED PIN to Digital
    ANSELAbits.ANSA4 = 0; // SET Button pin to Digital

    TRISAbits.TRISA2 = 0; // LED bit is output
    TRISAbits.TRISA4 = 1; // Button bit is input

    PORTAbits.RA4 = 0; // Button bit low initial
    PORTAbits.RA2 = 0; // LED bit low initial

    while (1) {
        while (PORTAbits.RA4 == 0); // Wait until button is pressed

        __delay_ms(100); // wait 100ms
        PORTAbits.RA2 = 1; // Light up the LED
        __delay_ms(100); // wait 100ms

        while (PORTAbits.RA4 == 1); // Wait until button release

        __delay_ms(100); // wait 100ms
        PORTAbits.RA2 = 0; // Turn off the LED
        __delay_ms(100); // wait 100ms
    }
    return (EXIT_SUCCESS);
}

 
Any help is appreciated.

illusion.00

#pragma config FOSC = ECH // Oscillator Selection (ECH, External Clock, High Power Mode (4-32 MHz): device clock supplied to CLKIN pin)
...
OSCCON = 0xF0; // Internal oscillator 8MHz and PLL enabled

 
Do you see why those two statements are incompatible?
 

illusion.00

#pragma config WDTE = ON // Watchdog Timer Enable (WDT enabled)

 
Turn off the watchdog timer until you are experienced enough to know why you want to use it. And then you still probably don't want to use it.
 

illusion.00

#pragma config MCLRE = ON // MCLR Pin Function Select (MCLR/VPP pin function is MCLR)

 
Do you have an external pullup resistor on the MCLR pin?
 

illusion.00

#pragma config LVP = ON // Low-Voltage Programming Enable (Low-voltage programming enabled)

 
If you're not actually using Low Voltage Programming you should probably turn it off.
 

illusion.00

#pragma config PLLEN = ON // PLL Enable (4x PLL enabled)
...
#define _XTAL_FREQ 8000000 // SET frequency as 8000000 bit because of the 8 MHz clock
...
OSCCON = 0xF0; // Internal oscillator 8MHz and PLL enabled

 
8MHz X 4xPLL = ? (hint: not 8MHz)
 

illusion.00

ADCON0 = 0; // Disable ADC module
ANSELAbits.ANSA2 = 0; // SET LED PIN to Digital
ANSELAbits.ANSA4 = 0; // SET Button pin to Digital

 
If you're just turning on an LED, turn off analog functions altogether:
ANSELA = 0;
ADCON0 = 0;
ADCON1 = 0;
CM1CON0 = 0; // these two lines aren't actually necessary
CM1CON1 = 0;
 

illusion.00

TRISAbits.TRISA2 = 0; // LED bit is output
TRISAbits.TRISA4 = 1; // Button bit is input

 
Don't leave inputs floating. Best to set everything to output and turn on the inputs you need:
TRISA = 0; // all output
TRISAbits.TRISA4 = 1; // Button bit is input
 

illusion.00

PORTAbits.RA4 = 0; // Button bit low initial
PORTAbits.RA2 = 0; // LED bit low initial

 
Search for the "Read-Modify-Write" problem. PIC's with data latches should use them.
Hence: Write to LATA, read from PORTA.
 

illusion.00

PORTAbits.RA4 = 0; // Button bit low initial

 
Why write to a pin that is set as INPUT?
 

Aside from everything pointed out above: How is your button wired to the PIC?  Do you have a pull-up resistor on your button line external to the PIC?
If you don't have a pull-up, this PIC does have weak internal pull-ups on the I/O that can be enabled.  Should probably work for a simple button.

/*Enable the internal weak pull-up on A4*/
WPUAbits.WPUA4 = 1;

If your input is not pulled in one direction or another (either high to VDD or low to GND), then you have what is called a "floating" input.  A floating input has no guarantee as to how it is going to be read and could be why you see "random" values from the pin.