Recurrent Neural Network

Recurrent Neural Networks (RNNs) process sequences by maintaining hidden state that carries information across time steps. They were foundational for NLP before transformers.

How RNNs work:

• Process input one step at a time
• Hidden state carries information forward
• Output depends on current input AND history
• Same weights applied at each step

RNN variants:

• Vanilla RNN: Simple but struggles with long sequences
• LSTM: Long Short-Term Memory, gates control information flow
• GRU: Gated Recurrent Unit, simplified LSTM
• Bidirectional: Process sequences both directions

Limitations addressed by transformers:

• Sequential processing (can't parallelise)
• Difficulty with very long dependencies
• Vanishing/exploding gradients (even with LSTM)

RNNs still useful for:

• Real-time streaming processing
• Resource-constrained environments
• Simple sequential tasks