PassBERT: Improving Real-world Password Guessing via Bi-directional Transformers

Password breaches remain one of the most prevalent cybersecurity threats. With billions of credentials leaked across various data breaches, understanding how attackers can exploit this information becomes critical for both offensive security research and defensive password policy design. This article examines the work "Improving Real-world Password Guessing Attacks via Bi-directional Transformers" by Ming Xu et al., presented at USENIX Security 2023, which introduces PassBERT—a pre-trained transformer-based framework for password guessing attacks.

The Password Breach Epidemic

Password security remains a critical concern in modern cybersecurity:

• 81% of company data breaches are caused by poor passwords
• 80% of hacking incidents involve stolen and reused login credentials
• 555 million stolen passwords exist on the dark web since 2017
• 27% of hackers have tried to guess other people's passwords
• 17% have successfully guessed passwords

Services like "Have I Been Pwned" allow users to check if their credentials have been compromised, while underground forums openly trade massive password databases containing billions of records.

Two Types of Password Guessing Attacks

Password guessing attacks can be categorized into two main types:

1. General Guessing Attacks

These attacks operate without any prior information about the target:

• Brute-force enumeration
• Markov chain-based models
• Dictionary attacks

2. Real-World Guessing Attacks

These attacks leverage extra information about the target:

• Previously leaked passwords from the same user
• Partial password knowledge
• Common password modification patterns

The paper focuses on three real-world guessing scenarios that are more practical and effective than general attacks.

Real-World Guessing Scenarios

1. Conditional Password Guessing (CPG)

Given: Partial password information (some characters known) Goal: Recover the complete password

Example:

*a*s*o*d → passWord

The attacker knows certain character positions and must fill in the blanks. This scenario occurs when attackers obtain partial password information through keyloggers, shoulder surfing, or partial memory dumps.

2. Targeted Password Guessing (TPG)

Given: A user's old password from a data breach Goal: Guess their new password on a different service

Example:

Dan1999 → D@n19!9

Users tend to create similar passwords across services with minor modifications. TPG exploits this password reuse behavior by predicting likely variations.

3. Adaptive Rule-based Password Guessing (ARPG)

Given: A base word and transformation rules Goal: Apply the most likely rules to crack passwords

Example:

Rule (a→@) + passWord → p@ssWord

ARPG learns which transformation rules (character substitution, appending numbers, capitalization changes) are most effective for a given password structure.

The PassBERT Framework

PassBERT adapts the BERT (Bidirectional Encoder Representations from Transformers) architecture for password modeling. Unlike unidirectional models (like GPT), BERT's bidirectional attention allows the model to understand password patterns from both directions—crucial for understanding password structure.

Pre-training Dataset

The model is pre-trained on RockYou2021.txt, a compilation containing:

• Historical password leaks
• Probable and commonly-used passwords
• Wordlists and dictionaries
• Approximately 8.4 billion entries

Architecture Specifications

Pre-training Objective: Masked Language Modeling (MLM)

Following BERT's approach, PassBERT is pre-trained using masked language modeling at the character level:

1. Randomly mask 15% of characters in a password
2. Train the model to predict the masked characters
3. The model learns contextual representations of password characters

This pre-training objective directly aligns with the CPG task, where the model must predict missing characters given partial password information.

Key Insight: Password-Language Correlation

Research by Wang et al. (2017) demonstrated a high correlation between natural language patterns and password structures—passwords follow Zipf's law similar to natural language. This explains why NLP techniques transfer effectively to password modeling.

Fine-tuning for Specific Attacks

After pre-training, PassBERT is fine-tuned for each specific attack scenario.

CPG Model Architecture

The CPG model directly uses the MLM capability—given a partially masked password, it predicts the probability distribution over all possible characters for each masked position:

P(pwd | pivot) = ∏ P(c_i | mask_i, pivot)
                c_i∈pwd, mask_i∈pivot

TPG Model Architecture

The TPG model predicts edit operations for each character position:

• Keep: Retain the original character
• Delete: Remove the character
• Replace1: Single character replacement (95 options)
• Replace2: Two-character replacement (95² options)

With maximum edit distance of 4, this allows generating likely password variations.

Example transformation:

ARPG Model Architecture

ARPG is formulated as a multi-label classification problem:

• Input: Base word from a wordlist
• Output: Which transformation rules apply

Rule sets used:

• PasswordPro: 3,120 rules
• Generated: 14,278 rules

Example rules:

Experimental Results

CPG Evaluation

Evaluation on Neopets and Cit0day datasets:

Where:

• CE: CWAE (previous state-of-the-art)
• noPT: PassBERT without pre-training (random initialization)
• VT: Vanilla BERT
• PT: PassBERT (full model)

PassBERT achieves 7.5% average improvement over the previous state-of-the-art.

TPG Evaluation

PassBERT and Vanilla BERT significantly outperform Pass2path, cracking 18%+ of accounts within 1,000 guesses compared to the previous 14.7%.

Interesting observation: Vanilla BERT performs slightly better on TPG, suggesting that fine-tuning dominates performance for this task.

ARPG Evaluation

ARPG results show PassBERT achieving cracking rates of:

• 55.6% on Neopets with PasswordPro rules
• 36.37% on Cit0day with PasswordPro rules
• 59.99% on Neopets with Generated rules
• 43.95% on Cit0day with Generated rules

These significantly outperform both standard rule-based attacks in Hashcat and previous neural approaches.

Hybrid Password Strength Meter

A practical application of PassBERT is a hybrid password strength meter that combines all three attack scenarios:

1. CPG Component: Assesses the predictability of individual characters
2. TPG Component: Estimates guesses needed based on leaked data
3. ARPG Component: Identifies base words and rule vulnerabilities

Example output for "p@ssw0rd123":

Character Strength Level: [p:low] [@:high] [s:low] [s:low] [w:medium] [0:medium] [r:low] [d:medium] [1:high] [2:high] [3:medium]

Potential Risks from Target Guessing Attacks:
The input "p@ssw0rd123" can be cracked when trying 825 guesses given the leaked "p@ssw0rd", make it more complex!

Discussion and Limitations

Key Findings

• Pre-training significantly improves password guessing performance
• High correlation between natural language and passwords enables effective transfer learning
• PassBERT achieves state-of-the-art on all three real-world guessing scenarios

Limitations and Future Directions

1. Pre-training Data Quality: RockYou2021 contains both passwords and dictionary words. The actual password ratio may be low, questioning whether improvements come from domain-specific pre-training or BERT's inherent capabilities.

2. LLM Extensions: Given the password-language correlation, larger language models (GPT-4, LLaMA) might perform even better, though computational costs would increase.

3. Token-level Modeling: Current work explores character-level relationships. Investigating token-level (chunk-level) semantics could capture higher-order password patterns.

4. Defensive Applications: The same techniques could improve password policies by identifying weak passwords that are vulnerable to learned attack patterns.

Conclusion

PassBERT demonstrates that pre-trained language models can significantly improve real-world password guessing attacks. By adapting BERT's bidirectional attention mechanism to character-level password modeling, the framework achieves state-of-the-art results across three practical attack scenarios.

The implications are twofold:

1. For attackers: More efficient credential attacks are possible when prior information is available
2. For defenders: Password policies must account for these sophisticated guessing strategies, and strength meters should incorporate similar modeling techniques

As deep learning continues to advance, the arms race between password attack and defense will increasingly depend on who better understands the patterns humans use to create passwords.

Reference: Xu et al., "Improving Real-world Password Guessing Attacks via Bi-directional Transformers," USENIX Security Symposium, 2023.

• Paper: USENIX Security '23
• Code: GitHub

• Slide: 0708_PassBERT.pdf