Optimizing Traversing and Retrieval Speed of Large Breached Databases

Abstract: Breached data refers to the unauthorized access, theft, or exposure of confidential or sensitive information. Breaches typically occur when malicious actors or unauthorized users breach secure systems or networks, resulting in compromised personally identifiable information (PII), protected or personal health information (PHI), payment card industry (PCI) information, or other sensitive data. Data breaches are often the result of malicious activities such as hacking, phishing, insider threats, malware, or physical theft. The misuse of breached data can lead to identity theft, fraud, spamming, or blackmailing. Organizations that experience data breaches may face legal and financial consequences, reputational damage, and harm to their customers or users. Breached records are commonly sold on the dark web or made available on various public forums. To counteract these malicious activities, it is possible to collect breached databases and mitigate potential harm. These databases can be quite large, reaching sizes of up to 150 GB or more. Typically, breached data is stored in the CSV (Comma Separated Value) format due to its simplicity and lightweight nature, which reduces storage requirements. Analyzing and traversing large breached databases necessitates substantial computational power. However, this research explores techniques to optimize database traversal speed without the need to rent expensive cloud machines or virtual private servers (VPS). This optimization will enable individual security researchers to analyze and process large databases on their personal computer systems while significantly reducing costs.