Job Title

Senior Backend Developer – Web CrawlingExperience: 5+ YearsLocation: Remote / BengaluruMode of Engagement: Full-timeNumber of Positions: 2 to 5Educational Qualifications: Bachelor’s degree in Computer Science, Information Technology, or related fieldIndustry: IT Services / Data Engineering / AI & AutomationNotice Period: Immediate to 30 Days PreferredWhat We Are Looking For5+ years of strong backend development experience with deep expertise in Python and scalable system design.Proven hands-on experience in large-scale web crawling using Scrapy, Requests, Playwright, and Selenium, including handling dynamic and JavaScript-heavy websites.Strong understanding of proxy/IP rotation, anti-bot mechanisms, session & cookie management, and enterprise-grade data extraction workflows.Ability to design, optimize, and maintain high-performance, distributed crawling infrastructures for structured and reliable data pipelines.ResponsibilitiesDesign, develop, and maintain large-scale web crawling and scraping systems.Build scalable backend services and crawling architectures using Python.Implement and optimize crawling frameworks using Scrapy and Requests.Handle dynamic and JavaScript-rendered websites using Playwright and Selenium.Develop and manage proxy rotation, IP management, and anti-bot bypass mechanisms.Implement authentication flows, cookie persistence, and session handling strategies.Build distributed crawling systems for enterprise-scale data extraction.Ensure data validation, normalization, and structured storage pipelines.Monitor crawler performance, logging, retries, error handling, and system stability.Collaborate with data engineering and AI teams for downstream data processing and automation.Qualifications5+ years of backend development experience with strong Python expertise.Proven experience with Scrapy, Requests, Playwright, and Selenium.Strong understanding of HTTP protocols, headers, sessions, cookies, and browser behavior.Experience implementing proxy rotation, IP management, and rate-limit handling.Familiarity with CAPTCHA handling and anti-detection strategies.Experience building large-scale or distributed crawling systems.Strong knowledge of databases such as PostgreSQL or MongoDB.Experience deploying and managing applications on AWS or similar cloud platforms.Strong analytical, debugging, and performance optimization skills.Excellent logical thinking and problem-solving ability.