Web Crawling for Varibute – A Case Study

Varibute required a comprehensive data extraction solution to gather commodity data from the website agmarket.gov.in. The project aimed to collect extensive data on commodity prices and arrivals from various states and markets to support their business operations and decision-making processes. The total data expected to be collected was around 35 million records.

Varibute faced several challenges in obtaining the necessary data:

• Data Volume: The need to extract and manage a vast amount of data (35 million records) posed significant logistical and technical challenges.
• Data Accuracy: Ensuring the accuracy and relevance of the extracted data was crucial for business operations.
• Efficiency: Manual data extraction was not feasible due to the volume and complexity of the data, necessitating an automated solution.
• Integration: The extracted data needed to be formatted and stored in a way that could be easily integrated into Varibute’s existing systems for analysis and reporting.

To address these challenges, the following solution was proposed and implemented:

• Crawler Development: A crawler engine was developed using JavaScript and jQuery to automate the data extraction process. The crawler was designed to navigate the agmarket.gov.in website and extract the required data efficiently.
• System Deployment: Three systems were deployed in the office, each equipped with Chrome browsers and the necessary crawling extensions.
• Data Management: Extracted data was stored in the cloud and provided to the client in JSON format to facilitate easy access and integration.
• Continuous Communication: Regular updates on the status of the crawling activity were provided to the client via WhatsApp to ensure transparency and address any issues promptly