`RequestList` with persistence doesn't handle malformed URLs correctly

[francescoaracu]

I'm trying to implement a crawler using BS (you can find the repo here).

I tried to run the crawler, adding the persistence parameter to my RequestList, but it only works if it's not present.

This is the error thrown in the logs:

[Mantisus]

Hi @francescoaracu!

Thank you for being so interested in Crawlee!

In this case, the issue isn't with persistence but with validating certain URLs in your list. Without persistence, this would have caused an error, but much later.

Until a fix is released, you can handle this on your side in load_urls_from_csv

from crawlee import Request
from pydantic import ValidationError

try:
    request = Request.from_url(row[0])
except ValidationError:
    print(f'Invalid URL encountered in CSV: {row[0]}')
    continue

[francescoaracu]

Thank you for being so interested in Crawlee!

Thanks man! I'm using the framework for my MSc thesis 😄

you can handle this on your side in load_urls_from_csv

I guess I should change the behaviour of my function, given that with this block I'm instantiating a Request object for each URL... or do I just need to put this block after the if row?

[Mantisus]

Thanks man! I'm using the framework for my MSc thesis 😄

Cool!

I guess I should change the behaviour of my function, given that with this block I'm instantiating a Request object for each URL... or do I just need to put this block after the if row?

This should work for you

from crawlee import Request
from pydantic import ValidationError

async def load_urls_from_csv(file_path: str):
    """Read URLs from CSV file asynchronously."""
    async with aiofiles.open(file_path, 'r') as file:
        reader = aiocsv.AsyncReader(file)
        # Skip header row
        await reader.__anext__()
        async for row in reader:
            if row:
                try:
                    request = Request.from_url(row[0])
                except ValidationError:
                    print(f'Invalid URL encountered in CSV: {row[0]}')
                    continue
                yield row[0]  # URL is in the first column
                # Yield control back to the event loop
                # break
                await asyncio.sleep(0)

[francescoaracu]

Thanks a lot for your suggestion @Mantisus!
Now, after some initial requests loading (like 3 minutes), with some errors encountered, crawling is starting successfully!

[Mantisus]

Glad to help.

Yes, to ensure persistence, it saves all requests to storage; this takes a little time before crawling begins.

[francescoaracu]

to ensure persistence, it saves all requests to storage

I assume that it saves the number of requests contained in the max_requests_per_crawl parameter in the crawler definition, or does it scan the whole .csv file?

[Mantisus]

I assume that it saves the number of requests contained in the max_requests_per_crawl parameter in the crawler definition, or does it scan the whole .csv file?

It scans the whole .csv file.

max_requests_per_crawl is a parameter specific to the crawler itself, which currently does not affect the queue.

When the crawler reaches the limit, it is stopped, but there may still be many URLs in the queue at that time.

This is discussed in part at #1371