Upcoming, it is best to make a decision how you desire the crawler to deal with different World wide web technologies. There exists an ongoing debate with regard to the intelligence of search engine crawlers. It isn't totally clear If they're full-blown headless browsers or simply glorified curl scripts https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD