iF6 ddlmZmZmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZddlmZddlmZddlmZddlmZddlZd ed efd Zd eeed ee fd Zdeeefd eeeffdZded ed efdZ d dddededeedee d ef dZ!dddededee d efdZ" d dddededee deedee d ee f dZ#deded e$fdZ%ded e d e fdZ&deded e fdZ'ded efdZ(dS)!)OptionalDictAnyList) CrawlRequestCrawlJob CrawlResponseDocumentCrawlParamsRequestCrawlParamsData WebhookConfigCrawlErrorsResponseActiveCrawlsResponse ActiveCrawlPaginationConfig)handle_response_error)prepare_scrape_options)AsyncHttpClient)normalize_document_inputNrequestreturnc^|jr|jstdd|ji}|jr |j|d<|jt |j}|r||d<|jCt|jtr |j|d<n|j d|d<| dd}| dd| dd| d dd d d d dddddddddd }| D]!\}}||vr| |||<"| |t|dd2tt|d|d<|S)NURL cannot be emptyurlprompt scrapeOptionswebhookT) exclude_none)r exclude_unsetscrape_options includePaths excludePathsmaxDiscoveryDepthsitemapignoreQueryParametersdeduplicateSimilarURLscrawlEntireDomainallowExternalLinksallowSubdomainsdelaymaxConcurrencyregexOnFullURLzeroDataRetention) include_paths exclude_pathsmax_discovery_depthr%ignore_query_parametersdeduplicate_similar_urlscrawl_entire_domainallow_external_linksallow_subdomainsr+max_concurrencyregex_on_full_urlzero_data_retention integration)rstrip ValueErrorrr!rr isinstancestr model_dumppopitemsupdategetattr)rdataopts request_datafield_mappingssnakecamels `/root/.hermes/hermes-agent/venv/lib64/python3.11/site-packages/firecrawl/v2/methods/aio/crawl.py_prepare_crawl_requestrKs ;0gk//110./// 7; D~( X)%g&<==  )$(D !" gos + + L%oDOO%o88d8KKDO%%4t%LLLUD!!!Xt$$$%t,,,''2#:$<2 4-+-2N',,..22 u L &**511DKKK w t,,8!''="A"ABBHHJJ] K data_listcg}|pgD]E}t|tr.t|}|t di|F|S)N)r=dictrappendr )rM documentsdoc_data normalizeds rJ_parse_crawl_documentsrUBsb "IO55 h % % 51(;;J   X33 33 4 4 4 rLbodyc |ds#t|dd|d|dd|dd|dd|d |d t|d gd S) NsuccesserrorUnknown error occurredstatus completedrtotal creditsUsed expiresAtnextrDr[r\r] credits_used expires_atr`rD)get ExceptionrU)rVs rJ_parse_crawl_status_responserfKs 88I  E*BCCDDD((8$$XXk1--'1%%22hh{++  &txx';';<<  rLclientcKt|}|d|d{V}|jdkrt|d|}|dr7t |d|dSt|d d ) aV Start a crawl job for a website. Args: client: Async HTTP client instance request: CrawlRequest containing URL and options Returns: CrawlResponse with job information Raises: ValueError: If request is invalid Exception: If the crawl operation fails to start z /v2/crawlNz start crawlrXidr)rjrrYrZ)rKpost status_coderjsonrdr re)rgrpayloadresponserVs rJ start_crawlrpZs%W--G[[g66666666Hs""h 666 ==??D xx EDHHUOODDDD DHHW&>?? @ @@rLrequest_timeoutjob_idpagination_configrrc K|d||d{V}|jdkrt|d|}t |}|d}|r|jnd}|r(|dr t ||d||| d{V}t|d |d |d |d |d|s|dnd|S)a[ Get the status of a crawl job. Args: client: Async HTTP client instance job_id: ID of the crawl job pagination_config: Optional configuration for pagination limits request_timeout: Timeout (in seconds) for each individual HTTP request. When auto-pagination is enabled (default) and there are multiple pages of results, this timeout applies to each page request separately, not to the entire operation Returns: CrawlJob with job information Raises: Exception: If the status check fails /v2/crawl/timeoutNrizget crawl statusrDTr`rqr[r\r]rbrcra)rdrlrrmrf auto_paginate_fetch_all_pages_asyncr ) rgrsrtrrrorVrnrRrys rJget_crawl_statusr{ss<0ZZ 5V 5 5ZOOOOOOOOHs""h(:;;; ==??D*400GI8IR%33dM  0  FO  +           x +&g^,<($1 ;WV__t    rLnext_urlc <K|||d{V}|jdkrt|d|}t |}t |d|d|d|d|d |d |d  S) a Fetch a single page of crawl results using the provided next URL. Args: client: Async HTTP client instance next_url: Opaque next URL from a prior crawl status response request_timeout: Timeout (in seconds) for the HTTP request Returns: CrawlJob with the page data and next URL (if any) Raises: Exception: If the request fails or returns an error response rwNrizget crawl status pager[r\r]rbrcr`rDra)rdrlrrmrfr )rgr|rrrorVrns rJget_crawl_status_pager~s(ZZ/ZBBBBBBBBHs""h(?@@@ ==??D*400G x +&g^,<( V_ V_   rLinitial_documentscK|}|}d}|r|jnd}|r|jnd} |r|jnd} t j} |r|||krn| t j| z | krn|||d{V} | jdkr8ddl} | d}| dd| jin| } t|}n#t$rYnawxYw|d D].}| t|| krn||/| t|| krn|d }|d z }||S) a Fetch all pages of crawl results asynchronously. Args: client: Async HTTP client instance next_url: URL for the next page initial_documents: Documents from the first page pagination_config: Optional configuration for pagination limits request_timeout: Optional timeout (in seconds) for the underlying HTTP request Returns: List of all documents from all pages rNrwri firecrawlzFailed to fetch next pagerl)extrarDr`)copy max_pages max_results max_wait_timetime monotonicrdrllogging getLoggerwarningrmrfrelenrQ)rgr|rrtrrrR current_url page_countrrr start_timerorlogger page_data page_payloaddocuments rJrzrzs*"&&((IKJ0AJ!++dI3DN#//$K7HR%33dM!!J %  !zY'>'>   %DN,<,cN d d d MMOO  7 BBLL    E %V, ' 'H'c)nn .K.K   X & & & &  ##i..K*G*G #6* a K %N s.C>> D  D cK|d|d{V}|jdkrt|d|}|ddkS)a  Cancel a crawl job. Args: client: Async HTTP client instance job_id: ID of the crawl job Returns: True if cancellation was successful Raises: Exception: If the cancellation operation fails rvNriz cancel crawlr[ cancelled)deleterlrrmrd)rgrsrorVs rJ cancel_crawlrsv]]#8#8#899999999Hs""h777 ==??D 88H   ,,rLc K|jr|jstd|jr|jstd|j|jd}|d|d{V}|jdkrt |d|}|ds#t|d d |d i}i}d d ddddddddddd }| D]\}} ||vr |||| <d|vr |d} | |d<d|vr |d|d<tdi|S)au Preview crawl parameters before starting a crawl job. Args: client: Async HTTP client instance request: CrawlParamsRequest containing URL and prompt Returns: CrawlParamsData containing crawl configuration Raises: ValueError: If request is invalid Exception: If the parameter preview fails rzPrompt cannot be empty)rrz/v2/crawl/params-previewNrizcrawl params previewrXrYrZrDr/r0r1r%r2r3r4r5r6r7r!r9) r"r#r$r%r&r'r(r)r*r,rr.rrrO) rr;r<rrkrlrrmrdrerAr ) rgrrnrorV params_data convertedmappingrIrHwks rJcrawl_params_previewr,s ;0gk//110./// >3!5!5!7!731222kW^<??? ==??D 88I  E*BCCDDD((62&&K "I''2!:"<24-+)2  G 22 u K  *51Ie K  #! )D#I )  ' 'Y ' ''rLcrawl_idchK|d|dd{V}|jdkrt|d|}|d|}|dg|d|d gd }t d i|S) a" Get errors from a crawl job. Args: client: Async HTTP client instance crawl_id: ID of the crawl job Returns: CrawlErrorsResponse with errors and robots blocked Raises: Exception: If the error check operation fails rvz/errorsNrizcheck crawl errorsrDerrors robotsBlockedrobots_blocked)rrrO)rdrlrrmr)rgrrorVrnrTs rJget_crawl_errorsrasZZ >X > > >????????Hs""h(<=== ==??Dhhvt$$G++h++!++ow{{CSUW7X7XYYJ  , , , ,,rLc K|dd{V}|jdkrt|d|}|ds#t |dd|dg}g}|D]}t |t rz||d |d |d |d |d dtdd|DS)z Get active crawl jobs. Args: client: Async HTTP client instance Returns: ActiveCrawlsResponse with active crawl jobs Raises: Exception: If the active crawl jobs operation fails z/v2/crawl/activeNrizget active crawlsrXrYrZcrawlsrjteamIdteam_idroptions)rjrrrTc&g|]}tdi|S)rO)r).0ncs rJ z%get_active_crawls..s&5]5]5]Bk6G6GB6G6G5]5]5]rL)rXr) rdrlrrmrer=rPrQr)rgrorV crawls_inrTcs rJget_active_crawlsr{sQZZ 233333333Hs""h(;<<< ==??D 88I  E*BCCDDD2&&IJ  a      eeDkk55155+;+;<<uuU||55++     5]5]R\5]5]5] ^ ^ ^^rL)N))typingrrrrtypesrr r r r r rrrrrutils.error_handlerrutils.validationrutils.http_client_asyncrutils.normalizerrrPrKrUr>rfrpfloatr{r~rzboolrrrrrOrLrJrs,,,,,,,,,,,,                          988888666666666666777777 )L)T))))XhtCy&9d8n tCH~ $sCx.    AoA AAAAA8593 (, 333 3 3 013 e_ 3  3333t(, !!! !!e_ !  !!!!P59 G (, GGG GGH~G 01 G e_ G (^GGGGT-------*2(2(AS2(Xg2(2(2(2(j-?-c-FY----4_O_8L______rL