hi dZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddlZ ddl mZmZdZdZdaej(dZej(dZejdd d Zd Zdd ZddZddZdddej8d d d dd f dZddZddZddZ Gdde!Z"Gdde!Z#y)a Descriptor archives are available from `CollecTor `_. If you need Tor's topology at a prior point in time this is the place to go! With CollecTor you can either read descriptors directly... .. literalinclude:: /_static/example/collector_reading.py :language: python ... or download the descriptors to disk and read them later. .. literalinclude:: /_static/example/collector_caching.py :language: python :: get_instance - Provides a singleton CollecTor used for... |- get_server_descriptors - published server descriptors |- get_extrainfo_descriptors - published extrainfo descriptors |- get_microdescriptors - published microdescriptors |- get_consensus - published router status entries | |- get_key_certificates - authority key certificates |- get_bandwidth_files - bandwidth authority heuristics +- get_exit_lists - TorDNSEL exit list File - Individual file residing within CollecTor |- read - provides descriptors from this file +- download - download this file to disk CollecTor - Downloader for descriptors from CollecTor |- get_server_descriptors - published server descriptors |- get_extrainfo_descriptors - published extrainfo descriptors |- get_microdescriptors - published microdescriptors |- get_consensus - published router status entries | |- get_key_certificates - authority key certificates |- get_bandwidth_files - bandwidth authority heuristics |- get_exit_lists - TorDNSEL exit list | |- index - metadata for content available from CollecTor +- files - files available from CollecTor .. versionadded:: 1.8.0 N) CompressionDocumentHandlerz!https://collector.torproject.org/z-(\d{4})-(\d{2})\.z%(\d{4}-\d{2}-\d{2}-\d{2}-\d{2}-\d{2})i'c.t tatS)z Provides the singleton :class:`~stem.descriptor.collector.CollecTor` used for this module's shorthand functions. :returns: singleton :class:`~stem.descriptor.collector.CollecTor` instance )SINGLETON_COLLECTOR CollecTorc/var/www/betterdocs.net/sherlock_api/venv/lib/python3.12/site-packages/stem/descriptor/collector.py get_instancer Ps #+ r Fc#ZKtj||||||D]}|yw)zv Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_server_descriptors` on our singleton instance. N)r get_server_descriptorsstartendcache_tobridgetimeoutretriesdescs r rr`s5n33E3&RY[bcd J)+c#ZKtj||||||D]}|yw)zy Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_extrainfo_descriptors` on our singleton instance. N)r get_extrainfo_descriptorsrs r rrks5n66uc8VU\^efd Jrc#XKtj|||||D]}|yw)zt Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_microdescriptors` on our singleton instance. N)r get_microdescriptorsrrrrrrs r rrv2n11%hQXYd J(*c #`Ktj||||||||| D]} | yw)zm Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_consensus` on our singleton instance. N)r get_consensus) rrrdocument_handlerversionmicrodescriptorrrrrs r r"r"sFn**5#xAQSZ\kmsu|FGd Js,.c#XKtj|||||D]}|yw)zt Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_key_certificates` on our singleton instance. N)r get_key_certificatesrs r r'r'rr c#XKtj|||||D]}|yw)zs Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_bandwidth_files` on our singleton instance. N)r get_bandwidth_filesrs r r)r)s2n00XwPWXd Jr c#XKtj|||||D]}|yw)zn Shorthand for :func:`~stem.descriptor.collector.CollecTor.get_exit_lists` on our singleton instance. N)r get_exit_listsrs r r+r+s1n++E3'7Sd Jr cheZdZdZdZddddej ddfdZd dZe dZ e dZ y) FileaF File within CollecTor. :var str path: file path within collector :var tuple types: descriptor types contained within this file :var stem.descriptor.Compression compression: file compression, **None** if this cannot be determined :var int size: size of the file :var str sha256: file's sha256 checksum :var datetime start: first publication within the file, **None** if this cannot be determined :var datetime end: last publication within the file, **None** if this cannot be determined :var datetime last_modified: when the file was last modified c||_|r t|nd|_tj ||_||_||_tjj|d|_ d|_ |rM|rKtjj|d|_ tjj|d|_ ytj|\|_ |_ y)Nr z%Y-%m-%d %H:%M)pathtupletypesr-_guess_compression compressionsizesha256datetimestrptime last_modified_downloaded_torr_guess_time_range)selfr/r1r4r5first_publishedlast_publishedr8s r __init__z File.__init__sDI!&uBDJ..t4DDIDK!**33MCSTDD >$$--o?OPdj""++N`_, this is guessed if not provided :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param int timeout: timeout when connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose :returns: iterator for :class:`~stem.descriptor.__init__.Descriptor` instances in the file :raises: * **ValueError** if unable to determine the descirptor type * **TypeError** if we cannot parse this descriptor type * :class:`~stem.DownloadFailed` if the download fails N rz/Unable to determine this file's descriptor typerz;Unable to disambiguate file's descriptor type from among %sz, T)r# published)setr1split ValueErrorlenjoinr9osr/existsdirnametempfilemkdtempreadshutilrmtreedownloadstem descriptor parse_file startswithtype_annotationnamegetattr)r; directorydescriptor_typerrr#rrt base_types tmp_directoryrr/rAs r rLz File.readsP $>t?S?S?U?Z?Z$[D+t4  y5( y3   ==sGG E0G$+Gc F|jjdd}|jtjk7r|r|j ddd}t jj|}t jj||}t jj|st j|t jj|rt|5}tjtj|j } t#j |j%j'} | | k(r |cdddS|st)|d| d| d  dddt*j,j.j1t2|jz||} |r|jj5| } t|d 5} | j7| ddd||_|S#1swYxYw#1swYxYw) a Downloads this file to the given location. If a file already exists this is a no-op. :param str directory: destination to download into :param bool decompress: decompress written file :param int timeout: timeout when connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose :param bool overwrite: if this file exists but mismatches CollecTor's checksum then overwrites if **True**, otherwise rases an exception :returns: **str** with the path we downloaded to :raises: * :class:`~stem.DownloadFailed` if the download fails * **IOError** if a mismatching file exists and **overwrite** is **False** /.rrNz? already exists but mismatches CollecTor's checksum (expected: z , actual: )wb)r/rCr3r PLAINTEXTrsplitrG expanduserrFrHmakedirsopenbinasciihexlifybase64 b64decoder5hashlibrL hexdigestIOErrorrPutil connectionrO COLLECTOR_URL decompresswriter9) r;rWrqrr overwritefilenamer/ prior_file expected_hash actual_hashresponse output_files r rOz File.download+s(yys#B'H ;000Za(+h""9-I 77<< 8 ,D 77>>) $kk) ww~~d :Q (()9)9$++)FG nnZ__%67AAC K ' QQ nruBDOPQ Q Qyy##,,]TYY-FQXYH!!,,X6h dD "[!"D K%QQ""s0A*H $H (H HH ctjtjtjfD]!}|j |j s|cStj S)z> Determine file comprssion from CollecTor's filename. )rLZMABZ2GZIPendswith extensionrb)r/r3s r r2zFile._guess_compressionbsP $((+//;;K;KL {,, -   r ctj|}|rutt|j \}}t j ||d}|dkr|t j ||dzdfS|t j |dzddfSt j|}|rJt j j|jdd}||t jdzfSy)z Attemt to determine the (start, end) time range from CollecTor's filename. This provides (None, None) if this cannot be determined. r z%Y-%m-%d-%H-%M-%Sr)seconds)NN) YEAR_DATEsearchmapintgroupsr6SEC_DATEr7group timedelta)r/ year_matchyearmonthr sec_matchs r r:zFile._guess_time_rangens!!$'JZ..01kdEeQ/e x((uqy!<==x((1a899%I (();=PQeUX//$?? @@ r )TNrF) __name__ __module__ __qualname____doc__r>rENTRIESrLrO staticmethodr2r:r r r r-r-sc":$"T4thwhhLP\]Xt5n ! !r r-c eZdZdZddZddZddZddZdddejdddddf d Z dd Z dd Z dd Z dd ZddZedZy)r aS Downloader for descriptors from CollecTor. The contents of CollecTor are provided in `an index `_ that's fetched as required. :var int retries: number of times to attempt the request if downloading it fails :var float timeout: duration before we'll time out our request NcJ||_||_d|_d|_d|_y)Nr)rr _cached_index _cached_files_cached_index_at)r;rrs r r>zCollecTor.__init__s(DLDLDDDr Frc #K|sdnd}|j|||D]"}|j||||||D]} | $yw)aL Provides server descriptors published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.server_descriptor.ServerDescriptor` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zserver-descriptorzbridge-server-descriptorrrNfilesrL r;rrrrrr desc_typefrs r rz CollecTor.get_server_descriptorssY*,2#7QI ZZ 5# .&&9eSGW^&_$ AAc #K|sdnd}|j|||D]"}|j||||||D]} | $yw)aZ Provides extrainfo descriptors published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.extrainfo_descriptor.RelayExtraInfoDescriptor` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails z extra-infozbridge-extra-inforNrrs r rz#CollecTor.get_extrainfo_descriptorssX*%+ 0CI ZZ 5# .&&9eSGW^&_$ rc #|K|jd||D]"}|j|d||||D]}|$yw)a Provides microdescriptors estimated to be published during the given time range, sorted oldest to newest. Unlike server/extrainfo descriptors, microdescriptors change very infrequently... :: "Microdescriptors are expected to be relatively static and only change about once per week." -dir-spec section 3.3 CollecTor archives only contain microdescriptors that *change*, so hourly tarballs often contain very few. Microdescriptors also do not contain their publication timestamp, so this is estimated. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.microdescriptor.Microdescriptor for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails r%rNrr;rrrrrrrs r rzCollecTor.get_microdescriptorssP<ZZ)5# 6&&#4eSG_f&g$ :<c #K|dk(r|s|sd} n@|dk(r|r|sd} n4|dk(r|s|sd} n(|rd} n#|r|dk7rtd|ztd|z|j| ||D]#} | j|| ||||| D]} | %y w) a Provides consensus router status entries published during the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param stem.descriptor.__init__.DocumentHandler document_handler: method in which to parse a :class:`~stem.descriptor.networkstatus.NetworkStatusDocument` :param int version: consensus variant to retrieve (versions 2 or 3) :param bool microdescriptor: provides the microdescriptor consensus if **True**, standard consensus otherwise :param bool bridge: standard descriptors if **False**, bridge if **True** :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.router_status_entry.RouterStatusEntry` for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails rznetwork-status-consensus-3z$network-status-microdesc-consensus-3znetwork-status-2zbridge-network-statusz7Only v3 microdescriptors are available (not version %s)zCOnly v2 and v3 router status entries are available (not version %s)rN)rDrrL) r;rrrr#r$r%rrrrrrs r r"zCollecTor.get_consensuss4!|OF.i A/&8i Aof$i )i W\RU\\]]^ahhii ZZ 5# .&&9eS:JV]ip&q$ sBB c #|K|jd||D]"}|j|d||||D]}|$yw)a Directory authority key certificates for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.networkstatus.KeyCertificate for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zdir-key-certificate-3rNrrs r r'zCollecTor.get_key_certificates$sQ(ZZ/ <&&#:E3RYel&m$ rc #|K|jd||D]"}|j|d||||D]}|$yw)a Bandwidth authority heuristics for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.bandwidth_file.BandwidthFile for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails zbandwidth-filerNrrs r r)zCollecTor.get_bandwidth_files<sP(ZZ(% 5&&#3UC7^e&f$ rc #|K|jd||D]"}|j|d||||D]}|$yw)a `TorDNSEL exit lists `_ for the given time range, sorted oldest to newest. :param datetime.datetime start: publication time to begin with :param datetime.datetime end: publication time to end with :param str cache_to: directory to cache archives into, if an archive is available here it is not downloaded :param int timeout: timeout for downloading each individual archive when the connection becomes idle, no timeout applied if **None** :param int retries: maximum attempts to impose on a per-archive basis :returns: **iterator** of :class:`~stem.descriptor.tordnsel.TorDNSEL for the given time range :raises: :class:`~stem.DownloadFailed` if the download fails tordnselrNrrs r r+zCollecTor.get_exit_listsTsN(ZZ E3 /&&:ucWX_&`$ rc|jr)tj|jz tk\r@|dk(rTtj tj tjtjfD]}|js|}nn|tj}|tjk7r |jnd}tdz|z}|jtjjj!||j"|j$}t'j(tjj*j-||_tj|_|jS)a Provides the archives available in CollecTor. :param descriptor.Compression compression: compression type to download from, if undefiled we'll use the best decompression available :returns: **dict** with the archive contents :raises: If unable to retrieve the index this provide... * **ValueError** if json is malformed * **IOError** if unable to decompress * :class:`~stem.DownloadFailed` if the download fails bestzindex/index.json)rtimerREFRESH_INDEX_RATErr{r|r}rb availablerrprqrPrnrorOrrjsonloads str_tools _to_unicode)r;r3optionrurlrxs r indexzCollecTor.indexls"   t/D/D!DHZ!Z  "''+:J:JKLaLab F    K    !++ +6+:O:O+O+''UWi . . :c'' (<(<(E(Ec4<|jr |jStS)N)rFUTURE)xs r z!CollecTor.files..sabahahVWV]V]ntr )key)rrrrsortedr _filesrrranyr1rSappend)r;rXrrmatchesrrs r rzCollecTor.filess$   t/D/D!DHZ!Z!)"2"24::<"DLtudG    AEEMQUUU] AGGOqww}  C`a`g`g(hS\)=)=o)N(h$iq N)is D c>t|tsgSg}|jD]\}}|dk(r|D]}dj||j dgz}|j t ||j d|j d|j d|j d|j d|j d |d k(s|D]:}|jtj|||j dgz<|S) z Recursively provies files within the index. :param dict val: index hash :param list path: path we've transversed into :returns: **list** of :class:`~stem.descriptor.collector.File` rr]r/r1r4r5r<r=r8 directories) isinstancedictitemsrFgetrr-extendr r)valr/rkvattr file_paths r rzCollecTor._filessD c4 i E J1 g uDhhttxx'7&889) ,,tItxx'8$((6:JDHHU]L^`d`h`hiz`{~B~F~FGW~XZ^ZbZbcrZst u u  JD ,,y''ddhhv6F5G.GH I J J Lr )rNNNNFNrNNNNr)r)NNN)rrrrr>rrrrrr"r'r)r+rrrrr r r r r s66 D#'dtXgXoXo{|QVafrvBC*X000!F Dr r rr)$rrirgr6rkrrGrerMrJrstem.descriptorrPstem.util.connectionstem.util.str_toolsrrrprrcompilerrrr rrrrr"r'r)r+objectr-r r r r rs-^  83  BJJ. / 2::C D   4A &  dtP_PgPgstINY^jnz{]6]@}}r