This is over twice as fast as the Python dirs code. Upcoming changes
will nearly double its speed again.
perfdirs results for a working dir with 170,000 files:
Python 638 msec
C 244
This encapsulates the "multiset of directories" structures that are
currently open-coded (and duplicated) in both the dirstate and
context modules.
This will be used, and optionally replaced by a C implementation,
in upcoming changes.
This can happen when a file with flags is removed or deleted in the working
directory and also not present in m2. The obvious solution is to add a
__delitem__ override to manifestdict that removes the file from flags if
necessary, but that has a significant performance cost in some cases, e.g.
hg status --rev rev1 --rev rev2 <file>.
This is not only a bit faster, but also aligns with callers' expectations
better since we can legitimately have manifestdict's _flags set to '' instead
of unset.
hg perfmergecalculate -r .
before: ! wall 0.139582 comb 0.140000 user 0.140000 sys 0.000000 (best of 59)
after: ! wall 0.126154 comb 0.120000 user 0.120000 sys 0.000000 (best of 74)
hg perfmergecalculate -r .^
before: ! wall 0.236333 comb 0.240000 user 0.240000 sys 0.000000 (best of 36)
after: ! wall 0.212265 comb 0.210000 user 0.210000 sys 0.000000 (best of 45)
Before this patch, the certificate of the SMTP server for STARTTLS or
SMTPS isn't verified.
This may cause man-in-the-middle security problem (stealing
authentication information), even though SMTP channel itself is
encrypted by SSL.
When "[smtp] tls" is configured as "smtps" or "starttls", this patch:
- uses classes introduced by preceding patches instead of "SMTP" or
"SMTP_SSL" of smtplib, and
- verifies the certificate of the SMTP server, if "[smtp]
verifycert" is configured as other than False
"[smtp] verifycert" can be configured in 3 levels:
- "strict":
This verifies peer certificate, and aborts if:
- peer certification is not valid, or
- no configuration in "[hostfingerprints]" and "[web] cacerts"
This is default value of "[smtp] verifycert" for security.
- "loose":
This verifies peer certificate, and aborts if peer certification is
not valid.
This just shows warning message ("certificate not verified"), if
there is no configuration in "[hostfingerprints]" and "[web]
cacerts".
This is as same as verification for HTTPS connection.
- False(no verification):
Peer certificate is not verified.
This is as same as the behavior before this patch series.
"hg email --insecure" uses "loose" level, and ignores "[web] cacerts"
as same as push/pull/etc... with --insecure.
Ignoring "[web] cacerts" configuration for "hg email --insecure" is
already done in "dispatch._dispatch()" by looking "insecure" up in the
table of command options.
Before this patch, "sslutil.validator" may returns successfully, even
if peer certificate is not verified because there is no information in
"[hostfingerprints]" and "[web] cacerts".
To prevent from sending authentication credential to untrustable SMTP
server, validation should be aborted if peer certificate is not
verified.
This patch introduces "strict" optional argument, and
"sslutil.validator" will abort if it is True and peer certificate is
not verified.
Original "smtplib.SMTP_SSL" has no route to pass "ca_certs" and
"cert_reqs" arguments to underlying SSL socket creation. This causes
that "getpeercert()" on SSL socket returns empty dict, so the peer
certificate for SMTPS can't be verified.
This patch introduces the "SMTPS" class derived from "smtplib.SMTP" to
pass "ca_certs" and "cert_reqs" arguments to underlying SSL socket
creation.
"SMTPS" class is derived directly from "smtplib.SMTP", because amount
of "smtplib.SMTP_SSL" definition derived from "smtplib.SMTP" is as
same as one needed to override it.
This patch defines "SMTPS" class, only when "smtplib.SMTP" class has
"_get_socket()" method, because this makes using SSL socket instead of
normal socket easy.
"smtplib.SMTP" class of Python 2.5.x or earlier doesn't have this
method. Omitting SMTPS support for them is reasonable, because
"smtplib.SMTP_SSL" is already unavailable for them before this patch.
Almost all code of "SMTPS" class is imported from "smtplib.SMTP_SSL"
of Python 2.7.3, but it differs from original code in point below:
- "ssl.wrap_socket()" is replaced by "sslutil.ssl_wrap_socket()" for
compatibility between Python versions
Original "smtplib.SMTP" has no route to pass "ca_certs" and
"cert_reqs" arguments to underlying SSL socket creation. This causes
that "getpeercert()" on SSL socket returns empty dict, so the peer
certificate for STARTTLS can't be verified.
This patch introduces the "STARTTLS" class derived from "smtplib.SMTP"
to pass "ca_certs" and "cert_reqs" arguments to underlying SSL socket
creation.
Almost all code of "starttls()" in this class is imported from
"smtplib.SMTP" of Python 2.7.3, but it differs from original code in
points below:
- "self.ehlo_or_helo_if_needed()" invocation is omitted, because:
- "ehlo_or_helo_if_needed()" is available with Python 2.6 or later, and
- "ehlo()" is explicitly invoked in "mercurial.mail._smtp()"
- "if not _have_ssl:" check is omitted, because:
- "_have_ssl" is available with Python 2.6 or later, and
- same checking is done in "mercurial.sslutil.ssl_wrap_socket()"
- "ssl.wrap_socket()" is replaced by "sslutil.ssl_wrap_socket()" for
compatibility between Python versions
- use "sock.recv()" also as "sock.read()", if "sock" doesn't have
"read()" method
with Python 2.5.x or earlier, "sslutil.ssl_wrap_socket()" returns
"httplib.FakeSocket"-ed object, and it doesn't have "read()"
method, which is invoked via "smtplib.SSLFakeFile".
When the revisions to graft are numerically close to the destination, this
avoids two walks up the DAG, which for a repository with over 470,000
changesets translates to around 2.2 seconds.
When the revisions to graft are numerically close to the destination, this
avoids one walk up the DAG, which for a repository with over 470,000
changesets translates to around 1.1 seconds.
This is a performance win for a number of reasons:
- We don't iterate over contexts, which avoids a completely unnecessary sorted
call + the O(number of files) abstraction cost of doing that.
- We don't check membership in a context, which avoids another
O(number of files) abstraction cost.
- We iterate over the manifests in C instead of Python.
For a large repo with 170,000 files, this improves perfpathcopies from 0.34
seconds to 0.07. Anything that uses pathcopies, such as rebase or diff --git
between two revisions, benefits.
This makes a big difference to performance in some cases.
hg --time locate 'set:symlink()'
mozilla-central (70,000 files):
before: 2.92 sec
after: 2.47
another repo (170,000 files):
before: 7.87 sec
after: 6.86
Now that we no longer sort all the walk results, using iteritems becomes
possible.
This is a relatively minor speedup: on a large repository with 170,000 files,
perfaddremove goes from 2.13 seconds to 2.10.
The only place where the order matters is in printing out added or removed
files. We already sort that set.
On a large repository with 170,000 files, this speeds up perfaddremove from
2.34 seconds to 2.13.
This will let us stop sorting all the results in an upcoming patch.
This also permits future refactorings where the same code consumes
dirstate.walk results but doesn't print anything out.
An argument could be made that printing out results as we go along is more
responsive UI-wise. However, at this point iterating through walk results is
actually faster than sorting them, so once we stop sorting all the results the
argument ceases to be valid.
dirstate.walk only does lstats and never returns stat objects for directories.
On a large repository with 170,000 files, this speeds perfaddremove up from
2.40 seconds to 2.34.
