Merge remote-tracking branch 'upstream/master' into jam

Author: marcothephoenix65

CHANGELOG.md

@@ -578,6 +578,18 @@ Please see [git log](https://github.com/python/typeshed/commits/main?after=f8f07
 - Allow dangerous identity comparisons to `Any` typed variables (Shantanu, PR [21142](https://github.com/python/mypy/pull/21142))
 - `--warn-unused-config` should not be a strict flag (Ivan Levkivskyi, PR [21139](https://github.com/python/mypy/pull/21139))
 
+### Mypy 1.20.2
+
+- Use WAL with SQLite cache and fix close (Shantanu, PR [21154](https://github.com/python/mypy/pull/21154))
+- Adjust SQLite journal mode (Ivan Levkivskyi, PR [21217](https://github.com/python/mypy/pull/21217))
+- Correctly aggregate narrowing information on parent expressions (Shantanu, PR [21206](https://github.com/python/mypy/pull/21206))
+- Fix regression related to generic callables (Shantanu, PR [21208](https://github.com/python/mypy/pull/21208))
+- Fix regression by avoiding widening types in some contexts (Shantanu, PR [21242](https://github.com/python/mypy/pull/21242))
+- Fix slicing in non-strict optional mode (Shantanu, PR [21282](https://github.com/python/mypy/pull/21282))
+- mypyc: Fix match statement semantics for "or" pattern (Shantanu, PR [21156](https://github.com/python/mypy/pull/21156))
+- mypyc: Fix issue with module dunder attributes (Piotr Sawicki, PR [21275](https://github.com/python/mypy/pull/21275))
+- Initial support for Python 3.15.0a8 (Marc Mueller, PR [21255](https://github.com/python/mypy/pull/21255))
+
 ### Acknowledgements
 
 Thanks to all mypy contributors who contributed to this release:

docs/source/command_line.rst

@@ -595,7 +595,7 @@ of the above sections.
     This flag causes mypy to suppress errors caused by not being able to fully
     infer the types of global and class variables.
 
-.. option:: --allow-redefinition-new
+.. option:: --allow-redefinition
 
     By default, mypy won't allow a variable to be redefined with an
     unrelated type. This flag enables the redefinition of *unannotated*
@@ -613,7 +613,7 @@ of the above sections.
             # Type of "x" is "int | str" here.
             return x
 
-    Without the new flag, mypy only supports inferring optional types
+    Without this flag, mypy only supports inferring optional types
     (``X | None``) from multiple assignments. With this option enabled,
     mypy can infer arbitrary union types.
 
@@ -646,24 +646,21 @@ of the above sections.
     Note: We are planning to turn this flag on by default in a future mypy
     release.
 
-.. option:: --allow-redefinition
+.. option:: --allow-redefinition-new
 
-    This is an alias to :option:`--allow-redefinition-old <mypy --allow-redefinition-old>`.
-    In mypy v2.0 this will point to
-    :option:`--allow-redefinition-new <mypy --allow-redefinition-new>`, and will
-    eventually became the default.
+    Deprecated alias for :option:`--allow-redefinition <mypy --allow-redefinition>`.
 
 .. option:: --allow-redefinition-old
 
-    This is an older variant of
-    :option:`--allow-redefinition-new <mypy --allow-redefinition-new>`.
+    This is an older, more limited variant of
+    :option:`--allow-redefinition <mypy --allow-redefinition>`.
     This flag enables redefinition of a variable with an
     arbitrary type *in some contexts*: only redefinitions within the
     same block and nesting depth as the original definition are allowed.
 
-    We have no plans to remove this flag, but we expect that
-    :option:`--allow-redefinition-new <mypy --allow-redefinition-new>`
-    will replace this flag for new use cases eventually.
+    We have no plans to remove this flag, but
+    :option:`--allow-redefinition <mypy --allow-redefinition>`
+    is recommended for new use cases.
 
     Example where this can be useful:
 

docs/source/common_issues.rst

@@ -232,8 +232,7 @@ improved performance. You can ensure the presence of orjson using the ``faster-c
 
     python3 -m pip install -U mypy[faster-cache]
 
-Mypy may depend on orjson by default in the future. To use faster, native parser, use the
-``native-parse`` extra. Native parser will be default in near future.
+Mypy may depend on orjson by default in the future.
 
 Types of empty collections
 --------------------------

docs/source/config_file.rst

@@ -713,7 +713,7 @@ section of the command line docs.
     Causes mypy to suppress errors caused by not being able to fully
     infer the types of global and class variables.
 
-.. confval:: allow_redefinition_new
+.. confval:: allow_redefinition
 
     :type: boolean
     :default: False
@@ -763,6 +763,13 @@ section of the command line docs.
     Note: We are planning to turn this flag on by default in a future mypy
     release.
 
+.. confval:: allow_redefinition_new
+
+    :type: boolean
+    :default: False
+
+    Deprecated alias for :confval:`allow_redefinition`.
+
 .. confval:: allow_redefinition_old
 
     :type: boolean
@@ -789,14 +796,6 @@ section of the command line docs.
            items = "100"  # valid, items now has type str
            items = int(items)  # valid, items now has type int
 
-.. confval:: allow_redefinition
-
-    :type: boolean
-    :default: False
-
-    An alias to :confval:`allow_redefinition_old`, in mypy v2.0 this will point to
-    :confval:`allow_redefinition_new`, and will eventually became the default.
-
 .. confval:: local_partial_types
 
     :type: boolean

misc/apply-cache-diff.py

@@ -19,19 +19,22 @@
 from librt.internal import ReadBuffer
 
 from mypy.cache import CacheMeta
+from mypy.defaults import SQLITE_NUM_SHARDS
 from mypy.metastore import FilesystemMetadataStore, MetadataStore, SqliteMetadataStore
 from mypy.util import json_dumps, json_loads
 
 
-def make_cache(input_dir: str, sqlite: bool) -> MetadataStore:
+def make_cache(input_dir: str, sqlite: bool, num_shards: int = SQLITE_NUM_SHARDS) -> MetadataStore:
     if sqlite:
-        return SqliteMetadataStore(input_dir)
+        return SqliteMetadataStore(input_dir, num_shards=num_shards)
     else:
         return FilesystemMetadataStore(input_dir)
 
 
-def apply_diff(cache_dir: str, diff_file: str, sqlite: bool = False) -> None:
-    cache = make_cache(cache_dir, sqlite)
+def apply_diff(
+    cache_dir: str, diff_file: str, sqlite: bool = False, num_shards: int = SQLITE_NUM_SHARDS
+) -> None:
+    cache = make_cache(cache_dir, sqlite, num_shards=num_shards)
     with open(diff_file, "rb") as f:
         diff = json_loads(f.read())
 
@@ -63,11 +66,14 @@ def apply_diff(cache_dir: str, diff_file: str, sqlite: bool = False) -> None:
 def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--sqlite", action="store_true", default=False, help="Use a sqlite cache")
+    parser.add_argument(
+        "--num-shards", type=int, default=SQLITE_NUM_SHARDS, help=argparse.SUPPRESS
+    )
     parser.add_argument("cache_dir", help="Directory for the cache")
     parser.add_argument("diff", help="Cache diff file")
     args = parser.parse_args()
 
-    apply_diff(args.cache_dir, args.diff, args.sqlite)
+    apply_diff(args.cache_dir, args.diff, args.sqlite, num_shards=args.num_shards)
 
 
 if __name__ == "__main__":

misc/convert-cache.py

@@ -15,6 +15,7 @@
 
 import argparse
 
+from mypy.defaults import SQLITE_NUM_SHARDS
 from mypy.metastore import FilesystemMetadataStore, MetadataStore, SqliteMetadataStore
 
 
@@ -26,6 +27,13 @@ def main() -> None:
         default=False,
         help="Convert to a sqlite cache (default: convert from)",
     )
+    parser.add_argument(
+        "--num-shards",
+        type=int,
+        default=SQLITE_NUM_SHARDS,
+        dest="num_shards",
+        help=argparse.SUPPRESS,
+    )
     parser.add_argument(
         "--output_dir",
         action="store",
@@ -37,17 +45,23 @@ def main() -> None:
 
     input_dir = args.input_dir
     output_dir = args.output_dir or input_dir
+    num_shards = args.num_shards
     assert os.path.isdir(output_dir), f"{output_dir} is not a directory"
     if args.to_sqlite:
         input: MetadataStore = FilesystemMetadataStore(input_dir)
-        output: MetadataStore = SqliteMetadataStore(output_dir)
+        output: MetadataStore = SqliteMetadataStore(output_dir, num_shards=num_shards)
     else:
-        fnam = os.path.join(input_dir, "cache.db")
-        msg = f"{fnam} does not exist"
-        if not re.match(r"[0-9]+\.[0-9]+$", os.path.basename(input_dir)):
-            msg += f" (are you missing Python version at the end, e.g. {input_dir}/3.11)"
-        assert os.path.isfile(fnam), msg
-        input, output = SqliteMetadataStore(input_dir), FilesystemMetadataStore(output_dir)
+        if num_shards <= 1:
+            db_files = [os.path.join(input_dir, "cache.db")]
+        else:
+            db_files = [os.path.join(input_dir, f"cache.{i}.db") for i in range(num_shards)]
+        for fnam in db_files:
+            msg = f"{fnam} does not exist"
+            if not re.match(r"[0-9]+\.[0-9]+$", os.path.basename(input_dir)):
+                msg += f" (are you missing Python version at the end, e.g. {input_dir}/3.11)"
+            assert os.path.isfile(fnam), msg
+        input = SqliteMetadataStore(input_dir, num_shards=num_shards)
+        output = FilesystemMetadataStore(output_dir)
 
     for s in input.list_all():
         if s.endswith((".json", ".ff")):

misc/diff-cache.py

@@ -19,13 +19,14 @@
 from librt.internal import ReadBuffer, WriteBuffer
 
 from mypy.cache import CacheMeta, CacheMetaEx
+from mypy.defaults import SQLITE_NUM_SHARDS
 from mypy.metastore import FilesystemMetadataStore, MetadataStore, SqliteMetadataStore
 from mypy.util import json_dumps, json_loads
 
 
-def make_cache(input_dir: str, sqlite: bool) -> MetadataStore:
+def make_cache(input_dir: str, sqlite: bool, num_shards: int = SQLITE_NUM_SHARDS) -> MetadataStore:
     if sqlite:
-        return SqliteMetadataStore(input_dir)
+        return SqliteMetadataStore(input_dir, num_shards=num_shards)
     else:
         return FilesystemMetadataStore(input_dir)
 
@@ -154,13 +155,16 @@ def main() -> None:
     parser = argparse.ArgumentParser()
     parser.add_argument("--verbose", action="store_true", default=False, help="Increase verbosity")
     parser.add_argument("--sqlite", action="store_true", default=False, help="Use a sqlite cache")
+    parser.add_argument(
+        "--num-shards", type=int, default=SQLITE_NUM_SHARDS, help=argparse.SUPPRESS
+    )
     parser.add_argument("input_dir1", help="Input directory for the original cache")
     parser.add_argument("input_dir2", help="Input directory for the target cache")
     parser.add_argument("output", help="Output file with the diff from original cache")
     args = parser.parse_args()
 
-    cache1 = make_cache(args.input_dir1, args.sqlite)
-    cache2 = make_cache(args.input_dir2, args.sqlite)
+    cache1 = make_cache(args.input_dir1, args.sqlite, num_shards=args.num_shards)
+    cache2 = make_cache(args.input_dir2, args.sqlite, num_shards=args.num_shards)
 
     type_misses: dict[str, int] = defaultdict(int)
     type_hits: dict[str, int] = defaultdict(int)

mypy-requirements.txt

@@ -6,3 +6,4 @@ mypy_extensions>=1.0.0
 pathspec>=1.0.0
 tomli>=1.1.0; python_version<'3.11'
 librt>=0.9.0; platform_python_implementation != 'PyPy'
+ast-serialize>=0.2.2,<1.0.0

mypy/__main__.py

@@ -6,6 +6,7 @@
 import sys
 import traceback
 
+from mypy.fscache import FileSystemCache
 from mypy.main import main, process_options
 from mypy.util import FancyFormatter
 
@@ -22,7 +23,8 @@ def console_entry() -> None:
         os.dup2(devnull, sys.stdout.fileno())
         sys.exit(2)
     except KeyboardInterrupt:
-        _, options = process_options(args=sys.argv[1:])
+        # Setting fscache prevents bogus errors when --package-root is set.
+        _, options = process_options(args=sys.argv[1:], fscache=FileSystemCache())
         if options.show_traceback:
             sys.stdout.write(traceback.format_exc())
         formatter = FancyFormatter(sys.stdout, sys.stderr, False)
@@ -36,7 +38,7 @@ def console_entry() -> None:
         try:
             import mypy.errors
 
-            _, options = process_options(args=sys.argv[1:])
+            _, options = process_options(args=sys.argv[1:], fscache=FileSystemCache())
             mypy.errors.report_internal_error(e, None, 0, None, options)
         except Exception:
             pass

mypy/binder.py

@@ -3,7 +3,7 @@
 from collections import defaultdict
 from collections.abc import Iterator
 from contextlib import contextmanager
-from typing import Literal, NamedTuple, TypeAlias as _TypeAlias
+from typing import Final, Literal, TypeAlias as _TypeAlias
 
 from mypy.erasetype import remove_instance_last_known_values
 from mypy.literals import Key, extract_var_from_literal_hash, literal, literal_hash, subkeys
@@ -42,9 +42,10 @@
 BindableExpression: _TypeAlias = IndexExpr | MemberExpr | NameExpr
 
 
-class CurrentType(NamedTuple):
-    type: Type
-    from_assignment: bool
+class CurrentType:
+    def __init__(self, type: Type, from_assignment: bool) -> None:
+        self.type: Final = type
+        self.from_assignment: Final = from_assignment
 
 
 class Frame:
@@ -199,8 +200,8 @@ def __init__(self, options: Options) -> None:
 
         # If True, initial assignment to a simple variable (e.g. "x", but not "x.y")
         # is added to the binder. This allows more precise narrowing and more
-        # flexible inference of variable types (--allow-redefinition-new).
-        self.bind_all = options.allow_redefinition_new
+        # flexible inference of variable types (--allow-redefinition).
+        self.bind_all = options.allow_redefinition
 
         # This tracks any externally visible changes in binder to invalidate
         # expression caches when needed.
@@ -339,7 +340,7 @@ def update_from_options(self, frames: list[Frame]) -> bool:
                 continue
 
             # Remove exact duplicates to save pointless work later, this is
-            # a micro-optimization for --allow-redefinition-new.
+            # a micro-optimization for --allow-redefinition.
             seen_types = set()
             resulting_types = []
             for rv in resulting_values: